Archive for the ‘Technology’ Category

« Older Entries

Speaking OFX Using Go

Thursday, May 11th, 2017

I’ve been working on a web-based mint.com/Gnucash/Quicken clone I’m calling ‘moneygo‘ off and on in my spare time. One important feature of these types of programs is the ability to automatically import transaction and balance information from a user’s financial institution. In the US, this functionality is primarily supported by banks via the OFX specification.

I spent some time exploring my options and was unable to find an OFX library written in Golang capable of not only parsing OFX responses, but generating the request files as well. Generating the request file is important because otherwise the user must obtain the OFX file out-of-band (i.e. most likely logging into their bank and downloading it with a web browser). It is much more convenient for your financial management software to do this automatically for you. I actually ended up briefly using libofx before deciding that the C-to-Go interface was too awkward to make it a reasonable solution – converting complex types from C to Go is not fun.

So, I did what any self-respecting programmer would do – downloaded the OFX spec and wrote my own OFX generator and parser in Go: ofxgo. This process was somewhat tedious at times(there are a lot of OFX elements, particularly for investment transactions), and I didn’t implement nearly all of the spec. However, the result is a library that I’ve been able to use to download bank, credit card, and investment transactions and balances for all of my financial institutions! Along the way, I also created a sample Golang command-line client which can be used to test the library with your bank, download and/or parse OFX, or even to help detect the correct settings to use with your bank.

I hope my ofxgo library will enable others to more easily access their financial data using OFX, and am looking forward to getting it fully integrated with moneygo. Don’t hesitate to let me know if you find bugs or would like to see improvements in this library by creating an issue or pull request on github.

Tags: , , , ,
Posted in Technology | No Comments »

Installing Seafile Client/Server on Arch Linux

Monday, January 21st, 2013

I have previously hacked around some on my own open source Dropbox clone, but I haven’t had the time lately. I came across a project called Seafile, which looks promising as an open-source Dropbox alternative. Unfortunately, they don’t package their installation for Arch Linux, so I had to get my hands a little dirty to try it out. I created four AUR packages for seafile and its dependencies:

For the curious, my installation instructions and packages are based on the directions at https://github.com/haiwen/seafile/wiki/Build-and-deploy-seafile-server-from-source and https://github.com/haiwen/seafile/wiki/Build-and-use-seafile-client-from-source. In these instructions, I assume you are using Arch Linux, and that you are familiar with how to use makepkg, pacman & friends to build and install packages from the AUR.

Build/install the Seafile package

You must install the seafile package above, along with all its dependencies. The easiest way to do this is using the AUR helper of your choice, but you can also do this manually if you wish. For most of the AUR helpers, this will look something like

pacaur -S seafile

or

yaourt -S seafile

Install the Seafile server

If you simply want to use the seafile client with seafile.com and are not interested in hosting your own server, skip to the next section. Otherwise, read on to install the seafile server on your machine.

Now that the seafile package and all its dependencies are installed, create a directory where you wish to serve the seafile files out of. This should be somewhere with plenty of space, as all the files served by your seafile installation will ultimately live here. This directory should be owned by the user you want your seafile server to run as. In fact, you should su into that user and run the rest of these commands from there. I’ll refer to the directory you just created as $SEABASE from here on out.

Download seahub, the web server portion of the seafile server.

mkdir -p $SEABASE/seafile-server
cd $SEABASE/seafile-server
wget http://seafile.googlecode.com/files/seahub-latest.tar.gz
tar xf seahub-latest.tar.gz
mv seahub-${version} seahub
cd ..

Run the setup script to generate all the configuration files:

seafile-admin setup

Fill out the requested fields following the provided directions. You may now start/stop the seafile server using the following commands:

seafile-admin start
seafile-admin stop

Note that the seafile-admin commands must be run from the $SEABASE directory.

For more information, consult the Seafile wiki page for installing from source (but remember that the top part of the page won’t apply to you since your AUR helper has downloaded, built, and installed all necessary packages for you).

Using the Seafile client

At this point, the Seafile client shouldn’t need further installation. Running

seafile-applet

should start the client, presenting a configuration screen on the first time it is run.

 

Enjoy! And, of course, let me know if you encounter any problems.

 

Update (2013.03.08): Updated seahub-latest.tar.gz download URL.

Tags: , , , , ,
Posted in Technology | 12 Comments »

How To Fix A Non-Booting Thinkpad From 5342 Miles Away

Wednesday, July 20th, 2011

For reference, 5342 miles is the distance it would take sound approximately 7 hours to travel, or 1.4 times the length of the Amazon River. Of course, distance is somewhat irrelevant with the Internet, but it sounds more impressive that way.

So, on to how it all started. I got a message saying “hi, something is wrong with my computer”. We’re okay so far – that’s a fairly regular occurrence in the life of someone in a technology-related field. Then she told me it wouldn’t go past the very first screen. After a few questions, I figured out she meant the BIOS boot screen, where it does the POST. She had already rebooted several times to no avail. So, the next course of action was to get to the CMOS/BIOS settings and see if we could reset them. After some trial and error, she reported that she couldn’t get that to come up (being a Thinkpad, all you should have to do is press the ThinkVantage button). It was starting to look really bad. At this point, I just about gave up and decided that her motherboard or some other crucial hardware component was failing and that she needed a new laptop.

Fortunately, I turned to Google instead. With a bit of Google-fu and following a few links, I found this somewhat sketchy page (the link is now dead), which explains in broken English that a broken USB port can cause a Thinkpad T400 not to boot. I’m not entirely sure why I didn’t click past that page immediately. Perhaps I looked into it because I remembered her mentioning using a USB drive earlier in the day. Maybe it was because at this point I had gone through just about everything else I knew to do to debug the problem. At any rate, I continued our game of 20 questions by asking her if any of her USB ports looked broken, or like they had pins that were bent in a way so that they were touching each other. It took us a bit of back-and-forth to iron out which port was the USB port she had plugged the USB drive into earlier in the day, and to identify any other USB ports which may be the damaged culprit.

She reported that she had one USB port, and another port beside it that wasn’t quite the same. After some more questions back and forth, we established that the other port was indeed a USB port, but that it was missing the plastic middle piece, much like what the sketchy link described. After she managed to take and send me a picture over her fairly slow Internet connection with the friend’s laptop she was using, I began to hope that there might actually be something to this. The pins in the second USB port were all bent over, and were almost certainly shorting out when she tried to boot it.

Broken USB

USB Port Missing Plastic Pin Support (On Right)

She began attempting to straighten the pins with tweezers and a sewing needle. After she managed to get the pins straight enough so they weren’t touching each other or the grounded sides of the port, she tried to boot the laptop again, and it worked! I am quite proud of the fact that we managed to debug and fix a seemingly defunct laptop from opposite sides of the equator, using only instant messages and one picture.

Tags: , , , , , ,
Posted in Personal, Technology | 2 Comments »

Dropbox Download Statistics/Graph

Tuesday, July 12th, 2011

Recently I added one of my machines to my Dropbox account. I installed the CLI version on an Ubuntu machine (using this script). While doing so, I noticed I could get statistics about the process as it was happening via the command `./dropbox.py status’, and naturally decided I should gather them and make pretty graphs to appease my curiosity. (Note: I’m particularly curious because I’ve been working on my own Dropbox clone as of late – Asink)

So, I started a fresh download of my Dropbox folder (containing 10691 files weighing in at 616 MB). The complete download took me right at an hour and a half (5500 seconds) to complete. During this time period, I polled dropbox’s `status` command once a second until the download was done. Each second I gathered the total number of files Dropbox reported it still had to download, the download speed it estimated, and the amount of time it estimated it would take to complete the download. I might add that part of what piqued my interest in this was that for a while Dropbox was telling me it would take nearly 100 days to completely download a mere 616 MB of files.

And so, without further adieu, the graph. I apologize for a) the somewhat awkward units – I wanted to overlay all the information on one graph so I normalized the units to make that work, and b) for the gap in the middle of the data – my statistic-gathering script decided to die on me and I didn’t notice for a few minutes:

Dropbox Download Statistics

Dropbox Download Statistics

Several things struck me as interesting about this graph. First is the fact that the download speed appears to begin high, drop off, and then pick up drastically near the end. I find this mildly confusing, and I think it could be caused by one of two things: a) Dropbox throttling download speeds when it detects you’re downloading a lot (hence the faster speeds near the beginning and end) or b) the speed is miscalculated by the Dropbox daemon, and is a bit off as you approach either of the extremes. Some may claim that this is my ISP throttling traffic, which *would* explain the dropoff after the initial burst, but fails to explain the drastic pickup at the end.

Second, Dropbox’s time-to-completion estimation is about as horrid as that of progress bars in Windows. Even though the number of files remaining to be downloaded is more-or-less linear, the time remaining estimation varies everywhere from 100 days (it caps out at 100 days *24*4 = 9600 quarter-hours at the beginning and again at ~2000 seconds). This estimation is also surprisingly bad given the relatively constant estimated download speed. One would think that you could come up with some relatively-easy calculation which would account for small variations in connection speed, and arrive at a much more accurate estimation than what they have.

Well, there you have it: a highly-unscientific and mildly interesting graph of my Dropbox downloads.

Tags: , ,
Posted in Technology | No Comments »

The Sorry State of Open-Source Dropbox Clones

Wednesday, June 22nd, 2011

For starters, I use and love Dropbox. It is one of, if not THE, best file synchronization tool I have used or know of. I’m going to make the claim that no open-source Dropbox clone comes close to its functionality at the moment. But first, here is a list of the requirements for my personal file synchronization utility which Dropbox satisfies:

  1. a) It must keep a version-controlled copy of all of the files I am synchronizing. b) I should be able to easily revert to an earlier version of a file if I wish, independently of the state of the version of the rest of my files.
  2. It must work automatically and behind-the-scenes to keep my files in sync. I don’t want to have to think about synchronization or backups -that should be handled for me.
  3. File synchronization should begin as soon as I save the file locally. If I update a file I’m sharing with a friend, I want them to get the new version as soon as possible, which brings me to…
  4. Sharing files with others should be easy and intuitive.
  5. It should have an easy-to-use API which makes it easy for others to build applications around.
  6. My files must be accessible on my local machine, even if I am disconnected from the Internet.

With all of the above compliments, why would I want to replace Dropbox? Well, even Dropbox isn’t perfect, and I have a few additional requirements my ideal personal synchronization tool would meet:

  1. A high level of privacy and security. Although Dropbox claims to keep your files private and secure, there have been a few cases where that seems to be wearing thin. Many people are searching for a replacement. It seems to me that encrypting files locally before transmission (and not storing the private key anywhere except locally) is a good solution.
  2. I should be able to control where my files are stored. This probably doesn’t matter for your average user, but as a superuser, I want control. I.e., I should be able to setup my own server and store my files there if I want. This also means my available storage space is only limited by the size of my hard drive. Although related with privacy and security, I think this warrants a separate mention.
  3. The ability to share a public link with people who don’t have Dropbox accounts. This would make file sharing much easier. (A shout out to Spence for this idea)
  4. A proper access control facility. This would probably ideally mirror Unix-like ACLs.
  5. I should be able to revert all my files to a certain point in time. This is not (I don’t think) currently an option with Dropbox.
  6. Finally, it should be open source for the usual reasons.

Existing “Solutions”

Looking at the set of above requirements, it becomes clear that a) I am very picky, and b) there is no existing solution which can satisfy all of my requirements. There are a great number of open-source Dropbox clones out there. Below is a partial list of what I see as the major competitors in this space, along with their shortcomings on the above items. Feel free to leave a comment if I have missed one, or made false claims about one I have listed.

  • SparkleShare – It’s based on git. Git is fantastic, but not for all the image and music files I want to sync. You’re limited to storing your files in a git repository.
  • ownCloud – No client synchronization tool. They tell  you to use use WebDAV (see below).
  • Syncany – Doesn’t support sharing files with others.
  • duplicity – Only synchronizes when you tell it to.
  • NFS and WebDAV – No access to files when you are offline. This is a must.

What’s Next?

After searching for a synchronization client which meets my requirements and failing, I did what every self-respecting techno-geek would do, and began writing my own. I know I will catch some grief for starting yet another project and not contributing to an existing one, but I believe that the above projects all have core design limitations which will inhibit them from meeting all of the above without changing key components of their infrastructure.

I am aiming to eventually hit all of the above requirements. In the first pass, I am aiming to meet Dropbox requirements numbers 1.a., 2, 3, 6 and non-Dropbox requirements 2 and 6. These are, in my opinion, the subset of the above requirements that is in the middle of the Venn Diagram containing easy-to-implement features on the one side and important features on the other.

For the moment, I am calling my project “Asink” for both the asynchronous method of file synchronization, the fact that it aims to be a sink for all of my data, and that I should be able to store everything and the kitchen sink in it. You can follow my development on github at http://github.com/aclindsa/asink. I am planning to write another post to explain my initial design decisions, as well as an update as soon as I have a version with the basic functionality properly working (hopefully within the next week).

Feel free to let me know what you think in the comments.

Tags: , , , , ,
Posted in Technology | 10 Comments »

« Older Entries