Dropbox Download Statistics/Graph

July 12th, 2011

Recently I added one of my machines to my Dropbox account. I installed the CLI version on an Ubuntu machine (using this script). While doing so, I noticed I could get statistics about the process as it was happening via the command `./dropbox.py status’, and naturally decided I should gather them and make pretty graphs to appease my curiosity. (Note: I’m particularly curious because I’ve been working on my own Dropbox clone as of late – Asink)

So, I started a fresh download of my Dropbox folder (containing 10691 files weighing in at 616 MB). The complete download took me right at an hour and a half (5500 seconds) to complete. During this time period, I polled dropbox’s `status` command once a second until the download was done. Each second I gathered the total number of files Dropbox reported it still had to download, the download speed it estimated, and the amount of time it estimated it would take to complete the download. I might add that part of what piqued my interest in this was that for a while Dropbox was telling me it would take nearly 100 days to completely download a mere 616 MB of files.

And so, without further adieu, the graph. I apologize for a) the somewhat awkward units – I wanted to overlay all the information on one graph so I normalized the units to make that work, and b) for the gap in the middle of the data – my statistic-gathering script decided to die on me and I didn’t notice for a few minutes:

Dropbox Download Statistics

Dropbox Download Statistics

Several things struck me as interesting about this graph. First is the fact that the download speed appears to begin high, drop off, and then pick up drastically near the end. I find this mildly confusing, and I think it could be caused by one of two things: a) Dropbox throttling download speeds when it detects you’re downloading a lot (hence the faster speeds near the beginning and end) or b) the speed is miscalculated by the Dropbox daemon, and is a bit off as you approach either of the extremes. Some may claim that this is my ISP throttling traffic, which *would* explain the dropoff after the initial burst, but fails to explain the drastic pickup at the end.

Second, Dropbox’s time-to-completion estimation is about as horrid as that of progress bars in Windows. Even though the number of files remaining to be downloaded is more-or-less linear, the time remaining estimation varies everywhere from 100 days (it caps out at 100 days *24*4 = 9600 quarter-hours at the beginning and again at ~2000 seconds). This estimation is also surprisingly bad given the relatively constant estimated download speed. One would think that you could come up with some relatively-easy calculation which would account for small variations in connection speed, and arrive at a much more accurate estimation than what they have.

Well, there you have it: a highly-unscientific and mildly interesting graph of my Dropbox downloads.

The Sorry State of Open-Source Dropbox Clones

June 22nd, 2011

For starters, I use and love Dropbox. It is one of, if not THE, best file synchronization tool I have used or know of. I’m going to make the claim that no open-source Dropbox clone comes close to its functionality at the moment. But first, here is a list of the requirements for my personal file synchronization utility which Dropbox satisfies:

  1. a) It must keep a version-controlled copy of all of the files I am synchronizing. b) I should be able to easily revert to an earlier version of a file if I wish, independently of the state of the version of the rest of my files.
  2. It must work automatically and behind-the-scenes to keep my files in sync. I don’t want to have to think about synchronization or backups -that should be handled for me.
  3. File synchronization should begin as soon as I save the file locally. If I update a file I’m sharing with a friend, I want them to get the new version as soon as possible, which brings me to…
  4. Sharing files with others should be easy and intuitive.
  5. It should have an easy-to-use API which makes it easy for others to build applications around.
  6. My files must be accessible on my local machine, even if I am disconnected from the Internet.

With all of the above compliments, why would I want to replace Dropbox? Well, even Dropbox isn’t perfect, and I have a few additional requirements my ideal personal synchronization tool would meet:

  1. A high level of privacy and security. Although Dropbox claims to keep your files private and secure, there have been a few cases where that seems to be wearing thin. Many people are searching for a replacement. It seems to me that encrypting files locally before transmission (and not storing the private key anywhere except locally) is a good solution.
  2. I should be able to control where my files are stored. This probably doesn’t matter for your average user, but as a superuser, I want control. I.e., I should be able to setup my own server and store my files there if I want. This also means my available storage space is only limited by the size of my hard drive. Although related with privacy and security, I think this warrants a separate mention.
  3. The ability to share a public link with people who don’t have Dropbox accounts. This would make file sharing much easier. (A shout out to Spence for this idea)
  4. A proper access control facility. This would probably ideally mirror Unix-like ACLs.
  5. I should be able to revert all my files to a certain point in time. This is not (I don’t think) currently an option with Dropbox.
  6. Finally, it should be open source for the usual reasons.

Existing “Solutions”

Looking at the set of above requirements, it becomes clear that a) I am very picky, and b) there is no existing solution which can satisfy all of my requirements. There are a great number of open-source Dropbox clones out there. Below is a partial list of what I see as the major competitors in this space, along with their shortcomings on the above items. Feel free to leave a comment if I have missed one, or made false claims about one I have listed.

  • SparkleShare – It’s based on git. Git is fantastic, but not for all the image and music files I want to sync. You’re limited to storing your files in a git repository.
  • ownCloud – No client synchronization tool. They tell  you to use use WebDAV (see below).
  • Syncany – Doesn’t support sharing files with others.
  • duplicity – Only synchronizes when you tell it to.
  • NFS and WebDAV – No access to files when you are offline. This is a must.

What’s Next?

After searching for a synchronization client which meets my requirements and failing, I did what every self-respecting techno-geek would do, and began writing my own. I know I will catch some grief for starting yet another project and not contributing to an existing one, but I believe that the above projects all have core design limitations which will inhibit them from meeting all of the above without changing key components of their infrastructure.

I am aiming to eventually hit all of the above requirements. In the first pass, I am aiming to meet Dropbox requirements numbers 1.a., 2, 3, 6 and non-Dropbox requirements 2 and 6. These are, in my opinion, the subset of the above requirements that is in the middle of the Venn Diagram containing easy-to-implement features on the one side and important features on the other.

For the moment, I am calling my project “Asink” for both the asynchronous method of file synchronization, the fact that it aims to be a sink for all of my data, and that I should be able to store everything and the kitchen sink in it. You can follow my development on github at http://github.com/aclindsa/asink. I am planning to write another post to explain my initial design decisions, as well as an update as soon as I have a version with the basic functionality properly working (hopefully within the next week).

Feel free to let me know what you think in the comments.

Installing dwm and Killing gdm on Ubuntu

April 29th, 2011

Lately I’ve been playing around with a few non-traditional window managers; particularly wmii and dwm, both of which are minimalist dynamic window managers from suckless.org. With both of them, you primarily use the keyboard to open programs and manage your running windows. The two have slightly different ways of doing things, but are quite similar (as you would expect coming from the same project). Being ‘dynamic’, they both automatically handle window placement when you open a new window, although they allow you to change this placement if you want.

So, why did I decide to use them? Well, a lot of it has been motivated by my recent foray into Arch Linux combined with a desire to use my mouse less (and maybe delay arthritis in my wrist for a few years at least). They’re also much more lightweight than Gnome or KDE – I don’t use most of the niceties in either one of those two very much, so there’s no sense in paying the performance cost if there are no other benefits. Also, and this may be a bigger reason than any of the others, I love tinkering.

At any rate, I installed them both on Arch Linux (which was fairly straightforward due to Arch’s simplicity), and then decided I wanted to see if I could get dwm working in Ubuntu on my desktop machine. This turned out to be a little more difficult, mostly because the “user-friendliness” of Ubuntu means that it has many non-standard customizations and more complex configurations to make everything as smooth and brainless as possible. Unfortunately, this means that for those of us with a technological-minded brain, it’s harder to get it to actually do what you want. Ironic, no?

First, I installed dwm itself. This is done by checking out the dwm source from their website, and building it with the ‘make’ command. Its fairly straightforward and well-documented on their website.

However, then came the problems: I didn’t want to boot into gdm, which Ubuntu does by default. I tried things like removing it from /etc/inittab (doesn’t exist on Ubuntu), editing /etc/init/gdm.conf to stop gdm on runlevel 2, which is the default in Ubuntu (this didn’t seem to do the trick), and looked everywhere in /etc for something promising. I didn’t find anything until I stumbled across this blog post. It describes how to change the command-line options for booting your kernel into a login terminal, and how to fix the audio problem which then arises (I had the same issue it describes).

Unfortunately, I had one more issue with Ubuntu. dwm makes use of a mod-key quite heavily. Most all of the commands you need to use to interact with the window manager via keyboard are prefixed with this key. I decided I wanted to bind it to the capslock key (who uses that key anyway?). Mostly because I had done this on my laptop, which runs Arch. I tried to do it using xmodmap, which is how I did it under Arch. I got some really quirky behavior if I ran xmodmap as part of my .xinitrc, as I do under Arch. The changes wouldn’t stick… unless I made the process running xmodmap sleep for 6 or 7 seconds first. I eventually figured out that this was because when X.org starts on Ubuntu, it runs the xkbcomp program, which overwrites what was set with xmodmap.

“If you can’t beat ’em, join ’em”.

I decided that in order to play nice with Ubuntu, I had to play by its rules, and use xkbcomp. After a bit of searching, I found an article which dove deeper into how it worked than I really wanted to go. It is not nearly as simple and easy to use as xmodmap. Happily, they’d already figured everything out for me. So, I used their methods to set the caps lock key to be Mod3, recompiled dwm to use that as the mod-key for all the window actions, and I was on my way!

April 16th, 2007

April 15th, 2011

It’s hard to believe it’s been 4 years. The vast majority of people who now go to Virginia Tech weren’t here 4 years ago. That day has very nearly left the collective memory of the student body.

I remember getting to class in TORG 2150 that morning a few minutes early, and wondering why my professor didn’t ever show up. Someone in the class looked on the Internet at the news. We all sat there, repeatedly refreshing all the major news sites in an attempt to figure out what was going on a few hundred yards away. By this point, we had been told we couldn’t go anywhere.

I remember not being able to call home to tell my family I was okay because too many others were calling at the same time.

I remember finally being able to leave the classroom after a few hours. We hurried back to our dorms and began checking in with friends and family to make sure those we knew were okay. All too many of them weren’t.

I remember going to the candlelight vigil — one of the, if not the, most moving experiences I have ever taken part in. Thousands of people, all standing together, doing nothing but holding their candles with Coke cups from the dining halls as makeshift wax-catchers. None of us could imagine what had just happened, but none of us could imagine being in a better community either.

I remember the spontaneous “Let’s Go….” “Ho-kies!” chants at the vigil, which meant more than they ever have, either before or after that night.

I remember the convocation, at which University President Steger and Poet Nikki Giovanni got longer and louder standing ovations than the President of the United States, who was in attendance. Even through all the raw emotions, we were able to chant “Let’s go Hokies!”.

I will never forget that day. But more importantly, I will always remember those 32, and the amazing community of Blacksburg to which they will always belong.

We ARE Virginia Tech.

Paying Homage to Ubuntu ShipIt

April 5th, 2011

Back in high school, when I had a 56k internet connection at home, my computer engineering teacher mentioned this thing called Linux in class. I was a curious goodie-two-shoes, and so I asked him after class what it was and how I could get it.

He told me about Red Hat and gave me a stack of CDs to take home and copy. Being my first time attempting to copy CDs, I failed, and ended up with a bunch of CDs whose file system contained an .iso image, rather than that .iso image becoming the file system. But, by the time I realized this, I had already returned the stack of CDs to my teacher and was too ashamed to ask for them again.

Next, I researched Linux a little more and found out that there were several other distributions, of which Mandrake was prominent at the time. So, I convinced my dad to download a Mandrake CD from his work one day, because his Internet connection was several times faster than 56k. Mandrake seemed to work quite well, and I played around with it in my spare time for a while.

Then one day, I found out that some guy (Mark Shuttleworth) was paying to ship people free CDs of a Linux distribution called Ubuntu (and they were shipping world-wide, no less). So, being free, I ordered 10. I told my teacher what I had found the next day, and he decried it a scam, and told me that the CDs would probably have spyware and/or viruses on them. I proved him wrong over the course of the next few weeks and months by installing Ubuntu on every computer I could get my hands on. Ubuntu was easier to deal with and more straight-forward than Mandrake (now Mandriva), and led me to really begin my relationship with Linux in earnest.

Since then, I have explored countless other Linux distributions and Unices, but I still credit Ubuntu and its ShipIt program with helping introduce me to the joy that is Linux.

Today, the Ubuntu ShipIt program has been discontinued. Their website displays this message:

ShipIt has closed

After delivering millions of Ubuntu CDs to millions of new users, our ShipIt programme has finally run its course. While we can no longer deliver free CDs through the programme, it’s still easy to get Ubuntu. You can download Ubuntu for free from Ubuntu.com or you can buy a CD straight from the Canonical shop.

Ubuntu ShipIt, I salute you.