cloud computing

Trusting cloud computing is not about trusting the cloud, it's about trusting the company running the cloud.

Early on the West-coast morning of Friday, January 30th, Ma.gnolia experienced every web service's worst nightmare: data corruption and loss. For Ma.gnolia, this means that the service is offline and members' bookmarks are unavailable, both through the website itself and the API. As I evaluate recovery options, I can't provide a certain timeline or prognosis as to to when or to what degree Ma.gnolia or your bookmarks will return; only that this process will take days, not hours.

One of the downsides with using a third party service (or someone else's cloud) for your computing is you have no control over things like this. Hopefully they'll be back up sooner rather than later.

So then why not then move Bittorrent out of the home/office and into the cloud? This weekend I was able to do just that with great success. Using Amazon's Elastic Compute Cloud (EC2) and TorrentFlux (a web-based Bittorrent manager which runs on top of Bit Tornado). I created a web-based, open-source Bittorrent "machine" that liberated my network and leveraged Amazon's instead. I can access it from anywhere, uploading Torrent files from wherever, and manage them from my iPhone. I also pay only for what I use - which by my estimates will hopefully be less than $30 per month (real data to follow). And the best part? My ratio is maintained while my Left 4 Dead games speed along without any annoying lag.

If you think you might be interested in doing the same thing, please read on for a thorough HOWTO.

Can you trust your data to the cloud? For users of an online storage service called The Linkup, formerly known as MediaMax, the answer turned out to be a resounding 'no.' The Linkup shut down on Aug. 8 after losing access to as much as 45% of its customers' data. 'When we looked at some individual accounts, some people didn't have any files, and some people had all their files,' The Linkup CeO Steve Iverson admits. None of the affected users will get their lost data back. Iverson called it a 'worst-case scenario.'

Not that I've got a perfect record with data integrity myself.

In addition to my Mac Pro, I now have a 12" Powerbook that I use on the go. Having two computers introduces the hassle of keeping them in sync. Since .Mac comes with a 60 day trial, I decided I'd give it a try. For what it does, it does it well: it kept my mail, bookmarks, calendar and address book in sync, but nothing else.

However, .Mac costs $100 a year and it didn't meet all my needs. It did, however, offer a whole bunch of solutions (in the form of cloud computing) that I didn't want because rather than used someone else's cloud, I'd rather roll my own open source cloud. I don't use flickr, gmail or wordpress.com for this very reason, so why would I want .Mac email, webhosting, galleries, etc? All I really want to do is keep to Macs in sync, including all my documents. I googled and the best I found was Geek Throwdown: How to sync two or more Macs?.

Enter Unison. Here's a quick guide:

1. I turned on remote login in system preferences on the PowerBook. This lets me SSH into it, which is a good thing because Unison operates over SSH.
2. Installed the OS X binary of Unison onto both machines. (Downloaded the GUI universal binary and then launched the application, from there, within the application I was able to install the text version. Again, I did this for both machines.)
3. Logged out of the PowerBook, then SSHed into it from the Mac Pro. I then deleted my entire home directory on the PowerBook (rm -rf).
4. I exited all running programs on the Mac Pro except a terminal.
5. Created a directory .unison in my home directory on my Mac Pro. Inside that directory, I created a file sync.prf. Here's the contents of that file, annotated to explain what each line means:
   

   # Roots of synrchonoization
   # I want to sync my entire home directory of the Mac Pro, the local machine with
   root = /Users/brianpuccio
   # ... my PowerBook, hostname beta, the entire home directory
   root = ssh://brianpuccio@beta.local//Users/brianpuccio
   
   # This synchronizes file modification times
   times = true
   
   # This turns off logging
   log = false
   
   # This tells unison to ignore some files and paths
   # http://alliance.seas.upenn.edu/~bcpierce/wiki/index.php?n=Main.WikiSandbox
   ignore = Name .FBCIndex
   ignore = Name .FBCLockFolder
   ignore = Name {Cache*,.Trash*,.VolumeIcon.icns,.HSicon,Temporary*,.Temporary*,TheFindByContentFolder}
   ignore = Name {TheVolumeSettingsFolder,.Metadata,.filler.idsff,.Spotlight,.DS_Store,.CFUserTextEncoding}
   # ~/.fseventsd/ is owned by root, don't have privledges to this, so ignore it
   ignore = Name .fseventsd
   # Unison is also the name of a usenet client http://www.panic.com/unison/
   # This ignores its very large and often changing cache, which is fine since I don't use it on the Powerbook
   ignore = Path {Library/Application Support/Unison/news.usenetserver.com}
   # This is Mail's cache of my IMAP accounts, since this was large and I kept having Unison crash on the
   # first few syncs, I omitted this path figuring Mail on the PowerBook would sync once it went online
   # It worked, so I left it alone and in here
   ignore = Path {Library/Mail/IMAP-*}
   # This is my aperture library and it is too big to fit on my PowerBook (and the PowerBook too slow to run Aperture)
   ignore = Path {Pictures/Aperture Library.aplibrary}
   # It seems like Unison should automatically ignore its own config folder, but it didn't for me, so I added this
   ignore = Path .unison

6. I ran Unison by issuing the command unison sync (sync because that is the name I cave the preference file, sync.pref.
7. The GUI launched, asked me for my password on the PowerBook, which I entered. It did a quick comparison (since the PowerBook should have a completely empty home directory) and then listed all the files and directories.
8. I made sure that Unison was set to sync each file and directory from the Mac Pro to the Powerbook (Unison uses left and right terminology, e.g., sync file from left to right).
9. I clicked Go and it churned along.

For the most part, it worked. All my files were moved over, my keychain and all its passwords, my browsing history and bookmarks, my Adium settings, accounts and chat histories, my Colloquy settings and chat logs, my dock, my background ... quite literally everything.

There are a couple of issues:

1. Window positions are copied over. Going from a 1920x1200 screen to a 1024x768 means some windows were too large. A quick window, zoom command fixed those. (But they then get synced back to the Mac Pro, where everything will now launch in the top left of the screen.) I'm just going to accept this.
2. Several files (such as ~/Library/Preferences/com.apple.dock.plist, ~/Library/Preferences/com.apple.dashboard.plistand ~/Library/Preferences/com.apple.finder.plist) are constantly modified by the OS and the very act of logging in to both machines means they both have their timestamps changed, which means Unison doesn't know which one you want to sync to the other, so it displays a ? instead and the default is to do nothing. Of course you can go through each of these one by one (or even en masse) and set that sync to be a Left to Right or Right to Left sync. I'm going to try using the force option and either favor a root explicitly or use the newer option.
3. Dropping to a terminal to do this (and I do it a few times a week, whenever I shift from one machine to another) is annoying. I'm created an Automator script to run Unison and added it to my dock.
4. Typing in my password for every sync is annoying. I now use an SSH key.
5. I have only tried this where UIDs and GIDs were the same. Good luck to you if they're different!

I will update this as I make improvements.

amazon's elastic compute cloud, "ec2", provides a flexible and scalable hosting option for applications. while ec2 is not inherently suited for running application stacks with relational databases such as lamp, it does provide many advantages over traditional hosting solutions.

in this article we get a sense of lamp performance on ec2 by running a series of benchmarks on the drupal cms system. these benchmarks establish read throughput numbers for logged-in and logged-out users, for each of amazon's hardware classes.

...

amazon uses xen based virtualization technology to implement ec2. the cloud makes provisioning a machine as easy as executing a simple script command. when you are through with the machine, you simply terminate it and pay only for the hours that you've used.