First off, if you haven’t tried Dropbox, you should check it out; sync all of your computers via the Dropbox servers, their basic free service gives you 2Gigs of space and works cross-platform (Windows, Mac, Linux). I use it daily at home and work, and just having a live backup of my main data for my work workstation, my home netbook, and any other computer I need to login to is a huge win. [PRBREAK][/PRBREAK]Plus, I have various ‘shared’ folders that distribute certain data to certain users that I’ve granted access to, this means work details can be updated and automatically distributed to the folks I want to review/use the data. I recommend everyone try it out, and see how useful it is, it’s turned into a game changer for me. So a few months ago they made headlines on supporting Linux as they released the client as open source. While this got hopes up for many, it was only the client that was open source, the server is still proprietary. While slightly disappointing, this is fine, they’re a company trying to make money. I don’t fault them for this, it’s just that a free, portable service like that would be a killer app.
Meanwhile at work I’m working on a solution to sync large data clusters online and the project manager described it as the need for ‘Dropbox on steroids’. Before I had thought it was more complicated, but after thinking about it, I realized he was right. Look, Dropbox is a great idea, but it obviously is just a melding of rsync, with something watching for file changes to initiate the sync, along with an easy to use front end. From there I just started looking at ways this could work, and there are more than a few; here’s how I made it work.
Linux now includes inotify, which is a kernel subsystem that provides file system event notification. From there all it took was to find an application that listens to inotify and then kicks off a command when it hears of a change. I tried a few different applications like inocron, inosync and iwatch, before going with lsyncd. While all of them could work, lsyncd seemed to be the most mature, simple to configure and fast. Lsyncd uses inotify to watch a specified directory for any new, edited or removed files or directories, and then calls rsync to take care of business. So let’s get started in making our own open source Dropbox clone with Debian GNU/Linux (lenny)
Ladies and gentlemen, start your engines servers!
First, you need 2 severs; one being the server and the other the client. (you could do this on one host if you wanted to see how it works for a proof of concept)
Install OpenSSH server
First you’ll need to install OpenSSH Server on the remote system:
apt-get install openssh-server
Configure SSH for Passwordless Logins
You’ll need to configure passwordless logins between the two hosts you want to use, this is how rsync will pass the files back and forth. I’ve previously written a HOWTO on this topic, so we’ll crib from there.
First, generate a key:
ssh-keygen -t rsa
UPDATE: actually, it’s easier to do it this way
ssh-keygen -N '' -f ~/.ssh/id_dsa
You shouldn’t have a key stored there yet, but if you do it will prompt you now; make sure you overwrite it.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
We’re not using passphrases so logins can be automated, this should only be done for scripts or applications that need this functionality, it’s not for logging into servers lazily, and it should not be done as root!
Now, replace REMOTE_SERVER with the hostname or IP that you’re going to call when you SSH to it, and copy the key over to the server:
cat ~/.ssh/id_rsa.pub | ssh REMOTE_SERVER 'cat - >> ~/.ssh/authorized_keys2'
UPDATE: now you can use ssh-copy-id for this instead (hat tip briealeida)
Set the permissions to a sane level:
ssh REMOTE_SERVER 'chmod 700 .ssh'
Lastly, give it a go to see if it worked:
You should be dropped to a prompt on the remote server. If not you may need to redo your .ssh directory, so on both servers:
`mv ~/.ssh ~/.ssh-old`
and goto 10
Install rsync and lsyncd
Next up is to install rsync and lsyncd. First, rsync is simple, and could already be installed (you don’t need to run it as a server, just the client), make sure you have it with:
apt-get install rsync
Next is lsyncd. There is no official Debian package yet, but it’s simple to build from source and install. First off, if you don’t have build essentials you’ll need them, as well as libxml2-dev to build the lsyncd source. Installing those is as simple as:
apt-get install libxml2-dev build-essential
Now we’ll get the lsyncd code (you can check for a newer version at http://lsyncd.googlecode.com) and build that:
tar -zxf lsyncd-1.26.tar.gz
make; make install
This install does not install the configuration file, so we’ll do that manually now:
cp lsyncd.conf.xml /etc/
Next up, we’ll edit the configuration file now located in /etc The file is a simple, well documented XML file, and mine ended up like so