Quilter: Open Source File Sharing

By Max Thayer

Legend has it that, having forgotten to bring the USB stick containing most of his code along on a bus ride to New York, Dropbox's founder Drew Houston began prototyping a project to sync files over the web -- the basis for what would eventually become Dropbox.

Cloudant's _changes feed and ability to store attachments makes similar technology effortless. Syncing is easiest when both sides understand replication, but even when they don't, Cloudant still makes it easy.

Recently, I built Quilter to sync filesystems, like an open source Dropbox. Cloudant can store files as attachments, from text files and word documents to images and movies, so I built Quilter to sync local folders with remote Cloudant databases.

Attachments are files attached to documents, exempted from indexing, and retrievable by URL. For example, this image is an attachment:

Though Quilter is still in beta, it happily and reliably syncs my image folders between computers, powering EggChair and TransCommits. Combined with the upcoming Cloudant Sync libraries for iOS and Android devices, feature parity with Dropbox becomes effortless -- and open source :D

Using Quilter

Quilter requires Node.js, so if you don't have it, go get it!

Then, just install!

sudo npm install -g quilter

Quilter syncs with CouchDB and Cloudant databases, so if you don't already have an account...

Click here to make one

Now you can start syncing folders:

# push files from your machine to a remote database
quilt push --local {folder} --remote {url}
# pull files from a remote database to your machine
# and continue to watch for changes
quilt pull --local {folder} --remote {url} --watch
# push and pull files to and from the local and remote
# and save the job to disk, so you can run it again later
quilt sync --local {folder} --remote {url} --save

To run all saved jobs, just run quilt. To see all saved jobs, try quilt jobs.

If you have jobs you want to run on startup, you can use cron to quilt on startup. I do that to ensure my folders are always syncing and up to date.

How Quilter Works

Quilter monitors my filesystem and a Cloudant _changes feed. Whenever local files change, they're uploaded to Cloudant. Whenever the files in Cloudant change, those changes show up in the _changes feed, and Quilter downloads them to my computer.

Both when pushing changes to Cloudant, and when pulling changes to your filesystem, Quilter uses a worker-queue pattern to make sure it doesn't get overloaded:

  • One worker watches the filesystem, or a Cloudant database. When it observes changes, it pushes information about that change into a queue. (Source for the push worker, and the pull worker)
  • Another worker watches the queue, consuming jobs as they appear by pushing files to the database, or pulling files down to your computer. (Source for the push queue and the pull queue])

Using a queue helps to isolate each moving part of Quilter, so each works only as hard as it can. No matter how quickly your files are updating, Quilter will just keep populating the queue, and consuming it at its leisure.

I use Forever to restart Quilter if it encounters problems, like losing internet connectivity. A command in my crontab then starts Quilt on startup, so it's always syncing my files and folders. Just like Dropbox.

Sharing Quilts

If I want to share these synced folders with others, I can use Cloudant's database sharing to create API keys for friends, grant permissions to their accounts, or replicate a subset of the data to another database and share that. For example, in this image...

... in that red circle, you could enter the username for a friend's Cloudant account, and assign them permissions to the database, whether to read, write, or even administer the database alongside you. Do this for enough folks and, boom, Dropbox.

Or, using filtered replication, you could replicate a specific subset of files -- say, only image files with 'Screen Shot' in the _id. In that case, you could write a filter function like this:

function (doc, req) {
  // ensure it's a quilter file
  if (doc.timestamp && doc.hash) {
    var type = doc._attachments.file.content_type.split('/')[0];
    // get only files whose content types match "image/*"
    // and whose `_ids` feature 'Screen Shot'
    if (type === 'image' && doc._id.indexOf('Screen Shot') !== -1) {
      return true;
    }
  }
  // if we don't return true, return false
  return false;
}

That'll return only images marked as screenshots. Neat, huh?

Quilter is open source, and still very much in beta. If you want to check it out, try it out, or contribute to it, peruse the repository!

As always, if you have any trouble, check our docs, post your question to StackOverflow, ping us on IRC, or if you'd like to discuss the matter in private, email us at support@cloudant.com.

Happy coding!

Addendum: For the moment, Quilter works best with files smaller than gigabytes. Beyond that, and syncing can take a while. Eventually, Quilter will store files in S3, so that it only stores metadata in Cloudant.

Sign Up for Updates!

Recent Posts