Weblatch: automated Webhooks with Cloudant and Zapier

By Benjamin Young

Webhooks is a simple HTTP usage pattern for implementing simple subscriptions via a simple "hook" system. They essentially amount to HTTP-based "callbacks."

The typical Webhook is set up this way:

  1. user tells a Web App to POST "event" data to a certain URL.
  2. Web App sends "event" data to the URL specified by the user in step 1.

Imagine taking event and action data from all the various web applications you use daily. What if you could find things you did across Github commits, Trello cards, Fogbugz tickets, Tweets about replication, and emails you sent to your engineering team? Combining Webhooks with Weblatch and a Cloudant database, that’s all quite possible!

But there be dragons. Webhooks are really just a name for a usage pattern. There is not a specification around what should be sent, in what format, or even how to agree on basic things like the Content-Type to be sent, etc. It's completely up to the implementers of the Webhook what gets sent, how often, and in what format.

Enter Weblatch

Webhook listeners are entirely at the mercy of the sending party's choices of Content-Type and the "shape" of the content being sent.

To deal with that, Weblatch adheres to Postel's law:

"Be conservative in what you send, be liberal in what you accept."

Weblatch makes use of the _update function and endpoint available in any Cloudant database. Once set up, nearly any Webhook provider can POST (or PUT) data into the Cloudant database of your choice--from which you can further MapReduce, full-text search, or otherwise explore.

Create an account and try this in your own Cloudant account

Every Day Webhooks with Zapier.com

Zapier.com is a "web automation" service. It provides "Triggers" and "Actions" from "over 250 web apps." That array of web apps covers my current use case, of seeing across our development tools and activity. Once that event data lands in Cloudant, I can MapReduce, full-text search, and otherwise "post-process" it into my own activity stream or analyze it for interesting patterns.

Thankfully, Zapier.com supports Webhooks! This gave me the ability to take any of those services and pipe their event data directly into a Cloudant database.

Step 1: Install Weblatch

The simplest method to installing Weblatch is via replication. If you have curl installed, it's as easy as posting some simple JSON to your Cloudant's _replicate endpoint (be sure to change USER:PASS@USER to match your setup):

curl https://USER:PASS@USER.cloudant.com/_replicate \
-H 'Content-Type: application/json' \
-d '{ "source": "https://labs.cloudant.com/stub-weblatch", \
"target": "weblatch", "create_target": true, "use_checkpoints": false}'

That curl line will create a weblatch database in your Cloudant account, and replicate the _design/weblatch Design Document into it. The Weblatch "update function" is now installed in the weblatch database.

Step 2: Hook it up!

  1. Visit Zapier.com and setup your account (if you've not yet).
  2. Create a Zap.
  3. Pick a Trigger Service (Twitter's an easy one, but pick whatever).
  4. Pick "Web Hook" as the Action Service and PUT as the Action.

Below is a screenshot of the Trello Zap I’ve set up to dump Trello Notifications into my weblatch database:

Setup Trello + Weblatch on Zapier

Here’s the URL field for easy copy and pasting (be sure to customize it!):

https://USER.cloudant.com/weblatch/_design/weblatch/_update/latch/trello:{{id}}

I highly recommend using an API Key and Secret to access Weblatch (rather than your Cloudant Account username and password).

Visit the Cloudant Dashboard area for your weblatch database. Click Permissions. Click "Generate API key" and copy and paste the new key and secret into the Basic Auth section. The new API Key name will show up on the right-hand side where you can set further permissions.

Now! Let's try it out! The last step at Zapier allows you to send test PUT's to your Weblatch endpoint. Click "Get samples...", then click "Send!" next to any (or each) of the samples.

Lastly, visit your weblatch database at Cloudant to see the document that got added, explore its JSON, and plan your MapReduce adventures!

How Weblatch Works

When you installed Weblatch, the replication process pulled over a simple Design Document (the document type you also store views inside of). In that document is an "update function" that allows HTTP requests to hit an endpoint hosted in your Cloudant database that can process incoming requests and create a document with the input.

Below is the current Weblatch code. It handles basic application/json content like that sent by Zapier.com's Webhook integration as well as Github's more unique application/x-www-form-url-encoded content sending style.

Let's read the code:

function(doc, req) {
  var new_doc = {};

  if (doc !== null) {
    // The "callback" nature of Webhooks lends more toward "appending" than
    // overwriting, so we're going to prevent that this time around.
    return [doc, 'A document already exists. No updates made.'];
  } else {
    // This section, more than any area will change based on the Webhook being
    // used with Weblatch.
    // This first section handles Github's particular way of sending Webhooks.
    if ('form' in req && 'payload' in req.form) {
      // Github-style Webhook...with URI encoded payload...
      new_doc = JSON.parse(decodeURIComponent(req.form.payload));
    } else {
      // The `req.body` key contains anything that came in the body of the HTTP
      // request. Let's see if it's JSON first, and if it is, make that our
      // new document.
      try {
        new_doc = JSON.parse(req.body);
      } catch(e) {
        // If it's not, we still want to know what came in via the Webhook.
        new_doc['body'] = req.body;
      }
    }
    // Now we need to either use the requested `id` or a generated `uuid`.
    new_doc['_id'] = req.id || req.uuid;
    // However! If we make it here, we save the document and return a useful
    // message (for anyone that may be watching (likely just robots...).
    return [new_doc, 'Document ' + new_doc['_id'] + ' stored successfully!'];
    // Now check your database contents.
  }
}

Each update function receives two parameters: doc and req. The doc is either null or the existing document requested from the database via the update handler URL. The request includes important, inbound HTTP headers, database information, and pre-parsed form content (if application/x-www-form-url-encoded is used).

Update functions must always return a two-part array containing:

  1. null or the JSON document to store in the database
  2. a message to display in response to the request

The first return line in the code above is a failure notification (Weblatch doesn't update any existing documents...by design). The last return line creates the new document using either the requested identifier (in req.id) or a UUID generated by the database (in req.uuid).

The Apache CouchDB documentation covers update functions in more depth.

Weblatch the Web!

There are untold numbers of other Webhook sending parties out there. It would be great to see Weblatch tested with as many of them as possible!

If you try Weblatch with your favorite Web Apps, please toss issues and wiki thoughts into the Weblatch repo. The more that Weblatch can catch, the more data from more applications you can mix into your MapReduce and Full-Text Search indexes to see the state of your world and add your own documents into Cloudant alongside the others for annotation, expansion, and organization.

Happy latching, everybody!

Sign Up for Updates!

Recent Posts