Coding for Eventual Consistency

By Max Thayer

Jan Lehnardt recently wrote Understanding CouchDB Conflicts, which explains some of the nit and grit behind eventual consistency in CouchDB and Cloudant, but underlying Jan’s article is the question, "How does eventual consistency change how I should write applications?"

Whether your database guarantees strong consistency (where every part of a distributed system sees the same data at the same time) or high availability (where all requests receive responses, regardless of system failure) changes what you can expect of it. No distributed database can guarantee both while remaining meaningfully distributed, so it’s important to understand how you build applications given the tradeoff.

So, as a developer, how should you code for eventual consistency?


Store State Locally

Cloudant implements an eventually consistent clustering model (quorum-based, inspired by the Amazon Dynamo paper), which guarantees high availability. When I first started using Cloudant, I wrote a simple script that, among other things, inserted a document and immediately tried to retrieve it again. Occasionally, the script would fail, and I didn't understand why -- until I understood eventual consistency.

My script inserted the document into one node, but broke when the retrieval attempt spoke to a different node, which didn't yet have the inserted document. By skipping the retrieval and instead using the local copy of the document I inserted, I got the script to stop breaking, and perform faster by avoiding unnecessary database requests.

When nodes of a database cluster handle requests, such as insertions and updates, they need to let the rest of the cluster know, and bring them up to speed. Some systems will "lock" portions of the cluster while that happens, to prevent conflicts like what my script experienced and enforce the property of isolation. The downside is that, while locked, you can't perform insertions or updates to the locked fields, rows, documents, nodes, etc.

Other systems, like Cloudant and CouchDB, never use locks. This makes every part of the system always available to handle requests. In exchange, try to afford the cluster a moment to bring itself up to date after any change. Alternatively, store relevant state locally, such as by using PouchDB, which will keep itself up to date with your cluster automatically.

These days, if I’m writing data, I use PouchDB to replicate any relevant subsets of the data into the browser, replicating them back as I make changes. This gives me on-disk access speeds, ensures any local changes I make are immediately available, and syncs changes to and from Cloudant in the background. (N.B. Make sure to use filters to minimize the size of the dataset you replicate into PouchDB; it’s optimized for working with small datasets)

Insert, Don't Update

Kyle Kingsbury wrote Call me maybe: Carly Rae Jepsen and the perils of network partitions to explore all the wondrous ways distributed systems crack under pressure. By and large, the lesson across every database Kyle examined boiled down to updates break things. So, don't do updates.

"But wait, Max," you say. "I store {votes,visits,transactions} as updates to a {user,site,cart} document. How else would you do it?" Relations, yo.

Let's take votes, for example. Say you ran Reddit, and you decided to store votes as part of each post, like this:

{
 	"_id": "...",
	"_rev": "...",
	"type": "post",
 	"title": "The Mighty Broberg",
 	"link": "https://.../broberg-flips-every-table",
 	"votes": [
		{
 			"user_id": "...",
 			"created_at": "...",
 			"magnitude": 1
		}, {
    	    	    	"user_id": "..."
    	    	}
	]
}

What happens when two folks vote on the same object at once? The cluster will update the post document, appending a vote to the votes field, and save them at the same time -- resulting in two valid but conflicting documents. Most systems reject one (resulting in a failed request), or overwrite one (resulting in a successful request and data loss). Cloudant and CouchDB store both conflicting documents, so you can resolve the conflict yourself, but it still picks one of the conflicting documents as the winner until you resolve the conflict by hand, or handle it using some automated process. To the user, until you resolve it, the conflict looks like data loss.

Instead, store votes independently, like this:

{
	"_id": "...",
	"_rev": "...",
	"type": "vote",
	"post_id": "...",
	"user_id": "...",
	"created_at": "...",
	"magnitude": 1
}

When someone votes on a post, create a document with a field like "type": "vote", and other data reflecting the vote. To prevent duplicates, make the "_id": some unique and dependent value like "user_id+post_id" so that attempts by a single user to vote more than once on a post will be rejected as conflicts. Then, use MapReduce to count up the number of votes for a particular post, like this:

{
	map: function (doc) {
		if (doc.type === 'vote') {
			emit(doc.post_id, doc.magnitude);
		}
	},
	reduce: '_count'
}

Now, any number of folks can vote on a post at the same time, and none of them would experience write failures or data loss. (For more on MapReduce indexes, see our docs)

So, insert. Avoid updates. Where you can't avoid updates, try to reduce the number of agents that might be interacting with that document at once. If you need to update information about a user, for example, try to ensure only the user, and/or only admins, can edit that document.

Create an account and try Cloudant DBaaS yourself

As always, if you have any trouble, check our docs, post your question to StackOverflow, ping us on IRC, or if you'd like to discuss the matter in private, email us at support@cloudant.com.

Sign Up for Updates!

Recent Posts