How To Customize Quorum With Cloudant Using MyCouch

By Daniel Wertheim

Originally posted on Daniel Wertheim's blog here. Thanks, Daniel!

Cloudant is a distributed database-as-a-service, where each logical database is represented by a specific number of shards in a cluster (referred to as Q). Each shard is furthermore stored N times in the cluster, hence each document will be stored on more than one node in the cluster. The locality of the document is determined by using consistent hashing keys, as is very well described in the Dynamo paper by Amazon. There’s a free-to-start multi-tenant solution, where you can be up and running in 5 minutes and pay by certain types of requests, and there’s also dedicated installations of Cloudant. The point is that you don’t need to worry. The infrastructure is in good hands, and instead of worrying about maintenance, you can focus on building your application. But one interesting thing to know about as a developer is the ability of configuring read and write quorum.

Write and Read Quorum, configured for consistency

Whenever a document is written (PUT, POST) to Cloudant, it is by default (as of this writing), written to two nodes. Whenever you read a document (GET), by default, you will not get the document returned unless it can be read from two nodes. By default, the number of nodes in total are three. This means that by default, it is configured for consistency. Why? Well, if you write a document, and it needs to be successfully written to two nodes out of three, a maximum of one node (due to a partition failure) can be without the document. As a result, if you read the document with a read quorum of two nodes out of three, you are guaranteed to get the previously stored document, since the write- and read-quorum will overlap each other in at least one node. The equation for this is written:

W + R > N --> Consistency

You keep on saying: "By default"

Yes, by default, Cloudant is configured with sensible defaults for consistency. But, you can (can does not mean you should) configure these settings on a per-operation basis. So when performing your write (PUT, POST) you can pass an additional query parameter, e.g., w=1. Doing this will increase the throughput of your writes, but you will get a higher risk of getting inconsistencies. The same goes for reads. When performing a read (GET) you can pass an additional query parameter, e.g., r=1, this time indicating how many nodes you need to read the document from, to be accepted as a successful read. This could be OK, for example, if you are reading static data – data that is infrequently updated.

The CAP theorem

The terms Consistency and Partition tolerance used above are two of the three ingredients of the CAP theorem, also known as Brewers theorem. The third ingredient is Availability. Hence CAP stands for:

  • C: Consistency

  • A: Availability

  • P: Partition tolerance

You can read more about the theorem in: "Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (Seth Gilbert & Nancy Lynch of MIT). In short, the theorem explains how in any distributed system, you can only achieve two out of the three CAP properties at the same time, and to be a successful distributed system, you need to be partition tolerant. So the choice stands between being either: Consistent or Available. Being consistent means that all nodes share the same view of the state; while being available means that all nodes might or might not share the same view of the state at a given point, but nodes are configured to eventually catch up and to be in sync, which is expressed as being "eventually consistent."

One big misunderstanding is often that a distributed database system is either Consistent and Partition tolerant (CP) or Available and Partition tolerant (AP). But take Cloudant as an example; here, you can configure these properties per operation. If you were to configure write quorum to less than three and read quorum to be one, you would potentially gain a higher chance of being available, since the node serving the request could be partitioned from the other two nodes and still deliver a response. This of course increases the risk of serving data that is more stale than data residing on any of the other two nodes, and this is something your domain experts need to decide. The classic example is the banking world and ATMs. In case an ATM is partitioned, they still allow withdrawals to a certain amount. This means that banks have decided to favor availability before consistency – and thereby risk a temporarily incorrect view of bank account balances – in order to remain operational for customers.

Using MyCouch with Cloudant

MyCouch is the asynchronous .NET client for use with CouchDB and Cloudant. It’s currently only in v0.15.0 so the functionality in this post will most likely be improved and simplified soon. MyCouch has the concept of a Client. The Client points to a certain database, identified by a URL, and with it, you get a simplified experience of performing reads, writes, queries, etc. The Client makes use of an IConnectionimplementation. This Connection is the "last stop" before performing the actual HTTP-request against CouchDB or Cloudant, and as of now, there’s only one simple implementation: BasicHttpConnection, but you can easily create your own, and inject that via the constructor to the Client.

Create A Custom IConnection To Configure Read Quorum

In this case I will extend the BasicHttpConnection and ensure that all GET requests for documents will have a read quorum of one. The usage of a vanilla Client would look like this (read more about how-to get connected in the documentation):

using (var client = new Client("https://someaccount.cloudant.com/dbname"))
{
  var response = await client.Documents.GetAsync("test1");
}

the difference will be the injection of a custom connection:

var cn = new CustomCloudantConnection("https://someaccount.cloudant.com/dbname");
using (var client = new Client(cn))
{
  var response = await client.Documents.GetAsync("test1");
}

and the custom connection that intercepts all GetDocumentRequest, looks like this:

public class CustomCloudantConnection : BasicHttpClientConnection
{
  public CustomCloudantConnection(string url)
    : this(new Uri(url)) {}

  public CustomCloudantConnection(Uri uri)
    : base(uri) {}

  protected override HttpRequest OnBeforeSend(HttpRequest httpRequest)
  {
    if (httpRequest.Method != HttpMethod.Get)
      return httpRequest;

    if (httpRequest.Headers.Contains(HttpRequest.CustomHeaders.RequestType))
    {
      var requestTypeName = httpRequest.Headers
        .GetValues(HttpRequest.CustomHeaders.RequestType)
        .First();

      if (requestTypeName == typeof (GetDocumentRequest).Name)
      {
        httpRequest.RequestUri = string.IsNullOrEmpty(httpRequest.RequestUri.Query)
          ? new Uri(httpRequest.RequestUri + "?r=1")
          : new Uri(httpRequest.RequestUri + "&r=1");
      }
    }

    return base.OnBeforeSend(httpRequest);
  }
}

This code makes use of the meta information associated with the HttpRequest via custom headers. The headers mycouch-type and mycouch-entitytype will be removed before the request is performed. This is done in BasicHttpConnection.OnBeforeSend.

That’s all for now. Some info about Cloudant read and write quorum and also a use-case for a custom connection in MyCouch.

Cheers,

//Daniel

Sign Up for Updates!

Recent Posts