Today we’re announcing the open-sourcing of our Android™ and iOS® sync libraries. Our press release discusses some of the business drivers for this, but, as project lead, I wanted to talk a little about the features and design choices we made.
It’s clear the future for mobile devices is having all your data available, all the time -- even when network connectivity is unavailable. When we started this process, we were therefore very clear on our goal:
Make a great database for device developers which can synchronise data across devices using CouchDB and Cloudant.
In essence, we did an experiment: produce a database which used enough of the Apache CouchDB™ data model to replicate data -- and so allow developers to easily add synchronisation features to their application -- but to also make a device-local database tuned to the needs of application developers. This experiment was successful, and so Cloudant officially created a Sync team within the company to bring the product to market.
Alongside the extensive testing we’ve done against the Cloudant service, the two main differentiating features of our Sync libraries are:
Cloudant’s HTTP interface is straightforward -- for those well-versed in HTTP. So one of the first things we decided to do was to create a native API for devices which didn’t attempt to mimic Cloudant’s HTTP interface but instead felt at home to mobile application developers. Our MVCC data model imposes certain constraints on this, of course, but we’ve tried to offer affordances unavailable over HTTP. The conflicts API is an example of this (Android docs).
We also decided to revisit the querying model used on devices. The querying requirements on local datastores often differ from those required for data stored in the cloud. Our customers often run large analytics tasks over their data using our incremental map-reduce platform. On a device, the queries are usually simpler, but also more fluid. So, to start with, we concentrated on building an easy-to-use, more familiar indexing and query framework (Android, iOS).
Finally, we’re releasing this under the Apache licence, meaning that it can be used in both proprietary applications and by other open source projects.
The Cloudant Sync libraries build on a number of battle-tested open source libraries. For both the Android and iOS libraries, the rough architecture is the following:
We use several open source libraries and pieces of code in Cloudant Sync, which are called out in the diagram above.
While it might seem like an odd choice to build a NoSQL database on top of SQLite, SQLite is incredibly well proven in production and is available on all of the platforms we’re targeting. Using SQLite as our durability layer was therefore a no-brainer.
We use MVCC code from TouchDB, which layers MVCC semantics onto SQLite. Again, this code is well-tested, particularly on iOS, so reinventing the wheel didn’t make sense.
The library is able to synchronise data directly with either Cloudant or CouchDB over HTTP(S) -- no need for any extra software. We based our iOS replication code on TouchDB’s, but wrote our own for Android to make better use of the facilities the Java platform offered.
On top of these core pieces, we wrapped a native-application-friendly API. At the moment we’re still hammering out the details of this as we use the library in different applications; expect this to change and improve on the way to 1.0.
The final piece was again leveraging SQLite to allow developers to easily build indexes around the data in the JSON data, by allowing developers to easily make use of SQLite’s fast indexing with the data from their JSON documents. Furthermore, each document in the database can be indexed in any number of ways, allowing for flexible querying semantics which are harder to achieve with CouchDB’s map-reduce views. We’ve found this to be a more natural fit to application use-cases in our testing.
Both libraries are on Github. Each repository has API tutorials and an example application:
With these libraries, we’re hoping to make the development of applications that sync across devices and platforms simpler and faster. Bringing together a distinctly native programming interface with data querying functionality focussed on the specialised needs of mobile applications, along with the powerful and proven replication functionality protocol of CouchDB, presents a uniquely easy to integrate solution.
We’re still in early days for this product, so now is a great time to get involved and influence the direction of the project as we explore the problem space. In particular, we’d love to hear feedback on the APIs -- where they work and where they don’t -- along with feedback on how the querying functionality fits into your applications. If we’re missing features useful to your application, let us know.
And finally, I hope you enjoy using the library.
The Android code is available on Github here and the iOS code is here. Both projects are available under the Apache 2.0 licence, which means that they can be safely integrated into both open source and proprietary projects without problem.
Open source is very important to Cloudant. As a service business, giving our users the tools to adapt how they access the service to their needs is important, and open source libraries such as the sync libraries are a part of this. To this end, we’ve also removed as many barriers to contributing as possible, making it as simple as a Github pull request -- no paperwork as we’re committed to maintaining the library under the Apache licence.
Both the Android and iOS libraries make use of code from TouchDB, and we’re incredibly grateful to be standing on the shoulders of those who’ve gone before us. However, we decided against forking the project in favour of reusing targetted portions of the TouchDB code.
TouchDB follows the CouchDB interface very precisely, going so far as to implement a HTTP layer and allowing access from CouchCocoa, an Objective-C CouchDB client. We wanted to experiment in creating a datastore following CouchDB’s data model but not its API. It therefore made sense to extract appropriate pieces from TouchDB rather than forking the project and immediately engaging in a large scale re-organisation and cutting-down of the library -- leaving something very different to the original project.
I think the way we’ve chosen to go avoids belittling the contribution of existing code while not sowing confusion about what the “right” fork of TouchDB is given the diverging project aims.
My experience as an app developer lead me to believe that the indexing and querying functionality we’ve implemented is a better fit to many applications’ needs. However, we know views are useful for many other use-cases, particularly for those already making use of views on Cloudant/CouchDB, so this feature is very much on our radar.
 For iOS, changes we have made to TouchDB can be seen in a checkout using:
$ git log Classes/common/touchdb/
$ git diff 1018859e0b63ffd9ed92587c01c37becd371f27d HEAD Classes/common/touchdb/
On Android, the code is less segregated, but all files with TouchDB code are marked accordingly.
Android is a trademark of Google Inc. IOS is a trademark or registered trademark of Cisco Systems, Inc., in the U.S. and other countries. The Android robot is reproduced or modified from work created and shared by Google and used according to terms described in the Creative Commons 3.0 Attribution License.