Jobot: Full-Text Search with Cloudant

By Max Thayer

For more on full-text search with Cloudant, check out the Zero to Full-Text Search webinar with my buddy Benjamin Young on Wednesday, October 2!

Did you know you can do full-text search indexing on Cloudant? It's built on Lucene, and exposes a powerful engine for ad hoc queries. I'll walk you through how it works using jobot as an example.

Jobot uses a server-side script to stream tweets about jobs into a Cloudant database, while the interface you see at http://jobot.maxthayer.org is a CouchApp. I built it for my friends just getting out of college to 1.) help them find jobs 2.) show them programming is awesome.

Full-text search works by defining an index, and then performing queries against it, just like secondary indexes. We define Jobot's indexes in a design document, in this case called _design/jobot, which looks like this:

{
  "_id": "_design/jobot",
  "indexes": {
    "text": {
      "index": "function (doc){
        index('default', doc.text)
      }"
    }
  }
}

That index field tells Cloudant to build our search index over each document's text field, so we can hit https://garbados.cloudant.com/jobot/_design/jobot/_search/text with our queries and get back any documents whose text field matches. So, what's a search query look like?

Say you love cupcakes. Who doesn't? Let's search for "pastry" by making a GET request to https://garbados.cloudant.com/jobot/_design/jobot/_search/text?include_docs=true&q=pastry, which gives us something like this:

{
  "bookmark": "g1AAAADjeJzLYWBgYM5gTmGQS0lKzi9KdUhJMtTLTczJLyrRS87JL01JzCvRy0styQGqY0pyAJJJ9f___88C892cSkRd4yqBYgyMn6qyiDUmjwVIMjQAKaBJ-xFGOZ7OBxv1_giJRh2AGIXkKk-1PWCjPotmZQEADmVGVg",
  "total_rows": 113,
  "rows": [
    {
      "doc": {
        "user": {
          "url": "http://www.hcareers.com/chef-jobs",
          "handle": "ChefJobsOnline",
          "name": "Chef Jobs"
        },
        "urls": [
          {
            "indices": [
              68,
              90
            ],
            "display_url": "workfor.us/Hcareers/1qn4f",
            "expanded_url": "http://workfor.us/Hcareers/1qn4f",
            "url": "http://t.co/yPEN5IwNY2"
          }
        ],
        "created_at": "Wed Sep 25 10:03:04 +0000 2013",
        "type": "tweet",
        "text": "Now hiring a Banquet Cook II/ Pastry Cook in Isle of Palms, SC, USA http://t.co/yPEN5IwNY2 #Job",
        "_rev": "1-02fc15375457517d687307cca2378e95",
        "_id": "382807457422004224"
      },
      "fields": {},
      "order": [
        1380103384000,
        127765
      ],
      "id": "382807457422004224"
    }
  ]
}

I truncated the output to only one document for readability, but hey! Pastry cook! We also get the whole doc, so we can see what other jobs this ChefJobsOnline person has shared. But this gig is in South Carolina; let's limit our pastry quest to a particular place, say London. To do that, we just adjust the q parameter to pastry AND london, and we get...

{
  "bookmark": "g2wAAAABaANkAB5kYmNvcmVAZGIxLm1hbG9ydC5jbG91ZGFudC5uZXRsAAAAAmEAYj____9qaAJGQAak2gAAAABiAABnY2o",
  "total_rows": 26,
  "rows": [
    {
      "doc": {
        "user": {
          "url": "http://www.thechangegroup.com/",
          "handle": "ChangeHsp",
          "name": "Change Hospitality"
        },
        "urls": [
          {
            "indices": [
              76,
              98
            ],
            "display_url": "tinyurl.com/mthaeue",
            "expanded_url": "http://tinyurl.com/mthaeue",
            "url": "http://t.co/gwsOA4uE0t"
          }
        ],
        "created_at": "Wed Aug 21 14:28:23 +0000 2013",
        "type": "tweet",
        "text": "New #job: Commis Pastry Chef – historic 5* Hotel – Central London,London .. http://t.co/gwsOA4uE0t #jobs #hiring",
        "_rev": "1-b83b7c4612d395574dee5b0cc3104cd9",
        "_id": "370190651524382720"
      },
      "fields": {},
      "order": [
        2.830493927001953,
        26467
      ],
      "id": "370190651524382720"
    }
  ]
}

Mmm, pastries. Well, now I'm just craving mince pies. Thanks, Jobot.

We used an AND operator in that query to return results which met multiple criteria. If we wanted results that met at least one but not necessarily all of our criteria, we would just omit the AND. For example, to search for jobs that are either pastry gigs or in London, we would use pastry london as opposed to pastry AND london.

Cloudant exposes a powerful query syntax for performing precise, multi-dimensional searches. Check out the full power of the engine in our For Developers section :D

As always, if you have any trouble, check our docs, post your question to StackOverflow, ping us on IRC, or if you'd like to discuss the matter in private, email us at support@cloudant.com.

Like that? Want more? Check out the Zero to Full-Text Search webinar with my buddy Benjamin Young on Wednesday, October 2!

Create an account and try it yourself

Sign Up for Updates!

Recent Posts