Grokking the cts API

As of the MarkLogic 10.0-3 release, the total number of built-in cts (“core text search”) functions comes in at 367! That already excludes deprecated functions. Given how central the cts functions are for building applications on MarkLogic, I thought it would help to provide some pointers in navigating this potentially overwhelming API.

But first of all, if you’re just getting started building a standard search application, you should start with the Search API (which uses and provides hooks into the cts API).

Having said that, here you go!

Just kidding. While word clouds can be fun, they’re not always very useful. (I generated the above based on each function’s number of search hits on this website, so I suppose the result is somewhat interesting; just don’t put too much stock in it.)

Let’s take a tour through the cts API, using some categories I’ve chosen. We’ll knock down all functions, without necessarily explaining how they work. You’ll want to refer to the cts API documentation for those details.

The following list summarizes my breakdown by category:

• Query Execution (3)
• Query Objects (220)
• Lexicon Functions (69)
• Lexicon Reference (21)
• Geospatial shapes (18)
• Sorting (7)
• Search result meta-data (6)
• Miscellaneous (23)

Now let’s take a quick tour through each one.

Query execution (3)

The most important function of them all is cts:search, which is concerned with executing cts queries (we’ll get to those next). A related and also important function is cts:contains which matches a given node sequence against a given cts query, returning true if it matches and false otherwise. cts:walk is used in a similar manner as cts:contains except it returns the actual match instead of just true or false. Three down, 384 to go!

Query objects (220)

MarkLogic extends the XPath data model with an object type called “cts:query”, which is the super-type of a number of more specific cts:query sub-types. Queries can be composed together using the cts query constructor functions. They can then be executed by passing them to cts:search() or passed to other functions, such as lexicon calls or functions in other libraries, including search:resolve(), jsearch’s where clause and many others. All of these function names end in “-query”. If you see a cts function whose name ends in “-query”, you can be assured that it’s a cts:query constructor.

Query constructors can be categorized into different kinds. I’m going to call them leaf, composite, and “special” (for lack of a better word).

Composite query constructors (12)

The composite query constructors build up new queries from other queries, whether leaf queries or other composite queries. Here they are broken down into a few sub-categories:

Leaf query constructors (35)

The leaf query constructors are for queries that can stand on their own, i.e. can be constructed without the help of another query constructor. The following list breaks them down into several categories, depending on what the query searches for (collection URIs, directories, words, values, etc.). I’ve marked some of the text with bold type to draw attention to the consistent naming conventions.

Words and values differ in how they compare content against the search. A JSON document containing {“Text”: “some content”} will match cts:word-query(“some”) but not cts:json-property-value-query(“some”).

Another thing worth noticing about the word, value, and range queries above is that they have consistent ways of scoping queries: by element, by attribute, or by field. So we see a function for each pairing of scope (element, attribute, or field) and object (word, value, or range). We’ll see something similar with the lexicon functions. Stay tuned.

This scoping applies to filtered search, i.e. we expect documents for element-***-query to return only XML documents while json-***-query would only return JSON documents. For unfiltered search, element-***-query and json-***-query will return both JSON and XML documents that match the query. Of course this does not apply to element-attribute-***-query since there is no such thing for JSON documents.

Special query constructors (5)

While the functions below each return a cts:query value, they don’t really fall into the above (leaf vs. composite) categories:

Okay, only 332 functions to go. (I promise the pace will pick up soon.)

Query accessors (168)

The query accessor functions aren’t very interesting at all—and there are 168 of them! They’re accessors for the various components of a cts:query value. You can recognize them using this failsafe technique: if you see a cts function whose name includes the string “-query-“, then it’s just an accessor. An example would be cts:word-query and its three accessors: cts:word-query-options, cts:word-query-text, and cts:word-query-weight. See a pattern?

Lexicon functions (69)

Lexicon functions are much more interesting. Whereas cts queries are about efficiently finding documents, lexicon functions are about efficiently retrieving unique values (or words or URIs, etc.) from across a potentially large number of documents. They all require a particular index setting to be enabled. For “search,” think cts:search. For “analytics,” think lexicon functions.

String lexicons (14)

Below are the 24 non-geospatial lexicon and lexicon wildcard functions grouped by lexicon type. Note the consistent naming conventions (at the end of the function names).

Lexicons are typically found at the database configuration page of the Admin UI, except for Field word lexicon as noted above.

Scalar type specific lexicons (18)

The range index above is a combination of element, attribute and field range index. “Range index” also includes the collection and uri lexicon. Indexes are found on the left-hand side of the Admin UI when you click on a database (Configure >> Databases >> {database name} >> *** Index. These functions can be used to generate aggregate reports.

Geospatial lexicons (17)

Requires corresponding geospatial index (element, element pair, element-child, element attribute pair). Which of these you use depends on how you chose to represent geospatial coordinates in your data.

Math-specific aggregates (19)

These are functions that will perform the mathematical computations for you.

*These functions take in a sequence (or an array) of values. The rest of the functions require a range index or collation.

Tuple meta-data (1)

This only contains the function cts:frequency.

Constructors (17)Lexicon reference functions (21)

These functions are often times used with the String and Scalar type-specific lexicon functions, as mentioned in the previous section.

Accessors (4)

Geospatial shapes and accessors (18)

Note that functions like cts:***-intersects and cts:***-contains are now deprecated. Switch to the geo library.

Most commonly, you use these shapes to construct geospatial queries. So first you construct a cts:region (using one or more of the above constructor functions). Then, you construct a geospatial cts:query (using a geospatial query function such as cts:element-geospatial-query), passing it the cts:region(s) you constructed. Finally, you pass the query to cts:search to run a geospatial search, or to a lexicon function to perform some geospatial-related analytics.

Sorting (7)

These constructors are typically used to specify which document information to use to “pre-sort” the response of cts:search, jsearch, and search:search.

Search result meta-data functions (6)

The result of a call to cts:search() is a sequence of nodes that reside in your database. But these node references also contain some special properties (five, to be precise) that extend beyond the XPath data model. They’re very handy for building search applications since they relate to things like search relevance:

Miscellaneous categories (23)

“Miscellaneous” is a popular category in my family’s monthly budget, but I digress. I’ll try to break down these last remaining functions into some sub-categories:

I’m not going to explain these (or fall on any swords defending their categorization). The important thing is that the cts API looks a lot less overwhelming to you now, right? There’s a hidden wisdom to it all—an underlying logic, a latent brilliance, a method to the madness…sorry, got a little carried away there.

Conclusion

Congratulations, you made it through the whole tour! As a reward, here’s a little code to look at. It’s the query I ran to generate the data for the Wordle shown at the beginning of the article. And, yes, it does use the cts API:

for $func-name in cts:element-attribute-values(xs:QName("function"),
                                               xs:QName("fullname"))
where starts-with($func-name,"cts:")
return
  concat($func-name,":",xdmp:estimate(cts:search(collection(),$func-name)))

And if you’re thinking to yourself that I must have a range index enabled on my database since I’m calling a value lexicon, you’re right. Well done.