New Enhancements for Semantics in MarkLogic 8.0-4

MarkLogic 8.0-4 Now Available

This type of release is not usually something I would write a blog post about—and technically, MarkLogic 8.0-4 is but another in the sequence of maintenance releases that round out the enormous leaps and bounds we made with the release of MarkLogic 8 last February. But we decided to pack a few extra things we’re really excited about that make it more than just an ordinary maintenance release.

There were a few key themes in MarkLogic 8 that we focused on such as improving the developer experience and making it possible to build more powerful, smarter applications (see those themes here). This release includes enhancements and new features that focus on those same themes.

Here are some things to keep your eye on in 8.0-4:

• The launch of JSearch, a higher level JavaScript API that makes common functionality like full-text search, faceted navigation, and aggregates more easily accessible. MarkLogic University has an on-demand course on JSearch that provides a great intro
• Improvements to the Management API with more RESTful calls to cover all of MarkLogic’s configuration and administration capabilities
• Enhancements to MarkLogic Content Pump (mlcp) to automatically convert CSV files into rich, semantic JSON documents (think: converting relational data or spreadsheets into documents)
• Improved Geospatial support by doubling the number of geospatial functions available, support for the latest standards (GML 3.2), and improved support for more complicated query operations (region comparisons for large numbers of polygons)
• And… Semantics! This was a big focus in the 8.0-4 release, and we discuss the details below

I encourage you to download MarkLogic 8.0-4 or read more about these features in the release notes.

Continuing Improvements for Semantics

We built a very strong semantics foundation in MarkLogic 7, and the initial MarkLogic 8 release added major enhancements to make semantics more powerful with the addition of semantic inference, and easier to use with the capability to handle the most up to date SPARQL query standards. MarkLogic 8.0-4 continues with the same trajectory, with a particular focus on improving the APIs.

Stephen Buxton, our Senior Director of Product Management, told me:

“As a whole, the new semantics features in MarkLogic 8.0-4 make it easier for developers to build semantic applications and for those more business-minded to use semantic tools to work with and analyze data stored in MarkLogic. We now have faster performance, full support for SPARQL 1.1 including new operators, and also support for the standard Java APIs for semantics—Jena and Sesame.”

These enhancements mean that developers have Java APIs that are easy to use (and may already be using), and they make it easy to integrate MarkLogic with third-party semantics tools provided by companies like Semaphore, Cambridge Semantics, and Pool Party. We also extended the APIs to include unique MarkLogic capabilities such as the ability to do SPARQL queries combined with contextual queries (e.g. full-text, geospatial, ranges, and/or bitemporal).

I remember when we first released MarkLogic 8, Broadridge’s head of architecture said, “Early testing has determined that MarkLogic 8 will be a game changer for us as well as the market as a whole.” MarkLogic 8.0-4 continues to deliver on that.

Semantics Now Integrated With Samplestack

Semantics can be an abstract topic to grasp at first, so in addition to adding new features, we also worked on providing a new way for developers to see how semantics works in practice. With MarkLogic 8, we released a sample application, aptly named “Samplestack,”.

Samplestack: An end-to-end three tiered sample application demonstrating how to implement a reference architecture with sample code.

The Samplestack 1.1 release now shows off one of the many ways in which semantics can be implemented in a working application. When users search in the Samplestack app, they can now navigate to related content easier by clicking on related tags that are suggested based on semantics.

Without semantics, users may get stuck if their search does not return an exact match for what they typed in the search box or returns a random list of seemingly unrelated things (just recall some past Google searches that ended in frustration). Semantics solves this problem by providing related tags that users can click on that are generated based on semantic relationships between tags the user selected and other words that users may not have known or even thought about.

How It Applies to Users

I talked to Kasey Alderete, the Product Manager who handles Samplestack development, and she explained it like this…

“Imagine I am using the Samplestack app and I do a search for ‘get value from string.’ This result brings back 200 results, but I don’t initially see answers relevant to my problem. I add a JavaScript tag (because I see things like PHP in the results which don’t apply to me), and this search brings back 115 results, but I still don’t see my answer. So, I check ‘Related Tags,’ which brings up ‘JSON’ as a suggestion. This revised semantic search further narrows the result set to 26 answers that are much more relevant.”

“In other words, I was able to search along a completely different yet related axis of information that didn’t even occur to me. Picture a Venn diagram with two circles, one for JavaScript and one for JSON. I may be searching in the JavaScript circle, when the content that best answers my question is in the JSON circle outside of the JavaScript circle. Knowing these two circles are semantically related allows me to shift my search to find the desired content.”

How This Works Under the Covers

If you’re a developer, now you’re probably wondering how this functionality is built using MarkLogic. Kasey added…

“Performing a semantic search is demonstrated in Samplestack by using a Server-Side JavaScript service extension. The Samplestack application loads data from DBpedia, which is expressed as RDF triples. These triples describe the relationships between tags. The extension executes a SPARQL query against the DBpedia data requesting related tags for a given tag.”

Here is a triple from Kasey’s example:

<http://dbpedia.org/resource/Category:JSON> 
<http://www.w3.org/2004/02/skos/core#broader> 
<http://dbpedia.org/resource/Category:JavaScript>

So that’s it! Developers can see now all of these parts working together in a full MarkLogic application. It’s freely available on GitHub and it only takes about 30 minutes to setup (and that includes getting a fresh download). For more details on MarkLogic 8.0-4, refer to the release notes in our docs.