Introducing MarkLogic 11

With MarkLogic 11, we’ve introduced features to help you analyze your multi-model data in new ways, as well as more ways to integrate with your data ecosystem. And, we’ve made MarkLogic even easier to deploy, manage, and audit – including in the Cloud.

We’re continuing to double down on removing complexity and being the single place for breaking down data and knowledge silos – with a platform that just keeps getting more and more robust and easier to manage.

Analyzing Multi-Model Data in New Ways with Optic API Updates

The reality of MarkLogic as a multi-model database is that it’s actually limited on the query side by the languages that people may use. If you use SPARQL, you get graph. If you use SQL, you get relational.

The Optic API, that we first released with MarkLogic 9, is purpose-built for querying multi-model data. In MarkLogic 11, we’ve extended it to support the needs of our customers who do geospatial analysis. Optic now supports indexing and querying of geospatial data, scalable export of large geospatial result sets, and interoperability with GIS tools, including support for OpenGIS and GeoSPARQL.

We’re also giving customers the same multi-model capabilities, all-encompassing on the retrieval side and requesting side that we have already on the ingestion and storage side. This is an important development because it unifies all the capabilities of MarkLogic in one approach to query – one that is using standards, one that is accessible for developers, and one on which we’re going to base our future connector strategy.

Connecting to BI and Other Tools with GraphQL

As a data platform, MarkLogic needs to connect to the world. A number of BI and other tools out there have started to embrace the GraphQL movement, and what that means is they are able to fetch data from GraphQL endpoints.

We want to expose data in a way that’s readily usable by existing products. If organizations are using relational models, if they’re using GraphQL, they can get the data as they need from MarkLogic; they can do the analytics, the analysis, whatever it is they’re doing with the data.

The GraphQL capability builds on top of the Optic capability. Using the GraphQL endpoints, we can expose data modeled and queried via the Optic API in GraphQL. So, if you have relational views or query-based views where you’ve written a complex Optic query that projects out the data that you want to access, you can expose that as GraphQL APIs.

It makes developers’ lives easier and it’s part of the bigger picture of increasing the MarkLogic platform’s interoperability with existing tooling. We’re leveraging the powerful multi-model querying capability but exposing that via industry standard querying mechanisms like GraphQL.

Managing and Monitoring Growing Data Volumes

We know our customers’ data is not going to shrink. The data flows they’re dealing with are growing all the time, the volumes of their archives are growing all the time, and the data volumes that we must handle on their behalf are growing way beyond linearly. We saw a graph the other day at a customer site that was showing not quite exponential growth, but probably not far off.

So how do we accompany our customers on that quest for feeding more data into the system?

Some of it is simple; they need to add more hardware to add more nodes. But, actually, if you think about that, it means we need to improve observability, auditability, and manageability – so our customers can understand from health checks, from monitoring, that they’re at the limit. They need to have an easy-to-use user interface to manage those things. In MarkLogic 11, we’ve revamped the Admin UI to help with that.

What it also means is the ability to handle larger result sets at query time. The adaptive memory algorithms that we’ve added, external sort and joins, are playing in that area. It’s a theme that we’ll continue in MarkLogic 12 with, for example, visual TDEs that we’re investigating. These will allow complex queries for large data sets to be evaluated at runtime instead of indexing time to reduce the memory requirements.

From a platform perspective, we’re also reviewing tooling, which operating systems, how to deal with the modern architecture of CPUs, ARM support, hypervisors reviews.

Very few of our customers today are running on-premises on physical hardware. The vast majority of our customers are running on VMs, be they in private cloud or public cloud, and the nature of the storage and nature of the computing resources available is different. And that means that over time, we will be reviewing our best practices, our limits, our recommendations, and our default configurations.

In the meantime, in MarkLogic 11, we have improved observability – health checks and memory diagnostics – to help MarkLogic operators have better understanding of their clusters so they can be more on point maintaining their systems.

We’ve also added cold storage detection. As customers embrace cloud architecture, the storage is rarely on the node. They’re using some form of premium or non-premium storage from the cloud provider and that means latencies. That means network availability of the mounted drives. It means all sorts of things, and the availability of such volumes or the performance of such volumes may lead to degradation of service on the MarkLogic front. So we’ve added in MarkLogic 11 the ability to detect cold storage and to essentially exclude the node from the cluster.

MarkLogic Server has always been enterprise-class software with high availability, strong security features, and so forth. MarkLogic 11 continues to improve on that, adding this storage failure detection and failover along with additional security features that support our customers’ needs.

Ensuring Data Security

Security has always been a very strong area for us. It’s something that MarkLogic is synonymous with in many of our customers. We continue to invest in security improvement, including creation of backups, improved certificate management for intra-cluster communication, and OAuth support for managing encryption keys.

Our long heritage of being an enterprise-class data management platform means we will continue to enhance security, availability, monitoring, and other features that our enterprise customers rely on. We take their trust in us very seriously.

Supporting Cloud Strategies

At the time of release, MarkLogic 11 will be available as always in cloud formation templates and as Docker images.

Our customers also want to use modern deployment methods like containerization via Kubernetes. So, we’ve now begun to support Kubernetes, which is really important as a deployment and orchestration mechanism. We’re also basing our own SaaS service on this. So, we are serious about it and we will release enhancements all along the next year.

A Look Ahead

With the acquisition of Semaphore last year, we’ve entered a new era for MarkLogic. We will not only continue to invest in the “DNA” of our MarkLogic Server and Semaphore products, but we’re also focused on evolving our unified platform.

The MarkLogic 11 release includes new capabilities on both these fronts:

• Investing in the core MarkLogic Server capabilities of multi-model database, search, security, and other enterprise features
• Extending our data platform capabilities as part of our customers’ data ecosystems, with new GraphQL support and Optic API enhancements, along with our connector strategy

We’re moving to a regular cadence of releases that support both these “DNA” and platform goals, which will provide our customers with certainty around the support they’re getting, and let them better plan their project lifecycles.

I don’t want to go into too much detail about future releases, but we’ll continue to invest in capabilities that deliver stability, robustness, and reliability – and are focused on helping our customers solve real-life problems to benefit their business.