Microservices, Persistence: Benefits and Risks

There’s a growing interest in Microservices and Martin Fowler describes how microservice architecture is a particular way of “designing software applications as suites of independently deployable services.”

The promise of microservices is it allows you to continuously increase the capabilities of your public site, or internal systems, by rolling out small, defined bits of functionality on a regular basis. That’s in contrast with making changes in large monolithic applications where a change may be harder to engineer, test, and deploy. I think there is a lot of power in the microservice message — but there are some potential pitfalls, such as polyglot persistence.

The good part of polyglot persistence is that each service can best determine the most effective way of storing its data; anything from static text files to semantic databases. This is good in the sense that you want the right tool for the right job. In surprisingly many cases, trying to shoehorn everything into a relational database does not work. On the other hand, having a unique storage or database solution for each service becomes a maintenance nightmare. In the past, it was common to have a database dedicated to an application, and this lead to a proliferation of databases that many enterprises are still consolidating. It can even be a challenge to inventory what application has which data, whether it’s the authoritative source, and (if not authoritative) how stale is the data?

Bad Persistence

Different data types go into different databases. Many Problems (like HA) have to be solved for each database. Multiple security models for each database. Hopefully you can get support for each database.

How to Handle Polyglot Persistence

I recently completed a proof of concept for a customer that allowed us to quickly create data-rich, enterprise-dashboard views from multiple, different data sources. Some views contained data from several services, which imported the data from relational database extracts, PDFs, and even semantic data. All that heterogenous data went into one MarkLogic database. The small, composable set of services built on MarkLogic made us agile and productive.

Because the data was now in one place, it was no longer stove-piped. Before, useful data was trapped in systems (like their enterprise application modeling tools), which may not seem like a database but are rich with information about their enterprise systems. Now, all that data is accessible through a set of services that are micro (in the sense they are small, contained, and independent of each other), but with the benefits inherent in consolidation. The data is one place and we can indicate its provenance (where it came from and when).

Good Persistence

Multiple types of data go into the MarkLogic Database, which is clustered, highly available and supported. Many problems (like HA) are solved jut once. One consistent security model; one vendor support contract.

Rather than equating the application’s need to store data in a specific format with a specialized database solution to support that format, MarkLogic offers a flexible data model foundation where different types of data can be stored in the same system. XML documents (of course!), but also JSON documents, binary documents, text, documents, semantic triples, and even relational. Rather than having multiple solutions for each type of data, MarkLogic just sucks it all in. It’s polyglot data (a good thing) rather than polyglot vendors or polyglot implementations (bad things).

If that company had picked a different vendor for each type of data (let alone for each microservice) for its project, it would be one way to guarantee its IT costs would bloom. Answering a business need often allows us to forget the long and short-term costs of “just” adding another database and/or service. But that service becomes your legacy infrastructure the minute it goes into production. Instead think of microservice and polyglot data without being polyglot vendor. The goal is to create something that will be a cornerstone for new functionality, lowering costs, and driving revenue, as opposed to being a millstone grinding on your budget.