Why MarkLogic Won the Best Data Lineage Solution Award

I’m delighted to write that on 7^th December 2018, MarkLogic was recognized as the Best Data Lineage Solution in the 2018 Data Management Review Awards. It’s a real honour to receive this award, and it also presents an opportunity to explain data lineage, its use in financial services and why MarkLogic® is indeed the Best Data Lineage Solution.

The Importance of Data Lineage

Data lineage is about tracing the lifecycle of data – where it originates from, the path it takes and the transformations it undergoes. It’s all providing visibility and ultimately about whether data can be trusted. It’s a critical component of any effective data architecture.

Financial services firms have been having to deal with a plethora of new regulations over the last 10 years. Although the costs of complying with new regulations may have peaked, updates to existing and new regulations will continue to come into force. Understanding the lineage of trades, models, client communications, payments, etc. is vital to not only staying compliant but staying compliant effectively. Too often, when the phone call from the regulator comes, organizations struggle to put together the information needed. This process always involves data lineage – tracing back from symptoms to cause. With data lineage built into a data architecture, this process is accelerated and more accurate.

MarkLogic and Data Lineage

Data lineage is built into MarkLogic from the ground up. Original data is imported “as is” and can then be progressively transformed using “envelope” patterns to enable and maintain data lineage. The original data is stored, and transformations are kept as metadata. The entire lifecycle of every piece of data can therefore be replayed and examined.

This process avoids the brittle process of ETL (Extract-Transform-Load) that is a characteristic of conventional relational systems, and that causes such difficulties when understanding data lineage. For example, was a specific data value part of the original record, or was it added during a subsequent ETL step?

Data Lineage in Action

Many of the largest global investment banks use MarkLogic for post-trade processing and regulatory reporting, and one case in particular helped MarkLogic win its category in the 2018 Data Management Review Awards. ABN Amro selected MarkLogic because of its ability to integrate complex trade data across asset classes and several trading systems quickly and cost-effectively.

ABN Amro needed a database that could provide a fully integrated, consolidated view of data to provide a single source of truth for reporting purposes as well as to provide alerts when behaviour is outside normal parameters. MarkLogic filled the bill because its fast development could support the bank’s timelines. Additionally, MarkLogic’s in-built lineage capabilities allows ABN Amro to minimize risk through “tech time travel”– rewinding the database and examining trade lineage. Furthermore, ABN Amro can more easily comply with future regulations as all required data is in one place, which provides a uniform and consistent view of orders and trades.

As Jaap Boersma, Principal Architect for Capital Markets Solutions, ABN Amro, said, “We need the ability to respond quickly to changing regulatory requirements. We chose MarkLogic because trade data is notoriously difficult to handle in relational databases. We found the breadth of MarkLogic’s multi-model database and associated features, such as security and ACID transactions, to be compelling.”

Thank You!

We at MarkLogic have worked hard to create a trusted, enterprise-class NoSQL database with data lineage built into its core. We are grateful that our revolutionary approach to Big Data challenges has helped so many of our customers. We are proud to be recognized as the Best Data Lineage Solution by the 2018 Data Management Review Awards.

To find out how we can help your organization build a trade data hub, please visit our website, or contact us by sending an email to tradedatahub@marklogic.com.