Search All Your Websites With a Master Data Hub

How many websites does your organization have? These days, it is common for one organization to own and manage multiple external websites. Take MarkLogic for example. We have our main website at marklogic.orbit-websites.com, developer community at developer.marklogic.com, and MarkLogic World at world.marklogic.com just to name a few, all serving one mission while targeting different audiences – business management, developers, event attendees, etc.

And like most companies infrastructure has evolved. These three websites run on different platforms, with siloed databases, providing a master search against them can be quite challenging. Like many companies, we have a tremendous amount of traffic that comes in from a “Google” search. But now that they are on the site, can they find all the information they are looking for? What if information resides on another external site — and not the one the visitors are on? You surely would not want them to go back to Google or go through each site of yours and conduct multiple searches — or worse, decide that you just don’t have what they are looking for.

To overcome this challenge and thus improve the end-user experiences of those visiting our sites, we knew the solution — build a master data hub on MarkLogic. I just had to have some free time to put it all together. I finally cleared the deck and got to it last fall.

NoSQL database MarkLogic makes it easy to bring separate siloed data into a single database and web developers managing multiple websites for a single organization or client should definitely take time to look into it. MarkLogic’s flexible data model allows data from different sources and formats provide a single platform for all your data.

Building a Master Data Hub

Our solution was to consolidate content from the different sites into a single MarkLogic instance and expose data back to each site via REST API where all search queries are made. Since MarkLogic indexes data as it is ingested and is completely transactional, any data flowed would be immediately available — so I didn’t have to worry about synchronizing databases.

My colleague Mike Wooldridge had built a MLPHP library – so integrating MarkLogic with websites or web applications built on PHP is extremely easy. However, since I was new to MarkLogic development, I did run into a few challenges including mapping different content types to different collections.

WordPress Plugin to MarkLogic

In another blog post, I will discuss technical details of our approach, what those challenges were, and how, we overcame them. Anyway… back to the solution part. Our developer site was already running on MarkLogic so it made a perfect sense to consolidate data there. We built a WordPress plugin that pushes data from the main site at www to developer site whenever new content is created or existing content is updated.

The end result is that visitors conducting searches on the main www site can now see results from the developer site, and those conducting searches on the developer site can see results from www site. Also, the search result page now has facets, making it easy to drill down on different content types, something that native WordPress search does not provide.

Sample search results page on marklogic.orbit-websites.com

Moreover, the WordPress plugin that we built is available to public for those who want to bring the same powerful search to your WordPress websites. The plugin has an option to replace native search with MarkLogic search for seamless integration.

What’s next for us? Consolidating the main site with developer site was just the first step. One of the challenges we still face is that the search does not include localized content. So we are looking at integrating our other public sites including the localized sites in German, Japanese, and French, into our data hub. We are also looking at fine-tuning the search results so that more relevant information is shown before others. All these customizations, fine-tuning, and data consolidation are possible now that the search is powered by MarkLogic. If this does not excite web developers out there like myself, I don’t know what would!

Stay tuned for a technical blog post on this topic which will be published on the developer blog. I will also add a link here when it is available.