MarkLogic World Hackathon: Building a Data Hub in the Cloud

Building on last year’s momentum, we held our annual hackathon again at MarkLogic World 2019, the annual conference for MarkLogic users.

Held in the conference’s Developer Lounge, conference attendees met after two days of information overflow and buckled down to show off what they could do with a MarkLogic Data Hub. Teamed up with strangers, Hackathon participants got their hands dirty by extending a data hub and deploying their application to the new MarkLogic Data Hub Service, while learning a ton along the way. We wrapped up late in the evening with presentations, judging, and prizes. Missed it? There’s always next year!

The Challenge

We gave teams an existing data hub with data about Fortune 500 companies and products and social media feeds. The teams got creative by further enhancing the data and digging deeper into data hub features. In front of their peers, teams demonstrated their solutions through a REST extension hosted on the MarkLogic Data Hub Service.

We placed participants onto teams of three, separating colleagues so participants could meet new people. We also distributed MarkLogic expertise so those with more experience teamed up with folks new to the product. Then, we paired teams with a MarkLogic mentor who would guide them through brainstorming, collaboration, and troubleshooting. There were MarkLogic representatives floating around the rooms, ready to help and answer questions at the raise of a hand.

Some of the participants never worked with MarkLogic before, and many were new to the idea of a data hub. And what a better way to learn than by sitting side-by-side with the folks responsible for engineering it?

“It’s a great opportunity to get some hands-on practice with new features while meeting other members of the developer community” said David Cassel, MarkLogic Developer and Architect at 4V Services, LLC.

Curious what the participants came up with?

And the Winner Is…

Team SHK, comprised of Stephen Brooks from Avalon Consulting, LLC, Hannah Sporer from Selective Insurance, and Koushik Chatterjee from Deloitte, took home the coveted first place prize. They tackled the hackathon challenge head-on, further enriching the Twitter data they were provided with sentiment analysis. They used the third-party MeaningCloud API to determine if the tweets were generally positive or negative. They gave a demonstration of their working custom REST endpoint deployed to the MarkLogic Data Hub Service.

Team SHK built a data service that served two purposes. First, given a location input, the service returns nearby companies and sentiments about that company from local social media users. This answered several business questions. Are people generally happy about Company X nearby? Would this be a good area to expand into or release a new product into? Second, given a company, the service provides sentiments from tweets about that company and the corresponding locations. In other words, where is Company Y successful with user interactions?

Figure 1: Given location or company inputs, Team SHK’s data service returned relevant information about companies nearby and sentiment analysis of the tweets.

Figure 2: Given a tweet, the MeaningCloud API returns sentiment analysis.

Figure 3: Within the data hub, a harmonize flow combines the twitter data with the sentiment analysis.

Given more time, Team SHK would have enriched their data hub with weather-related data, harmonizing the local weather at the time of the tweet, to filter sentiments based on weather and determine the weather when comments were made. One of their goals was to produce actionable business data, such as determining what time of year and in what weather conditions did a product became popular. For example, if a drink manufacturer found that a particular beverage was really popular on hot, humid, summer days, sunny scenes may be particularly relevant in advertisements for that beverage. In addition, Team SHK would have liked to implement a date range restriction so that the data service would return results within a date range.

They won based on their creative solutions in addressing business needs. In particular, they used MarkLogic 9 features and Data Hub concepts and functionality to create a data service, and of course, deployed the hub and service to the MarkLogic Data Hub Service.  Great job, Team SHK!

Figure 4: Team SHK (from left to right) with their MarkLogic mentor Ruth Stryker far left: Stephen Brooks, Koushik Chatterjee, and Hannah Sporer.

We also want to give a shout out to the second place team, Team Craft Data Hub, which included Gabriel Aviles, Orbis Technologies, Inc.; David Cassel, 4V Services; and Bruce Hecht, IEEE. The team came up with the concept for “a craft beer data hub” after enjoying a few at the Hackathon! They wanted to answer questions about a particular beer of choice, such as: what company made this beer, and what other products do they produce? They augmented the data given with Fortune 1000 data and deployed to the MarkLogic Data Hub Service as well. Pretty impressive!

We’re already thinking about MarkLogic World 2020 and how to make the Hackathon experience an even better one. As always, our goal is to combine learning with a fun environment. One of the members of the winning team told us after: “It was great having coaches and others to help answer our questions.  Before the hackathon, I did not realize how straight forward it was to enrich data using microservices or online APIs.  I think this is a powerful feature that I’ll use again in the future.”

Are you up for the challenge?

Want to try your hand at the challenge?

Download the tech:

• MarkLogic 9.0
• Data Hub Framework v4.3.1
• Data hub shell (.zip)

Take the courses to get up to speed:

• Using the MarkLogic Data Hub
• Data Services
• MarkLogic Data Hub Service Administration

And see what you come up with! Want to deploy to the MarkLogic Data Hub Service too? Go for it!