Hacking Monitoring History

Monitoring History is a browser-based application that ships with MarkLogic Server and lets you visualize important performance metrics. It displays the data as interactive line charts created with the Highcharts JavaScript library:

Host-specific charts give you insights into your disk I/O, CPU, and memory performance. Charts for application servers tell you the history of your request rates, latency rates, and cache accesses, among other things. Database charts let you track the number of document fragments, the sizes of different data directories, lock rates, and more.

If you’re running a local MarkLogic instance, you can access Monitoring History here: http:// localhost:8002/history

Monitoring History leverages MarkLogic’s powerful Management API for retrieving some of the hundreds of performance metrics that are tracked in MarkLogic (when you have performance monitoring turned on). When designing Monitoring History, we selected metrics that would be the most helpful to administrators managing a MarkLogic cluster. But we also wrote the JavaScript code so that it would be straightforward to add new types of charts in the future (for example, if new metrics were exposed by the Management API).

If you’re willing to edit some configuration information, you can hack your Monitoring History application to add new charts for any of the hundreds of the metrics that the Management API exposes. You can also adjust the look and feel of the charts in the application. The main configuration file for Monitoring History is located here and is the file that we’ll be editing in the examples that follow:

Note that if you have a MarkLogic cluster, this configuration information is host-specific, since it lives on the file system. This means that if you edit configViews.js on node1 and look at Monitoring History on node1, you will see monitoring data for the entire cluster based on the updated configuration. However, if you look at Monitoring History on node2, you will see the default configuration.

Adding a New Chart

First, we’ll go over adding a new chart that tracks Environment Program Cache metrics across all servers in the cluster (suggested by Erin Miller, Lead Performance Engineer at MarkLogic). The Environment Program Cache records hits and misses that occur in the module cache for ad-hoc XSLT calls.

To view the server metrics you can choose from when adding a chart, access the Management API endpoint here:

http://localhost:8002/manage/v2/servers?format=html&view=metrics

(You can edit the URL above to view metrics for a different resource type by changing servers to hosts, databases, or forests.)

The server metrics we want for our new chart have the IDs env-program-cache-hits and env-program-cache-misses, and we will reference those in our configuration information.

Where will we display the new chart in the application? The collections of charts in Monitoring History are organized into views. In the configViews.js file, you can add a chart to a view by adding a new object under that view’s ML.config.views property.

We’ll add the chart to the Overview, which is the first view you see when you open the application. The Overview is configured under ML.config.views['configOverview']. To add the new chart for our metrics, we add a new property to that view’s charts object.

Here’s the code to add:

envProgramCache: {
  title: 'Environment Program Cache Hits/Misses',
  category: 'serversMore',
  categoryTitle: 'Servers (Environment Program Cache Data)',
  metricsType: 'servers-metrics',
  type: 'summary',
  series: [
    {name: 'Hits', metric: 'env-program-cache-hits'},
    {name: 'Misses', metric: 'env-program-cache-misses'}
  ],
  detail: 'configServer',
  ajaxIndex: 1
}

Here are what the different properties control:

With these properties added, the application has all the information it needs to retrieve the metric data from the Management API, build the chart using the Highcharts library, and place it in one of the views. Here’s what the new chart looks like when we refresh the browser:

If you browse the configViews.js configuration file, you can see that some charts have additional properties that control different aspects of their display. For more information about the available properties, see the comments at the top of the configViews.js file.

Changing Chart Heights

Besides adding charts, you can change the appearance of existing charts. Say you want to increase the height of certain charts to get better resolution of the lines. To do this, you can edit the dimensions property in the configViews.js file. This property can have the value full, fullTwoThirds, or fullHalf. By default, all the charts in all the views are displayed with fullHalf dimensions:

ML.config.views['configOverview'] = {
  title: 'Overview',
  ajax: {  // AJAX settings…  },
  type: 'overview',
  detail: '',
  dimensions: 'fullHalf',
  charts: {
  // etc.

We use the fullHalf height so that the viewer can see multiple charts at once in most monitor views. You can change the dimensions value to one of the other options (fullTwoThirds or full) to get larger versions of all the charts in a view.

You can also add a dimensions property to a specific chart. For example, you can make the first chart in the Overview large (full) and keep the rest of the charts the default smaller size (fullHalf). Here’s what the code looks like:

ML.config.views['configOverview'] = {
  title: 'Overview',
  ajax: {  // AJAX settings },
  type: 'overview',
  detail: '',
  dimensions: 'fullHalf',
  charts: {
    labelsChart: ML.configLabels, // Labels chart on top
    diskIOChart: {
      title: 'Disk I/O',
      dimensions: 'full',
      category: 'disks',
      categoryTitle: 'Disks',
      // etc.

And here is the larger Disk I/O chart that appears when we refresh the browser:

For complete control over your chart dimensions, you can specify your own custom dimensions in the config.js file located here:

The dimension settings are defined under the ML.config['UI'] property.

Updating Chart Styles

You also have control over the look and feel of charts in Monitoring History with your configuration file. The application uses the Highcharts JavaScript library to display the charts, and you can change chart styles by adding or editing Highcharts options associated with each chart.

You can see an example of these options at the top of configViews.js, where the styles for the Labels chart are defined in the chartOpts property of ML.configLabels. These options set spacing and margins, turn off the chart legend, activate tool tips on the chart lines, and more.

You can add a chartOpts property to any other chart to change that chart’s styles. For instance, to change the background color and thicken the line width for the Disk I/O chart that we just enlarged, you can add the following chartOptsproperty:

diskIOChart: {
  title: 'Disk I/O',
  chartOpts: {
    chart: {
      backgroundColor: '#000000',
      plotOptions: {
        line: {
          lineWidth: 3
        }
      }
    }
  },
  dimensions: 'full',
  category: ‘disks’,
  categoryTitle: ‘Disks’,
  // etc.

These style options are merged with a default set of Highchart options defined in the application’s config.js file. The resulting chart looks like this:

For more ideas on how to tweak Highchart styles in Monitoring History, check out the Highcharts documentation.

Conclusion

These examples give insight on how Monitoring History creates the charts for visualizing MarkLogic performance metrics. By updating the main configuration file for the application, you can add charts or change how existing charts appear.  All you need to do is edit some JavaScript object properties.

Note that changes you make to configViews.js will not survive if you upgrade your MarkLogic installation, so save a separate copy of your edits if you want to keep them.

You can download a configViews.js configuration file with the edits described in this article. Search for “Hacking Monitoring History” in the file to find the edits.

Happy hacking!