Integrate Audit trail module with Elasticsearch and Kibana

The default logging module of Sitefinity CMS is JSON logger, but you can also integrate Sitefinity CMS Audit trail module with external logging tools.

Elasticsearch is an open-source Java-based web server, used by the Audit trail module for storing and querying logged records, instead of using the textual log files.

Kibana is Elasticsearch web server that uses static HTML pages and client-side AJAX calls for querying and visualizing records from Elasticsearch in the form of advanced reports and custom dashboard screens.

• Elasticsearch is a Java-based server, which requires Java Development Kit (JDK). Download the JDK from the Oracle Downloads page and install it.
• You set the Path Environment variable. For more information, see the Oracle JDK installation instructions.

To integrate with Elasticsearch and Kibana servers, you must install the ElasticSearchAuditTrailLogger module. You do this by installing the Telerik.Sitefinity.Audit.Elasticsearch NuGet package from Sitefinity CMS NuGet Repository.

During setup, you need to run the Elasticsearch server on the web server machine, on a dedicated machine, on premise, in the cloud, or in a cluster of machines.

1. Download and install Elasticsearch using the Elasticsearch 7.12 Installation Guide.
   
2. Open the ~\<ElasticsearchPath>\config\elasticsearch.yml file for editing.
   
3. Locate the Network section, remove the comment tag and change the port to the one you want to use.
   

1. In Sitefinity CMS backend, navigate to Administration » Modules & Services and install the Audit Trail – Elasticsearch module.
   
2. Navigate to Administration » Settings » Advanced » ElasticsearchAudit.
   
3. In the ElasticsearchUri field, enter the root URL of your Elasticsearch site.
   The default value is http://localhost:9200/.  
   

NOTE: If you no longer need JSON logging, navigate to Administration » Settings » Advanced Settings » Audit and disable the default JSON logging by deselecting the EnableJsonLogging checkbox.

RECOMMENDATION: Because Sitefinity CMS Elasticsearch audit logger invokes Elasticsearch RESTful services synchronously, to improve performance, we recommend that you ensure minimum network latency between your web server and your Elasticsearch server. If this is an issue, you can install a dedicated logging agent - for example, Logstash, on the webserver machine, to provide out-of-process buffering and persistence of the logged messages.

1. Download and install Kibana using the Kibana 7.12.1 Installation Guide.
   
2. Open the ~\<KibanaPath>\config\kibana.yml file for editing.
   Locate the elasticsearch.hosts setting and set it to the root URL of your ElasticSearch server.

NOTE: For more information, see Kibana Setup Guide.

3. Navigate to the ~\<KibanaPath>\bin folder and open the kibana.bat file.
   This file hosts Kibana on the port set in the configuration file. 

NOTE: The default value of Elasticsearch server is 9200. You can also set another port to be used by Kibana in the server.port setting in the kibana.yml file. The default value is 5601.

Open your hosted Kibana site and configure an index pattern. 


For more information, see the Kibana’s tutorial Define your index patterns.

NOTE: Because Kibana is a client-side HTML application, which invokes the Elasticsearch REST API directly from the client's browser, the Elasticsearch server must be accessible from the machine of every user who is allowed to view the audit log. To avoid unrestricted access to the audit log (either through Kibana or just by calling the REST services of the Elasticsearch server), employ some network protection mechanism like a firewall (for example, white listing user IPs) or a dedicated VPN. In addition, make sure that these restrictions do not prevent Sitefinity CMS from connecting to Elasticsearch.

Elasticsearch can usually be upgraded using a rolling upgrade process so upgrading does not interrupt service. For more information, see Elasticsearch rolling upgrades.

Elasticsearch can read indices created in the previous major version. If you have indices created in 5.x or before, you must reindex or delete them before upgrading to 7.12.1. Elasticsearch nodes will fail to start if incompatible indices are present. Snapshots of 5.x or earlier indices cannot be restored to a 7.x cluster even if they were created by a 6.x cluster. For information about upgrading old indices, see Reindex before upgrading.

Upgrade to 6.x guide

This is is a sample guide for the upgrade procedure to version 6.x.

1. Back up your data.
   For more information, see Snapshot And Restore.
   
2. Address any 6.0 breaking changes that affect your applications.
   For more information, see Breaking Changes.
3. Check the Elasticsearch dependencies log.
   
For more information, see Configuring Elasticsearch.

4. Stop sending data to your cluster.
   
5. Shut down your cluster and install Elasticsearch 6.x on all nodes. 
   For more information, see Full cluster restart upgrade.
   Note that X-Pack is installed automatically with the default distribution from 6.3 onward.
   
6. Restart your Elasticsearch cluster.
   
7. Create a new index using:
   PUT /<new_index>
8. Reindex the old index using the following code:
   POST _reindex
   
9. Stop the Elasticsearch 6.x server.
   
10. Copy the new_index from <elasticsearch-6.8.0_path>\data\nodes\...\indices to <elasticsearch-7.5.1_path>\data\nodes\...\indices.
    
11. Start the Elasticsearch 7.5.1 server.
    
12. Reindex the new index using the following code:
    POST _reindex