Flowmon ADS integration with Elasticsearch

ElasticSearch gathers more and more enthusiasm on the IT market. Released versions of ElasticSearch put the project into the group of most important solutions in Open Source community. Growing number of leading market companies decide to learn more about the solution what becomes a real alternative for Big Vendors products.

The reason if obvious. Most important feature of ElasticSearch is to aggregate large number of events like IT system logs. On the top of it we can easily scale the system, search and visualize our data. These features put together show that there is little competition of the market. What’s more, there are companies on the market that have build well featured products focusing on IT system logs. Solutions like X-Pack, Energy Logserver add highly expected features to already comprehensive solution. Most important is to cover security aspects in our logs. OpenSource ElasticSearch present all the data to whoever asks. X-Pach and Energy Logserver allow creation of access roles with user authentication fulfilling the security policy. Other premium features are: Alarming and Reporting.

Having all system logs in one place with being able to fire an alarm for various criteria is a major step forward in management of IT security. Unfortunately logs are very static and do not show dynamic changes in our networks in the area of security.

Let’s look at the SIEM offerings on the market. All Out-Of-Box SIEM products perform similar task: they concentrate on two main groups of data sources – IT logs and network traffic. What we can learn from this approach? We can add Network Behavior Analysis to our existing Elastic ecosystem.

Flowmon ADS is a perfect choice for that. ADS is a comprehensive security module that work on the top of Flowmon Collector. ADS stands for Anomaly Detection System. All the security rules use live network traffic out of netflow or IPFIX data. Embedded rules work on several fields:

• Network errors and misconfiguration detection – ADS will fire an alarm whenever RFC traffic is recognized, misconfigured IPv6 are working in the network,  strange broadcase packages are detected … and many more
• “Zero day” exploits detection – ADS use network behavior to learn what is typical and what is not in terms of: IP addresses, ports, utylizations, country reputation, applications, flows etc. If we learn what is normal, we can throw an alarm when we see the risk of change
• Unwanted traffic detection – ADS learns about the network and all new application and protocols are a question of security. If one start mail exchanger in Your network, ADS will automaticly initiate Alarm saying where the traffic came from, what is the IP, MAC and VLAN for it.

Flowmon ADS is a new approach for Network Analytics market. We take the attention from the network packages itself and focus on security aspects. We’ve learned that our customers do not have resources for time consuming traffic filtering in Netflow Collector tools. We deliver the incidents with its priority, risk and evident.

Figure 1: List of event types in Flowmon ADS

That is a real value for the customer. So what about joining two systems together? Can we correlate ELK LogManagement and Flowmon ADS events together? ADS events will be enriched with the “end results” of the traffic. In one place we should see Application communication and Application logs what could lead us to new conclusions. Let’s try!

Flowmon can easly export events to SIEM products using CEF format. On other hand, Elastic can read that using Logstash CEF plugin https://www.elastic.co/guide/en/logstash/current/plugins-codecs-cef.html. So let’s search !

Figure 2: Flowmon ADS event search in

All the data is there. We see events categorizations, amounts, src, dst, ports … all in proper Kibana fields. That allows performing searches, aggragations and visualizations. Look at the result:

Figure 3: Visualisation of collected ADS events

If we know how searching in ELK works we immediately see the benefits. Query, subquery, drilldown, trend analysis – that is all it is about.

Once we enable premium features like Alerting from X-Pack and Energy Logserver we can configure Alerts on the top of ADS incidents – checking the possible impact of the network security issue and OS/Application logs. This information can be passed to SOC with a proper PDF report or a scripted action.

Figure 4: PDF Kibana report from ADS data

So where we are now? Elastic gives us large scale for central log management, Flowmon ADS adds dynamic security feed into static log view. We can immediately initiate actions to SOC and Ubrella systems. Do we still perform LogManagement or we touched SIEM? The answer is up to you!

ABOUT PARTNER

EMCA SA is full-service IT company specializing in a IT infrastructure monitoring, network monitoring and big data analytics, machine data and enterprise security. EMCA SA provides software licensing, deployment and professional services, and post-sale support.

Flowmon Networks Gold Partner