Creating custom logs from NetFlow

Did you know you can create logs with any flow information and export it to 3rd party systems like SIEM. Check this post to see how to do it and what we have prepared for you.

When managing security information and events from many different systems, people are using SIEM (Security Information and Event Management) systems to collect this information and events from syslog messages and process them in one place. Flowmon can send syslog messages to these systems, but if you want to, you can define new types of logs based on filtering the communication.

These logs are created from flows, so the filtering should be enough sufficiently restrictive and should not return too many results. As Flowmon is processesing enhanced flow, it can send DHCP logs with useful information from a network, and in contrast with normal DHCP logs, these can be configured just how users need them to be. For this purpose, Flowmon provides you with scripts which will send customizable logs to the SIEM system.

Scripts downloading

Default scripts provide URL, DNS, DHCP reports and from Flowmon v10.00.05 also TLS reports. These scripts consist of 3 types of files. Msg_logger.py is sending logs to a configured SIEM system, the format_msg_*.sh files apply given filters on Flowmon databases and return prepared logs. Run_*.sh files provide easy running and scheduling of these scripts. Contact us for script download.

Unzip the package and upload tar.gz file to the directory

/home/flowmon and run the following command from this directory:

tar -zxvf scripts.tar.gz

[flowmon@internal ~]$ tar -zxvf scripts.tar.gz
./bin/
./bin/run_dhcp.sh
./bin/run_tls.sh
./bin/run_dns_query.sh
./bin/run_dns_response.sh
./bin/run_url.sh
./bin/format_msg_dhcp.sh
./bin/format_msg_tls.sh
./bin/format_msg_dns_query.sh
./bin/format_msg_dns_response.sh
./bin/format_msg_url.sh
./bin/msg_logger.py

The scripts will be unzipped to the

/home/flowmon/bin directory.

Customizing scripts to the local environment

Edit format_msg_*.sh files

format_msg_dhcp.sh
format_msg_dns_query.sh
format_msg_dns_response.sh
format_msg_tls.sh
format_msg_url.sh

Next, all of the format_msg_*.sh files need to be edited to correspond with your Flowmon settings and with your network environment. All the variables that need to be changed are at the beginning of these files.

You need to edit:

• PROFILE - the name of the profile you want the data from
• CHANS - the name of your channel
• MAX_LINES – the maximum number of results; be careful not to overload your SIEM system
• USER_FILTER - edit the filter to correspond with your network; see Creating custom scripts for more information
• Optionally edit PREFIX, SUFFIX, MACROS and NAMES; see Creating custom scripts

PROFILE="live"
CHANS="192-168-3-84_p3000"
MAX_LINES=1000
USER_FILTER="not hurl = \"\" and (src net 192.168.70.0/24 and apptag 3:80)" ​

PREFIX="FMC: CEF:0|FLOWMON|Flowmon Monitoring
Center|${VER}|URLREPORT|URL Reporting|1|"
SUFFIX=" deviceCustomString1=${HOSTNAME}
deviceCustomString1Label=ADSHostName"

MACROS=("sa" "ts" "hhost" "hurl" "sp" "dp" "ipkt" "ibyt" "flg" "da")
NAMES=("src=" "start=" "HTTPHost=" "URL=" "SRCPORT=" "DSTPORT="
"PACKETS=" "BYTES=" "FLAGS=" "targets:")​

Edit run_*.sh files

run_dhcp.sh
run_dns_query.sh
run_dns_response.sh
run_tls.shrun_url.sh

Next, some parameters in all of the run_*.sh files need to be edited to send logs to your SIEM system: 

Parameter

Description

Example

c

IP address of your SIEM system

-c 192.168.50.69

p

Optionally add udp port, default is set to 514 in msg_logger.py file

-p 514

logger “URL reporting started”
sleep 120
/home/flowmon/bin/format_msg_url.sh | /home/flowmon/bin/msg_logger.py -c 192.168.50.69 -p 5554

Running scheduled scripts

Open crontab and schedule the running of these scripts every 5 minutes. It needs to run every 5 minutes, because the scripts get data from backend files that are updated every 5 minutes.[flowmon@internal ~]$ crontab -e

# URL reporting to SIEM
*/5 * * * * /home/flowmon/bin/run_url.sh
# DNS reporting to SIEM
*/5 * * * * /home/flowmon/bin/run_dns_query.sh
*/5 * * * * /home/flowmon/bin/run_dns_response.sh
# DHCP reporting to SIEM
*/5 * * * * /home/flowmon/bin/run_dhcp.sh
# TLS reporting to SIEM
*/5 * * * * /home/flowmon/bin/run_tls.sh

Creating custom scripts

To create new custom scripts for logging to the SIEM, just copy and rename any of the existing format_msg_*.sh and run_*.sh files that are correctly running, and logging in your local environment and change any of the above mentioned settings. But be aware of the following:

The most important setting is the USER_FILTER in format_msg_*.sh file, which must be tested in the Flowmon Monitoring Center (FMC) on the Analysis page, as it needs to be in a format that the Flowmon backend can process. The part of the original command that can be processed by the Flowmon backend database can be found in the FMC - Analysis - Advanced Analysis - Statistics command.

To see the desired output from your filtered flows in the SIEM system, set MACROS and NAMES in the format_msg_*.sh file. Macros define the output in FMC Analysis and they can be found in the FMC - Analysis - Advanced Analysis - Statistics command. Names are user friendly names of these macros that will be displayed in your logs.

One important restriction is the number of results returned by the filter. Every line of the results is sent as a separate log, so the SIEM system can be very easily overloaded by these logs and some of them can get lost. To prevent this situation, test your filter in FMC - Analysis and be sure that it does not return more than a couple of hundred results during a 5 minute time frame.

After creating a functional and sufficiently restrictive filter to get only the most important logs, create a new scheduled job in crontab to run your script every 5 minutes.

We hope these scripts will help you create your own logs from flows you are interested in.