How to Optimize Cloud Monitoring Costs Using Flow Logs in Progress Flowmon

This blog post discusses some of the best practices for balancing the costs of cloud traffic monitoring while maintaining a reasonable level of visibility. Progress Flowmon 12 has introduced the processing of native flow logs from Google Cloud and Microsoft Azure, plus it has enhanced support for Amazon Web Services (AWS) flow logs. This opens up interesting options for reducing the costs of your cloud traffic monitoring by leveraging flow logs in parts of your cloud infrastructure where a reduction in visibility is not an issue.

Why does cloud traffic monitoring need optimizing?

The simple fact is that cloud infrastructure providers charge by the amount of traffic transferred, and it makes no difference if it’s your actual data or monitoring metadata. At the same time, you still want to maintain a certain level of visibility into your traffic because by losing visibility, you lose control over your environment.

To prevent your cloud infrastructure costs from ballooning due to monitoring, you want to achieve a balance where you keep the overall amount of traffic for monitoring purposes to a minimum without compromising on visibility.

What options does Flowmon offer?

Flowmon 12 improves and expands flow log analysis support across the primary public cloud services. Flow logs are a form of network telemetry available natively in public cloud environments. Once configured, system administrators can use Flowmon to analyze flow logs to provide visibility into cloud traffic. This can be done across multiple cloud providers and incorporate on-premises infrastructure to support hybrid multi-cloud monitoring for much less cost than using separate native monitoring options.

Figure 1 below shows the overall concept of a hybrid deployment with a Flowmon Collector as a central aggregator gathering metadata from Probes (in this case both hardware and cloud, although only cloud ones will feature in the later discussion) and native flow logs.

Figure 1 – A sample hybrid deployment utilizing native flow logs, cloud traffic mirroring, and on-prem traffic

The Flowmon Collector can be both a hardware, virtual or cloud appliance, and standard sizing rules apply. A hardware Collector or virtual Collector deployed on-premise will reduce costs for computing resources in the cloud but will come with additional network egress costs. In the case of a cloud appliance, the situation gets reversed. Choosing one over the other will depend very much on the circumstances of your specific deployment.

Your deployment should include the Flowmon probes if you need:

• Highly reliable and accurate data for advanced network analytics and network performance metrics.
• In-depth visibility into network traffic, including application-layer visibility (L7 protocols).
• Advanced functionalities provided by Flowmon Application Performance Monitoring (APM) and Flowmon Packet Investigator (FPI).

However, operating Flowmon Probes in the cloud comes with the following caveats:

• You need either a cloud-native or 3rd-party vTAP/mirroring/brokering solution, which will introduce additional costs proportional to the volume of data passing through the monitored infrastructure.
• Running a Flowmon Probe in the cloud consumes computing resources and will incur extra costs.
• Depending on the cloud zones and regions, deployments may need multiple Probe appliances to service the monitored infrastructure.

In places where deep visibility and advanced features are not necessary, you can leverage native flow logs, which are very lightweight and consume much fewer resources.

Balancing resource consumption and visibility in Google Cloud

The following section focuses in more detail on the particularities of Google Cloud infrastructures. The cloud requirements for Azure and AWS will be very similar.

Figure 2 below shows a sample deployment with Google-specific terminology.

Figure 2 – A sample deployment with compute instances and cloud logging

In Google Cloud’s terminology, this is a Shared VPC deployment, where the cost of monitoring is impacted mainly by regionality or the deployment zone of resources. For this type of deployment, you will need the following services:

• vTAP/Mirroring – Google Cloud VPC Packet Mirroring
• Native Flow Logs – Google Cloud VPC Flow Logs
• with Google Cloud Logging and
• Google Cloud Pub/Sub

Packet mirroring and Probes

In a shared VPC deployment in Google Cloud, a copy of the network traffic from selected compute instances gets sent to an internal TCP/UDP load balancer and from there to one or more Flowmon Probes. You can use filters to mirror only parts of this traffic.

The mirrored traffic is counted toward Mirrored VM’s egress network traffic, and this needs to be considered during instance sizing. Your Probes must be deployed within the same region as Mirrored VMs (ideally in the same zone) to reduce these network egress charges.

Thus, the overall cost consists of:

• overprovisioning of egress network bandwidth for Mirrored VMs,
• internal TCP/UDP load balancer charges,
• cloud Compute charges for Flowmon Probe instance(s),
• network egress charges for mirrored data,
• network egress charges for flow data.

Flow logs

Flow logs are generated in subnets and published by logging to the pub/sub messaging infrastructure. Flowmon Collector appliances are subscribed to that infrastructure to retrieve and process the flow data. Hence their overall cost consists of:

• flow logs generation charges,
• logging charges,
• Pub/Sub charges,
• network egress charges.

Collectors

In Google Cloud terminology, Flowmon Collector appliances are cloud compute instances. The cost of standalone appliances consists of:

• Cloud Compute charges,
• network egress charges.

Cost and infrastructure examples of Google Cloud monitoring

This example (Figure 3) consists of three subnets – a DMZ for access to external networks, an application subnet to host the website, and an application database subnet with an internal load balancer sitting between the two.

Figure 3: Example Infrastructure

The assumption here is that we expect 1Gbps throughput to and from external networks, 5Gbps in communication between the two application subnets, and 10 Gbps between communicating instances in the database subnet. At the same time, we want:

1. Basic visibility into application internals
2. In-depth visibility at the perimeter
3. A reasonable costs vs. benefits ratio

The options are to use either only Flowmon Probe and Collector appliances, only Google Cloud VPC flow logs and Collector appliances, or a mix of both. NOTE: All costs shown below are for illustrative purposes only and are subject to change by Google Cloud at any time. However, the costs do show the scale of the difference between the options.

Option 1

If you choose to go with the first option and deploy a Probe on every subnet (Figure 4), you could estimate your monthly Cloud Compute costs at $2,868 with total monthly network costs for traffic mirroring $51,840 (assuming minimal cross-zone traffic). Indeed, you would get perfect visibility, but the monitoring costs are astronomical.

Note that you can reduce the mirroring costs by applying filters and only mirroring selected protocols or hosts, but this naturally comes with a loss of visibility plus increased management overhead.

Figure 4: Example Infrastructure - all Subnets monitored by Flowmon Probe

Option 2

If you monitor all your subnets using only flow logs and send them to your central Collector, your monthly Cloud Compute cost could be estimated at $1,448, and monthly flow logs cost $2,726.

Your network egress costs would be negligible, and your total costs would be highly favorable, but with very low visibility – especially on the perimeter, which is risky, to say the least.

Figure 5: Example Infrastructure - all Subnets monitored by flow logs

Option 3

This option combines the best of both worlds, using Flowmon Probe appliances with packet mirroring on the perimeter to provide complete visibility into incoming and outgoing traffic, complemented by flow logs for low-granularity visibility into the internal network between tiers.

With packet mirroring and a Probe on your DMZ subnet, flow logs in your two internal subnets, cloud logging, pub/sub messaging, and a virtual Collector, Cloud Compute costs can be estimated at $1,800, network costs for mirroring at $3,240 and flow logs cost at $870 per month.

In this way, you achieve an optimum balance between visibility and cost, reserving traffic mirroring for only where it is needed and covering the rest with basic visibility.

Figure 6: Example Infrastructure - Combination of Flowmon Probes & flow logs

Comparison of Google Cloud monitoring costs for different deployment methods

This table shows that Option 3 provides a reasonable compromise in monitoring visibility at just under 11% of the cost of Option 1. This trade-off in visibility versus cost will appeal to many organizations. Remember that you can deploy additional Flowmon Probes to monitor mission-critical applications as required. This will increase the costs incrementally. How much of an increase will depend on how many additional Probes get deployed.

Conclusion

The additions and enhancements to flow log monitoring in Flowmon 12 deliver more cost-effective options for all organizations using AWS, Azure, and Google Cloud. The scenarios and figures we have presented here are focused on Google Cloud, but deployments for AWS and Azure will be similar. Visit the Flowmon 12 page to find out more details, or contact your Progress Flowmon Partner Account Manager or sales representative to find out more.