Fluent Bit and Beats: Two Approaches to a Task
When users are building an observability or security strategy one of the first questions that comes up is how they can collect information (logs, metrics, traces, events) from their systems and route this information to a backend for analysis.
Two popular tools for collecting this type of information are Beats and Fluent Bit. In this guide we will provide an overview of the two, explain their different methodologies for collecting data and the impact of that difference, and demonstrate how to migrate from Beats to Fluent Bit.
What are Beats?
Beats are lightweight data shippers from Elastic. They are actually a family of individual data shippers that each perform a specific task such as file collection, windows collection, or metric collection. A short list of Beats is included below:
|Beat Name||Description and Use|
|Filebeat||Used to collect log data from files and send them to a centralized location.|
|Metricbeat||Used to collect system and application metrics, such as CPU usage, memory usage, and network traffic, and send them to a centralized location for analysis.|
|Packetbeat||Used to capture and analyze network data, such as HTTP requests and responses, DNS queries, and MySQL queries.|
|Winlogbeat||Used to collect Windows event logs and send them to a centralized location.|
|Auditbeat||Used to collect audit data on Linux systems, such as process and file system activity.|
|Heartbeat||Used to collect service availability data with active probing|
Beats follow a methodology of collecting information from a variety of sources and routing it to a telemetry pipeline, such as Calyptia Core, Data Prepper, or Logstash. The data collected is formatted into a log-friendly format such as JSON for easy transformation and compatibility with Elasticsearch.
What is Fluent Bit?
Fluent Bit is a fast, lightweight, and highly scalable log, metric, and trace processor and forwarder. It is a vendor-neutral and an Apache 2.0 open source project within the Cloud-Native ecosystem. \
Fluent Bit has been deployed over 7 billion times and seamlessly integrates with other CNCF and open source ecosystems, such as Prometheus and OpenTelemetry, as well as closed source and proprietary systems such as Elasticsearch and Splunk.
What are the differences between Beats and Fluent Bit?
The core functionality of Beats and Fluent Bit is the same: collect various types of data and deliver them to a destination. However, the two differ significantly in their approach to that task.
Each Beats shipper is engineered for a specific type of data collection—Filebeat for logs, Metricbeat for metrics, etc. Fluent Bit follows a single-agent approach, handling multiple types of data. As a result, if you need to gather more than one type of data you would need to deploy multiple Beats shippers compared to deploying a single Fluent Bit agent.
Beats and Fluent Bit also differ significantly in where they can send the data they collect. As an Elastic product, Beats only support Elasticsearch as an end destination. Fluent Bit takes a vendor-neutral approach and supports any number of destinations.
The table below lays out some common scenarios users encounter when dealing with telemetry data and the Beats and Fluent Bit approaches.
|Scenario||Beats Approach||Fluent Bit Approach|
|Collecting logs and metrics from a Linux Machine||Install Filebeat and Metricbeat, manage two configuration files, and start two processes||Add two inputs in a single Fluent Bit configuration process, start a single process|
|Sending data to Elasticsearch, Kafka, and other end destinations||Not supported – Beats only support Elasticsearch as an end destination||Supported with multiple output plugins|
|Process data midstream||Route data to Logstash or Elastic Ingest node||Process using Lua Scripting, Filters, OR route data to Logstash, Elastic ingest node, or Data Prepper|
Support for Prometheus and OpenTelemetry Metrics
Another notable difference between Fluent Bit and Beats is their approach to metrics. Within Beats, and specifically Metricbeat, all metrics are treated as a log-based metric. With Fluent Bit metrics can either be log-based metrics or Prometheus / OpenTelemetry metrics. This allows for users to plug into broader CNCF ecosystems without having to install another agent or collector.
Agent Configuration Comparison: Logs and Metrics from a Linux machine
Suppose you want to ship logs and metrics data using Beats. You need to install Filebeat and Metricbeat separately on each server. Additionally, you then need to configure the
filebeat.inputs: - type: log Paths: - /var/log/httpd/access_log
- module: system metricsets: - cpu - memory - network - process_summary - socket_summary - diskio - filesystem period: 10s processors: - add_host_metadata: ~ - add_cloud_metadata: ~
You’ll also need to add the output elasticsearch to each configuration file further to ship data. We would need to ensure that the output is kept in sync to make sure both inputs flow to the end destination.
In comparison, for Fluent Bit, you only need to install the agent and configure one file for shipping logs and metrics.
[INPUT] Name tail Path /var/log/httpd/access_log Tag apache.access DB /var/log/flb_apache.db [INPUT] Name cpu Tag cpu [INPUT] Name mem Tag memory
Sending Data to Fluent Bit from Beats
In some cases where Fluent Bit may not have the required support for a particular collector, a user may make use of the new in_elasticsearch plugin from Fluent Bit. This allows Fluent Bit to listen on a port and receive incoming payloads sent to Elasticsearch or HTTP bulk API.
For example in a beats configuration, you can set the output yaml to the following configuration. Fluent Bit returns 8.0 as the version, which may require you to set the allow_older_versions depending on the oss metricbeat version you are using. Lastly, the current integration does not support modules or index lifecycle management (ILM), which will need to be turned off.
output.elasticsearch: hosts: ["Fluent Bit host:PORT"] allow_older_versions: true ilm: false
Configuring Fluent Bit
In Fluent Bit you will need to configure the in_elasticsearch plugin and then populate the output section with any destination you desire. This can include Data Prepper, OpenSearch, Elasticsearch, or any other of 50+ destinations supported by Fluent Bit. In our example, we will use the standard output plugin to see logs in the terminal output
[INPUT] name elasticsearch listen 0.0.0.0 port 9200 buffer_max_size 20M buffer_chunk_size 5M [OUTPUT] name stdout match *
Both Fluent Bit and the Beats data shippers can both be used for collecting and routing logs, metrics, traces, and event data from systems and routing it to a backend for storage and analysis. However, the two take dramatically different approaches to that task. Built by Elastic, Beats shippers are very efficient at routing that data to Elasticsearch, but are incapable of connecting to other destinations.
Fluent Bit, on the other hand, takes a vendor-neutral approach and can route the data it collects to any number of destinations. In addition, Fluent Bit is a single agent that is capable of handling a variety of different data types while Beats shippers are specialized and require users to deploy multiple agents to collect multiple data types. As we saw above, Fluent Bit can even integrate with Beats.
The Fluent Bit agent serves as a key component of thousands of telemetry pipelines for organizations around the world. As the creators of Fluent Bit, Calyptia offers Calyptia Core, which simplifies the management of complex telemetry pipelines with a low-code and UI approach to connecting your existing agents, open source protocols, SaaS platforms, and your storage and analytic backends. To see how Calyptia Core can transform your observability and SIEM efforts request a demo and a free trial.