Fluent Bit and Beats: Two Approaches to a Task

When users are building an observability or security strategy one of the first questions that comes up is how they can collect information (logs, metrics, traces, events) from their systems and route this information to a backend for analysis.

Two popular tools for collecting this type of information are Beats and Fluent Bit. In this guide we will provide an overview of the two, explain their different methodologies for collecting data and the impact of that difference, and demonstrate how to migrate from Beats to Fluent Bit.

What are Beats?

Beats are lightweight data shippers from Elastic. They are actually a family of individual data shippers that each perform a specific task such as file collection, windows collection, or metric collection. A short list of Beats is included below:

Beats follow a methodology of collecting information from a variety of sources and routing it to a telemetry pipeline, such as Calyptia Core, Data Prepper, or Logstash. The data collected is formatted into a log-friendly format such as JSON for easy transformation and compatibility with Elasticsearch.

What is Fluent Bit?

Fluent Bit is a fast, lightweight, and highly scalable log, metric, and trace processor and forwarder. It is a vendor-neutral and an Apache 2.0 open source project within the Cloud-Native ecosystem.

Fluent Bit has been deployed over 7 billion times and seamlessly integrates with other CNCF and open source ecosystems, such as Prometheus and OpenTelemetry, as well as closed source and proprietary systems such as Elasticsearch and Splunk.

What are the differences between Beats and Fluent Bit?

The core functionality of Beats and Fluent Bit is the same: collect various types of data and deliver them to a destination. However, the two differ significantly in their approach to that task.

Each Beats shipper is engineered for a specific type of data collection—Filebeat for logs, Metricbeat for metrics, etc. Fluent Bit follows a single-agent approach, handling multiple types of data. As a result, if you need to gather more than one type of data you would need to deploy multiple Beats shippers compared to deploying a single Fluent Bit agent.

Beats and Fluent Bit also differ significantly in where they can send the data they collect. As an Elastic product, Beats only support Elasticsearch as an end destination. Fluent Bit takes a vendor-neutral approach and supports any number of destinations.

The table below lays out some common scenarios users encounter when dealing with telemetry data and the Beats and Fluent Bit approaches.

Support for Prometheus and OpenTelemetry Metrics

Another notable difference between Fluent Bit and Beats is their approach to metrics. Within Beats, and specifically Metricbeat, all metrics are treated as a log-based metric. With Fluent Bit metrics can either be log-based metrics or Prometheus / OpenTelemetry metrics. This allows for users to plug into broader CNCF ecosystems without having to install another agent or collector.

Agent Configuration Comparison: Logs and Metrics from a Linux machine

Suppose you want to ship logs and metrics data using Beats. You need to install Filebeat and Metricbeat separately on each server. Additionally, you then need to configure the filebeat.yml and metricbeat.yml files.

filebeat.yml

filebeat.inputs:
- type: log
      Paths:
    - /var/log/httpd/access_log

metricbeat.yml

- module: system
  metricsets:
    - cpu
    - memory
    - network
    - process_summary
    - socket_summary
    - diskio
    - filesystem
  period: 10s
  processors:
    - add_host_metadata: ~
    - add_cloud_metadata: ~

You’ll also need to add the output elasticsearch to each configuration file further to ship data. We would need to ensure that the output is kept in sync to make sure both inputs flow to the end destination.

In comparison, for Fluent Bit, you only need to install the agent and configure one file for shipping logs and metrics.

fluentbit.conf

[INPUT]
    Name    tail
    Path    /var/log/httpd/access_log
    Tag     apache.access
    DB      /var/log/flb_apache.db

[INPUT]
    Name    cpu
    Tag     cpu

[INPUT]
    Name    mem
    Tag     memory

Sending Data to Fluent Bit from Beats

In some cases where Fluent Bit may not have the required support for a particular collector, a user may make use of the new in_elasticsearch plugin from Fluent Bit. This allows Fluent Bit to listen on a port and receive incoming payloads sent to Elasticsearch or HTTP bulk API.

Configuring Beats

For example in a beats configuration, you can set the output yaml to the following configuration. Fluent Bit returns 8.0 as the version, which may require you to set the allow_older_versions depending on the oss metricbeat version you are using. Lastly, the current integration does not support modules or index lifecycle management (ILM), which will need to be turned off.

output.elasticsearch:
  hosts: ["Fluent Bit host:PORT"]
  allow_older_versions: true
  ilm: false

Configuring Fluent Bit

In Fluent Bit you will need to configure the in_elasticsearch plugin and then populate the output section with any destination you desire. This can include Data Prepper, OpenSearch, Elasticsearch, or any other of 50+ destinations supported by Fluent Bit. In our example, we will use the standard output plugin to see logs in the terminal output

[INPUT]
    name elasticsearch
    listen 0.0.0.0
    port 9200
    buffer_max_size 20M
    buffer_chunk_size 5M

[OUTPUT]
    name stdout
    match *

Conclusion

Both Fluent Bit and the Beats data shippers can both be used for collecting and routing logs, metrics, traces, and event data from systems and routing it to a backend for storage and analysis. However, the two take dramatically different approaches to that task. Built by Elastic, Beats shippers are very efficient at routing that data to Elasticsearch, but are incapable of connecting to other destinations.

Fluent Bit, on the other hand, takes a vendor-neutral approach and can route the data it collects to any number of destinations. In addition, Fluent Bit is a single agent that is capable of handling a variety of different data types while Beats shippers are specialized and require users to deploy multiple agents to collect multiple data types. As we saw above, Fluent Bit can even integrate with Beats.

The Fluent Bit agent serves as a key component of thousands of telemetry pipelines for organizations around the world. As the creators of Fluent Bit, Calyptia offers Calyptia Core, which simplifies the management of complex telemetry pipelines with a low-code and UI approach to connecting your existing agents, open source protocols, SaaS platforms, and your storage and analytic backends. To see how Calyptia Core can transform your observability and SIEM efforts request a demo and a free trial.