Fluent Bit: One Telemetry Agent for All your Data Needs

The following video was recorded at KubeCon + CloudNativeCon North America in November 2023. We are grateful to the Cloud Native Computing Foundation for allowing us to post it here.

Synopsis

People don’t wake up one day and say “I want to do logging.” Collecting and analyzing data logs is a means to another end. For most of us, our goal is to get insights into our system performance, and to get insights you need to collect information from those applications and systems, hence logging. 

But logging brings challenges:

• Log formats differ across systems and applications

• Logs can be collected in different ways, including from file systems, APIs, and network endpoints

• The sheer volume of logs produced can be difficult to manage

In order to get insights from our logs, we need to gather those logs from a variety of sources, convert them into structured data, and send them to storage for analysis. And we need to do this at scale.

This is where Fluent Bit comes in. Fluent Bit is a unified solution for collecting, processing, and routing telemetry data. In addition to logs, Fluent Bit can also handle metrics and traces. It has a fully pluggable architecture that allows us to connect different sources with different destinations and perform different types of processing to the data in flight. Fluent Bit provides more than 100 connectors built in the same binary, and it’s fully cross-platform for Linux, Windows, MacOS, and BSD.

Because Fluent Bit supports metrics (Prometheus) and traces (OpenTelemetry) as well as logs, it also alleviates “agent fatigue”—which occurs from needing to install multiple agents to collect different types of telemetry data that in turn needs to be delivered to multiple backends. Often each backend platform comes with its own collection agent. Because Fluent Bit is vendor-neutral and supports logs, metrics, and traces, you only need to install and manage one agent.

Fluent Bit is ideal for running in containerized environments like Kubernetes. It can even automatically enrich Kubelet logs with additional metadata from the Kubernetes API server, providing valuable information that would normally be unavailable to downstream systems.

Architecturally, Fluent Bit also provides both in-memory and file-based buffering to help ensure that no data is lost when the volume of data being generated by your sources exceeds your endpoints’ ability to consume it. 

The recently released Fluent Bit v2.2 includes new features and enhancements including:

• A new sysinfo filter

• cgroups v2 support for collecting metrics from Docker 

• YAML support in configuration files

• New collector support for metrics from macOS and Windows clients

• New native integration for Loki

• Collect Kubernetes cluster events with support for namespaces

• A 6X performance improvement in certain complex pipeline configurations 

Learn more about Fluent Bit

• Join the Fluent Slack Community: To learn more about Fluent Bit, we recommend joining the Fluent Community Slack channel where you will find thousands of other Fluent Bit users.

• Check Out Fluent Bit On-Demand Webinars: Our free on-demand webinars that range from introductory sessions to deep dives into advanced features.