Fluent Bit: One Telemetry Agent for All your Data Needs
The following video was recorded at KubeCon + CloudNativeCon North America in November 2023. We are grateful to the Cloud Native Computing Foundation for allowing us to post it here.
People don’t wake up one day and say “I want to do logging.” Collecting and analyzing data logs is a means to another end. For most of us, our goal is to get insights into our system performance, and to get insights you need to collect information from those applications and systems, hence logging.
But logging brings challenges:
Log formats differ across systems and applications
Logs can be collected in different ways, including from file systems, APIs, and network endpoints
The sheer volume of logs produced can be difficult to manage
In order to get insights from our logs, we need to gather those logs from a variety of sources, convert them into structured data, and send them to storage for analysis. And we need to do this at scale.
This is where Fluent Bit comes in. Fluent Bit is a unified solution for collecting, processing, and routing telemetry data. In addition to logs, Fluent Bit can also handle metrics and traces. It has a fully pluggable architecture that allows us to connect different sources with different destinations and perform different types of processing to the data in flight. Fluent Bit provides more than 100 connectors built in the same binary, and it’s fully cross-platform for Linux, Windows, MacOS, and BSD.
Because Fluent Bit supports metrics (Prometheus) and traces (OpenTelemetry) as well as logs, it also alleviates “agent fatigue”—which occurs from needing to install multiple agents to collect different types of telemetry data that in turn needs to be delivered to multiple backends. Often each backend platform comes with its own collection agent. Because Fluent Bit is vendor-neutral and supports logs, metrics, and traces, you only need to install and manage one agent.
Fluent Bit is ideal for running in containerized environments like Kubernetes. It can even automatically enrich Kubelet logs with additional metadata from the Kubernetes API server, providing valuable information that would normally be unavailable to downstream systems.
Architecturally, Fluent Bit also provides both in-memory and file-based buffering to help ensure that no data is lost when the volume of data being generated by your sources exceeds your endpoints’ ability to consume it.
The recently released Fluent Bit v2.2 includes new features and enhancements including:
A new sysinfo filter
cgroups v2 support for collecting metrics from Docker
YAML support in configuration files
New collector support for metrics from macOS and Windows clients
New native integration for Loki
Collect Kubernetes cluster events with support for namespaces
A 6X performance improvement in certain complex pipeline configurations
Learn more about Fluent Bit
Join the Fluent Slack Community: To learn more about Fluent Bit, we recommend joining the Fluent Community Slack channel where you will find thousands of other Fluent Bit users.
Check Out Fluent Bit On-Demand Webinars: Our free on-demand webinars that range from introductory sessions to deep dives into advanced features.
You might also like
A practical guide for avoiding data loss and backpressure problems with Fluent Bit
Learn how to detect and avoid backpressure problems with Fluent Bit by balancing memory-based and filesystem-based buffering.
Calyptia joins Chronosphere to build the future of observability
Today we are excited to announce that Calyptia is now part of Chronosphere. This will bring new benefits and opportunities to both Calyptia and Chronosphere customers while we continue to provide a vendor neutral approach to control observability data to any backend.