Background image - Fluent Bit bird on white background

Logging Deep Dive and Best Practices

Written by Eduardo Silva, Anurag Gupta, José Lecaros in Fluent Biton November 30, 2023

Logging Deep Dive and Best Practices

The following video was recorded at KubeCon + CloudNativeCon North America in November 2023. We are grateful to the Cloud Native Computing Foundation for allowing us to post it here.


The Challenges of Logging

Log collection and analysis is an essential part of understanding what is going on with our systems. However, logging brings challenges:

  • Log formats differ across systems and applications

  • Logs can be collected in different ways, including from file systems, APIs, and network endpoints

  • The sheer volume of logs produced can be difficult to manage

This is where Fluent Bit comes in. Fluent Bit is a unified solution for collecting, processing, and routing telemetry data. In addition to logs, Fluent Bit can also handle metrics and traces. 

Logging Best Practices

In this section, Anurag Gupta demonstrates how Fluent Bit can be used to solve various problems, including

  • Enriching logs with additional information, such as hostname

  • Keeping debug and trace logs out of expensive storage and analytical backends

  • Redacting or removing sensitive data such as credit card numbers

  • Converting logs to metrics for easier monitoring on dashboards and reporting

Monitoring and Operations

In the final section of the presentation, Jose Lecaros demonstrates various ways to monitor your Fluent Bit data pipelines and troubleshoot potential issues including:

  • Using Grafana to visualize the metrics that Fluent Bit generates about its own health and status

  • Identifying issues where your destination endpoint cannot handle the current volume and how Fluent Bit provides both in-memory and file system buffering to reduce back pressure and ensure that no data is lost.

Learn more about Fluent Bit

  • Join the Fluent Slack Community: To learn more about Fluent Bit, we recommend joining the Fluent Community Slack channel where you will find thousands of other Fluent Bit users.

  • Check Out Fluent Bit On-Demand Webinars: Our free on-demand webinars that range from introductory sessions to deep dives into advanced features.

You might also like

Fluent Bit

A practical guide for avoiding data loss and backpressure problems with Fluent Bit

Learn how to detect and avoid backpressure problems with Fluent Bit by balancing memory-based and filesystem-based buffering.

Continue reading
Chronosphere + Calyptia

Calyptia joins Chronosphere to build the future of observability

Today we are excited to announce that Calyptia is now part of Chronosphere. This will bring new benefits and opportunities to both Calyptia and Chronosphere customers while we continue to provide a vendor neutral approach to control observability data to any backend.

Continue reading
Processing Custom IIS server logs with Fluent Bit, Wasm, and Rust

Processing Custom IIS server logs with Fluent Bit, Wasm, and Rust

Create a custom processing script for IIS logs written in Rust and implemented using the Fluent Bit Wasm plugin.

Continue reading