Background image - Fluent Bit bird on white background

Logging Deep Dive and Best Practices

Written by Eduardo Silva, Anurag Gupta, José Lecaros in Fluent Biton November 30, 2023

Logging Deep Dive and Best Practices

The following video was recorded at KubeCon + CloudNativeCon North America in November 2023. We are grateful to the Cloud Native Computing Foundation for allowing us to post it here.


The Challenges of Logging

Log collection and analysis is an essential part of understanding what is going on with our systems. However, logging brings challenges:

  • Log formats differ across systems and applications

  • Logs can be collected in different ways, including from file systems, APIs, and network endpoints

  • The sheer volume of logs produced can be difficult to manage

This is where Fluent Bit comes in. Fluent Bit is a unified solution for collecting, processing, and routing telemetry data. In addition to logs, Fluent Bit can also handle metrics and traces. 

Logging Best Practices

In this section, Anurag Gupta demonstrates how Fluent Bit can be used to solve various problems, including

  • Enriching logs with additional information, such as hostname

  • Keeping debug and trace logs out of expensive storage and analytical backends

  • Redacting or removing sensitive data such as credit card numbers

  • Converting logs to metrics for easier monitoring on dashboards and reporting

Monitoring and Operations

In the final section of the presentation, Jose Lecaros demonstrates various ways to monitor your Fluent Bit data pipelines and troubleshoot potential issues including:

  • Using Grafana to visualize the metrics that Fluent Bit generates about its own health and status

  • Identifying issues where your destination endpoint cannot handle the current volume and how Fluent Bit provides both in-memory and file system buffering to reduce back pressure and ensure that no data is lost.

Learn more about Fluent Bit

  • Join the Fluent Slack Community: To learn more about Fluent Bit, we recommend joining the Fluent Community Slack channel where you will find thousands of other Fluent Bit users.

  • Check Out Fluent Bit On-Demand Webinars: Our free on-demand webinars that range from introductory sessions to deep dives into advanced features.

You might also like

Fluent Bit

Statement on CVE-2024-4323 and its fix

We'd like to make sure you’re aware of a security vulnerability (known as CVE-2024-4323) that impacts Fluent Bit versions 2.0.7 through 3.0.3. The latest version of Fluent Bit, version 3.0.4, fixes this issue.

Continue reading
Fluent Bit

Explaining the Fluent Bit processor

Fluent Bit 2.1.2 introduced customization logic for input and output plugins called Processors (not to be confused with the Stream Processor). Here's an example of its use.

Continue reading
Fluent Bit or Fluentd

Fluent Bit and Fluentd – a child or a successor?

Fluent Bit may have started as a sibling to Fluentd, but it is fair to say that it has now grown up and is Fluentd's equal. Learn which is right for your needs and how they can be used together.

Continue reading