Fluent Bit and Fluentd – a child or a successor?

This post is an excerpt from the forthcoming book Fluent Bit with Kubernetes by Phil Wilkins and published by Manning Publications. It appears here by agreement with the publisher. You can also download an advance copy of the book for free.

While Fluent Bit may have started as a sibling to Fluentd, with the support for OTel and other features arriving in the late 1.x versions or as part of v2.0+, it is fair to say that it has now grown up and is Fluentd's equal. This, of course, spawns several possible questions:

• Do I need to learn Fluentd to learn Fluent Bit?

• Is Fluentd a legacy solution now?

To get to grips with Fluent Bit, you don't need to know anything about Fluentd. But, if you understand Fluentd at a high level, you'll find that getting to grips with Fluent Bit is easy.

There is no dependency between the products. In many respects, while the two products have a lot of overlap, they are also complementary.

Is Fluentd a legacy technology?

The question of Fluentd’s place and whether it is a legacy technology is an architectural question. Like which technology to use, the answer will always be, “It depends.” The drivers and capabilities incorporated into Fluent Bit mean that it fits into the modern cloud-native ecosystem centered on Kubernetes like a glove with the means to address all its demands with some features not presently in Fluentd. Fluent Bit has a smaller, lighter footprint, making it more suited to containerized use cases. 

Another factor is the OpenTelemetry support. At the time of writing, we have not seen a roadmap to equip Fluentd with support for OTel. This makes Fluent Bit by far the better choice when deploying into container-orchestrated environments such as Kubernetes and working with services such as Istio. 

Nothing is stopping us from deploying Fluent Bit into non-cloud-native environments. Such environments typically have a wider portfolio of technologies to work with. This lends itself more to Fluentd for the foreseeable future, given the number of adapters it has available, and the skills to create custom plugins are more readily available – you simply need to grasp Ruby. While WASM can enable extensions to Fluent Bit with languages like Java and Ruby, it still demands additional skills for a technology that is still proving itself to the mainstream.

When it comes to whether Fluentd is history, the answer is no. Some may disagree.

But the reality is that major vendors have invested in and have used Fluentd for a long time. It's not the sort of investment you walk away from. Furthermore, Fluentd and Fluent Bit have different technologies, and while they have some common ideas, they do go about those ideas differently. Many of the key contributors behind the development of Fluentd are the same people working on Fluent Bit. Both solutions are being propelled forward to meet the demands and innovation around CNCF. Cloud-native ideas and CNCF influence the world of software; not all cloud software deployments are as tightly bound to Kubernetes as others. 

Put simply, Fluent Bit can do a lot and be applied to many use cases, but today, Fluentd fits some use cases better than Fluent Bit and vice versa.

Fluentd and Fluent Bit: Working together

One important thing to remember is that Fluent Bit and Fluentd can work together. As a result, we can capture from more sources by using Fluentd and introduce the data into a Fluent Bit deployment. This is achieved because the Forward plugin has a defined standard that Fluent Bit and Fluentd both use. However, some external products have also adopted this protocol so they can be connected directly to Fluent Bit (for example, Docker has a Log Driver supporting the Forward protocol). 

This isn’t the only way we can capture data from third-party products without having specific input plugins. Fluent Bit supports the OpenTelemetry standard (OTLP) protocol, which means applications configured to generate compliant Logs, Traces, and Metrics can also be connected directly with Fluent Bit. Traditionally, application logging frameworks are associated with creating log files; some do include the ability to use Forward protocol, and the OpenTelemetry project has also provided telemetry frameworks that can be used.

What does this mean for us? Well, for new or modernized solutions, we can consider alternatives to more traditional monitoring strategies such as log files, etc., and do so without necessarily having to resort to custom plugins (although custom plugins do offer a way of implementing any data transformation in a highly efficient manner). When capturing events, we should take a step back and consider our options and the pros and cons rather than accepting the default approaches. 

If you enjoyed this article, be sure to download your free copy of Fluent Bit with Kubernetes by Phil Wilkins. 

To learn more about Fluent Bit, visit Fluent Bit Academy, your destination for best practices and how-to’s on advanced processing, routing, and all things Fluent Bit. Here’s a sample of what you can find there:

• Getting Started with Fluent Bit and OpenSearch

• Getting Started with Fluent Bit and OpenTelemetry

• Fluent Bit for Windows

About Calyptia and Chronosphere

From the creators of Fluent Bit, Calyptia lets you build turnkey data pipelines to seamlessly collect, transform, and route data to your Observability and SIEM backends. Calyptia simplifies your infrastructure complexity and provides new capabilities for data collection, transformation, and multi-routing. 

Calyptia is now a part of Chronosphere, the only cloud native observability solution that helps teams quickly resolve incidents before they impact the customer experience and the bottom line. Combined with the power of Calyptia, Chronosphere’s end-to-end observability platform helps teams rein in costs, improve developer productivity, and gain a competitive advantage.