The Fluent Ecosystem - More than just Fluentd and Fluent Bit

The Fluent Ecosystem

Written by Eduardo Silva in Fluent Biton June 2, 2022

The Fluent Ecosystem

This is the first of a series of posts based upon my keynote presentation “Fluent Bit in 2022: Logs, Prometheus and OpenTelemetry” at FluentCon 2022. In this first post, we look at how the original Fluentd project evolved into a full ecosystem of observability solutions, still firmly grounded in open source.

The Fluentd project started more than 10 years ago to solve a very specifc problem: how to collect and move a large amount of data. We never intended Fluentd to be a drop-in replacement for other data collectors. Instead, we envisioned it as something that you could plug into your existing architecture and solve your problems related to moving data from one point to another.

Now, ten years later Fluentd is still moving data in thousands of systems, But the Fluent ecosystem has grown and evolved. The Fluent ecosystem is more than a single tool—more than just Fluentd. It’s more than Fluentd plus Fluent Bit. It’s a concept that you should be able to integrate with different systems, different architectures, and other projects. Pretty much from the beginning, we identified the importance of connecting with competitors’ products, not replacing them.

The Fluent ecosystem is sort of like the Swiss Army Knife for observability.

We try to connect with everything that is around us because the problem now is not just moving logs from one place to another. There are traces. There are metrics. But also you don’t want to remove your infrastructure agents. That is a major and expensive undertaking. We have users and customers who deploy 100,000 servers, So when you are talking about switching technology at that scale, it’s really expensive. It takes a lot of time, a lot of planning, and a lot of maintenance.


So for us Fluent is a full ecosystem that allows you to get into this observability journey and start connecting the dots between different sources of information, different destinations, and different types and formats of data. We try to be data agnostic. Today it’s logs, metrics, and traces. Maybe tomorrow another kind of critical information might exist, and you can be sure that the Fluent ecosystem will support it.

Watch the full presentation from FluentCon

Open Source is Key to Observability

I firmly believe that everything in observability must be open source. I say this not just because we are an open-source company. I say this because of the value of contributions from different vendors, companies, and individuals.

But it’s also really important that you don’t get into vendor lock-in. When you first start down the observability path, from a management perspective, you think that what you need to accomplish is data analysis, that you need to centralize all your information. But that is only part of the solution.

Our value lies in providing the tool that allows you to choose your own backend, or multiple backends, or cloud services in a smooth way. We try to be agnostic in all ways possible in terms of projects, products, and platforms.

In my next post, I’ll explore how Fluent Bit was created as a response to the era of containers and microservices.

You might also like

Fluent Bit or Fluentd

Fluent Bit and Fluentd – a child or a successor?

Fluent Bit may have started as a sibling to Fluentd, but it is fair to say that it has now grown up and is Fluentd's equal. Learn which is right for your needs and how they can be used together.

Continue reading
Calyptia + Lua + AI

Transform your logs in-flight with Lua, AI, and Calyptia

Learn how Calyptia lets you create custom processing rules to transform your data using Lua and how Calyptia integrates AI to simplify data processing.

Continue reading
Fluent Bit v3

Fluent Bit v3 gives users greater control of their data and telemetry pipelines

New release allows filtering of Windows and MacOS metrics, supports SQL for parsing logs, adds support for HTTP/2, and more.

Continue reading