Announcing Calyptia Fluentd distribution!

Today, we’re happy to announce the initial release of Calyptia Fluentd v1.0. Fluentd is an extremely popular open-source project that is a graduated project part of the cloud native computing foundation (CNCF).

Calyptia Fluentd is a new distribution of Fluentd that includes more metric-focused features, along with packages optimized for performance and integration with more services. In the future, Calyptia Fluentd will also give users simple ways to communicate with the Calyptia Cloud service.

How to install?

We’ve added documentation within the official Fluentd documents how to install on the following platforms:

• Debian GNU/Linux or Ubuntu

• RHEL/CentOS

• macOS

• Windows.

Supported platforms are:

Key Features

Ruby 3

Ruby 3 is the latest version of ruby that gives improvements in performance and more memory safety features compared to older ruby versions. Calyptia Fluentd packages Ruby 3.0 in the package by default and brings all the benefits to default plugins such as tail, filtering, and others.

Systemd (Linux)

Calyptia-Fluentd supports systemd management on Linux platforms.

start

$ sudo systemctl start calyptia-fluentd.service

stop

$ sudo systemctl stop calyptia-fluentd.service

Launchctl (macOS)

Calyptia-Fluentd supports launchctl management on macOS.

start

$ sudo launchctl load /Library/LaunchDaemons/calyptia-fluentd.plist

stop

$ sudo launchctl unload /Library/LaunchDaemons/calyptia-fluentd.plist

Windows Service (Windows)

Calyptia-Fluentd also integrates as a Windows Service on Windows. Note that the following commands request Administrator privileges.

start

using net.exe

> net start fluentdwinsvc

PowerShell Cmdlet

PS> Start-Service fluentdwinsvc

stop

using net.exe

> net stop fluentdwinsvc

PowerShell Cmdlet

PS> Stop-Service fluentdwinsvc

Plugin Management

Calyptia-Fluentd provides calyptia-fluentd-gem command to manage plugins.

For example, for users who want to install fluent-plugin-mongo:

$ sudo calyptia-fluentd-gem install fluent-plugin-mongo

Performance

Performance is extremely important to Fluentd users, and Calyptia Fluentd is no different. We’ve added new versions of ruby that include performance benefits and are always looking at ways we can optimize for major use cases. We’ve also included benchmarks for Calyptia Fluentd below in the following scenarios:

• tailing a flat file(in_tail)

• consuming syslog (in_syslog)

• consuming Windows EventLog (in_windows_eventlog2)

Tailing a Flat File — Benchmark scenario for in_tail

Environment

• Forwarder | CentOS 8 on AWS t2.medium instance

• Aggregator | CentOS 8 on AWS t2.medium instance

Scenario

increase generating lines rate step by step

• baseline(0 line/sec)

• 500 lines/sec

• 1000 lines/sec

• 2000 lines/sec

• 5000 lines/sec

Generate logs with dummer which is ltsv format

Dummer generates ltsv format lines:

configure 'sample' do
  output "message.log"
  delimiter "\t"
  labeled true
  field :id, type: :integer, countup: true, format: "%04d"
  field :time, type: :datetime, format: "[%Y-%m-%d %H:%M:%S]", random: false
  field :level, type: :string, any: %w[DEBUG INFO WARN ERROR]
  field :method, type: :string, any: %w[GET POST PUT]
  field :uri, type: :string, any: %w[/api/v1/people /api/v1/textdata]
  field :reqtime, type: :float, range: 0.1..5.0
  field :foobar, type: :string, length: 8
end

Worker CPU Usages

In higher traffic environments we see Calyptia Fluentd using lower CPU time consumption compared to other distributions.

Worker Memory Usages (RSS)

Consuming Syslog — Benchmark scenario for in_syslog

Environment

• Forwarder | CentOS 8 on AWS t2.medium instance

• Aggregator | CentOS 8 on AWS t2.medium instance

Scenario

Increase generating lines rate step by step

• baseline(0 line/sec)

• 500 messages/sec

• 1000 messages/sec

• 1500 messages/sec

Generate syslog message with loggen

Worker CPU Usages

Worker Memory Usages (RSS)

Calyptia Fluentd RSS memory usage is lower compared to other distributions with the same configuration.

Consuming Windows EventLog — Benchmark scenario for in_windows_eventlog2

Environment

• Collector | Windows Server 2019 on AWS t2.medium instance

• Benchmark tool written in C#

• Aggregator | Ubuntu Focal (20.04 LTS) on AWS t2.medium instance

Scenario

increase generating Windows events size step by step

• 512 bytes 120000 events total

• 1024 bytes 120000 events total

• 2048 bytes 120000 events total

monitoring Ruby processes with typeperf during about 18 minutes

Worker CPU Usages

Worker Memory Usages (Working Set)

How do I get started?

You can start by downloading Calyptia-Fluentd from Calyptia Fluentd download, and start building configuration in our visualization tool (sign-up required).