Decorative blue background

Calyptia Core adds support for Redpanda

Written by Erik Bledsoe in Calyptia Coreon January 2, 2024

Calyptia Core adds support for Redpanda

Calyptia Core now supports Redpanda as a destination for high-volume streaming data pipelines. Calyptia Core enables users to seamlessly manage their telemetry data from source to destination.

The new support for Redpanda adds to the dozens of other integrations already available as either sources or destinations available in Calyptia Core.

Redpanda is a simple, powerful, and cost-efficient streaming data platform that is compatible with Kafka APIs. Sometimes referred to as a Kafka clone written in C++, Redpanda offers performance improvements and eliminates much of the complexity of Kafka. 

In addition to support for Redpanda, Core also provides integrations for Kafka as well as for Confluent, a commercial version of the open-source Kafka from the project’s creators. 

Here’s a quick overview of how to integrate Redpanda into your data pipeline with Calyptia Core.

Getting Started

This assumes that you already have a Redpanda account and a running cluster and familiarity with the  Redpanda Console. If not, Redpanda offers a free trial or you can spin up a Redpanda cluster in Docker.

Likewise, we assume you already have a Calyptia Core account. If not, we also offer a free trial as well as a Docker option for quick evaluation

Gather the following from your Redpanda account:

  • Your Redpanda broker(s) address(es) and port number(s)

  • Account username and password 

  • Topic(s) to which you want to send data

Creating Your Pipeline in Core

From within your Calyptia Core interface, navigate to your running Core instance, select the option to create a new pipeline, and give it a name

Screen capture showing the steps for creating a new pipeline

Select your data source. For the purposes of this exercise, we’ll be using the Mock Data plugin, but you, of course, could also use any of the dozens of other data sources that Calyptia supports.

screen capture showing how to select a data source

Here is a sample of the mock data that we will use:


With your data source now defined, select Redpanda as your destination. 

You’ll need to provide your Redpanda broker address and port number. Core supports adding multiple brokers separated by commas, but for this exercise, we’ll be using a single instance. 

Provide the topic or topics to which you want to send your data. Note: the topic must already exist in your Redpanda cluster. Core does not create topics. 

Select the format of your data (either json or msgpack). We are using json. 

The security protocol and SASL mechanism are set to the Redpanda defaults, but if your cluster differs you’ll need to adjust accordingly. 

Finally, provide the username and password that Core will use to connect to your Redpanda cluster. 

Setting up Redpanda as the data destination

After you save and deploy your pipeline in Calyptia Core, switch to your Redpanda console and open the message topic you created and you’ll see messages flowing in. 

Screen capture of Redpanda console shoing data flowing in

Next Steps: Discover How Core can Transform your Data Pipelines

Of course, Calyptia Core can do much more than just collect and route data (although, as you‘ve seen Core makes it really easy to do so). You could also apply any of dozens of processing rules that allow you to act upon and transform your data in flight before they are sent to their destination. 

Calyptia’s built-in processing rules include the ability to

To learn more about how Calyptia enables turnkey data collection, aggregation, transformation, and forwarding from any source to any destination, schedule a demonstration.

You might also like

Fluent Bit

Statement on CVE-2024-4323 and its fix

We'd like to make sure you’re aware of a security vulnerability (known as CVE-2024-4323) that impacts Fluent Bit versions 2.0.7 through 3.0.3. The latest version of Fluent Bit, version 3.0.4, fixes this issue.

Continue reading
Fluent Bit

Explaining the Fluent Bit processor

Fluent Bit 2.1.2 introduced customization logic for input and output plugins called Processors (not to be confused with the Stream Processor). Here's an example of its use.

Continue reading
Fluent Bit or Fluentd

Fluent Bit and Fluentd – a child or a successor?

Fluent Bit may have started as a sibling to Fluentd, but it is fair to say that it has now grown up and is Fluentd's equal. Learn which is right for your needs and how they can be used together.

Continue reading