Decorative blue background
Background

Calyptia Core adds support for Redpanda

Written by Erik Bledsoe in Calyptia Coreon January 2, 2024

Calyptia Core adds support for Redpanda

Calyptia Core now supports Redpanda as a destination for high-volume streaming data pipelines. Calyptia Core enables users to seamlessly manage their telemetry data from source to destination.

The new support for Redpanda adds to the dozens of other integrations already available as either sources or destinations available in Calyptia Core.

Redpanda is a simple, powerful, and cost-efficient streaming data platform that is compatible with Kafka APIs. Sometimes referred to as a Kafka clone written in C++, Redpanda offers performance improvements and eliminates much of the complexity of Kafka. 

In addition to support for Redpanda, Core also provides integrations for Kafka as well as for Confluent, a commercial version of the open-source Kafka from the project’s creators. 

Here’s a quick overview of how to integrate Redpanda into your data pipeline with Calyptia Core.

Getting Started

This assumes that you already have a Redpanda account and a running cluster and familiarity with the  Redpanda Console. If not, Redpanda offers a free trial or you can spin up a Redpanda cluster in Docker.

Likewise, we assume you already have a Calyptia Core account. If not, we also offer a free trial as well as a Docker option for quick evaluation

Gather the following from your Redpanda account:

  • Your Redpanda broker(s) address(es) and port number(s)

  • Account username and password 

  • Topic(s) to which you want to send data

Creating Your Pipeline in Core

From within your Calyptia Core interface, navigate to your running Core instance, select the option to create a new pipeline, and give it a name

Screen capture showing the steps for creating a new pipeline

Select your data source. For the purposes of this exercise, we’ll be using the Mock Data plugin, but you, of course, could also use any of the dozens of other data sources that Calyptia supports.

screen capture showing how to select a data source

Here is a sample of the mock data that we will use:

{"registertime":1500599586519,"userid":"User_9","regionid":"Region_5","gender":"MALE"} 
{"registertime":1493882456812,"userid":"User_9","regionid":"Region_3","gender":"OTHER"} 
{"registertime":1514584804940,"userid":"User_9","regionid":"Region_8","gender":"FEMALE"} 
{"registertime":1498613454415,"userid":"User_7","regionid":"Region_9","gender":"FEMALE"} 
{"registertime":1510970841590,"userid":"User_8","regionid":"Region_8","gender":"OTHER"}

With your data source now defined, select Redpanda as your destination. 

You’ll need to provide your Redpanda broker address and port number. Core supports adding multiple brokers separated by commas, but for this exercise, we’ll be using a single instance. 

Provide the topic or topics to which you want to send your data. Note: the topic must already exist in your Redpanda cluster. Core does not create topics. 

Select the format of your data (either json or msgpack). We are using json. 

The security protocol and SASL mechanism are set to the Redpanda defaults, but if your cluster differs you’ll need to adjust accordingly. 

Finally, provide the username and password that Core will use to connect to your Redpanda cluster. 

Setting up Redpanda as the data destination

After you save and deploy your pipeline in Calyptia Core, switch to your Redpanda console and open the message topic you created and you’ll see messages flowing in. 

Screen capture of Redpanda console shoing data flowing in

Next Steps: Discover How Core can Transform your Data Pipelines

Of course, Calyptia Core can do much more than just collect and route data (although, as you‘ve seen Core makes it really easy to do so). You could also apply any of dozens of processing rules that allow you to act upon and transform your data in flight before they are sent to their destination. 

Calyptia’s built-in processing rules include the ability to

To learn more about how Calyptia enables turnkey data collection, aggregation, transformation, and forwarding from any source to any destination, schedule a demonstration.



You might also like

Fluent Bit

A practical guide for avoiding data loss and backpressure problems with Fluent Bit

Learn how to detect and avoid backpressure problems with Fluent Bit by balancing memory-based and filesystem-based buffering.

Continue reading
Chronosphere + Calyptia

Calyptia joins Chronosphere to build the future of observability

Today we are excited to announce that Calyptia is now part of Chronosphere. This will bring new benefits and opportunities to both Calyptia and Chronosphere customers while we continue to provide a vendor neutral approach to control observability data to any backend.

Continue reading
Processing Custom IIS server logs with Fluent Bit, Wasm, and Rust

Processing Custom IIS server logs with Fluent Bit, Wasm, and Rust

Create a custom processing script for IIS logs written in Rust and implemented using the Fluent Bit Wasm plugin.

Continue reading