Redpanda 24.1 Raises Streaming Data Performance to New Heights

May 07, 2024 at 09:00 AM EDT

Latest Release from Redpanda Introduces Write Caching, Simplified Security Controls, and the General Availability of Data Transforms

Streaming data pioneer Redpanda today announced the release of Redpanda 24.1, a powerful update that provides users more options for optimizing performance, extends inline processing capabilities and ensures the predictability and security of their systems. With a flexible write caching option, Redpanda 24.1 significantly improves performance on a wider variety of hardware and storage infrastructure. The release also includes new ways to balance cluster resources, role-based access control, and the general availability of Redpanda's data transformation capabilities.

“We’ve released another first – a microsecond-level latency storage engine mode for the most demanding applications,” said Alex Gallego, founder and CEO of Redpanda. “We were the first to build a fully compatible alternative to Apache Kafka. The first to deliver a Bring Your Own Cloud deployment option to the streaming world, fully auditable with minimal permissions. The first to deliver streaming, in-broker transformations based on WebAssembly – also GA in this release. And today we are the first to allow applications to selectively opt-in the fastest, most predictable storage engine available for the Kafka API. If you want to go fast, we’re it.”

“Redpanda is the fastest, most reliable streaming platform for every workload,” said Erik LaBianca, CTO at marketing technology innovator Seventh Sense. “Using Redpanda moves our bottlenecks to producers and consumers, where they belong.”

98% Performance Improvement with Write Caching

Write caching slashes latencies, enabling the fastest streaming-data performance for Redpanda by acknowledging data in memory before permanently storing it to disk. Crucial for use cases that prioritize real-time ephemeral event data, write caching delivers much faster performance than the impressive speed Redpanda achieves with guaranteed data durability. Internal benchmarks on write caching indicate a latency improvement of up to 98%. Redpanda users can now opt for the fastest possible streaming performance for applications that have more relaxed durability requirements. This tradeoff offers several benefits:

The ability to fine-tune hardware price-performance by running Redpanda on leaner infrastructure.
Support for high-volume write workloads, better accommodating scenarios where message batching is not possible.
Users can blend write caching and guaranteed data safety within a single Redpanda deployment, providing enhanced flexibility to clusters running a mix of workloads.

Streamlined Processing with Redpanda Data Transforms

Redpanda 24.1 also introduces enhancements that empower developers to build more efficient and reliable data pipelines. Built on WebAssembly, Redpanda Data Transforms enable developers to process individual messages on the fly, enriching and filtering data as it flows through the system. Now generally available, Redpanda Data Transforms has added features for self-hosted deployments:

Rust SDK: Rust support adds to previous Golang support, giving developers greater choice of the language they use for specific tasks. Additional languages are on the roadmap.
Write to Multiple Topics: Developers can now apply more sophisticated logic for pre-processing data. Data Transforms now support the ability to take one input topic and create custom code that fans out writes to multiple output topics.
Audit Logs: Data Transforms are now easier to debug, thanks to runtime logs written to an internal topic. Transform logs are accessible using the CLI and in Redpanda Console.

More Predictable Performance with Topic-Aware Partition Balancing

Topic-aware partition balancing optimizes resource allocation by intelligently distributing data across a cluster, while better accounting for the performance demands of each workload. It ensures no single server becomes overloaded, enabling efficient scaling as data volumes climb.

Simplified Access Control with RBAC

Especially useful for large organizations in regulated industries, role-based access control gives admins an easy and efficient way to onboard new employees, audit access and adapt to changing usage patterns and compliance needs. RBAC is currently available for Redpanda Enterprise edition only.

Available Now

Access the 24.1 release of Redpanda Community edition on GitHub. Visit redpanda.com or join the Redpanda Community slack with any questions. The fastest way to try Redpanda is to spin up a Redpanda Serverless cluster in less than a minute – starting with a free 14-day trial.

About Redpanda

Redpanda is the streaming data platform for developers. API-compatible with Apache Kafka, Redpanda introduces a breakthrough architecture and disruptive capabilities that make it an easy, fast, scalable, and cost-effective engine of record for both real-time and historical enterprise data. Innovators like ShareChat, Jump Trading, Vodafone, Moody’s, The Hotels Network and Alpaca rely on Redpanda to process hundreds of terabytes of data a day. Backed by premier venture investors Lightspeed, GV and Haystack VC, Redpanda is a diverse, people-first organization with teams distributed around the globe. To learn more, visit our website at redpanda.com and follow us on X (formerly Twitter) at @redpandadata.