Redis Streams as a Poor Man's Kafka for Telemetry Pipelines

When you're building a product that ingests high-frequency events: API requests, metrics, logs, the first instinct for a lot of engineers is "we need Kafka." For early and mid-stage products, that's usually premature. Kafka is a phenomenal piece of technology, but it's also a cluster to provision, a set of brokers to monitor, partition rebalancing to reason about, and often a managed-service bill before you even know your real write volume. We hit this exact fork in the road while building our telemetry ingestion pipeline, and this is a full write-up of what we built instead, why it works, and, just as importantly, where it will eventually stop being enough.

The problem we were solving

Our SDK sits inside a customer's API as middleware. Every request it observes generates a telemetry event: status code, latency, route, response size, whether the request was blocked and why. Multiply that across every customer's entire fleet of running instances, and you get a write pattern that is bursty, high-frequency, and completely decoupled in time from when the customer's own API actually needs to respond to its caller.

That last point matters more than it sounds like it should. The telemetry pipeline has one hard constraint above everything else: it must never add latency to the customer's real request path. A customer's API call succeeding or failing cannot depend on whether our telemetry database happens to be healthy at that moment. That single constraint eliminates the naive approach (synchronously writing each event to a database from inside the request) immediately.

Beyond that "never block" constraint, we had three more requirements:

Events shouldn't be lost during a burst of traffic that outpaces the database's write throughput
Events shouldn't be lost if the database itself has a brief outage or hiccup
Events shouldn't be lost if a worker process handling them crashes mid-processing

None of these individually demand Kafka. But together, they demand something with real delivery guarantees, not just "fire an async task and hope." We already run Redis in our stack for other purposes, so before reaching for new infrastructure, we asked: how far can Redis Streams actually take us?

Why not just use a task queue?

Before landing on Streams, it's worth explaining why a simpler background-job queue (Celery, RQ, or even just asyncio.create_task fire-and-forget) wasn't sufficient. A generic job queue is built around the idea of a queue: an item is added, a worker picks it up, and once processed, it's gone. That model is fine for one-shot tasks like "send this email." It's a poor fit for telemetry because:

There's no built-in concept of multiple independent consumer groups reading the same data for different purposes
Failure recovery is usually bolted on rather than a first-class part of the data structure
You don't get an inspectable, ordered log you can replay or audit; once a job is done, it's typically gone

A stream, by contrast, is a persisted, ordered, append-only log, much closer to what Kafka gives you conceptually, with consumer groups, acknowledgments, and pending-entry tracking built directly into the data structure. Redis added exactly this primitive with the XADD/XREAD/XREADGROUP family of commands, and it turned out to be a much closer match to our actual requirements than a plain job queue.

The pipeline, end to end

Here's the full path an event takes, from the moment it happens inside a customer's API to the moment it's durably stored in our database:

Client-side batching. The SDK doesn't send one HTTP request per event. It batches events in memory and flushes them periodically (or when the batch reaches a size threshold), which dramatically cuts down the number of outbound network calls the customer's process has to make.
Collector ingestion. The batch is POSTed to our telemetry endpoint. The collector's only job at this point is to take that batch and call XADD to append it onto a Redis stream (we call ours telemetry:events). It does not touch MongoDB directly on the happy path. This is the key architectural decision: the collector's response time to the customer is bounded by "how fast can Redis accept an append," not "how fast can Mongo write."
Consumer group processing. A pool of worker processes runs a loop calling XREADGROUP against a named consumer group on that stream. Each worker pulls a batch of pending entries it hasn't seen yet.
Database write. The worker transforms the batch into documents and writes them with a single insert_many call rather than one write per event, batching here again, this time on the database side, to cut down on round trips.
Acknowledgment. Only after the insert_many succeeds does the worker call XACK to mark those specific stream entries as processed. If the write throws before the ACK, the entries remain in the group's pending-entries list, unclaimed and un-lost.

The reason this ordering, write first, then ACK, is not optional is that it's what gives you an at-least-once delivery guarantee. If a worker dies between the insert and the ACK, the entries are still sitting there, pending, waiting for something to claim them. The alternative ordering (ACK, then write) would give you at-most-once delivery, which silently loses data on any crash, exactly what we were trying to avoid.

Layer one: falling back when Redis itself is down

Every dependency you add to a critical path is also a new thing that can fail. If Redis itself becomes unreachable, we don't want telemetry ingestion to simply stop. The collector detects the failed XADD and falls back to writing the batch directly to MongoDB instead.

This is explicitly a degraded mode, not a silent equivalent path. You lose the batching efficiency on the database side (each fallback write is smaller and more frequent), and you lose strict ordering guarantees relative to whatever else is flowing through the stream at the time. But the alternative, refusing to accept telemetry, or worse, blocking the customer's request while Redis recovers, is strictly worse. A Redis outage becomes "our data pipeline runs a little less efficiently for a while," not "we lose data" or "we slow down customer requests."

Layer two: the dead-letter queue

Not every failure is transient infrastructure flakiness. Sometimes a specific event is malformed, or triggers an edge case in the write path, or hits some deserialization bug we haven't found yet. If a worker keeps retrying that one bad entry forever, it can effectively stall the whole consumer: it never ACKs, never moves on, and the pending list backs up behind it.

So we built a dead-letter queue on top of the same stream primitives. The rule is simple: each stream entry tracks how many delivery attempts it's had. Once an entry has failed processing a fixed number of times, we settled on three, instead of retrying it again in the main pipeline, it gets promoted onto a separate dead-letter stream and ACKed off the main one, so it stops blocking anything downstream.

A dedicated sweep process runs on its own schedule, separate from the main ingestion workers, and does two things: it periodically retries entries sitting in the dead-letter stream (in case the underlying issue was actually transient and has since resolved), and it discards entries that have been sitting there for too long to be worth keeping, we use roughly a three-day cutoff. Past that point, an event that still can't be processed is far more likely to represent a genuine bug worth debugging from logs than something worth continuing to retry automatically.

The result is a pipeline that degrades gracefully in the face of bad data: one malformed event gets isolated and quarantined rather than poisoning the throughput of everything behind it, and you still have a place to go look at exactly what failed and why, instead of it just vanishing.

Layer three: recovering from a crashed worker with XAUTOCLAIM

This is the failure mode that's easiest to get wrong if you haven't worked with Redis consumer groups before, and it's subtle enough that it's worth walking through carefully.

When a worker calls XREADGROUP and receives a batch of entries, Redis doesn't just hand them over and forget about it. It records those entries in a per-consumer data structure called the Pending Entries List (PEL). An entry sits in the PEL from the moment it's delivered until the moment it's explicitly ACKed. This is exactly what gives you the at-least-once guarantee: the entry isn't considered "done" until someone says so.

But here's the trap: if that worker process crashes (gets OOM-killed, the container restarts, the deploy rolls it over) before it ACKs those entries, they don't automatically get freed up or redelivered to another worker. They just sit in that dead consumer's PEL. Forever. Invisible to every other worker in the group, because as far as Redis is concerned, that consumer still "owns" them and might still be working on them. Nothing about a crash implicitly notifies Redis that ownership should be reassigned.

If you don't handle this explicitly, the failure mode is brutal precisely because it's silent: there's no error thrown, no exception logged, nothing shows up in a dashboard. A batch of telemetry events simply never makes it to the database, and unless you go digging into consumer group internals with something like XPENDING, you'd have no reason to suspect anything is wrong. It looks exactly like a quiet, healthy system that's just quietly losing data.

XAUTOCLAIM is the command that solves this. It scans a consumer group's pending entries for ones that have been sitting unacknowledged longer than a configurable idle-time threshold, and reassigns, "claims," them to a different, currently-alive consumer, which can then process them from scratch. We run this on a dedicated background loop, separate from the main ingestion workers, sweeping periodically for anything that's been abandoned. Combined with the DLQ delivery-attempt counter, an entry that gets claimed, reprocessed, and fails again eventually routes into the dead-letter path instead of being claimed and reclaimed indefinitely.

If you take away one thing from this entire post: if you're using Redis Streams consumer groups for anything where losing data quietly is unacceptable, a recurring XAUTOCLAIM sweep is not an optional hardening step. It is the thing that makes the "at-least-once" promise of consumer groups actually true in practice, rather than true only in the case where every worker happens to shut down cleanly.

Bounding memory: why MAXLEN matters

Streams are append-only by default, which means they grow forever unless you tell Redis otherwise, and unlike a disk-backed log, Redis is holding all of that in memory. An unbounded stream on an in-memory data store is a slow-motion outage waiting to happen: if your workers ever fall meaningfully behind (a bad deploy, a downstream outage, a spike far beyond normal traffic), the stream keeps growing to accommodate everything that hasn't been ACKed yet, and eventually you're staring at an out-of-memory Redis instance.

We cap the stream with MAXLEN, set to 100,000 entries, using the approximate trimming mode rather than exact trimming. The approximate mode matters for performance: exact trimming requires Redis to do more bookkeeping on every single append to guarantee the length is never off by even one, whereas the approximate mode trims in efficient chunks and tolerates the length being slightly over the cap between trims. For a telemetry pipeline, that imprecision is irrelevant; the cap exists to prevent runaway memory growth, not to enforce an exact accounting figure.

The tradeoff this introduces is explicit and worth stating plainly: if consumers fall behind by more than the cap, the oldest unprocessed entries get trimmed away and are gone, not moved to a DLQ, not recoverable, just dropped. We made a deliberate call that this is an acceptable failure mode for telemetry specifically. It is much better to lose some of the oldest events under an extreme, sustained backlog than to let an unbounded stream take down the Redis instance entirely and lose everything, including all the healthy, already-buffered data for every other customer. Telemetry data has value in aggregate and in recency; losing a slice of it under a rare worst-case backlog is a far smaller cost than a full outage.

Putting it all together: five background loops

In production, this whole system runs as five independent background loops, wired into the application's lifespan so they start on boot and shut down cleanly:

The main telemetry worker loop: XREADGROUP, batch insert, XACK
The DLQ sweep loop, retries entries from the dead-letter stream on a schedule
The DLQ retry/discard loop, ages out entries that have exceeded the retention cutoff
A cleanup loop, unrelated to the stream itself, handles retention-based deletion of old telemetry documents from the database according to each customer's plan
A notifications loop, also unrelated to the stream, but running in the same process pool pattern, handling alert delivery

None of these loops are particularly complex in isolation. What makes the overall system reliable isn't any single clever mechanism, it's that each of the three failure modes we identified up front (Redis down, one bad event, one crashed worker) has its own dedicated, independently testable handling path, rather than one generic try/except wrapped around everything and hoping for the best.

What Redis Streams gets you, and what it doesn't

To be direct about the tradeoffs, because it's easy for a post like this to read as "just use Redis Streams, Kafka is unnecessary," that's not the claim. Redis Streams genuinely gets you:

At-least-once delivery, via ACK-based tracking and the PEL
Multiple independent consumers reading the same stream via consumer groups
Built-in backpressure and crash-recovery primitives (PEL + XAUTOCLAIM) without any extra infrastructure beyond Redis itself
A workable dead-letter pattern built from the same primitives, no second broker required

What it genuinely does not give you, compared to Kafka:

Real long-term retention and replay at scale. You are bounded by MAXLEN and by how much memory you're willing to dedicate to Redis; Kafka is designed around disk-backed, cheaply-scalable retention measured in days or weeks by default
Native multi-datacenter replication and partition-level scaling across a large cluster
The surrounding ecosystem: Kafka Connect, ksqlDB, schema registries, and the broader tooling built around treating the log as a first-class data platform rather than a pipeline plumbing detail

For a telemetry ingestion pipeline where the actual requirement is "reliably move events from A to B without losing them under realistic failure conditions," that tradeoff comes out clearly in Redis's favor, especially at the traffic volumes most products operate at before they have product-market fit figured out. If we ever needed to replay a full year of raw events for a new analytics feature, or needed true multi-region active-active replication of the stream itself, that's the point where reaching for Kafka (or a dedicated log-oriented store) would stop being premature and start being the right call.

The takeaway

Not every high-throughput pipeline needs a dedicated streaming platform on day one. If you're already running Redis for anything else in your stack, Streams plus consumer groups get you a surprising amount of the reliability model people usually reach for Kafka to get: ordered delivery, at-least-once semantics, and crash recovery, for close to zero additional operational surface area. The parts you have to build yourself on top of the primitive (the DLQ sweep, the XAUTOCLAIM reclaim loop, the MAXLEN bound, and the failure-mode decisions each of those implies) are exactly the parts that determine whether your pipeline is actually reliable or just looks reliable until the first crash. Understanding those tradeoffs deeply is worth more than which specific tool you end up choosing.