Kafka in 2026: When You Need It and When You Do Not

# Kafka in 2026: When You Need It and When You Do Not

Kafka is a distributed event streaming platform built for high-throughput, fault-tolerant message delivery at massive scale. It processes millions of events per second, stores them durably on disk, and replays them on demand. Most companies do not need this. The ones that do are processing real-time analytics, coordinating between dozens of services, or running systems where losing a single event means losing money.

What you need to know

• Kafka shines at throughput above 100k events per second and retention requirements measured in days or weeks
• For most SaaS startups, Redis Streams, SQS, or a simple Postgres-backed job queue covers 90% of the use cases Kafka gets proposed for
• Operating Kafka yourself is a significant engineering burden; Confluent Cloud and Upstash reduce this, but add cost
• The replay capability is Kafka's most underrated feature; raw throughput is what usually gets cited but rarely what teams actually need
• If your first instinct is Kafka and your team is under 10 engineers, that instinct is probably wrong

The core argument

I have seen Kafka get proposed in three different contexts across projects I have worked on, and it was the right call exactly once. The other two times it was a combination of "it sounds serious" and "the architecture diagram looks impressive." Neither of those is a reason to run distributed event streaming infrastructure.

The one time it made sense was a pipeline that needed to ingest events from multiple upstream services, process them asynchronously, and make them available for both a real-time dashboard and a historical analytics query. That is a genuine Kafka use case. The data needed to flow to two different consumers with different latency requirements, and it needed to be replayable in case either consumer fell behind.

What it is not a use case for: sending welcome emails, running background jobs, triggering webhooks, processing form submissions, or notifying a single downstream service about a state change. All of those are covered by SQS, Redis Streams, or a Postgres table with a polling worker. When I built the notification pipeline for Nexli, I used a simple Redis-backed queue. No cluster management. No topic partitioning decisions. It works and it runs in production. The pattern I see most often: an engineer reads a blog post about Kafka, adds it to the architecture proposal, and the founder approves it because it sounds enterprise-grade. Six months later, the ops team is debugging replication lag and partition leader elections instead of shipping features. Kafka is not a status symbol. It is a tool with a specific job.

Common mistakes

1. Proposing Kafka because the company "will need it eventually." You might. But adding it before you hit the scale threshold means paying the operations tax on a future that may never come.

1. Under-provisioning the broker count. Kafka clusters need at least three brokers for fault tolerance. Developers who start with a single broker for testing end up with that single broker in production when the deadline hits.

1. Skipping schema management. Publishing raw JSON without a schema registry works until a consumer reads a field that changed three weeks ago. Avro with the Confluent Schema Registry is not optional at production scale.

1. Setting retention too short. The replay capability that justifies Kafka's operational cost only helps if the data is still there when you need it. Default retention settings are often too conservative.

1. Not monitoring consumer group lag. Kafka will let your consumers fall behind by millions of events. Without lag monitoring and alerting, you discover this problem when a customer asks why their dashboard is an hour behind.

Where to start

1. Map your actual throughput. Count the events your system produces per minute. If the answer is under 10,000 per minute, SQS or Redis Streams almost certainly covers you. If the answer is genuinely in the millions per minute, Kafka deserves a serious look.

1. Evaluate managed options first. If Kafka is the right call, Confluent Cloud or Upstash Kafka will cost more per event but save weeks of engineering time on cluster setup, monitoring, and upgrades. For a startup, that tradeoff almost always favors managed.

1. Define the replay requirement explicitly. The most defensible reason to choose Kafka over simpler queues is the ability to replay events to a new consumer. Write down whether you actually need that. If the answer is no, you probably do not need Kafka.

Related reading

• Message Queues Compared: SQS, Kafka, RabbitMQ, Redis Streams
• Event Driven Architectures: When They Help and When They Hurt
• Background Job Queues: The Architecture Decision Founders Skip
• The Outbox Pattern: A SaaS Reliability Cheat Code