Message Queues Compared: SQS, Kafka, RabbitMQ, Redis Streams

What you need to know

• SQS is the default choice for AWS architectures that need managed, reliable message delivery without operational overhead. Start here unless you have specific requirements it cannot meet.
• Kafka is the right choice for high-throughput event streaming where multiple consumers need to process the same events independently, and where message retention and replay are requirements.
• RabbitMQ provides more sophisticated routing than SQS and is appropriate for complex message routing scenarios that SQS cannot express cleanly.
• Redis Streams is the pragmatic choice for teams already operating Redis who want a lightweight job queue without adding a new infrastructure component.
• The complexity order from lowest to highest operational overhead is: SQS, Redis Streams, RabbitMQ, Kafka (self-hosted).

The core argument

The message queue selection decision is primarily about operational complexity versus capability. SQS is a managed service: AWS runs it, scales it, and handles the operational complexity. The tradeoff is that SQS has limited routing capabilities, does not retain messages after consumption, and has a message size limit that matters for some use cases. For the majority of SaaS background job requirements, these limitations do not apply, and SQS is the pragmatic choice.

Kafka enters the picture when the use case requires capabilities SQS cannot provide: high-throughput event streaming, message retention beyond delivery, or multiple independent consumers processing the same stream. These requirements are real for data-intensive products, analytics pipelines, and event sourcing architectures. They are not real for most SaaS background job use cases. The mistake is choosing Kafka because it sounds like the right enterprise architecture when the actual use case is sending email confirmations and processing webhooks. In my experience, the teams that adopt Kafka at seed stage spend six months fighting the operational complexity before they have the scale to justify it.

Redis Streams occupies a useful middle ground. A product that already uses Redis for caching can add a message queue with no new infrastructure by using Redis Streams with consumer groups. The producer appends to a stream with XADD, the consumer reads with XREADGROUP, and the consumer acknowledges processed messages with XACK. The failure handling pattern is similar to SQS: unacknowledged messages can be reclaimed by other consumers after a timeout. The ceiling is lower than Kafka and the routing is simpler than RabbitMQ, but for moderate-volume queues within a service boundary, the zero-new-infrastructure advantage is significant.

Common mistakes

1. Choosing Kafka for background job processing. Kafka is not a job queue. It is an event streaming platform. Using it for standard background job processing introduces Kafka's operational complexity without leveraging its streaming and retention capabilities. Use SQS, RabbitMQ, or a managed platform for job queues.

1. Not configuring dead letter queues on SQS. An SQS queue without a DLQ sends messages that exceed the maximum receive count to the void. Configure a dead letter queue before processing any production messages.

1. Not planning for SQS visibility timeout. The SQS visibility timeout determines how long a message is hidden from other consumers while being processed. If a job takes longer than the timeout, the message becomes visible again and is processed twice. Set the visibility timeout to be longer than the maximum expected job execution time.

1. Running a single Kafka broker in production. A single Kafka broker has no fault tolerance. Kafka requires at least three brokers for production reliability. This minimum three-broker requirement is part of the operational cost that makes Kafka inappropriate for small-scale use cases.

1. Not designing for idempotency across all queue types. All four queue systems provide at-least-once delivery under certain conditions. A consumer that receives the same message twice must handle it without corrupting data. Idempotent message handlers are a requirement regardless of which queue you use.

Where to start

1. Start with SQS if you are on AWS and do not have specific requirements that exclude it. Create a queue with a matching DLQ, configure a visibility timeout appropriate for your job duration, and deploy a simple consumer. This takes two hours and works for most startup background job needs.

1. Evaluate Redis Streams if you already operate Redis. If SQS is not available (non-AWS infrastructure) or if you want to minimize the number of managed services, Redis Streams covers standard job queue patterns with the Redis installation you already have.

1. Only consider Kafka when you need event streaming semantics. If you need multiple consumer groups reading from the same stream, message replay, or high-throughput event ingestion, evaluate Confluent Cloud or MSK (Managed Streaming for Apache Kafka) before running Kafka yourself.

Related reading

• Inngest vs Trigger vs Temporal for Background Jobs
• Job Failure Recovery: How Good SaaS Companies Sleep at Night
• SaaS Webhook Reliability: From At-Most-Once to At-Least-Once to Exactly-Once
• Resilience Patterns: Circuit Breakers, Retries, Bulkheads