Kafka vs. Pub/Sub vs. SQS: Pick the Right Event Bus

For: A CTO or senior engineer at a 50–300-person B2B SaaS company whose event-driven architecture is working fine at current load but is about to absorb a new real-time feature — audit logging, cross-service order state sync, or customer-facing activity feeds — and needs to decide whether to extend their existing SQS queues, adopt Kafka, or wire up Google Pub/Sub before the next sprint kicks off

Pick based on retention model, not throughput. If any future consumer might need to replay history — a new analytics service, an audit trail, a re-hydrated cache — you need a log (Kafka or Google Pub/Sub). If every message has exactly one consumer and is safe to delete on ack, SQS is fine and cheaper to operate. Everything else in the Kafka vs SQS vs Pub/Sub debate is secondary to that one architectural fork.

This post is for the CTO or staff engineer who has SQS working today, is about to add a real-time feature (activity feed, audit log, cross-service order sync), and is trying to decide whether to extend what works or introduce a log-based broker before the sprint starts. I'll skip the marketing-page feature matrix and focus on the dimensions that determine whether you'll be re-platforming in eighteen months.

The one question that decides it

Ask this: six months from now, when a new service needs to consume this event stream from the beginning, can it?

With SQS, the answer is no. Once a message is acknowledged and deleted, it is gone. SQS is a destructive queue: messages exist to be consumed once and removed. If you add a second consumer later, you either fan out at publish time (SNS → multiple SQS queues) or you rebuild history from your database.

With Kafka, the answer is yes. Messages sit on a partitioned, append-only log for as long as your retention policy allows — days, weeks, or forever with tiered storage. A new consumer group starts at whatever offset it wants, including offset zero, and reads independently of every other consumer.

With Google Pub/Sub, the answer is yes, but with an asterisk. Pub/Sub is a queue by default, but subscriptions support message retention (up to 7 days by default, 31 days maximum with retained acked messages) and Seek to a timestamp or snapshot. It's log-like inside a bounded window, not an infinite log.

That single distinction — durable log vs destructive queue — is the difference between event streaming and message queueing in production, and it dictates almost everything else worth comparing.

Head-to-head: the dimensions that matter

Where each one actually fits

SQS: keep it if the workload is a task queue

SQS is right when the semantics are "a producer drops work, a pool of workers picks it up, the message is done." Image processing. Email sends. Webhook deliveries. Async database writes. One producer, one logical consumer, no need to look at the message ever again after it's processed.

SQS falls apart the moment two services need the same event. The standard workaround — SNS topic fanning out to multiple SQS queues — works, but it's fan-out at publish time. If you add a third consumer tomorrow, the events it missed yesterday are gone. You cannot subscribe historically. That's the trap teams walk into: SQS looks like it's scaling fine, then product asks for an activity feed that shows the last 90 days, and there is no source of truth outside the primary database.

SQS is also weak on ordering. FIFO queues cap at 300 transactions per second per message group without batching. If your ordering key is coarse (e.g., tenant_id) and one tenant is hot, you'll hit that ceiling.

Kafka: pick it when the stream itself is the source of truth

Kafka is the right choice when events are first-class data, not just transport. Order state machines. Audit logs. CDC pipelines feeding a warehouse. Real-time analytics. Anything where you'll want to reprocess history because you changed a downstream aggregation, or add a new consumer that needs the full picture.

The operational cost is real. Self-hosting Kafka means managing brokers, ZooKeeper (or KRaft), partition rebalancing, consumer lag monitoring, and disaster recovery. Managed offerings — AWS MSK, Confluent Cloud, Redpanda Cloud, Aiven — remove most of that, at a price. For a 50–300-person SaaS company, the honest answer is: don't self-host Kafka unless you already have platform engineers who want to. Use MSK or Confluent.

Kafka is bad at being a simple task queue. Consumer group rebalancing, offset management, and partition count planning are overhead you don't want for "send this email." Don't use Kafka for jobs that would be one line of SQS.

Google Pub/Sub: the pragmatic middle if you're on GCP

Pub/Sub gives you the multi-consumer subscription model (each subscription gets its own copy of every message, with independent acks) without the operational weight of Kafka. Add a new subscription tomorrow and it starts receiving new messages immediately. Enable message retention and you can seek back up to seven days by default, thirty-one with retained acked messages.

The catch: it's not a log in the Kafka sense. You can't rewind to any arbitrary offset from a year ago. If your replay window is bounded (24 hours, a week, a month), Pub/Sub is often the cleanest choice. If you're doing event sourcing where the log is the database, it's not.

Pub/Sub is also GCP-native. If your infrastructure is on AWS, you're introducing cross-cloud egress and a second control plane. That's usually not worth it unless you're already multi-cloud.

A decision tree that actually works

1. Does more than one logical consumer need this stream, now or plausibly within a year? If no → SQS. Move on.
2. Will any future consumer need to replay events older than 30 days? If yes → Kafka (managed). If no → continue.
3. Are you on GCP, or comfortable running a second control plane? If GCP → Pub/Sub is the lowest-friction answer. If AWS-only and replay window is short → SNS + multiple SQS queues can bridge the gap without a new broker.
4. Is the stream itself the system of record (event sourcing, audit, CDC)? Kafka. Not negotiable.
5. Do you need strict global ordering at high throughput? Kafka with a well-chosen partition key. SQS FIFO's per-group throughput ceiling will bite you.

The mistakes I see most often

Adopting Kafka for a task queue. A team reads about LinkedIn's architecture, decides Kafka is the "grown-up" choice, and ends up with three brokers, one topic, and a consumer that would have been fifty lines of SQS. They now have an on-call rotation for a broker.

Fan-out via SNS+SQS as a permanent architecture. This is a fine tactical move to add a second consumer without re-platforming. It becomes a problem when you have eleven SQS queues subscribed to four SNS topics, no replay, and every new consumer requires a CloudFormation change and a database backfill script.

Using Kafka retention as a database. Kafka can retain forever with tiered storage. That doesn't mean it should be your query layer. It's a log. Query it with a stream processor (Flink, Kafka Streams, ksqlDB) or materialize into a real database. Consumers that seek to offset zero on every restart because "it's cheap" will eventually be very slow.

Ignoring the partition key. On both Kafka and Pub/Sub with ordering keys, the partition/key choice determines ordering guarantees and hot-spotting. user_id is usually fine. region is usually a disaster. Think about skew before you ship.

How CodeNicely can help

Most of our work in this area comes from teams whose event flow is fine at current scale but is about to absorb a feature the original architecture wasn't designed for. On Vahak, a logistics marketplace with high-frequency state changes across trucks, loads, and shipper/transporter notifications, the interesting problem wasn't raw throughput — it was making sure every downstream service (matching, notifications, analytics, dispute resolution) could evolve independently without breaking the others or losing history. That's the same call you're making now: destructive queue vs replayable log, and how to migrate without a big-bang cutover.

If you're weighing whether to extend SQS with SNS fan-out, introduce Kafka behind a managed service, or move a subset of streams to Pub/Sub, we can review your current event topology, map it against the features on your roadmap, and give you a concrete migration path — including which streams stay on SQS and which need a log. See our digital transformation and engineering offerings for how we typically engage.

Frequently Asked Questions

Can I use Kafka and SQS together in the same architecture?

Yes, and it's often the right answer. Use Kafka for streams that multiple services consume or that need replay (order events, audit trail, CDC). Use SQS for task queues fed off those streams (send email, resize image, call third-party API). A common pattern is Kafka as the durable event backbone with SQS as the last-mile worker queue.

Is Google Pub/Sub really equivalent to Kafka for microservices?

For most microservice event-driven patterns with a bounded replay window, yes — subscriptions give you the multi-consumer semantics you actually need. Pub/Sub is not equivalent for event sourcing, long-term audit, or analytics pipelines that need to reprocess a year of history. Match the tool to the retention horizon.

How much operational overhead does managed Kafka actually add?

Less than self-hosted, more than SQS or Pub/Sub. On MSK or Confluent Cloud you still own topic design, partition counts, consumer group monitoring, schema management (usually with a schema registry), and consumer lag alerting. It's a real platform investment, not a "point and click" service.

What about NATS, RabbitMQ, or Redpanda?

Redpanda is Kafka-wire-compatible with lower operational overhead and is a legitimate alternative to MSK if you want Kafka semantics without the JVM. NATS JetStream is excellent for lightweight streaming, especially at the edge. RabbitMQ is a mature message broker but is closer to SQS in retention model than to Kafka. The retention-model question in this post applies to all of them: log or queue.

How do I estimate the cost and timeline for migrating from SQS to Kafka?

It depends heavily on how many streams, how many consumers, and whether you need zero-downtime cutover. Rather than a generic estimate, contact CodeNicely for a personalized assessment based on your current topology and roadmap.

The event bus you pick today is one of the harder decisions to undo. Get the retention model right first — log or queue — and the rest of the choices (managed vs self-hosted, cloud vendor, ordering strategy) get much easier. Get it wrong, and you'll spend a quarter rebuilding history from a database that was never meant to be a source of truth for the stream.