Kafka vs. Kinesis vs. Pub/Sub: Pick One Before You're Stuck

For: A CTO or senior engineer at a Series B SaaS company who is instrumenting real-time event pipelines for the first time and has shortlisted Kafka, Kinesis, and Pub/Sub — but cannot find a comparison that goes beyond managed-vs-self-hosted and language SDKs to address the axis that will actually bite them: whether their failure mode is message ordering, replay depth, or egress cost at scale

Pick Kafka if your dominant risk is deep replay and multi-cloud portability. Pick Kinesis if you are already deep in AWS and your event volume is predictable enough to avoid shard-split pain. Pick Google Pub/Sub if you want zero operational overhead and you are confident your data will never need to leave GCP or be replayed beyond seven days. That is the answer. The rest of this post explains why the axis that will bite you at month 12 is not throughput — it is replay window depth, partition management under bursty load, or egress cost after your architecture has calcified around one vendor.

Most Kafka vs Kinesis comparisons stop at "managed vs self-hosted" and language SDK support. That framing is useless once you are past the pilot. Below is the framing that actually matters when you are instrumenting your first real-time pipeline and cannot afford to migrate two years in.

The three failure modes nobody benchmarks

Every streaming platform can move millions of events per second on paper. The question is not whether it can. The question is what breaks first when your assumptions do.

Failure mode 1: Replay depth (Kafka wins, Pub/Sub loses quietly)

You will need to replay events. Not maybe — definitely. A downstream consumer will have a bug that silently corrupts a materialized view. Someone will ship a schema change that drops a field. You will onboard a new analytics workload that needs six months of history to backfill a feature store. This is not exotic; it is table stakes for any event-driven SaaS.

Kafka stores messages on disk for as long as you configure retention — days, months, years. Retention is a broker setting. You control it. Storage is your problem, but so is the flexibility.

Kinesis Data Streams retains data for up to 365 days (extended retention), but you pay per shard-hour for it and the per-GB retrieval cost climbs steeply past 24 hours. Kinesis was designed as a buffer, not a system of record.

Google Pub/Sub retains unacknowledged messages for up to 7 days by default. Pub/Sub Lite and topic snapshots give you more, but the primitive itself is not built for deep replay. If your architecture assumes "the log is the source of truth," Pub/Sub will fight you.

Failure mode 2: Bursty load and partition management (Kinesis punishes)

Kinesis charges by shard. Each shard supports 1 MB/s write and 2 MB/s read. When you exceed that, you get ProvisionedThroughputExceededException. The fix is to split shards — which is a manual operation on classic Kinesis, or on-demand mode's automatic scaling, which has its own lag (roughly minutes to react to sustained load, not seconds).

If your traffic is spiky — say, a fintech reconciliation batch, a Black Friday e-commerce spike, or a logistics dispatch surge at shift change — Kinesis's shard model will either drop writes or force you to over-provision. On-demand mode helps but is not free and is not instantaneous.

Kafka partitions are more forgiving because throughput per partition depends on your broker sizing, not a hard AWS-enforced ceiling. Pub/Sub abstracts partitioning away entirely — it scales elastically at the cost of not letting you reason about per-partition ordering the way Kafka does.

Failure mode 3: Egress cost and lock-in (Pub/Sub and Kinesis both punish, quietly)

Here is the one that surprises people two years in. You build everything around Pub/Sub. Then a customer requires their data live in AWS. Or your analytics team wants Databricks on Azure. Or you decide to run a DR region in a different cloud. Suddenly you are paying GCP egress on every event you replay or mirror out.

Same story with Kinesis. Cross-region replication is doable but not free, and cross-cloud is worse. Kafka running on your own infrastructure (or on Confluent, or MSK with careful VPC peering) gives you portability. That is not a small thing — it is often the difference between a two-week migration and a nine-month one.

Head-to-head on the axes that matter

When each one is the right call

Pick Kafka when

• The log is your system of record. You are doing event sourcing, CDC into a warehouse, or feeding a feature store where reprocessing months of history is a normal operation.
• You need multi-cloud or hybrid deployment. Kafka speaks a wire protocol, not a vendor API. You can run it on MSK today and move to Confluent Cloud or self-hosted tomorrow without rewriting producers and consumers.
• Your team can afford the operational tax — or you buy it managed. MSK and Confluent Cloud remove most of the ZooKeeper/KRaft, broker-rebalancing, and JVM-tuning pain, but you still need to think about partitions, consumer groups, and rebalance storms.

What Kafka is bad at: low-volume, bursty, fire-and-forget workloads where you just want a queue. It is overkill and the operational surface area is a liability. Also bad at ad-hoc pub/sub fanout to hundreds of ephemeral subscribers — Pub/Sub does that natively; Kafka needs consumer group gymnastics.

Pick Kinesis when

• You are already all-in on AWS and the rest of your stack is IAM, Lambda, DynamoDB, Firehose, and Redshift. The integration surface is genuinely tighter than Kafka on AWS.
• Your throughput is predictable, or spiky in ways you can model. On-demand mode helps with the second case but is not magic and will lag real traffic spikes.
• You do not need replay beyond a few days as a hot-path operation.

What Kinesis is bad at: shard math surprises. If a single hot partition key (a whale customer, a viral product ID) sends more than 1 MB/s, you are stuck — repartitioning requires changing the partition key strategy in the producer. Also bad at deep replay economics: the 365-day retention exists, but you will feel the bill.

Pick Pub/Sub when

• You want to write producer and consumer code and never think about capacity again. Pub/Sub genuinely delivers on this — it is the least operationally demanding of the three.
• You are on GCP and your downstreams are Dataflow, BigQuery, or Cloud Functions.
• Your events are fire-and-forget. Notifications, task queues, fanout to microservices — Pub/Sub is excellent here.

What Pub/Sub is bad at: being a system of record. The 7-day retention is a design statement, not an accident. Ordering was retrofitted (ordering keys) and is not the natural model. And the egress cost of ever leaving GCP with your event history is the sleeper issue nobody warns you about at pilot time.

The real decision framework

Instead of asking "which is fastest" or "which is easiest," ask these three questions in order:

1. Will I need to replay more than 7 days of events as a routine operation? If yes, Kafka. Full stop. Kinesis extended retention is an escape hatch, not a plan.
2. Is my traffic bursty in ways I cannot predict? If yes, avoid Kinesis unless you accept on-demand mode's cost profile. Kafka with room to grow, or Pub/Sub, will treat you better.
3. Is there any scenario in the next three years where my data or workloads need to live in another cloud? If yes, Kafka. The cost of egressing years of event history out of Pub/Sub or Kinesis is not theoretical — teams discover it during acquisition due diligence, enterprise contract negotiation, or a compliance-driven region move.

If all three answers are no — short retention, predictable load, single-cloud forever — pick the managed option native to your cloud and move on. The operational simplicity is real and worth it.

A few things nobody tells you at the pilot stage

Ordering keys are not free. Pub/Sub ordering keys serialize delivery per key. If you have a hot key, you have a bottleneck. Same math as Kinesis partition keys, just presented differently.

Exactly-once is a spectrum, not a checkbox. Kafka has transactional producers and idempotent writes. Kinesis has KPL deduplication with caveats. Pub/Sub added exactly-once delivery in 2022 but only within a subscription. Read the fine print for each before you promise a downstream team "exactly once."

Schema management is your problem regardless. Confluent Schema Registry, AWS Glue Schema Registry, and GCP's schema service all exist. They do not talk to each other. Picking a platform locks in schema tooling too.

Consumer lag monitoring is not the same as system health. Kafka lag is measured in offset gap. Kinesis in MillisBehindLatest. Pub/Sub in oldest_unacked_message_age. If your on-call runbook says "check lag," make sure everyone knows what lag means on your platform.

Where teams actually go wrong

The single most common regret we see when teams come to us for platform modernization work is not that they picked the wrong streaming platform. It is that they picked one before they understood their replay and portability requirements — because at pilot time, nobody was asking those questions. They optimized for "easiest to get running this quarter" and inherited a constraint that shaped every architectural decision for the next three years.

If you are building a fintech ledger, a logistics dispatch system, or a healthcare event backbone — the kinds of systems where replaying history is a compliance requirement, not a nice-to-have — do not let the pilot-ease argument win. We have seen this play out in accounting SaaS and logistics marketplaces where the event log becomes load-bearing much faster than anyone predicted.

Frequently Asked Questions

Can I use Kafka on AWS instead of Kinesis?

Yes, and many teams do. AWS MSK is managed Kafka on AWS with tight IAM and VPC integration. Confluent Cloud also runs on AWS. You get Kafka's replay depth and portability while staying inside AWS networking. The trade-off is slightly higher operational surface than Kinesis and you lose some of the native Lambda/Firehose ergonomics.

Is Google Pub/Sub really unsuitable for event sourcing?

Not unsuitable, but not natural. Pub/Sub was designed as a messaging service, not a durable log. Pub/Sub Lite and topic snapshots extend its capabilities, but if event sourcing is central to your architecture, Kafka's semantics match the pattern better. Teams that pick Pub/Sub for event sourcing usually end up building compensating infrastructure around it.

How do I estimate the egress cost of leaving one platform later?

Model your total event volume per month in GB, then apply your cloud provider's egress rate for cross-region or cross-cloud transfer. Then multiply by your retention window. For most SaaS companies with years of event history, this number is uncomfortably large. If cross-cloud is even a possibility, factor it into the pick decision, not the migration decision.

What about Redpanda, NATS, or Pulsar?

All three are legitimate. Redpanda is Kafka-API-compatible with a simpler operational model (no ZooKeeper, no JVM). Pulsar has better multi-tenancy and geo-replication primitives. NATS JetStream is lighter-weight for smaller footprints. If you are seriously evaluating streaming, they belong on the shortlist. This post focused on the three most common shortlisted options at Series B.

How long does it take to migrate from one to another?

It depends on how deeply the platform's semantics are embedded in your producer and consumer code, and on how much historical data must be moved. For a serious assessment of migration scope for your specific stack, talk to CodeNicely for a personalized assessment.