5 Mistakes Teams Make When Adding Real-Time to a Batch System

For: A backend engineering lead at a mid-stage SaaS company whose product was built on nightly or hourly batch jobs and is now being asked by the product or ops team to make dashboards, alerts, or fulfillment flows 'real-time' — without a greenfield rewrite

Most real-time migrations don't fail in the streaming layer. They fail because the data model underneath was designed to be overwritten by a nightly job, and no amount of Kafka partitioning fixes a schema that was never meant to accept partial, ordered, incremental truth. The five mistakes below are the ones we see repeatedly when a batch-era SaaS backend gets asked to power live dashboards, alerts, or fulfillment — and they are all structural, not infrastructural.

If you're leading backend at a mid-stage SaaS company and product is asking for "real-time" without a greenfield rewrite, read this before you spin up another consumer group.

The setup: why batch-to-real-time is harder than it looks

A batch system encodes an assumption so deep most engineers don't notice it: the source of truth is whatever the last job wrote. Tables get truncated and reloaded. Aggregates get recomputed from scratch. Idempotency is trivial because you rerun the whole thing. Ordering doesn't matter because everything lands at once.

The moment you introduce an event stream — a Kafka topic, a Kinesis shard, a webhook fan-out — every one of those assumptions inverts. Now you need per-record idempotency, per-key ordering, partial state updates, and a way to reconcile streaming writes with the batch job that is still running at 2am and does not know your stream exists. This is the actual problem. The streaming infrastructure is the easy part.

Mistake 1: Treating the streaming layer as an add-on instead of a schema change

The mistake: The team stands up Kafka (or Kinesis, or Pub/Sub), writes a consumer, and points it at the existing tables. The tables were designed for bulk overwrite — no updated_at at the row level, no version column, no event sequence number, no soft deletes. The consumer does UPDATE ... WHERE id = ? and calls it done.

What causes it: Real-time gets framed as a delivery problem ("how do we get events from A to B faster") instead of a state problem ("how does our data model represent partial, ordered updates").

Symptom in production: Dashboards briefly show correct numbers, then jump backward at 2:07am when the nightly ETL finishes. Support tickets say "the number changed after I refreshed." You'll find yourself explaining to the CS team why the fulfillment status regressed from shipped back to picking.

How to recover: Before you tune a single consumer, add three things to every table the stream writes to: a monotonic version or sequence column, a last_event_ts, and a source-of-write marker (stream vs batch). Then rewrite the batch job to be a reconciler, not an overwriter — it may only update rows where the batch's timestamp is newer than last_event_ts, or it writes to a separate reconciled column that the read model merges. This is the smallest change that stops the two systems from silently fighting.

Mistake 2: Assuming exactly-once because the broker promises it

The mistake: Enabling exactly-once semantics in Kafka or using SQS FIFO and treating the problem as solved. Then writing consumers that call external APIs, send emails, or increment counters without idempotency keys.

What causes it: Broker-level exactly-once only covers producer-to-broker-to-consumer within the broker's transactional boundary. The moment your consumer talks to Stripe, sends a webhook, writes to a second database, or fires a Slack alert, you're back to at-least-once. Consumers restart. Pods get evicted. Rebalances happen. A message will be processed twice.

Symptom in production: Duplicate charges. Two "order shipped" emails to the same customer thirty seconds apart. An alerting channel that pages on-call twice for the same incident. Aggregate counters that drift upward over weeks and nobody can explain why.

How to recover: Every side effect needs an idempotency key derived from the event itself — event ID, or a hash of (entity_id, event_type, version). Store processed keys in a table with a TTL longer than your worst-case redelivery window. For counter aggregates, don't increment — upsert the event's contribution keyed by event ID, then sum. It's more expensive, and it's the only thing that actually works when the pod dies mid-batch.

Mistake 3: Ignoring ordering until a customer notices

The mistake: Partitioning topics by something convenient (round-robin, timestamp, hash of message ID) instead of by the entity whose state is being mutated. Or partitioning correctly but running consumers with internal parallelism that reorders within a partition.

What causes it: In batch, ordering is implicit — everything for a customer lands together and gets processed as a set. In streaming, if events for user_id=42 land on partitions 3 and 7, or get picked up by two worker threads inside the same consumer, the "account closed" event can be processed before the "payment received" event that preceded it by 200ms.

Symptom in production: Rare, hard-to-reproduce bugs where entity state is impossible — a subscription marked canceled that still has an active renewal, a shipment marked delivered before it was dispatched. It shows up as a support ticket, not a monitoring alert, because from the pipeline's perspective every message was processed successfully.

How to recover: Partition by the entity key that owns the state transition — usually tenant_id or entity_id, never message ID or timestamp. Enforce single-threaded processing per partition key, even if that means reducing throughput. If ordering across entities matters (rare, but real for financial ledgers), you need a single-partition topic or a sequencer service — and you should question whether streaming is the right pattern for that flow at all. Add an event sequence number produced at the source, and reject or park events that arrive out of sequence for a given key.

Mistake 4: Building the real-time dashboard on top of the same OLTP tables

The mistake: The dashboard team wants live numbers, so they point the frontend at the same Postgres tables the stream is writing to, with a five-second poll or a WebSocket subscription. Now every dashboard viewer is running aggregate queries against tables that are also handling transactional writes.

What causes it: It looks free. The data is already there. Why build a read model when you can just query? This works fine at low scale and collapses the first time a big customer opens the dashboard during a traffic spike.

Symptom in production: P99 latency on the transactional path degrades whenever someone opens the dashboard. Long-running aggregate queries hold locks that block streaming writes. The dashboard itself becomes inconsistent because the aggregates it computes don't match the row-level events users just saw. Eventually somebody adds a read replica, which helps for a quarter and then hits the same wall.

How to recover: Build a separate read model — a materialized view, a rollup table maintained by the stream, or a purpose-built store like ClickHouse or a time-series DB for time-bucketed metrics. The stream writes to both the transactional table (for correctness) and the read model (for the dashboard). Yes, this is CQRS. You don't have to call it that. The point is that the dashboard's query pattern and the transactional write pattern have nothing in common and should not share indexes, locks, or a query planner.

Mistake 5: No plan for replay, backfill, or schema evolution

The mistake: Shipping the streaming pipeline without answering three questions: How do we rebuild the read model from scratch? How do we backfill events for a customer who was onboarded before we turned this on? What happens when we add a field to an event payload?

What causes it: The pipeline works in staging, passes load tests, and ships. Six weeks later, a bug in the consumer logic corrupts the read model for a subset of tenants. Now you need to reprocess three weeks of events. Your retention is seven days.

Symptom in production: A silent data quality incident that can only be fixed by manual SQL scripts. A schema change that requires coordinating a deploy of the producer, all consumers, and a database migration in a specific order — because no consumer is tolerant of unknown fields, and no producer is tolerant of consumers that haven't caught up yet.

How to recover: Three disciplines, none optional. First, event retention should be measured in weeks or months for anything that feeds a read model, not days — use tiered storage or archive to S3 and make sure you can replay from the archive. Second, treat event schemas like public APIs: additive changes only, use a schema registry (Confluent, Buf, or a homegrown one on Protobuf/Avro), and version the payload. Third, make your consumers idempotent enough that replaying the last 30 days is a routine operation, not a crisis — because when you need to replay, it will be under pressure.

The pattern underneath all five

Notice what these mistakes have in common. None of them are about Kafka configuration, consumer lag, or throughput tuning. They are about the batch-era data model refusing to accept the constraints a stream imposes: order, partiality, replay, and coexistence with a batch job that still runs.

The productive way to think about a batch to real-time migration is not "how do we add streaming." It's "how do we evolve the data model so that both batch and streaming writers can safely coexist, and then move flows across one at a time." Almost every team that fails does it in the opposite order — infrastructure first, model second — and spends the next two quarters firefighting drift.

A few tactical things that make the transition survivable:

• Dual-write with reconciliation, not cutover. Run the batch job and the stream in parallel for weeks. Diff the outputs. Fix the stream until the diff is empty for a rolling window, then retire the batch write.
• Pick one flow, not the whole system. Migrate the highest-value read (usually a dashboard or an alert) end-to-end before touching anything else. Learn the failure modes on something reversible.
• Instrument the seams. The bugs live at the boundary between stream and batch, and between consumer and side effect. Log event ID, sequence, and processing outcome at every boundary. You will need this at 3am.
• Design for the incident, not the happy path. Assume you will need to replay, that a consumer will process a message twice, and that a batch job will run while the stream is live. If any of those breaks correctness, the design is wrong.

Teams working through this kind of legacy modernization often underestimate how much of the work is data model surgery versus infrastructure. It's a useful reframe when the ask from product is "just make it real-time." For context on how this plays out in domains where ordering and idempotency are non-negotiable, our work with Vahak on logistics event flows and GimBooks on accounting ledgers shows the same pattern in very different verticals.

Frequently Asked Questions

Do we need Kafka to move from batch to real-time?

No. Kafka is a good fit when you need durable event replay, high fan-out, and multiple independent consumers. For simpler cases — a single downstream service, moderate throughput — a managed queue (SQS, Pub/Sub) or change-data-capture from your existing database (Debezium, AWS DMS) is often the right first step. Choose the simplest tool that gives you ordered, durable, replayable delivery for your key granularity.

Can we keep the batch jobs and add real-time on top?

Yes, and for most mid-stage SaaS backends this is the right path. The critical rule is that batch and stream must not both be authoritative writers to the same field. Either the batch job becomes a reconciler that only fixes drift, or you migrate specific fields entirely to stream-owned and remove them from the batch job's write set. Coexistence works; silent competition doesn't.

How do we handle events that arrive out of order?

Attach a monotonic sequence number or a high-resolution timestamp at the source (not at ingest time). Consumers should compare the incoming event's sequence against the last processed sequence for that entity key and either apply, reject, or buffer. For most SaaS use cases, last-writer-wins with a version guard is sufficient. For financial or inventory flows, you likely need a stricter windowed reordering buffer.

What's the right way to test a streaming migration before rollout?

Shadow traffic. Run the new stream-based pipeline in parallel with the batch job, write its output to a separate table or namespace, and diff continuously. Alert on divergence. Only cut over reads once the diff has been stable at zero for a meaningful window under real production load — including peak hours and any known nightly batch overlap.

How long does a batch-to-real-time migration take?

It depends heavily on how coupled your data model is to batch assumptions, how many downstream consumers exist, and what your correctness bar is. If you'd like a grounded assessment for your specific system, talk to CodeNicely for a personalized review of your architecture and migration path.