Migrate a Live Webhook-Heavy Integration Without Dropping Events

For: Engineering lead at a 30–150-person B2B SaaS company who owns a payment, CRM, or data-pipeline integration built on inbound webhooks — and has been told to swap the upstream provider or re-platform the receiving service without losing a single event mid-flight

To migrate a live webhook-heavy integration without dropping events, run both the old and new receivers in parallel behind a fan-out proxy, promote the new receiver to primary only after it has processed a full retry window of duplicate traffic idempotently, and keep the old endpoint returning 200s for at least 2x the sender's maximum retry horizon after cutover. The event loss risk people plan for — the cutover second itself — is not where migrations actually break. They break in the retry tail afterward, when the sender re-delivers events the old system already acknowledged and the new system processes them as if they were fresh.

This playbook assumes you own the receiver, not the sender. Stripe, HubSpot, Shopify, Segment, Twilio, GitHub — you don't get to pause their outbound queue. You have to migrate underneath live traffic while the sender keeps firing, retrying failed deliveries on its own schedule, and expecting a 2xx within its timeout budget.

Who this is for

You lead engineering at a B2B SaaS company. You have one or more integrations built on inbound webhooks — a Stripe payments pipeline, a HubSpot CRM sync, a Segment event stream, a GitHub App, a Twilio delivery-status feed. The events are stateful: they mutate customer records, trigger billing, kick off workflows. You've been asked to either (a) swap the upstream provider (Stripe to Adyen, Segment to Rudderstack), or (b) re-platform the receiving service itself (Lambda to Kubernetes, monolith to a dedicated ingestion service, on-prem to cloud). Missing events is not acceptable. Duplicate processing is also not acceptable.

Every migration guide you've read assumes you control both sides. You don't. That's the whole problem.

The insight that changes the plan

The dangerous window in a webhook migration is not the cutover. It is the retry tail after the cutover.

Here's the sequence that quietly duplicates events in production:

1. Sender delivers event E to old endpoint at T-0. Old endpoint 200s. Sender marks E as delivered.
2. Between T-0 and T-cutover, something in the sender's internal queue gets flaky — a node restarts, a delivery attempt is re-queued, a status flag isn't flushed. Some senders will re-deliver E within their retry window (24h for Stripe, 72h for many others) even though they got a 200.
3. At T-cutover you move DNS or update the webhook URL. Old endpoint gone.
4. At T-cutover + 6h, sender re-delivers E to the new endpoint. New endpoint has no memory of E. Processes it. Charges the customer twice, or sends a duplicate onboarding email, or writes a duplicate row.

Idempotency keys catch this — if the new receiver shares an idempotency store with the old one. Almost no team plans for that. They build a fresh idempotency table on the new stack because it's a fresh stack. That's the bug.

The whole playbook below is organized around not having that bug.

Step 1: Inventory the sender's actual behavior, not its documentation

Before you touch anything, you need a real answer to five questions per sender:

• Maximum retry horizon. How long after the original delivery attempt will this sender still be retrying? Stripe: up to 3 days. Shopify: 48 hours. GitHub: about 8 hours. Custom internal senders: whatever a junior engineer wrote in 2019.
• Retry trigger. Non-2xx only? Timeout only? Or does it retry on internal queue re-processing even after a 200?
• Delivery ID / idempotency header. What field uniquely identifies a delivery attempt vs. an event? Stripe distinguishes event.id (stable) from delivery attempts. GitHub uses X-GitHub-Delivery. Some senders give you nothing stable — flag those as high risk.
• Ordering guarantees. None, in practice. But confirm.
• Signature scheme and whether it's tied to endpoint URL. Stripe signing secrets are per-endpoint. If you register a second endpoint, you get a second secret. Your parallel receiver must handle both.

Write this down as a table. It becomes the reference for every decision after this.

Anti-pattern: reading the sender's docs and assuming they match reality. Look at 30 days of your own webhook logs. Group by event ID. Count how many events show up more than once, and how far apart. You will almost always find non-zero duplicates already — that's the baseline you're migrating against.

You'll know this step is done when you can state, for every sender, the exact retry horizon in hours and the exact field you'll use as an idempotency key across both old and new receivers.

Step 2: Build a shared idempotency store before you build anything else

This is the single most important decision in the migration. The old receiver and the new receiver must share an idempotency layer that survives the entire cutover plus the sender's maximum retry horizon plus a safety margin. If your retry horizon is 72 hours, this store needs to hold delivery IDs for at least 7 days.

Concretely:

• A Redis or DynamoDB table keyed on (sender, event_id) or (sender, delivery_id), whichever is stable.
• TTL set to at least 2x the maximum retry horizon.
• Written to on successful processing, not on receipt. If you write on receipt and then crash, you'll drop events.
• Checked by both the old receiver (as a bolted-on middleware if it doesn't already have one) and the new receiver.
• The value stored should include the processing result or a pointer to it, so a duplicate delivery can return the same response the sender got the first time.

If the old receiver already has an idempotency table but it lives in the old service's private database, you have two options: either expose it via an internal API that the new receiver calls, or replicate it to the shared store as a one-time backfill plus ongoing dual-write. Backfill first, then start dual-writing, then verify the new store is at parity, then flip the check.

Anti-pattern: letting each receiver keep its own idempotency table "just for the migration." You will forget to reconcile them. Duplicate processing will happen. Nobody will notice for a week.

You'll know this step is done when you can send the same webhook payload to both the old and new receivers and both return the same processed response, having consulted the same idempotency store.

Step 3: Stand up the new receiver in shadow mode

Do not change the webhook URL at the sender yet. Instead, put a fan-out proxy (or a fan-out at the load balancer / API gateway) in front of the old endpoint. For every incoming webhook:

1. Deliver synchronously to the old receiver. Return its response to the sender. This is the source of truth for the sender's perspective.
2. Asynchronously (fire-and-forget, but durable — SQS, Kafka, whatever you have) copy the request to the new receiver.
3. The new receiver processes normally. Its writes go to a parallel or shadow set of resources — a separate schema, a feature-flagged code path, a shadow table.

Run this for at least one full sender retry horizon, ideally longer. During this window you're checking three things:

• Parity: for every event the old receiver processed, did the new one produce the same downstream state? Diff the outputs. Log the diffs. Fix them.
• Signature validation: the new receiver is validating with its own signing secret (if the sender allows multiple registered endpoints) or with the same secret via the proxy (if it doesn't). Both paths have gotchas — the proxy path means the signature is computed against the original URL, not the new one, so your new receiver must not check the Host header.
• Idempotency: when the sender retries a webhook (which they will, naturally), does the new receiver correctly de-dup via the shared store?

Anti-pattern: shadow mode that writes to real downstream systems. If your new receiver in shadow mode is calling Stripe, sending emails, or writing to the production customers table, you're not shadowing — you're double-processing. Every side effect must be gated behind a shadow-mode flag.

You'll know this step is done when you have a full retry-horizon window of parity between old and new receivers, with a documented list of every diff and why it's acceptable (or fixed).

Step 4: Promote the new receiver to primary, keep the old one warm

Now the actual cutover. This is where most guides stop and where the real risk starts.

Two possible paths depending on what you control at the sender:

Path A: The sender supports multiple registered endpoints. Register the new endpoint. Both endpoints now receive every event independently from the sender. The old endpoint keeps processing normally (via the shared idempotency store, so it de-dups against the new one's writes and vice versa). Monitor for a full retry horizon. Then de-register the old endpoint at the sender. Keep the old receiver's HTTP handler alive but make it a no-op that returns 200 and writes to the idempotency store — for another full retry horizon.

Path B: The sender supports only one endpoint. Update the URL at the sender. Your fan-out proxy is now the front door. Flip the proxy config so the new receiver is the primary (synchronous, its response goes back to the sender) and the old receiver is the shadow. The old receiver continues to process for a full retry horizon so that any in-flight retries the sender fires against the new URL — for events the old receiver already processed — are correctly de-duped via the shared store.

In both paths, the critical rule: the old receiver's writes must remain idempotent against the new receiver's writes for the entire retry horizon after cutover. This is why the shared idempotency store from Step 2 exists.

Anti-pattern: tearing down the old receiver on cutover day because "the new one is live now." The sender's queue does not know about your cutover. It will retry old deliveries. Those retries need somewhere to land that de-dups correctly.

You'll know this step is done when the new receiver has been primary for at least one full retry horizon with zero de-dup misses in the shared store and zero parity diffs.

Step 5: Bleed off the old receiver

Now you can start dismantling — carefully, in this order:

1. At sender: de-register the old endpoint (Path A) or confirm the URL update has propagated for the full retry horizon (Path B).
2. Keep the old receiver running as an idempotency-only endpoint (200 + write to shared store) for another full retry horizon.
3. Watch traffic to the old endpoint. When it goes to zero for a full 24-hour window, you can turn it off.
4. Keep the shared idempotency store alive with its full TTL for one more retry horizon after the old receiver dies. This handles the case where a late retry hits the new receiver and needs to be de-duped against an event the old one processed.
5. Only then can you decommission the old service, delete the shadow schema, and remove the fan-out proxy.

Anti-pattern: celebrating cutover on day one. Migration is not done at cutover. It is done when the retry tail is fully drained and the idempotency store has aged past its useful window.

You'll know this step is done when old endpoint traffic has been zero for longer than the sender's maximum retry horizon, and no de-dup hits from the shared store have referenced the old receiver in that window.

Step 6: Instrument the whole thing so you'd actually notice a problem

None of the above matters if you can't see it going wrong. Minimum instrumentation:

• Per-sender counter of received events, processed events, de-duped events, signature failures, and 5xx responses. Alert on any of these deviating from the trailing 7-day baseline by more than 20%.
• Business-metric canaries: count of successful payments, count of CRM contacts created, count of pipeline events landed downstream. These should be within noise of pre-migration baselines. If they drop, you're dropping events regardless of what your webhook counters say.
• End-to-end synthetic events. Fire a known test event from the sender (or a spoofed sender if the real one supports it) every 5 minutes and confirm it lands in the downstream system with the correct state.
• A dead-letter queue on the new receiver, alarmed. Events that fail signature validation, fail parsing, or throw during processing land here — not in the void.

You'll know this step is done when you can stop watching the migration in real time and trust the alerts to page you if something goes wrong.

Failure modes I've seen

The signing-secret trap. Team registers a second endpoint at Stripe for parallel testing, gets a new signing secret, deploys the new receiver with the new secret, forgets that the fan-out proxy is forwarding traffic from the old endpoint (with the old secret). New receiver rejects everything as invalid signature. Silent, because the old receiver is still succeeding. Only caught when someone checks the shadow parity dashboard.

The idempotency-key-that-isn't. Team uses the sender's id field as the idempotency key. Turns out the sender resets that ID on retry — the stable field was actually event.id, and id was delivery.id. Every retry looked like a new event. Duplicates everywhere.

The Lambda cold-start timeout. New receiver is on Lambda. Cold starts push occasional responses past the sender's timeout (Stripe times out webhooks at ~10 seconds for the first response). Sender retries. Old receiver, still warm, also processes. Both receivers succeed, both write to downstream. Idempotency store catches most of it, but not the ones where the Lambda 200'd just after the sender gave up.

The clock-skew replay window. Sender signs webhooks with a timestamp and expects the receiver to reject anything older than N minutes to prevent replay attacks. New receiver's system clock is off by 90 seconds. About 8% of legitimate retries get rejected. Looks like intermittent flakiness, gets diagnosed after two weeks.

The DNS-TTL surprise. Team updates the webhook URL at the sender assuming DNS propagation is instant. Sender's DNS resolver caches for an hour. Half the traffic keeps hitting the old IP for 60 minutes after the "cutover." This is fine if you followed Step 4 and kept the old receiver hot. Catastrophic if you didn't.

How CodeNicely can help

Most webhook migrations we're pulled into are fintech and logistics — domains where dropping a single event is not a rounding error, it's a customer-visible failure. The engagement that maps most directly to this playbook is GimBooks, the YC-backed accounting SaaS. GimBooks runs a heavy inbound integration surface — payment providers, GST filings, bank feeds — and any migration on that side needs the parallel-receiver, shared-idempotency approach described above, because the downstream side effects (invoices, tax filings) are legally binding and cannot be duplicated or lost.

Where we tend to help: designing the shared idempotency layer, building the fan-out proxy, writing the parity diffing, and running the retry-tail monitoring for the full window after cutover. We don't touch cost or timeline in a blog post — those depend on your sender surface area and your current receiver architecture. If you want a scoped assessment, talk to us about your integration surface and we'll come back with a specific migration plan.

For teams doing this as part of a broader re-platforming (monolith to services, on-prem to cloud), the webhook migration is usually the highest-risk sub-project inside a larger legacy modernization effort, and it's worth sequencing first so the rest of the migration has clean event ingestion to build on.

Frequently Asked Questions

How long should the old webhook endpoint stay alive after cutover?

At minimum, 2x the sender's maximum retry horizon. For Stripe that means at least 6 days. For a sender with a 24-hour retry window, at least 48 hours. During that period the old endpoint should still be running, still writing to the shared idempotency store, and still returning 200s — even if all it does is de-dup and no-op.

Can we skip the parallel/shadow phase if we have good idempotency keys?No. Idempotency keys protect you from duplicate processing but not from missed events, silent parity bugs, or signature-validation errors on the new receiver. The shadow phase is where you catch bugs that only show up under real production traffic patterns, before those bugs affect a real event.

What if the upstream provider only supports one webhook endpoint at a time?

Use a fan-out proxy at your edge (nginx, a small Lambda, an API gateway route). The sender sees one URL — your proxy — and the proxy delivers synchronously to whichever receiver is currently primary and asynchronously to the other. This lets you swap primary/shadow via config rather than by re-registering endpoints at the sender.

How do we handle events that were in-flight at the exact moment of cutover?

They land at whichever receiver the sender's DNS resolves to at that instant, and the shared idempotency store handles the rest. If both receivers process the same event (because of a race between DNS propagation and the fan-out proxy), the second one's downstream side effects are suppressed by the idempotency check. This is the entire reason Step 2 exists.

Do we need this playbook for outbound webhooks we send to customers?

Different problem. Outbound webhooks — where you're the sender — give you full control over the retry queue, so you can pause deliveries, drain the queue, cut over, and resume. The specific difficulty this playbook addresses only exists when you don't control the sender. For outbound migrations, a queue-drain-and-resume approach is usually enough.