Home › Blog › Businesses › Saas › Idempotent Webhook Consumers: A Step-by-Step Guide
SaaS technology
Businesses SaaS August 19, 2026 • 11 min read

Idempotent Webhook Consumers: A Step-by-Step Guide

For: A backend engineer at a Series B SaaS company whose payment or fulfillment webhook handler has started double-processing events under retry load, and whose team has just had a production incident where a Stripe retry caused a duplicate charge or double-shipment

To make a webhook consumer idempotent, store the provider's event ID (e.g. Stripe's evt_...) in a processed_events table with a unique constraint, and insert that row inside the same database transaction as your business side effects. If the insert fails on the unique constraint, roll back and return 200. That's it. The rest of this post is the details that keep this from breaking under concurrent retries — and the mistake most teams make (hashing the payload instead of using the event ID) that silently corrupts data.

This tutorial assumes you just had an incident: a Stripe retry, an SQS redelivery, or a Shopify webhook double-fire caused a duplicate charge or a double-shipment. Your handler is correct for a single delivery. It has no deduplication layer. Let's fix that.

Prerequisites

The core insight before we start

Two things trip up most implementations:

  1. Deduplicate on the provider's event ID, not a payload hash. Two legitimately distinct events can carry identical payloads — a customer buying the same $10 SKU twice in 30 seconds generates two charge.succeeded events with nearly identical bodies but different evt_ IDs. Hash the payload and you'll silently drop the second charge. Use the event ID and you won't.
  2. The idempotency check must live inside the same transaction as the side effect. If you check SELECT ... WHERE event_id = ?, then do the work, then INSERT, you have a race window. Two concurrent workers picking up the same retried event will both see "not processed", both do the work, both try to insert. Use a unique constraint and let the database enforce the invariant atomically.

Step 1: Create the processed_events table

CREATE TABLE processed_events (
  event_id      TEXT PRIMARY KEY,
  provider      TEXT NOT NULL,
  event_type    TEXT NOT NULL,
  processed_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  result        JSONB
);

CREATE INDEX idx_processed_events_processed_at
  ON processed_events (processed_at);

Notes on the schema:

Run it:

psql $DATABASE_URL -f migrations/001_processed_events.sql

Expected output: CREATE TABLE and CREATE INDEX.

Step 2: Wrap the handler in a transaction with the dedup insert first

Here's the shape. Node + pg, but the SQL is what matters.

import { Pool } from 'pg';
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

async function handleStripeWebhook(event) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // 1. Attempt to claim the event. Fails if already processed.
    const claim = await client.query(
      `INSERT INTO processed_events (event_id, provider, event_type)
       VALUES ($1, $2, $3)
       ON CONFLICT (event_id) DO NOTHING
       RETURNING event_id`,
      [event.id, 'stripe', event.type]
    );

    if (claim.rowCount === 0) {
      // Already processed. Return the stored result if you cached it.
      await client.query('ROLLBACK');
      return { status: 'duplicate', event_id: event.id };
    }

    // 2. Do the actual work inside the same transaction.
    const result = await processEvent(client, event);

    // 3. Optionally persist the result for future retries.
    await client.query(
      `UPDATE processed_events SET result = $1 WHERE event_id = $2`,
      [result, event.id]
    );

    await client.query('COMMIT');
    return { status: 'processed', ...result };
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

Two things worth calling out:

Step 3: Make processEvent do its work through the transaction client

This is the step teams get wrong. If your business logic calls pool.query directly (a new connection) instead of using the passed client, its writes are not part of the transaction and will commit independently.

async function processEvent(client, event) {
  if (event.type === 'charge.succeeded') {
    const charge = event.data.object;
    await client.query(
      `INSERT INTO payments (stripe_charge_id, amount, customer_id, status)
       VALUES ($1, $2, $3, 'succeeded')
       ON CONFLICT (stripe_charge_id) DO NOTHING`,
      [charge.id, charge.amount, charge.customer]
    );
    return { charge_id: charge.id };
  }
  // ... other event types
}

Belt and suspenders: notice the second ON CONFLICT DO NOTHING on stripe_charge_id. Even if the outer idempotency layer fails, the database still refuses to double-insert a payment. Always add unique constraints on the natural business keys — Stripe charge IDs, order numbers, external transaction IDs.

Step 4: Handle non-DB side effects (the hard part)

Databases are easy. The tricky side effects are the ones you can't roll back:

Rule: never do a non-transactional side effect inside the webhook transaction. Instead, enqueue the intent inside the transaction, and let a worker do the external call.

async function processEvent(client, event) {
  if (event.type === 'charge.succeeded') {
    const charge = event.data.object;

    await client.query(
      `INSERT INTO payments (stripe_charge_id, amount, status)
       VALUES ($1, $2, 'succeeded')
       ON CONFLICT (stripe_charge_id) DO NOTHING`,
      [charge.id, charge.amount]
    );

    // Enqueue the email — same transaction, same rollback semantics.
    await client.query(
      `INSERT INTO outbox (topic, payload) VALUES ($1, $2)`,
      ['send_receipt', { charge_id: charge.id }]
    );

    return { charge_id: charge.id };
  }
}

This is the transactional outbox pattern. Your worker reads from outbox, sends the email, deletes the row. Because the outbox insert commits atomically with the payment, you either have both or neither — no ghost emails for rolled-back events.

The worker itself needs to be idempotent too, usually via a provider-side idempotency key (SendGrid, Postmark, Stripe API, Twilio all accept one). Use outbox.id as that key.

Step 5: Return the right HTTP status codes

Providers retry on non-2xx. Get this wrong and you'll cause the exact retries you're trying to survive.

app.post('/webhooks/stripe', async (req, res) => {
  let event;
  try {
    event = stripe.webhooks.constructEvent(
      req.rawBody, req.headers['stripe-signature'], process.env.STRIPE_WEBHOOK_SECRET
    );
  } catch (err) {
    // Bad signature — do NOT retry. 400.
    return res.status(400).send('invalid signature');
  }

  try {
    const result = await handleStripeWebhook(event);
    return res.status(200).json(result);
  } catch (err) {
    console.error('webhook processing failed', { event_id: event.id, err });
    // Transient failure — DO retry. 500.
    return res.status(500).send('processing failed');
  }
});

Rules of thumb:

Step 6: Test it under concurrent load

Unit tests won't catch the race. You need to actually hammer the endpoint with the same event ID in parallel.

# Save a real event payload from Stripe CLI first:
# stripe listen --print-json > sample.json

# Fire the same event 20 times in parallel:
seq 1 20 | xargs -n1 -P20 -I{} curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Stripe-Signature: $SIG" \
  --data-binary @sample.json \
  http://localhost:3000/webhooks/stripe

Then check the database:

SELECT COUNT(*) FROM payments WHERE stripe_charge_id = 'ch_test_123';
-- Expected: 1

SELECT COUNT(*) FROM processed_events WHERE event_id = 'evt_test_123';
-- Expected: 1

SELECT COUNT(*) FROM outbox WHERE payload->>'charge_id' = 'ch_test_123';
-- Expected: 1

If any of these come back > 1, your transaction boundary is wrong. The most common cause: a side effect using a different DB connection than the one holding the transaction.

Step 7: Add a retention job

The processed_events table will grow forever. You only need to retain event IDs for as long as the provider might retry. Stripe retries webhooks for up to 3 days by default; check your providers.

-- Run daily via cron / pg_cron / a scheduled job
DELETE FROM processed_events
WHERE processed_at < NOW() - INTERVAL '30 days';

30 days gives you comfortable margin over any mainstream provider's retry window plus room to debug. Don't be aggressive here — the storage cost is trivial compared to the cost of a duplicate charge.

Common errors and how to diagnose them

"I still see duplicate rows in payments"

Your side effect is running on a different connection than the transaction. Grep your handler for anything that calls pool.query, knex(), or your ORM's default connection — all of these bypass the transaction. Every DB call in processEvent must go through the passed client.

"Duplicates only appear under load"

You're checking-then-inserting instead of using ON CONFLICT. Two workers both see "not processed", both do the work, one insert wins and one fails — but both side effects already happened. Fix: attempt the insert first, use its result as the claim.

"Stripe keeps retrying even though I return 200"

You're returning 200 after the response body has already errored, or your load balancer is timing out before your handler completes. Check that the handler finishes in under 10 seconds (Stripe's timeout). If your work is slow, respond 200 immediately after claiming the event and enqueue the actual processing to a worker — the claim in processed_events is enough to guarantee at-most-once.

"ON CONFLICT DO NOTHING is not returning rows I expect"

Remember: RETURNING only returns rows that were actually inserted. On conflict, rowCount is 0 and rows is empty. That's the signal for "duplicate", not an error.

"My tests pass but production still double-processes"

You probably have two webhook consumers running — a leftover staging endpoint, a canary deployment, or a queue subscriber and an HTTP handler both consuming the same event. Idempotency at the DB layer will catch this only if they share a database. Audit your provider dashboard for the list of registered endpoints.

What this approach is bad at

Honest tradeoffs:

Teams building payment, logistics, or lending flows tend to learn these tradeoffs the hard way. If you're modernizing a payments or fulfillment pipeline and want to see how these patterns play out in production, the GimBooks accounting and Cashpo lending case studies walk through similar transactional guarantees at scale.

Frequently Asked Questions

Should I use the provider's Idempotency-Key header or the event ID?

Different things. The Idempotency-Key header is for your outbound requests to the provider (e.g. creating a Stripe charge). The event ID is for inbound webhooks the provider sends you. For deduplicating webhook consumers, always use the event ID from the payload (event.id in Stripe, X-Shopify-Webhook-Id in Shopify).

What if my webhook provider doesn't send a stable event ID?

Rare, but it happens with older systems. Fall back to a composite key: (event_type, external_resource_id, event_timestamp). Avoid hashing the whole payload — you'll drop legitimate distinct-but-identical events. If the provider offers a signed timestamp, include it to disambiguate.

Can I do this with a Redis SETNX instead of a database table?

You can, but you lose the atomic-with-side-effect guarantee unless your side effects are also in Redis. The whole point of putting the dedup row in Postgres is that it commits or rolls back with the payment insert. If you use Redis for dedup and Postgres for the payment, a crash between them leaves you inconsistent. Redis is fine as a fast-path filter before the DB check, not as the source of truth.

How do I handle events that arrive out of order?

Idempotency and ordering are separate problems. Store the event's timestamp and version in your domain model, and reject writes that would move state backwards (e.g. don't apply charge.succeeded if you've already recorded charge.refunded). Most providers include a created timestamp; use it.

Do I need this if my webhook volume is low?

Yes. Duplicate processing is a correctness bug, not a scale bug. A single retry on a single duplicate charge is one angry customer and a chargeback. The code above adds one insert per webhook — it's not a performance concern at any volume that's realistic for a webhook consumer.

Keep reading

SaaS

Your Integration Is Not Done When the Data Flows

Passing QA is not the same as being production-ready. The real failure surface of an integration isn't the happy path — it's what happens when a record breaks, and who owns it when it does.

 SaaS

Feature Flags Are Not Config. Treating Them That Way Breaks You.

A feature flag is a decision with an owner and a death date. A config value is state. When you store them in the same table, you lose the guardrails that make flags safe — and production pays the bill.

 SaaS

Rate-Limit a Multi-Tenant API Without Punishing Good Tenants

One free-tier tenant shouldn't be able to spike your p99 and take down enterprise customers sharing the same limiter. Here's a runnable per-tenant sliding window in Redis, with the sorted-set logic that makes boundary bursts impossible.

Found this useful? CodeNicely publishes engineering and product playbooks weekly. Browse the archive or tell us what you're building.