Idempotent Webhook Consumers: A Step-by-Step Guide

For: A backend engineer at a Series A SaaS company whose payment or subscription webhook handler is silently double-processing events — charging customers twice, double-crediting accounts, or firing duplicate emails — because their provider (Stripe, Paddle, or similar) guarantees at-least-once delivery and their handler was written assuming exactly-once

To make a webhook consumer safely idempotent, store the provider's event ID and commit your side effect in the same database transaction, using a UNIQUE constraint to reject duplicates. Checking for the ID before the transaction and inserting it after leaves a race window where two concurrent retries both pass the check and both write. That single detail is what separates a handler that looks correct in staging from one that survives Stripe's retry behavior in production.

This tutorial walks through a working implementation in Node.js + Postgres. The pattern translates directly to Python, Go, or Rails — the database mechanics are what matter.

Why at-least-once delivery breaks naive handlers

Stripe, Paddle, Shopify, GitHub, and most modern providers guarantee at-least-once delivery. If your endpoint returns anything other than a 2xx within their timeout (Stripe: 30s), or if the acknowledgement packet is lost, they retry. Retries can arrive seconds or hours later. Two retries can arrive within milliseconds of each other if your server was briefly slow.

A handler like this is the common failure mode:

app.post('/webhooks/stripe', async (req, res) => {
  const event = verifyStripeSignature(req);
  if (event.type === 'invoice.payment_succeeded') {
    await creditAccount(event.data.object.customer, event.data.object.amount);
    await sendReceiptEmail(event.data.object.customer);
  }
  res.sendStatus(200);
});

Nothing prevents creditAccount from running twice. Nothing prevents two receipt emails. And logs will show two clean 200 responses for two apparently distinct requests — because Stripe's retry uses the same event ID but a different HTTP request ID.

Prerequisites

• Node.js 18+ and npm
• Postgres 13+ running locally (or Docker: docker run -p 5432:5432 -e POSTGRES_PASSWORD=dev postgres:15)
• psql installed
• A Stripe test account (optional — we'll simulate deliveries with curl)

Install dependencies:

npm init -y
npm install express pg stripe

Step 1: Create the idempotency table

The table is the entire foundation. It needs three things: a unique constraint on the event ID, a status column, and a place to cache the response.

CREATE TABLE processed_webhooks (
  event_id       TEXT PRIMARY KEY,
  provider       TEXT NOT NULL,
  event_type     TEXT NOT NULL,
  status         TEXT NOT NULL CHECK (status IN ('processing','completed','failed')),
  response_body  JSONB,
  received_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  completed_at   TIMESTAMPTZ
);

CREATE INDEX idx_processed_webhooks_received_at ON processed_webhooks(received_at);

Run it:

psql -U postgres -d webhooks_demo -f schema.sql

Expected output:

CREATE TABLE
CREATE INDEX

The PRIMARY KEY on event_id is what actually enforces idempotency. Everything else is bookkeeping.

Step 2: Verify the signature before anything else

Idempotency is meaningless if attackers can send arbitrary events. Verify the provider signature before you touch the database.

const stripe = require('stripe')(process.env.STRIPE_SECRET);

function verifyEvent(req) {
  const sig = req.headers['stripe-signature'];
  return stripe.webhooks.constructEvent(
    req.rawBody,
    sig,
    process.env.STRIPE_WEBHOOK_SECRET
  );
}

Note: you need the raw request body, not the parsed JSON. Configure Express with express.raw({ type: 'application/json' }) on this route only.

Step 3: The transactional handler (the important part)

This is where most implementations quietly get it wrong. The insert of the idempotency row and the side effect must live in one transaction. If the side effect fails, the row rolls back. If a duplicate arrives, the INSERT fails with a unique-violation and we return the cached response.

const { Pool } = require('pg');
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

async function handleWebhook(event) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // Attempt to claim this event. If it already exists, this throws.
    try {
      await client.query(
        `INSERT INTO processed_webhooks
           (event_id, provider, event_type, status)
         VALUES ($1, $2, $3, 'processing')`,
        [event.id, 'stripe', event.type]
      );
    } catch (err) {
      if (err.code === '23505') { // unique_violation
        await client.query('ROLLBACK');
        return await getCachedResponse(event.id);
      }
      throw err;
    }

    // Do the side effect INSIDE the transaction.
    const result = await applyBusinessLogic(client, event);

    await client.query(
      `UPDATE processed_webhooks
         SET status = 'completed',
             response_body = $2,
             completed_at = now()
       WHERE event_id = $1`,
      [event.id, result]
    );

    await client.query('COMMIT');
    return result;
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

Two things to notice:

• applyBusinessLogic receives the same client — the account credit, the ledger entry, whatever it does, all happen in this transaction. If any of it throws, the idempotency row is rolled back too, so a retry will re-process the event. That is what you want.
• Duplicates are detected by letting Postgres reject the insert. There is no read-then-write window. Two concurrent retries race for the insert; exactly one wins, the other gets 23505.

Step 4: Handle side effects that can't be transactional

Sending an email is not a database write. You cannot roll it back. Two patterns work:

Option A: Outbox pattern (recommended)

Instead of calling the email service directly, insert a row into an outbox table inside the same transaction. A separate worker reads the outbox and dispatches emails, marking rows sent. The email send itself needs its own idempotency key (most providers — Postmark, SendGrid, Resend — accept one).

async function applyBusinessLogic(client, event) {
  const invoice = event.data.object;
  await client.query(
    `UPDATE accounts SET balance = balance + $1 WHERE customer_id = $2`,
    [invoice.amount, invoice.customer]
  );
  await client.query(
    `INSERT INTO outbox (kind, payload, idempotency_key)
     VALUES ('receipt_email', $1, $2)`,
    [invoice, `receipt:${event.id}`]
  );
  return { credited: invoice.amount };
}

Option B: Post-commit dispatch with provider-side idempotency

Commit the transaction first, then call the email/SMS/Slack API using event.id as the idempotency key the provider accepts. If your process crashes between commit and dispatch, a Stripe retry will hit the duplicate-detection path and never re-dispatch. That's a real gap — Option A closes it.

Step 5: Return cached responses for duplicates

When the unique violation fires, you still need to return a 2xx so the provider stops retrying.

async function getCachedResponse(eventId) {
  const { rows } = await pool.query(
    `SELECT status, response_body FROM processed_webhooks WHERE event_id = $1`,
    [eventId]
  );
  if (rows.length === 0) return { status: 'unknown' };
  return rows[0];
}

Edge case: if the first request is still processing when the retry arrives (long-running side effect, provider retried before you committed), the duplicate insert also fails. You should return a 409 or a 5xx so the provider retries later — by which time the first request will have committed or rolled back.

app.post('/webhooks/stripe', async (req, res) => {
  let event;
  try { event = verifyEvent(req); }
  catch { return res.sendStatus(400); }

  try {
    const result = await handleWebhook(event);
    if (result.status === 'processing') return res.sendStatus(409);
    res.status(200).json(result);
  } catch (err) {
    console.error(err);
    res.sendStatus(500); // provider will retry
  }
});

Step 6: Test the race condition

Prove the fix works. Fire two concurrent requests with the same event ID:

EVENT='{"id":"evt_test_123","type":"invoice.payment_succeeded","data":{"object":{"customer":"cus_1","amount":5000}}}'

curl -X POST http://localhost:3000/webhooks/stripe -H 'Content-Type: application/json' -d "$EVENT" &
curl -X POST http://localhost:3000/webhooks/stripe -H 'Content-Type: application/json' -d "$EVENT" &
wait

Then check the account:

psql -c "SELECT customer_id, balance FROM accounts WHERE customer_id='cus_1';"

Expected output:

 customer_id | balance
-------------+---------
 cus_1       |    5000

Balance is 5000, not 10000. One request committed; the other hit 23505 and returned the cached response.

Step 7: Add observability

You want to know when duplicates arrive — it's a signal about your infrastructure, not just a defensive measure. High duplicate rates usually mean your handler is slow enough to hit provider timeouts, or your load balancer is retrying.

-- daily duplicate rate
SELECT
  date_trunc('day', received_at) AS day,
  count(*) FILTER (WHERE status = 'completed') AS processed,
  count(*) FILTER (WHERE status = 'failed') AS failed
FROM processed_webhooks
GROUP BY 1 ORDER BY 1 DESC;

For the duplicate count itself, log every 23505 hit with the event ID and emit a metric. If duplicates spike, investigate handler latency first.

Step 8: Retention

The table grows forever otherwise. Providers usually don't retry after a fixed window (Stripe: up to 3 days for most events, longer for some). Keep 30 days for safety and archive the rest.

DELETE FROM processed_webhooks WHERE received_at < now() - interval '30 days';

Run it nightly with pg_cron or a scheduled job.

Common errors and how to debug them

Duplicate side effects still happening

Nine times out of ten, the side effect is not actually inside the transaction. Check that every await client.query in your business logic uses the same client passed in — not a fresh pool.query. A quick way to catch this: wrap the pool so pool.query throws inside handler code during tests.

Deadlocks under load

If your side effect updates rows that other transactions also touch (e.g., a shared counter), you'll see 40P01 deadlock errors. Fix by ordering row locks consistently, or by using SELECT ... FOR UPDATE on a canonical row (like the customer row) at the top of the transaction.

Handler timing out on slow side effects

Stripe times out at 30 seconds. If your business logic takes longer, acknowledge the webhook fast and process asynchronously: insert the raw event into a jobs table inside the transaction, return 200, and have a worker do the real work with its own idempotency check on the job row.

Same event, different signature

Occasionally you'll see the same event ID with a slightly different payload — usually because the provider re-serialized. Trust the event ID, not the payload hash. If your business requires content-level dedup, hash the semantic fields (amount + customer + type), not the raw body.

Testing shows no duplicates because your client library retries silently

Some HTTP clients retry on connection resets before your app sees anything. Fire duplicates with raw curl or two separate processes, as in Step 6, not with a client library.

What this pattern is bad at

Honest tradeoffs:

• Latency floor. Every webhook now does at least two writes (insert, update) plus your business logic in one transaction. If you're processing tens of thousands of events per minute, the row-level contention on hot customer rows will hurt. Sharding by customer or moving to an outbox-first design helps.
• Failed events retry forever. If your business logic has a permanent bug for a specific event, the provider will retry until it gives up. Add a failed terminal state after N attempts and alert on it.
• Cross-provider dedup. If two providers can generate logically-equivalent events (e.g., Stripe + a manual admin action), event-ID dedup won't catch it. You need a domain-level idempotency key on the underlying operation.

For teams building payment or subscription infrastructure, this pattern is table stakes. We've applied variants of it across fintech projects like GimBooks and lending flows like Cashpo where a duplicated ledger write is a customer-support incident, not a bug ticket.

Frequently Asked Questions

Can I use Redis instead of Postgres for the idempotency store?

Yes, but you lose the ability to co-commit the idempotency record with the side effect. That reintroduces the race window this whole pattern exists to close. Redis works if your side effect is itself in Redis, or if you accept a small duplicate rate and rely on downstream idempotency (e.g., provider-side keys on the email API). For payments, use Postgres.

What's the difference between the event ID and an idempotency key?

The event ID is issued by the webhook provider (Stripe's evt_...) and identifies a delivery attempt's logical event. An idempotency key is one you generate and send when calling an outbound API to make that call safe to retry. In a webhook consumer you use the provider's event ID as your dedup key; when your consumer then calls another API, you generate a fresh idempotency key for that outbound call.

How do I handle events that arrive out of order?

Idempotency and ordering are separate problems. Providers don't guarantee order — a subscription.updated can arrive before subscription.created. Store the event's timestamp and compare against the current state before applying: if the event is older than the last applied state, record it as processed but skip the side effect. Stripe includes created on every event for exactly this.

Should the webhook endpoint be behind authentication?

Signature verification is the authentication. Don't put webhook endpoints behind API keys or session auth — the provider can't send those. Do rate-limit by source IP if your provider publishes their IP ranges (Stripe does), and reject any request missing a valid signature before you spend a database round-trip on it.

Does this pattern work for GitHub, Shopify, and Slack webhooks too?

Yes. GitHub sends X-GitHub-Delivery, Shopify sends X-Shopify-Webhook-Id, Slack sends an event_id in the payload. Any provider with at-least-once delivery gives you some stable identifier — use that as your primary key. The transactional pattern is provider-agnostic; only the signature verification changes.