Idempotent Webhooks: Handle Retries Without Duplicate Orders

For: A backend engineer at a seed-to-Series-A e-commerce or fintech startup who has just found duplicate orders in production after their payment provider retried a webhook their server returned 500 on

If your payment provider retried a webhook and you woke up to duplicate orders, the fix is not a Redis SETNX on the event ID. The fix is to make the database write that creates the order and the row that marks the event as processed happen inside the same transaction, and only return 200 after that transaction commits. Any gap between "I did the work" and "I told Stripe I did the work" is the exact millisecond window a retry will land in. This tutorial walks through building that handler in Node.js and Postgres, end to end, with the failure modes you'll hit in production.

Why your current handler is broken

Here's the pattern that ships to production and breaks a week later:

app.post('/webhooks/stripe', async (req, res) => {
  const event = stripe.webhooks.constructEvent(req.body, sig, secret);
  if (event.type === 'checkout.session.completed') {
    await createOrder(event.data.object); // writes to DB
    await sendFulfillmentEmail(...);      // calls SendGrid
  }
  res.status(200).send('ok');
});

Three things can go wrong, and all of them cause Stripe to retry:

• createOrder succeeds, sendFulfillmentEmail throws, your process crashes before returning 200. Stripe retries. You get a second order.
• The DB write commits, but the load balancer kills the connection before the 200 reaches Stripe. Stripe retries.
• Two Stripe deliveries arrive in parallel (yes, this happens on redrives). Both handlers race, both create an order.

Deduplicating on event.id in application memory or Redis with a TTL is a band-aid. Redis and Postgres can disagree. The only source of truth that matters is the same database that holds your orders.

Prerequisites

• Node.js 18+ and a Postgres 13+ instance (local is fine — docker run -e POSTGRES_PASSWORD=pw -p 5432:5432 postgres:15)
• A Stripe account with the CLI installed (brew install stripe/stripe-cli/stripe)
• Basic familiarity with Express and pg

Install deps:

npm init -y
npm i express stripe pg

Step 1: Create the processed_events table

This is the core of the pattern. One row per webhook event you've handled. It lives in the same database as your orders so it can share a transaction.

CREATE TABLE processed_events (
  event_id      TEXT PRIMARY KEY,
  event_type    TEXT NOT NULL,
  processed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  result        JSONB
);

CREATE TABLE orders (
  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  stripe_session_id TEXT UNIQUE NOT NULL,
  amount_cents  INTEGER NOT NULL,
  status        TEXT NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

Two idempotency guards, not one:

1. processed_events.event_id is the primary key. Trying to insert the same Stripe event ID twice will fail with a unique violation.
2. orders.stripe_session_id has a UNIQUE constraint. Even if you had a bug and skipped the event log, you still couldn't write two orders for the same checkout session.

Belt and suspenders. In fintech, you want both.

Step 2: Write the handler as a single transaction

The whole handler — insert into processed_events, insert into orders, any ledger writes — lives inside BEGIN / COMMIT.

// handler.js
const { Pool } = require('pg');
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

async function handleCheckoutCompleted(event) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // 1. Claim the event. If another worker already claimed it, this throws.
    try {
      await client.query(
        'INSERT INTO processed_events (event_id, event_type) VALUES ($1, $2)',
        [event.id, event.type]
      );
    } catch (err) {
      if (err.code === '23505') { // unique_violation
        await client.query('ROLLBACK');
        return { status: 'already_processed' };
      }
      throw err;
    }

    // 2. Do the actual work.
    const session = event.data.object;
    await client.query(
      `INSERT INTO orders (stripe_session_id, amount_cents, status)
       VALUES ($1, $2, 'paid')
       ON CONFLICT (stripe_session_id) DO NOTHING`,
      [session.id, session.amount_total]
    );

    await client.query('COMMIT');
    return { status: 'processed' };
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

module.exports = { handleCheckoutCompleted };

Three things to notice:

• The INSERT into processed_events is the claim. If it fails with 23505, another delivery of the same event already ran. We roll back and return quickly.
• The order write is inside the same transaction. If anything after the claim throws, we roll back — including the claim — so the retry can legitimately re-run.
• No email sending, no Slack notification, no third-party API calls inside the transaction. Those come later, outside the DB. More on this in Step 5.

Step 3: Wire it into Express

// server.js
const express = require('express');
const Stripe = require('stripe');
const { handleCheckoutCompleted } = require('./handler');

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);
const app = express();

app.post('/webhooks/stripe',
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    const sig = req.headers['stripe-signature'];
    let event;
    try {
      event = stripe.webhooks.constructEvent(
        req.body, sig, process.env.STRIPE_WEBHOOK_SECRET
      );
    } catch (err) {
      return res.status(400).send(`Webhook Error: ${err.message}`);
    }

    try {
      if (event.type === 'checkout.session.completed') {
        await handleCheckoutCompleted(event);
      }
      // COMMIT has succeeded before we get here.
      res.status(200).json({ received: true });
    } catch (err) {
      console.error('handler failed', event.id, err);
      // 500 tells Stripe to retry. That's what we want.
      res.status(500).send('handler error');
    }
  }
);

app.listen(3000, () => console.log('listening on 3000'));

Signature verification comes first — never process an unsigned webhook. Then the handler runs, and only after COMMIT returns does the 200 go out.

Step 4: Test it locally with the Stripe CLI

Forward events to your local server:

stripe listen --forward-to localhost:3000/webhooks/stripe

In another terminal, fire a test event:

stripe trigger checkout.session.completed

Expected output on your server logs:

POST /webhooks/stripe 200 45ms

Check the database:

psql $DATABASE_URL -c 'SELECT event_id, event_type FROM processed_events;'
psql $DATABASE_URL -c 'SELECT id, stripe_session_id, status FROM orders;'

You should see exactly one row in each.

Now simulate a retry. Copy the event ID from your logs and use stripe events resend:

stripe events resend evt_1Nxxxxxxxxxxxx

Expected behavior: server returns 200, but no new row appears in orders. The handler hit the unique violation on processed_events, rolled back, returned already_processed, and Express replied 200.

Step 5: Handle side effects outside the transaction

Fulfillment emails, Slack alerts, shipping API calls — none of these belong inside the DB transaction. They're slow, they fail independently, and you can't roll them back.

The right pattern: write an outbox row inside the same transaction, then a separate worker drains it.

CREATE TABLE outbox (
  id           BIGSERIAL PRIMARY KEY,
  event_id     TEXT NOT NULL REFERENCES processed_events(event_id),
  task_type    TEXT NOT NULL,
  payload      JSONB NOT NULL,
  sent_at      TIMESTAMPTZ
);

Inside your transaction, after inserting the order:

await client.query(
  `INSERT INTO outbox (event_id, task_type, payload)
   VALUES ($1, 'send_fulfillment_email', $2)`,
  [event.id, JSON.stringify({ session_id: session.id, email: session.customer_email })]
);

A separate worker polls outbox WHERE sent_at IS NULL, sends the email, and marks it sent. Emails become idempotent at the worker level (SendGrid supports its own idempotency keys). If the worker crashes mid-send, it retries — worst case, one duplicate email, never a duplicate order.

This is the transactional outbox pattern. It's the reason your ledger never disagrees with what you told the customer.

Step 6: Handle concurrent deliveries with row locking

Stripe occasionally delivers the same event twice in parallel — especially after a redrive or during a region failover. Both handlers reach INSERT INTO processed_events at the same instant. Postgres serializes them: one wins, one gets 23505. This is fine.

But if you're doing more complex work — say, updating a running balance on a wallet row — you need an explicit lock so the two transactions don't interleave. Use SELECT ... FOR UPDATE:

await client.query('SELECT balance FROM wallets WHERE user_id = $1 FOR UPDATE', [userId]);
// now safely compute and update balance

The row lock is held until COMMIT. The second concurrent transaction blocks, then when it wakes up, its claim insert fails with 23505 and it exits cleanly.

Step 7: Add observability before you deploy

You will never debug a duplicate-order incident without logs of every event ID you saw and what you did with it. Bare minimum:

console.log(JSON.stringify({
  event_id: event.id,
  event_type: event.type,
  outcome: result.status, // 'processed' or 'already_processed'
  duration_ms: Date.now() - start
}));

Send these to your log aggregator with the event ID as an indexed field. When a customer says "I got charged twice," you search by stripe_session_id, find every webhook delivery for that session, and prove exactly what happened.

Also add a metric for already_processed count. If it spikes, Stripe is retrying more than usual — usually a sign that your p99 latency crossed their timeout (Stripe times out webhooks at 30 seconds; keep handlers well under a second).

Common errors

"duplicate key value violates unique constraint processed_events_pkey"

This is the expected outcome for a retry. Catch err.code === '23505', roll back, return 200. If you're seeing this in error logs at high volume, you probably forgot to catch it and are returning 500 — which causes Stripe to retry again, forever.

Orders exist but processed_events is empty

You wrote the order outside the transaction, or you're using two different database connections. Both writes must go through the same client from pool.connect(), not through pool.query() directly.

Handler times out, Stripe retries, second delivery also times out

You're doing slow work inside the transaction — most commonly, calling a third-party API. Move it to the outbox worker. The webhook handler should do nothing except claim the event, write to your DB, and return.

Emails sent twice even though orders are correct

Your outbox worker isn't marking rows sent atomically with the send. Use UPDATE outbox SET sent_at = now() WHERE id = $1 AND sent_at IS NULL RETURNING id before sending, and skip if no row returned. Or use the email provider's idempotency key feature.

Signature verification fails intermittently

You're using express.json() before the webhook route, which mutates the raw body. Webhook routes need express.raw({ type: 'application/json' }) and must be mounted before any global JSON parser.

What this pattern is bad at

Honest tradeoffs:

• It couples your webhook throughput to Postgres write throughput. If you're processing thousands of events per second, the processed_events table becomes a hot spot. Partition by month, or move to a dedicated events database.
• The outbox adds latency. A polling worker adds seconds of delay before the customer's email goes out. For most fintech workflows this is fine — settlement is not real-time — but if you need sub-second fulfillment, use LISTEN/NOTIFY on the outbox insert.
• It doesn't help if the event payload changes between retries. Stripe won't do that, but some providers (looking at you, older PayPal APIs) will send a fresh payload on retry. In that case, always trust the latest payload but keep the idempotency key on your business identifier, not the event ID.

Applies beyond Stripe

The same pattern works for Razorpay, Adyen, PayPal, Shopify, GitHub, Slack — anything that sends webhooks with an ID and retries on non-2xx. Teams building lending platforms and accounting SaaS deal with this constantly; see how it plays out in production systems like Cashpo's lending stack or GimBooks' accounting flows, where a duplicated ledger entry isn't a UX bug — it's a compliance incident. If you're building the payments or ledger layer of a fintech product from scratch, the architectural choices around idempotency and outbox are worth getting right before you have volume, not after.

Frequently Asked Questions

Should I use Redis instead of Postgres for idempotency keys?

Only as a fast pre-check, never as the source of truth. Redis and your orders database can diverge — Redis loses a key, and now a retry creates a duplicate order. The unique constraint on processed_events in the same Postgres instance as your orders is the only guarantee that survives every failure mode.

What HTTP status should I return if I've already processed the event?

Return 200. From the payment provider's perspective, the event has been handled — that's what 2xx means. Returning 409 or 200 both stop retries in practice, but 200 is standard and won't trigger alerts on the provider's dashboard.

How long should I keep rows in processed_events?

At least as long as the payment provider will retry. Stripe retries for up to 3 days on failed deliveries. Keep 30-90 days for debugging, then archive to cold storage. Never delete without archiving — you'll want the audit trail during a customer dispute.

Does this work for webhooks that don't send an event ID?

Yes, but you have to synthesize one. Hash the raw payload plus a business identifier (order ID, transaction reference) and use that as your idempotency key. It's slightly weaker than a provider-generated ID because two logically distinct events with identical payloads collapse, but for most webhook shapes this is safe.

Can you help us audit our current webhook infrastructure?

Yes — for a review of your webhook, ledger, and reconciliation architecture, contact CodeNicely for a personalized assessment. We work with fintech and e-commerce teams on payment infrastructure, idempotency guarantees, and the outbox patterns discussed here.