Idempotent Webhook Handlers: Stop Processing Events Twice

For: A backend engineer at a seed-to-Series-A fintech startup who wired up Stripe or Razorpay webhooks in a weekend sprint and is now seeing duplicate charges, double-credited wallets, or duplicate order records in production whenever the payment provider retries a delivery

If your Stripe or Razorpay webhook is double-crediting wallets or creating duplicate orders on retries, the fix is not an in-memory Set of event IDs or a Redis SETNX check before your handler runs. It is to store the event ID in the same database transaction that applies the business logic, with a UNIQUE constraint that forces a rollback if the event has already been processed. Anything less loses to a race between two concurrent retries.

This post walks through building an idempotent webhook handler for a fintech wallet, using Postgres and Node.js. The same pattern works in Python, Go, or Java — the guarantees come from the database, not the language.

Why your current handler is broken

Here is the shape of the bug. Stripe sends payment_intent.succeeded. Your server accepts it, starts crediting the user's wallet, but takes 12 seconds because a downstream call is slow. Stripe times out at 10 seconds, marks delivery as failed, and retries. Now two workers are running the same handler concurrently. Both read wallet.balance = 500, both write wallet.balance = 500 + 1000 = 1500. The user has been credited twice for a single payment.

Common "fixes" that do not actually work:

• Checking a Redis key before processing. Two retries arrive within milliseconds. Both check Redis, both see "not processed", both proceed. Redis SETNX before the DB write does not bind the check to the write.
• An in-memory Set of event IDs. Dies with the process. Useless behind a load balancer.
• Checking SELECT * FROM processed_events WHERE id = $1 at the top of the handler. Classic time-of-check-to-time-of-use race. Two transactions can both see "not present" and both insert.

The correct approach: insert the event ID into a processed_events table with a UNIQUE constraint, inside the same transaction as the wallet credit. If the insert fails with a unique-violation, the whole transaction rolls back and you return 200 OK. The database is the arbiter — no application-level check can beat two concurrent transactions.

Prerequisites

• Node.js 18+ and npm
• Postgres 13+ running locally (or Docker: docker run -e POSTGRES_PASSWORD=pw -p 5432:5432 postgres:15)
• Stripe CLI installed (brew install stripe/stripe-cli/stripe or equivalent)
• A Stripe test-mode account and webhook signing secret

Step 1: Set up the schema

Create two tables. One for wallets, one for the idempotency ledger.

-- schema.sql
CREATE TABLE wallets (
  user_id       TEXT PRIMARY KEY,
  balance_cents BIGINT NOT NULL DEFAULT 0,
  updated_at    TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE processed_events (
  event_id      TEXT PRIMARY KEY,
  event_type    TEXT NOT NULL,
  processed_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  result        JSONB
);

INSERT INTO wallets (user_id, balance_cents) VALUES ('user_123', 0);

Apply it:

psql postgres://postgres:pw@localhost:5432/postgres -f schema.sql

Expected output:

CREATE TABLE
CREATE TABLE
INSERT 0 1

The PRIMARY KEY on processed_events.event_id is the only durable guard we need. It is enforced by Postgres at commit time. No application code can defeat it.

Step 2: The naive (broken) handler, for reference

Before writing the correct version, look at what most teams ship first:

// broken.js — do NOT use
app.post('/webhook', async (req, res) => {
  const event = req.body;
  const exists = await db.query(
    'SELECT 1 FROM processed_events WHERE event_id = $1',
    [event.id]
  );
  if (exists.rows.length) return res.sendStatus(200);

  await db.query(
    'UPDATE wallets SET balance_cents = balance_cents + $1 WHERE user_id = $2',
    [event.data.object.amount, event.data.object.metadata.user_id]
  );
  await db.query(
    'INSERT INTO processed_events (event_id, event_type) VALUES ($1, $2)',
    [event.id, event.type]
  );
  res.sendStatus(200);
});

Two simultaneous retries will both pass the SELECT check, both run the UPDATE, and both insert (or one inserts and one 500s after the money has already been credited twice). This is the bug.

Step 3: The correct handler

// handler.js
const express = require('express');
const { Pool } = require('pg');
const Stripe = require('stripe');

const app = express();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const stripe = Stripe(process.env.STRIPE_SECRET_KEY);
const WEBHOOK_SECRET = process.env.STRIPE_WEBHOOK_SECRET;

app.post('/webhook',
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    let event;
    try {
      event = stripe.webhooks.constructEvent(
        req.body, req.headers['stripe-signature'], WEBHOOK_SECRET
      );
    } catch (err) {
      return res.status(400).send(`Signature error: ${err.message}`);
    }

    const client = await pool.connect();
    try {
      await client.query('BEGIN');

      // 1. Claim the event. If it exists, unique_violation fires.
      try {
        await client.query(
          'INSERT INTO processed_events (event_id, event_type) VALUES ($1, $2)',
          [event.id, event.type]
        );
      } catch (err) {
        if (err.code === '23505') {  // unique_violation
          await client.query('ROLLBACK');
          console.log(`Duplicate event ${event.id} — skipping`);
          return res.sendStatus(200);
        }
        throw err;
      }

      // 2. Apply business logic in the SAME transaction.
      if (event.type === 'payment_intent.succeeded') {
        const pi = event.data.object;
        await client.query(
          `UPDATE wallets
             SET balance_cents = balance_cents + $1,
                 updated_at = NOW()
           WHERE user_id = $2`,
          [pi.amount, pi.metadata.user_id]
        );
      }

      await client.query('COMMIT');
      res.sendStatus(200);
    } catch (err) {
      await client.query('ROLLBACK');
      console.error('Handler failed:', err);
      // Return 500 so Stripe retries.
      res.sendStatus(500);
    } finally {
      client.release();
    }
  }
);

app.listen(3000, () => console.log('listening on :3000'));

The critical property: the INSERT into processed_events and the UPDATE on wallets live inside a single BEGIN/COMMIT. If a second retry arrives while the first is still running, Postgres serializes them. The second one hits unique_violation and rolls back the wallet update it was about to make.

Step 4: Run it locally with the Stripe CLI

export DATABASE_URL=postgres://postgres:pw@localhost:5432/postgres
export STRIPE_SECRET_KEY=sk_test_...

# In one terminal: forward events and grab the signing secret
stripe listen --forward-to localhost:3000/webhook

Expected output:

> Ready! Your webhook signing secret is whsec_abc123... (^C to quit)

Copy that secret and export it, then start the server:

export STRIPE_WEBHOOK_SECRET=whsec_abc123...
node handler.js

Trigger a test event in another terminal:

stripe trigger payment_intent.succeeded \
  --add payment_intent:metadata.user_id=user_123

Check the wallet:

psql $DATABASE_URL -c "SELECT * FROM wallets WHERE user_id = 'user_123';"

Expected output (amount will vary by fixture):

 user_id  | balance_cents |          updated_at
----------+---------------+-------------------------------
 user_123 |          2000 | 2024-...

Step 5: Prove idempotency by replaying

Find the event ID in the Stripe CLI output (looks like evt_1P...) and resend it:

stripe events resend evt_1P...

Server log:

Duplicate event evt_1P... — skipping

Balance unchanged:

 user_id  | balance_cents
----------+---------------
 user_123 |          2000

The event was accepted, acknowledged with 200, but the wallet was not touched. This is what idempotency looks like from the outside.

Step 6: Prove it under concurrency

The real test is two simultaneous deliveries. Simulate this by disabling signature verification temporarily and firing two curl requests in parallel with the same fake event ID:

# concurrency-test.sh
BODY='{"id":"evt_test_concurrent_1","type":"payment_intent.succeeded","data":{"object":{"amount":5000,"metadata":{"user_id":"user_123"}}}}'

curl -s -X POST localhost:3000/webhook-unsafe -d "$BODY" &
curl -s -X POST localhost:3000/webhook-unsafe -d "$BODY" &
wait

(Wire up a temporary /webhook-unsafe route that skips constructEvent and parses JSON directly. Do not ship this.)

Run it, then check the balance. It should have gone up by exactly 5000, not 10000. If you run the same test against the broken handler from Step 2, you will see 10000 roughly half the time — the race window is small but real.

Step 7: Handle poison messages

What if the event is malformed or triggers a bug? You do not want Stripe to retry forever. Two guardrails:

• Return 200 for permanent failures (validation errors, unknown user IDs where the correct action is "drop"). Log to your error tracker. Do not let the provider retry things that will never succeed.
• Return 5xx only for transient failures — DB down, downstream API timing out. Stripe retries with exponential backoff for up to 3 days.

Also record the outcome in the result JSONB column so you can audit what happened for any event ID later. This is where the processed_events table earns its keep beyond deduplication.

Step 8: Same pattern for Razorpay, PayPal, and others

Razorpay sends an x-razorpay-signature header you verify with HMAC-SHA256 over the raw body and your webhook secret. The event payload has no top-level id, but each event has payload.payment.entity.id (or the equivalent for the resource type). Use that as your idempotency key. PayPal provides event.id in the same shape as Stripe. The transaction pattern is identical.

For accounting and invoicing platforms that fan out to multiple providers, we usually namespace the event ID: stripe:evt_1P..., razorpay:pay_M.... Prevents collisions if two providers ever pick the same string.

Common errors and troubleshooting

"deadlock detected" in Postgres logs

Usually means you are updating multiple rows in inconsistent order across concurrent transactions. Fix: always update rows in a deterministic order (e.g., ORDER BY user_id before a batch update). For single-row wallet updates, this rarely happens.

Signature verification fails intermittently

You are almost certainly not using the raw body. Express's express.json() middleware mutates the payload, which breaks HMAC verification. Use express.raw({ type: 'application/json' }) on the webhook route only. Frameworks like Next.js API routes need export const config = { api: { bodyParser: false } }.

Handler is fast but Stripe still retries

Check your ack latency. Stripe's timeout is generous but if you are doing heavy work synchronously (sending emails, calling ledgers, updating a search index), move it to a background queue. The webhook handler should do exactly two things: (1) claim the event ID, (2) apply the state change. Everything else — notifications, analytics, downstream fanout — belongs in a worker reading from an outbox table you wrote to in the same transaction.

Duplicate rows in processed_events

Should be impossible with a PRIMARY KEY. If you see it, you are probably using ON CONFLICT DO NOTHING without checking the row count, and treating that as "success" without knowing whether you actually did the work. Prefer the try/catch on 23505 pattern above — it is unambiguous.

The event arrived out of order

Idempotency is not ordering. If refund.created arrives before payment_intent.succeeded, your handler will fail the refund because the payment does not exist yet. Return 5xx so the provider retries, or reconcile via a scheduled job that reads the provider's API as the source of truth. This is a separate problem from duplicate delivery.

What this approach is bad at

Being honest about tradeoffs:

• The processed_events table grows forever. Add a retention policy — most providers do not retry after 3–7 days, so you can safely archive rows older than 30 days. Partition by month if volume is high.
• Long-running transactions hold locks. If your business logic calls a slow external API inside the transaction, you will hurt throughput. Move external calls to a post-commit worker via an outbox row.
• It does not help with non-database side effects. If you send an email inside the handler, a rollback does not un-send it. Same rule: side effects go to a queue, not into the transaction.

For higher-stakes flows — lending disbursements, KYC state changes, ledger postings — we usually pair this with a double-entry ledger and reconciliation jobs. If you are building something in that neighborhood, our work on Cashpo's lending stack and payments modernization projects lean on the same primitives.

Frequently Asked Questions

Can I use Redis instead of Postgres for the idempotency key?

Only if your business logic writes are also in Redis, which is rare for financial state. The whole point is that the idempotency check and the business write commit or roll back together. If your wallet lives in Postgres and your dedup key lives in Redis, you have two systems that can disagree — Redis records the event as processed but the Postgres write failed, or vice versa. Keep them in the same store.

What is the difference between an idempotency key and a webhook event ID?

An idempotency key is a client-generated token you send to an API (like Stripe's Idempotency-Key header) to make outbound calls safe to retry. A webhook event ID is a provider-generated token you receive from an API to make inbound handlers safe under redelivery. Different directions, same underlying idea. Both should be stored and enforced by your database.

How long should I keep processed event IDs?

Long enough to cover the provider's maximum retry window plus a safety margin. Stripe retries for up to 3 days, Razorpay for up to 24 hours, PayPal up to 25 days on some event types. Thirty days is a safe blanket policy for most fintech setups. Archive older rows to cold storage if you need audit history.

Do I need this if I only get one webhook a day?

Yes. The bug does not care about volume — it cares about whether a single event can be delivered twice, which every major provider explicitly warns can happen. One duplicate wallet credit on a real customer is worse than a thousand duplicates on test data.

How do I retrofit this onto a handler that has been running for months?

Add the processed_events table, backfill it with event IDs from your provider dashboard for the last 30 days, then deploy the new handler. Duplicates already in your DB need a separate reconciliation script — usually a query that finds wallets whose SUM(credits) does not match the provider's payment totals for that user. If the surface area is large or you are unsure of the blast radius, talk to CodeNicely for a personalized assessment.