What Is a Webhook? And Why Your Integration Keeps Breaking

For: An operations or product lead at a mid-size SaaS company whose third-party integration — payment provider, CRM, or shipping carrier — keeps missing events in production, and whose developer has explained it as 'a webhook issue' without making the fix obvious

A webhook is a push notification between servers: instead of your app repeatedly asking a provider "any news?", the provider calls a URL you own the moment an event happens. That is the easy part. The reason your integration keeps breaking is almost never that the provider failed to send the event — it is that your handler processes the same event twice when the provider retries, and your database now believes a customer paid twice, shipped twice, or upgraded twice. Fix the idempotency, not the delivery.

The rest of this post explains what a webhook actually is, why it beats polling, the three failure modes that cause most production incidents, and a minimal example of a handler that survives them.

The problem webhooks solve

Say your SaaS needs to know when Stripe captures a payment, HubSpot updates a contact, or FedEx marks a package delivered. You have two options.

Polling: hit the provider's API every 30 seconds asking "anything new?" This works, but you waste 99% of those requests on empty responses, you hit rate limits, and you learn about events up to 30 seconds late. Bad for anything time-sensitive.

Webhooks: you register a URL with the provider once. When something happens on their side, they make an HTTP POST to your URL with a JSON body describing the event. You respond with a 2xx status code to acknowledge receipt. That is the whole protocol.

In the webhook vs polling tradeoff, webhooks win on latency and efficiency. They lose on control — you cannot ask for events on your schedule, you inherit the provider's retry behavior, and if your endpoint is down, you depend on the provider to redeliver.

The delivery-truck analogy

Polling is you driving to the post office every 10 minutes to check for mail. Webhooks are the postal service delivering mail to your house.

The catch: if you are not home when the postal worker arrives, they might leave the package on your porch, try again tomorrow, or give up after three attempts — depending on the carrier. Some carriers deliver the same package twice if their tracking system glitches. Your job is to (a) always be home, (b) have a place to put the package, and (c) not accept the same package twice.

That is webhooks. Being home means your endpoint returns 2xx quickly. Having a place to put it means storing the event before processing. Not accepting duplicates means idempotency.

A minimal worked example

Here is a Stripe webhook handler in about 20 lines. Note what it does before it does any real work.

app.post('/webhooks/stripe', async (req, res) => {
  const sig = req.headers['stripe-signature'];
  let event;

  // 1. Verify the signature. Reject forgeries.
  try {
    event = stripe.webhooks.constructEvent(req.rawBody, sig, WEBHOOK_SECRET);
  } catch (err) {
    return res.status(400).send('Invalid signature');
  }

  // 2. Check if we've already processed this event ID.
  const seen = await db.processedEvents.findOne({ id: event.id });
  if (seen) return res.status(200).send('Already processed');

  // 3. Acknowledge fast. Do the real work async.
  await db.processedEvents.insertOne({ id: event.id, receivedAt: new Date() });
  await queue.enqueue('stripe-event', event);

  res.status(200).send('OK');
});

Four things matter here:

• Signature verification before anything else. Webhook URLs are public. Without verification, anyone can POST a fake "payment succeeded" event to you.
• Deduplication by event ID is what makes the handler idempotent. Stripe, Shopify, GitHub, and most mature providers include a unique event ID on every delivery. Same ID on a retry = same event.
• Fast acknowledgment. Most providers time out webhook calls at 5–30 seconds. If your handler runs a 45-second report generation inline, the provider marks it failed, retries, and you get duplicate events on top of your slow handler.
• Real work happens on a queue. The webhook endpoint's only job is to accept, validate, dedupe, and enqueue. The queue worker does the mutation.

The three failure modes that actually break production

1. The non-idempotent handler (the big one)

Your handler receives payment_succeeded for order #4471. It marks the order paid, emails the customer, and triggers fulfillment. Response takes 8 seconds. The provider timed out at 5 seconds, marks the delivery failed, and retries. Now you have two "payment received" emails, two fulfillment jobs, and if your accounting logic adds to a running total instead of setting a state, you have double-counted revenue.

An idempotent webhook handler can receive the same event 1 time or 100 times and produce the same end state. The pattern above — dedupe by event ID before doing work — is the cheapest way to get there. For extra safety, make the downstream operations idempotent too: UPDATE orders SET status = 'paid' WHERE id = ? AND status != 'paid' is safe to run twice. UPDATE revenue SET total = total + ? is not.

2. Silent endpoint failures

Your server returned a 500 for six hours during a deploy. The provider retried for a while, then gave up. You now have a gap in your event history and no alert.

Fixes: monitor your webhook endpoint's error rate as a first-class SLO. Enable the provider's dead-letter or event-replay feature (Stripe has one, Shopify has one, most mature APIs do). Reconcile periodically by pulling a list of recent events from the provider's REST API and cross-checking against what you processed. Reconciliation is the seatbelt; webhooks are the airbag.

3. Out-of-order delivery

Webhooks are not ordered. You can receive subscription.updated before subscription.created, or a shipment's delivered event before in_transit. Providers do not guarantee order, and network retries make disorder likely.

Fix: use the timestamp on the event, not the order of arrival, to decide whether to apply it. If you receive an update older than the current state, drop it. Store last_event_at on the record and compare.

Webhook retry logic: what providers actually do

Every provider retries differently, and you must read their docs. A rough map:

• Stripe: retries for up to 3 days with exponential backoff on any non-2xx response.
• Shopify: 19 retries over 48 hours, then removes the webhook subscription entirely if it keeps failing.
• GitHub: retries 8 times over ~8 hours.
• Twilio: does not retry status callbacks by default — you have to enable it.

The one that surprises teams is Shopify auto-unsubscribing. If your endpoint is broken for a weekend, the webhook is just gone Monday morning, and no one notices until a customer complains about a missing order.

When to use webhooks vs when not to

Use webhooks when: events are infrequent relative to how often you would poll, latency matters (payments, shipping updates, auth events), or the provider explicitly recommends them.

Skip webhooks when: you need guaranteed ordering, you cannot expose a public HTTPS endpoint, or the data changes so frequently that a batched pull is cheaper. For bulk sync of a slowly-changing catalog, a nightly REST pull is often simpler and more reliable than a webhook firehose.

Use both when the integration is critical. Webhooks for real-time reaction, a scheduled reconciliation pull for correctness. Every mature payments integration I have seen does this.

How CodeNicely can help

When we built GimBooks — an accounting SaaS for Indian SMBs — the integration surface was the hard part, not the ledger logic. GST filings, payment gateways, invoicing partners: each with its own webhook contract, its own retry behavior, and its own idea of what "delivered" means. Getting the idempotency layer and the reconciliation job right was what let the product scale without a support queue full of "my invoice shows paid twice" tickets.

If your team is debugging an integration that keeps dropping or duplicating events, that is the shape of work we do. Our integration and modernization engagements typically start with an audit of the webhook handlers, the retry assumptions, and the reconciliation gap — then a fix that leaves you with clear observability, not more magic. Full IP ownership, no lock-in.

Frequently Asked Questions

What is the difference between a webhook and an API?

An API is a door you knock on — your code calls the provider's server. A webhook is the reverse — the provider's server calls yours. Same HTTP, opposite direction. Most integrations use both: webhooks to hear about events, the REST API to fetch details or reconcile.

Why is my webhook handler receiving the same event multiple times?

Because the provider thought your first response failed. Either you returned a non-2xx status, you timed out (usually >5–30 seconds), or the network dropped the acknowledgment. This is normal and expected. The fix is not to stop the retries — it is to make your handler idempotent by deduplicating on the event ID.

How do I test webhooks in local development?

Use a tunnel like ngrok, Cloudflare Tunnel, or the CLI tools most providers ship (Stripe CLI, Shopify CLI). They forward the provider's HTTP POST to your localhost. For automated tests, replay saved event payloads directly against your handler — you do not need a real provider in the loop.

Should I use webhooks or polling for a new integration?

Webhooks if the provider supports them and events are moderately infrequent. Polling if you need guaranteed ordering, cannot expose a public endpoint, or the API explicitly recommends it. For anything financial or customer-facing, use webhooks for latency and a scheduled reconciliation pull for correctness.

How long does it take to fix a broken webhook integration?

It depends entirely on how the current handler is written, how many event types are in play, and whether you have historical data to backfill. For a personalized assessment, contact CodeNicely and we will scope it against your specific stack.

The bottom line

Webhooks are simple in concept and unforgiving in production. The delivery part almost always works — providers retry aggressively. The part that breaks is your handler, quietly processing the same payment or status update twice because no one told you that "at-least-once delivery" is the default guarantee and "exactly-once" is your problem to solve. Verify the signature, dedupe by event ID, acknowledge fast, do the work on a queue, and reconcile on a schedule. That is 90% of production webhook reliability.