A trustworthy webhook design treats the callback as a promise: define a stable event envelope, guarantee at-least-once delivery with idempotency keys, retry failed deliveries with exponential backoff, and sign every payload with a verifiable, timestamped HMAC. Skip any one of these four commitments and subscribers will eventually build workarounds that route around your API entirely.
Quick answer: Design webhooks like an API you don't control — a versioned event envelope, at-least-once delivery with idempotent consumers, exponential-backoff retries, and HMAC signatures inside a replay window. Document the contract with the same rigor as any inbound endpoint, because a subscriber's integration lives or dies on it.
Why a Webhook Is an API You Don't Control
A webhook flips the usual client-server relationship: your server becomes the client, and a subscriber's endpoint becomes the server whose uptime, timeout tolerance, and error handling you can't dictate. That inversion means every discipline you'd apply to an inbound endpoint — versioning, contracts, meaningful error codes — still applies, just pointed the other way.
Most platform teams treat webhooks as a footnote: a POST URL field in a settings page, described in a paragraph of prose buried in the docs. That's a mistake with a long tail. A webhook is the one integration surface where your bugs become someone else's production incident, silently, hours after you shipped.
Consider what a subscriber has to build against a poorly specified webhook:
- Guessing which fields are guaranteed to be present versus optional
- Reverse-engineering your retry timing by watching duplicate deliveries arrive
- Writing defensive code for payload shapes that changed without a changelog entry
- Building their own polling fallback because they stopped trusting the callback
Each of those is a tax you're imposing on every integration built on your platform. The fix starts with the same rigor you'd bring to specifying an inbound API contract — except now you're specifying what you promise to send, not what you'll accept. If you haven't yet nailed down the foundational decisions of treating your API as a product, our complete guide to API product design is the place to start before layering webhooks on top.
Designing the Event Envelope and the Thin-vs-Fat Payload Tradeoff
A well-designed event envelope wraps every webhook in a consistent, versioned structure — a unique event ID, a type, a timestamp, and a data payload — so subscribers can write one parser for every event type your platform ever emits. Getting the envelope right up front avoids a breaking change later, once hundreds of integrations depend on today's ad hoc shape.
A minimal, defensible envelope looks like this:
{
"id": "evt_01HZK3F9T8N6QABC",
"type": "invoice.paid",
"created_at": "2026-07-09T14:03:22Z",
"api_version": "2026-06-01",
"data": {
"object": "invoice",
"id": "inv_9f8s7d6f",
"amount_paid": 4800,
"currency": "usd"
}
}
Note what's doing the work here: id lets a subscriber deduplicate; type lets them route without parsing the body; api_version lets you evolve the shape without silently breaking old integrations. Choosing good event type names is really an exercise in resource modeling around the nouns in your domain — invoice.paid only reads cleanly because invoice is already a well-understood resource elsewhere in your API.
Thin vs Fat Payloads
The biggest early decision is how much of the underlying object to embed in data. A thin payload sends only IDs and references, forcing the subscriber to call your API back to fetch the full object. A fat payload embeds the entire object state at the moment the event fired.
| Dimension | Thin payload | Fat payload |
|---|---|---|
| Subscriber effort | Must make a follow-up API call | Zero follow-up calls needed |
| Freshness | Always current (fetched live) | Reflects state at event time only |
| Bandwidth / payload size | Small, fast to deliver | Larger, more storage on subscriber's end |
| Race-condition risk | Low — data fetched fresh | Higher — object may have changed again before processing |
| Your API load | One extra call per event received | None |
| Best fit | High-value, low-frequency events (e.g. account deletion) | High-frequency, latency-sensitive events (e.g. payment status) |
Neither option is universally correct — most mature platforms land on a hybrid: a fat-enough payload to cover the common case, plus a stable resource URL in the envelope so a subscriber can always fetch the authoritative current state if theirs is stale. Webhooks aren't answering an ad hoc query the way a REST, RPC, or GraphQL endpoint might — you're pushing a fixed shape, so pick the shape deliberately rather than defaulting to "whatever the internal object model looks like this quarter."
Delivery Guarantees: At-Least-Once, Ordering, and Idempotency
Nearly every production webhook system promises at-least-once delivery, never exactly-once — network partitions, timeouts, and retries make duplicate delivery unavoidable, and pretending otherwise just moves the bug to whoever discovers it in production. Design the contract so a duplicate delivery is harmless, not a rare edge case you hope never happens.
Martin Kleppmann's Designing Data-Intensive Applications frames this precisely: distributed systems can offer at-least-once or at-most-once semantics cheaply, but exactly-once requires an idempotent consumer cooperating with the producer — the guarantee is a property of the whole pipeline, not something the sender can promise alone. Two concrete commitments follow from that:
- Include a stable, unique event
idin every delivery, including retries of the same logical event, so subscribers can deduplicate with a simple "have I seen this ID" check. - Never guarantee ordering across event types. A
subscription.canceledevent can arrive before asubscription.updatedevent fired seconds earlier, especially across retries. If ordering genuinely matters, include a monotonic sequence number or the underlying resource'supdated_attimestamp so subscribers can order themselves.
Cloud messaging systems like AWS SNS/SQS document this same tradeoff publicly: at-least-once delivery with no ordering guarantee on standard queues, and ordering only available at a throughput cost on FIFO variants. Your webhook system is making the identical tradeoff, whether or not you've written it down — writing it down is the whole difference between a documented contract and a support ticket.
Retry and Backoff: Designing for Failure as the Default Path
Assume every delivery will fail at least once in the lifetime of an integration, and design the retry policy as core functionality, not an afterthought. A well-behaved retry schedule uses exponential backoff with jitter, respects the HTTP status code the subscriber returned, and eventually gives up with a clear notification rather than retrying forever.
Status codes should drive very different behavior:
| Response | What it means | Retry behavior |
|---|---|---|
2xx | Delivered and accepted | Mark delivered, stop |
4xx (except 429) | Subscriber rejected the payload — likely a bug on their end or a stale signing secret | Retry a small number of times, then flag for manual review |
429 | Subscriber is rate-limiting you | Back off and retry per any Retry-After header |
5xx / timeout | Subscriber's server is down or overloaded | Retry with exponential backoff on the full schedule |
A representative backoff schedule might look like:
| Attempt | Delay before this attempt | Cumulative elapsed time |
|---|---|---|
| 1 | Immediate | 0 |
| 2 | ~30 seconds | 30 seconds |
| 3 | ~5 minutes | ~5.5 minutes |
| 4 | ~30 minutes | ~36 minutes |
| 5 | ~2 hours | ~2.6 hours |
| 6+ | ~12-24 hours, then stop | 1-3 days |
GitHub's own webhook infrastructure follows this same shape publicly: failed deliveries are retried on a backoff schedule, and an endpoint that fails consistently over an extended stretch eventually gets flagged or disabled rather than retried indefinitely. That's the right instinct — an endpoint that's been failing for three days isn't a transient blip, and continuing to hammer it wastes both sides' resources.
Build a dead-letter view into your dashboard so subscribers (and your own support team) can see failed deliveries, inspect the payload, and manually replay a specific event once the underlying issue is fixed. Without it, a subscriber's only recourse after a bad deploy is asking you to somehow resend three days of missed events.
Signature Verification and Replay Protection
Every webhook payload should carry a cryptographic signature so the subscriber can prove it actually came from you and wasn't tampered with in transit or replayed by an attacker who captured an old request. The standard mechanism is an HMAC computed over the raw request body plus a shared secret, sent as a header alongside a timestamp.
The construction, at minimum:
- Generate a per-subscriber signing secret at webhook setup, rotatable without downtime (support two active secrets during rotation).
- Compute
HMAC-SHA256(secret, timestamp + "." + raw_body)— the raw, unparsed body, not a re-serialized version, since re-serialization can silently change field ordering or whitespace and invalidate the signature. - Send both the timestamp and the resulting signature as headers (e.g.
X-Webhook-Timestamp,X-Webhook-Signature). - Have the subscriber recompute the HMAC on their end using constant-time comparison to avoid timing-attack leakage, and reject anything that doesn't match.
This is exactly the construction defined by RFC 2104, the original IETF HMAC specification, and it's what Stripe, GitHub, and most mature webhook systems implement in practice. Stripe's own documentation is a useful reference point: it recommends rejecting any webhook whose timestamp is more than five minutes old, which is the replay-protection half of the equation — without a timestamp check, a captured, validly-signed payload can be resent indefinitely.
A signature without a timestamp check only proves the payload wasn't tampered with. It doesn't prove it wasn't replayed — verify both, every time.
The open Standard Webhooks specification (backed by webhook-infrastructure vendors and adopted in some form by companies including Zapier and Twilio Segment) formalizes this same envelope-plus-signature pattern into a shared, framework-agnostic standard, precisely because so many platforms were independently reinventing a slightly-wrong version of it.
Documenting the Contract So Subscribers Can Trust It
A webhook is only as trustworthy as its documentation: subscribers need the exact envelope shape, every event type you emit, the retry schedule, and the signature algorithm written down as a versioned contract — not inferred from watching production traffic. Treat the webhook spec with the same seriousness as your inbound endpoint docs, because from the subscriber's side, it is an endpoint they had to build.
Think about the contract from the subscriber's side, the same way you'd map a customer's journey through the highs and lows of onboarding. The first webhook a new integrator receives is often the moment they decide whether your platform is trustworthy infrastructure or a source of ongoing pain.
Framing each event type around the job a subscriber is hiring it for — the same lens our complete guide to Jobs to Be Done applies to end-user features — works just as well here. It's how you decide which events are worth emitting at all, versus which just add noise.
A useful webhook contract includes, at minimum:
- A changelog of every field added, deprecated, or removed per
api_version - A sandbox/test mode that lets subscribers trigger a real delivery on demand, rather than waiting for a live event
- Explicit documentation of which fields are guaranteed present versus optional, per event type
- The full retry schedule and status-code handling table, so subscribers know what "no response yet" actually means
This is precisely the kind of specification that's easy to describe in a paragraph and hard to keep accurate by hand once a platform has a few dozen event types. Prodinja's API Designing tool is built around exactly this gap.
It lets you specify a webhook's payload shape and headers as a formal endpoint contract — the same way you'd spec an inbound route. What you generate is a source of truth a subscriber can read to know exactly what you'll POST them, rather than a paragraph of prose that quietly drifts out of date.
Key Takeaways
- A webhook is an inverted API — you're the client now, and the same contract discipline (versioning, clear errors, documentation) still applies, just pointed outward.
- Design the event envelope first: a stable
id,type,created_at,api_version, anddatashape lets subscribers write one parser for every event you'll ever emit. - Choose thin vs fat payloads deliberately — thin payloads stay fresh but cost the subscriber an API call; fat payloads are convenient but can go stale before they're processed.
- At-least-once delivery is the honest guarantee — build idempotent consumers on both sides rather than pretending exactly-once is achievable.
- Never promise cross-event ordering; include a sequence number or timestamp if a subscriber needs to reconstruct order themselves.
- Retry with exponential backoff, respect status codes, and give up visibly with a dead-letter view rather than retrying forever or silently dropping events.
- Sign every payload with HMAC and check a replay window — a signature alone proves authenticity, not freshness.
- Document the contract as precisely as any inbound endpoint — subscribers are building real production code against your paragraph of prose.
Frequently Asked Questions
What's the difference between at-least-once and exactly-once webhook delivery?
At-least-once means a subscriber may receive the same event more than once and must handle duplicates gracefully; exactly-once means each event arrives precisely one time. Nearly all production webhook systems, including cloud messaging infrastructure like AWS SNS/SQS, only offer at-least-once cheaply — exactly-once requires an idempotent consumer cooperating with the sender, not a promise the sender can make alone.
How many times should a webhook retry before giving up?
Most mature systems retry for somewhere between one and three days using exponential backoff, then stop and flag the endpoint for review rather than retrying indefinitely. The exact ceiling matters less than having one at all, paired with a dead-letter view so a failed delivery can be manually replayed once the subscriber fixes their endpoint.
How do I verify a webhook signature is legitimate?
Recompute the HMAC over the raw request body and shared secret using the same algorithm the sender used (commonly HMAC-SHA256 per RFC 2104), compare it to the signature header using constant-time comparison, and separately check that the accompanying timestamp is recent — Stripe's documented five-minute tolerance is a reasonable default. Signature-only checks miss replay attacks; timestamp-only checks miss tampering.
Should webhook payloads include the full object or just an ID?
It depends on event frequency and freshness needs: thin, ID-only payloads stay accurate because the subscriber fetches live data, while fat payloads save a round trip but can reflect stale state by the time they're processed. Many platforms land on a hybrid — enough embedded data for the common case, plus a resource URL for subscribers who need to confirm current state.
What is a webhook replay attack and how do I stop it?
A replay attack resends a previously valid, correctly-signed webhook payload that an attacker captured in transit, hoping the subscriber's endpoint processes it again as if it were new. Preventing it requires checking a timestamp header against a short tolerance window (commonly around five minutes) in addition to verifying the signature itself, and rejecting anything outside that window even if the signature checks out.