Deduplication

Providers occasionally send the same webhook twice — during their own retry loops, failovers, or race conditions. Without deduplication you might double-charge a customer, fire the same notification twice, or corrupt a sync. Hookstream dedupes at ingest using a KV-backed window so each unique event only fans out to your connections once.

Three dedup strategies

Pick the one that matches how your provider guarantees uniqueness.

payload_hash

Hookstream computes a SHA-256 hash of the raw request body and checks KV for a match. Perfect when the entire payload is identical on retry (most providers). Zero configuration beyond enabling it.

header_field

Uses a specific header — typically an idempotency key like X-GitHub-Delivery or Idempotency-Key — as the dedup key. Best when the provider supplies an explicit unique id.

body_field

Extracts a value from the JSON body using dot-notation (data.id, event.uuid, etc.). Useful when the idempotency id lives inside the payload instead of a header.

Duplicate events are still stored (with dedup_status: "duplicate") so you can see them in the dashboard — they just don't get delivered to your destinations. That makes debugging much easier than silently dropping them.

The dedup window

The window controls how long a key is remembered. Default is 300 seconds (5 minutes), which covers most providers' retry windows.

Pick a window longer than your provider's maximum retry interval:

• Stripe: 3 days max retry → a window of 1–7 days if you're truly paranoid, but in practice 15 minutes catches almost everything.
• GitHub: retries within an hour → 1 hour window.
• Most SaaS: retries within 1–5 minutes → the default 5 minutes is fine.

Configure dedup

{
  "dedup_strategy": "header_field",
  "dedup_field": "x-github-delivery",
  "dedup_window_seconds": 3600
}

{
  "dedup_strategy": "payload_hash",
  "dedup_window_seconds": 300
}

{
  "dedup_strategy": "body_field",
  "dedup_field": "data.id",
  "dedup_window_seconds": 300
}

Implementation notes

• Dedup keys live in Cloudflare KV with a TTL equal to the dedup window. Storage cleans itself up — no maintenance.
• Header lookup is case-insensitive.
• If header_field or body_field can't extract a value (missing header, invalid JSON), hookstream falls back to payload_hash so the event still gets deduped.

Next Steps