> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hookstream.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Deduplication

Prevent duplicate webhook processing with payload hash, header, or body field strategies.

Providers occasionally send the same webhook twice — during their own retry loops, failovers, or race conditions. Without deduplication you might double-charge a customer, fire the same notification twice, or corrupt a sync. Hookstream dedupes at ingest using a KV-backed window so each unique event only fans out to your connections once.

## Three dedup strategies

Pick the one that matches how your provider guarantees uniqueness.

### `payload_hash`

Hookstream computes a SHA-256 hash of the raw request body and checks KV for a match. Perfect when the entire payload is identical on retry (most providers). Zero configuration beyond enabling it.

### `header_field`

Uses a specific header — typically an idempotency key like `X-GitHub-Delivery` or `Idempotency-Key` — as the dedup key. Best when the provider supplies an explicit unique id.

### `body_field`

Extracts a value from the JSON body using dot-notation (`data.id`, `event.uuid`, etc.). Useful when the idempotency id lives inside the payload instead of a header.

Duplicate events are still *stored* (with `dedup_status: "duplicate"`) so you can see them in the dashboard — they just don't get delivered to your destinations. That makes debugging much easier than silently dropping them.

## The dedup window

The window controls how long a key is remembered. Default is 300 seconds (5 minutes), which covers most providers' retry windows.

Pick a window longer than your provider's maximum retry interval:

- Stripe: 3 days max retry → a window of 1–7 days if you're truly paranoid, but in practice 15 minutes catches almost everything.
- GitHub: retries within an hour → 1 hour window.
- Most SaaS: retries within 1–5 minutes → the default 5 minutes is fine.

## Configure dedup

**Pick a strategy**

Use `header_field` if the provider sends an idempotency header — it's the cheapest and most correct. Fall back to `payload_hash` if they don't.
  **Set it on the source**

Patch the source with `dedup_strategy`, optionally `dedup_field`, and a window:

    ```json header-field
    {
      "dedup_strategy": "header_field",
      "dedup_field": "x-github-delivery",
      "dedup_window_seconds": 3600
    }
    ```

    ```json payload-hash
    {
      "dedup_strategy": "payload_hash",
      "dedup_window_seconds": 300
    }
    ```

    ```json body-field
    {
      "dedup_strategy": "body_field",
      "dedup_field": "data.id",
      "dedup_window_seconds": 300
    }
    ```
  **Verify with a duplicate**

Replay a real webhook (or send the same request twice via `curl`). The second event shows up in the dashboard with a `duplicate` badge and no delivery attempts.

## Implementation notes

- Dedup keys live in [Cloudflare KV](https://developers.cloudflare.com/kv/) with a TTL equal to the dedup window. Storage cleans itself up — no maintenance.
- Header lookup is case-insensitive.
- If `header_field` or `body_field` can't extract a value (missing header, invalid JSON), hookstream falls back to `payload_hash` so the event still gets deduped.

## Next Steps

**[Sources API](/api/sources)**

Full source schema, including every dedup field.
  **[IP Filtering](/guides/ip-filtering)**

Combine dedup with a network-level allowlist.