Delivery Pipeline

The delivery pipeline is what hookstream does with an event once it arrives: verify it, route it through matching connections, transform the payload, deliver to each destination, retry on failure, and dead-letter anything that never makes it through.

Pipeline Overview

graph TD
  A[HTTP Request] --> B[Ingest<br/>verify + dedup + store]
  B --> C[Route<br/>evaluate connections]
  C --> D[Filter<br/>content-based rules]
  D --> E[Transform<br/>JSONata]
  E --> F[DeliveryScheduler DO<br/>per destination]
  F --> G{Deliver}
  G -->|success| H[Delivered]
  G -->|failure| I[Retry<br/>alarm-driven]
  I --> G
  I -->|exhausted| J[DLQ + Issue]

Events flow through six stages: ingest, route, transform, deliver, retry, DLQ.

Ingest Stage

The ingest endpoint (https://hookstream.io/v1/ingest/<source_id>) accepts any HTTP method. Each request runs through:

1. Signature verification — HMAC-SHA256, HMAC-SHA1, or Standard Webhooks.
2. IP filter — allow/deny by IP or CIDR.
3. Dedup check — KV lookup by payload hash, header, or body field.
4. Event storage — D1 for metadata, D1 or R2 for the payload.
5. Real-time broadcast — pushed to the per-org EventStream DO for dashboards and the CLI listen command.
6. Pipeline dispatch — delivery is scheduled via ctx.waitUntil() so ingestion returns 200 immediately.

The whole ingest path is designed to return as fast as possible — everything downstream happens after the response is sent.

Delivery Stage

Every destination owns a dedicated DeliveryScheduler Durable Object. When an event is dispatched:

1. The DO checks the circuit breaker state.
2. If closed or half-open, it attempts delivery via the configured provider.
3. On success — the attempt is recorded and a delivery.succeeded event is broadcast.
4. On failure — a retry alarm is scheduled and the circuit breaker's failure count is incremented.
5. On final failure — the event moves to the DLQ and an issue is created or updated.

Because every destination has its own DO, failures on one destination never slow down deliveries to another.

Retry Logic

Retries are alarm-driven inside the DO. The retry policy is configurable:

• Exponential backoff (default) — each attempt uses the next interval in the array: 30s → 5m → 30m → 2h → 24h.
• Linear backoff — base interval × attempt number.
• Fixed backoff — the first interval, every time.

Default intervals: [30, 300, 1800, 7200, 86400] seconds. See the retry strategies guide for tuning advice.

Circuit Breaker

The circuit breaker protects destinations from cascading failures:

• Closed (normal) — deliveries proceed; consecutive failures are tracked.
• Open (tripped) — after 5 consecutive failures, deliveries are paused and new events are queued. A cooldown alarm is set.
• Half-Open (testing) — after cooldown, one probe delivery is sent. Success closes the circuit and flushes the queue. Failure re-opens it.

Circuit state is visible in the dashboard and queryable via GET /v1/destinations/:id/circuit. See the circuit breaker guide for more.

While the circuit is open, events are queued in the DO — they aren't lost. When the circuit closes, the queue flushes in order. But if traffic is heavy and the destination stays down, the queue can grow quickly. Monitor your issues page.

Dead Letter Queue

Events that exhaust every retry are moved to the DLQ. They're still visible in the dashboard and API, and can be manually retried individually or in bulk. An issue is created automatically for each DLQ event, with severity based on failure count. See the DLQ guide for replay patterns.

Next Steps