Skip to main content

Reliability

The contract

This is what your endpoint needs to satisfy — everything else on this page is a consequence of these five facts:

SuccessAny 2xx status code. Resets the destination's failure streak to zero.
FailureAnything else — a non-2xx status, a transport error, or a timeout. Classified as final or retried, below.
TimeoutEach attempt gets 30 seconds to respond. No response by then counts as a retryable transport failure, same as a dropped connection.
Response bodyNever read to decide success or failure. Stored for GET /events/deliveries/{id}, truncated to 10,000 characters.
Response sizeA response over 1 MB is discarded unread — it's recorded as a retryable failure with no status code, not a success.

Acknowledge fast, then do the real work asynchronously — see the handler examples in Quickstart and Signature Verification.

Retries

Not every failed delivery attempt is retried — whether an attempt is retried depends on how it failed:

  • Final — not retried, delivery goes straight to failed: the destination answered with 400, 401, 403, 404, 410 or 422. These are treated as the destination telling us the request itself is wrong (bad payload, bad auth, endpoint gone), so sending it again unchanged would not help.
  • Retried, on the schedule below: 408, 429, any 5xx status, any transport-level failure — DNS resolution failure, connection refused, timeout, or the connection dropping mid-response — and a response body over the 1 MB limit (see The contract above).

Failed deliveries are retried automatically on the schedule below. The "Cumulative elapsed" column is the total time since the first attempt — the question "how long until you give up on me" that a per-attempt list can't answer on its own.

AttemptDelay before attemptCumulative elapsed
10s (immediate)0s
25s5s
35m5m 5s
430m35m 5s
52h2h 35m 5s
65h7h 35m 5s
710h17h 35m 5s
810h27h 35m 5s
910h37h 35m 5s

Delays carry up to 10% random jitter to avoid synchronized retry storms, so treat these figures as nominal rather than exact.

After 9 attempts (about 37h 35m 5s after the first), a delivery is marked failed. Independently of any single delivery's outcome, a destination is automatically disabled after 50 consecutive failed deliveries.

Once a delivery reaches a terminal state (succeeded, failed, superseded, or discarded), fetch its full attempt history — including each response status and body — from GET /events/deliveries/{id}.

Auto-disable

Independently of any single delivery's outcome, a destination is automatically disabled after 50 consecutive failed deliveries. This protects both sides: your endpoint stops being hammered by a backlog it clearly cannot process, and Helix stops burning retry budget on a destination that is not coming back on its own.

A disabled destination does not resume automatically — re-enabling is an explicit PATCH { "status": "enabled" } on the destination, once you've confirmed the endpoint is healthy again. See Backfill for catching up on what was missed while disabled.

Leaving auto_disabled status by any path — customer or admin — resets the consecutive-failure counter to zero, including a transition straight to disabled rather than enabled. So auto_disabled → disabled → enabled starts counting from zero again, the same as auto_disabled → enabled directly. An ordinary enabled ↔ disabled toggle that never passed through auto_disabled still never resets the counter.

Disabling a destination discards its pending deliveries — it does not pause them

This applies whether the destination goes to disabled by your own PATCH (for example, planned maintenance) or to auto_disabled on its own. A delivery that is still pending, waiting on its next scheduled retry, when that attempt comes due against a destination that is not enabled is finalized as discarded — it is not held for when you re-enable the destination. Re-enabling only lets new events through; anything discarded while the destination was off has to be recovered explicitly with Backfill.

Replay

POST /events/deliveries/{id}/replay re-sends a single delivery immediately, skipping the debounce window. Use it when you've fixed a bug in your handler and want to reprocess a specific failed delivery without waiting for a new event.

A few rules keep replay from producing confusing outcomes:

  • Only a delivery that has finished — succeeded, failed, discarded, or superseded — can be replayed. One still pending or delivering returns 409.
  • A superseded delivery is refused with 409 too, but for a different reason than the line above: superseded means a newer delivery for the same subject collapsed over this one via debounce before it was ever sent — not that another replay claimed it. (A delivery another replay has already claimed is delivering, same as any in-flight delivery.) Replay the delivery that superseded it instead.
  • Two replay requests racing the same delivery also produce a 409 for the loser — the winner's claim moves the delivery straight to delivering, so the loser's request finds it no longer in a replayable status.
  • A delivery that is discarded because its subscription is inactive, or because its destination organization has lost access to the event's feed (see Concepts: Feed access is re-checked at send time), cannot be replayed either — the request is refused with 409, and the error names the reason.
  • Replay counts toward destination health exactly like an ordinary delivery: a success clears the failure streak, a failure adds to it and can trip auto-disable.
  • Replay never re-enables an already auto-disabled destination on its own — that stays an explicit status PATCH. A replay against a destination that is not enabled is not sent: the delivery becomes discarded, unless it had already succeeded, in which case it stays succeeded.
  • A replay can only improve a delivery's recorded outcome (failed → succeeded), never regress a succeeded one to failed.
  • If Temporal (the workflow engine that runs replay) is unreachable, the request answers 503 with a Retry-After header within a few seconds, rather than a generic 500 after a long timeout; the delivery's claim on the replay reverts so a retried request isn't blocked by the failed one.

Backfill

POST /events/destinations/{ref}/backfill is the recovery path that makes auto-disable safe: when a destination has been off — disabled, or simply down — this replays a historical window of events against its current subscriptions.

curl -X POST https://api.feeds.onhelix.ai/events/destinations/{ref}/backfill \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "since": "2024-01-01T00:00:00Z", "until": "2024-01-02T00:00:00Z" }'
{
"success": true,
"data": { "eventsScanned": 4213, "deliveriesCreated": 187 }
}
  • Only events your destination's current subscriptions match are replayed — the same matching rule live routing uses — and feed access is re-checked per event, so a feed you've since lost access to is skipped.
  • The window may not exceed the 30-day retention period, and since must precede until; both violations are 400s. Both bounds must carry a Z or an explicit offset; a zone-less timestamp is a 400.
  • The scan is also capped at 10,000 in-scope events per call. A window that holds more is refused outright with a 400 rather than partially backfilled — narrow the window and run backfill again in parts, oldest-first, until you've covered the whole gap.
  • The response reports events scanned and deliveries created, so you can see the blast radius before triggering it against a large window.
  • Backfilled deliveries skip the debounce window — the events are already historical, so there's nothing to collapse.
  • Backfill creates at most one delivery per (type, source, subject) in the window — the latest matching event, not one per historical change — and starts those deliveries oldest-first.
  • Re-running an overlapping window creates nothing new.
  • If Temporal (the workflow engine that runs backfill) is unreachable, the request answers 503 with a Retry-After header within a few seconds, rather than a generic 500 after a long timeout; the backfill claim reverts so a retried request isn't blocked by the failed one.

Retention

A delivery and its attempt history are kept for 30 days after the delivery was created. Once a delivery has both reached a terminal status (succeeded, failed, superseded, or discarded) and passed that 30-day mark, it is deleted permanently — after that, GET /events/deliveries/{id} returns 404, and the attempt history, response bodies, and everything else on that row are gone for good. There is no extended-retention tier; export delivery history yourself if you need to keep it longer.

An event that was never routed to any destination is the one exception: it is kept past 30 days rather than pruned, so a routing bug doesn't destroy the only record of what was missed.

Idempotency

The envelope's id field (see Concepts) is stable across every retry attempt and every replay of the same logical delivery. Use it as your dedupe key:

if (await alreadyProcessed(event.id)) return;
await handle(event);
await markProcessed(event.id);

This is also why webhook-id (see Signature Verification) matches the envelope id — it is the same delivery identifier, surfaced in the header so you can dedupe before even parsing the body.

A replay reuses the original delivery's id — it is not a new logical delivery, it is the same one sent again. If your dedupe store already has that id recorded from the earlier attempt, a naive id-only dedupe key will silently discard the replay instead of reprocessing it. If you want a replay to actually reprocess (for example, after fixing a bug in your handler), key your dedupe on (id, webhook-timestamp) instead — the timestamp changes on every send, including a replay — or explicitly clear the id from your dedupe record before triggering the replay.

Test deliveries

POST /events/destinations/{ref}/test sends one delivery.test event to a destination, now, through the same path a real delivery takes: the same CloudEvents envelope, the same webhook-id / webhook-timestamp / webhook-signature headers signed with the destination's active secret(s), the same custom headers, timeout and address checks. It answers with what happened — the HTTP status your endpoint returned, or the transport failure — so you can prove your route, your auth and your signature verification before the first real event.

It is deliberately narrow:

  • One fixed type. delivery.test is not in the catalog and cannot be subscribed to. Its data is a short message plus the destination's id and name. There is no way to request a synthetic news.item.added with a realistic payload — the old system's per-type fixtures were a maintenance cost that grew with every type, and this replaces them with one envelope that exercises everything a real one does except the payload shape. For the payload shape, use the per-type pages (e.g. news.item.added) and JSON Schemas from GET /events/types.
  • Not recorded. No delivery row, no attempt, no change to the destination's health counters. It is a probe, not traffic.
  • Rate limited. It makes an outbound request to your URL, so it shares the validate endpoint's limit of 10 requests per minute per organization.

Your handler should return 2xx for delivery.test (after verifying the signature as it would for any event) and otherwise ignore it — do not treat it as an item. Once real deliveries have occurred, replay is the way to re-exercise your handler against real payloads.

Next steps