Reliability
The contract
This is what your endpoint needs to satisfy — everything else on this page is a consequence of these five facts:
| Success | Any 2xx status code. Resets the destination's failure streak to zero. |
| Failure | Anything else — a non-2xx status, a transport error, or a timeout. Classified as final or retried, below. |
| Timeout | Each attempt gets 30 seconds to respond. No response by then counts as a retryable transport failure, same as a dropped connection. |
| Response body | Never read to decide success or failure. Stored for GET /events/deliveries/{id}, truncated to 10,000 characters. |
| Response size | A response over 1 MB is discarded unread — it's recorded as a retryable failure with no status code, not a success. |
Acknowledge fast, then do the real work asynchronously — see the handler examples in Quickstart and Signature Verification.
Retries
Not every failed delivery attempt is retried — whether an attempt is retried depends on how it failed:
- Final — not retried, delivery goes straight to
failed: the destination answered with400,401,403,404,410or422. These are treated as the destination telling us the request itself is wrong (bad payload, bad auth, endpoint gone), so sending it again unchanged would not help. - Retried, on the schedule below:
408,429, any5xxstatus, any transport-level failure — DNS resolution failure, connection refused, timeout, or the connection dropping mid-response — and a response body over the 1 MB limit (see The contract above).
Failed deliveries are retried automatically on the schedule below. The "Cumulative elapsed" column is the total time since the first attempt — the question "how long until you give up on me" that a per-attempt list can't answer on its own.
| Attempt | Delay before attempt | Cumulative elapsed |
|---|---|---|
| 1 | 0s (immediate) | 0s |
| 2 | 5s | 5s |
| 3 | 5m | 5m 5s |
| 4 | 30m | 35m 5s |
| 5 | 2h | 2h 35m 5s |
| 6 | 5h | 7h 35m 5s |
| 7 | 10h | 17h 35m 5s |
| 8 | 10h | 27h 35m 5s |
| 9 | 10h | 37h 35m 5s |
Delays carry up to 10% random jitter to avoid synchronized retry storms, so treat these figures as nominal rather than exact.
After 9 attempts (about 37h 35m 5s after the first), a delivery is marked failed. Independently of any single delivery's outcome, a destination is automatically disabled after 50 consecutive failed deliveries.
Once a delivery reaches a terminal state (succeeded, failed,
superseded, or discarded), fetch its full attempt history — including
each response status and body — from
GET /events/deliveries/{id}.
Auto-disable
Independently of any single delivery's outcome, a destination is automatically disabled after 50 consecutive failed deliveries. This protects both sides: your endpoint stops being hammered by a backlog it clearly cannot process, and Helix stops burning retry budget on a destination that is not coming back on its own.
A disabled destination does not resume automatically — re-enabling is an
explicit PATCH { "status": "enabled" } on the destination, once you've
confirmed the endpoint is healthy again. See
Backfill for catching up on what was missed while disabled.
Leaving auto_disabled status by any path — customer or admin — resets
the consecutive-failure counter to zero, including a transition straight to
disabled rather than enabled. So auto_disabled → disabled → enabled
starts counting from zero again, the same as auto_disabled → enabled
directly. An ordinary enabled ↔ disabled toggle that never passed through
auto_disabled still never resets the counter.
This applies whether the destination goes to disabled by your own PATCH
(for example, planned maintenance) or to auto_disabled on its own. A
delivery that is still pending, waiting on its next scheduled retry, when
that attempt comes due against a destination that is not enabled is
finalized as discarded — it is not held for when you re-enable the
destination. Re-enabling only lets new events through; anything
discarded while the destination was off has to be recovered explicitly with
Backfill.
Replay
POST /events/deliveries/{id}/replay
re-sends a single delivery immediately, skipping the debounce window. Use it
when you've fixed a bug in your handler and want to reprocess a specific
failed delivery without waiting for a new event.
A few rules keep replay from producing confusing outcomes:
- Only a delivery that has finished —
succeeded,failed,discarded, orsuperseded— can be replayed. One stillpendingordeliveringreturns 409. - A
supersededdelivery is refused with 409 too, but for a different reason than the line above:supersededmeans a newer delivery for the same subject collapsed over this one via debounce before it was ever sent — not that another replay claimed it. (A delivery another replay has already claimed isdelivering, same as any in-flight delivery.) Replay the delivery that superseded it instead. - Two replay requests racing the same delivery also produce a 409 for the
loser — the winner's claim moves the delivery straight to
delivering, so the loser's request finds it no longer in a replayable status. - A delivery that is
discardedbecause its subscription is inactive, or because its destination organization has lost access to the event's feed (see Concepts: Feed access is re-checked at send time), cannot be replayed either — the request is refused with409, and the error names the reason. - Replay counts toward destination health exactly like an ordinary delivery: a success clears the failure streak, a failure adds to it and can trip auto-disable.
- Replay never re-enables an already auto-disabled destination on its own —
that stays an explicit status
PATCH. A replay against a destination that is not enabled is not sent: the delivery becomesdiscarded, unless it had already succeeded, in which case it stayssucceeded. - A replay can only improve a delivery's recorded outcome (failed → succeeded), never regress a succeeded one to failed.
- If Temporal (the workflow engine that runs replay) is unreachable, the
request answers
503with aRetry-Afterheader within a few seconds, rather than a generic500after a long timeout; the delivery's claim on the replay reverts so a retried request isn't blocked by the failed one.
Backfill
POST /events/destinations/{ref}/backfill
is the recovery path that makes auto-disable safe: when a destination has
been off — disabled, or simply down — this replays a historical window of
events against its current subscriptions.
curl -X POST https://api.feeds.onhelix.ai/events/destinations/{ref}/backfill \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "since": "2024-01-01T00:00:00Z", "until": "2024-01-02T00:00:00Z" }'
{
"success": true,
"data": { "eventsScanned": 4213, "deliveriesCreated": 187 }
}
- Only events your destination's current subscriptions match are replayed — the same matching rule live routing uses — and feed access is re-checked per event, so a feed you've since lost access to is skipped.
- The window may not exceed the 30-day retention period, and
sincemust precedeuntil; both violations are 400s. Both bounds must carry aZor an explicit offset; a zone-less timestamp is a 400. - The scan is also capped at 10,000 in-scope events per call. A window that holds more is refused outright with a 400 rather than partially backfilled — narrow the window and run backfill again in parts, oldest-first, until you've covered the whole gap.
- The response reports events scanned and deliveries created, so you can see the blast radius before triggering it against a large window.
- Backfilled deliveries skip the debounce window — the events are already historical, so there's nothing to collapse.
- Backfill creates at most one delivery per
(type, source, subject)in the window — the latest matching event, not one per historical change — and starts those deliveries oldest-first. - Re-running an overlapping window creates nothing new.
- If Temporal (the workflow engine that runs backfill) is unreachable, the
request answers
503with aRetry-Afterheader within a few seconds, rather than a generic500after a long timeout; the backfill claim reverts so a retried request isn't blocked by the failed one.
Retention
A delivery and its attempt history are kept for 30 days after the delivery
was created. Once a delivery has both reached a terminal status
(succeeded, failed, superseded, or discarded) and passed that 30-day
mark, it is deleted permanently — after that,
GET /events/deliveries/{id}
returns 404, and the attempt history, response bodies, and everything else
on that row are gone for good. There is no extended-retention tier; export
delivery history yourself if you need to keep it longer.
An event that was never routed to any destination is the one exception: it is kept past 30 days rather than pruned, so a routing bug doesn't destroy the only record of what was missed.
Idempotency
The envelope's id field (see
Concepts) is stable across
every retry attempt and every replay of the same logical delivery. Use it as
your dedupe key:
if (await alreadyProcessed(event.id)) return;
await handle(event);
await markProcessed(event.id);
This is also why webhook-id (see
Signature Verification)
matches the envelope id — it is the same delivery identifier, surfaced in
the header so you can dedupe before even parsing the body.
A replay reuses the original delivery's id — it is not a new
logical delivery, it is the same one sent again. If your dedupe store already
has that id recorded from the earlier attempt, a naive id-only dedupe key
will silently discard the replay instead of reprocessing it. If you want a
replay to actually reprocess (for example, after fixing a bug in your
handler), key your dedupe on (id, webhook-timestamp) instead — the
timestamp changes on every send, including a replay — or explicitly clear
the id from your dedupe record before triggering the replay.
Test deliveries
POST /events/destinations/{ref}/test
sends one delivery.test event to a destination, now, through the same
path a real delivery takes: the same CloudEvents envelope, the same
webhook-id / webhook-timestamp / webhook-signature headers signed with
the destination's active secret(s), the same custom headers, timeout and
address checks. It answers with what happened — the HTTP status your
endpoint returned, or the transport failure — so you can prove your route,
your auth and your signature verification before the first real event.
It is deliberately narrow:
- One fixed type.
delivery.testis not in the catalog and cannot be subscribed to. Itsdatais a short message plus the destination's id and name. There is no way to request a syntheticnews.item.addedwith a realistic payload — the old system's per-type fixtures were a maintenance cost that grew with every type, and this replaces them with one envelope that exercises everything a real one does except the payload shape. For the payload shape, use the per-type pages (e.g. news.item.added) and JSON Schemas fromGET /events/types. - Not recorded. No delivery row, no attempt, no change to the destination's health counters. It is a probe, not traffic.
- Rate limited. It makes an outbound request to your URL, so it shares the validate endpoint's limit of 10 requests per minute per organization.
Your handler should return 2xx for delivery.test (after verifying the
signature as it would for any event) and otherwise ignore it — do not treat
it as an item. Once real deliveries have occurred, replay is the
way to re-exercise your handler against real payloads.
Next steps
- API Reference — the full
/events/*surface.