Webhook reliability for government data feeds: retries, signatures, dead-endpoint detection

A push product is an implicit SLA. If our webhooks stop and we don't notice, you find out days later from a missed deadline. So the reliability plumbing isn't a roadmap item — it shipped on day one, and this post documents it.

1. Signatures

Every POST carries X-SamHooks-Signature: HMAC-SHA256 over the raw body. Four lines to verify (snippet). Unsigned webhook products are asking you to trust every host on the internet.

2. Retries with a floor under them

Failed deliveries retry up to 5 times with exponential backoff via a queue, not a cron loop. After the 5th failure the message goes to a dead-letter queue and is recorded failed-final — visible in GET /v1/status, never silently dropped.

3. Dead-endpoint detection

20 consecutive final failures auto-disable the endpoint. Disabled means: webhooks pause, your RSS feed keeps flowing, the status API says disabled, and recovery is a self-serve PUT — documented procedure, no support ticket.

4. Watching the watcher

An hourly monitor checks upstream freshness (is the extract stale?), delivery backlog, newly disabled endpoints, and billing anomalies — and pages the operator. The same signals render publicly at /status/. We publish the log instead of an uptime percentage.

This is also why we exist vs pull-only APIs: reliability work is most of a push product. Compare: SamHooks vs GovConAPI, then quickstart.