Webhook delivery for Tidewire
Rebuilt how shipment events reach customer systems, with fair queues and honest retries. Tidewire: freight visibility platform for shippers across ASEAN.
99.98%
- Tidewire
- Senior engineer, platform team
- 5 months
- 2024.05, v2.0
- Rust, PostgreSQL, SQS, TypeScript
Problem
Tidewire tells shippers where their containers are by sending webhooks: arrived at port, cleared customs, out for delivery. One large customer's endpoint went down for an afternoon, retries piled up in a shared queue, and every other customer's events arrived four hours late.
Customers had no way to see what was sent, what failed or why. Every incident became a support thread with screenshots of logs.
Constraints
- About 40 million events a day, spiking around vessel arrivals.
- Customer endpoints range from serverless functions to a single PHP box in a warehouse office.
- Events for one shipment must arrive in order.
Architecture
Code
Full jitter keeps a recovering endpoint from being hit by every retry in the same second.
/// Delay before attempt n (1-based). Caps at 30 minutes, gives up after 24 hours.pub fn next_delay(attempt: u32, rng: &mut impl Rng) -> Option<Duration> { if attempt > 18 { return None; // dead-letter, customer sees it in the dashboard } let cap = 1_800_000u64; // 30 min in ms let base = 500u64.saturating_mul(1 << attempt.min(16)); let ceiling = base.min(cap); Some(Duration::from_millis(rng.gen_range(0..=ceiling)))}
Result
- delivered within 60 seconds
- 99.98%
- delay caused to other customers by one outage
- 0 min
- webhook support tickets
- -62%
- replay from the customer dashboard
- 1-click
The attempt log became a product feature with its own page in the customer console, and it is now the first thing sales demos.