Building Reliable Shipping Webhooks: A Developer's Playbook

In the world of e-commerce, ensuring that shipping information flows smoothly is crucial. A single missed webhook can mean a customer never gets their tracking update — or worse, your system might end up double-shipping an order. Typically, carrier webhook delivery rates hover around 95-98%. This might sound sufficient, but at scale, it means you're inevitably going to lose some events. So, how do you build a robust webhook system that can handle the real world with grace?

Understanding the Core Problem

Carrier APIs, such as those provided by USPS, FedEx, and UPS, send webhook notifications to update you about tracking information, label statuses, and delivery confirmations. However, there's a catch: webhooks can fail silently. The carriers operate on a fire-and-forget model, meaning they send the webhook notification and assume it reaches you. If your endpoint is down, even for a brief 30 seconds during a deployment, those events vanish into the ether.

Designing an Effective Architecture

To ensure your webhook system can handle these challenges, a few architectural principles are essential.

Idempotency: The Non-Negotiable Rule

To avoid processing the same event multiple times, use idempotency keys. This ensures each webhook is processed exactly once. Typically, you can use a combination of the event ID or the tracking number and status as your idempotency key. Store these processed event IDs in a fast-access storage like Redis with a 24-hour time-to-live (TTL). Every time a webhook comes in, check if it's already been handled. If it has, simply return a 200 status and skip further processing.

Immediate Acknowledgment, Deferred Processing

When a webhook arrives, return a 200 status immediately. Then, push the payload to a message queue like BullMQ, SQS, or RabbitMQ. Processing inline and taking longer than a few seconds can lead to carriers retrying the webhook, causing duplicates. By acknowledging first and processing later, you allow yourself the flexibility to handle the payload without the pressure of immediate completion.

Implementing a Reconciliation Job

Given that 2-5% of webhooks may be lost, it's wise to have a safety net. Set up a cron job to run every 15 minutes that polls the carrier APIs for shipments that haven't received updates as expected. For instance, query your database for shipments in transit that haven't had a webhook update in the last four hours, and then use the carrier's tracking API to check on those shipments. This proactive step helps you catch any lost notifications.

Signature Verification for Security

Security is paramount. Always verify the signatures of incoming webhooks. Each carrier has its own method: USPS uses HMAC-SHA256, FedEx uses a shared secret header, and UPS utilizes OAuth-signed payloads. This step is crucial because webhook endpoints are public URLs and could be targeted by malicious actors. Skipping this in production is a risk you shouldn't take.

Crafting a Robust Retry Strategy

Failures aren't just on the carrier's side. Sometimes your processing fails due to reasons like a database outage or service error. Implementing a retry strategy with exponential backoff and jitter can help manage these issues. Start with an immediate retry for the first attempt. Subsequent retries should progressively back off: 30 seconds plus a random jitter, then two minutes plus a jitter, followed by a 15-minute gap, and finally an hour before alerting your team if the problem persists.

Monitoring: Your System's Health Check

To maintain a healthy webhook system, keep an eye on three key metrics. First, monitor the webhook receive rate, which should remain steady. Second, track the processing latency, aiming for 99% of requests to be completed under 500 milliseconds. Finally, check the reconciliation hit rate. If this is catching more than 5% of events, it indicates a problem with your webhook intake system that needs addressing.

In the pursuit of seamless e-commerce operations, building a reliable webhook system is a critical component. With the right architecture and practices, you can minimize errors and ensure your shipping information is as dependable as your service. If you're looking for a robust solution to manage your shipping webhooks, consider exploring the infrastructure provided by Atoship. Their platform is designed to handle the complexities of shipping notifications efficiently, helping you maintain smooth operations. Explore Atoship's webhook infrastructure →