Idempotent sends: designing safe retries for a messaging API

A customer who gets the same "Your order has shipped" message twice notices. Duplicates are rarely caused by bugs in the sending code; they come from retries after a timeout. This guide shows how to make sends idempotent, which errors to retry and how, and how to handle delivery statuses that arrive out of order.

Why duplicates happen

Your system sends a request, the API accepts the message, and the response gets lost: the connection drops, a load balancer times out, or your process restarts before it records the result. From your side, the send failed. If you simply send again, the customer gets the message twice.

The fix is to make "send this notification" an operation that can be repeated safely. The client attaches a key that identifies the logical send, and the server remembers which keys it has already accepted.

How Idempotency-Key works

UPPU's API v1 requires an Idempotency-Key header on every send. The key is 1 to 64 characters long (letters, digits, dot, underscore, colon or hyphen) and stays unique per channel for good:

• The first send with a key is accepted with 202.
• Sending again with the same key and the same body returns 200 with the current state of the first message and the header Idempotent-Replayed: true. No second message goes out.
• The same key with a different body gets 409 idempotency_key_reused. This protects you from reusing a key by mistake.
• A send refused for a policy reason (for example window_closed or consent_required) keeps no key, so you can retry with the same key once the condition has changed.

POST /v1/messages            Idempotency-Key: order-1001-shipped   body A   -> 202 Accepted (first send)
POST /v1/messages            Idempotency-Key: order-1001-shipped   body A   -> 200 OK, Idempotent-Replayed: true
POST /v1/messages            Idempotency-Key: order-1001-shipped   body B   -> 409 idempotency_key_reused

Derive keys from business events

One logical send gets one key, and that key must survive retries and restarts. Two good ways:

• Derive it from the event: order-1001-shipped means "the shipped notification for order 1001". It's stable without storage and readable in logs.
• Store it with the event: generate a UUID or ULID when the event is created and save it next to the event, before the first attempt.

Build every body value from the event too. If you put the attempt time or a retry counter into the body, for example in client_reference or a consent's collected_at, the second attempt carries a different body and gets 409.

def shipped_notification(order):
    key = f"order-{order.id}-shipped"                     # one key per logical send, stable across restarts
    body = {
        "channel_id": order.whatsapp_channel_id,
        "to": {"contact_id": order.contact_id},
        "content": {"type": "template", "template": {
            "name": "order_update", "language": "en",
            "body": {"parameters": [order.first_name, str(order.number)]}}},
        "client_reference": f"order-{order.id}",          # from the event, never from the attempt
    }
    return key, body

What to retry, and how

Every error carries a retryable flag, and it's true only for media_not_ready, rate_limited, channel_backlog, internal_error and service_unavailable. Use exponential backoff with jitter so that many workers don't retry in lockstep, and pace sends yourself: UPPU allows 20 sends per second per key, and the general RateLimit-* headers don't show that limit.

import json
import os
import random
import time
import urllib.error
import urllib.request

BASE = os.environ.get("UPPU_API_BASE", "https://messaging-staging.uppu.net/v1")


def _json(raw):
    try:
        return json.loads(raw or b"null")
    except ValueError:                                    # e.g. an HTML error page from a proxy
        return None


def send_once(body, key):
    request = urllib.request.Request(
        f"{BASE}/messages",
        data=json.dumps(body, ensure_ascii=False).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {os.environ['UPPU_API_KEY']}",
            "Content-Type": "application/json",
            "Accept": "application/json",
            "Idempotency-Key": key,
        },
        method="POST",
    )
    try:
        with urllib.request.urlopen(request, timeout=20) as response:
            return response.status, _json(response.read()), response.headers
    except urllib.error.HTTPError as error:
        return error.code, _json(error.read()), error.headers


def send_with_retries(body, key, attempts=6):
    """Retry with the SAME key and the SAME body. Network errors, 429 and 5xx are retried."""
    delay = 1.0
    for _ in range(attempts):
        try:
            status, data, headers = send_once(body, key)
        except (urllib.error.URLError, TimeoutError):
            status, data, headers = None, None, {}
        if status in (200, 202):
            return data                                   # 200 means: an earlier attempt got through
        error = (data or {}).get("error") or {}
        if status is not None and not error.get("retryable", status >= 500):
            raise RuntimeError(f"{status} {error.get('code')}")   # fix the cause; do not retry blindly
        retry_after = headers.get("Retry-After") if headers else None
        time.sleep(float(retry_after) if retry_after else delay + random.uniform(0, delay))
        delay = min(delay * 2, 30)
    raise RuntimeError("gave up for now; retry later with the same key and body")

Asynchronous failures need a new key

A 202 means the message was accepted, not delivered. The outcome arrives later as a message.status webhook. If that status is failed, for example because the recipient's number isn't on WhatsApp, the logical send is over. Retrying with the same key would only replay the failed message. If you decide to try again, that is a new logical send and needs a new key.

One status needs care: unknown means the request may have reached Meta, but no answer came back. Don't resend automatically: the message may already be on the customer's phone. UPPU retries transient Meta errors itself; a deliberate resend after unknown needs a new key and accepts the risk of a duplicate.

Ordering statuses with previous_status

Statuses move forward only. In UPPU the order is: queued, then accepted or unknown, then sent, failed, delivered and read. Webhook events are delivered at least once and in order per contact, but a retried event can arrive after a later one. Each message.status event carries previous_status and your client_reference, so you can tell where it fits:

• Store the furthest status you have seen, and never move a message backwards.
• If an event's previous_status doesn't match what you have stored, an earlier event is still on its way; apply the new one if it moves forward.
• When in doubt, fetch the message with GET /v1/messages/{id} and reconcile.
• De-duplicate events by their webhook-id; see how to verify webhook signatures.

RANK = {"queued": 0, "accepted": 1, "unknown": 1, "sent": 2, "failed": 3, "delivered": 4, "read": 5}


def apply_status(stored, event):
    """Keep the furthest status; an event that arrives late never moves a message backwards."""
    new = event["data"]["message"]["status"]
    if RANK.get(new, -1) > RANK.get(stored, -1):
        return new
    return stored

A short checklist

1. One logical send, one key, created before the first attempt.
2. Same key and same body on every retry; nothing in the body that depends on the attempt.
3. Retry only network errors, 429 and 5xx; honor Retry-After; back off with jitter.
4. Treat 200 with Idempotent-Replayed: true as success.
5. Treat window_closed as a business outcome, not an outage.
6. A new key only for a new logical send, such as a deliberate resend after failed.
7. Apply statuses forward-only and reconcile when events arrive out of order.

In UPPU's API v1

UPPU's REST API v1 has a public OpenAPI document and an error-code page. Idempotency-Key is required on sends, so retries never create duplicates; webhooks are signed with Standard Webhooks (HMAC-SHA256) and delivered at least once, in order per contact, in up to 12 attempts over about 76 hours. Start with the developer overview.