Your agentic email webhook handler is retrying into a duplicate send: idempotency, ordering, and the retry rules that stop it

Most duplicate agent emails aren't a delivery problem — they're a handler that treats a retry as a new event.

When an autonomous agent replies twice to the same email, the duplicate send happens because your webhook handler processed the same inbound delivery twice without an idempotency barrier. Production webhook infrastructure operates on an at-least-once delivery guarantee, meaning solid agentic email webhook retry logic requires your application to absorb identical retries safely without producing duplicate outbound side effects.

If your system experiences unexpected failures, retries are functioning as designed rather than malfunctioning. Webhook senders implement retries because networks drop packets, proxy servers drop idle connections, and receiver databases encounter transient locks. If a provider used at-most-once delivery instead, transient hiccups on your server would silently discard customer emails. Because retries are inevitable, your webhook endpoint must treat idempotency as an architectural baseline rather than an optional safeguard.

When handlers lack proper idempotency mechanisms, failures manifest in three distinct failure modes:

• Duplicate outbound messages: An agent reads an incoming request, executes its language model chain, and sends two identical replies to an external recipient because the original webhook delivery timed out before the status code was returned.
• Orphaned calendar holds: A retried event causes the agent to re-request a slot hold. The second hold collides with the first or locks out subsequent slots, triggering availability errors down the line.
• Corrupted audit trails: The database records two conflicting decisions for a single action, creating an inconsistent log when an engineer inspects what the agent actually decided.

To eliminate duplicate sends, you must implement a reliable deduplication key, enforce strict HTTP status code semantics, maintain an exponential backoff schedule with jitter, and configure your state machine to handle out-of-order event arrivals.

What a delivery actually contains, and which field you dedupe on

A reliable webhook delivery carries metadata, an event payload, and a cryptographic signature. A standard delivery envelope includes the event name, a unique event identifier, a timestamp, and an authentication header. For example, AgentDraft signs every webhook delivery with an X-AgentDraft-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256> header computed over <t>. followed by the raw request body, using the workspace's signing secret.

A common engineering mistake is attempting to use the signature header or its timestamp (t=<unix seconds>) as the deduplication key. Do not do this. The delivery provider updates the timestamp header on every delivery attempt. Hashing or indexing that header generates a new key for every retry, completely bypassing your deduplication logic.

The only valid primary deduplication key is the payload's top-level event_id (or id). If you integrate with an external sender that fails to provide a stable event identifier across retries, your fallback key must be a deterministic hash (such as SHA-256) of the immutable fields within the event body (for instance, sha256(message_id + thread_id + timestamp_received)).

Below is a minimal PostgreSQL schema demonstrating a production-grade processed_deliveries table:

CREATE TABLE processed_deliveries (
    workspace_id VARCHAR(64) NOT NULL,
    event_id VARCHAR(128) NOT NULL,
    event_type VARCHAR(64) NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    response_status INT NOT NULL,
    PRIMARY KEY (workspace_id, event_id)
);

CREATE INDEX idx_processed_deliveries_ttl 
ON processed_deliveries (processed_at);

This table must be written in the exact same database transaction that records your agent's initial state transition or outbound intent. A naive check-then-act pattern implemented in application code contains an inherent race condition:

# BROKEN: Application-level check-then-act race window
if not db.query("SELECT 1 FROM processed_deliveries WHERE event_id = :id", id=event_id):
    # Context switch or parallel retry arrives here
    agent.send_reply(to=sender, body=draft)
    db.execute("INSERT INTO processed_deliveries (event_id) VALUES (:id)", id=event_id)

If two identical delivery attempts hit two worker processes simultaneously, both queries return empty results, and both workers issue outbound replies. As documented in the PostgreSQL documentation on unique constraints, declaring a PRIMARY KEY (workspace_id, event_id) or a UNIQUE constraint ensures that the database storage layer serializes conflicting writes. The second concurrent transaction encounters a unique constraint violation and aborts before the side effect executes.

Deduplication records should not be retained indefinitely. A practical retention floor is 7 days, which comfortably exceeds the maximum retry lifetime of any enterprise webhook provider. You can purge older records asynchronously with a partition-drop or a scheduled daily query:

DELETE FROM processed_deliveries 
WHERE processed_at < NOW() - INTERVAL '7 days';

The status code is the retry policy: 2xx, 4xx, 5xx, and 429

Webhook senders treat your HTTP response code as the explicit control instruction for its retry engine. Standard semantics defined in RFC 9110 HTTP Semantics dictate how clients interpret success, client failure, and temporary server fault conditions. Returning the wrong status code either creates message dropouts or triggers infinite retry loops.

A frequent anti-pattern in high-throughput handlers is returning 200 OK immediately upon HTTP receipt, then offloading the event to an in-memory queue for background execution. If that background worker encounters an unhandled exception or terminates during a host restart, the delivery is permanently lost. The remote sender treats the event as acknowledged and ceases delivery attempts. Handlers must persist the raw event to a durable table or a durable message bus before returning an HTTP 2xx status.

Conversely, returning 400 Bad Request when your database times out tells the sender that the payload itself is corrupt. The sender drops the event. Under RFC 6585 Additional HTTP Status Codes, a 429 Too Many Requests response informs the caller that rate limits were exceeded and pairs with a Retry-After header to specify a pause window. Reserve 4xx status codes strictly for unrecoverable delivery errors, such as invalid signatures, JSON syntax errors, or mismatched workspace IDs.

Backoff, jitter, and the retry storm you cause by returning 500 too fast

When an internal database lock triggers an uncaught exception, returning 500 Internal Server Error informs the upstream sender to retry. However, if your service returns 500 across thousands of concurrent webhooks, a fixed-interval retry schedule triggers a self-inflicted Distributed Denial of Service (DDoS) attack.

Industrial webhook delivery systems utilize truncated exponential backoff paired with full jitter, calculating retry intervals using the standard formula outlined in the AWS Architecture Blog on Exponential Backoff and Jitter:

t_retry = Min(t_max, t_base * 2^(attempt))
sleep_duration = Uniform(0, t_retry)

Where t_base is commonly 1 second and t_max is capped around 300 seconds (5 minutes). Incorporating a randomized offset across the interval prevents your fleet of recovering workers from being repeatedly hit by synchronized waves of retried traffic.

After a configured threshold (typically between 5 and 8 failed attempts), the sender marks the delivery dead and moves the payload to a Dead-Letter Queue (DLQ). When this happens, an engineer must inspect the payloads or replay them using administrative APIs.

Because you cannot configure the sender's retry schedule directly, you must configure your receiver's internal processing timeouts to remain shorter than the sender's HTTP connection timeout. If your upstream sender abandons connections after 15 seconds, configure your database queries and inference steps to timeout strictly at 10 seconds. This guarantees your handler can catch its own deadline exhaustion and return a clean, intentional 503 Service Unavailable or 429 Too Many Requests with an explicit Retry-After: 30 header, rather than allowing the upstream load balancer to forcefully sever an open socket mid-write.

Ordering: retries arrive out of order, and your state machine has to survive it

At-least-once delivery semantics do not guarantee in-order delivery. If message 1 fails its initial delivery attempt and is queued for a 60-second retry, a subsequent message 2 sent 5 seconds later will arrive at your webhook handler before the retry of message 1.

If your agent blindly appends incoming payloads to an internal conversational array, your application state degrades:

# BROKEN: Assuming arrival order equals chronological order
thread = db.load_thread(payload.thread_id)
thread.messages.append(payload.message)
db.save_thread(thread)
# Results in message 2 appearing before message 1!

To establish resilience against out-of-order execution, enforce monotonically increasing sequence numbers or validate the sender's chronological timestamp (sent_at) on a per-thread basis. When an out-of-order event is detected, your handler must choose between two strategies:

1. Insert with index positioning: Store messages in a normalized relation where ordering is derived from the sender's creation timestamp (ORDER BY sent_at ASC) rather than auto-incrementing surrogate database IDs.
2. Defer processing: If your agent must respond to an unbroken sequence of state transitions, store the early-arriving event in an unsequenced_events buffer and return 200 OK. Process it only when the missing predecessor event has successfully committed.

Furthermore, when an agent emits an outbound reply as a result of an inbound event, it must attach an idempotency key to that outgoing HTTP request. According to the Stripe API documentation on idempotent requests, passing an explicit client-generated idempotency key allows the upstream payment gateway or messaging layer to identify retry attempts and prevent accidental double-billing or duplicate sends. Passing Idempotency-Key: agent-reply-{event_id} to your outbound mail provider ensures that even if your application crashes and restarts, the downstream mail service safely collapses the duplicate request.

Signature verification belongs before the dedupe check, not after

Do not read the deduplication key or execute database queries before authenticating the webhook signature. Querying your storage layer using untrusted payload headers allows unauthenticated actors to mount denial-of-service attacks against your database connection pool. The sequence of operations must proceed in a strict pipeline:

1. Read the raw byte stream of the incoming request body.
2. Extract the signature and timestamp headers.
3. Verify that the timestamp falls within a reasonable replay boundary (for instance, within 300 seconds).
4. Compute the HMAC-SHA256 signature using your shared secret and compare it against the provided header using a constant-time comparison utility.
5. Execute your deduplication check against the storage layer.
6. Parse the JSON payload and execute business logic.

A critical operational hazard is relying on HTTP middleware that automatically decodes JSON before signature verification occurs. Deserializing and re-serializing JSON often alters spacing, key ordering, and byte boundaries, which invalidates the computed HMAC signature. Handlers must verify against the raw byte string.

AgentDraft signs every webhook delivery with an X-AgentDraft-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256> header computed over <t>. followed by the raw request body, using the workspace's signing secret. The receiver verifies it; AgentDraft recommends that receivers reject any timestamp more than 300 seconds (5 minutes) from the current time to block replays, and its documented reference verifier does so. Signed webhook delivery is available on every plan, including the free Developer tier.

The following Python implementation demonstrates this exact verification flow:

import hmac
import hashlib
import time

def verify_agentdraft_webhook(raw_body: bytes, sig_header: str, secret: str) -> bool:
    # Header format: t=1729600000,v1=a1b2c3d4...
    pairs = dict(item.split("=") for item in sig_header.split(","))
    timestamp_str = pairs.get("t")
    signature_hex = pairs.get("v1")

    if not timestamp_str or not signature_hex:
        return False

    # Enforce the 300-second freshness window
    current_time = int(time.time())
    delivery_time = int(timestamp_str)
    if abs(current_time - delivery_time) > 300:
        return False

    # Prepare signature payload: '<t>.' + raw_body
    signed_payload = f"{timestamp_str}.".encode("utf-8") + raw_body
    computed_signature = hmac.new(
        secret.encode("utf-8"),
        signed_payload,
        hashlib.sha256
    ).hexdigest()

    # Constant-time comparison prevents timing attacks
    return hmac.compare_digest(computed_signature, signature_hex)

For more details on avoiding signature-level replay attacks, review our guide to agentic email webhook replay protection. Validating signatures protects against unauthorized origin forging, while your database deduplication key guarantees idempotent handling.

Testing retry logic without waiting for production to break

Most development teams test webhook handlers exclusively along the happy path: they trigger a single test event via a CLI, observe a 200 OK response, and deploy to production. This test profile completely misses real-world failure dynamics.

To safely test your agentic email webhook retry logic before deploying to production, incorporate these four validation scenarios into your continuous integration (CI) pipeline:

• Manual payload replay: Capture a genuine signed webhook delivery payload, forward it to your local staging port twice in immediate succession, and assert that your external communication mock receives precisely one outbound request.
• Crash simulation: Insert a conditional fault injection hook that raises an unhandled process termination immediately after your outbound mail client returns success, but before the deduplication record commits. Replay the payload and assert that your downstream service's idempotency key absorbed the second call without creating a second message.
• Explicit 4xx validation: Post an intentionally truncated JSON payload to your webhook endpoint. Verify that the handler returns an HTTP 400 Bad Request rather than catching the parse error and throwing a 500 Internal Server Error.
• Reversed event sequences: Send a simulated message.received event carrying an advanced timestamp or sequence index, immediately followed by the preceding event. Verify that your application normalizes the storage indices correctly without corrupting conversational context.

You can execute these integration tests using a local sandbox or staging workspace. AgentDraft's Developer tier is free with no card required: 1 seat, 3 agents, 1 mailbox, 1 connected calendar, 50 bookings a month, and 7-day audit retention. Signed webhook delivery is included on the Developer tier, so a free workspace is a working sandbox for testing webhook handlers.

Where the retry boundary sits when the agent also books and asks for approval

A production agent's execution path frequently involves more than just parsing an email; it updates shared calendars and triggers consequential business processes. Idempotency must span the entire action chain.

Consider an autonomous scheduling agent. If an incoming email prompts the agent to reserve a meeting, an unhandled webhook retry must not generate duplicate holds or overbook a client. AgentDraft coordinates holds and commits through a priority-aware conflict engine so multiple agents can act on the same calendar without double-booking. AgentDraft holds expire after 30 seconds by default. A committed booking can be bumped by a higher-priority agent only within a 30-second window, after which it is frozen. A single booking is capped at 480 minutes (8 hours), buffers included, and at 99 five-minute buckets per request; a longer request is rejected with 422 booking_too_long. The cap is service-wide; no workspace or plan setting raises it.

AgentDraft's conflict engine locks time in five-minute buckets: a 30-minute booking writes 6 bucket rows, and the 480-minute maximum booking spans 96. Because these guarantees execute at the storage layer inside a single atomic DynamoDB transaction, a retried webhook attempt cannot double-allocate conflicting five-minute slots.

Similarly, when an agent processes an email that triggers a high-impact operation—such as issuing a financial refund, approving an enterprise contract, or provisioning cloud infrastructure—the retry handler must not spam your operations team with duplicate approval requests. AgentDraft lets an agent pause any consequential action for human sign-off: it opens an approval request carrying a one-line summary and a JSON evidence payload, a person approves or denies it in the dashboard with an optional note, and the agent reads the outcome back. The gated action does not have to be one AgentDraft performs — a deploy, a migration, or a refund is gated the same way. Every transition lands in the append-only audit trail and fires an approval.* webhook.

Approvals are decided in the AgentDraft dashboard. AgentDraft emails the workspace owner a notification linking to the queue, but the decision itself is made signed in — there are deliberately no approve-from-email links, because an unauthenticated one-click approve is an attack surface. Slack, Discord, Teams, SMS and push delivery are not available today. The requesting agent decides for itself when to open an approval request. AgentDraft does not yet provide a policy engine that auto-requires approval by action class, amount threshold, or role, and there are no escalation chains or multi-approver quorums — a single workspace human resolves each request.

AgentDraft records state-changing agent actions in an append-only audit trail. When you inspect an agent's execution history following a network disruption, this audit trail provides chronological certainty, proving that a repeated event was processed as an idempotent retry rather than an uncoordinated secondary action.

A checklist you can apply to the handler you have open right now

Before deploying an agentic email webhook handler to production, review your implementation against this engineering checklist:

• [ ] Signature verification runs first: The HMAC-SHA256 signature is calculated over the raw request body bytes and validated using constant-time comparison before any database queries execute.
• [ ] Clock drift is bounded: The timestamp in the signature header is verified against your server's clock, rejecting deliveries with greater than 300 seconds of drift.
• [ ] Deterministic deduplication key: The primary key for idempotency is derived from the payload's unique event_id rather than the signature timestamp.
• [ ] Atomic storage boundary: Deduplication records are committed in the same database transaction that registers the state change, enforced by a unique database constraint.
• [ ] Deliberate status codes: 2xx is returned only after durable writes complete; transient issues return 5xx or 429; malformed payloads return 4xx.
• [ ] Outbound call idempotency: All downstream external API requests (such as outbound email replies) forward the incoming event_id inside an Idempotency-Key HTTP header.
• [ ] Out-of-order tolerance: State transitions are governed by thread-level sequence indicators or message timestamps rather than HTTP arrival ordering.
• [ ] Automated cleanup: A retention policy purges historical deduplication keys after a minimum of 7 days to conserve database storage.

Frequently Asked Questions

How should human-in-the-loop approval workflows handle retried webhooks?

When a retried webhook triggers an approval flow, the handler must check whether an approval request for that delivery's idempotency key already exists before creating a new one. AgentDraft lets an agent pause any consequential action for human sign-off: it opens an approval request carrying a one-line summary and a JSON evidence payload, a person approves or denies it in the dashboard with an optional note, and the agent reads the outcome back. The gated action does not have to be one AgentDraft performs — a deploy, a migration, or a refund is gated the same way. Every transition lands in the append-only audit trail and fires an approval.* webhook. Approvals are decided in the AgentDraft dashboard. AgentDraft emails the workspace owner a notification linking to the queue, but the decision itself is made signed in — there are deliberately no approve-from-email links, because an unauthenticated one-click approve is an attack surface. Slack, Discord, Teams, SMS and push delivery are not available today.

Should the webhook handler return 200 before or after it finishes processing?

Your handler must return an HTTP 2xx code only after the incoming event has been safely persisted to durable storage. If your agent uses asynchronous processing, write the incoming event to an operational database table or a durable event log inside a transaction, and then immediately return 200 OK or 202 Accepted. Returning 200 before persisting the event risks permanent data loss if your worker crashes. Conversely, holding the HTTP connection open while waiting for slow LLM inference to complete causes connection timeouts and unnecessary retries from the webhook sender.

What should I use as the idempotency key if the sender's event id changes between retries?

If an upstream provider generates a new event identifier on every retry attempt, you cannot use that field as a deduplication key. You must construct a deterministic fallback hash from the immutable business fields within the message payload. Create a SHA-256 hash across the sender's email address, the inbound message ID header, the thread ID, and the original message timestamp. Use this computed hash as the unique constraint in your deduplication table.

How long do I need to keep dedupe records for agentic email webhook retry logic to be safe?

You must keep deduplication records for at least the length of your webhook provider's maximum retry window. Most webhook senders abandon failed deliveries within 24 to 72 hours. Retaining deduplication rows for 7 days provides a safe operational margin against extended retry schedules while allowing engineers to replay historical webhooks manually during incident recovery. After 7 days, deduplication records should be deleted via scheduled database tasks.

Is a 429 from my handler treated differently from a 500 by the sender?

Yes. A 500 Internal Server Error signals an unexpected system failure, prompting the sender to retry using its standard exponential backoff schedule. A 429 Too Many Requests explicitly indicates rate-limit exhaustion. When returning a 429 status, your handler should include a Retry-After: [seconds] header. Reputable webhook delivery engines read this header and pause delivery attempts for that specific duration, preventing excessive traffic spikes against your recovering infrastructure.

Idempotency is the contract, retries are just the delivery mechanism

Webhook infrastructures operate on a split contract: the delivery system guarantees at-least-once transmission, and your handler guarantees idempotent execution. Attempting to suppress retries at the sender layer is impossible over unreliable networks. Durability and consistency are achieved at the storage layer by combining constant-time signature verification, unique database constraints on event_id, and outbound idempotency keys.

For complete details on event envelopes and webhook header formats, consult the AgentDraft documentation. Once you make your webhook handler idempotent, the next challenges to address are managing human sign-offs and avoiding booking collisions. AgentDraft gives AI agents per-agent email inboxes with inbound webhooks, replies, and audit evidence. The number of inboxes a workspace can run at once is set by plan: 1 on Developer and Individual, 5 on Team, 25 on Scale, unlimited on Enterprise. AgentDraft syncs Google Calendar today; Microsoft 365 / Outlook calendar sync is planned, not yet shipped. AgentDraft is a proprietary hosted API; it is not open source and is not offered as a self-hosted or on-premise product. Enterprise SSO (SAML/SCIM via WorkOS) is on the AgentDraft roadmap and not available today; agents authenticate with bearer API keys and humans with passkeys. AgentDraft does not hold formal compliance certifications (SOC 2, HIPAA, ISO 27001, etc.). It does keep an append-only audit trail. AgentDraft's Developer tier is free with no card required: 1 seat, 3 agents, 1 mailbox, 1 connected calendar, 50 bookings a month, and 7-day audit retention. Signed webhook delivery is included on the Developer tier, so a free workspace is a working sandbox for testing webhook handlers. To run your handler against live infrastructure, sign up for AgentDraft at https://agentdraft.io/pricing on the free Developer tier, which requires no card and includes 1 user, 3 agents, and one mailbox per workspace.