A2A 1.0 · Ed25519 receipts · SOC 2 in progress
Build October 26, 2026Status
AgentDraft▸agentdraft.io
API key

Human Review of LLM Output Before Workflow Continuation: Where the Gate Belongs and How It Survives Retries

A gate that lives in a prompt is not a gate. Here is how to place human review of LLM output before workflow continuation as a durable, idempotent state transition your agent can retry safely.

Reliable human review of LLM output before workflow continuation requires an externalized state machine instead of an in-context instruction. When an agent must verify a generated response, draft an email, or confirm a calendar mutation before acting, the approval state must live in a durable datastore with a distinct lifecycle, explicit idempotency keys, and an append-only audit trail.

If you tell an LLM in its system prompt to "ask a human before executing tools," the workflow breaks when context is truncated, the worker restarts, or a retry loop executes. Prompt rules fail under load. Human review of LLM output before workflow continuation succeeds when the runtime halts execution, writes a pending record to an external store, and refuses to re-enter the execution branch until an authenticated identity sets that record to an approved status.

The fix: make the gate a state transition, not a prompt instruction

Language models do not possess execution state. When you ask a model to "pause and confirm with the user," it emits text indicating intent. If your orchestrator relies on the model to evaluate whether it received permission, you invite hallucinated consensus. Under network retries, context compression, or system prompt edits, models often decide that permission was granted in a prior turn.

State belongs outside the model. To enforce genuine human review of LLM output before workflow continuation, the gate must exist outside the model context. The approval step is an immutable record in a database, defined by an identifier and an explicit state machine: pending, approved, denied, or expired.

The continuation condition in your agent runtime becomes a deterministic boolean check against this external record:

  • If status == "approved", proceed to execute the downstream action.
  • If status == "denied", branch immediately to the compensating cleanup path or notify the caller.
  • If status == "pending", halt execution, suspend the process, and register a listener or poll schedule.
  • If status == "expired", tear down any transient allocations and abort the run.

It halts immediately. AgentDraft lets an agent pause any consequential action for human sign-off: it opens an approval request carrying a one-line summary and a JSON evidence payload, a person approves or denies it in the dashboard with an optional note, and the agent reads the outcome back. The gated action does not have to be one AgentDraft performs—a deploy, a migration, or a refund is gated the same way. Every transition lands in the append-only audit trail and fires an approval.* webhook.

This design eliminates unapproved emails and corrupted calendar states. When an agent acts in the real world, the authorization boundary cannot be negotiated inside the conversation buffer. Externalizing the gate converts a prompt constraint into a testable storage constraint.

What the review record must carry before the workflow can continue

Reviewers need context. Human review of LLM output before workflow continuation fails when reviewers are asked to evaluate opaque actions. Presenting an operator with a button that says "Approve agent execution" leads to review fatigue and erroneous approvals. The approval record must store the full contextual payload required for an informed decision.

Every review record must persist the following schema:

  • approval_id: A globally unique, non-guessable string identifying this exact request.
  • agent_id: The authenticated identity of the agent requesting authorization.
  • action_class: The operational domain of the gated tool call (for example, email.send, calendar.commit, or billing.refund).
  • summary: A single-line human-readable description of what will execute.
  • evidence_payload: A structured JSON payload detailing the proposed change (such as an email body diff, recipient addresses, or calendar bucket allocations).
  • status: The current state of the request (pending, approved, denied, expired).
  • reviewer_id: The verified user ID of the human who resolved the ticket.
  • reviewer_note: An optional string explaining the decision.

The evidence payload is the core requirement. If an agent drafts an email via a per-agent mailbox, the reviewer must inspect the exact subject line, the plain text body, any CC recipients, and any tracking headers. If the agent proposes scheduling an appointment, the payload must contain the precise time buckets, duration, attendee identities, and priority rankings.

Status transitions must be terminal and single-writer. Once an approval record moves from pending to approved or denied, it cannot transition again. If two administrators review the same ticket concurrently, the first write sets the terminal state. The second write returns an HTTP 409 Conflict, conforming to the conflict semantics defined in RFC 9110 (HTTP Semantics). This prevents conflicting downstream tasks from launching off a single gate.

Avoid storing pending approval states in the agent's working memory or local thread memory. If your worker process crashes, restarts during a deployment, or gets evicted by an orchestrator, in-memory approvals vanish. The agent then spins up, runs its loop again, and issues duplicate approval requests. The state belongs in a persistent, shared datastore accessible across workers.

Idempotency: why your gate fires twice and how to stop it

A frequent failure mode in human-in-the-loop workflows is double execution caused by network timeouts. An agent requests an approval, waits 60 seconds, encounters a network drop or gateway timeout, and retries the HTTP request. If the endpoint is non-idempotent, the server creates a second approval record. When an operator approves both items in the dashboard, the downstream tool executes twice.

Idempotency prevents this. To eliminate duplicate requests, derive a deterministic idempotency key from the intent: the agent ID, the action class, and a cryptographic digest of the evidence payload. When calculating this digest, JSON fields must be ordered deterministically so equivalent objects produce identical hashes. Follow the standard specification in RFC 8785 (JSON Canonicalization Scheme) before computing your SHA-256 hash. If an agent sends a POST request with an idempotency key that matches an active or completed record, the endpoint returns the existing record with an HTTP 200 instead of creating a duplicate with an HTTP 201.

Idempotency is equally vital when receiving notifications. When an approval status changes, webhook deliveries follow at-least-once transport semantics. Your receiving endpoint must deduplicate events by event ID and verify authenticity before releasing a workflow lock.

AgentDraft signs every webhook delivery with an X-AgentDraft-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256> header computed over <t>. followed by the raw request body, using the workspace's signing secret. The receiver verifies it; AgentDraft recommends that receivers reject any timestamp more than 300 seconds (5 minutes) from the current time to block replays, and its documented reference verifier does so. Signed webhook delivery is available on every plan, including the free Developer tier.

Review the reference Python verification below for processing signed payloads safely at your inbound webhook endpoint:

import hmac
import hashlib
import time

def verify_agentdraft_webhook(raw_body: bytes, signature_header: str, secret: str) -> bool:
    pairs = dict(item.split("=", 1) for item in signature_header.split(","))
    timestamp = int(pairs.get("t", 0))
    expected_v1 = pairs.get("v1", "")

    # Reject deliveries outside the 300-second drift tolerance
    if abs(time.time() - timestamp) > 300:
        return False

    payload_to_sign = f"{timestamp}.".encode("utf-8") + raw_body
    computed_v1 = hmac.new(
        secret.encode("utf-8"),
        payload_to_sign,
        hashlib.sha256
    ).hexdigest()

    return hmac.compare_digest(computed_v1, expected_v1)

Your receiver must enforce this check before updating execution state, ensuring that spoofed network traffic cannot force workflow resumption.

LLM output validation: what to check before the human ever sees it

Human attention is an expensive compute resource. Do not route raw model output to an operator if a standard schema parser can reject it. Implement a strict, deterministic pre-validation step between raw LLM generation and gate creation.

Separate model generation failures into two distinct classes:

  • Malformed output occurs when the model returns unparseable JSON, omits mandatory fields, breaches length limits, or produces invalid types. These failures do not route to human review; they trigger an immediate internal retry against the LLM with a schema correction hint.
  • Consequential output occurs when the model returns valid, structurally conformant data that initiates an operational side effect, such as writing to a calendar or sending a client email. This class halts automatic execution and routes to the gate.

Your pre-checks should enforce invariants using standard application logic without secondary LLM evaluations. Check that email addresses match permitted organizational domains, ensure scheduling timestamps fall strictly within business hours and in the future, check financial values against static maximum ceilings, and scan payload strings for sensitive internal identifiers. If any heuristic fails, reject the payload automatically and feed the validation error back to the generator.

Keep these validation rules deterministic, code-based, and version-controlled. If you use an LLM-as-a-judge prompt to validate your primary LLM output, you introduce nondeterminism into your security gate. Log the failure. When an automated heuristic rejects an agent's proposal, log the rejection in your application logs to preserve calibration metrics.

Agentic workflow gates: where in the flow the pause belongs

Where you place agentic workflow gates determines the stability of your runtime. Placing the gate at the beginning of an agent's run, such as asking an operator to approve a broad high-level plan, is ineffective. The model's plan frequently diverges from the actual arguments passed to downstream tools three steps later. The gate belongs immediately before the irreversible call, evaluating the precise final payload.

In multi-step coordination, gate the commit phase, not the exploratory phase. Consider calendar management. Letting an agent place a temporary hold on a time slot allows it to verify availability, assemble context, and construct an approval request without losing the resource to competing agents. Once the human inspects the proposed meeting details, the agent issues the final commit.

AgentDraft coordinates holds and commits through a priority-aware conflict engine so multiple agents can act on the same calendar without double-booking. AgentDraft holds expire after 30 seconds by default. A committed booking can be bumped by a higher-priority agent only within a 30-second window, after which it is frozen. A single booking is capped at 480 minutes (8 hours), buffers included, and at 99 five-minute buckets per request; a longer request is rejected with 422 booking_too_long. The cap is service-wide; no workspace or plan setting raises it.

The boundary is rigid. Because atomic DynamoDB operations govern bucket allocation, this boundary is rigid. The 99-bucket ceiling directly matches the underlying limits of AWS distributed storage, detailed in the Amazon DynamoDB TransactWriteItems documentation. The conflict engine is race-free at the storage layer, not in application code. AgentDraft's conflict engine locks time in five-minute buckets: a 30-minute booking writes 6 bucket rows, and the 480-minute maximum booking spans 96.

This mechanism requires an explicit design decision regarding hold timeouts:

  • Synchronous review relies on an active operator who evaluates notifications within the default 30-second hold window, committing the allocation before TTL expiry.
  • A two-phase commit treats the initial hold as validation, releasing the hold after inspection. Human approval authorizes the target schedule, after which the agent issues a fresh hold-and-commit sequence. If another agent secured the slot during the pause, the runtime flags the conflict for rescheduling.

Avoid placing gates around read-only tools such as checking availability or querying a customer profile. Gating safe operations trains operators to approve requests mechanically without reading the parameters.

Who approves, and what the reviewer is allowed to do

Establishing human authorization requires an authenticated identity boundary. Approvals are decided in the AgentDraft dashboard. AgentDraft emails the workspace owner a notification linking to the queue, but the decision itself is made signed in—there are deliberately no approve-from-email links, because an unauthenticated one-click approve is an attack surface. Slack, Discord, Teams, SMS and push delivery are not available today.

Humans sign in to the dashboard with a passkey (WebAuthn), with a magic link as the bootstrap and recovery path. Passkey credentials follow the W3C Web Authentication specification. This ensures an approval event is tied to an authenticated session instead of an interceptable token in an email body.

The requesting agent decides for itself when to open an approval request. AgentDraft does not yet provide a policy engine that auto-requires approval by action class, amount threshold, or role, and there are no escalation chains or multi-approver quorums—a single workspace human resolves each request. If your deployment requires hierarchical approval rules, implement those checks inside your orchestration logic before invoking the approval endpoint.

Enforce the principle of least privilege using scoped credentials. AgentDraft agents authenticate with bearer API keys prefixed avs_live_, stored argon2id-hashed per RFC 9106. Each key carries explicit scopes (availability:read, bookings:read, bookings:write, mailbox:read, mailbox:write, rules:read, approvals:request); new keys default to availability:read and bookings:write, and approvals:request must be granted per key.

By default, agents cannot generate approval tickets unless explicitly provisioned with the approvals:request scope. Limiting key capabilities prevents an agent running in an untrusted sandbox from manipulating unauthorized governance queues.

The audit record: proving what the agent did and who said yes

Human-in-the-loop controls are incomplete without verifiable history. When an automated process alters external state, you must be able to prove which user approved the action, at what time, and under what specific evidence payload. AgentDraft records state-changing agent actions in an append-only audit trail.

Auditors evaluate deterministic records linking the authorization ID to the final action payload:

{
  "audit_id": "aud_01J9X8G5K3W8D6Q4M1N7P0R2S3",
  "workspace_id": "ws_live_77a9b2c8",
  "agent_id": "ag_01J9X8C2B3",
  "approval_id": "appr_01J9X8D4F5",
  "event_type": "approval.resolved",
  "timestamp": "2026-10-26T14:32:05.112Z",
  "actor": {
    "type": "human",
    "user_id": "usr_99a8b7c6",
    "auth_method": "webauthn"
  },
  "action_class": "calendar.commit",
  "decision": "approved",
  "reviewer_note": "Verified client availability via phone call.",
  "evidence_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}

Every state-changing operation emits an audit record. Audit retention is per-tier and enforced on read as well as on write, so the retention claim holds even though deletion is lazy. On the free Developer tier, retention is 7 days. Plan your tier selection around your team's compliance and incident response cycles, ensuring records remain accessible throughout retrospective windows.

AgentDraft does not hold formal compliance certifications (SOC 2, HIPAA, ISO 27001, etc.). It does keep an append-only audit trail. This trail operates as an immutable log structured specifically for tracing agentic execution paths, avoiding unparsed raw log streams.

Wiring it up: framework notes and the evaluation checklist

Whether you orchestrate through LangChain, CrewAI, AutoGen, the OpenAI Agents SDK, n8n, or Model Context Protocol (MCP) servers, integrating approval gates follows a standard pattern: interrupt execution, store the transaction identifier, and register a resumption callback.

If you build against the Model Context Protocol specification, expose the gate through two decoupled tools:

  1. request_action_approval: Accepts the human-readable summary, action class, and evidence payload. Writes the record to the persistence layer and returns an approval_id.
  2. check_approval_status: Accepts an approval_id and returns the current terminal state (pending, approved, denied, expired).

Decouple the two calls. Do not merge these operations into a single blocking tool call. A blocking call forces your agent framework to hold open a synchronous execution loop, consuming worker memory while waiting hours for human intervention. Decoupling the tools allows the agent process to terminate gracefully while pending, freeing system resources until an inbound webhook or scheduled poller resumes the run.

Before launching an agent with human-in-the-loop gates into production, test your system against this checklist:

  • If the agent runtime process crashes while waiting for human input, does it resume safely upon reboot without opening a second approval request?
  • Are approval requests uniquely identified by an idempotency key derived from the evidence payload hash?
  • Does your inbound webhook handler verify the X-AgentDraft-Signature header and reject requests with timestamp drift exceeding 300 seconds?
  • Does a denied response trigger a clean compensation flow without leaving transient holds or database locks stranded?
  • Are agent API keys restricted so that the approvals:request scope is only granted to models executing gated operations?
  • Is the gated operational payload stored immutably, ensuring the action executed matches the exact content shown to the human reviewer?

AgentDraft plans are per workspace. Developer (free): 3 agents, 1 mailbox, 50 bookings a month. Individual: 3 agents, 1 mailbox, 500 bookings a month. Team: unlimited agents, 5 mailboxes, 5 seats, 10,000 bookings a month. Scale: unlimited agents, 25 mailboxes, 25 seats, 100,000 bookings a month. Enterprise: custom, with no mailbox cap. Freeform outbound email (not tied to a booking) starts at Team.

AgentDraft gives AI agents per-agent email inboxes with inbound webhooks, replies, and audit evidence. The number of inboxes a workspace can run at once is set by plan: 1 on Developer and Individual, 5 on Team, 25 on Scale, unlimited on Enterprise. AgentDraft's free Developer tier includes one mailbox, which the workspace owner can move between its 3 agents; giving several agents their own inbox at the same time needs Team (5 mailboxes) or above.

For calendar coordination across your agents, note that AgentDraft syncs Google Calendar today; Microsoft 365 / Outlook calendar sync is planned, not yet shipped. AgentDraft is a proprietary hosted API; it is not open source and is not offered as a self-hosted or on-premise product. Enterprise SSO (SAML/SCIM via WorkOS) is on the AgentDraft roadmap and not available today; agents authenticate with bearer API keys and humans with passkeys. AgentDraft publishes a public conflict-resolution benchmark for its own engine; it does not provide load-testing or throughput stress-testing tools for your architecture.

Frequently asked questions

What solution supports human review of llm output before continuing workflow?

Durable human review requires an external persistence layer that models approval requests as explicit database records. The agent emits an approval request containing a structured evidence payload, pauses execution, and waits for a human to approve or deny the action via an authenticated interface. The workflow continues only when the record transitions to an approved state.

How do I stop my agent from continuing the workflow before a human approves?

Do not rely on system prompt instructions to halt the model. Terminate or suspend the agent execution thread immediately after dispatching the approval request. Configure your execution engine to resume solely upon receiving a verified status update, such as a cryptographically signed webhook or an explicit poll returning a terminal status of approved.

Should the approval gate go before the plan or before the action?

The gate belongs immediately before the irreversible action, not before the preliminary plan. Reviewing high-level plans leaves room for hallucinations during execution. Presenting the reviewer with the final tool arguments, such as the exact recipient, message body, or calendar bucket allocation, ensures that what the human reviews matches what executes.

What happens to a pending approval if the agent process restarts?

If approvals are stored in an external datastore instead of local memory, a process restart causes no data loss. When the worker comes back online, it reads the pending approval ID from its run state, queries the API for status, and resumes or waits accordingly. Idempotency keys prevent the restarted agent from generating duplicate tickets.

How long should an approval request stay open before it expires?

Expiration windows depend on resource volatility. For ephemeral state locks like calendar holds, AgentDraft holds expire after 30 seconds by default, requiring short review windows or an asynchronous two-phase hold pattern. For actions lacking real-time contention, such as customer email drafts, TTLs between 24 and 48 hours are standard before invalidating the run.

The gate is a row, not a request to the model

Double-booked appointments, unauthorized outbound messages, and untraceable database mutations stem from handling operational safety inside LLM context windows. Natural language instructions are not execution boundaries. To build resilient agentic systems, treat approval gates as state transitions backed by an immutable database row, enforce idempotency across your endpoints, verify incoming webhooks, and record every decision in an append-only audit trail.

AgentDraft's Developer tier is free with no card required: 1 seat, 3 agents, 1 mailbox, 1 connected calendar, 50 bookings a month, and 7-day audit retention. Developer-tier outbound email must reply within a booking thread and is capped at 5 sends per agent per day. Signed webhook delivery is included on the Developer tier, so a free workspace is a working sandbox for testing webhook handlers. View tier details and register at https://agentdraft.io/pricing.