An AI agent calls an ERP to create a purchase request. The ERP commits, but its response is lost. The agent sees a timeout. Should it retry?
Doing nothing may leave the workflow stuck even though the request exists. Retrying with a new identity may create a duplicate. Asking the model to decide delegates a consequential reliability choice to a component that lacks the target’s committed state.
This is the transaction-safety problem for tool-using agents. Ordinary distributed-systems ambiguity can be more expensive than model variability. Any agent that sends messages, changes records, provisions access, reserves stock, or initiates financial activity needs deterministic controls around its reasoning: durable operation identity, idempotent tools, error-aware retries, checkpoints, and recovery paths.
A timeout means unknown
Separate three write outcomes:
- confirmed success: the target returns a stable identifier for a committed effect;
- confirmed rejection: validation or policy proves nothing was applied;
- unknown outcome: the caller cannot establish whether the target committed.
Connection reset, timeout, worker crash, and lost acknowledgement usually belong in the third category. The orchestrator should preserve the same operation identity and query or retry according to the tool contract. It must not ask the model to regenerate the call: regeneration can change parameters and turn one ambiguous operation into a new one.
HTTP defines an idempotent method as one whose intended server effect is the same for repeated identical requests as for one. Agent tools need an equivalent contract even when they use POST, RPC, messaging, or a legacy connector.
Create one durable ID per business intent
Generate the operation ID before the first external write and persist it with workflow state. Reuse it across retries, queue redelivery, restarts, failover, and manual resume.
A useful scope combines tenant, workflow instance, step, business object, and requested effect. Do not derive the key solely from the payload. Two legitimate orders may have identical parameters, while harmless metadata may differ between retries. The caller should state intent rather than forcing the target to infer duplication.
The tool stores the key, normalized intent, status, result identifier, and retention deadline with the business change where possible. On repetition:
- same key and same intent returns the original result;
- same key with different intent is rejected;
- an in-progress key returns a stable pending state;
- an expired key follows a documented rule.
Retention must exceed the maximum replay window. A key that expires after an hour cannot protect yesterday’s restored workflow.
Put retry and recovery semantics in the tool contract
The model may choose an approved tool. The orchestration layer decides whether and how it can retry.
| Contract field | Required meaning |
|---|---|
| Effect class | Read, reversible, compensable, or irreversible |
| Preconditions | Expected resource version, status, or ownership |
| Success receipt | Stable target ID, version, timestamp, and effect |
| Retryable failures | Named timeouts, rate limits, and transient errors |
| Status lookup | How an ambiguous outcome is reconciled |
| Compensation | Action, required inputs, approval, and deadline |
| Point of no return | Stage after which automatic reversal is unsafe |
Retries need deadlines, bounded attempts, backoff, and jitter—but especially classification. Retrying an authorization denial or malformed request only amplifies load. Retrying a non-idempotent write after an unknown outcome can create damage. The broader schema, permission, and validation layer is covered in enterprise tool-calling patterns.
Use durable state and explicit compensation
A write step can move through PLANNED, VALIDATED, APPROVED, DISPATCHED, and COMMITTED, with separate branches for OUTCOME_UNKNOWN, RECONCILING, and MANUAL_REVIEW. Persist the dispatched state before the call and the target receipt after confirmation. If a worker dies in between, reconciliation queries the target by operation key.
When a later step fails, a distributed workflow may need compensation. If an agent reserved inventory and opened a case before credit validation failed, recovery may release the reservation and close the case. A sent message cannot be unsent; its compensation may be a correction or human follow-up.
Microsoft’s compensating-transaction pattern notes that compensation is application-specific, may not restore the exact original state, and can itself fail. Record the information needed to compensate as each forward step commits. Put irreversible legal, physical, or financial actions last, behind deterministic checks and required human approval.
Reconcile and test business effects
Exactly-once execution across every target is usually an illusion. Design for convergence: delivery may repeat, the operation ID remains stable, the target deduplicates, ambiguous outcomes are queried, and periodic reconciliation compares workflow state with target state.
Inject failures after dispatch, after target commit, before receipt storage, during queue redelivery, and halfway through compensation. Test an idempotency key reused with different parameters and a replay after retention expiry. Assert business outcomes—one case, one message, one reservation—not merely successful HTTP responses.
Track duplicate effects, age of unknown outcomes, reconciliation backlog, compensation success, manual repairs, and irreversible actions missing approval. Include them in agent service-level objectives.
VDF AI Networks provides node status, inputs, outputs, timeouts, retries, fallbacks, and execution history. VDF AI Agents adds scoped tools, approval rules, and an audit trail. The enterprise tool must still implement idempotency, status lookup, preconditions, and compensation; VDF carries the stable operation identity and receipts through the governed workflow.
Sources and further reading
- RFC 9110 idempotent method semantics
- AWS Builders’ Library: Safe retries with idempotent APIs
- Microsoft compensating transaction pattern
- Production agent identity and containment
Preparing an agent to write into production systems? Book a VDF AI architecture review to define operation identities, safe retries, reconciliation, compensation, and approval points.