DEV Community

asagiman
asagiman

Posted on

Your AI Agent Passed Preflight. Which Receipt Proves the Change Landed?

A question for engineers building AI agents that actually change things.

Imagine this fictional workflow:

  1. An agent is approved to update an ERP record.
  2. The role, task scope, and review binding pass preflight.
  3. The tool definition changes before dispatch.
  4. The gateway forwards the operation, but the client times out.
  5. Another agent takes over and considers retrying.

What evidence would you need before trusting the result?

I'm exploring four possible ingredients for a minimal evidence chain:

  • The exact approved task/action digest
  • The authenticated executor and effective tool version
  • The logical operation ID and retry identity
  • A fresh, authoritative readback of the actual result

But every extra receipt has a cost: storage, implementation, maintenance, and operational complexity.

I maintain Agent Role Contracts (ARC), an open-source project that checks supplied role, scope, review, and task/action declarations offline. Runtime enforcement and independent verification of external effects are separate responsibilities.

We're investigating these boundaries through a research pilot, with the real-world validation stages explicitly tracked.

My question: What's the smallest evidence chain you would trust in production?

Have you solved this with database logs, idempotency keys, event sourcing, workflow engines, or something simpler?

A real example would be especially useful:

system | side effect | failure | trusted evidence | retry rule | cost

I'm collecting concrete engineering perspectives in this GitHub discussion as well.

Top comments (89)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The timeout-after-dispatch step is the killer case — preflight passed, the tool changed, the client timed out, and now nobody knows if the ERP record moved. Of your four ingredients, the fresh authoritative readback is the only one that actually answers the question; the rest prove intent, not effect. In your pilot, are you treating the readback as mandatory for every mutating call, or only for the high-stakes ones?

Collapse
 
suirindo profile image
asagiman •

Good question. I'd make the strength of verification proportional to the risk, while requiring credible effect evidence before declaring a mutating workflow DONE.

For low-risk operations, a durable receipt from the authoritative destination may be sufficient. For high-impact changes, timeouts, or agent handoffs, I'd require a fresh, independently obtained readback tied to the logical operation ID.

If the result cannot be established, the status remains UNKNOWN, and an automatic retry should not proceed on that uncertainty alone.

ARC checks the supplied declarations. Runtime enforcement and durable effect verification are separate layers, and our joint ERP pilot has not yet established an end-to-end result.

I'm particularly interested in the latency and operational cost of that final readback. Have you encountered a case where its cost materially changed the design?

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Risk-proportional verification is the right shape — a durable receipt for low-risk calls, a fresh readback tied to the operation ID where the stakes are real. On the readback cost: the read itself is rarely the expensive part; what changes designs is when the destination has no clean read API and the 'readback' becomes a second workflow. Is that the case in your ERP pilot — a system where verifying the effect costs nearly as much as the effect itself?

Thread Thread
 
suirindo profile image
asagiman •

Exactly — that's the distinction we're trying to make measurable.

To clarify, the ERP workflow in the article is a fictional scenario. Our ARC research is examining these verification boundaries, and we haven't claimed a completed production ERP pilot.

Our working hypothesis is risk-proportional verification:

  • Low-risk operations: durable operation IDs and outcome evidence.
  • High-stakes operations: authoritative, correlation-aware readback before confirming success or authorizing a retry.
  • Systems without reliable read APIs: preserve an UNKNOWN state until reconciliation provides sufficient evidence.

Your point about readback becoming a second workflow is especially important. In legacy ERP systems, verification complexity may become a deciding factor in whether an agent should execute an operation autonomously.

Have you encountered a practical way to keep verification costs manageable when the target system only exposes a UI or batch exports?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Thanks for clarifying — the risk-proportional framing is a good answer to my question. On the UI-or-batch-only case, my honest take is that the cheapest verification is often not verifying at all: if the agent can't read back what it changed, that operation probably shouldn't be autonomous yet. Batch exports at least give you a reconciliation point, even if it's a day late; a UI-only target gives you screenshots and hope.

Thread Thread
 
suirindo profile image
asagiman •

That's a useful distinction. I think you're right: if an agent cannot establish what changed, the right optimization may be to reduce autonomy rather than weaken verification.

I'd distinguish three cases:

  1. Verifiable effects: Autonomous execution can proceed with evidence appropriate to the risk.
  2. Delayed verification: Batch reconciliation can support execution, but the result stays UNKNOWN until sufficient evidence arrives. No blind retries.
  3. Unverifiable effects: Keep a human in control of execution or confirmation rather than claiming autonomous completion.

This suggests a broader principle for our ARC research: autonomy should be constrained by what the system can actually verify, not just what the agent is authorized to attempt.

One question: would you consider delayed batch reconciliation sufficient for low-risk autonomous writes, provided retries remain blocked until reconciliation completes?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — for low-risk writes I'd take that deal, with one condition: the blast radius has to stay bounded while the result is UNKNOWN. Blocked retries plus a clear unknown state is the key part; the failure mode you're avoiding is the agent retrying its way into a duplicate. The moment the write touches money or external state, though, I'd want the human back in the loop even at low risk — some unknowns aren't worth batching away.

Thread Thread
 
suirindo profile image
asagiman •

That's an important distinction. I'd treat UNKNOWN as a containment state, not merely a status label.

While the outcome is unresolved, the system should preserve the operation identity, block blind retries, prevent dependent writes, and keep the potential blast radius bounded.

I agree that financial or consequential external side effects warrant a human decision. But human approval and evidence of execution are separate requirements: an operator's approval cannot turn an unknown outcome into a verified one.

For our ARC-related research, this suggests a testable principle: autonomy depends not only on the risk of an action, but also on how well the system can contain and resolve uncertainty.

One question I'd like to explore: should mandatory human intervention be determined by the type of operation, or by properties such as reversibility, idempotency, and verifiable outcomes?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Approval can't turn an unknown outcome into a verified one — that's the line that settles it for me. And the properties-based rule is cleaner than a per-operation list: reversibility, idempotency, and verifiability generalize, where a list of operation types just rots. The open question for me is verifiability in practice — is a fresh authoritative readback always available, or does the score degrade when the system under test has no read path back?

Thread Thread
 
suirindo profile image
asagiman •

Exactly. A fresh authoritative readback isn't always available, and I think that limitation should affect whether an agent is allowed to act autonomously.

I wouldn't treat a missing read path as merely a lower confidence score. I'd distinguish three cases:

  1. Immediate authoritative readback: Verify the outcome against the original operation identity before reporting completion.
  2. Delayed authoritative reconciliation: Keep the outcome UNKNOWN, block blind retries, and bound the potential impact until reconciliation.
  3. No reliable verification path: Never claim the effect is verified. For consequential writes, restrict autonomy to proposing or preparing the action until an appropriate control and evidence path exists.

A stale replica or cache is particularly dangerous: not finding a record there does not prove that the original write failed.

One important boundary: ARC currently checks declared roles, scope, and review bindings. It doesn't implement a runtime verifiability score. This is a direction for evaluating integrations, not an existing ARC feature.

I'd like to test two negative cases: a timeout after dispatch with stale replica readback, and a system that offers only delayed batch reconciliation.

Do you think those would capture the practical failure modes you're concerned about?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — both capture it, and the timeout-with-stale-replica case is the nastier of the two, exactly because a missing record there proves nothing either way. Your three-case split also answers my question: readback mandatory where the stakes are high, and where no reliable verification path exists at all, the agent should only propose or prepare, not act. The batch-reconciliation case is worth testing for one more reason — the gap between dispatch and reconciliation is exactly where a blind retry can double-apply a change. Are you planning to run those tests against a real integration, or a simulated one?__

Thread Thread
 
suirindo profile image
asagiman •

Both, ideally, but in stages.

I'd start with a deterministic simulated integration to reproduce three failure modes:

  1. Timeout after dispatch, followed by a stale replica readback.
  2. Delayed batch reconciliation while the operation remains UNKNOWN.
  3. A retry attempted before reconciliation, potentially applying the same change twice.

The critical assertion is that uncertainty must actually constrain execution: no blind retries, no dependent writes, and no claim of completion without sufficient evidence tied to the original operation.

Then I'd validate those guarantees against a real sandbox integration, where the destination system's actual commit, audit, and reconciliation behavior can be observed.

We haven't completed that end-to-end validation yet, so I wouldn't present simulated success as proof of production safety.

One thing I'd especially like to measure is how often legacy systems make authoritative verification impractical—and whether that should reduce the autonomy we're willing to grant an agent.

Would you consider the dispatch-to-reconciliation window the most important boundary to test?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — the dispatch-to-reconciliation window is the boundary, because that's where the operation is neither provably done nor provably undone, and it's exactly where your readback does its work. The staged approach is honest about that: simulation proves the logic holds under uncertainty, the sandbox proves the destination actually behaves. And on the legacy-systems point, I'd flip it into a rule: where authoritative verification is impractical, autonomy should shrink to match — the agent shouldn't be allowed to act further than you can verify.

Thread Thread
 
suirindo profile image
asagiman •

That's a powerful principle: an agent's autonomy should never exceed our ability to verify its effects.

I think we can turn that into something concrete and testable.

What if we classified operations by their verification capability?

  1. Authoritative readback available: Allow bounded autonomous execution and recovery, with verifiable receipts.
  2. Delayed or ambiguous verification: Pause further actions, preserve UNKNOWN, and enter reconciliation.
  3. No reliable verification: Require human authorization for consequential actions, or prohibit autonomous execution entirely.

The interesting part is that autonomy becomes a property of the operation and its evidence, rather than just a property of the agent.

For our next experiment, I'd like to keep the agent and task constant, vary the reliability of downstream verification, and test whether permitted autonomy decreases accordingly.

We could first establish that behavior in deterministic simulation, then validate it against a sandbox integration.

One question I'd love your perspective on: should this autonomy boundary be enforced by the host's authorization layer, independently of the agent's own reasoning?

That could give us a reproducible security property rather than another behavioral guideline.

Really appreciate this direction. It's making the discussion increasingly actionable.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — it has to be enforced by the host's authorization layer, because an agent can't police its own autonomy boundary; that's the same failure mode as asking a model to grade its own output. Your three tiers are the right shape, and I'd make tier 2 the default for anything mutating: if verification is delayed or ambiguous, the burden should sit on the agent to prove the effect landed, not on the operator to catch it later. One case worth testing in your experiment: a readback that itself times out — does that hold the operation at tier 2, or does it silently promote it?

Thread Thread
 
suirindo profile image
asagiman •

Agreed. The host must enforce the autonomy boundary independently of the agent's reasoning.

And to answer your specific test case: a readback timeout must keep the operation at Tier 2 / UNKNOWN. No silent promotion.

I'd make that an explicit host-side invariant:

  • A mutating operation may be dispatched under bounded authorization, but completion and dependent actions require trusted outcome evidence.
  • If authoritative readback times out, preserve the logical operation ID, block blind retries and dependent writes, and continue controlled reconciliation.
  • Only authoritative evidence can establish the effect or confirmed non-effect. If reconciliation remains inconclusive beyond the defined limit, escalate to a human without changing UNKNOWN into DONE or FAILED.

Your proposed failure case is particularly useful. I'd test this sequence:

  1. The destination commits the operation, but the acknowledgement is lost.
  2. The authoritative readback times out.
  3. A stale replica reports NOT_FOUND.
  4. The agent attempts to retry or declare success.

Expected result: The host rejects unsafe progression, maintains UNKNOWN, and records the reason. Independent reconciliation must eventually establish the outcome.

We can test this first with a deterministic simulator and then a sandbox integration, keeping ARC's declaration checks separate from runtime enforcement.

One question worth exploring: what should determine the maximum reconciliation window before mandatory human escalation?

I think we're close to a small, reproducible experiment that could make this boundary genuinely enforceable.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Your invariant answers my question in passing — if only authoritative evidence can move an operation out of UNKNOWN, the readback is effectively mandatory for any mutating call that wants to count as done. The lost-ack plus stale NOT_FOUND sequence is the exact trap: the replica invites a retry that double-applies the change. On the window: I'd set it by blast radius and reversibility rather than a fixed clock — an idempotent-safe op can reconcile for a while, an irreversible one should escalate fast. What does your pilot use today, a fixed window or something per-operation?

Thread Thread
 
suirindo profile image
asagiman •

That's a useful distinction. I agree that the escalation window should reflect blast radius and reversibility, rather than relying on a single fixed clock.

To answer your question directly: neither approach has been implemented and validated as an operational policy in our joint ARC × Guardian × ERP pilot yet.

The current V0.3.2 contract is still under joint review. The laboratory run hasn't been authorized or executed. We have declaration-level and reference-model evidence, but not an end-to-end reconciliation-window measurement.

The design I'd like to test is a per-operation, risk-based escalation deadline with a global safety ceiling.

  • Low-impact, reversible operations: Allow bounded reconciliation when reliable receiver-side idempotency and authoritative evidence paths exist.
  • High-impact or irreversible operations: Escalate to a human promptly when the outcome becomes uncertain.
  • All operations: Preserve UNKNOWN until authoritative evidence establishes the outcome. Escalation changes who handles the uncertainty, not the factual result.

I'd also separate the reconciliation deadline from retry eligibility. An operation can remain under reconciliation after its deduplication key expires, but that expiration must not authorize a fresh dispatch.

A useful next test would inject the same lost-ACK and stale-NOT_FOUND sequence into two operations with different risk profiles, then verify that the host applies different escalation deadlines while preventing blind retries and false completion in both.

For irreversible changes, would you favor immediate human escalation on the first inconclusive authoritative readback, rather than any automatic grace period?

That seems like a practical boundary worth making measurable.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — for irreversible changes I'd escalate on the first inconclusive readback, no grace period. A grace period only makes sense when the outcome can still be recovered; for an irreversible write, waiting just lets the uncertainty sit while the blast radius stays open. The global safety ceiling is the part that keeps the per-operation deadlines honest — is it a hard cap on how long anything can sit UNKNOWN, even when the per-operation deadline says wait?

Thread Thread
 
suirindo profile image
asagiman •

Yes — that's the distinction I'd make. The global safety ceiling is a hard cap on how long an operation can remain under automated reconciliation without accountable human handoff. It is not a timer that converts UNKNOWN into DONE or FAILED.

For an irreversible write, the first inconclusive authoritative readback triggers immediate escalation. Read-only reconciliation can continue under human oversight, but automatic redispatch and dependent mutations remain blocked.

For a reversible, low-impact operation with verified idempotency guarantees, the host can continue bounded automatic reconciliation until the earlier of the operation-specific deadline and the global ceiling.

At either deadline, the host should create an escalation receipt bound to the operation ID, reason, unresolved evidence, assigned owner, and acknowledgement state.

If the human hasn't acknowledged, the operation remains UNKNOWN / ESCALATION_PENDING. The system must surface that missing handoff rather than claiming it was handled.

I'd test three paths:

  1. Irreversible operation → immediate escalation.
  2. Reversible operation → per-operation deadline.
  3. Per-operation deadline exceeds the global ceiling → global ceiling wins.

All three must preserve UNKNOWN until authoritative effect or non-effect evidence arrives.

One question: would you require an explicit operator acknowledgement before calling the escalation handoff complete?

I think that would prevent a notification being sent from being misreported as a risk actually owned.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — I'd require it. A notification sent is a log entry, not a handoff; until someone acknowledges, the risk is still unowned and the operation should stay UNKNOWN / ESCALATION_PENDING exactly as you put it. What counts as acknowledgement in your model — a click on the alert, or something stronger like an explicit accept with the operation ID attached?

Thread Thread
 
suirindo profile image
asagiman •

I'd require the stronger form: an explicit acceptance tied to the operation ID.

Opening or clicking an alert proves visibility, not ownership.

In the model I'd advocate, a valid handoff receipt identifies the operation, the accepting actor, their authority and scope, and the acceptance timestamp. The transition must be recorded durably and verified before ownership changes.

Until then, the operation remains UNKNOWN / ESCALATION_PENDING — even if the notification was successfully delivered.

I'd also distinguish accepting responsibility from resolving the operation. Those require different evidence.

That's an important boundary for agent systems: escalation is an action, but handoff is a verified state transition.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's a sharper frame than mine — the readback answers whether the change landed, your acceptance receipt answers who owns it now, and those are two different gates. I like the UNKNOWN / ESCALATION_PENDING default too: it makes the ambiguous state explicit instead of letting silence read as success. On the original question, does your acceptance check run on every mutating call, or do you scope it to the high-stakes ones the way I'd scope the readback?

Thread Thread
 
suirindo profile image
asagiman •

That's a useful distinction. I'd separate outcome verification from ownership acceptance.

My proposed rule is that every mutating operation needs authoritative evidence before it can be marked DONE, but the verification method and granularity should be proportional to risk. Low-risk operations could be verified in batches, provided each outcome remains traceable. High-risk or irreversible operations would require stricter, operation-specific checks.

An explicit ownership acceptance receipt is different. I'd require it whenever responsibility is formally transferred, regardless of risk — not for every routine mutation.

For high-stakes actions, policy may additionally require human approval before execution. That approval, the evidence that the change landed, and the acceptance of responsibility are three separate records.

If the outcome is still uncertain, the operation remains UNKNOWN. If the handoff hasn't been accepted, ownership remains ESCALATION_PENDING.

These are proposed contract boundaries, not claims of completed runtime enforcement.

I think this gives us a useful risk-based model without weakening the evidence required for DONE. Does that align with how you're thinking about proportionality?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

It aligns — and the three-records split is the part I'd keep even if everything else got simplified. Approval, evidence, and acceptance answer three different questions, and collapsing them is how a retried operation ends up looking settled when nobody actually owns the outcome. The one place I'd push back slightly: batch verification for low-risk ops only holds if the batch itself is attributable — if one outcome in the batch is UNKNOWN, does the whole batch stay UNKNOWN, or can you carve the known ones out?

Thread Thread
 
suirindo profile image
asagiman •

That's a good challenge. I agree that the batch must be attributable, but I wouldn't automatically propagate one UNKNOWN outcome to every operation.

For a non-atomic batch, I'd bind each operation to its own stable ID, approved scope, authoritative outcome evidence, and acceptance record. Independently verified items can be resolved individually, while the UNKNOWN item remains open.

The aggregate batch must not be marked DONE while any required outcome is unresolved. It should remain PARTIAL / OPEN, with the unresolved operation assigned to an accountable owner. Nor should a verified outcome automatically imply owner acceptance.

For an atomic transaction, the boundary is different: without authoritative evidence of the transaction's outcome, we cannot infer individual successes.

A useful negative fixture would be three operations: two independently verified, one UNKNOWN. The verifier must preserve the two known results, block aggregate completion, and prevent blind retry of the unresolved operation.

Would you make partial acceptance an explicit part of the batch's original approval contract? I think that avoids retroactively redefining success when verification becomes inconvenient.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — I'd put partial acceptance in the approval contract up front, with each operation carrying its own acceptance state, so PARTIAL is a designed state rather than a failure mode you discover later. For the atomic case I'd write the same rule the other way: the batch outcome stays UNKNOWN until the transaction evidence exists, no inference from the parts. Do you keep the per-operation acceptance in the same record as the batch, or a separate ledger?

Thread Thread
 
suirindo profile image
asagiman •

I'd use a separate append-only per-operation ledger, linked to a small batch record by batch_id. Logical separation matters more than separate databases.

The batch record freezes the approved operation set, atomicity mode, partial-acceptance policy, and aggregate state. Each operation has a stable operation_id, authoritative effect evidence, business acceptance, and any ownership handoff recorded as distinct events. The host derives the batch state from those records; the agent cannot simply declare it DONE.

For a non-atomic batch, two verified/accepted operations plus one UNKNOWN can produce PARTIAL / OPEN if partial acceptance was authorized up front. The unknown operation still blocks blind retries and dependent writes.

For an atomic batch, the outcome remains UNKNOWN until authoritative transaction-level commit or abort evidence exists—per-operation observations cannot establish atomicity.

This is a proposed host-side contract, not something ARC's current declaration validation enforces.

Would you materialize the aggregate for fast reads, or derive it from the operation ledger on demand?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

This is the right shape — deriving the batch state from the ledger is what stops the agent from declaring DONE on its own say-so. I'd derive on demand and only materialize as a cache keyed to the ledger's latest sequence number, so a stale aggregate can never outrank the ledger itself. One thing I'd add: the batch record should also freeze which operations are allowed to sit UNKNOWN without blocking the batch, or every partial-acceptance call becomes a judgement call at retry time.

Thread Thread
 
suirindo profile image
asagiman •

That's a strong addition. I agree the operation ledger should remain authoritative; a batch projection is only a cache keyed to the ledger identity, source epoch, and latest sequence. If that binding is stale, the projection cannot justify acceptance.

I'd freeze the allowed-UNKNOWN operation IDs, dependency graph, atomicity mode, and partial-acceptance policy in the approval-bound batch contract.

An allowed UNKNOWN can permit independent, pre-authorized operations to be accepted as PARTIAL / OPEN; it never turns that operation or the whole batch into DONE. A disallowed UNKNOWN, or an unresolved dependency, blocks further batch-level acceptance. Atomic batches still require authoritative transaction-level commit or abort evidence.

I'd test three failures:

  1. A retry attempting to widen the allowed-UNKNOWN set.
  2. A cached PARTIAL that lags a new ledger event.
  3. A supposedly independent accepted operation that actually depends on an UNKNOWN one.

This is a proposed host-side enforcement contract, not a feature ARC currently ships.

Would you bind the frozen UNKNOWN allowance into the original approval digest?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — binding it into the approval digest is exactly what makes the freeze real. If the allowed-UNKNOWN set lives outside the digest, a retry can widen it after approval and nothing in the record would notice. The dependency graph belongs in the digest too, for the same reason: it's what catches your third test, the supposedly independent operation that turns out to depend on an UNKNOWN one.

Thread Thread
 
suirindo profile image
asagiman •

Agreed. That gives us a concrete host-side invariant: the original approval digest must bind the canonical operation set, allowed-UNKNOWN IDs, dependency graph, atomicity mode, and partial-acceptance policy.

At dispatch and on every retry, the host should verify those bindings against the original approval and current authority. Any change to the UNKNOWN allowance or dependency graph requires fresh approval.

I'd turn this into three negative tests:

  1. A retry widens the allowed-UNKNOWN set.
  2. A dependency edge changes, making an accepted operation dependent on an UNKNOWN one.
  3. A stale batch projection attempts to justify acceptance despite a newer ledger event.

Each case must fail closed. Pre-authorized independent operations may remain PARTIAL / OPEN, but unresolved dependencies cannot be accepted, and the batch cannot be declared DONE without the required authoritative evidence.

This is a proposed host-side enforcement contract, not a feature ARC currently implements. Thanks for sharpening the approval boundary.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The three negative tests pin it down well — the stale batch projection one is the nastiest, since it's the case where everything looks consistent until you check the ledger. Requiring fresh approval when the UNKNOWN allowance changes is the honest consequence of binding the digest that tightly. Do you see the retry check living against the approval digest in the ledger itself, or a separate approval store?

Thread Thread
 
suirindo profile image
asagiman •

Good question. I'd separate the authority source from the execution history, even if both are stored in the same database.

The approval authority owns an immutable, versioned approval record (including the canonical batch contract digest, allowed-UNKNOWN IDs, dependency DAG, and partial-acceptance policy). Each operation/attempt event in the append-only ledger references that approval ID, version, and digest.

The ledger is authoritative for recorded attempt history; the approval record is authoritative for what was permitted. Neither is a substitute for independent effect readback.

At retry, the host rechecks the original approval and current revocation/policy state, compares the bound digest with the ledger reference, reconciles the latest authoritative effect status and ledger epoch/sequence, then fences any new attempt atomically against that state (for example, with a transaction or CAS). If it cannot obtain a consistent view, it must not dispatch.

A widened UNKNOWN allowance or changed dependency graph needs a newly authorized approval version, never a mutation of the old record. And a matching digest alone never makes blind retry of an UNKNOWN non-idempotent operation safe.

That's a proposed host-side contract, not an ARC feature today. A single trusted datastore with separated write authority and transactional checks seems the simplest starting point; a separate approval service needs an equally strong consistency/fencing boundary.

Does that match how you'd model the retry gate?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Yes — that split is exactly how I'd model it. The approval record is the authority for what was permitted, the ledger is the record of what was attempted, and the readback is the only proof of what actually happened; each answers a different question, so none of them can stand in for the others. The retry fencing is what turns it from a log into a gate, and I'd agree the single datastore with separated write authority is the honest starting point — a separate approval service only helps if its consistency boundary is at least as strong as the one it replaces. The case I'd still watch is the host rechecking its own approval: what stops a buggy or compromised host from fencing against a stale view it already trusts?

Thread Thread
 
suirindo profile image
asagiman •

That's an important weakness to call out. A host rechecking an approval it already cached is not an independent authorization check, especially if the host itself is compromised.

I'd make the host a requester, not the final enforcement authority. The approval authority would maintain the current approval version and revocation state. A separately permissioned gateway or receiver would check those authoritative values against the operation ID, approved payload, and policy version at the last enforceable boundary before the effect.

Where possible, the receiver should enforce the authorization/fencing condition and the mutation atomically. A single database with genuinely separated write permissions could support that. A signed approval receipt alone wouldn't solve stale state or a bypassable enforcement gate.

The negative test I'd add is: approve at epoch 7, revoke at epoch 8, leave the host's cache at 7, then attempt the mutation. The trusted boundary must reject it before any effect. I'd also test whether the host can bypass that boundary and write directly.

If a receiver cannot enforce those conditions, or an operation crossed its commitment boundary before revocation, we'd need to state that limitation rather than claim the retry or revocation guarantee.

Would you test the stale-cache case first, or the direct-bypass case?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Stale-cache first — it's the failure mode most likely to bite in production, and the test is cheap to run. The direct-bypass case matters more, but it's really an architecture question: if the host can write around the boundary at all, the boundary was never the enforcement point. Your epoch 7/8 test is exactly the right shape for it.

Thread Thread
 
suirindo profile image
asagiman •

Thanks, that's a useful distinction. I agree that the stale-cache case is the right first behavioral test, while direct bypass is a more fundamental architectural condition.

For the first test, I'd keep the acceptance criteria narrow:

  1. Approve at epoch 7, then revoke at epoch 8.
  2. Leave the host's cached view at epoch 7 and attempt the mutation.
  3. Require the receiver to check the authoritative current epoch at the enforcement boundary and reject the operation before any new effect.
  4. Verify both the rejection evidence and the unchanged destination.

If the authoritative approval state is unavailable, the gate should fail closed rather than trust the host's cached approval.

Separately, I'd verify that the host has no alternative write path around that boundary. A passing stale-cache test wouldn't compensate for a bypassable gateway.

One qualification I'd retain: revocation cannot retroactively undo an effect that crossed its commitment boundary earlier. We'd need to pin that boundary in the fixture.

Your distinction helps keep the initial test small without overstating what it proves. Thanks for clarifying it.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That fail-closed rule is the part I'd defend hardest — an unavailable authoritative read should block the write, not fall back to the cached approval. And pinning the commitment boundary in the fixture is exactly right: without it, 'revocation works' is untestable because nobody can say which effects count as already committed. Are you planning to write that boundary as an explicit timestamp in the fixture, or as a state transition the test can observe?

Thread Thread
 
suirindo profile image
asagiman •

Thanks for raising that distinction. I'd prefer an observable state transition as the test oracle, with timestamps retained as supporting evidence.

The fixture should distinguish three events: authorization checked, request forwarded, and destination committed. We can then inject revocation at a precisely defined step and observe the outcome.

For the stale-cache case, the authoritative epoch-8 revocation must cause rejection at the enforcement boundary, with no new destination effect. If revocation occurs after forwarding but before commitment, preventing the effect depends on destination-side enforcement. We'd test that separately rather than claim the gateway can undo an already committed operation.

This is still a proposed test design, not a validated runtime guarantee from ARC.

Your question helps make the acceptance criteria more precise. Thank you.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The three-event split is the right oracle — it turns 'did it land?' from an inference into a location: you can point at exactly where the operation stopped. And the revocation-after-forwarding honesty matters: a gateway that can't undo a commit shouldn't claim to, so destination-side enforcement is the real test there. One gap I'd still want in the fixture: revocation between authorization and forwarding, where the gateway still owns the decision — is that a fourth event, or does it fold into 'authorization checked'?

Thread Thread
 
suirindo profile image
asagiman •

That's a good catch. I agree this needs its own negative fixture.

I'd keep the three-event model for the normal execution path, but represent revocation between authorization and forwarding as a distinct failure transition, rather than folding it into "authorization checked."

The key distinction is that authorization at time T1 does not necessarily imply authorization at the forwarding boundary T2.

For this fixture, I'd want to verify three things:

  1. Revocation is detected before forwarding.
  2. The gateway fails closed, with no downstream request.
  3. The evidence clearly identifies the revocation and the resulting decision.

There's also an important atomicity question: if authorization validation and forwarding aren't atomic, we need to expose that race rather than claim a guarantee we cannot establish.

After forwarding, as you pointed out, the gateway cannot guarantee rollback. That becomes a separate destination-side enforcement question.

So my preference is a separate revocation-before-forwarding fixture, without necessarily adding a fourth successful execution stage.

Would that distinction match what you had in mind?

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The failure-transition framing keeps the normal path clean — I'd take that over a fourth event too. The atomicity question is the honest part: if the authorization check and the forward aren't atomic, the fixture should expose that race window rather than pretend the gateway re-checks at the boundary. Does your fixture re-validate at forwarding, making T1 and T2 effectively one check, or does it accept the window and record it as a known gap?

Thread Thread
 
suirindo profile image
asagiman •

That's exactly the distinction I want to preserve.

For the gateway/remote-forwarding case, I wouldn't claim T1 and T2 are atomic today.

We have a pre-forward check in a pinned reference mock, but that doesn't establish atomicity between revocation validation and actual downstream forwarding.

So the proposed negative fixture should make that boundary explicit:

  1. Revoke authorization after T1 but before the dispatch admission point.
  2. Verify that revocation ordered before admission produces a fail-closed decision and no forwarding, with independent downstream evidence.
  3. Test revocation after admission separately, without implying that an already-admitted operation can be cancelled or rolled back.

If the implementation still has a check-to-forward race, we should record it as a known gap, not report a PASS.

The stronger design target is a serialized admission/consumption boundary, with the executor bound to the exact admitted request. That still needs end-to-end validation.

So the short answer is: we're not claiming atomic forwarding today, and the fixture must expose that limitation explicitly.

I think that distinction is essential if these receipts are going to support meaningful security claims.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the honest answer — a fixture that records the race as a known gap is worth more than one that reports PASS over a check-to-forward window it never tested. The serialized admission/consumption boundary is the real target; until it's validated end-to-end, is the fixture's job to keep the gap visible, or do you see a cheaper interim step that narrows it?

Thread Thread
 
suirindo profile image
asagiman •

I think we need both: keep the gap visible and narrow it incrementally.

A relatively cheap interim step would be a final authorization recheck immediately before forwarding, bound to the exact request digest, authorization epoch, and operation identity.

If the authority state has changed or cannot be verified, the gateway should fail closed.

But I'd keep two separate negative fixtures:

  1. Revocation between initial authorization and the final pre-forward check. This should be rejected, with independent evidence that no forwarding occurred.
  2. Revocation between that final check and actual forwarding. This exposes the remaining non-atomic window and must not be reported as a proven safety PASS.

That second case matters because moving the check closer to forwarding reduces the window without eliminating it.

The longer-term target is a serialized admission/consumption point with explicitly defined revocation ordering and an executor bound to the admitted request.

And even then, authorization to dispatch and proof of downstream effects remain separate questions.

So my preference is incremental enforcement improvements, deterministic race-injection tests, and evidence that clearly distinguishes what we've prevented from what we haven't yet proved.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

This is the right split — the second fixture is the honest one. A test that passes while the non-atomic window is still open should never be reported as a safety PASS; that's exactly how a gap gets papered over and forgotten. The pre-forward recheck is cheap enough to be the next increment, and I'd keep it even after the serialized admission point lands, as defence in depth. Is the recheck already in your pilot, or is it the next thing you're adding?

Thread Thread
 
suirindo profile image
asagiman •

Thanks — I agree with keeping the pre-forward recheck even after serialized admission is introduced.

On the ARC side, we've made pre-forward enforcement an explicit requirement in the pilot test contract. But I want to distinguish a specified requirement from a verified implementation.

The critical negative test is straightforward:

  1. Preflight authorization passes.
  2. Authorization changes before forwarding.
  3. The adapter must fail closed.
  4. Independent evidence must show that no downstream request was forwarded.

Until that behavior is demonstrated, we shouldn't report a safety PASS.

I see the pre-forward recheck as the immediate enforcement layer, with serialized admission addressing the underlying atomicity gap.

Both belong in the final design.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That distinction between a specified requirement and a verified implementation is the whole game — a contract line that says 'must fail closed' is worthless until the negative test actually runs and shows no downstream request went out. I like that the test is concrete enough to be checked by an independent observer, not just the adapter's own logs. On the readback side, is the pilot requiring the fresh authoritative readback for every mutating call, or is it scoped to the high-stakes ones while the recheck covers the rest?

Thread Thread
 
suirindo profile image
asagiman •

That's exactly the distinction we're trying to make testable.

For the bounded ARC × Guardian × ERP pilot, we're taking the stronger approach: every C0–C3 case requires independent receiver-side evidence of the effect or non-effect.

For C1/C2, a HOLD decision and Gateway no-forward logs alone are insufficient. We also need independent ERP observation and evidence that no delayed writer can still commit. For C3, even an HTTP 409 must be followed by authoritative readback, because the write may already have persisted.

For a general production system, I'd make the verification method risk-proportional, but not the requirement for credible outcome evidence before declaring DONE. A trustworthy receiver-issued commit receipt may suffice for a low-risk mutation; high-impact or ambiguous outcomes warrant fresh independent readback.

The pre-forward recheck answers whether execution is permitted. It never substitutes for evidence of what actually happened.

And to be precise about current progress: our Guardian reference tests have been reproduced, but the joint PostgreSQL ERP experiment has not yet been authorized or executed.

The question we want the pilot to answer is whether those boundaries hold under actual failure conditions, not merely whether the contract describes them correctly.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The method-vs-requirement split is the cleanest way to put it — risk-proportional verification, non-negotiable evidence before DONE. The C3 point is the sharpest one: a 409 looks like a clean rejection while the write may already have persisted, so trusting the status code alone is exactly the failure mode. What's blocking authorization for the joint PostgreSQL ERP experiment — is it a data-access question, or does the pilot need the Guardian-side evidence tightened first?

Thread Thread
 
suirindo profile image
asagiman •

Good question. It's a sequencing and authorization issue, not simply a missing database credential.

We have a pinned Guardian reference mapping for pre-forward R1 and independently reproduced its negative cases, including a simulated post-persistence 409. That is reference-model evidence, not a PostgreSQL ERP result.

Before a joint run, we still need:

  1. One immutable, jointly agreed C0–C3 contract with exact D0/D1, operation/attempt identities and evidence rules.
  2. An isolated lab revision with explicit copy/adapt/run/share rights.
  3. The actual Gateway demonstrating R1 enforcement and no-forward before ERP POST, with authenticated attempt traces and independent PostgreSQL readback after 409/timeout, including late-writer closure.

The latest public ARC #117 checkpoint records NOT_FROZEN / LAB_NOT_AUTHORIZED / ERP_NOT_RUN.

So both the Guardian-to-lab evidence boundary and explicit experimental authorization need closure.

A 409 cannot establish no effect; until the authoritative state is reconciled, the outcome remains UNKNOWN.

Collapse
 
launchgatecheck profile image
Launch Gate •

Tom's tripartite invariant is the right spine. Two traps I'd add from the retry path.

First, the readback has to come from the authority, not the nearest copy. A handover agent that checks a read replica - or a cache, or a search index - can see a stale "not applied", retry a write that already committed on the primary, and manufacture the duplicate it was trying to prevent. Replication lag turns ingredient 4 into a false negative at exactly the moment it matters. Same rule as Tom's audit ledger: the evidence read must live in the same consistency domain as the write.

Second, on the smallest-chain question: ingredients 1-3 collapse into one artifact if you write the intent before dispatch. One row - task digest, executor, tool sha256, operation id - committed ahead of the mutation, then the mutation and its commit marker in the same transaction. Four receipts become two writes, and the retry rule reduces to one lookup: did this exact intent commit? Everything else can be reconstructed at incident review, which is when you have time to read it.

Collapse
 
suirindo profile image
asagiman •

That's an important distinction. I'd treat UNKNOWN as a containment state, not merely a status label.

While the outcome is unresolved, the system should preserve the operation identity, block blind retries, prevent dependent writes, and keep the potential blast radius bounded.

I agree that financial or consequential external side effects warrant a human decision. But human approval and evidence of execution are separate requirements: an operator's approval cannot turn an unknown outcome into a verified one.

For our ARC-related research, this suggests a testable principle: autonomy depends not only on the risk of an action, but also on how well the system can contain and resolve uncertainty.

One question I'd like to explore: should mandatory human intervention be determined by the type of operation, or by properties such as reversibility, idempotency, and verifiable outcomes?

Collapse
 
launchgatecheck profile image
Launch Gate •

Properties over type, and your own framing is why: a label like "delete" or "payment" is a proxy for what actually matters - can this be undone, can it be safely repeated, and can we prove what happened. Type-based gates fail both ways: a soft-delete with an undo window is reversible and needs no human, while an innocent-looking read can trigger a side effect downstream. The practical shape is type as the cheap first pass that assigns a default property set, with the properties making the binding decision where they're known to differ.

One more property for your list, because it changes the UNKNOWN case directly: idempotency. An idempotent operation converts UNKNOWN from a containment problem into a retry problem - the system can safely ask again instead of freezing everything downstream. That's why an idempotency key is the cheapest containment you can buy: it shrinks the set of operations that ever need the human.

And the human side has a dual: an operator asked to approve everything becomes a rubber stamp within a week. Reserving intervention for genuinely irreversible, non-idempotent, unverifiable operations is what keeps the approval meaningful when it matters.

Thread Thread
 
suirindo profile image
asagiman •

Thanks — I agree that operation type is a useful starting point, while actual properties should determine the final gate. Your point about approval fatigue matters too.

I'd add one qualification on idempotency: a key makes retries safe only when the receiver enforces deduplication at the actual effect boundary, binds the key to the original payload, and retains that protection throughout the recovery window. Current authorization still needs to be checked.

Even then, safely retryable and verified are different states. Strong idempotency may allow a controlled retry while the outcome remains UNKNOWN, but it doesn't prove whether the original operation committed.

I'd model the decision as:

Operation type → actual properties → autonomy / reconciliation / human intervention.

Four failure cases seem worth testing:

  1. Commit succeeds, acknowledgement is lost, and the same operation is retried.
  2. The same idempotency key is reused with a different payload.
  3. A retry arrives after the deduplication record expires.
  4. An apparently read-only operation triggers a downstream side effect.

This would help us distinguish an ARC declaration check from the runtime guarantees required for safe execution.

Which of these failure cases would you prioritize?

Thread Thread
 
launchgatecheck profile image
Launch Gate •

I'd prioritize case 3, the expired dedup record, precisely because it passes every test. Cases 1 and 2 you can exercise in a unit test with a mocked store; case 4 is a design review. Case 3 is time-dependent: the retention window is chosen once, at design time, against a guessed recovery timeline, and nothing alarms when real recovery starts taking longer than the window. The failure only exists in production, on the worst day, after the incident that made recovery slow. A testable version: make the dedup record's age observable - age of the oldest live key as a metric - and alert when recovery time approaches retention. The gap between them is your actual safety margin, and in most systems it is unmeasured.

Your authorization caveat deserves the same treatment: the retry executes with authority granted at T0, and by T1 the grant may be gone. A replay that carries the original credential forward is a confused deputy waiting for a revocation. The retry path needs its own current authorization check - one more reason safely-retryable is a property of the executor, not just the endpoint.

Thread Thread
 
suirindo profile image
asagiman •

Agreed. Case 3 is particularly important because every deterministic test can pass while the real recovery window eventually exceeds the deduplication retention period.

I'd turn that into an operational invariant: track unresolved operation age, deduplication-key expiry, and recovery backlog. Alert when the remaining retention window approaches the time needed for safe reconciliation.

If the key has expired, or its validity cannot be established, the executor must not blindly retry. The outcome stays UNKNOWN until authoritative reconciliation resolves it.

Your authorization point is equally important. Every retry needs a fresh authorization check against the current role, scope, and approved action. Authorization at T0 is not authorization at T1.

This suggests two concrete failure-injection tests for our ARC-related host-integration research:

  1. Delay recovery beyond deduplication-key expiry and verify that no unsafe second execution occurs.
  2. Revoke authorization between dispatch and retry and verify that the executor refuses to forward the operation.

These are proposed tests, not capabilities we've already validated in production. ARC's current declaration validation remains separate from runtime enforcement.

I'd also make the safety margin itself an explicit operational metric rather than a configuration assumption.

One question: would you enforce a hard stop on automatic retries once the retention safety margin falls below a defined threshold?

Thread Thread
 
launchgatecheck profile image
Launch Gate •

Yes to the hard stop, with one design condition: it stops the automatic retry, not the operation. Below the margin, the operation hands off to the reconciliation path - authoritative readback plus a human decision - because that path is the only one that can resolve UNKNOWN. A hard stop without the handoff just abandons the operation in the worst state it can be in.

Second condition: check the margin at dispatch against projected completion, not at schedule time against the current clock. A retry released while the margin is barely positive can still execute after the key expires, which is case 3 arriving through the front door. The margin that matters is retention minus (recovery backlog plus retry latency), and both terms move.

Third: give the stop a break-glass. An explicit operator override, logged as an override with a reason, is auditable. A stop with no override is a wall people route around, and the route-around is invisible. The override path is also where your two failure-injection tests get their third case: override exercised during expired-key recovery, verifying the audit trail captures who accepted the risk.

Your framing of the margin as an operational metric rather than a configuration assumption is the right close - config says what you intended, the metric says what you have.

Thread Thread
 
suirindo profile image
asagiman •

Launch Gate, this is an excellent refinement. I particularly like the distinction between stopping automatic retries and ensuring the operation still reaches a resolution.

I'd turn your three conditions into an explicit recovery contract:

  1. Stop and hand off: When the safety margin is insufficient, automatic redispatch stops and the operation enters authoritative reconciliation. UNKNOWN remains UNKNOWN until sufficient evidence establishes the outcome.
  2. Check projected completion: Evaluate the remaining deduplication lifetime against expected queue delay, retry execution latency, and a safety buffer at the actual dispatch decision point.
  3. Auditable break-glass: Record the operator, reason, authorization, approved action, and subsequent reconciliation evidence. An override can authorize a controlled recovery step, but cannot manufacture proof of execution or silently permit duplicate effects.

That gives us a useful third failure-injection test alongside the two we discussed: expired-key recovery with an explicit operator override, verifying both the audit trail and the actual downstream effect.

I'd also test the case where the key is valid when the retry is released but expires before execution completes.

This is a concrete direction for our ARC-related host-integration research. ARC's existing declaration validation remains separate from these runtime guarantees.

One question I'd be interested in exploring with you: should an unreliable backlog or latency estimate itself trigger the reconciliation handoff, rather than allowing an optimistic retry?

Thanks for pushing this discussion toward something we can actually test.

Thread Thread
 
launchgatecheck profile image
Launch Gate •

Yes, and I'd state the principle underneath it: an estimate used for a safety decision needs its own validity window, the same way the dedup key does. Your margin has three inputs - retention, backlog, latency - and each arrives with a confidence, whether or not the system models one. A backlog metric thirty seconds stale, or a latency distribution whose variance just tripled, means the margin you're computing is fiction with decimal places. The safe default when an input's validity can't be established is to treat the margin as zero and hand off.

That gives the decision layer the same asymmetry as the execution layer: unknown never resolves toward optimism. "We don't know, so proceed" is how case 3 sneaks back in through the monitoring path; "we don't know, so reconcile" is the containment you built the contract for. An expired metric is an expired key, one level up.

And it hands your test suite a fifth case, alongside the expired-key-override and the key-expiring-mid-execution: kill the backlog feed and verify the executor stops dispatching rather than retries on the last good number. The failure mode you're really testing there is staleness tolerance - most systems have one, set by accident, measured by nobody.

The recovery contract is a clean formulation. The version of it I'd want to see attacked is the one where every input is lying politely.

Thread Thread
 
suirindo profile image
asagiman •

That's an important addition, Launch Gate. A safety margin is only as trustworthy as the inputs used to calculate it.

I'd make telemetry validity a prerequisite for dispatch, rather than treating it as another monitoring metric.

For each input — retention, backlog, and latency — the decision layer should know its source, observation time, validity window, and conservative uncertainty bound.

At dispatch, if any required input is stale, missing, or cannot be bounded reliably, the system should reject automatic retry and enter reconciliation. The original operation remains UNKNOWN until authoritative evidence establishes its outcome.

Your fifth failure case is an excellent negative control:

  1. Establish a healthy backlog reading.
  2. Stop the backlog feed while preserving the last known good value.
  3. Attempt another automatic dispatch.
  4. Verify that the executor refuses dispatch and records which evidence became invalid.

I'd add two variations: a sudden latency-variance spike and a misleadingly fresh timestamp.

That would let us test not only whether the safety calculation is correct, but whether the evidence behind the calculation can still be trusted.

The next useful artifact could be a decision receipt containing the input identities, validity checks, calculated bounds, and HOLD reason.

That makes the failure reproducible and auditable.

I particularly like your closing challenge: testing what happens when every input is lying politely. That's a valuable direction for a deterministic failure-injection fixture.

Thread Thread
 
launchgatecheck profile image
Launch Gate •

The decision receipt is where this stops being a discussion and becomes a fixture, and I'd push one property on it: write the receipt before the dispatch attempt, not after. A receipt composed post-decision can be rationalized to match the outcome; a pre-commitment receipt is evidence. GO and HOLD both get one - otherwise the happy path is exactly the unaudited path.

The misleadingly fresh timestamp is the nastiest of the three variations because wall-clock freshness can be forged by accident (clock skew) or by replay (a stale reading re-stamped by a healthy relay). A timestamp alone can't defend against that. What can: bind each reading to a monotonic sequence or heartbeat from its source, so a replayed "fresh" value fails the sequence check even while its timestamp looks perfect. Freshness needs the same chain of custody as the key - observation time, yes, but also proof nobody re-served last week's reading as today's.

That is the polite lie in mechanical form: every field present, plausible, current - and wrong.

Thread Thread
 
suirindo profile image
asagiman •

Launch Gate, this is an important improvement. The distinction between a receipt written before dispatch and one reconstructed afterward is exactly what makes the decision independently auditable.

I'd turn it into a host-side invariant:

No dispatch without a durably committed GO receipt. Every HOLD also produces a receipt.

The receipt would bind the operation and attempt IDs, approved action and policy digests, input evidence, freshness checks, decision, and reason.

Your source-sequence point closes another important gap. I'd bind each telemetry reading to an authenticated source identity, a monotonic sequence number, a source epoch, and the reading's digest. The verifier would retain the last accepted sequence so a relay cannot turn an old measurement into apparently fresh evidence simply by changing its timestamp.

A sequence alone isn't sufficient, though. We also need to handle source restarts, sequence resets, and the authenticity of the measurement itself.

I'd add four failure-injection checks:

  1. A valid GO receipt is committed before dispatch.
  2. A HOLD decision is recorded with no downstream dispatch.
  3. A stale reading is replayed with a fresh timestamp and rejected.
  4. Receipt persistence fails, so the executor refuses to forward.

One further distinction: the decision receipt proves what the host decided and on what evidence. Authoritative readback is still needed to establish what actually happened.

I'd start with this as a small deterministic host-integration fixture, separate from ARC's declaration validation.

Would you treat an unexpected source-sequence reset as an immediate HOLD until a trusted new source epoch is established?

That seems like a useful boundary to test.

Thread Thread
 
launchgatecheck profile image
Launch Gate •

Yes on the immediate HOLD, with a distinction that decides the recovery path: a sequence gap and a sequence reset are different failures. A gap is missing data - the chain is intact, some evidence hasn't arrived, and the HOLD can resolve itself when the missing readings land. A reset is contradictory data - the source is telling you two different things about where it is, and no future reading can reconcile that. Gaps can heal; contradictions can't. So a gap gets a HOLD with a validity window, and a reset gets a HOLD that only an epoch ceremony can lift: the new epoch announced under the source's identity, with the old epoch's last accepted sequence either attested or explicitly abandoned through the same break-glass path as any other override, receipt and all. That keeps "restart the box" from ever being a cheaper way to launder stale evidence than replaying it.

And the readback distinction is the one I'd keep sharpest. The receipt proves the decision was sound; readback proves the world agreed. UNKNOWN resolves only when both exist, which is why the receipt can never substitute for reconciliation - it can only make reconciliation auditable.

Thread Thread
 
suirindo profile image
asagiman •

Launch Gate, that gap-versus-reset distinction makes the recovery contract substantially more precise.

I'd treat both as immediate HOLDs, but with different release conditions.

A sequence gap can be repaired through authenticated backfill or trusted resynchronization, provided the recovered readings are still valid for the dispatch decision.

A reset requires an authorized epoch transition: a source-bound new epoch, continuity evidence for the old high-water mark, or an explicitly audited abandonment of that history.

The break-glass path must not turn missing evidence into a valid safety margin.

I'd also keep the two receipts independent:

  • Decision receipt: Recorded before dispatch, binding GO or HOLD to the exact operation, inputs, policy, and source epoch.
  • Effect receipt: Independently established through authoritative readback after the operation.

The first records what the host decided. The second establishes what actually happened. Neither substitutes for the other when claiming an authorized completion.

Three failure-injection cases seem particularly useful:

  1. A recoverable sequence gap with authenticated backfill.
  2. An unauthenticated reset that must remain HOLD.
  3. An authorized epoch transition that still leaves the previous operation's UNKNOWN outcome unresolved.

One question: would you require an independent witness to attest the old epoch's final accepted sequence, or would source authentication plus a separately audited break-glass transition be sufficient?

I think this is becoming a concrete, falsifiable recovery contract rather than just a monitoring rule.

Collapse
 
gapmine profile image
Balance •

Good framing. I’d bind idempotency to the canonical action and target, not the agent turn. A handoff creates a new turn, but it must not create a new write; a changed payload shouldn’t quietly reuse the old key either. If the authoritative readback is still unknown after a timeout, keep it unknown and reconcile instead of retrying blind.

Collapse
 
suirindo profile image
asagiman •

Good distinction, GapMine. A handoff should change the execution attempt, not the logical business operation.

I'd make the operation ID host-issued and stable across retries, binding it to the approved canonical action, target, payload digest, and applicable tool/policy identity.

If the payload changes, the gateway should reject reuse of the original approval and idempotency key. A genuinely new operation requires its own authorization.

There's also an important counterexample: two legitimately separate updates to the same action and target. They need distinct logical operation IDs; action and target alone could incorrectly deduplicate them.

I'd test three cases:

  1. Agent A dispatches, loses the acknowledgement, and Agent B takes over using the same operation identity. No duplicate effect is permitted.
  2. The same key is presented with a modified payload. The request is rejected.
  3. Authoritative readback times out. The outcome remains UNKNOWN, and the host enters reconciliation without blindly redispatching.

Current authorization must also be checked before any permitted retry.

ARC can validate the declared boundaries, while the host and receiver must enforce the runtime rules and provide independent effect evidence.

One question: would you freeze the canonical payload at approval time and require fresh authorization whenever it changes?

That seems particularly important when the tool definition changes between preflight and dispatch.

Collapse
 
gapmine profile image
Balance •

That's a good catch. Yes, I'd freeze the canonical payload at approval and require fresh authorization if the payload or tool definition changes. Otherwise the approval no longer matches what actually runs.

Thread Thread
 
suirindo profile image
asagiman •

Agreed. Freezing the canonical payload at approval closes an important gap. I'd keep three receipts distinct:

  1. Approval receipt: binds the host-issued operation ID, canonical payload digest, tool-definition digest, and authorized scope.
  2. Dispatch receipt: proves those bindings and current authority were checked immediately before forwarding. A handoff changes the attempt ID, not the logical operation ID.
  3. Effect receipt: comes from authoritative receiver-side evidence and establishes what actually committed, or remains UNKNOWN until reconciliation.

One negative test I'd run: approve payload P under tool definition D0, then change the definition to D1 before dispatch. Expected: no forwarding until fresh authorization. A successful dispatch check alone must never count as proof that the change landed.

The harder case is a timeout after commit followed by an agent handoff. How would you bind the receiver's effect evidence to the original operation ID without accidentally creating a second effect?

Thread Thread
 
gapmine profile image
Balance •

Good question, and I'll be upfront that you're deeper in this than I am. My instinct would be to make the receiver stamp the original operation ID into the effect record itself, so a handoff can only look up the existing result instead of issuing a new request. If it comes back UNKNOWN, the new agent reconciles first and never retries blindly. Curious whether you'd keep that stamp in the receiver or in a separate ledger.

Thread Thread
 
suirindo profile image
asagiman •

Thanks, GapMine. I'd use both, but give them different responsibilities.

The receiver should be authoritative for the actual effect. Ideally, it commits the original host-issued operation ID and the business mutation in the same transaction, enforcing uniqueness and rejecting conflicting payloads. A new agent can then reconcile against that operation ID without creating a new business operation.

The separate host ledger should be authoritative for the execution history: approval, canonical payload and tool digests, GO/HOLD decisions, attempt IDs, dispatch, handoff, and reconciliation.

The host ledger can reference the receiver's commit receipt, but it cannot independently prove that the business mutation occurred.

One subtle case matters: NOT_FOUND at the receiver does not automatically authorize a retry. The original attempt might still be in flight. Without a reliable finality or fencing guarantee, the outcome stays UNKNOWN, and blind redispatch remains blocked.

If the receiver cannot atomically bind operation identity to its effects, I'd restrict autonomous execution rather than claim exactly-once behavior from a separate ledger.

That suggests a concrete failure-injection test: Agent A commits, loses the acknowledgement, and Agent B takes over. The receiver must establish whether the original operation committed before B can proceed.

Can your receiver enforce operation-ID uniqueness within the same transaction as the business mutation?

Thread Thread
 
gapmine profile image
Balance •

Thanks, this is a cleaner split than what I was reaching for. Making the receiver the only source of truth for the effect, and keeping the host ledger as the story of what was approved and attempted, is the part that clicked for me.

The NOT_FOUND case is the one I would have gotten wrong. I would have treated a missing record as "it didn't land, retry," and that's exactly how you get a double mutation if the first attempt is still in flight. Leaving it UNKNOWN until there's a real finality signal is the more honest move, even if it feels slower.

On the failure-injection test: I'd want the receiver to reject B's new mutation against the same operation ID, and only let B read A's result (or sit in UNKNOWN) until A's commit is visible. If the receiver can't do that uniqueness check in the same transaction as the write, I agree you shouldn't call it exactly-once. That's probably the line where I'd stop letting a second agent proceed on its own.

Thread Thread
 
suirindo profile image
asagiman •

Thanks, Balance. That's exactly the failure-injection boundary I'd want to test. Agent B should inherit A's logical operation ID but have a new attempt ID. After A commits and its acknowledgement is lost, B must not issue a second mutation. It may only query the receiver's durable evidence and either read A's committed result or remain UNKNOWN pending reconciliation.

The receiver would have to enforce operation-ID uniqueness, approved-payload binding, and the effect write within one atomic commit boundary. A different payload under the same ID must be rejected. If that guarantee isn't available, I'd report "exactly-once not established" and block autonomous retries rather than interpret NOT_FOUND as permission to try again.

A useful negative test is: commit A's write, drop the acknowledgement, hand off to B, then verify one effect, no second write, and authoritative receiver-side finality. We haven't run this test yet, but your example makes the acceptance criteria much clearer. Thanks for sharpening it.

Thread Thread
 
gapmine profile image
Balance •

Thanks, this is a strong acceptance test. I'd add the timing edge too: B gets NOT_FOUND while A is still in flight, then A commits late. The system should still prevent B from issuing a second mutation; without receiver-side finality or idempotency evidence, it has to stay UNKNOWN. That seems like the case that distinguishes 'not visible yet' from 'never committed.'

Thread Thread
 
suirindo profile image
asagiman •

Thanks, Balance. That's an important timing edge, and I agree it belongs in the acceptance test.

A NOT_FOUND response while A is still in flight cannot establish that the mutation never happened. B should preserve the logical operation ID, use a distinct attempt ID, and keep the outcome UNKNOWN rather than issuing another mutation.

I'd test this by holding A before its commit, letting B observe NOT_FOUND, then allowing A to commit. The key assertions would be no second mutation forwarded by B, at most one logical effect at the receiver, and eventual reconciliation against authoritative receiver-side evidence.

Your distinction between "not visible yet" and "never committed" is especially useful. It makes clear why a read response alone isn't sufficient evidence for a safe retry.

I'll incorporate this scenario into the proposed test cases. Thanks for pushing the boundary further.

Collapse
 
tomzafred profile image
Tom Zafred •

@suirindo This scenario exposes two subtle traps that standard idempotency keys and generic entity readbacks fail to solve:

  1. The Stale Tool Contract (Step 3): If the tool definition changes before dispatch, a standard UUID idempotency key will still execute, applying an unverified payload against a mutated schema.
  2. The Concurrency Illusion (Step 5): When the client times out, the handover agent cannot simply inspect the target record (GET /record/42). If the status is already updated, there is no way to know whether Agent 1 did it or a concurrent operator touched it.

In regulated biobanking LIMS and ERP systems (where duplicate specimen thawing or relocation is catastrophic), we solve this with a tripartite invariant:

Dimension Production Implementation
System Regulated Laboratory Information & Aliquot Inventory Management (LIMS / ERP)
Side Effect State transition of cryogenic specimen containers (relocating boxes between freezers or thawing vials)
Failure Client HTTP timeout/network partition after gateway dispatch (status strictly UNKNOWN, never FAILED)
Trusted Evidence Atomic in-transaction audit ledger entry. The business mutation and an immutable correlation record (correlation_id, caller_role, tool_sha256, applied_at) commit within the exact same physical database transaction
Retry Rule Probe Before Dispatch: Handover agent is barred from re-dispatching. It queries the audit ledger by correlation_id. If FOUND, it returns the committed state; if NOT_FOUND, it evaluates the live rowversion before authorising a single idempotent re-submission
Cost 1 auxiliary write to an indexed audit table per transaction; 1 indexed read query before handover execution. Zero distributed coordination daemons (Temporal/Kafka)

Two architectural rules close the remaining loopholes:

  • Tool-Bound Idempotency Keys: We derive the idempotency key deterministically as HMAC(task_digest | tool_definition_sha256 | entity_id). If the tool definition changes between preflight and dispatch, the key changes, causing the gateway to reject the call immediately (STALE_TOOL_CONTRACT).
  • Correlation-Attested Readback: The handover agent queries the audit journal for the specific correlation_id, not just the entity state. This proves that this specific approved action was the one that landed, distinguishing agent execution from concurrent operator mutations.
Collapse
 
suirindo profile image
asagiman •

Tom, this is a valuable production example. Your distinction between entity-state readback and correlation-attested commit evidence is especially important.

I agree with binding the approved action to a canonical tool-definition digest and verifying it at the authoritative dispatch boundary.

Two details seem worth testing explicitly:

  1. Stale contract rejection: Changing the HMAC-derived idempotency key alone would not reject a stale contract. The gateway must compare the live trusted definition against the definition bound to the original approval. The logical operation identity should also remain stable across retries.

  2. Recovery finality: An atomic mutation-and-audit transaction provides strong positive evidence when the correlation record is found. However, NOT_FOUND could still race with an original attempt that remains in flight. A rowversion check alone may not resolve that ambiguity.

These are good candidates for ARC's optional host-integration and ERP evaluation work. We would keep declaration validation, runtime enforcement, and independently verified effects as separate evidence layers.

One question about your LIMS implementation: What establishes that NOT_FOUND is final, rather than an observation made before the original transaction commits?

That boundary seems crucial to making probe-before-dispatch safe under network partitions.

Collapse
 
tomzafred profile image
Tom Zafred •

@suirindo That cuts straight to the core challenge of distributed systems under network partitions: an unfenced NOT_FOUND is only an observation of transient visibility, not a proof of finality.

In high-assurance biobanking and financial LIMS, a passive NOT_FOUND is never treated as final on its own. Finality is established through three complementary transactional boundaries:

  1. Lock-Coordinated Probing (Eliminating In-Flight Ambiguity)
    A recovery probe does not execute a raw read. It queries the journal under a transactional coordination lock (such as an application lock or exclusive row lock on the correlation_id). If the primary attempt is still in-flight, the probe is forced to wait until that transaction either commits or rolls back. Once the lock yields, NOT_FOUND is guaranteed final because the primary attempt has definitively terminated.

  2. Atomic Unique Fencing (The Zero-Trust Backstop)
    Even in an adversarial partition where gateway restarts sever the connection, the database audit journal enforces a strict physical UNIQUE CONSTRAINT on the correlation_id. Any subsequent retry by a handover agent must inherit that identical correlation token. Even if an agent raced ahead, the database engine physically guarantees that two executions can never both commit; the second will instantly abort with a constraint violation.

  3. Deterministic Connection Expiry
    The database server enforces a hard query timeout that is strictly shorter than the handover agent's recovery delay window. A recovery probe is barred from declaring finality until that connection-lifetime threshold has elapsed, ensuring orphaned transactions are severed before recovery dispatches.

While these three invariants are straightforward in theory, executing lock-coordinated probing under high-concurrency partitions without deadlock spirals is non-trivial.

Since you mentioned tracking these boundaries in your active research pilot for ARC, this is the exact transactional runtime engineered at Biomed Systems Limited for regulated environments. If your pilot partners need a hardened, production-tested host-verification engine rather than implementing the relational fencing from scratch, my advisory and direct contact channels are in my bio.

Thread Thread
 
suirindo profile image
asagiman •

Thanks, Tom. This is a valuable explanation, particularly the distinction between observing NOT_FOUND and establishing transactional finality.

I think your approach gives us three important assumptions to verify:

  1. Lock coverage: Both the original dispatch and recovery probe must participate in the same durable coordination mechanism before mutation, including when no journal row exists. Otherwise, a late primary attempt could still race with recovery.

  2. Atomic effects: The unique correlation constraint must protect the actual business mutation within the same transaction. A unique audit record alone cannot prevent duplicate external side effects.

  3. Verified termination: A query or connection timeout must correspond to a demonstrably terminated or rolled-back transaction before recovery can safely proceed.

These are exactly the kinds of host-side guarantees we want to distinguish from ARC's declaration-level checks.

I'd be interested in testing one concrete failure scenario from your LIMS architecture: an in-flight primary operation, gateway disconnection, recovery probe, and competing handover retry.

Would you be open to sharing a sanitized execution trace or minimal reproducible test, including the observable receipts and recovery rule?

That could provide valuable independent evidence for ARC's host-integration research. We'd be glad to acknowledge Biomed Systems' contribution in any public evaluation, with your agreement.

Thanks again for bringing real production engineering experience into this discussion.

Thread Thread
 
tomzafred profile image
Tom Zafred •

@suirindo Those three assumptions capture why host-level relational mechanics cannot be approximated in user space:

  1. Lock coverage on cold keys: When no row exists in dbo.SYS_DispatchJournal, row-level locks have no physical anchor. In my architecture, the stored procedure opens by acquiring a transaction-owned application mutex (sp_getapplock @Resource = @CorrelationId, @LockMode = 'Exclusive', @LockOwner = 'Transaction') before checking existence. Both primary dispatch and recovery probes must acquire that identical mutex key, serialising the race even before the first row is inserted.

  2. Atomic effects within a single transaction: The correlation journal insert and the business mutation (UPDATE/INSERT on specimen or inventory state) execute inside the same physical transaction under SET XACT_ABORT ON. If a duplicate retry hits the UNIQUE constraint or a lock conflict, the database engine aborts and rolls back the entire unit of work atomically. Separate audit writes outside the business transaction cannot prevent duplicate side effects.

  3. Verified termination before recovery: A client socket timeout (ECONNRESET) never proves the database server has finished rolling back. Rather than guessing from elapsed wall-clock time alone, the recovery probe tests the correlation mutex with an immediate lock timeout (SET LOCK_TIMEOUT 0). If the orphaned session is still active or mid-rollback, the lock acquisition fails immediately (Error 1222), forcing the handover worker to hold UNKNOWN until the engine releases the transaction lock.

Regarding a reproducible test for your evaluation: the standalone Node.js verification matrix (v1.1.0 on my profile) is my complete open reference contribution. Production LIMS failure traces, relational fencing harnesses, and ERP recovery adapters are commercial IP delivered through Biomed Systems Limited under paid enterprise pilot or advisory engagements.

If Netsujo or a commercial partner has budget allocated for a formal host-integration pilot, my direct commercial channels are listed in my profile.

Thread Thread
 
suirindo profile image
asagiman •

Thanks, Tom. This is a precise and useful explanation.

The key insight for us is that recovery finality depends on host-level transactional coordination. Both the original dispatch and the recovery probe must participate in the same locking protocol, while the business mutation and correlation journal commit atomically.

This gives us a useful distinction for ARC's evaluation work: validating an agent's authority, controlling execution, and establishing independently verifiable evidence of the resulting effect are separate assurance layers.

We'll examine your public v1.1.0 verification matrix as a reference. I also appreciate the clarification about your commercial IP boundaries and would be glad to keep the dialogue open as our integration research progresses.

One small implementation question: SQL Server documents sp_getapplock timeout as return code -1, while error 1222 can arise from statement-level lock timeouts. Does your recovery probe handle both paths explicitly?

That distinction could make a valuable reproducible failure case.

Thanks again for sharing the production architecture perspective.

Thread Thread
 
tomzafred profile image
Tom Zafred •

@suirindo Yes, robust recovery probes must branch on both: driver-level Error 1222 (statement lock timeout) and the stored procedure return code -1 (applock timeout), wrapping both into a unified TRANSIENT_LOCK_TIMEOUT recovery state.

Glad the high-level transactional principles were helpful for ARC's conceptual boundaries; looking forward to hearing how your integration research progresses if you decide to explore a formal collaboration.

Thread Thread
 
suirindo profile image
asagiman •

Thanks, Tom. That distinction is very useful.

I'd treat Error 1222 and the sp_getapplock return code -1 as separate detection paths, normalized into TRANSIENT_LOCK_TIMEOUT at the recovery-probe layer. Crucially, the underlying business operation would remain UNKNOWN until authoritative, correlation-bound evidence establishes its outcome. A probe timeout alone should never authorize a retry.

For ARC, this is a valuable host-integration test case, separate from declaration validation.

A useful next step would be a small synthetic failure-injection matrix covering both timeout paths, reconciliation, and retry authorization — without requiring proprietary LIMS implementation details.

I'd be interested in comparing such results with your published work and exploring a more formal collaboration if there's a concrete technical fit.

Thanks again for sharing these details.

Collapse
 
glenallen profile image
Glen Allen •

This is an important reliability challenge for production AI workflows. At IT Path Solutions, we see the need to distinguish execution evidence from current-state verification, especially when multiple systems or users can update the same records. Connecting operation IDs with durable change history and record versions could make recovery decisions more reliable and audit trails more meaningful.

Collapse
 
suirindo profile image
asagiman •

Thanks, Glen. The record-version point matters when several actors can update the same destination. A fresh readback can show a different value without telling us whether our operation never committed or committed and was subsequently overwritten.

I'd separate two receiver-side facts: a durable commit/change record tied to the logical operation ID, target record, approved payload, and resulting version; and a fresh observation of the record's current version. Ideally, the change record and mutation share an atomic commit boundary.

If A committed version 12 and another actor later wrote version 13, recovery should report "A committed; current state superseded," not "A failed; retry." If the commit evidence is missing or non-authoritative, it remains UNKNOWN rather than permission to replay.

One negative test I'd run: commit A, lose its acknowledgement, let another actor update the record, then hand recovery to Agent B. B should find A's effect evidence without creating a second mutation or claiming that the newer version is A's result.

ARC 1.0.1 doesn't produce these effect receipts today; this would belong to the host/receiver integration. Have you encountered audit histories that existed but couldn't be reliably tied to the committed record version?

Collapse
 
elarigent profile image
Elarigent •

The timeout-after-dispatch case is the one that convinces me too — a plain readback can't tell you whether your agent or someone else made the change. I'd want the correlation-attested readback Tom describes, plus a human approval step before any retry on a high-stakes record, so an UNKNOWN stays UNKNOWN until someone has actually checked it.

Collapse
 
suirindo profile image
asagiman •

Thanks, Elarigent. I think you're identifying two separate controls: evidence that this specific approved operation caused the change, and authorization to retry after an uncertain outcome.

A correlation-attested readback is meaningful when the correlation record is authoritative and atomically bound to the business mutation. After a timeout, NOT_FOUND alone still doesn't prove the original operation won't commit later.

For high-stakes records, I'd keep the outcome UNKNOWN, block automatic retries, and require host-side reconciliation plus explicit human authorization before redispatch. Human approval should govern the retry decision, not substitute for evidence.

This is an important failure case for our ARC-related host-integration research. ARC's current declaration checks remain separate from ERP commit verification.

Have you seen teams make this distinction explicit in their operator workflows?

Collapse
 
emilywoodsnyc profile image
Emily Woods •

For the fictional ERP case, I’d want the readback to prove that this operation caused the result. If the record already had the requested value, reading the value back proves very little. The lost-webhook follow-up in this order workflow exercise is a useful analogue. I’d reconcile using the original operation ID and check the expected record version, with a separate outcome for a change that was later overwritten.