Agent Session Recovery and Terminal Evidence

Short answer

Session recovery needs more than a transcript. A durable agent host must answer at least four independent questions before it offers retry or continuation:

1. Liveness: is work for this session still executing somewhere?
2. Provenance: did the apparent unfinished tail come from a real user prompt, a system-injected notification, a scheduler, a tool or another internal source?
3. Delivery: did execution already produce a reply that is persisted but still owed to the user?
4. Finality: is there already trustworthy terminal evidence that the operation completed, failed or was interrupted?

Treating those as one question creates dangerous failure modes. A persisted system notification can look like an unanswered user message after projection; a healthy background turn can look resumable if the host only tracks foreground prompt state; a completed reply can be regenerated if crash recovery uses recent activity instead of exact turn ownership; and an append-only terminal ledger can permanently preserve a wrong completion classification if recovery logic stamps a result without attributable evidence.

Two current upstream fixes show complementary parts of this boundary. Qwen Code PR #12007, merged on 18 September 2026 as ff38a9c430bdfbbfa03896c806c78778ee6fc24e, hardens provenance, active-work admission and terminal reconciliation. Hermes PR #120377, merged on 23 September 2026 as 1136f135dd640fff082af7859386637d347dbcd2, replaces a crash-time recency heuristic with durable active-turn ownership and closes the gap between a persisted reply and its durable delivery obligation. Hermes stable is now v2026.9.24 / Agent v0.21.5; GitHub ancestry confirms #120377 is included in that tag rather than remaining current-main-only behavior. Neither change revises UHP, MCP, ACP or A2A wire semantics.

Choose delivery, continuation or investigation after a crash

Start recovery from the exact execution/turn identity, then inspect its ownership and delivery records. The Hermes handoff fix shows why a recently touched transcript is insufficient.

Durable evidence	Safe recovery class	Action
Live owner still holds this turn	Active	Reattach/observe; refuse a competing continuation
Final reply plus outstanding delivery obligation	Deliverable	Recover delivery of that reply; do not regenerate it
Interrupted user-origin turn with no terminal result	Candidate continuation	Revalidate authority and admit new execution explicitly
System-origin tail only	No user continuation inferred	Preserve notification; avoid synthesizing a new prompt
Conflicting terminal/ownership evidence	Indeterminate	Preserve records and investigate before execution

Consider a crash after reply persistence but before adapter acknowledgement. The desired operation is delivery reconciliation using the original reply and attempt identity. Starting another model turn can produce a different answer and duplicate external effects. If the channel cannot prove whether delivery happened, record uncertainty rather than calling either resend or suppression guaranteed safe.

A useful rehearsal places crashes on both sides of the ownership-to-delivery handoff and checks that at least one durable record still names the obligation. This proposed fixture complements control-delivery evidence; this site has not executed the upstream crash matrix.

The recovery stack

A robust recovery implementation benefits from separating these records and decisions instead of deriving everything from one projected chat history:

Surface	What it should establish	What it cannot establish by itself
Persisted transcript/history	What content was recorded and in what order	Whether the last entry represents live user intent, internal injection or settled execution
Provenance metadata	Who or what produced a record	Whether the producing turn is still active
Active-turn ownership	Which exact turn, if any, still owns execution	Whether a completed reply is already durably owed for delivery
Delivery ledger/obligation	Which produced output is still owed, attempting or acknowledged	Whether model execution itself remains live
Recovery classifier	Whether persisted evidence looks clean, interrupted, deliverable or resumable	A terminal fact unless its inputs preserve the required provenance/finality evidence
Terminal ledger/snapshot	Durable settled outcome for an execution identifier	Whether a still-live turn may safely be superseded
Admission policy	Whether retry/continue may start now	Historical truth about what happened before admission

The key design rule is do not collapse history, provenance, liveness, delivery and finality into one heuristic.

Hermes failure case: recent activity is not recovery ownership

Hermes PR #120377 exposes a particularly costly recovery shortcut. On an unclean gateway start, the older path called SessionStore.suspend_recently_active(120). Every session touched in the previous 120 seconds could be marked resume_pending / restart_interrupted, even when its turn had already completed and its reply had already been delivered. Auto-resume could then run a new model turn and send an unsolicited second answer.

The fix removes that recency sweep. Unclean recovery now uses durable active-turn markers that identify the exact turns that were in flight. That changes the recovery question from “was this chat recently active?” to “does durable state prove this exact turn still owned work when the process died?”

The reusable invariant is:

Crash recovery should act on durable ownership evidence, not temporal proximity.

A timestamp window can be useful for bounding stale evidence, but it is not an ownership proof. Any heuristic broad enough to include completed work can turn recovery into duplicate execution.

The execution-to-delivery handoff must be durable

The same Hermes fix closes a second crash window between execution and delivery. The gateway previously cleared its active-turn marker in the runner’s finally path before the adapter durably recorded the reply in the delivery ledger. A process death in that window could leave a final assistant reply persisted in the transcript but with neither an active-turn marker nor a delivery obligation.

PR #120377 moves ownership of that marker across the handoff. For turns whose reply is delivered by the adapter, the marker remains until the adapter has recorded the delivery obligation, or until the system can prove that nothing is owed because the reply was streamed, empty, suppressed, errored or cancelled. Other runner paths still clear their own markers.

That creates a useful two-phase recovery boundary:

execution owns turn marker
        │
        ▼
final reply persisted
        │
        ▼
adapter records delivery obligation
        │
        ▼
turn marker may clear
        │
        ▼
delivery ledger owns redelivery/reconciliation

There must not be a crash interval in which neither side owns the outcome.

Recover evidence before re-executing semantics

At unclean startup, Hermes now distinguishes four materially different states:

Durable state at restart	Correct recovery action
Turn already finished and delivered	Do nothing
Final assistant reply persisted after the active-turn marker, but not yet ledgered	Adopt the stored reply into delivery state and deliver it; do not regenerate it
Reply already in the delivery ledger but acknowledgement is uncertain	Let delivery reconciliation/redelivery handle it under the ledger’s semantics
Active-turn marker exists and no persisted final reply exists	Resume the interrupted turn once, subject to the existing freshness/stuck-loop rules

This is more than an implementation detail. A persisted final reply is evidence that execution already produced semantics. Re-running the model is not equivalent to delivering that stored result: the new generation can differ, repeat side-effecting intent, consume more resources, or violate user expectations.

A strong recovery system therefore prefers this order:

1. recover already-produced durable evidence;
2. reconcile pending delivery;
3. resume or regenerate only when durable state shows that execution did not finish.

Suppression and time evidence belong to the same recovery contract

The review follow-up for #120377 also applies live-delivery policy during crash adoption. Internal bare silence markers such as [SILENT], SILENT or NO_REPLY, and diagnostic wake replies muted by chat policy, are not converted into recovered outbound messages merely because the process crashed. Human-turn silence markers retain the same user-facing handling as the live path.

The active-turn marker’s start time is also written as timezone-aware UTC and compared as epoch time. That matters because restart may occur under a different process timezone. Recovery freshness should not reinterpret the same durable instant differently after a deployment, container move or host configuration change.

These are instances of one broader rule: recovery must apply the same semantic delivery policy and time interpretation as live execution. A crash must not create a second policy regime.

Qwen’s failure case: a system notification looked like an orphaned prompt

Qwen records a background notification before its automatic turn runs. That is useful for durability, but it creates a subtle recovery shape when the notification is persisted and the automatic turn never answers it: the history ends in a user-role entry that has no model response.

In the affected projection, the source record’s subtype: 'notification' and provenance: 'system' were not preserved in the Content representation consumed by recovery. The tail could therefore resemble a real unanswered user prompt and be classified as interrupted_prompt with canContinue: true.

PR #12007 makes the classifier ignore the narrow trailing shape used by cold system-injected <task-notification> entries before it classifies the underlying turn. The fix is intentionally conservative: it does not turn every user-role notification-looking string into system provenance, and the upstream review records cases where a genuine interrupted prompt must still remain recoverable.

This exposes a general architectural requirement: a lossy projection should not silently become the authority for provenance-sensitive recovery decisions. If a host projects rich records into a simpler message type, it should either carry the provenance alongside that projection or explicitly bound the heuristics used when provenance is absent.

“Continue” is an execution admission decision

The second Qwen bug was not only a misleading banner. Notification and cron turns use their own streaming loops and do not necessarily install the ordinary foreground pendingPrompt state. Checking only that foreground marker could therefore say canContinue: true while automatic work was still running.

That matters because accepting a continuation enters the normal prompt-admission path. In Qwen, normal user input has priority and aborts the notification/cron controllers. A false recovery offer could therefore terminate healthy automatic work.

PR #12007 changes Session.getRecoveryStatus() to gate continuation on the broader active-turn predicate rather than pendingPrompt alone.

The reusable invariant is:

A recovery UI must not offer an action that admission would use to supersede work the recovery classifier failed to notice.

In systems with multiple execution classes, “is a prompt pending?” is weaker than “does this session currently have an authoritative turn owner?” Background notifications, scheduled jobs, goal continuations, settling prompts and history mutations may all need to participate in that ownership test.

Terminal evidence must fail closed

Qwen’s daemon keeps an append-only prompt terminal ledger. Recovery reconciliation uses persisted history to decide whether an unresolved prompt should receive a terminal entry such as completed or interrupted.

That makes false positives especially costly: once an append-only ledger records a wrong terminal, later loads can treat the incorrect classification as durable evidence.

PR #12007 therefore aligns the ledger with the same classifiable tail used by recovery, excludes system-provenance records from evidence attribution, and prevents a completed stamp when the remaining visible write is user-role without the required attributable completion evidence.

A useful general rule is:

When recovery cannot prove completion, prefer unknown/interrupted evidence over fabricating completion.

That does not mean every ambiguous session should be retried automatically. A fail-closed terminal classifier and a conservative admission policy are separate controls: the first avoids inventing history; the second decides whether another execution may safely begin.

All recovery consumers must use the same evidence boundary

A recovery classifier is rarely consumed by one UI. Qwen’s PR documents affected paths beyond the Web Shell banner, including non-interactive continuation, --continue-interrupted, TUI /resume and OpenTUI session switching.

That breadth matters. Fixing only the visual banner would leave machine-driven or headless paths capable of acting on stale recovery state. In the pre-fix path described upstream, some continuation surfaces could re-submit a persisted <task-notification>...</task-notification> envelope as retry input even though it was not a real user prompt.

Hermes shows the same consistency requirement at a different boundary: boot recovery, transcript persistence, active-turn ownership and delivery reconciliation must agree on which component owns an outcome. Fixing only the resume decision while leaving the marker→delivery handoff non-durable would still permit lost or regenerated replies.

The general consistency requirement is:

• normalize the classifiable history tail once;
• preserve exact active-turn ownership across process death;
• hand completed output to durable delivery state before releasing execution ownership;
• reuse the same suppression and delivery policy during recovery;
• gate every continuation/retry surface on the same active-turn ownership rule;
• preserve the distinction between “show a notice,” “deliver stored output,” “offer continuation” and “automatically execute.”

Recovery semantics are part of the control plane, not merely presentation state.

Verification boundary and known limitations

For Qwen PR #12007, upstream reports integration-level verification through the same history projection and recovery layers used by the application. Its recorded post-fix suites include 561 relevant core tests, 1,019 tests in the CLI Session.test.ts file and 38 prompt-terminal-ledger tests. The PR is explicit that no live qwen serve + browser reproduction was performed for the Web Shell banner, so those numbers are implementation/integration evidence rather than full end-user runtime certification.

For Hermes PR #120377, upstream used real gateway subprocesses with fake messaging platforms and SIGKILL/restart on the same HERMES_HOME. In the reported C12 quiet-restart cell, the pre-fix path emitted 22 unsolicited resumed replies on a 41-cell run (and 52 on the lane’s full run), while the fixed path emitted 0 new messages. The persisted-but-not-ledgered case delivers the stored answer after the fix instead of regenerating it. The review follow-up reports the gateway suite at 8,701 passed, while documenting unrelated load/timing flakes separately.

Hermes also keeps an explicit unresolved boundary: sent_ack_lost-fk_tg can still duplicate under heavy load if a streamed final is accepted by the platform before the turn is persisted and the process dies in that interval. The accepted text then dies with the process while the durable active-turn marker can still cause a resume. That STREAM_CRASH_AFTER_ACCEPT gap is separate from the recency-fan-out and marker→ledger fixes, so #120377 should not be described as universal exactly-once messaging.

Two provenance-related Qwen gaps also remain documented upstream and tracked in Qwen issue #12042:

1. a failed live notification turn with no distinguishing reminder part can still be shape-identical to a cold persisted notification and can therefore be classified as clean; and
2. a genuine user prompt whose entire text is exactly the notification-envelope shape can still be treated as clean by the recovery banner path even though the terminal ledger now fails closed.

These limitations point to the same general principle: recovery documentation should state not only what a fix proves, but which crash windows and ambiguous evidence shapes remain unresolved.

Relation to process-death workflow recovery

Loop engineering documents a related but different Qwen pattern from PR #12159: checkpoint ownership, durable journal evidence and settled workflow history after process death. That design answers “who owned this unfinished workflow and how do we truthfully convert abandoned intent into history?”

The recovery boundary here answers a lower-level question: “given persisted session content, delivery state and possibly live automatic work, what does the tail mean, what output is still owed, and is another turn allowed to start?”

The patterns compose:

Durable workflow/session state
        │
        ├── ownership/liveness proof
        ├── author/source provenance
        ├── persisted produced output
        ├── delivery obligation
        ├── interruption classifier
        └── terminal evidence
                 │
                 ▼
      deliver / retry / continue / refuse / report

Persistence without ownership can double-run work. Ownership without provenance can misclassify internal records as user intent. Produced output without a durable delivery handoff can be lost or regenerated. Provenance without terminal evidence can lose or fabricate outcomes.

Relation to UHP and ACP

This page describes harness/host recovery architecture. It does not add a UHP task state or an ACP session method.

• UHP standardizes the external product/server/harness execution boundary. A UHP implementation may sit in front of a harness with sophisticated internal recovery, but UHP conformance does not certify the harness’s private recovery classifier, delivery ledger, active-turn marker or notification provenance model.
• ACP standardizes client↔agent session interaction. Qwen exposes recovery through its own host and ACP-related integration paths, but PR #12007 does not revise the ACP v1 schema or make a new ACP v2 semantic rule. Hermes PR #120377 is likewise gateway/session-delivery behavior rather than an ACP change.
• MCP and A2A are not changed by either fix. These recovery invariants may matter inside products that use those protocols, but the upstream changes do not revise their wire contracts.
• Harness internals still own how background notifications, cron turns, transcript projections, active-turn markers, delivery obligations, terminal ledgers and retry admission are represented unless an external protocol explicitly standardizes them.

See UHP vs ACP, Lifecycle, Qwen Code, Hermes and Loop engineering for the surrounding boundaries.

Practical review checklist

When evaluating an agent host’s session-recovery design, verify:

1. Does provenance survive persistence and projection? User, system, scheduler and tool records should not become indistinguishable accidentally.
2. What counts as an active turn? Include automatic/background execution, not only an RPC prompt handle.
3. Is recovery ownership exact? Recent activity or an updated_at window is not a substitute for a durable identifier of the turn that was actually in flight.
4. Who owns the execution→delivery handoff? There must not be a crash interval where the active-turn marker is gone but the delivery obligation is not yet durable.
5. Can recovery deliver persisted output instead of regenerating it? Completed semantics should not be re-executed merely because delivery was interrupted.
6. Who owns continuation admission? A UI hint should not bypass the same liveness checks used by the execution path.
7. Is terminal evidence durable and attributable? Do not stamp completion from message shape alone.
8. Do all consumers share one classifier and policy boundary? Web, TUI, headless, API and restart paths should not disagree about the same session state.
9. Can retry replay state-changing work? Recovery should not silently duplicate an operation because presentation or delivery state was wrong.
10. Are timestamps restart-safe? Persist instants in an unambiguous representation and do not reinterpret them under a new timezone.
11. What ambiguity remains? Document cases where provenance, persistence or acknowledgement is still insufficient for exactly-once behavior.
12. What was actually tested? Separate unit/integration evidence from live daemon/browser/runtime validation.
13. Is the behavior version-bounded? Distinguish stable release behavior from post-release current-main changes.
14. Is a host feature being mistaken for a protocol rule? Keep UHP/ACP/MCP/A2A semantics separate from implementation recovery policy.

Primary sources

• Qwen Code PR #12007 — session recovery and notification classification
• Qwen merge commit ff38a9c4
• Qwen issue #12042 — remaining recovery/provenance gaps
• Qwen Code v0.24.0
• Hermes PR #120377 — unclean gateway recovery and delivery handoff
• Hermes merge commit 1136f135
• Hermes Agent v0.21.5 / v2026.9.24
• Hermes v0.21.5 tag commit f97608f1