Routing, waits and recovery semantics

This page records decision B04: the runtime semantics for routing and ownership, active-event policies, wait registration and wake, lease recovery, retry, cancellation and causal bounds. It resolves the runtime choices left by B02 and B03, and its normative “must” requirements apply to every runtime implementation; the production runtime is internal/runtime. The executable specification is the shared runtime suite, using a public test adapter. It runs against a volatile reference coordinator and against the production runtime on SQLite. Route handlers use the public extension contract.

Routing and ownership

Acceptance durably retains the immutable Event and assigns its sequence before acknowledging ingress. Routing is a separate checkpoint. A duplicate (source, delivery_id) returns the original Event without rerunning committed effects. Correlation never substitutes for delivery identity.
Trusted ingress configuration selects one target Scope. An explicit Work ID selects that Work and its fixed scope; any supplied correlation must resolve to the same Work. Reject contradictory identity/scope rather than overriding it. Source namespaces include account/resource identity; uniqueness is (source, external_id) across scopes, including terminal Work. Matching a route cannot grant cross-scope authority.
Resolve effective routes root to leaf. Within a scope, route names must be unique and declaration order is significant. A descendant’s same-name route replaces the entire ancestor route in its original slot; new names append. Evaluate in that order and stop at the first match, including a match choosing drop. A handler error stops processing without trying later routes. No match commits unmatched, retaining the Event. There is no broadcast to every matching handler.
Handlers see detached Event/Scope/optional Work snapshots, do deterministic local computation and return decisions. They never call services or write coordination state during matching. A revision conflict requires reread/re-evaluation, with stable proposed IDs. Runtime atomically revalidates ownership and applies create/append-keys, context, queue/intents and MarkRouted. No executor launch or remote action runs in that transaction.
Concurrent Events resolving to one key create at most one Work. Different keys owned by one Work serialize route effects by acceptance sequence, not occurrence time, handler completion or goroutine order. Key attachment and route checkpoint commit together. Attachment adds keys, never removes/transfers them; an owned-by-other key aborts all effects and leaves the Event unrouted for diagnosis/retry. Acquire multiple key/Work guards in canonical order or use equivalent transactional conflict handling.
On key attachment, earlier accepted, still-unrouted Events for the new key join that Work’s ordered backlog. Already checkpointed unmatched, drop or terminal Events stay checkpointed; attachment never silently resurrects them. An operator can emit a distinct, causally linked replay-request Event. Previously unresolved Events do not acquire a retroactive dependency on unrelated Work.
Per-Work sequencing does not impose global routing order. A blocked handler, execution, wait or retry for A must allow B’s independent routing and scheduling to progress. Short serializable storage commits can serialize physically; no global lock may span handler/executor I/O or execution lifetime. Scope capacity can legitimately defer starts, but not unrelated acceptance/routing.
An uncorrelated Event can explicitly target existing Work, or a first-match handler can explicitly create fresh Work with stable ID and optional keys. Otherwise it is unmatched. Absence of a key never means “use whichever Work is running.” Multiple keys on one Work remain independent external identities with one lifecycle.
Terminal Work is immutable. Late Events for its keys get terminal, with association and audit intact. Reopen is an extension policy producing a successor with a new Work ID and distinct generation-qualified key, e.g. resource/42/generation/reopen-7. The adapter must consistently map subsequent deliveries to that generation. A causally linked core.work.created Event records predecessor/successor relation as opaque payload; ParentID still means child, not predecessor. Old keys stay on the old Work. No reopen policy means no automatic successor. Terminal failed Work follows this rule too.

Bindings are inherited by name with the nearest definition replacing the whole binding; inherited routes resolve against the effective bindings in their target scope. Child Work inherits its parent’s scope; an explicitly authorized management create may select another scope. Cross-scope subscriptions need explicit read/alias authorization and do not inherit write authority. Opaque extension config is replaced at its named entry, not recursively merged by core. B05 supplies the concrete config schema, secret-reference overlays, authorization ceilings, disabling entries and reload snapshots; it preserves route order and fixed Work scope. A validated config generation is pinned for each decision/attempt; replay uses a committed decision when present, otherwise the current validated generation and records it with the checkpoint.

Active-event policies and acknowledgements

Policy applies when the selected Work has a running attempt. Waiting Work first evaluates active Waits; ordinary nonmatching Events accumulate as context without bypassing a wait. Ready Work queues eligible input. There is one nonterminal attempt per Work, including pending/parked. Scope concurrency and retry eligibility are checked before Claim; storage Claim itself does not implement those policies.

Policy	Durable processing acknowledgement	Dispatch/replay and completion
queue	Event association, ordered pending input and route checkpoint commit together. No immediate second attempt.	Next eligible attempt takes pending input in acceptance order. Preserve consumed-input association/history so replay cannot enqueue it again.
inject	Commit an intent tied to Event, target attempt and lease epoch, plus route checkpoint. This is distinct from executor acknowledgement.	Invoke `Inject` outside Tx. Success means accepted for delivery, not consumed. Retain input for a subsequent attempt or wake unless a result of that execution proves consumption (`Result.ConsumedInputs` on any outcome; see generic inject acknowledgement). `ErrNotAccepting` settles the intent as not delivered (no retry; input retained). Repeat presentation of unproved input is explicit. Recover ambiguous sends using the same Event identity; no exactly-once external delivery claim.
interrupt	Commit pending input and interrupt intent with target attempt/epoch before calling `Interrupt`.	Fence the old attempt before any replacement: mark it cancelled/release authority under policy and retain pending input. A cancellation of an attempt for interruption leaves Work ready; cancellation of Work is terminal. Do not overlap process resources until stop/adoption exclusion is established. Replayed control targets the same old attempt, never a new one.
drop	Commit `drop` with Event, association, selected route, config generation and reason.	Do not queue, inject or start from this Event. Preserve its envelope and checkpoint for audit; replay is a no-op. Acceptance acknowledgement still follows durable acceptance.

An operator can also interrupt one running attempt without changing any route (POST /work/{id}/interrupt, operator controls). The control commits a core.manual.interrupt Event and an interrupt intent against the running attempt and epoch, and the same dispatch fences the attempt and keeps the Work ready with its pending input. The Event is a control, not input. The call is refused unless the running attempt was pinned with the interrupt capability and its executor implements it, and a failed dispatch never takes an error fallback. Whichever path fenced it, the attempt records the Event whose interrupt ended it (interrupt_event_id in run history).

An active inject/interrupt route must bind to the corresponding mechanically advertised and implemented capability. Reject the binding before scheduling unless an explicit configured queue or drop fallback exists. Record the fallback reason and effective policy. Unknown policy, coalesce, fallback cycles and missing implementations are validation errors. Never infer queue on unsupported control. A runtime failure/ambiguous result from an otherwise supported control is retried with stable intent identity or uses an explicitly configured error fallback; it is not an unsupported-capability shortcut. Post-completion dispatch never targets another attempt using the old intent.

Completion and routing are serialized against Work revision and authority. Before making Work terminal, completion must settle any accepted, unrouted input already resolving to that Work (it can return a retryable conflict until routing finishes). If input routing wins, unconfirmed injection/queued input keeps Work ready after the current attempt completes, then starts N+1. If terminal completion wins before new acceptance, a later Event gets terminal; a successor requires the explicit generation policy above. If completion races an in-flight Inject, retain the Event until consumption is proved; a successful return alone cannot erase it. Work may only complete when no pending eligible input remains. Generic inject acknowledgement settles delivery and consumption acknowledgement.

Queue is lossless ordered pending input; it does not replace older Events. Pending-job newest-delivery replacement, timer replacement and detached-busy behavior are extension behaviour built on these semantics (pending-job coalescing). This decision deliberately does not add a coalesce option.

Wait registration, observation and exactly one wake

A Wait is a durable subscription on its Work’s keys, with a type allowlist (empty means any type), exclusive AfterSequence and optional one-shot timeout. An Event matches only when its sequence is greater than the boundary, its key is in the Wait, its type passes the filter, and it is not one the Work’s own executions published (an Event carrying the Work’s ID and an execution ID: its turn ending, an Event it emitted). A Work is never news to itself. Types/keys are OR within each set and AND between sets. Target Work/authorized alias must agree. Occurrence timestamps and payload conditions are not core predicates. At most one active Wait exists for an attempt; multiple prerequisites are translated by the extension to a suitable Event, not a core condition language.

AfterSequence is the execution’s acknowledged observation watermark, not the sequence at registration. The caller cannot claim a watermark beyond accepted history. Core scans already accepted matching Events strictly after it in the registration transaction, including Events already routed while the attempt was active. Registration joins parking/terminal-wait outcome, lease release, Work waiting, timeout/subscription records and this scan. An immediate match may register and wake in that same transaction; old executor authority is still invalidated. A match at or below the watermark is not reused. For an already-met external condition whose evidence precedes the watermark, the adapter observes/reconciles and emits a fresh canonical observation/alias rather than lowering the boundary or directly waking Work.

For a dependent prerequisite, never attach the prerequisite owner’s correlation to the waiting Work. The extension persists an authorized subscription in its namespaced KV containing stable subscription ID, dependent Work/Wait ID, external source key/type translation and last accepted delivery checkpoint. Registration installs it and scans existing observations atomically; a provider read done outside Tx needs post-registration reconcile to close the gap. The adapter translates each matching source Event to an alias Event with dependent Work ID, one dependent-owned key, original Event as causation, and stable delivery identity (subscription ID, original Event ID). It accepts the alias and advances its subscription checkpoint atomically. Many dependents can get distinct aliases without sharing ownership. Cross-scope aliases require explicit policy authorization and minimal payload; cancelled/completed subscriptions emit no new runnable activity. State-change/merge/check/job semantics stay in adapters. The reference alias case proves ownership and dedup, not a provider subscription implementation.

GitHub PR, issue and turn dependencies (including cross-scope ones authorized by read_scopes) and scheduled holds use these subscriptions and timeout events through an extension wait translation (engineering tools, waits and playbooks). The adapter supplies the source key, types and an opaque filter. Core neither matches payloads itself nor learns a condition kind. Nor does it know a turn’s vocabulary: it publishes the turn-ended Event for any status a turn declares (execution scheduling), and what review or approved means is for the extension that waits on it or projects it. A boundary test keeps playbook states and tracker words (building, review, issues, pull requests) out of domain and internal/runtime.

Timeout templates are derived from the registration trigger, carry Wait identity and use stable firing identity. Generated timeout/retry Events use normal durable acceptance and causal limits. Timer presence is required at Wait registration, not on later historical transitions: a timer may have fired and been deleted before the wake processor runs. The runtime persists enough template/delivery identity to recognize that accepted timeout after deletion. TimeoutTimerID remains on history; it is not a live foreign-key requirement after firing.

Wake arbitration is serializable per Wait/Work:

Select the matching accepted Event with the lowest acceptance sequence, including an accepted timeout for this Wait. A timeout already accepted before a normal match wins, even if the normal match occurred earlier externally.
At Now == FireAt the timer is due. If a normal matching Event is already accepted and the timeout is not, that Event wins even at the deadline. Otherwise accept the due timeout and wake from it in one transaction. Real schedulers may accept the due timeout first; the durable sequence then decides. A clock deadline alone never wakes without an accepted Event.
Atomically change active → woken with winner Event ID, consume/cancel the timeout, remove subscription eligibility and move Work to ready. Mark/enqueue the wake input once. Replaying the winner, a loser, a stale timer fire or wake scan cannot produce a second transition. Preserve Wait history.
Cancellation first: active → cancelled, delete pending timers/subscriptions, cancel attempts and terminalize Work together. Later matches/timeouts are audit-only. Wake first: retain woken history; subsequent cancellation still fences the pending/resumed attempt and prevents future starts. Neither transition rewrites the other.
Resume-capable execution stays parked until the scheduler claims the same attempt at a higher epoch with a new token. Its original immutable TriggerID remains; pass the winner as the Resume request trigger and record wake linkage separately. Without resume, the waiting attempt is terminal and wake creates N+1 triggered by that same accepted Event. No direct executor callback, poll result or clock shortcut bypasses Event acceptance.

The production wait/timer implementation separates the actual accepted Resume request trigger from trusted causal continuation authority. A stable core.wait.woken bridge takes the winner as its parent and records the Wait and winner in its payload: a wake on an independent external root starts shallow, while a wake the attempt’s own output caused stays on its chain and deepens it. Causal refusal stops that continuation with durable diagnostics and holds the Work with the reason until an operator releases it. Ordered accumulated input is passed as RunRequest.Inputs, and recovery reproduces the pinned batch. Execution timeout measures active segments, preserving remaining budget across suspension rather than charging multi-day waiting time or resetting the full budget at each Resume.

Gated calls

A toolbelt gate (extension.Registration.Gate, toolbelt gates) can turn an extension operation call into a Wait before the operation runs. The plan is declared exactly like a translated wait, under the same fence, watermark, authority release and registration handshake, so everything above holds for it: the caller parks on the plan’s types and dependencies, a timeout is an ordinary wait timeout Timer, and exactly one wake resumes it. The gated operation never ran, so resuming is not a retry by the runtime: the caller makes the call again, and the gate decides again. Whether the gate lets it through then (an approval it has recorded, say) is the gating extension’s semantics, not the core’s. A gate’s refusal is the call’s result and changes no runtime state.

Gated starts

A start gate (extension.Registration.StartGate, start gates) can hold Work before its first attempt starts, the way a budget holds a turn. The hold is a scheduling hold, not a Wait: no execution exists to park, input keeps queueing, and nothing polls. The holding extension settles it later, from whatever it observes (a person’s answer, say), either letting the next start through once or cancelling the Work. Why a gate holds Work is the extension’s semantics, not the core’s.

Lease recovery, retry and cancellation

Lease expiry is exclusive; Now == ExpiresAt has lost authority. Expiry/release makes the same running attempt recoverable, not permission to launch another attempt. Whoever claims an attempt holds and renews its lease from the claim, through launch and observation, until the attempt completes, parks, is stopped or loses it; losing it stops the launch. Recovery claims that attempt with a higher epoch, persists recovery-in-progress authority and invokes Recover outside Tx using opaque executor metadata. A launch that lost its lease before it started anything is the one exception: still launching, nothing pinned with its segment, and either recorded as never handed to its executor or bound to a sandbox or container that holds every process until pinned. A segment that records neither is adopted. Recovery relaunches that same attempt under the new epoch. Nothing ran, so the attempt isn’t failed and no retry is spent. Competing recoverers cannot both hold a live epoch. A live lease cannot be stolen. Parked wait continuation uses Resume and is distinct from detached adoption.

Adoption succeeds only when the executor verifies the existing process/finished handle and replaces old credentials before allowing mutations. It never calls Run; only a relaunch of a launch that started nothing does. A finished detached result is committed under the recovered lease. Unknown/dead/unverifiable metadata returns ErrNotAdoptable; core records the interrupted attempt as failed under the new fence, then applies retry policy. “Unverifiable” does not establish that a process stopped: resource/process exclusion or confirmed termination remains necessary before replacement launch. Recovery crash repeats recovery of the same attempt after expiry, with another epoch. Stale results, tool calls, emissions and heartbeat renewals fail even if owner names or process IDs are reused.

Failure atomically ends the attempt, leaves Work ready, records eligibility and arms the retry Timer. The default uses three total failure attempts, waits 5 minutes after failure 1 and 10 after failure 2, then stops automatic retries while retaining nonterminal Work. With an explicitly configured four-attempt budget, failure 3 waits 15 minutes; the suite uses that budget to exercise all three backoffs. No jitter in this reference policy. The failure count is policy state, separate from monotonic Execution.Attempt. A successful continuation or explicit manual retry resets the automatic failure budget by configured policy; attempt numbering never resets. Executor rate-limit classification can select a generic retry-after hint (known reset + 2m, otherwise 30m) without adding provider fields/branches to core. Repeated policy failures must stay bounded; the runtime validates hints and rejects invalid or excessive ones rather than retrying immediately (RetryDelay in internal/runtime/scheduler_policy.go).

Backoff expiry emits core.retry.due with failed trigger ancestry; it does not Claim directly. Routing the accepted event makes Work eligible only if its failed-attempt generation is current, Work remains nonterminal and no live attempt/wait blocks it. Events queued during backoff do not implicitly clear eligibility. A manual accepted core.retry.requested Event can clear backoff/exhaustion, cancel the old timer and request the next attempt. Duplicate/stale retry Events cannot clear a later failure’s backoff. Claim is still subject to scope capacity and executor binding/resource checks. An explicit terminal-on-exhaustion policy can fail Work; subsequent retry needs a successor.

Cancellation commits cancellation of active/pending/parked attempts, Waits, timers and queue eligibility before terminalizing Work. Tokens lose authority at that commit. Interrupt/cleanup happens afterward and cannot restore authority. Retry first then cancel: accepted due Event survives as history, eligibility/attempt are removed or cancelled. Cancel first then retry: due processing cannot rearm or Claim. Recovery/adoption first then cancel: the new epoch is fenced too. Completion first then cancel: preserve the completed terminal record; cancellation never rewrites terminal Work. These orders use trusted transaction admission time and revisions, not process message timing.

Budgets

Retry bounds failures; budgets bound everything else a Work consumes. The host checks them where it admits a turn, so no prompt or playbook keeps an executor within them. Each executor reports its usage in named units on its result (usd, tokens, minutes, or its own) and core counts turns and attempts itself; core adds the amounts up per Work, per scope and across every scope, over the Work’s life or a UTC day or week, and never decides what counts as a cost. A turn does not start once a ledger has reached its limit: the Work is held with the reason and stays nonterminal, with input queueing, until an operator releases the hold (one more turn) or raises the limit. Crossing a warning threshold or a limit accepts a root core.budget.warning or core.budget.exceeded Event in the scope, which routes can act on and which never becomes the Work’s input. Execution scheduling has the mechanism.

Causal bounds and rejection evidence

Only trusted external ingress/configured independent schedule/operator activity creates roots. Executors, route-derived activity, extension operations, subscription aliases, child creation and executor-generated timers must have trusted causal ancestry. Core derives the immediate parent from the accepted trigger/authority; clients cannot select a root, sever ancestry through a child, invent a missing parent or emit a privileged core family. Child creation and its core.work.created Event are atomic. An executor-created timer inherits its arming trigger even if it fires days later. An independent configured recurring schedule may create root occurrences; an execution-created recurrence stays on its original chain. Recovery/resume retains the root rather than resetting it; a wake on independent external input is charged to that input’s root instead (#185, timers and waits).

Defaults: root depth 0, maximum derived depth 32; at most 128 new derived Events per root per fixed UTC minute (floor(trusted Unix seconds / 60)). At the cap the next new Event is rejected; the next minute restores rate capacity but never resets depth. Siblings, aliases to other scopes, child Work, builtins and timers share the root’s budget. The fixed window permits up to twice the minute allowance across a boundary; it is deliberately not a rolling-window promise. Configuration accepts only positive bounded limits (B05 settings); a child cannot raise the root’s pinned limits.

Deduplication precedes budget charging. Counters, accepted Event and associated child/route/timer-fire effects commit atomically; rejected effects all roll back. Persist root/depth and bucket counts or reconstruct from retained ancestry; memory-only counters fail restart. Compaction must preserve enough ancestry/counters to enforce pinned limits. Check limits at every derived acceptance, including FireTimer, not only the toolbelt emit operation. Arming a future timer need not spend a token before it fires, but cannot discard ancestry. Over-limit timer firing records rejection and disables that firing; no tight retry loop and no wake without an Event. Control-plane cancellation/lease fencing must still operate without needing another derived-event token.

The daemon applies the configured limits to routing, timers, control and toolbelt emissions, including translated emissions (extension.Operation.Emit). A report-about-a-report loop under issue reporting is stopped at causal_max_depth by AcceptDerivedInTx, with a durable depth_limit diagnostic and no cooperation from the extension. A burst on one chain hits causal_per_minute. See issue reporting.

Rejected derived input gets a durable, redacted diagnostic keyed by attempted delivery/parent, with reason, root/depth/budget and relevant Work/Execution/trace, without payload/secrets. It is not another recursively routed Event and consumes no chain token. Execution-facing calls fail explicitly; timers/routes stop the rejected branch. Independent roots continue. A crash between refusal and diagnostic must be recoverable through the durable originating intent/timer. The runtime writes the diagnostic to the core.causal.rejections KV namespace in the refusing transaction (internal/runtime/causal.go). The reference stores safe rejection identity/reason in KV and demonstrates bounded acceptance; it does not claim crash-safe diagnostic delivery.

Generic inject acknowledgement

This section settles how an executor acknowledges injected input and how completion races with injection. Claude Code and the shell executor are two different implementations of it.

Three Inject answers. nil means accepted for delivery, never consumed. extension.ErrNotAccepting means definitely not delivered: the process finished, was adopted without an input channel, or this executor instance does not hold it. Any other error is ambiguous. Core records not_accepting and stops dispatching that intent. The input was never removed from the Work’s pending input, so the next attempt or wake presents it. This neither infers queue (the route policy stays inject) nor drops anything. An ambiguous error keeps the behaviour in the active policy table: retry the same intent and Event identity, or apply the configured error fallback. Delivering one Event twice must not make an executor report it consumed twice.
One proof, every outcome. Result.ConsumedInputs names injected Events the process took in. Claude Code uses the stream-json echo. The shell executor uses an inbox read and acknowledged by the process. The Codex executor uses an inbox its segment acknowledges once a follow-up turn has carried the Events (Codex executor). The proof is applied atomically with the completed, handoff, waiting or failed result. Core consumes an input only when it holds an inject intent for that execution (any lease epoch) whose dispatch was attempted, its status is neither not_accepting nor already consumed, and the input is still pending. Anything else is ignored, so the proof grants no authority. The intent becomes consumed.
Completion races.
- A proof while the dispatch acknowledgement is in flight (the process echoed and the turn ended before DispatchControl committed) still consumes. The late acknowledgement keeps consumed, and replay is a no-op.
- An injection accepted too late to be taken in has no proof, stays pending and starts attempt N+1, or is the next wake’s input. This is the explicit repeat presentation.
- Completion before any dispatch leaves the input pending. The old intent then goes stale and never retargets the next attempt.
Waits. A waiting result’s proof is applied before the registration scan. The wake scan skips input the Work has already consumed. A parked turn is therefore never woken by, or handed again, what it took in. The next new message wakes it, and only that message is the input.
Failures. A failed attempt’s proved injections are consumed like its claimed inputs. A retry sees neither again; unproved ones remain for it.
Resume, N+1 and tokens. These follow the wait rules above. A resume-capable binding parks and resumes the same attempt at a higher epoch with a new token. The old token is refused, and the executor replaces any credential file before acknowledging. A binding without resume ends its waiting attempt and starts N+1 with the winner and the accumulated context (RunRequest.Inputs). No service-specific runtime code is involved.

The executor conformance suite includes conformance.CheckInjectExecutor, run by the reference inject double, extensions/shell, extensions/claudecode and extensions/codex. It has negative controls for an executor that over-claims, accepts input after its result, accepts input for an execution it does not hold, loses the proof on recovery, or never consumes. Production-scheduler races on reopened SQLite are in inject_ack_test.go. The composed daemon proof is TestDaemonSlackSupportConversation.

Pending-job coalescing and standing Work

One pending rerun is permitted while a job runs; a delivery takes the place of a waiting timer request and the newest delivery takes the place of older ones.

No core coalesce option. The queue stays lossless and ordered; coalesce remains a validation error. One pending run comes from existing semantics: one nonterminal attempt per Work, and Scheduler.Start hands one attempt every pending input. A job is one Work, so any number of triggers accepted while it runs become exactly one next run, and every trigger stays in that attempt’s consumption history.
Extension trigger selection. SchedulerOptions.WorkKinds[kind].Trigger is a pure function an extension registers for its Work kind. It picks which input of the batch triggers the attempt (Execution.TriggerID, RunRequest.Trigger), and an out-of-range answer selects the first. A wake winner always takes precedence. The schedule extension picks the newest delivery, else the earliest request.
Standing Work. WorkKindPolicy.Standing marks an objective that outlives its attempts. A completed attempt leaves it ready and idle instead of completing it. A failed attempt arms no automatic retry, because its next routed input is its next attempt. Only cancellation makes it terminal; after that, later input is terminal as for any Work.
Run-once Work. WorkKindPolicy.RunOnce marks a kind that does all its work in one attempt: nothing arrives to continue a turn it hands back. A handoff outcome on it settles as completed, so the Work completes when it has no pending input (or, if it is also standing, waits idle for its next one) instead of waiting for an event that will never come.
Route config reaches its handler. extension.RouteInput.Route is the evaluated route with its opaque handler config. One handler can serve several routes (one per binding) deterministically.
Busy jobs. The detached-busy delivery rule is a drop route selected by trigger kind (schedule route config triggers), and the busy status retry is an executor retry hint, which standing Work honours (the failure budget stays off for it). Job runs are announced as schedule.run.* Events for job: waits.

Executable cases: TestScheduleTriggersCoalesce and TestWorkKindPolicyContract on SQLite, and TestSelectTrigger. See scheduled jobs.

Cited causation

A translated emission (extension.EmitPlan) may cite a cause (EmitPlan.Cause) other than the attempt’s trigger. A fix requested by the daily triage needs it: the fix must trace through causation to the report it fixes, while the triage’s own trigger is a timer slot. The rule keeps loop-breaking a core property that needs no extension cooperation:

Eligible causes. An accepted Event in the emitting Work’s scope whose correlation key lies in the registration’s own source namespace. Citing the trigger itself is the default. Any other Event, a missing one included, is refused before anything commits.
Chain position. The new Event’s causation is the cause and its root is the cause’s root. Its depth is max(depth(cause), depth(trigger)) + 1, checked against the stricter of the two roots’ causal_max_depth. Every emission is still deeper than the attempt’s trigger, so any execution → emission → execution cycle grows in depth and stops at the limit however its emissions cite.
Budgets. The Event is charged to the per-minute bucket of the cause’s root and of the trigger’s root, and refused when either is full. A burst cannot spread across cited chains to escape the trigger’s chain budget.
Evidence. A refusal writes the same durable diagnostic (depth_limit or rate_limit, with the cited parent). TestCitedCausationKeepsCoreLimits shows a fix from a trigger at the depth limit refused although it cites a shallow report. TestCitedCauseContract covers eligible causes and the double charge.

Implementation: checkCausal/acceptCausalInTx in internal/runtime/causal.go with the trigger as a floor; AcceptDerivedInTx is unchanged (no floor).

Executable cases

Shared case	Required result
`concurrent_correlations_and_attempts`	Barrier-started same-key routes converge on one Work; competing starts produce one active attempt.
`independent_work_progress`	A deliberately blocked handler allows another key’s route and start before release.
`first_match_replay_and_drop_audit`, `multi_key_attachment_conflict_and_order`	No second handler; durable drop; atomic key append/conflict and shared owner. Storage suite additionally rejects out-of-order checkpoints.
`uncorrelated_and_terminal_successor`, `scope_route_inheritance`, `unsupported_policy_explicit_fallback`, `unmatched_and_contradictory_target`	Explicit fresh identity; immutable terminal ownership; stable override slots; unsupported controls fail or record explicit fallback.
`wait_event_registration_orders`, `wait_timeout_acceptance_tie`, `wait_cancel_orders`, `wait_watermark_and_timeout_only`	Both serial orders; already accepted match; deleted timeout still recognized; one winner; parked continuation vs N+1.
`dependency_subscription_alias`	Foreign key cannot be stolen; deduplicated causal alias wakes dependent.
`completion_inject_orders`	Accepted unprocessed input blocks terminal completion; committed injection intent retains unconfirmed input; later post-terminal input stays audited. The generic consumption acknowledgement and its races are covered by the inject acknowledgement tests.
`lease_loss_adoption_and_stale_result`	Exact expiry, higher epoch on same attempt, failed unverifiable adoption, stale outcome and post-cancel recovery rejected.
`retry_backoff_exhaustion_and_manual_event`, `retry_cancel_orders`	5/10/15 with explicit four-attempt budget, bounded exhaustion, accepted manual N+1, both cancellation orders.
`causal_depth_rate_replay_and_clock`, `concurrent_chain_budget`	Children/timer-typed derived Events share depth/rate, replay costs nothing, driver reconstruction retains bounds, next minute resets only rate.

Evidence

Run go test -race ./..., go vet ./... and go build ./.... Tests use explicit clocks and channel barriers; wall-clock deadlines only detect a stalled test.

Nineteen shared scenarios are executable. They run against the reference coordinator, against it over reopened SQLite (storage_runtime_test.go) and against the production runtime on SQLite in the runtime’s own tests (TestProductionRuntimeAllSharedCases, through runtimetest); the daemon kill drills exercise the same rules through SQLite reopen and process boundaries. Negative controls run the actual shared cases against a driver suppressing wake and one bypassing causal bounds, and require failure diagnostics. The storage suite continues to prove callback rollback, ownership, per-Work ordering and lease fencing separately. Passing the reference alone does not prove process control, throughput, durability or real service behavior.

History

01-04 recorded B04 with the shared runtime suite and the volatile reference coordinator. It delivered no production scheduler, daemon or backend; the suite was to be adapted to the real runtime in 02-03 to 02-05 and rerun through SQLite reopen and process boundaries in 02-09.
At 01-04 these were left open: coalescing (05-01), generic consumption acknowledgement (04-02), local auth and config syntax (01-05) and production instrumentation (02-08). Pending-job replacement, timer replacement and detached-busy behaviour were extension requirements for 05-01 and 05-02. Production audit persistence and export of causal rejections were assigned to 02-03 and 02-08.
The executable case table originally named the ticket that would implement each case for real: concurrency, independent progress, first match, key attachment and causal cases in 02-03 to 02-05 (with physical SQLite throughput in 02-01); configuration and fallback cases in 01-05, 02-03 and 02-04; wait cases in 02-05; alias persistence, read-reconcile and cross-scope auth in 02-05 and 03-04; real Inject/Interrupt I/O and dispatch ambiguity in 02-04; detached process and credential replacement in 02-04, 02-09 and 03-03; retry cases in 02-04 and 02-05.
01-05 (B05) specified inheritance and tombstones, pinned generations with current authorization ceilings, secret resolution and local HTTP-over-Unix-socket bearer authentication; runtime enforcement followed in 02-06.
02-04 implemented retry hint validation and executor fixtures.
02-05 added the core.wait.woken continuation bridge, RunRequest.Inputs and active-segment execution timeouts (timers and waits).
03-04 routed GitHub dependencies and scheduled holds through extension wait translation.
03-06 introduced Result.ConsumedInputs as consumption proof; 04-02 generalised it to every outcome.
03-08 applied the causal bounds to translated toolbelt emissions; from then the daemon applied the configured limits to toolbelt emissions as well as to routing, timers and control.
04-02 settled generic inject acknowledgement, the open “executor inject contract” in the planning document and the “generic inject acknowledgement and completion races” decision. The Slack support workload forced it, with Claude Code and the shell executor as two different implementations. It added no service-specific runtime code.
05-01 resolved the coalescing question 01-04 left open (“whether coalescing is needed for one pending job rerun”) without a coalesce option, and added standing Work and extension trigger selection.
05-02 settled the remaining job behaviours (detached-busy drop routes, busy retry hints, schedule.run.* Events) without a coalesce option.
05-03 added cited causation for Workflow D, the triage-requested fix.