Execution scheduling, recovery and control

The scheduler starts execution attempts for ready Work, recovers them after lost authority, applies retry policy and dispatches inject/interrupt controls. It implements B04 execution policy over the public storage and executor contracts. The scheduler, retry policy and control dispatch compose with the production router. They contain no process, service, SQLite or configuration-loader dependency. A host drives these operations: in the daemon, the host loop in internal/bootstrap starts ready Work, observes attempts, renews leases, dispatches controls and confirms physical stops (daemon).

Starting attempts

NewScheduler(store, now, router, options) takes a trusted clock, owner, configuration generation and mechanically validated executor registrations. Defaults are four concurrent attempts, a one-minute lease, a one-hour execution timeout and three total failure attempts. Scope limits apply to the selected scope and its ancestors. Constructor options are copied. Binding inheritance resolves before claim; the commit rechecks Work, input and Scope snapshots.

Claim atomically selects ordered pending input, creates the next monotonic attempt, claims its lease, pins behavior and records consumed-input history. The durable launch intent precedes Run. Launch then launches the claimed attempt outside storage, under the claim’s lease; Start is the two together. A host that must not block on a launch claims on its pass and launches elsewhere, holding the lease from the claim with Hold(ctx, lease, ticks, report). Hold renews the lease on every tick (a host ticks every third of the lease duration) until it is stopped or a renewal is refused. When the lease is lost, the returned context ends with cause ErrLeaseLost (it matches storage.ErrLeaseLost), so a launch under it stops at once rather than at its next fenced write. One holder holds an attempt’s lease from its claim, through its launch and observation, until it completes, parks, is stopped or loses the lease. A launch that takes longer than the lease therefore keeps it. Executor calls, capability methods and retry-hint callbacks run outside coordination transactions; a blocked executor cannot retain the storage lock. No process-local global lock coordinates claims. Real serializable storage prevents competing schedulers from creating simultaneous attempts or exceeding configured capacity.

The attempt snapshot retains its configuration generation, full binding, executor registration, capabilities, retry policy, timeout deadline, control error fallback and handoff policy. HistoricalExecutors[generation][registration] must reinstall the exact older implementation for recovery/control after reload. A missing generation is unsupported; it never selects a replacement executor silently. Future attempts use current validated bindings and policy. Secret resolution and authorization ceilings belong to the configuration composition, not the scheduler; secrets are not part of durable attempt snapshots. The same goes for the run environment layer: SchedulerOptions.Environment asks the extensions of the attempt’s scope for it (extension.RunEnvironment) before Run and Resume, never for a recovery, and puts it in ExecutionContext.Env. It is never persisted, and a failure fails the attempt as run environment unavailable (ErrEnvironmentUnavailable) before the executor is called.

A binding may set timeout_seconds, which replaces SchedulerOptions.Timeout for its attempts (pinned with the binding in the attempt state; a resumed segment keeps its remaining budget). When a Work opted into binding grants, the binding the attempt pins already has their config applied, and the attempt state’s Grants names the ones it took, so recovery and resume keep them.

SchedulerOptions.WorkKinds registers an extension’s lifecycle policy for the Work kind it creates. Standing Work returns to ready (idle) after a completed attempt and arms no automatic retry after a failed one; RunOnce Work does all its work in one attempt, so an attempt that hands back settles as completed; Trigger selects which input of the pending batch triggers the attempt. Scheduled jobs use both for the one-pending-run rule (runtime semantics).

Attempt segments

The attempt state keeps the attempt’s segments, the stretches it held a lease, measured exactly as the timeout budget measures them. Start begins one in its claim transaction: the first start, or a resume of a parked attempt, carrying the wake’s event ID when a wake started it. A resume wait’s registration ends the open segment in the park transaction, where the remaining budget is stored. Recover ends the open segment and begins one marked recovered at the same instant, with no gap. Recovery keeps the deadline, so the budget charges the time between a lost lease and its recovery, and the segments charge it too. This holds whether the lease expired after a crash or Release cleared it in a graceful stop. A timeout processed after the lease expired recovers first, so it leaves a zero-length recovered segment. Every other end, a non-resume wait included, is the execution’s terminal finish, which the history record uses to close the last segment. While an attempt is parked, its ended segments and its remaining budget sum to its timeout.

Each segment records its launch, phase by phase (Segment.Launch, at most MaxLaunchPhases): toolbelt, secrets, environment, sandbox or container (the hook binding it), and executor (the executor’s own launch). The steps the executor’s launch runs inside it are recorded too: workspace (the daemon’s resource broker acquiring it), sandbox plan, container plan, container create (which pulls the image when it isn’t there) and the sandbox pin or container pin. Each phase is written, fenced by the lease, as it begins and again as it ends, with its outcome (ok, failed or cancelled), so a slow phase shows while it runs. A write the lease refuses only leaves the trail short. The hooks reach the recorder through runtime.LaunchPhases(ctx) on the context they are bound with, and the extension’s container and sandbox bindings take it as Phase. A segment also records how far its launch got towards its executor (Invoke). The claim that begins it sets pending. The fenced write just before the executor is called sets called, together with whether a sandbox or container was bound first (Isolated).

An attempt keeps its last 50 segments (MaxSegments), with the count and active time of the dropped ones accumulated. An attempt started before segments were recorded gains none, so its history shows no segments rather than a partial record. Segments are lease-held time, not proof that a process ran: an adopted detached process may have run through the gap, and an attempt that crashed between claim and launch held a lease without running.

Recovery and relaunch

Start and Recover return an Attempt containing its lease, request and handle. An executor must successfully use the fenced PersistRecovery callback before returning a successful handle. The scheduler validates bounded opaque JSON recovery metadata and checks current authority around launch acknowledgement. Missing acknowledgement or an ambiguous operation records a safe failure and retains resource exclusion. Executors own closing the process launch/persist crash window through deterministic discovery, prelaunch intent or cleanup, as required by B03.

Recover claims the same nonterminal attempt at a higher epoch and persists recovery-in-progress before calling Recover outside storage (ClaimRecovery then Launch; PollClaims is Poll for a host that launches recoveries itself). It never calls Run to replace a process. The exception is an attempt whose launch lost its lease before it started anything. That attempt is still launching (or resuming), and nothing is pinned with its open segment. Either its executor is known never to have been called (Invoke is pending), or it was called after a sandbox or container was bound (Isolated), which holds every process until it is pinned. A segment with no Invoke, such as one an earlier release began, proves neither and is adopted. Such an attempt is relaunched: the recovery’s segment is marked relaunched, and Launch calls Run (or Resume) again under the new epoch. A relaunch fails nothing, uses no retry and waits out no backoff. Losing the lease was the host’s failure, not the execution’s. The old launch can’t do harm: its fenced writes are refused, a refused container pin removes its own container, and the new launch’s binding removes any stopped container the execution left. A launch that pinned a container or sandbox is adopted as before. So is any unisolated launch whose executor was called, whether or not it recorded metadata: the executor contract lets it start its process before persisting, so it may be running, and adoption fails closed when it can’t prove otherwise. A timeout that recovers authority to time an attempt out never relaunches it. An unapplied toolbelt declaration carries over to the new epoch. Renew/release use the exact lease; release makes that same attempt recoverable. Renew sets the expiry to now plus the lease duration unless the stored expiry is later (the clock stepped back), which it keeps: a renewal never shortens a lease, and the storage conformance case lease_renew_survives_clock_step_back holds every backend to it. Adoptable(lease) tells a host whose lease was lost whether the attempt still belongs to recovery: its Execution is running and its Work open, at this epoch or a newer one. The daemon stops a process after lost authority only when it is not. Old outcomes, metadata callbacks and renewals fail after expiry, release, cancellation or a newer epoch. Observation cancellation cancels Wait observation only and does not assert that a detached process stopped.

Lease fencing and physical resource exclusion are separate. Durable attempt holds survive restart, including terminal attempts with uncertain external processes, and count against Work/global/ancestor capacity. ErrNotAdoptable, transport errors, timeout and a successful interrupt request do not prove that resources stopped. Replacement remains blocked until the trusted host supplies ConfirmStopped(executionID, expectedEpoch). A successfully observed structured terminal result establishes finished attempt resources. Confirmations cannot target an attempt that has since been adopted at another epoch. A binding’s sandbox (B09) is bound by SchedulerOptions.Sandbox beside the resource; its plan’s digest is pinned with the open segment (Segment.Sandbox) through a lease-fenced callback, and a recovered segment carries it. The generic scheduler does not inspect process IDs or executor metadata: the shell executor supplies verified-stop evidence through Stop(execution, epoch), and the daemon renews observed leases and performs that stop/confirm after lost authority to an ended attempt, observation failure or a failed recovery. A lease that merely lapsed on a running attempt is left for recovery to adopt.

Outcomes, handoff and retry

Observe waits outside storage, then applies a lease-fenced structured result. Completed Work cannot overtake accepted unrouted input; queued or unconfirmed injected input leaves it ready for N+1. Results preserve common outcome, usage and artifact-reference history. Failed attempts return Work to ready with bounded retry eligibility, unless an explicit terminal-on-exhaustion policy fails Work. An executor whose handle fails with an extension.OperationError leaves its code and message on the failed attempt’s summary (executor observation failed: <code>: <message>, runtime.FailureCause), and the host’s observe diagnostic logs it as cause; any other executor error’s text is kept out, since it may carry provider output. Timeout requires the persisted deadline to be due and current lease authority, then records failure and retains the uncertain resource hold. A host must recover expired authority before applying such a mutation; a clock deadline does not grant a stale lease authority.

An outcome whose data declares a turn (core.turn.v1 {status, comment}, status any word of the right shape: a lower-case letter, then up to 63 lower-case letters, digits, - or _) publishes a core.execution.turn Event in the transition that stores it, whatever the outcome kind: one per correlation key of the Work (one without a key for keyless Work), derived from the attempt’s trigger, with payload core.execution.turn.v1 {status, work, execution} and delivery turn/<execution>/<lease epoch>[/<key>], so a repeated transition is a no-op and a resumed attempt’s next turn is a new Event. It is recorded as unmatched, never input: another Work’s dependency subscription on the key is what it is for, and, like every Event a Work’s own execution published, it never wakes that Work’s Wait (timers and waits). The core knows no status vocabulary: building, review, help, done and finished are the work-issue playbook’s words, which the Claude Code executor offers by default and the tracker and GitHub projectors map to labels and holds. A shell job or a person can end a turn in a status of its own, approved say, for another Work to wait on. Outcomes stored under the source-named <source>.status.v1 of earlier releases are rewritten to core.turn.v1 the first time the SQLite store opens (a metadata key records it, so later opens skip the scan), and nothing reads the old name any more. Causal refusal skips it and keeps the outcome.

Handoff ends the attempt and leaves Work ready for eligible input. Automatic continuation is off by default. If explicitly enabled, a stable derived core.execution.handoff Event and queue checkpoint join the outcome transaction. Causal refusal records a diagnostic and stops that continuation branch while preserving the handoff outcome. A bare waiting outcome cannot bypass event-based suspension; predicate registration, structured waiting results and fenced acknowledgement are described in timers and waits.

Failures arm a durable retry Timer template with the failed attempt generation and trigger ancestry. Default delays are five and ten minutes; an explicit four-attempt policy adds fifteen minutes. Failure counts are separate from attempt numbers. Up to 100 configured delays yield 101 total attempts, matching the B05 schema. Generic retry hints select reset time plus two minutes, or thirty minutes for an unknown reset; invalid/past/excessive hints fail explicitly. Hinted failures still consume the bounded failure budget. Provider classification stays outside core.

A failed result marked extension.Result.Permanent is one the executor knows trying again cannot cure, such as a binding whose resource is misconfigured. It arms no retry Timer and exhausts the failure budget at once, with BlockedReason permanent_failure (telemetry’s retry.blocked_reason, which a manual retry clears), so the Work is exhausted (failed, with TerminalOnExhaustion) at its first attempt rather than after the delays run out. A manual retry still runs once the config is fixed. Standing Work arms nothing for a hintless failure anyway. Shell, Codex and Claude Code mark invalid_config and a workspace refused as extension.ErrInvalid this way. TestSchedulerPermanentFailureArmsNoRetry covers it, with an unmarked failure arming its first delay as the negative control.

An executor’s retry hint also applies to standing Work: a hinted failure arms the retry timer and blocks scheduling until it fires, without exhausting the Work, while a hintless failure of standing Work arms nothing. A hint’s delay does not index the failure-budget delays, so a standing Work with many failures cannot overrun them.

Executor errors implement the optional public extension.RetryHintProvider; they do not import runtime implementation types. Complete/Fail accept a trusted host’s finished structured result. A toolbelt outcome or wait declaration is recorded under the current fence but applied only when the host observes the executor finishing or suspending that segment through Complete, so process/resource exclusion is established before replacement capacity is granted; a client declaration alone is not stop evidence.

ProcessRetry consumes an already accepted trusted retry Event. The timer driver must first accept through AcceptDerivedInTx and atomically remove/fire the timer. A due clock alone never clears eligibility. core.retry.requested (B04) and core.manual.retry (B05) are manual aliases. The event must follow the current failure’s acceptance-sequence boundary, and a processed Event cannot clear a later failure. Manual retry resets the automatic failure budget, not attempt numbering. Old due generations, terminal Work and replay remain historical; live attempts/Waits prevent retry. Ordinary queued input during backoff does not clear it.

Control dispatch

DispatchControl claims a durable intent, performs Inject/Interrupt outside storage, and acknowledges only with the same live dispatch token. It uses the pinned attempt executor, capabilities, target and epoch. A new executor registration cannot grant control capability to an older attempt. Inject acknowledgement means accepted, not consumed: pending input remains eligible for subsequent presentation. Ambiguous delivery retries the same intent/Event identity, with no exactly-once transport promise. An explicit pinned queue/drop error fallback is recorded; provider error text is not persisted. Late acknowledgements, completed targets and superseded epochs cannot retarget a newer attempt. An ErrNotAccepting answer settles the intent as not_accepting (no retry, input retained). An intent for an attempt still launching, resuming or being recovered waits, undispatched, until a later pass finds it launched. A result’s ConsumedInputs, on any outcome and even while the dispatch acknowledgement is in flight, settles it as consumed (generic inject acknowledgement).

Interrupt fences the old attempt and leaves Work ready before sending control, but retains its resource hold until stopping is verified. Work cancellation atomically fences attempts, Waits, owned timers and queue eligibility through the router. Physical cleanup follows that commit. The host must not treat acknowledgement of a stop request as proof of resource exclusion.

Capacity pools

Capacity is split by executor, so scheduled jobs and agent turns do not take slots from each other. SchedulerOptions.ExecutorPools gives an executor registration its own pool, bounded by PoolLimits[pool]; an executor without a pool counts against GlobalLimit. An attempt counts only against its own pool, and Start refuses with a CapacityError (it matches ErrCapacity) naming the full pool, or the scope for a scope limit. Scope limits still count every pool.

The job executors (shell, github-hold) are in the jobs pool, bounded by settings.job_concurrency (default 4). Each executor declares its pool where it is registered (inproc.Executor.Pool, an installed package’s manifest executors[].pool), so an installed executor can be a job too. settings.concurrency (default 4) then bounds agent turns (claude-code) and every executor that declares no pool. With the agent limit reached a due job still starts, subject to job_concurrency, and the reverse holds. A job and a turn can run at once up to the sum of the two limits. A config that relies on concurrency to bound jobs as well should add job_concurrency. An agent job (a claude-code binding selected by a schedule route) is an agent turn and counts against concurrency.

A scope’s limits.concurrency narrows this further: the daemon passes each scope whose limit is below its parent’s (or settings.concurrency at the root) as ScopeLimits, and the scheduler checks the attempt’s scope and every ancestor, counting attempts from both pools.

An attempt holds a slot while it is unfinished and not parked, and also after it ends until its resource exclusion is confirmed. The capacity check reads only those attempts: the unfinished ones (storage.Tx.LiveExecutions) and an index of attempts with unconfirmed exclusion (core.scheduler.unexcluded), which every attempt-state write keeps current. A store written before the index existed is indexed by the first Poll; until then the check reads every attempt.

A start held back by a limit is reported with the code of its setting: capacity_concurrency, capacity_job_concurrency or capacity_scope. open-conduit doctor has a capacity check: each pool’s running attempts against its limit, and each ready Work waiting for one with the limit holding it back. The Work’s detail (GET /v1/work/{id}, the CLI and the TUI) carries a waiting annotation with the same limit. GET /v1/capacity gives the same accounting to clients: for each pool (agents for settings.concurrency, jobs for settings.job_concurrency), its limit (zero: unbounded), the attempts holding a slot (in_use) and the ready Work its limit held back (waiting). Work held back by a scope limit counts in no pool. in_use is read from storage on each request. waiting is the daemon’s process-local result of its last start pass, so it reads 0 after a restart until the first pass, and for one pass it can disagree with in_use. Ready Work skipped as idle (no pending input) is cleared from it.

Budgets

Budgets cap what Work consumes, in named units, and Start enforces them in its claim transaction, after the capacity check (SchedulerOptions.Budgets, from settings.budgets; the guide covers configuring them). A runtime.Budget is at most Max of Unit per Work, per scope or across every scope (Per), over the ledger’s life or each UTC day or week (Period), optionally narrowed to one scope (ScopeID, which config refuses on a per: all budget, since that ledger is shared by every scope).

Usage is reported, not interpreted. Core counts turns (every start, a resume included) and attempts (every new execution) itself. Every other unit comes from the executor’s Result.Usage: Usage.Units names any amounts it reports, and Usage.Amounts() adds the cost in its currency’s lower-case code and the input plus output tokens. ValidateResult refuses a unit outside extension.UnitPattern, a negative or non-finite amount, and the reserved turns and attempts, as units or as a currency. Claude Code reports usd and tokens; the shell executor reports the segment’s wall time in minutes. Out-of-process executors carry units on the wire result.

putAttemptState stages the part of an attempt’s usage that is not yet staged (AttemptState.Charged, compared with the stored state, so a stale copy never charges twice) in the core.budget.pending namespace, so a finish, park, failure or timeout each charge it once. Staged usage is charged to the ledgers (core.budget.ledger, one record per Work, scope or everything and window) at the next Start or Scheduler.SettleBudgets, which the daemon runs every pass; without budgets it is dropped. A ledger counts from the time its budget is first configured.

Start refuses a turn when an applicable ledger has reached its Max. The refusal commits a hold of kind budget (Hold.Kind) with the reason, and a core.budget.held control Event addressed to the Work, then returns ErrBudgetHeld (it matches ErrNotEligible; the daemon reports over_budget). Input keeps queueing while it is held. Releasing a budget hold (Control.ReleaseWork) leaves a pass that covers the first admission after it, over budget or not, so a pass is never left over for a later limit; the turn after is checked again. An admitted turn is charged one turns, and one attempts when it creates an execution.

A charge that crosses a budget’s WarnAt fraction or its Max accepts a routable root core.budget.warning or core.budget.exceeded Event (schema core.budget.v1, runtime.BudgetNotice) in the Work’s scope, once per ledger window, kind, limit and scope (a per: all budget notifies each scope in its own scope). It names the Work in its data instead of addressing it, so it never becomes the Work’s input. The hold path accepts the exceeded notice too, for a limit lowered below what was already used. open-conduit doctor has a budgets check listing held Work with its reason, and held Work shows in the web view’s Needs you list, the TUI and GET /v1/work/{id} (hold.kind is budget).

Start gates

A toolbelt registration can carry a start gate (extension.Registration.StartGate). Before the first attempt of a Work starts, and only once capacity allows that start (asking can have effects, such as a request a person sees), Start asks every registered start gate, in registration-name order and outside any transaction, with the Work, the binding the attempt would run on and the Event that would be its first input. Later attempts of the same Work are not asked. A gate answers in one of three ways:

Let it start. The next gate is asked, and once all have, the start goes on to its capacity and budget checks as usual.
Hold it (extension.StartHold{Reason}). In the claim transaction, after the capacity check and before a budget counts the turn, the start commits a hold of kind gate (Hold.Kind) naming the registration (Hold.Holder) and its reason, and a core.gate.held control Event addressed to the Work. It then returns ErrGateHeld (it matches ErrNotEligible). Input keeps queueing while it is held, and held Work is not asked again.
Fail to decide (an error, or a panic). The start returns ErrStartGate and nothing is held, so the next pass asks again: the host fails closed.

Only the holding registration settles its hold: Control.SettleGateHold, which an extension reaches as inproc.Host.SettleHold (work.settle_hold for a package). Admitting it deletes the hold and leaves a pass, so the next start skips the start gates once. The pass is spent only by a start that is admitted: a budget that holds that start leaves it for the start the budget’s release lets through. Refusing it cancels the Work. Either way a core.gate.released or core.gate.refused control Event records the settlement with its reason, idempotent per key. An operator’s ReleaseWork refuses a gate’s hold with ErrGateHold. They can cancel the Work instead. The core knows nothing of why a gate holds Work. The human extension’s approval gates on kinds and bindings are the first (human decisions).

Evidence

SQLite scheduler tests run eleven applicable shared runtime cases through the production scheduler with persistent and independently reopened handles. A small explicit timer-contract adapter accepts the retry Event and fires its stored timer for these tests. Additional cases cover blocked Run progress, global/scope/ancestor capacity, same-attempt recovery, stale callbacks, uncertain-launch restart, resource holds, cancellation orders, inject/completion races, interruption, deadlines and handoff. Policy regressions cover generation-pinned retry/capabilities, missing launch acknowledgement, retry bounds and explicit control error policies. TestProductionSchedulerExecutorPools and TestDaemonJobConcurrency cover capacity pools. TestProductionSchedulerBudgets covers budget warnings, the hold, its reason and the one-turn release, with TestProductionSchedulerBudgetsOffByDefault as its negative control. TestProductionSchedulerStartGateHoldsTheFirstStart covers a start gate’s hold, its history, that only its registration settles it and that the settled start runs without asking again. TestProductionSchedulerStartGatePassSurvivesABudgetHold and TestProductionSchedulerStartGateWaitsForCapacity cover the pass outliving a budget hold and the gates waiting for capacity. TestProductionSchedulerStartGateRefusalCancels covers refusal, and TestProductionSchedulerStartGateFailsClosed covers an undecided gate, with ungated Work starting as its negative control. Launch regressions cover the lease across a slow launch. In TestProductionSchedulerHeldLaunchOutlastsItsLease, a container that takes three lease durations to create keeps its lease under Hold and runs; its negative control loses the lease at the pin without a holder. TestProductionSchedulerRelaunchesALaunchThatPinnedNothing covers the relaunch of a launch that lost its lease before its pin or before its executor: not failed, no retry spent, the old epoch’s container removed. Its negative controls are all adopted: a lease lost after the pin, an unisolated launch that recorded metadata, one whose executor started its process before recording any, and segments with no Invoke, as an earlier release wrote them. TestProductionSchedulerRecordsLaunchPhases covers the phase trail, and TestProductionSchedulerHoldEndsOnLostLease covers the hold’s cancellation. TestStartReadyHandsLaunchesOffTheLoop shows the daemon’s start step returning, and the next Work launching, while another launch is stuck.

History

Ticket 02-04 implemented the scheduler, retry policy and control dispatch. The daemon, authenticated toolbelt, durable timer driver and shell process implementation were left to later tickets; secret resolution and authorization ceilings were assigned to 02-06. Its tests used a small timer-contract adapter because the production due-event driver arrived in 02-05, and claimed no real process, daemon crash drill, service integration or hosted CI execution.
Real shell cleanup, adoption and resource evidence were assigned to 02-07 and 02-09. 02-07 added the shell executor’s Stop(execution, epoch) evidence; 02-09 added daemon lease renewal and stop/confirmation, and made Recover carry an unapplied toolbelt declaration to the new epoch.
Ticket 02-05 supplied wait predicate registration, structured waiting results and fenced acknowledgement.
Ticket 04-02 added the not_accepting and consumed inject settlements.
Ticket 05-01 added SchedulerOptions.WorkKinds (standing Work and trigger selection).
Ticket 05-02 added binding timeout_seconds and retry hints for standing Work; before it, a hint’s delay indexed the failure-budget delays.
Capacity pools (#13) replaced a single settings.concurrency bound on every running attempt, under which two long agent turns held every due job back and two long jobs held back agent work. Existing configs kept working unchanged.