Durable acceptance and reconciliation

Ingress is how external input becomes accepted Events in Open Conduit. It implements the acceptance parts of B02 storage, B03 ingress and B04: every push, reconcile page, timer firing and toolbelt emission is accepted through one transactional boundary, deduplicated by delivery identity and left unrouted for the router. The implementation depends on public contracts only and has no service, Git, SQLite, config-loader or reference-double dependency. Ingress never launches an executor or changes Work state: in the daemon, the host loop in internal/bootstrap drives ReconcileDue and hands Pending input to the router. SQLite runtime tests exercise the production boundary.

One acceptance boundary

NewIngress(store, now, options) takes a public storage.Store and trusted clock. Accept(ctx, targetScope, event) returns an accepted envelope only after commit. AcceptInTx(tx, targetScope, event) is the transaction-level form that timer, toolbelt, manual and introspection producers use to join event acceptance to KV, Wait or outcome changes. Its returned record is not a durable acknowledgement until the caller’s outer transaction commits. These are trusted internal entrypoints, not unauthenticated management or execution-facing APIs; callers must validate declarations, authority and derived causation/budgets at their own public boundary.

Acceptance preserves the first envelope using B02 (source, delivery_id) deduplication. Internal IDs and receipt timestamps may differ on retries. Trusted target Scope metadata is persisted separately and atomically with acceptance, because domain.Event has no Scope field. Explicit Work, Execution’s Work, owned correlation and requested target Scope must agree; contradictory identity is rejected. Scope assignment cannot change on duplicate delivery. Events and route processing metadata remain distinct: accepted input starts unrouted and only a later processing transaction may join its effects to MarkRouted.

Pending(ctx, afterSequence, limit) scans unrouted storage Events, including raw atomic storage.FireTimer acceptance, so its cost follows the unrouted backlog rather than retained history. limit is 1–200 examined rows, not a promised number of pending results. It returns unrouted entries, their original Scope, a continuation sequence and More; start another traversal at zero to find older still-pending input. It derives Scope from explicit Work/Execution/correlation for raw accepted events without acceptance metadata. Truly unbound input has unbound_scope; later contradictory ownership has identity_conflict while preserving the recorded target Scope. Such a row remains visible for operator diagnosis and does not hide independent later Events. Backend failures still fail the read. No fallback Scope or domain routing decision is invented.

The lost-response proof kills a subprocess after Accept commits and before any external acknowledgement. Reopen finds the unrouted input and an identical delivery retry returns the original record without another Event. Marking its processing checkpoint removes it from pending results without changing the envelope.

Webhook delivery IDs are not signed. GitHub (X-GitHub-Delivery), Linear (Linear-Delivery) and Bitbucket (X-Request-UUID) send the delivery ID in a header their signature doesn’t cover, so someone who captured a signed delivery can send it again under a fresh ID and dedup won’t catch it. The harm is small: a webhook only says what changed, and Push reads the issue, PR or comment back from the provider, so a replay reports current state, not the captured state. Where the provider signs a timestamp, the extension refuses old deliveries: Slack checks X-Slack-Request-Timestamp (five minutes) and Linear the body’s webhookTimestamp (ten minutes, required). GitHub, Jira and Bitbucket sign no timestamp, so their replays are bounded only by that read-back.

Registration, canonical identities and push

Register(ctx, name, source, adapter, interval) validates declared event families and persists source ownership before enabling the in-process registration. The source namespace and configured Scope are stable across restart. A namespace cannot silently acquire another Scope (ErrIngressNamespaceOwned, diagnostic ingress_namespace_owned), and a name cannot silently select a new namespace (ErrIngressNameTaken, ingress_name_taken); both also match extension.ErrConflict. Registering the same namespace and Scope under a new name is a renamed configuration entry: the stored source, checkpoint and history move to the new name and the old name is released. Configuration publication/removal and deliberate ownership moves belong to the B05 configuration loader. Registrations are immutable for one runtime instance; reconstruct and register the same identities after restart.

Every family must be declared recoverable or include a concrete loss limitation. Ingress cannot declare core.* families or use reserved core.* source namespaces. Push calls adapter normalization outside storage, validates the whole batch and accepts its recoverable observations and explicitly non-recoverable direct Events in one transaction. Any invalid member rolls back the entire batch, including canonical identity ownership. Direct Events cannot masquerade as a recoverable observation family. Provider errors return the safe sentinel ErrIngressProvider; credential/provider error text is not persisted or exposed.

For each source, the core retains separate immutable coordination entries linking (item_id, version) and (event_source, delivery_id) to a canonical identity signature. That signature covers the normalized opaque envelope except internal Event ID and actual receipt time. A repeated item/version cannot change delivery identity or canonical content, and two logical item/version changes cannot claim one delivery identity. Provider normalization still owns stable external occurrence time, payload, family and correlation. The core hashes their opaque representation and never interprets domain fields. Identity entries remain while their Events can be retried; there is no second event log and no silent identity-expiration policy.

Versions are equality tokens, never ordered counters. Push accepts canonical input and records identity but does not update either the complete authoritative snapshot or a staged reconcile generation. A late push therefore cannot rewind a complete snapshot. Push during a scan cannot rewrite its staged pages. Delivery dedup preserves the first accepted envelope on overlap; a subsequent complete reconcile supplies the authoritative source view. Reconciliation is not a total order of external history and cannot recover transitions a provider no longer exposes. Providers must distinguish repeated A → B → A occurrences with different version identities, or expose canonical tombstones when exact deletion history matters.

External extensions’ ingress runs through the same Ingress.Push/Reconcile path. A crash mid-push accepts nothing, and a crash mid-page advances nothing. Because canonical identity binds (item, version) to the envelope, a release that changes payloads must mint new observation versions and delivery IDs (schema evolution); otherwise every page fails with a deterministic conflict.

Transactional snapshot generations

Reconcile(ctx, name) makes exactly one external page call, even for a multi-page source. It first acquires a durable unpredictable scan lease, then calls the adapter with saved source/generation/cursor/checkpoint outside storage. Commit rechecks the lease token and a freshly sampled trusted time; an expired or superseded holder returns ErrScanLeaseLost and cannot publish its stale response. Another worker may recover the source after lease expiry. Page calls have a bounded context deadline; adapters must honor cancellation.

Each source stores complete progress separately from active generation progress. A request for a new generation receives the last complete checkpoint. A resumed generation receives its durably staged page checkpoint. Returned pages must retain one nonempty ScanID, avoid repeated cursors, and cannot claim completion while providing another cursor. Duplicate items with conflicting versions fail without poisoning prior staging. All item/family/envelope checks and canonical identity claims join staging and its page cursor transaction. Failed pages advance neither cursor nor complete checkpoint.

Incomplete pages stage full observations in separate per-item KV entries. They emit no changes or absence. On the final exhaustive page, one transaction diffs the entire staged generation against per-item authoritative heads, accepts changed canonical Events, accepts generic absence Events, publishes the new heads, advances the complete checkpoint and removes staging. An unchanged snapshot emits no new change. An exhausted partial scan (Complete=false, no next cursor) discards only its staging and retries later; the previous complete snapshot/checkpoint remain authoritative. A process crash leaves either the prior page state or all of the committed page; completion cannot publish half a diff.

Absence is core.ingress.absent, sourced from the reserved core.ingress namespace. Its stable delivery identity hashes source namespace, scan generation and item ID; its payload is core.ingress.absent.v1 with source, item and last_version. It preserves the last canonical observation’s CorrelationKey and explicit WorkID, since provider item IDs need not equal domain correlations. It does not retain an old ExecutionID, invent an external deletion timestamp or alter/finish Work. Pure routing can find the existing objective through the same correlation path; route handlers decide its significance.

If a provider can no longer resume an opaque generation/cursor, RestartScan(ctx, name, expectedScanID) is the explicit recovery operation. It rejects a stale expected generation or live scan lease, discards only staging/cursors, preserves accepted history, canonical identities and the complete snapshot/checkpoint, and makes a fresh scan due immediately. No destructive KV edit is required. A caller disconnected during a page call may leave its scan lease until expiry; this remains recoverable without stealing a live call.

Bounded scheduling and inspection

ReconcileDue(ctx, limit) performs at most 1–200 external page calls. Each registered source appears at most once in one poll. Durable next-due times survive restart. Completed scans schedule the next interval, unless an operator reconcile request (open-conduit scopes reconcile) committed after the generation’s first page was claimed: then the source stays due, so the request is answered by a generation that starts after it. Failed/partial scans use a deterministic 5-second exponential backoff capped by the configured interval. Retry after the same partial page does not reset complete progress. There is no sleeping goroutine, network loop or scheduler timer in this package: the daemon’s host loop drives bounded polls with its trusted clock. Configured intervals range from one second to one year.

A durable lexical round-robin cursor makes progress fair across sources, including under a fake clock that remains fixed between polls. An invalid/unavailable source cannot consume an unbounded number of page calls or starve independent sources. Active lease contention skips that source. Non-retryable storage/context errors still return explicitly rather than becoming a successful scan.

Default limits are 1,000 observations/direct Events per page, 10,000 staged items, 1,000 pages, 8 MiB encoded page/batch and 32 MiB staged observation bytes. ScanID, cursor, checkpoint, item and version tokens are bounded to 4 KiB; registration names to 128 bytes and source namespaces to 1 KiB. Options permit explicit bounded adjustments. Identity/history entries and authoritative heads are per item/version rather than an ever-growing serialized source blob; no artifact/log content enters coordination KV. The implementation enumerates bounded staging during publication and makes no high-throughput claim.

Inspect provides sorted source/config/health records: source identity/Scope, declared families and loss limitations, complete checkpoint, active generation/cursor, due time, safe last-error code, the provider’s failure behind a provider_failed code, last completion and LimitedRecovery. These metadata must be credential-free at their adapter source; bodies, opaque observations, secrets and raw provider errors are absent.

A failed page call keeps what the provider reported, so a failing source links to its cause. Its class is timeout (no page within the page timeout, 30 seconds by default), cancelled, transport (the package’s connection closed), protocol (a refusal with one of the protocol’s own codes, such as overloaded), provider (the provider’s own authored code, an extension.OperationError) or unauthored (an error the provider did not author). With it go the code and a message of at most 256 bytes on one line. Both come only from authored errors (protocol.Encode), which are secret-free by contract; an unauthored error’s own text is never recorded, and stays in the extension’s log. The next successful page clears it with the code. open-conduit doctor presents them through its reconcile_lag check and the per-integration checks (doctor). Ingress state transitions are reported through the lifecycle instrumentation described in observability.

Evidence

Run the ingress integration cases with go test -race -count=1 ./internal/runtime -run 'Ingress'. The SQLite fixtures demonstrate reconcile-only convergence, complete/staged checkpoint reopening, repeated/partial/mixed generations, push/reconcile/late-delivery dedup, source/scope isolation, declaration/canonical identity rejection, page-lease fencing, pending identity-conflict isolation and post-commit/pre-ack process kill recovery. B02 weakened-backend controls still run unchanged.

History

Ticket 02-02 implemented durable acceptance, push, reconciliation and inspection. Routing/dispatch, derived-event budgets and execution lifecycle were left to 02-03 onward. It claimed no real provider, webhook transport, router, executor or ownership transfer, and its verification ran make check and GOTOOLCHAIN=go1.24.2 make check alongside the ingress integration cases.
At that stage AcceptInTx had no callers outside ingress; it was provided for later timer, toolbelt and manual producers. Configuration publication and ownership moves were assigned to 02-06’s B05 loader. Runtime management/CLI presentation of Inspect was assigned to 02-06/02-08, and lifecycle OpenTelemetry/export delivery to 02-08; the 02-02 tests proved durable transitions, not instrumentation coverage.
Ticket 05-04 made an operator reconcile request keep the source due until a generation that starts after the request completes.
Ticket 07-02 confirmed that external extensions’ ingress runs through the same Push/Reconcile path and added the payload schema evolution rule.