How a write becomes durable

Open-time recovery sweep

The staged-write rewire eliminates one drift class by construction at the writer layer: an op that fails before pushing to the in-memory accumulator (validation errors, missing endpoints, parse-time D₂ rejection) leaves Lance HEAD untouched on every staged table. This is the case the partial_failure_leaves_target_queryable_and_unblocks_next_mutation test pins.

A second, narrower drift class — the finalize → publisher window — is closed across one open cycle by the open-time recovery sweep:

MutationStaging::finalize runs stage_* + commit_staged per touched table sequentially, then the publisher commits the manifest. Lance has no multi-dataset atomic commit, so the per-table commit_staged calls are independent operations: if commit_staged on table N+1 fails after commit_staged on tables 1..N succeeded, or if the publisher's CAS pre-check rejects after every commit_staged succeeded, tables 1..N are left at Lance HEAD = manifest_pinned + 1.

Recovery protocol (lifecycle of every staged-write writer — MutationStaging::finalize, schema_apply::apply_schema_with_lock, branch_merge_on_current_target, ensure_indices_for_branch, optimize_all_tables):

1. Phase A: writer writes a sidecar JSON to __recovery/{ulid}.json BEFORE its first HEAD-advancing commit (commit_staged, or compact_files for optimize_all_tables, which advances the Lance HEAD via a reserve-fragments + rewrite commit rather than a staged write). The sidecar names every (table_key, table_path, expected_version, post_commit_pin) it intends to commit + the writer kind + actor_id.
2. Phase B: writer's per-table commit_staged loop runs.
3. Phase C: publisher commits the manifest.
4. Phase D: writer deletes the sidecar.

Phase letter convention. Throughout the recovery code, log messages, failpoint names (e.g. branch_merge.post_phase_b_pre_manifest_commit), and the per-writer integration tests, "Phase A/B/C/D" refers exclusively to the four-step lifecycle above. The per-table staged-write contract (stage_* then commit_staged, two steps) is referred to by those API verbs — never by phase letters — so a reader of recovery.rs, failpoints.rs, or this document only encounters phase letters in the per-writer context.

A failure between Phase A and Phase D leaves the sidecar on disk. The next Omnigraph::open (gated on OpenMode::ReadWrite) runs the recovery sweep in crates/omnigraph/src/db/manifest/recovery.rs:

• For each sidecar in __recovery/, compare every named table's Lance HEAD to the manifest pin. Classify per the all-or-nothing decision tree (RolledPastExpected / NoMovement / UnexpectedAtP1 / UnexpectedMultistep / InvariantViolation).
• If any table is InvariantViolation (Lance HEAD < manifest pinned — should be impossible), abort with a loud error and leave the sidecar on disk for operator review.
• Otherwise, if every table is RolledPastExpected, roll forward: a single ManifestBatchPublisher::publish call extends every pin atomically. SchemaApply sidecars are eligible only when schema-state recovery promoted the matching staging files in the same recovery pass; otherwise full open-time recovery rolls them back and refresh-time recovery leaves them for the next read-write open.
• Otherwise roll back: per-table Dataset::restore to the manifest-pinned table version, then a single ManifestBatchPublisher::publish of the restored HEAD — symmetric with roll-forward, so manifest == HEAD after recovery (no residual drift). This convergence is what lets a failed-then-retried schema apply succeed instead of failing one version higher each iteration. The audit row's to_version records the logical rolled-back-to version (manifest_pinned); the manifest is published at the restore commit (manifest_pinned + 1, same content).
• After a successful roll-forward or roll-back, an audit row is recorded — _graph_commits.lance carries a commit tagged actor_id = "omnigraph:recovery", and a sibling _graph_commit_recoveries.lance row carries recovery_kind, recovery_for_actor (the original sidecar's actor), operation_id, per-table outcomes. Operators run omnigraph commit list --filter actor=omnigraph:recovery to find recoveries.
• Sidecar deleted as the final step.

Triggers for the residual: transient Lance write errors during finalize (object-store retry budget exhaustion, disk full); persistent publisher contention exceeding PUBLISHER_RETRY_BUDGET = 5 retries.

Long-running servers: the write entry points (load_as, mutate_as, apply_schema_as, branch_merge_as) and Omnigraph::refresh run roll-forward-only recovery in-process (recovery::heal_pending_sidecars_roll_forward) — the common Phase B → Phase C residual closes on the next write, without a restart and without an explicit refresh. The heal lists __recovery/ (one list_dir; empty in the steady state) and, per sidecar, acquires the same per-(table_key, table_branch) write queues every sidecar writer holds from before write_sidecar until after delete_sidecar — so it serializes against a live writer instead of rolling its in-flight sidecar forward from under it (a sidecar whose queues can be acquired belongs to a writer that finished or died; an existence re-check after the wait skips the finished case). Lock order is queues → coordinator, matching every writer's commit→publish path. Pinned by the four tests/failpoints.rs::*_after_finalize_publisher_failure_heals_without_reopen tests (load, mutation, schema apply, branch merge). The maintenance entries need the heal for more than liveness: without it, a schema apply re-plans rewrites from the manifest pin and orphans the drifted Phase-B commit (dropping its rows), and a branch merge publishes the drift as an unattributed side effect — both while the stale sidecar lingers to misclassify later. Sidecars that would require a Dataset::restore (mixed / unexpected state) are deferred to the next OpenMode::ReadWrite open: restore is unsafe under concurrency because Lance's check_restore_txn accepts the restore against in-flight Append/Update/Delete commits and silently orphans them (pinned by tests/staged_writes.rs::lance_restore_loses_to_concurrent_append_via_orphaning). When such a deferred sidecar blocks a write, the commit-time drift guard says so explicitly ("a pending recovery sidecar requires rollback — reopen the graph read-write") instead of pointing at omnigraph repair, which refuses while a sidecar is pending. Continuous in-process recovery for the rollback path is the goal of a future background reconciler. ensure_indices does not heal at entry itself — it runs inside the load / schema-apply flows after their entry heal, and its strict preconditions still fail loudly on drift when invoked directly.

The publisher-CAS contract is unchanged: a concurrent writer that advances any of our touched tables between snapshot capture and publisher commit produces exactly one winner. The residual above is about our abandoned commits in the failure path, not about concurrency races.

Sidecar I/O failure semantics (all sidecar I/O goes through the backend-generic StorageAdapter; the contracts below are pinned by the storage-fault failpoints recovery.sidecar_{write,delete,list} / recovery.record_audit and their tests in tests/failpoints.rs and tests/recovery.rs):

• Phase A put fails (S3 PutObject / fs write): the writer aborts before its first HEAD-advancing commit — no sidecar, no drift, nothing to recover; a transient fault never wedges later writes.
• Phase D delete fails (S3 DeleteObject): swallowed with a warning — the write already published, so failing the caller would report an error for a durable write. The stale sidecar is consumed by the next write's entry heal (or the next open) via the stale-sidecar audit-recovery path, recorded as RolledForward.
• __recovery/ list fails (S3 ListObjectsV2): loud at every consumer — the write-entry heal fails the write, the open-time sweep fails the open. Silently skipping recovery would be consumer tolerance of drift.
• Corrupt / unparseable sidecar: refused loudly by heal and open alike; the file stays on disk for operator inspection (read-only opens still work — the sweep is skipped there).
• Audit append fails after a roll-forward publish: that recovery attempt errors and keeps the sidecar; re-entry sees the already-published manifest, records exactly one RolledForward audit row, and deletes the sidecar (the retry tolerance documented on record_audit).

Backend notes (the adapter is one implementation over object_store for every backend): local writes stage through name#<digits> temp files that the backend filters from listings and refuses to address — crash residue of that shape is invisible to the sweep, harmless, and reclaimed by delete_prefix/manual cleanup. Storage errors are backend-wrapped text without a typed NotFound discriminant — callers that need missing-vs-error (the cluster store) probe exists() first. exists() itself is object-store semantics everywhere: only objects (or non-empty prefixes) exist, and a permission failure is a loud error, not a silent false.