How a write becomes durable

L2 — Multi-dataset coordination via __manifest

OmniGraph is not a single Lance dataset; it is a graph of datasets coordinated through one append-only manifest table.

• Manifest table: __manifest/ Lance dataset.
• Layout:
  • nodes/{fnv1a64-hex(type_name)} — one Lance dataset per node type
  • edges/{fnv1a64-hex(edge_type_name)} — one Lance dataset per edge type
  • __manifest/ — the catalog of all sub-tables and their published versions, and the graph commit lineage (RFC-013 Phase 7)
  • _graph_commits.lance / _graph_commit_actors.lance — legacy / branch-ref carriers. Since RFC-013 Phase 7 the graph lineage lives in __manifest (graph_commit / graph_head rows, written in the publish CAS); _graph_commits.lance no longer receives commit rows, but is retained to carry the Lance branch refs that create_branch / list_branches / the cleanup orphan reconciler operate on. A graph created before Phase 7 (internal schema v3) keeps its lineage here until its first read-write open, which migrates it into __manifest via migrate_v3_to_v4.
  • (legacy _graph_runs.lance / _graph_run_actors.lance from pre-v0.4.0 graphs are inert; the run state machine was removed. The internal schema migration sweeps stale __run__* branches on first write-open; the inert dataset bytes themselves remain until a prefix-delete storage primitive lands)
• Manifest row schema (object_id, object_type, location, metadata, base_objects, table_key, table_version, table_branch, row_count):
  • object_type ∈ table | table_version | table_tombstone | graph_commit | graph_head
  • table_key ∈ node:<TypeName> | edge:<EdgeName> (empty for graph_commit / graph_head lineage rows)
  • table_branch is null for the main lineage and the branch name otherwise
  • Graph lineage rows (RFC-013 Phase 7): one immutable graph_commit row per commit (object_id = the commit ULID; metadata JSON carries parent / merged-parent / actor / timestamp) plus one mutable graph_head:<branch> pointer per branch (graph_head:main for main). The in-memory commit DAG is a projection of these rows.
• Snapshot reconstruction: latest visible table_version per (table_key, table_branch) minus tombstones — rows where object_type = table_tombstone, whose own table_version (acting as the tombstone version) is >= the entry's table_version.
• Atomic publish: multi-dataset commits publish so that a single write to __manifest flips all the new sub-table versions visible at once.
• Row-level CAS on the merge-insert join key: object_id carries an unenforced-primary-key annotation so Lance's bloom-filter conflict resolver rejects two concurrent commits that land the same object_id row. Without this annotation, Lance's transparent rebase would admit silent duplicates from racing publishers.
• Optimistic concurrency control on publish: a publish asserts the manifest's current latest non-tombstoned version for each touched table is exactly what the caller observed; mismatches surface as an ExpectedVersionMismatch manifest conflict naming the table and the expected/actual versions. Concurrent advances surface as a conflict rather than being silently rebased through.