Hyphae
GitHub

Transactions and proofs

Every commit in Hyphae — SQL, structure, or search — passes through one commit scheduler and lands at one commit sequence number (CSN) across every engine. This page covers how to stage a transaction that spans engines, what each durability class actually guarantees, and how a result can carry a proof someone else verifies without trusting the machine that produced it.

Explicit all-engine transactions

An explicit transaction mixes SQL, structure, search, and vector stages and commits them all under one CSN, or none of them. The step sequence is status, any of stage_sql, stage_structure, stage_search, stage_vector (repeatable, in any order), then a terminal commit or rollback. Each stage returns its own provisional result inside the transaction, before the commit: 

hyphae transaction --data-dir "$D" execute --steps-json '[
  {"operation":"stage_sql","statement":"INSERT INTO notes (id, body, stars) VALUES (?, ?, ?)",
   "parameters":[3,"transactional note",5]},
  {"operation":"stage_structure","mutation":{"operation":"counter_add","keyspace":20,"key":"visits","delta":1}},
  {"operation":"commit"}
]'

If the process dies or a commit acknowledgement is lost, durable evidence on disk resolves the outcome — never guess or blindly replay a commit whose acknowledgement you never saw: 

hyphae transaction --data-dir "$D" status --id <transaction_id>

In the SDKs, a cancelled or transport-failed commit becomes a terminal outcome_unknown — resolved the same way, through transaction status, never by re-sending the commit.

One delta batch, not a full-state reload

Underneath, a local all-engine transaction is built from one detached native delta batch captured off a single immutable snapshot: a bounded catalog cache, a relational overlay, a scalar-structure overlay, and a lexical overlay, rather than a reload of the whole directory's state on every stage. The batch stays bound to the exact live database handle that created it — staging or committing through a different handle, a reopened handle, or another directory fails closed. Work scales with the keys, index entries, and WAL bytes actually touched, not with the total row, structure, or document count already in the directory. Full contract: delta all-engine transaction v1.

Three durability classes

Every mutating command accepts --durability strict|group|memory. None of them carry a universal sub-millisecond promise — fsync, cold I/O, and unbounded queries are measured and reported separately, never folded into one latency claim:

  • Strict — acknowledged after this transaction's own WAL fsync. The strongest guarantee; the most fsync latency.
  • Group — acknowledged after a shared cohort fsync. Multiple commits share one fsync, trading a small acknowledgement delay for throughput.
  • Memory — acknowledged with no fsync at all. An acknowledged Memory commit can be lost on crash — but never torn: recovery drops whole commits from the volatile WAL suffix only, never a partial one.

A commit receipt's durability field names which class actually acknowledged that write (see Getting started). The cross-engine commit protocol — atomicity across crash and recovery, strict-durability acknowledgement, first-committer-wins, and contiguous visibility — has a machine-checked TLA+ model (docs/formal/HyphaeCommit.tla). The model checks the protocol as specified; it is evidence about the design, not a proof of the Rust implementation, whose fidelity is carried separately by physical crash-matrix tests.

Checkpoints, backups, vacuum

A checkpoint publishes one synchronized all-engine recovery boundary — run it before a backup, and after any bulk load. The verified full backup/restore cycle validates at every step before promising anything: 

hyphae checkpoint --data-dir "$D"
hyphae backup create --data-dir "$D" --out ./backup      # → created (verified at creation)
hyphae backup verify --backup ./backup                   # → verified (without opening live state)
hyphae restore --backup ./backup --data-dir ./restored   # → restored (staging + doctor + atomic activation)
hyphae doctor --data-dir ./restored                       # → healthy, snapshot_verified: true

A Native backup is physical and synchronized, described by a NATIVE_BACKUP.json manifest with an exact inventory. Restore never merges or overwrites: it rebuilds into a sibling staging directory, runs mandatory doctor validation, then activates atomically — the destination must be new. There is no online or incremental backup; that is a declared non-capability, and media policy (where backups live, how long they are kept) belongs to your application, not Hyphae. vacuum rebuilds live roots into a smaller page generation and publishes the result atomically, for reclaiming space after deletes; compact --target structures|search compacts one root family on a schedule you choose. See Operations for doctor's full diagnostic scope.

Result proofs: verifiable without trusting the machine

An eligible read — a catalog lookup, a SQL query, or a product read — can emit a canonical proof plus a witness. Any third party verifies both completely offline, with nothing but a 32-byte trusted anchor supplied independently of the proof itself — no access to your data directory, no network call back to your machine: 

# Generate: the query executes and is bound to the artifacts
hyphae proof generate --data-dir "$D" \
  --operation-json '{"operation":"sql","statement":"SELECT id, body FROM notes WHERE id = ?","parameters":[1]}' \
  --proof-out query.hynproof --witness-out query.hynwitness
# → { "anchor": "f68a8eae03a3ea69...", "kind": "sql", "proof_bytes": 620 }

# Verify (another machine, no data access): full semantic re-execution
hyphae proof verify --proof query.hynproof --witness query.hynwitness \
  --anchor f68a8eae03a3ea69...
# → { "status": "verified", "scope": "semantic_reexecution",
#     "semantic_reexecution_performed": true }

The verifier validates both artifacts, requires the independently supplied anchor, re-executes the operation under bounded reference semantics, and compares the exact result — it does not merely check that the proof is internally well-formed. CLI-provable operations are catalog_list, catalog_describe, and sql; the SDKs additionally expose prove, prove_sql, and verify_proof. Formats: native result proof v2 and native directory witness v2.

Witnesses and the external trusted anchor

A witness bundles every directory below the native data root needed to re-execute the proved operation, carrying the same directory lineage, history epoch, and visible CSN as the proof. The ExternalTrustedAnchor that verification requires binds the 24-byte directory lineage and nonzero history epoch, the visible CSN and catalog version, the complete immutable all-engine root digest, and the durable checkpoint's visible CSN and manifest digest — the producer checkpoints the exact root that produced the result while it holds the directory lock, and the verifier requires its own reopened, retained authority to equal that anchor exactly.

Self-consistency is not trust. A proof that only checks against itself proves nothing about the world outside the machine that generated it. Verification always requires the anchor from somewhere else — an escrow, a second party, a published ledger — obtained independently of the proof and witness you are handed. Hyphae's own verifier will not skip this step, and neither should yours.