Search

Hyphae's search engine owns documents, lexical indexes, doc values, aggregations, and transactional search visibility inside the same directory, catalog, and commit sequence as SQL and structures — it is not an OpenSearch or Elasticsearch REST facade, and it embeds no models of its own. Your application supplies vectors; Hyphae ranks and fuses them.

Call it "an integrated lexical/vector search engine," not "full-text search compatibility" or a "Lucene-class query surface." The grammar is bounded and typed, not an open query language.

Collections and documents

A search collection declares a stable object ID, source ownership, fields, a stored/source policy, analyzers, a doc-values policy, and optional vector indexes. Every document has a stable object ID and MVCC version, and may be search-owned or linked to an object in another engine. provision creates the physical lexical index and the exact and ann vector indexes in one step; --dimension at creation fixes the vector width for every later ingest.

hyphae catalog --data-dir "$D" create-search-collection \
  --database 10 --schema 11 --collection 13 --analyzer 12 \
  --name main.public.note_search --dimension 2

hyphae search --data-dir "$D" provision --collection 13

Ingest is idempotent: repeating an --idempotency-id with identical content is a safe no-op. Doc-value fields are fixed by the collection definition: the CLI's create-search-collection declares category (text), price (integer) and rating (float), and a document carrying any other doc-value field name fails with invalid_request. Document IDs share the engine's stable object-ID namespace, so use IDs that cannot collide with catalog objects (the examples start at 1001).

hyphae search --data-dir "$D" ingest --collection 13 --idempotency-id 1 \
  --documents-json '[{"id":1001,
    "text":"offline search engine with proofs",
    "doc_values":{"category":"note","price":5},
    "vectors":{"exact":[1.0,0.0],"ann":[1.0,0.0]}}]'

Two ways to query: direct lexical, integrated hybrid

A direct lexical query targets the physical index — find it in the catalog as __product_lexical_<collection> — with --kind term, phrase, prefix, or fuzzy:

hyphae search --data-dir "$D" query --index 23 --query offline --kind term --limit 5

The integrated query binds a lexical branch and a vector branch to one catalog snapshot, with typed doc-value filters, sort, facets, and metric aggregations:

hyphae search --data-dir "$D" integrated --collection 13 \
  --lexical search \
  --vector-target exact --vector 0.7 --vector 0.7 --vector-strategy exact \
  --filter-json '{"operation":"compare","field":"category","operator":"equal","value":"article"}' \
  --facets-json '[{"field":"category","limit":5}]' --limit 10

{
  "approximate": false,
  "hits": [ { "object_id": "1007", "score": 0.0325...,
              "doc_values": { "category": "article", "price": 4 } } ],
  "facets": [ { "field": "category", "buckets": [ {"value":"article","count":1} ] } ],
  "vector_branches": [ { "strategy": "exact_filtered", "exact_reranked": true } ],
  "snapshot": { "visible_csn": 20, "root_digest": "ac0c116c..." }
}

Lexical scoring: BM25, and BM25F with field boosts

Default ranking is BM25 with per-collection k1/b (defaults 1.2 / 0.75, stored as micro-unit integers so identical definitions score identically on every host), query-term deduplication, descending score, then bytewise document-ID ascending as the tie-break. Declaring an ordered list of (field, weight) pairs — at most 64 fields, weights 1..=1,000,000,000 micros — switches the branch to versioned BM25F: the reserved name body scores the canonical indexed text, any other name scores that field's string doc value (missing or non-string reads as empty). Field boosts, the term operator, and prefix expansion are mutually exclusive in one request.

Filters

Verified on an 8-document corpus:

hyphae search --data-dir "$D" integrated --collection 13 \
  --filter-json '{"operation":"in","field":"category","values":["memo","receipt"]}' --limit 10
# → the 4 documents whose category is "memo" or "receipt"

hyphae search --data-dir "$D" integrated --collection 13 \
  --filter-json '{"operation":"is_null","field":"nonexistent_field"}' --limit 10
# → all 8 documents; none carry that field, so none are excluded

hyphae search --data-dir "$D" integrated --collection 13 \
  --filter-json '{"operation":"like","field":"category","pattern":"a%"}' --limit 10
# → the 4 documents whose category starts with "a" ("article")

Lexical matching: term operator, prefix, fuzzy, phrase

These options select candidates before scoring, and — like field boosts — are mutually exclusive with each other in one request: --lexical-and requires every analyzed term; --minimum-match <n> requires a floor on distinct matched terms; --lexical-prefix expands the final term as a bounded prefix (at most 64 distinct expansions, MAX_LEXICAL_PREFIX_TERMS, else limit-exceeded); --fuzzy <1..2> expands every term within that Levenshtein edit distance over the same bounded vocabulary; --phrase demands the exact consecutive analyzed sequence (verification re-analyzes only the BM25 candidates, never the whole corpus).

hyphae search --data-dir "$D" integrated --collection 13 \
  --lexical "offline search engine benchmark" --minimum-match 3 --limit 10
# → 3 hits sharing at least 3 of the 4 query terms

hyphae search --data-dir "$D" integrated --collection 13 \
  --lexical "offlne serch enigne" --fuzzy 2 --limit 10
# → 7 of 8 documents match despite every query term being misspelled;
#   only the one document sharing no term within edit distance 2 is excluded

hyphae search --data-dir "$D" integrated --collection 13 \
  --lexical "offline search engine" --phrase --limit 10
# → exactly the 2 documents containing "offline search engine" as a
#   consecutive phrase

Vector branches and hybrid fusion

A vector branch chooses exact (the oracle — ranks the effective base-plus-delta set exactly), ann (incremental HNSW; discloses "approximate": true plus candidate evidence), or adaptive. Mutations never rebuild the whole HNSW graph. When both branches are present, fusion combines them: the default is deterministic weighted reciprocal-rank fusion (k = 60); weighted-score blends each branch's weight with its normalized score; relative-score min-max normalizes each branch over its own admitted candidates first, so the best candidate of every branch contributes exactly its weight regardless of that branch's score scale.

hyphae search --data-dir "$D" integrated --collection 13 \
  --lexical "offline search engine" \
  --vector-target exact --vector 1.0 --vector 0.0 --vector-strategy exact \
  --fusion relative-score --highlight-fragments 2 --highlight-bytes 64 --limit 5

{
  "hits": [
    { "object_id": "1001", "score": 2.0,
      "fragments": ["offline search engine with proofs and receipts"],
      "doc_values": { "category": "article", "price": 5 } },
    { "object_id": "1007", "score": 1.8339... }
  ],
  "vector_branches": [ { "strategy": "exact_filtered", "exact_reranked": true } ]
}

An optional max_distance cutoff drops hits past a canonical metric distance before fusion, so a garbage vector match never earns a reciprocal rank or a normalized score. Branch receipts still report pre-cutoff candidate counts, so recall risk stays observable.

Facets, range facets, autocut, offset

Terms facets bucket a field's distinct values. Range facets bucket a numeric doc value into caller-defined half-open intervals — at most 8 ranges per request, at most 64 ranges each — returning exactly one bucket per declared range in request order, never count-sorted:

hyphae search --data-dir "$D" integrated --collection 13 --lexical offline \
  --range-facets-json '[{"field":"price","ranges":[{"upper":10},{"lower":10,"upper":20},{"lower":20}]}]' \
  --limit 10
# → "range_facets": [ { "field": "price",
#      "buckets": [ {"range_ordinal":0,"count":3}, {"range_ordinal":1,"count":1}, {"range_ordinal":2,"count":1} ] } ]

--offset skips leading hits ahead of the limit window after every other stage (fusion, filter, sort, rerank, dedupe, autocut); the ranked window must still stay within the 1,024-hit bounded ceiling. --autocut <1..16> finds the first steep knee in the score curve and cuts there instead of a fixed count — the argument trades recall for precision, steeper is more conservative:

hyphae search --data-dir "$D" integrated --collection 13 \
  --lexical "offline search engine proofs receipts" \
  --vector-target exact --vector 1.0 --vector 0.0 --vector-strategy exact \
  --autocut 1 --limit 10
# → 7 hits: the 8th candidate (score 0.0147, well below the other seven's
#   ~0.03 band) is cut. The same query with --autocut 4 keeps all 8 —
#   a gentler decay does not clear the steeper knee threshold.

Highlighting and pagination

--highlight-fragments <1..4> and --highlight-bytes <16..512> bound the number and byte size of returned fragments per hit — see the fragment in the relative-score example above. Pagination is --limit plus --offset; there is no separate cursor API, and the offset + limit bound above applies to both.

The document cap: 250,000 shipped, 1,000,000 measured not shipped

MAX_PRODUCT_SEARCH_COLLECTION_DOCUMENTS is 250,000 in 3.0.0, raised from 100,000 after a B+tree batch-rewrite fix removed a structural degeneration in the durable scorer. This is a receipted number, not a soft default — a collection at the cap fails closed rather than degrading. A separate re-measurement at 1,000,000 documents found the query ladder linear in document count (BM25 23.2 ms, filtered+facet 42.6 ms, phrase 24.2 ms, fuzzy 54.2 ms — a 3.3–4.5× latency increase for 4× the documents), but 1,000,000 is measured, not shipped: it waits on the ANN delta stage and vector-batch ingest. The buffer pool sizing that made the 1M ladder linear rather than degenerate is part of the same evidence: at 1,000,000 documents a two-term BM25 query plans over 1,500 posting segments, and the default 8,192-frame pool (see Operations) keeps them resident instead of re-verifying them on every query.

Durable format, conceptually

The physical implementation is one immutable copy-on-write native B+tree. INDEX DOCUMENT creates a document; REPLACE DOCUMENT and DELETE DOCUMENT require an exact live identity and atomically maintain stored source, collection statistics, term metadata, and postings in one transaction. The first accepted lifecycle mutation upgrades the tree root from format marker HYSEABT1 to HYSEABT2; historical v1 roots stay readable, and v1 rejects every tombstone. Postings use format HYPOST01 or the current HYPOST02. A search collection's manifest uses header format HYPSMAN2, chunked at a bounded 1,024 entries per chunk. None of this is exposed through the CLI or SDKs — it exists so an independent verifier can audit the directory byte-for-byte. Full contract: search document lifecycle v1.