Retrieval: finding the relevant knowledge, in any language

A retrieve rule ranks a layer's items against the person's message and keeps the relevant ones. SEIF finds relevance two ways, and a project can use both:

Rule strategies: lexical (words), semantic (meaning only), hybrid (both, fused), static (stable order) and external (your own ranking service). A retrieve rule without an explicit strategy uses hybrid when the project configures search.semantic, else lexical.

By words

runtime/engine/text-analysis.mjs reads text the same way for relevance (what an agent receives) and for the Compass, learn and find (what the owner sees):

1. Normalize: NFKC, lower case, accents folded for Latin, Greek and Cyrillic (kana keep their marks, so が stays が); R$ 20,00 equals 20.
2. Segment with the ICU word segmenter (Intl.Segmenter, built into Node; no dependency): spaces for most languages, dictionaries for Japanese, Chinese, Thai, Lao, Khmer and Myanmar.
3. Drop stop words of English, Portuguese, Spanish, French, Italian, German, Dutch, Russian, Japanese, Chinese, Korean and Thai (question words included), plus the project's own search.ignore.
4. Map synonyms to one word: the project's search.synonyms groups, and built-in groups for price, opening hours and ID in six languages (turn them off with search.builtinSynonyms: false).
5. Emit terms: each word; for alphabetic words of 5+ letters a 5-letter prefix (a language-neutral stem: prazos meets prazo, renouvellement meets renouveler); for Chinese, Japanese and Korean also character bigrams (免許証 meets 免許, 갱신은 meets 갱신).

Ranking is BM25 (k1 1.2, b 0.75); titles count three times; ties break by locator. With a non-empty question, an item that shares no term with it is NOT_RELEVANT. In the Compass and learn, coverage is computed per word (a word counts when any of its terms is in the section, with the weight of its rarest term), so refund meets Refunds whole.

What words cannot do: meet a word that shares no letters with the question (fee and price, valor and preço without a synonym), or cross languages. The Busca screen shows, for any question, the words used, the words ignored, the synonyms applied and what was found, so the reason is never hidden.

Proof: tests/engine/multilingual-search.test.mjs (Japanese, Chinese, Korean, Thai, Russian, French, German and Spanish questions find the right section; project synonyms and ignored words), tests/engine/resolution-guarantees.test.mjs.

By meaning, with SEIF owning the index

The earlier external strategy sent only metadata (ref, title, locator, digest) to a ranking service, so that service had to keep its own index of the content, in sync by digest: an indexing contract SEIF did not define, which is why embeddings did not really work. Search by meaning removes the gap by making SEIF the owner of the index:

• Configure an endpoint (seif.yaml → search.semantic: endpoint, model, optional secretRef, minSimilarity, onUnavailable). Any service that answers POST /embeddings in the OpenAI format works. Use a multilingual model: text-embedding-3-small (OpenAI), voyage-multilingual-2 (Voyage), bge-m3 (Ollama, free and local).
• SEIF computes and keeps the vectors of each item once per (endpoint, model, content digest) in this computer's state folder. Changing a file re-embeds only that file.
• A Build pins them (embeddings.json, its digest in the manifest, verified on load and carried in Registry artifacts). A deployed runtime embeds only the question; question vectors are reused for the next layers and turns.
• Relevance by meaning is a similarity, recorded per item (similarity, reason RETRIEVED_BY_MEANING) with the model in the trace. Without minSimilarity, an item is relevant when the best match stands out from the median of the candidates by at least 0.10 and the item is within 0.06 of the best; so a question about something the knowledge does not hold finds nothing, rather than the nearest thing.
• Hybrid fuses the word and meaning orders by reciprocal rank (k 60); an item is kept if it shares words with the question or stands out by meaning.
• Failure is explicit: if the endpoint fails, onUnavailable: lexical (default) continues by words and records fallback.reason; fail blocks with RETRIEVAL_UNAVAILABLE.
• Privacy: the knowledge content is sent to the endpoint to be embedded, and the question at request time. Choosing the endpoint is the owner's decision; with Ollama nothing leaves the computer. On the shared web host, projects reach public addresses only, so a local Ollama works in the desktop app and the CLI.

Calibration (October 2026, Clínica Aurora knowledge in Portuguese, 16 questions in Portuguese, English, Spanish, French, Japanese, Korean, Chinese and German, three of them about things the clinic does not do): bge-m3 16/16 correct, including the three gaps; nomic-embed-text (an English model) failed most non-English questions, mostly by finding nothing rather than the wrong thing. The model matters: Busca → Testar conexão reports how similar the same question is in Portuguese and English (0.92 for bge-m3) and warns when the model is weak across languages.

Proof: tests/engine/multilingual-search.test.mjs (with a deterministic stand-in endpoint: a Japanese question finds a Portuguese section by meaning; fallback and fail; the Compass and learn verdicts; a Build pins every vector and the runtime embeds only the question; tampered vectors fail closed).

Turning it on

Conexões → Busca → Ligar a busca por significado uses the model the platform includes (on app.seifprotocol.com: bge-m3, multilingual, running on the SEIF server, no key; the knowledge does not go to a third party). Opções avançadas chooses another service (OpenAI, Voyage, Mistral, Ollama on this computer, any OpenAI-compatible endpoint) and where the vectors live. Testar antes de salvar checks the service, how well it relates the same question in two languages, and the vector store, without changing anything.

Vectors in your Qdrant

search.semantic.store: { kind: qdrant, url, collection, secretRef } keeps the index in a Qdrant collection instead of only on this computer:

• SEIF creates the collection (cosine, the model's dimension; another dimension is refused), writes one point per item (id derived from the embedding index and the content digest, payload index and digest, both indexed) and fills it with every vector this computer or a Build already holds, so a store added later is not left empty.
• Another computer, the web and the CLI read the vectors from the collection instead of embedding the material again.
• Similarity comes from Qdrant, filtered to the items the rule may retrieve (a layer's eligible digests); nothing else is ever returned.
• A Build still pins its vectors: if Qdrant cannot be reached, the same cosine is computed locally and the trace records store.error.
• The Qdrant key is a project secret (env:SEIF_SECRET_QDRANT), never written to project files.

To try it on your machine, scripts/dev/qdrant.sh runs Qdrant (with an API key, published on 0.0.0.0 so the tailnet reaches it); the key it makes is the project secret SEIF_SECRET_QDRANT.

Proof: tests/engine/multilingual-search.test.mjs (a stand-in Qdrant: filled from the local cache, read by another computer that embeds only the question, filtered search, fallback when unreachable, dimension mismatch refused).

Cross-topic references (links the author declares)

Relevance ranking finds the sections that match a question; a cross-topic reference is the author saying, directly, that one section leads to another. A knowledge section can name other sections it points at — in frontmatter (references:) or its seif marker (<!-- seif: references=<locator1,locator2> -->). When the referring section is in hand, resolve.mjs turns each live reference into a frontier door with reason REFERENCED, which the agent may open on its own: the same within-reach mechanism as an on-demand frontier, but intentional rather than ranked.

• The reference honours the profile seal: a target in a layer this profile excludes or that is not applicable is never offered, not even its title. Anything already in hand or already a door is skipped, and the frontier keeps the same cap (MAX_FRONTIER). A door already open for another reason (e.g. NOT_RELEVANT) is relabelled REFERENCED, because an explicit link outweighs relevance ranking.
• Authored safely: the owner links sections from a picker of existing sections (never a raw locator); a dangling link is refused at authoring (UNKNOWN_REFERENCE) and a link that breaks later (its target renamed or removed) is reported by doctor as DANGLING_REFERENCE, not silently dropped. See SEIF in the agent you already use → Cross-topic references.

External ranking

For teams with their own search service. The service receives the question (treat it as a processor of that text) and the eligible items' metadata (ref, title, locator, digest; never the content), so it must already hold its own index keyed by digest. It returns ordered {ref, score} items and a version; unknown or duplicate refs and non-finite scores fail validation; it cannot grant eligibility, override forbidden rules or add material. The Build records that this ranking is not pinned. Prefer search by meaning unless you already run such an index.

Budget

Consumer, Profile and per-layer budgets, maxItems and material limits apply to resolution. Required material is protected per item: if the budget trims any item of a required layer, resolution is BLOCKED with REQUIRED_CONTEXT_TRIMMED. Tests and regression report token growth, layer contributions, trimmed material and required-context failures. Token counts are character-based ESTIMATED values (ceil(chars/4)), not provider tokenizer counts.

An application may supply contextUsage: {tokens, model} with consumption reporting when it measured the context for that model; the trace labels this MEASURED_FOR_MODEL. Without it, measured usage stays UNKNOWN.