Context Tests

A Context Test is a deterministic structural assertion over one resolution — no model, no LLM judge.

{ "id": "refund-known-customer", "name": "Refund question from a known customer",
  "consumer": "support-agent", "objective": "answer-customer",
  "input": { "message": "How do I get a refund on our annual Pro plan?", "customerId": "CUST-1042" },
  "at": "2026-10-08T12:00:00Z", "required": true,
  "expect": { "status": "RESOLVED", "include": ["policy", "customer-profile"], "exclude": ["engineering-notes"],
              "includeItems": ["docs/refunds.md"], "maxTokens": 3000 } }

A test is FAIL if any assertion fails, else UNKNOWN if any is unknown, else PASS. environment pins the resolver policy basis; incident tests always require it. at pins the evaluation time so currentness rules are reproducible; omit it to evaluate "now".

Reading a result. Failures come first, then unknowns, under an Expected / actual / why summary. When a failing or unknown assertion concerns a layer and the tests ran against the current design, the Workstation names any unsaved rule change on that layer (profile, field, saved → draft) — the most likely cause — with Inspect rule →, which opens the rule with this test's input, and Discard unsaved changes. If no unsaved change touches the layer, it says so: look at the layer's sources or the saved design. The Inspector shows each layer resolved for the test, with Inspect rule for this scenario →.

Create by capturing the current preview (rendered layers become must render, profile-excluded layers become forbidden, status and budget are pinned), then refine in TEST. Run against the current design (live sources) or any Build. Build validation runs every test against exactly the material being pinned; a required test that does not PASS blocks a release-quality Build unless you choose Build anyway (the Build is then marked not release-quality).

Written from a conversation (seif learn). A test created from a real conversation (the Prova tab, or seif learn --write) uses an item-level shorthand instead of the full expect block: must: [<locator>] for a question the agent should receive (a COVERED verdict) and forbid: [<locator>] for one its profile keeps out on purpose (a SEALED verdict). These compile to expect.includeItems / expect.excludeItems. Such a test is written required: true by default — so a release that loses the guarantee is blocked — unless the owner chooses watch only, which writes required: false as a deliberate, named downgrade. Its provenance is the frontmatter field source: learned from a conversation, and it is named after the agent it proves. See learn.

From a historical incident (0.6.0)

Observe → Historical investigation → Improve → Create regression test. An intact retained request, evaluation instant and available deterministic dependencies are required. The engineer supplies expected or forbidden layers/materials; the observed historical outcome is never automatically treated as correct. The draft records the historical trace, observation digest, Build and input digests, expectations: USER_SUPPLIED and DRAFT_REQUIRES_REVIEW. Review confidential input before saving it into the versioned project. The test pins the original resolution environment and instant; provenance also preserves the deployment environment.

In Test, choose Confirm expected behavior after reviewing the input and assertions. The design-time reviewHistoricalTest command accepts { project, testId } and returns the same reviewed test for headless authoring. provenance.status becomes REVIEWED, with review.at and review.scenarioDigest. The digest binds input, consumer, objective, environment, instant, required flag, expectations, fixtures and incident provenance. Editing any of these invalidates confirmation. Renaming a test does not change its semantics. Confirmation records the author's intent; the test must still pass to validate a correction.

Unconfirmed required incident tests evaluate guardReview as UNKNOWN, block release-quality Builds and block promotion under the default required-test policy. Old incident drafts also require confirmation; their status is not silently upgraded. The assertion does not authenticate the author: project review records remain authored metadata, while Registry authorization and deployment approvals remain separate.

The supported pinned-source journey removes a required policy, observes the generated test FAIL, restores it in a candidate and observes PASS. Comparison evaluates the corrected Build against the same scenario. Production provider contents are not copied into a project: use separately authored synthetic fixtures. Missing input or restricted retention blocks generation rather than inventing a scenario. Provenance labels document the test author's declared basis; they do not independently authenticate imported project files.

Versioned tools

Context and tool delivery documents tool declarations and profile rules, v2 tool Builds, executor bindings, synthetic tests, operator approval, SDK integration and effect-free replay. Tool-free Builds preserve the v1 contract.