SEIF Reliability Impact
Open Deploy → Reliability → Impact to review the last 30 days of retained evidence for the selected environment. Impact answers what the context delivery system observed and tested. It separates application evaluations and unmeasured operational savings.
Reading the numbers
| Indicator | Evidence and denominator | What it does not prove |
|---|---|---|
| Resolution failures / unknown / stale context | Matching retained traces in the observation window | Reliability of all application traffic; discarded traces are absent |
| Consumption mismatches | Traces with an application consumption report | Independent proof of model processing |
| Distinct candidates blocked | Environment promotion-block audit records, deduplicated by candidate | Production incidents avoided; standalone failed tests are not counted |
| Incident-linked guard failures blocked | Required reviewed guards failing an actual promotion gate; unique incident/candidate pairs | A confirmed production incident; missing historical observations are labelled |
| Verified structural recurrences blocked | The above, plus a matching retained historical observation containing a resolution failure or consumption mismatch | Business damage avoided or improved model intelligence |
| Exact reproduction | Distinct attempted traces with at least one exact receipt / distinct attempted traces | A pass for unattempted incidents or independent application replay |
| Sufficient replay evidence | Existing historical investigation checks over at most the latest 100 incident signals | Actual exact execution; the Runtime identity still needs a reproduction attempt |
| Investigation duration | Explicitly reported elapsed sessions, including idle time | Active labor or time saved |
| Application failure rate | Latest non-unknown outcome per retained trace / traces with a success or failure label | Failures among unlabelled traffic or causal attribution to a context correction |
Trace windows filter execution time; application outcomes use the latest retained report at report generation. This is not a historical reconstruction of labels as of the window end. Invalid legacy evaluation metadata is excluded from cohorts. Each rate contains numerator, denominator and basis. A zero denominator yields null, displayed as UNKNOWN. The complete application-traffic denominator is UNKNOWN. Bounds and sampling are returned explicitly. Incident signals include stale/unknown context and reported application failures; they are broader than confirmed structural incidents.
Avoided production incidents, work saved and financial return are null in V1. There is no default multiplier that turns blocked tests into incidents or investigation duration into savings.
Inspectable chains
The report joins actual records: historical Trace → explicitly measured Investigation, when present → reviewed Guard → candidate Build → Regression → blocked promotion → later healthy deployment with a passing reviewed guard referring to the same incident. Missing links remain null. Changed/expired historical observations cannot support the stronger verified count. Later append-only reports can preserve a verified original snapshot. A later deployment is an observed association, not proof that every failure was corrected.
The report returns at most 50 chains. Aggregates read all matching operational records, up to 10,000 per kind; larger windows fail with IMPACT_WINDOW_TOO_LARGE instead of silently sampling. Choose a narrower window. Artifact verification and replay diagnostics never invoke live providers or models.
Explicit investigation measurement
An administrator enables Allow explicit investigation duration and interaction reports for an environment. Each investigator then chooses Start measured investigation on a Registry historical trace and Record elapsed investigation when finished. No timer or report is started implicitly. Counted interactions are the start, section navigation, comparison/reproduction actions and material/test dialog openings; this is not a count of keystrokes or all application activity.
Only report identity, trace reference, elapsed milliseconds, interaction count, actor and consent revision are retained. The measurement identifies client-reported elapsed time, never labor. New measurements are refused after consent is disabled; existing measurements remain under their bounded retention. A repeated reportId is idempotent; a changed payload conflicts. Measurement records are bounded by the environment's maxTraces.
Application evaluation integration
The existing outcome report optionally carries seven opaque identifiers: evaluator, version, dataset, experiment, cohort, model, configuration. Supplying them explicitly opts into evaluation metadata collection. Do not use customer identifiers, answers, prompts or secrets. Numeric metrics remain application-defined, bounded to 20 finite values. Impact aggregates the latest labelled outcome by Build and the complete evaluation coordinate, showing sample counts and metric means/ranges. Descriptive comparisons require matching evaluator, version, dataset and experiment, report changed model/configuration/cohort factors, and explicitly leave controlled-experiment and causal status unverified. Up to 100 comparisons are returned with truncation labelled.
Treat Build differences as descriptive evidence. Changes to prompts, models, providers, model configuration, user mix, dataset sampling and application code may explain outcome differences. An independent controlled experiment is required for attribution; V1 never labels a delta causal. The deterministic reference application demonstrates the integration, not real model performance or a customer pilot.
Storage and retention diagnostics
Impact shows replay blockers and logical environment content usage. An unrestricted Registry Admin also sees measured SQLite, WAL and shared-memory file sizes, reusable database pages and distinct reachable Project artifact bytes. Database figures cover all organizations, not just the selected environment. Runtime deployment files, backups, unreferenced objects and filesystem allocation overhead are excluded. Physical storage is not bounded by the recorder content budget.
API
| Method | Route | Contract |
|---|---|---|
| GET | /v1/reliability/impact?project=P&environment=E&since=ISO&until=ISO | Ordered window of up to 90 days; defaults to the last 30 days. Requires environment, Build and trace read access. |
| POST | /v1/reliability/impact-policy | {project, environment, enabled, expectedRevision}; Admin configuration with stale-write protection. |
| POST | /v1/traces/:id/investigation-measurements | {reportId, elapsedMs, interactions}; human trace reader plus explicit environment consent. |
Replay diagnostics additionally require trace.content; permission restrictions are labelled rather than producing a misleading zero. No trace messages, request values, test expectations or secrets are returned by the Impact report. Existing authentication, organization, Project and environment boundaries apply. Reports never promote, deploy or approve.
Run runtime/node scripts/prove-reliability-impact.mjs for a repeatable independent-process HTTP journey and a JSON evidence report. Execution time is measured; engineering effort and labor savings remain unmeasured.