Results and reliability
Interpret behavioral outcomes, infrastructure status, checks, model usage, and repeated trials.
CrashLab keeps separate claims separate. A concise outcome is useful for a gate, but the retained result preserves the evidence needed to understand it.
Trial outcomes
| Outcome | Meaning |
|---|---|
pass | Required checks passed and the infrastructure produced trustworthy evidence. |
fail | A required behavioral check failed. |
inconclusive | Infrastructure, target execution, or required verification prevented a trustworthy behavioral decision. |
An inconclusive trial is not a pass and is not automatically an agent regression. For example, a model-provider outage should be repaired and rerun rather than counted as proof that the prompt became worse.
Execution phases
Every trial records a causal lifecycle:
- prepare the sandbox and bindings;
- run the target agent;
- verify protected behavior; and
- record the decision.
The result viewer attaches runtime checks to the phase that failed and behavioral checks to verification. This makes it possible to distinguish “the agent produced the wrong state” from “the verifier never ran.”
Expected, observed, evidence
A check should explain:
- what must be true;
- what the verifier observed;
- why that observation passed or failed; and
- where the supporting state or event evidence lives.
crashlab view <stored-run> renders the human diagnosis. --json emits the
bounded coding-agent projection. --raw is the complete authoritative result,
including internal provenance and evidence.
Reliability denominators
A repeated run reports several different rates:
- End-to-end success: passing trials divided by every scheduled trial.
- Behavioral success: passing behavior divided only by trials whose infrastructure was healthy and whose target completed.
- Infrastructure availability: infrastructure-healthy trials divided by scheduled trials.
- Target completion: completed targets divided by scheduled trials.
- Criterion reliability: passes for one check divided by trials in which that check could be evaluated.
CrashLab includes uncertainty for finite samples. Three passes out of three are encouraging evidence, not proof of a certain 100% rate.
Latency, tokens, and cost
When the target or provider exposes usage, results retain latency, token categories, and cost alongside behavior. Provider/model identity and inference settings are part of trial provenance.
If an evaluation uses a semantic judge, CrashLab reports its request count, tokens, latency, model, and cost separately from target-model execution. Trial, repeated-run, and comparison views never add judge usage to target-provider totals. Aggregates sum only comparable retained assessments.
Missing pricing or billing information is reported as unavailable. CrashLab does not silently turn missing cost into zero. When cost is estimated from a pricing snapshot rather than provider billing, the result records that distinction.
Regression checks
crashlab check compares baseline and candidate revisions under the same
selected cases and evaluation source. It preserves complete results for both
variants plus criterion, reliability, latency, token, and cost deltas.
The target source determines whether the result supports a regression claim. Candidate-owned evaluation code may intentionally remain identical across both variants. When no included target source changed, the comparison is a same-source control/reliability sample.
The comparison policy is customer-selected. A strict policy, regressions-only policy, and report-only policy answer different workflow questions; none changes the underlying evidence.