Run your first evaluation // CrashLab

Run a configured candidate, inspect retained evidence, and compare a proposed change.

This guide assumes crashlab/ contains a real target and evaluation plan. Run these commands from anywhere inside the agent project. The explicit trust flag acknowledges that its Python world and verifier execute on the host.

Run one fresh trial

crashlab run --repetitions 1 --trust-evaluation-code

CrashLab snapshots the configured candidate, creates a fresh world, prepares an isolated target, runs the agent, verifies protected evidence, records the decision, and cleans up simulation-owned Docker resources. The result is stored automatically under .crashlabs/runs/.

Inspect the retained result

The command prints a short stored-run selector. Use it directly:

crashlab view <stored-run>
crashlab view <stored-run> --json

The human report explains the outcome. The JSON projection is bounded for coding agents and includes failed phases, checks, expected and observed behavior, evidence references, and exact focused rerun commands. Use --raw only when you need the complete authoritative result.

Measure reliability

crashlab run --repetitions 3 --trust-evaluation-code

Every repetition receives a fresh world and independent trial. CrashLab keeps end-to-end success, behavior among infrastructure-healthy trials, infrastructure availability, target completion, criterion reliability, latency, tokens, and cost as separate measurements.

Check the candidate against the baseline

crashlab check --trust-evaluation-code

The configured baseline and candidate run against the same selected evaluation cases. CrashLab records source identity, uncommitted candidate changes, criterion-level deltas, reliability, and the caller-selected gate result.

Sharing candidate-owned evaluation code across variants is expected. If no included target source changed, interpret the result as a same-source control/reliability sample rather than evidence about a regression.

Exit codes

Exit	Meaning	What to do
`0`	The selected run or policy passed	Continue your normal review workflow.
`1`	Behavior or the selected policy blocked the change	Inspect failed checks, fix the target, and rerun.
`2`	Infrastructure prevented a trustworthy decision	Repair the sandbox, provider, or verifier issue; do not treat it as an agent regression.

Focus one case

When diagnosis returns a focused argv command, execute it exactly. A typical repository-owned package rerun is:

crashlab run --case <case-id> --trust-evaluation-code

Before shipping, execute the complete configured reliability_check returned by onboarding again. A focused pass is evidence about one case, not the entire gate.