Coding-agent development loop

CrashLab is designed to stay in the coding loop. The coding agent operates the same configured commands and evidence paths as a human developer; it does not need a separate testing API.

1. Run the configured candidate

Execute the exact next_commands.smoke_run argv array returned by crashlab init --json. For a repository-owned evaluation package it typically looks like:

crashlab run --repetitions 1 --trust-evaluation-code --json

Machine-readable result data goes to stdout. The short selector is printed to stderr as stored-run: …. Capture the streams separately; do not scrape the human terminal report. The authoritative result is retained automatically under .crashlabs/runs/ whether the behavior passes, fails, or is inconclusive.

If setup fails before a trial exists, stdout contains a bounded crashlab.command-error.v1 document instead of a result and there is no retained selector. For missing_host_secret, set the named variable on the host and rerun the same command; never add the credential value to the repository.

For human-driven work, omit --json:

crashlab run --repetitions 1 --trust-evaluation-code

2. Ask for a bounded diagnosis

crashlab view <stored-run> --json

The crashlab.agent-diagnosis.v1 projection contains:

• one outcome and exit meaning;
• failed or skipped execution phases;
• non-passing checks with expected and observed behavior;
• evidence references and bounded infrastructure diagnostics;
• affected case IDs and, for comparisons, the baseline or candidate variant;
• target-model and semantic-judge usage/cost as separate metrics when present; and
• exact focused rerun and full-check commands.

Use crashlab view <stored-run> --raw only if diagnosis points to evidence not included in the bounded projection.

A passing comparison can retain non-passing baseline findings as improvement evidence. Those findings are labeled variant: "baseline"; they do not change the root passing outcome into a candidate failure. The diagnosis preserves the outer case ID and uses it in the focused argv commands even when a nested trial does not repeat that identity.

3. Classify before editing

If the result is inconclusive, fix the infrastructure issue first: Docker, provider access, target startup, or required verification. Do not modify the agent merely to turn an unavailable verifier into a pass.

If the result is fail, inspect the specific check and evidence. Change the smallest relevant part of the agent system: source, prompt, tool schema, model, inference settings, or orchestration. Do not weaken the verifier or patch the target adapter to hide a regression.

4. Rerun the focused case

Execute next_commands.rerun_focused as an argv array. It preserves the project configuration, selected case, and evaluation-code trust decision. A typical command is:

crashlab run --case <case-id> --trust-evaluation-code

Repeat diagnosis and repair until the focused behavior is healthy. Every rerun receives a fresh world.

5. Run the complete regression check

Execute the exact next_commands.reliability_check array returned during onboarding. For a repository-owned package, the typical command is:

crashlab check --trust-evaluation-code

The full check compares the configured baseline and candidate with the configured repetitions and customer-selected policy. A focused rerun cannot substitute for this final coverage.

Candidate-owned evaluation code is intentionally shared across both target variants. If no included target source changed, the check is a same-source control/reliability sample, not regression evidence. Commit initial setup and make the intended agent change before the historical comparison.

Useful project commands

crashlab runs --json           # list retained runs
crashlab view                  # latest retained result
crashlab view <prefix> --json  # bounded coding-agent diagnosis
crashlab check --case <id> --trust-evaluation-code

A practical agent instruction

Run the configured CrashLab evaluation. If it does not pass, inspect the stored
run with `crashlab view <selector> --json`. Fix only the target or its setup,
execute the diagnosis's focused argv command, then execute the complete
configured reliability-check argv array. Preserve the retained evidence and
report any inconclusive infrastructure failure separately from behavioral
regressions. Never weaken the verifier to make the gate green.