Troubleshoot a local evaluation

CrashLab deliberately distinguishes a behavioral failure from an evaluation that could not produce trustworthy evidence. Start with the command's exit code and retained selector instead of changing the agent immediately.

First, verify the installed boundary

crashlab --version
command -v crashlab-viewer
docker version
docker buildx version
docker info >/dev/null

The Homebrew formula owns the CLI, viewer, Python runtime, and uv used for evaluation packages. If either CrashLab executable is missing or mismatched, repair the installation as one product:

brew update
brew reinstall crashlabsai/tap/crashlab

Do not install a second Python copy of CrashLab beside the formula to work around an evaluator error. That creates two release sources and makes diagnosis harder.

No evaluation is discovered

Run from the intended agent repository, not the CrashLab source repository:

crashlab inspect . --json

A configured repository has a visible crashlab/evaluations.py. If it is absent, install the managed setup skill with crashlab skills install, open the agent repository in your coding agent, and ask it to Set up CrashLab. A generated folder is not complete until it imports the real agent and replaces every world, gateway, coverage, and verifier placeholder.

If the package exists in a monorepo, pass the agent project explicitly to inspection or initialization. Do not add a second TOML selector at the root:

crashlab inspect path/to/agent --json
crashlab init path/to/agent --json

Evaluation package validation fails

Review the repository-owned Python before acknowledging that it is trusted host code, then run:

crashlab eval validate --trust-evaluation-code --json

If the error says the evaluation lock is stale, regenerate only the evaluation package lock and commit it with the evaluation source:

uv lock --project crashlab
crashlab eval validate --trust-evaluation-code --json

Do not use --allow-unlocked-evaluation merely to silence a stale lock. That is an explicit provenance downgrade for packages that truly cannot be locked.

A provider credential is missing

JSON mode returns a bounded crashlab.command-error.v1 document whose category is missing_host_secret and whose next step names the missing environment variable. Export that value in the trusted host session or CI secret store, then rerun the same command.

CrashLab does not load a repository .env automatically. Never place the real key in the evaluation package, target environment, Dockerfile, command line, or retained result. The target should receive only the short-lived gateway token.

The run is inconclusive

An inconclusive result means infrastructure, target execution, or required verification prevented a trustworthy behavior decision. Use the selector that the run printed:

crashlab view <stored-run>
crashlab view <stored-run> --json

The diagnosis identifies the failed phase and gives a focused next command. Common boundaries are:

Use --raw only when the bounded diagnosis points to evidence it did not include. Raw results may contain customer data, tool arguments, and repository paths.

The result is a behavioral failure

Exit 1 means CrashLab obtained trustworthy evidence and a required check or selected regression policy blocked the change. Read the expected value, observed value, explanation, and cited evidence for that check. Change the smallest relevant part of the target agent, then execute the diagnosis's focused rerun argv. Run the complete configured check before shipping.

Do not weaken the verifier, change the world to match the bug, or treat repeated tries until one passes as a fix.

No stored result exists

Failures before a trial is created—such as an invalid package selector, missing trust acknowledgement, or missing host secret—return a command-error document and exit 2; they do not invent a run in .crashlabs/runs/. Correct that error and rerun. Once a trial starts, passing, failing, and inconclusive results are retained automatically.

The interactive workspace does not open

Bare crashlab needs a TTY and the crashlab-viewer installed by the formula. Coding agents and CI should use explicit commands with --json rather than launching the TUI. If the viewer is installed but terminal input or layout is broken, include the terminal application, window dimensions, macOS version, and output of crashlab --version in a bug report.

Cleanup reports a problem

Normal commands clean only Docker resources created by that simulation set. Never run a global Docker prune as a CrashLab recovery step: it can delete unrelated customer work. Preserve the bounded cleanup diagnostic and report the CrashLab version, exact command, and redacted resource names. A process killed with SIGKILL can bypass normal cleanup and currently requires careful, identity-specific maintainer recovery.

If the diagnosis still does not identify a safe next step, follow the support guide and remove credentials, customer source, protected seeds, and sensitive evidence before opening an issue.