CrashLab

CrashLab runs your real agent against a controlled, stateful world before you ship it. It verifies outcomes and safety properties from protected evidence, keeps model variability visible, and stores every result in the repository for humans and coding agents to inspect.

Start with one useful workflow

brew install crashlabsai/tap/crashlab
crashlab skills install

cd /path/to/your-agent

Then ask your coding agent:

Set up CrashLab

For an unknown repository, crashlab init is not the first command. The bundled skill inspects the project, starts from the compatible Python/uv or Node/pnpm source scaffold when one is unambiguous, connects one real workflow, and only then initializes and runs the project. The scaffold is deliberately unable to pass until its world, target hook, provider gateway, and verifier describe the real agent; its coverage placeholders must also explain that evaluation before setup is complete.

What CrashLab proves

CrashLab is for behavior at the boundary between probabilistic agents and deterministic systems:

• tool and MCP operation selection, arguments, dispatch, and returned results;
• coherent reads and writes against simulated APIs or application state;
• workflow outcomes, invariants, minefields, and causal milestones;
• subagent coordination and recovery from injected failures;
• reliability across fresh trials; and
• model identity, latency, token use, and attributable cost when available.

It does not replace unit tests for deterministic application code, and it does not evaluate a model in isolation. The model, prompts, tools, framework, orchestration, source, and runtime configuration are all part of the target agent system.

The development loop

change the agent
      ↓
run one focused evaluation
      ↓
inspect the failed phase, check, and evidence
      ↓
fix the agent
      ↓
compare the candidate with the baseline

CrashLab keeps the authoritative result under .crashlabs/runs/ and returns a short selector. crashlab view <selector> --json gives a bounded diagnosis that a coding agent can use without parsing internal result schemas.