Evaluation model

CrashLab evaluates an assembled agent system inside a controlled experiment. It does not reduce the test to a prompt and expected string.

evaluation case
  ├─ world        protected state and functional dependencies
  ├─ target       the real agent system being changed
  ├─ verifier     protected checks over state and causal evidence
  └─ limits       time, turns, resources, and provider requests
        │
        └─ fresh trial → retained result

Evaluation

An evaluation describes a user-visible workflow to test, the world in which it occurs, the target boundary, the checks that decide whether it worked, and the limits on execution.

A useful first evaluation is small enough to diagnose but meaningful enough to catch a real regression. “Create one support ticket with the correct priority without deleting the customer” is useful. “Return plausible text” usually is not.

Target

The target is the agent system under test:

• source and orchestration;
• prompts and instructions;
• tools, MCP clients, and subagents;
• framework adapter;
• provider, model, and inference settings; and
• runtime configuration.

Changing the model changes the target. CrashLab records the observed model and provider provenance so model changes can be compared under the same evaluation.

World

A world is a fresh, stateful simulation of the dependencies and state the agent needs. It can model APIs, MCP servers, files, databases, queues, and business rules. Reads and writes are coherent: a successful mutation changes what later reads observe.

The world is protected from the target. It owns the initial seed, performs allowed operations, records causal events, and exposes a final snapshot to the verifier. The target cannot award itself a pass by printing an invented tool result or modifying expected answers.

Verifier

A verifier reads protected evidence after the trial. Evidence can include:

• initial and final world state;
• the world diff;
• dependency operation arguments, outcomes, and mutation effects;
• normalized agent events;
• target output; and
• runtime and provider status.

Verifiers may use deterministic code, model-based judging, or both. CrashLab does not mandate who owns them or which ones gate a change. It records their identity, source relationship, and results. A required verifier that does not run cannot silently pass the evaluation.

Criterion kinds

Prefer outcomes over one prescribed trajectory. Alternate reasoning or tool paths should pass when they produce valid state and respect real safety or causal constraints.

Layered semantic verification

Use deterministic code for observable state, invariants, budgets, and causal events. Use a semantic judge only for meaning that those checks cannot establish, such as whether a free-form answer accurately communicates a policy without requiring one exact phrase.

A semantic criterion receives a versioned behavior clause and a small catalog of strict-JSON evidence items with stable IDs. A substantive judgement must cite exact IDs. Not-applicable or insufficient evidence is inconclusive, as are judge provider outages, refusals, and malformed responses. Broken evaluator evidence selection is a verifier error rather than model uncertainty.

Semantic criteria default to advisory. Evaluation authors may make them required; CrashLab records that choice without imposing one ownership or release policy. The judge's model, settings, latency, tokens, and estimated cost are retained on the criterion assessment separately from target-model usage.

Case, trial, and repetition

An evaluation plan contains one or more named cases. Each case binds one scenario and its fresh world, target, and verifier set.

A trial is one execution of one case. A repetition schedules another independent trial with a fresh world. Repetitions measure nondeterministic reliability; they do not retry a failed trial until it happens to pass.

Evaluation package

Repository-specific evaluation logic lives in a versioned package selected independently from the target:

crashlab/
  evaluations.py
  pyproject.toml
  uv.lock
  ...world, adapter, and verifier modules

The conventional package exports PLAN from evaluations.py and uses its uv.lock. Package code imports from crashlab.alpha; crashlab.core is internal.

The target and evaluation may come from the same repository, a baseline-pinned revision, or independent projects. CrashLab records the exact source and digest that ran rather than assigning ownership policy on the customer's behalf.