Author an evaluation package

An evaluation package contains domain-specific code that should not live in CrashLab core. Start with one behavior that would matter in a pull request, then expand coverage as the agent evolves.

Package shape

crashlab/
  evaluations.py
  pyproject.toml
  uv.lock
  target_hook.py       # Python/uv scaffold
  target_hook.mjs      # Node/pnpm scaffold (alternative)
  README.md

Both supported scaffolds use a Python evaluation package because the protected world and verifier execute in CrashLab's host runtime. Only the target hook and immutable target snapshot differ by source runtime.

CrashLab infers evaluations.py:PLAN and uv.lock from this layout while still digesting both for exact provenance.

Use the exact Python scaffold command returned by repository inspection. A root uv project commonly uses:

crashlab eval scaffold --runtime python-uv --json

A nested requirements-based application may instead use:

crashlab eval scaffold --runtime python-requirements \
  --target-project python-backend \
  --requirements-file requirements.txt \
  --json

For a repository-root Node/pnpm agent with a pinned pnpm version and lockfile, use the corresponding target shape:

crashlab eval scaffold --runtime node-pnpm --json

The mounted hook receives CRASHLAB_TARGET_ROOT as the immutable repository root. In a pnpm monorepo, resolve runtime packages from the package.json that owns the selected agent; do not assume nested dependencies are hoisted beside the hook. Use the repository's existing locked TypeScript loader when the production entrypoint is TypeScript.

The scaffold is intentionally incomplete. Its world operations, host-owned provider gateway, production target hook, and protected verifier fail until a coding agent replaces them with repository-specific implementations. It is an authoring aid, never passing evaluation evidence. The generated verifier shows the typed protected-evidence fields and supplies a package-local result helper, so authors do not need to reverse-engineer the alpha result constructor.

The scaffold also includes EvaluationCoverage, VerifierCoverage, and related human-facing descriptions. Replace every placeholder even though coverage does not execute the world or call a model. CrashLab uses it to explain what is about to run and how it will be checked. Its title and summary are retained with the execution request and result, so local and request-driven execution use the same customer-facing identity.

Use python-uv for a root or nested pyproject.toml plus uv.lock, and python-requirements for a selected requirements.txt project. CrashLab records requirements installs as unlocked rather than pretending they are reproducible. Use node-pnpm only for a repository-root package.json plus pnpm-lock.yaml and a pinned packageManager field. Do not reshape a monorepo, add duplicate root package metadata, or regenerate the target's dependency lock merely to fit a scaffold.

No CrashLab-specific package manifest is required. pyproject.toml and uv.lock remain necessary because the protected evaluation runtime must be reproducible.

Design the smallest useful case

A first case should define:

1. a deterministic initial world state;
2. the target's natural production entrypoint;
3. functional tool, MCP, API, filesystem, or subagent boundaries;
4. at least one required outcome;
5. at least one invariant or minefield for collateral behavior; and
6. limits for time, turns, resources, and provider requests.

The world must be coherent. If ticket.create succeeds, a later ticket.get must observe the new ticket. A list of fixed mock responses is not a stateful world.

Describe coverage before execution

Coverage should answer five questions without constructing a trial:

1. What customer-visible behavior does this evaluation protect?
2. What resources and initial facts exist in the fresh simulated world?
3. What can the agent read or change, and which declared operations implement those capabilities?
4. What does each verifier check, by what method, and from which protected evidence?
5. Which real target and orchestration path will run?

Use human names and descriptions, not internal IDs or enum values. Keep the coverage honest and compact: it is an explanation of the authored evaluation, not a second verifier. Runtime criteria and protected evidence remain authoritative after execution.

Keep boundaries explicit

World

Own the seed, dependency operations, state mutations, and causal event log outside the target. Validate operation arguments and results at the boundary.

Target adapter

Invoke the real prompts, tools, model, and orchestration. Adapt transport when needed, but do not patch the target into a known-good answer. Run target source in Docker unless the customer explicitly chooses and records the unsafe process opt-out.

Import an existing source-controlled agent, factory, graph, application entrypoint, or runnable example. Do not construct a new agent, prompt, or tool graph inside the evaluation hook and present it as the configured target. When a framework/library repository has no runnable agent, evaluate the application that uses it or stop and request the intended entrypoint. A shipped example is a valid target only when the hook imports and preserves its source-controlled behavior.

Python hooks return the explicit CrashLab payload rather than copying field names from a framework result:

return TargetOutput(
    summary=str(result.final_output),
    data={"run_id": str(result.run_id)},
)

When a model SDK treats its base URL as an API prefix, declare that prefix on the host gateway instead of patching it into target code. For example, an SDK that appends /responses should receive container_base_url_path="/v1" while the gateway allowlists the complete /v1/responses path. The target receives a short-lived token and private URL; the long-lived provider credential remains on the host.

Verifier

Read protected initial/final state and events. Verify outcomes and real safety properties without prescribing incidental reasoning steps. Treat attempted tool use, handler dispatch, mutation, and returned results as different evidence.

For a natural-language property that cannot be reduced to those structured checks, add SemanticBehaviorVerifier from crashlab.alpha. Supply one SemanticBehaviorSpec, a protected SemanticJudge, and a selector that returns uniquely named SemanticEvidenceItem values from the EvidenceBundle.

Keep the evidence catalog bounded and intentional. Do not send an unfiltered raw trace to the judge. Invalid catalog construction is evaluation code failure; judge unavailability is a typed inconclusive assessment. The default severity is advisory, but the evaluation may explicitly select Severity.REQUIRED. The supplied OpenAI Responses implementation uses a HostSecret, exact model/path allowlists, strict structured output, no tools, bounded requests, and store=false; a customer may implement the provider-neutral SemanticJudge protocol instead.

Lock and validate

The current frozen package runtime supports uv.lock:

cd crashlab
uv lock
cd ..

crashlab eval validate \
  --trust-evaluation-code \
  --json

Validation checks the conventional descriptor, lock freshness, package digest, entrypoint, plan, and cases in a fresh frozen environment. An unlocked package requires an explicit --allow-unlocked-evaluation opt-out and is recorded as such.

Confirm the inferred project

crashlab init --json

The visible crashlab/ directory already connects the repository, so initialization does not write another configuration file. It validates the inferred package and returns the exact smoke and comparison commands. Treat each returned command as an argv array, not a shell string:

crashlab run --repetitions 1 --trust-evaluation-code
crashlab check --repetitions 1 --trust-evaluation-code

Prove the verifier can reject bad behavior

Before relying on a case, exercise counterexamples:

• wrong or missing state mutation;
• collateral changes to protected records;
• forbidden or destructive operation attempts;
• forged success text without dependency dispatch;
• missing or reordered causal milestones;
• duplicated mutation after a retry; and
• unavailable required verifier evidence.

A verifier is only as valuable as the failures it can distinguish.