Choose your first workflow
Pick a small agent behavior that produces trustworthy value quickly.
The smallest useful evaluation protects one decision you would care about in a pull request. It should be small enough to understand, realistic enough to exercise the production agent, and strict enough to reject a plausible bad outcome.
A good first evaluation
Look for a workflow with all four of these properties:
| Property | Useful first case | Avoid initially |
|---|---|---|
| Trigger | One concrete user request or event | An open-ended multi-day workflow |
| Side effect | One or two observable tool or state changes | A broad set of loosely related actions |
| Outcome | A protected fact that code can inspect | “The answer seems good” with no evidence boundary |
| Failure | At least one meaningful invariant or forbidden action | Only checking that the process exited successfully |
Examples include routing one support request without sending a reply, looking up one record before drafting an answer, or creating one ticket without changing unrelated customer data.
Write the behavioral contract first
Before implementing the world, answer these questions in plain language:
- What starts the workflow?
- What production agent entrypoint receives it?
- What can the agent read and change?
- What final state must exist for the evaluation to pass?
- What must never happen?
- Which facts must stay protected from the target?
Those answers become the evaluation's coverage description, world boundary, and verifier checks. They also let another developer decide whether the test protects the behavior they think it protects.
Prefer observable effects
Use deterministic checks for tool arguments, state changes, invariants, request budgets, and causal milestones. Add a bounded semantic judge only when the property genuinely depends on meaning that structured evidence cannot establish. Do not use a model judge to replace an exact check that code can perform.
Define reliability after the case is useful
One trial is the fastest way to debug setup. It is not a reliability claim. Once the world, target, and verifier behave correctly, run several independent trials and preserve the denominator. Start with the configured three-trial check, then increase repetitions only when the cost and decision risk justify it.
Continue with Set up an agent when you can describe one workflow at this level.