Run CrashLab in CI

CrashLab's explicit commands are non-interactive and preserve distinct exit codes for pass, behavioral block, and infrastructure uncertainty. CI should run the same repository-owned evaluation package that developers and coding agents use locally.

Before adding the gate

Commit a complete crashlab/ package and run it locally first. CI is not the place to discover the agent entrypoint, replace scaffold placeholders, or decide which behavior should matter. The baseline revision must also contain the target source needed by the configured comparison.

Store model-provider credentials in the CI platform's secret store. Never put a credential in repository variables, workflow arguments, Docker build arguments, the evaluation package, or an uploaded result.

GitHub Actions example

The 0.1 packaged release supports macOS through Homebrew. A minimal trusted-branch gate looks like this:

name: Agent regression

on:
  workflow_dispatch:
  pull_request:

permissions:
  contents: read

jobs:
  crashlab:
    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: macos-15-intel
    environment: crashlab-evaluations
    timeout-minutes: 45
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
        with:
          fetch-depth: 0
      - uses: docker/setup-docker-action@77e84dbf09b47d1e29270283c22f16145aa85ca1 # v5.4.0
      - uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
      - run: docker version && docker buildx version
      - run: brew install crashlabsai/tap/crashlab
      - run: crashlab eval validate --trust-evaluation-code --json
      - run: crashlab check --trust-evaluation-code --json

Use the provider variable actually required by your agent. A full Git history is needed when the inferred baseline is the repository's default branch. The example uses GitHub's Intel macOS runner because Docker's setup action does not support GitHub-hosted Apple Silicon runners: nested virtualization is unavailable there. For a self-hosted Mac, omit those setup actions only when a Docker daemon and Buildx are already running. Keep third-party actions pinned to reviewed commit SHAs.

Create the crashlab-evaluations environment with required reviewers before adding provider secrets. The job should wait until a maintainer reviews the code that will receive those credentials. The same-repository condition also rejects fork pull requests; keep it even when the environment requires approval.

Preserve the decision, not raw customer data

crashlab check --json writes the bounded comparison projection to stdout and returns:

Do not turn exit 2 into success. Retry only after classifying the provider, Docker, target, or verifier failure.

Authoritative results remain under .crashlabs/runs/ on the runner. They may contain agent output, tool arguments, state changes, and local paths. Prefer the bounded JSON diagnosis in job logs. Upload raw run files only to an access- controlled artifact store with an explicit retention policy, after reviewing the data your evaluation can retain.

Keep the coding-agent loop local

CI is the final gate, not the fastest debugging interface. A coding agent should run the focused evaluation locally, inspect the stored diagnosis, repair the target, and run the full comparison before opening or updating a pull request. CI then confirms that the reviewed commit passes the same configured contract.