Agent evaluations — Yard documentation

Technical reference synchronized with Yard’s source. File names and development commands refer to the native app repository.

Yard evaluates committed implementations against a captured acceptance contract. The normal Build → QA → Review → Done lifecycle, human approval policy and exact reviewed commit remain in force. Deterministic checks establish acceptance; supplementary model judgments provide evidence for review. There is no weighted score that can offset a failed mandatory requirement.

Configure a project

Existing projects retain their behavior with evaluations.mode = "off". In the project's engine settings under Verification, choose observe to collect evidence or enforce to require every captured criterion to pass before delivery. Configurations live in .yard/configs/<slug>.json; older .claude/yard-board/configs paths remain supported. Unknown configuration keys are preserved.

For example, merge the following settings into a project's existing configuration:

{
  "verify_commands": ["php artisan test"],
  "evaluations": {
    "mode": "observe",
    "checks": {
      "registration": {
        "command": ["php", "artisan", "test", "--filter=RegistrationTest"],
        "timeout_seconds": 60
      }
    },
    "bindings": {
      "42": {
        "AC-1": ["registration"],
        "AC-2": ["registration"],
        "AC-3": ["verify-1"]
      }
    }
  }
}

Task 42 supplies the requirements, in its ordinary description:

## Acceptance Criteria

- [ ] [AC-1] A valid registration persists exactly one user and returns the specified response.
- [ ] [AC-2] A duplicate email is rejected without creating another user.
- [ ] [AC-3] Existing behavior remains compatible with the full regression suite.

## Criterios de aceptación is also supported. All checklist entries in that section are mandatory. Explicit IDs are recommended; entries without them receive AC-1, AC-2, etc. in order. Code fences are not interpreted as requirements. Reordering an implicit checklist changes its contract. An existing checkmark does not establish that a criterion passed.

Bindings select operator-configured check IDs. Requirements cannot supply commands. verify-N refers to the corresponding nonempty verify_commands entry; these results are reused, so several criteria bound to the same check do not run it repeatedly. Additional checks take argument arrays; shell syntax only works when the operator explicitly configures a shell executable and arguments.

Choose meaningful behavioral checks that fail for incorrect implementations. A command that exits zero without testing the requirement provides no useful acceptance assurance. The feature map inspector continues to select project scenarios and identify uncovered paths; its metadata validity alone is not a passing execution.

Execution and evidence

Before dispatch, Yard captures the current task text hash, criterion IDs/descriptions, check definitions, verification commands, evaluator version, mode and judge configuration/rubric hash. Frozen contracts are stored in the host's native runtime directory, outside the repository write scope. The contract persists across retries and runs while the delivery episode remains active. Intentional requirement/configuration changes need an operator to review and refresh the contract in the task inspector or explicitly use evals freeze. A reopened delivered task captures a new contract. The task source remains the only backlog; the contract is an execution snapshot.

After an advancing QA stage, the engine verifies the committed task branch in its own detached worktree. Both local and GitHub merge gates use the same executor after integrating the current base into their scratch worktree. They still require the reviewed SHA and all existing approvals. Changing task requirements, check definitions or rubric content requires a refreshed contract. Changing the branch or base requires fresh verification. Checks cannot commit, reset or change tracked source or introduce uncommitted source and still deliver the previously reviewed implementation.

Result	Meaning
`passed`	Every mandatory criterion has passing bound checks and project verification passed.
`failed`	A check demonstrated a failure.
`inconclusive`	Evidence is incomplete or a check could not run reliably.
`not_run`	The criterion has no binding, evaluation is disabled, or an earlier failure prevented execution.

Project verification failures block delivery in every mode. In enforce, missing criteria, unmapped checks, timeouts and inconclusive mandatory results also block. In observe, these remain visible without changing the existing merge policy. Observation-mode merges count as delivered; they count as acceptance verified only when their final integration evidence actually passed. Cancellation and existing verification-command timeout behavior preserve the stopped worktree. Native QA failures return to Ready with a task comment listing the failed criterion IDs and evaluated commit. Inconclusive or missing acceptance evidence moves to Blocked so another agent does not spend tokens retrying without corrected configuration or new evidence.

Reports live in .yard/runtime/<slug>/evaluations, with unique artifacts and a latest-report pointer per task. The evidence journal records evaluation.contract and evaluation.finished events correlated with run, task, attempt and stage. Bounded stdout/stderr is redacted. The task inspector shows each criterion, the recorded commit, evidence freshness and advisory judgments.

CLI

Use the packaged engine executable or swift run --package-path Engine/tools/macos yard-agents-engine:

yard-agents-engine evals contract 42 --root "/path/to/My Project"
yard-agents-engine evals freeze 42 --root "/path/to/My Project"
yard-agents-engine evals run 42 --branch loop/issue-42 --root "/path/to/My Project"
yard-agents-engine evals report 42 --root "/path/to/My Project"

contract previews without freezing; freeze explicitly replaces the saved contract. run checks an existing branch without publishing or merging. Normal board execution freezes automatically. Use the actual branch name for the configured task source; Yard's integrated board namespaces it.

GitHub evaluated delivery requires a single originating task linked to the PR. For a Yard/plugin task delivered through GitHub, the reviewer additionally supplies --evaluation-task <task> to the merge helper; the engine validates its explicit PR/repository link. Local delivery retains --task <task> --branch <branch> --expect-head <SHA>.

Rubric judge and human calibration

Optional evaluations.judge settings:

{
  "enabled": true,
  "host": "codex",
  "model": "gpt-6-luna",
  "reasoning_effort": "low",
  "rubric": "docs/evaluation-rubric.md",
  "timeout_seconds": 60
}

The rubric is a bounded project Markdown file captured by hash. The judge receives deterministic results and a bounded code diff. It runs without tools using the selected Codex or Claude CLI account, cites evidence and can abstain. A Codex judge does not require Claude to be installed or authenticated. When host is omitted, the judge uses the project's provider if it is Codex or Claude; otherwise it uses Codex. Set the judge's host explicitly when comparing different builder providers with the same evaluator.

Judge model selection is independent of the builder's model. An omitted model uses an economical default for the selected judge provider; Codex judging uses low reasoning by default, with an optional reasoning_effort override. The judge's settings do not replace the builder's stored model or reasoning settings. Failed, missing or invalid judge output is inconclusive. Judgment is advisory and never approves a merge or turns a failed deterministic criterion into a pass. Review truncated or incomplete evidence before drawing conclusions.

Calibrate a rubric/model combination with human-labeled examples:

yard-agents-engine evals calibrate-judge --rubric docs/evaluation-rubric.md \
  --dataset evals/judge-labels.json --output /tmp/yard-judge-calibration.json

The dataset has schema_version: 1 and 4–100 examples, each with unique id, evidence object, human expected (passed, failed, inconclusive) and split (calibration or holdout). Include both positive and negative examples and a held-out set. The report shows agreement, held-out agreement, false passes, dataset/rubric hashes and individual verdicts. A calibrated report means agreement on those examples; it does not authorize automatic promotion. Recheck calibration when changing the judge's provider, model, reasoning settings or rubric. The standalone evals judge --rubric <file> --evidence <json> command evaluates a supplied evidence bundle.

Improve quality, then latency and cost

Reproducible benchmarks compare complete agent configurations on identical tasks, with independent final-state graders and reference solutions. Start with the executable five-case corpus, then add production tasks and observed failures from each target repository. Keep held-out tasks out of prompt tuning. The old imported plugin eval files are separate historical resources, not the native benchmark runner.

Delivery metrics accumulate attempts, evaluators and cost over a delivery episode. A reopened task starts a new episode. Missing Ready timestamps, cost, human intervention observation or post-delivery feedback remain unknown, with coverage shown. Use existing feedback and deployment recording to link real defects to confirmed merge commits.

Operators can explicitly close an abandoned episode or record human involvement:

yard-agents-engine evals abandon 42 --reason "Superseded by a different requirement"
yard-agents-engine evals intervention 42 --kind clarification --action started --id scope-question
yard-agents-engine evals intervention 42 --kind clarification --action finished --id scope-question

Append --coverage-complete only when explicitly attesting that all human involvement for the episode has been recorded. An isolated interaction never establishes autonomous operation. Yard's integrated board records Ready transitions for new work; external sources without that timestamp show dispatch-to-delivery duration and unknown Ready coverage.

Compare lost acceptance cases before promoting a cheaper or faster configuration. Keep existing worker/wave limits and benchmark retry/time budgets bounded; feed back specific failed criteria rather than asking an agent to retry without new evidence. Settings and skill changes continue through the existing review/calibration mechanisms. Benchmarks do not automatically rewrite them.