Benchmark degli agenti

Technical reference synchronized with Yard’s source. File names and development commands refer to the native app repository.

Yard can replay programming tasks against a pinned Git commit and compare complete Build → QA → Review executions. Protected final-state graders determine whether the implementation satisfies the acceptance criteria. A stage saying “passed” is not a passing grade.

The benchmark does not dispatch the project board, create pull requests, publish, merge or deploy. Each trial gets its own shallow repository containing only the initial commit, with no remote. The source checkout and its uncommitted files are never modified. Commands are argument arrays, so paths containing spaces work.

Try the twenty-two reference cases without models

From the Yard repository root:

python3 Engine/tools/macos/prepare-resources.py
python3 Engine/evals/feature-delivery/seed.py /tmp/yard-eval-fixture
swift run --package-path Engine/tools/macos yard-agents-engine benchmark validate \
  --manifest /tmp/yard-eval-fixture/manifest.json
swift run --package-path Engine/tools/macos yard-agents-engine benchmark run \
  --manifest /tmp/yard-eval-fixture/manifest.json \
  --output /tmp/yard-eval-reference-evidence --validation-only

Use new fixture and output directories for another run. Existing directories are preserved. Seeding creates a deterministic local Git repository and writes its actual commit SHA into the manifest. Node is the only runtime dependency of these examples; the seeder records its discovered executable path.

The twenty-two example cases span simple validation through concurrency, atomic persistence, transactional inventory and signed webhooks. The manifest labels difficulty and reserves five cases as holdouts. Each case must fail on the initial repository and pass with its reference patch. Reference validation never invokes an agent or a judge and spends no model tokens. The small examples exercise the benchmark; they are not evidence of model quality on a production application. Add representative tasks from the repositories your factory actually delivers.

Compare agent configurations

Review the generated manifest before starting a real run. Its default variants use Codex gpt-6.1-terra with high reasoning and Fast service tier in all stages. They can be changed to the exact configurations under investigation. Authentication uses the existing CLI accounts; no credentials belong in the manifest. The two providers are example variants, not mandatory dependencies: keep or add only Codex variants to compare Codex configurations without installing or authenticating Claude.

swift run --package-path Engine/tools/macos yard-agents-engine benchmark run \
  --manifest /tmp/yard-eval-fixture/manifest.json \
  --output /tmp/yard-eval-agent-evidence
swift run --package-path Engine/tools/macos yard-agents-engine benchmark compare \
  --report /tmp/yard-eval-agent-evidence/report.json

The default corpus runs twenty-two tasks, one variant and one repetition: twenty-two trials, each with up to two attempts and three agent stages per attempt. The timeout and known-cost limit apply to the whole trial, including retries. Stage invocations share the remaining time budget. A known spend above max_cost_usd stops the trial. Providers that do not report spend leave cost_budget_status as unknown; Yard cannot enforce an unreported dollar limit. Reports show cost coverage and never substitute zero for unknown costs.

Model and effort pins, manifest and input hashes, initial commit, initial tree, engine/OS details, per-stage evidence, grading results and changes are retained. Variant order alternates between repetitions. All trials start from the same commit and run in fresh repositories; retries within a trial retain their work and receive the previous attempt's concrete findings.

Yard creates native checkpoint commits after successful Build and QA stages, including newly created tests and excluding private files and runtime state. The next stage receives the existing implementation as the same trial's work. Review examines changes since the pinned initial commit, including these checkpoints. Checkpoint SHAs and saved paths remain in the stage evidence.

To compare prompts or skill revisions, a variant can include stage_instructions, an object mapping build, qa and review to lists of Markdown paths relative to the manifest. The runner snapshots their complete text before any agent starts, includes it in that stage's prompt, records each hash and incorporates those versions into config_id. Changing the original files during the experiment does not change later invocations. Use explicit instruction snapshots when testing the production factory's skills: the default benchmark stage policy is specialized for isolated execution and does not run the real delivery/approval workflow.

Manifest contract

The version 1 manifest contains:

A case has an id, prompt, reference_patch, optional boolean holdout, and criteria. Every case needs at least one deterministic criterion. A criterion has id, description, command as an argument array, and files listing its operator-owned input files. It can set a timeout_seconds of at most 600 seconds. For example:

{
  "id": "tenant-access",
  "prompt": "Deny invoice access across tenant boundaries, including admins.",
  "reference_patch": "references/tenant-access.patch",
  "criteria": [{
    "id": "AC-1",
    "description": "Owners and admins can read only invoices in their tenant.",
    "command": ["node", "{inputs}/graders/tenant-access.cjs"],
    "files": ["graders/tenant-access.cjs"]
  }]
}

The engine substitutes {inputs} and {workspace} within individual arguments, without invoking a shell. A grader runs with its working directory in the input snapshot. YARD_BENCHMARK_WORKSPACE points at the agent checkout and YARD_BENCHMARK_INPUTS points at the snapshot. Exit 0 means passed, 1 means the expected behavior is absent, and other codes mean inconclusive infrastructure or grader failure. Grader timeouts are inconclusive. An agent reaching its trial timeout is a budget failure. Cancellation stops the experiment and retains its partial report.

All input files and reference patches use canonical relative paths beneath the manifest directory. They are bounded and snapshotted before execution. Hashes are checked after agent invocations and around grading. Changing an expected result or grader invalidates the trial. The snapshot is outside every agent checkout; it is integrity-checked, not a hostile-code security boundary. Agents run as the local user, so untrusted repositories or tools still require a suitable isolated machine or container. Do not give benchmark agents production credentials.

Supplementary model judgments

A criterion may specify judge instead of command, while the case retains at least one deterministic criterion. Its judge.rubric names a snapshotted Markdown file. An optional judge.calibration file is also hashed and retained. The judge receives bounded evidence and can abstain with inconclusive; it cannot edit the checkout or authorize a delivery. The CLI records model judgments separately from the deterministic result. They do not turn failed tests into a passed trial.

Use evals calibrate-judge --rubric <file.md> --dataset <file.json> to compare a judge with labeled human examples before relying on it. A rubric file or an unexamined model score is not proof of calibration. The benchmark's supplementary judgments are advisory until that agreement has been assessed; calibration is not repeated for every trial.

Configure the judge under each variant's config.evaluations.judge. Codex and Claude both support judging without tools. For example, a Codex-only benchmark can use this judge configuration:

{
  "evaluations": {
    "judge": {
      "host": "codex",
      "model": "gpt-6-luna",
      "reasoning_effort": "low",
      "timeout_seconds": 60
    }
  }
}

The judge can use a different provider or model from the builder. If omitted, its provider follows the variant's Codex/Claude host, falling back to Codex for other hosts; its model uses the judge provider's economical default and Codex reasoning defaults to low. Builder model and reasoning pins remain unchanged. Set the same explicit judge configuration across variants when that evaluator should remain fixed, and recalibrate if the judge's provider, model, reasoning or rubric changes.

Read the results

report.json contains controls, ordered trials and a paired comparison against the first variant. Each trial retains attempts, stage results, deterministic criterion evidence, supplementary judgments, costs/tokens when reported, and a changes.patch. Workspace repositories and newly created files remain available for inspection. A harness run can complete while some task trials fail; inspect the trial statuses and pass rates rather than interpreting command exit zero as all features accepted.

Comparisons expose task/repetition pairs, candidate-only and baseline-only successes, infrastructure exclusions, holdout pairs, task-consistency intervals, and duration/cost differences for pairs that both completed successfully. Cost deltas require full cost coverage on both sides. Cost per accepted result includes the measured spend of failed trials as well as successful ones.

The consistency interval counts unique tasks that passed every repetition. The paired sign test uses one improvement/regression direction per unique task, so repeated attempts do not inflate its sample size. These remain descriptive samples: a small corpus cannot establish broad model superiority. Check lost acceptance cases before optimizing latency or cost, inspect transcripts and grader fairness, and keep holdout tasks out of prompt tuning. The runner never automatically promotes a model or rewrites installed skills. Production traffic cohorts remain observational and are not mixed into these paired benchmark results.

Codex Fast service

Set config.codex.service_tier to fast to request Fast without changing the model or reasoning effort. Yard passes both service_tier="fast" and features.fast_mode=true to Codex, including tool-free judge requests. Omit the setting to inherit the Codex preference. The example corpus and continuous campaigns request Fast by default, including the campaign analysis phase. The requested tier is recorded in benchmark variant settings and their hash; it is a request, not a guarantee that the provider served that tier or that a particular speedup will occur. Availability and higher usage are described in Codex Speed. Keep service tiers consistent when comparing skill changes, or report the tier change as an additional experimental variable.