Evals

Examples

An example is a folder under .reqfile/<ID>/examples/:

.reqfile/COLOCATION/examples/distant-test/
  example.yaml   # what the checks should say
  files/         # the case, the only part the checks see
    src/checkout/discount.ts
    test/unit/checkout/discount.test.ts

expected: violation                          # violation | ok
findings: [test/unit/checkout/discount.test.ts]   # optional: where the checks must flag
rationale: The test sits in a mirror tree while its sibling tests are colocated.
origin: authored                             # or real:<commit>, for a case met in real use
known: miss                                  # optional: a failure tracked without failing
split: holdout                               # optional: kept out of tuning (or any name), reported apart

Examples double as the requirement's documentation: a violation and an ok case side by side say what the must means.

reqfile eval

reqfile eval puts each case's files/ alone in a fresh repository and runs the requirement's effective checks on it, with $REQFILE_ASSETS and the Jev settings of its folder. The label stays outside, so no check can read its answer. A violation flagged elsewhere than its findings counts as missed. known: miss or known: false_alarm records a failure the checks are known to have, reported without failing, until the checks improve. A requirement taken with use is measured on its definition's examples and its own.

UNUSED_CODE  asserted: 2 examples, 2 as labeled
FAIL_FAST  measured, no assertion: 3 of 4 violations caught (95% interval 30 to 95%; 1 missed, 0 not selected, 0 uncertain); 0 of 2 correct examples flagged (95% interval 0 to 66%; 0 uncertain)
  rust-ok-discard: missed
SIMULATION  no evidence: no labeled examples

3 requirements: 1 asserted, 1 measured, 1 without evidence; 6 examples. 0 failing their labels.

Each requirement falls in one of three classes:

• asserted: deterministic checks only, which must match every label, or reqfile eval exits 1;
• measured: with a decision check, or a command reporting probabilities, results are counted, never asserted: judgments vary, so exit 0 does not mean the check works. A detector earns its place, or mode: blocking, by the rates it reaches on examples it was not tuned on;
• no evidence: no labeled examples, so nothing is known about its checks.

Violations no check looked at are reported as “not selected”, apart from those it looked at and missed. Each rate comes with its 95% Wilson interval: on a few examples it is wide, and two rates whose intervals overlap are not evidence of a difference. A case that cannot run exits 3.

Capture a mistake

Checks become precise over time when their failures become evidence. reqfile example add copies files of the repository into a new example, with the commit it came from:

reqfile example add FAIL_FAST swallowed-retry --expected violation \
  --finding src/sync/retry.ts --rationale "Why it has this label." src/sync/
reqfile eval --only FAIL_FAST

The learning loop

A requirement's must is a person's promise; its checks are the machine's reading of it; its examples are where the two meet. Checks converge on what people mean because every disagreement between them ends as an example:

1. One precise question per check. The must may be broad; each check answers one narrow question about it: a lint rule, a script's criterion, a decision's question with its violation_when and ok_when.
2. A mistake becomes an example. A false alarm someone disputes, or a violation found by sampling what passed, goes into .reqfile/<ID>/examples/ with reqfile example add. Misses are silent: reqfile check --log records what passed too, so sample it, and review the uncertain findings.
3. Someone else labels it. Whoever changes the check does not choose the label. When two labelers disagree, the must is ambiguous: a person rewords it, rather than the check being tuned to one reading.
4. One change, measured. reqfile eval shows the failure; one change to the check is kept only if the held-out examples improve beyond the noise and no other case regresses.
5. Rates are the contract. Asserted checks match every label; measured ones report rates with intervals. A check becomes blocking only once its measured rates meet a target chosen beforehand.

People own the must, the labels of disputed cases, and the choice between requirements that contradict each other. Agents fix code, capture mistakes as examples, and iterate on checks within the evidence. The output says which is which: a violation is work for the agent; a decide line, a contested label or a rate below its target is a question for a person.

Improving a check without fitting its examples

• Held-out examples. split: holdout (or reqfile example add --holdout) keeps an example out of tuning; reqfile eval reports the tuning and held-out examples on separate lines. Choose about a third of each class before looking at results, and revert a change that raises the tuning rates and not the held-out ones. Any other name of lowercase letters, digits and dashes, such as split: confirmation-1, makes a split of its own, reported apart by that name.
• Fast iterations. reqfile eval --format json gives each requirement's counts per split and every case with its outcome, the files flagged, its known and its error. --command-override 'FROM=>TO' measures another command for one run, such as a package's working copy instead of its published runner.
• Noise. reqfile eval --repeat N runs each case N times and lists the cases whose outcome changes; 3 to 5 runs are enough to see it. A change improves a check only when it moves more cases than are unstable. An asserted requirement whose outcome changes between runs fails.
• Uncertainty is the review queue. Findings between pass_below and violation_above are reported as uncertain: those are the cases worth a person's label.
• Thresholds. Choose thresholds on the tuning examples and confirm them on the held-out ones.
• Headroom. When every example is classified as labeled, the benchmark cannot show that a change helps. Add cases a person judges hard, and say why in rationale.
• No leaks. No example's text, names or identifiers go into a command, a decision question or a threshold.
• Labels first. Before changing a check for a failing case, reread its label against the must; a wrong label is fixed in example.yaml.

What each step establishes

Exit 0 means the checks passed, not that the requirements hold.

Adapter evidence

Eval executes resolved adapter leaves with their original checker assets, while project bindings and scenario code come from the isolated example. The runner does not inject fixture labels into bindings. Commands remain trusted executable code; eval is not a filesystem sandbox. Preserve independent confirmation splits when changing checkers; a format migration does not upgrade an alpha checker to a reviewed policy.