finboard · assistant evals

Anatomy of a Gate Run

One eval run is a measurement that has to stay comparable to every run before it. That constraint — not the model call — is what shapes the pipeline. This page explains the mechanism; the sibling pages render what it produced.

Eval Ledger — fix a model, walk time Footprint Ladder — fix an era, walk models Run Inspector — drill into one run Run Composer — compose the next run this page carries no measurements — by construction

The lifecycle

Three gates — selection, multiturn, narration — share this lifecycle: the same inputs, a per-item measure loop, the same two record tiers, the same refusals. The loop as drawn — tool schemas → tool call → dispatch/validate → retry — is the structured pipeline selection and multiturn run; narration swaps that middle box out: it sends the production narration messages with no tools over a pass schedule (the temp-0 pass plus the default-temperature passes), grades the prose against supplied capability results, and its lines record constrained: false. Where each gate’s oracle comes from differs too — the table further down.

INPUTS MEASURE RECORD — TWO TIERS READ evals/sets/ committed · dispatch-verified fixture store isolated · gitignored runtime + model loopback · digest read per item prompt surface + tool schemas → model → tool call → dispatch/validate → grade retry within budget N (narration: no tools · passes) findings/ gitignored · per-run stem free text lives here and only here runs.jsonl committed · mirrors public one allowlisted line no free-text field exists refused partial run · unknown key user path · nothing appended Run Inspector Eval Ledger Footprint Ladder one run fix model, walk time fix era, walk models --local-details thresholds applied in the page, never stored in the ledger The fork is the design: everything uncertifiable for publication stays left of it, and the committed line carries only what an allowlist can prove safe.
The two record tiers are a disclosure boundary, not a convenience split — which is why the Run Inspector needs an explicit, loopback-pinned flag to reach across it, and why serve.py stages a view rather than serving evals/ directly.

Why two runs are comparable

Every input that can move a score belongs to exactly one named digest, so a corpus, grading or protocol change can never masquerade in the trend as a prompt or model effect (spec 14 §5a). An offline coverage test fails until each reachable harness module is claimed.

SOURCE DIGEST ROLE the runtime strings the model reads manifest over every set file graded grader + fixture-builder sources execution scalars + harness execution sources runtime /api/tags prompt_sha eval_set_sha grading_sha protocol_sha model_digest the axis you walk a prompt era — what the Eval Ledger plots against the comparability key ( gate,   model, model_digest,   eval_set_sha,   grading_sha,   protocol_sha ) match ⇒ the two runs sit on one trend Not in any digest: where the weights are stored, the endpoint, the host’s model directory. Identity is hashed; location is not.

How each digest is built

All sha256. Two shapes: a digest over a set of files, and a digest over live artifacts. Sorting is load-bearing in the first — declaration order cannot change a digest, only content can.

digestconstructionmoves when
eval_set_sha the corpus manifest: sha256 over "<path>:<file sha>" lines, one per graded set file, in sorted order any graded set file is edited
grading_sha the same sha-of-shas shape over the gate’s declared grader and fixture-builder sources a grader or the fixture it grades against changes
protocol_sha canonical JSON of the protocol scalars, a newline, then sha-of-shas over the gate’s execution sources. The scalar set is closed both ways — a missing scalar is refused, and so is an extra one, which would otherwise be silently dropped from the digest a scalar (context window, retry budget, constrained flag, temperature, runtime version) or any execution source changes
prompt_sha three component digests over live runtime values — the selection surface, the tool schemas, the narration surface — NUL-separated so concatenation is injective, then combined what the model reads changes. A comment edit elsewhere in the same module does not move it: the axis covers the surface, not the file
model_digest read from the runtime’s tags endpoint — the manifest digest of the weights themselves the weights differ. Content-addressed, so the same tag in a different store is the same digest

One bound is recorded rather than papered over: the production capability layer is the system under test, not a provenance axis. Digesting it would make every line incomparable across any product commit and destroy the trend the ledger exists to draw, so that risk is carried by the re-run clause in LLM-ADR-002/004 — a human obligation, not a hash.

What stops a bad line

The ledger writer is the sole writer. Every refusal is fail-closed: it raises and appends nothing.

  • A partial run — fewer distinct graded items than the corpus manifest carries, or a manifest missing a file the gate must grade. A partial score can never pool into a trend or back a validated claim.
  • An unknown key at any depth — including the operator’s note, by name. No allowlist can certify arbitrary prose safe to publish, so the schema has no free-text field at all.
  • A missing key — a page must never have to guess whether a field was absent or unmeasured.
  • A value outside its vocabulary — hosts are generic hardware labels, never machine or user names.
  • An absolute user path in any string, anywhere.

The narration branch

The one place a human is load-bearing, because prose faithful to a typed result is not a computable predicate.

measured the run as recorded review the generations a verdict on every one — the reader never attests attest verifies findings + raw digests validated computed by the page, never stored joins on (attests,  findings_sha)
Clearing the flagged triage queue is not reading the run: an unflagged run is unread, not clean. The reader prints the attest command only once every generation carries a verdict — and will not run it for you.

Where the oracle comes from

An oracle is whatever decides correct; its one requirement is that it comes from somewhere other than the thing being tested. The three gates are three different answers to that question, which is why they are graded so differently.

gateoracleconsequence
selection the real dispatch path — hand-authored intent labels, dispatch-verified: the label verifier replays every expected capability + arguments through the live capability layer against the seeded fixture and checks the expected outcome and answer against what it returns graded mechanically; a label that drifts from what the product computes fails the verifier, so the corpus cannot silently disagree with the product — the hand-authored part is the intent, which is what the gate measures
multiturn the same, over a conversation set the same — the dangerous mode is a follow-up silently answered against the previous turn’s scope
narration none that is mechanical — the automated grader is a screen, and the verdict is the developer’s renders measured — never a pass — until an attestation line joins it

The selection oracle is part of the product, which is fine for the question the gate asks — it is independent of the model, the system under test. It is not independent of the product: a wrong dispatch would be graded against confidently. That is the same bound recorded above, wearing its other hat.

This page renders no measurement and fetches nothing: what it describes does not change from run to run, and a committed copy of ledger numbers would be a claim the ledger could not back (spec 14 D3, INV-6). For anything measured, open the Eval Ledger, the Footprint Ladder, or the Run Inspector. Terms — docs/llm-assistant/glossary.md in the repo defines every one of these words once. Named, not linked: the glossary is outside the staging view this page is served from, so a link here would point at nothing. Authorities: docs/llm-assistant/eval-telemetry.md (spec 14) and LLM-ADR-005; the shapes above are implemented in evals/harness/provenance.py and evals/harness/ledger.py.