finboard · assistant evals

Run Composer

Pick a gate and its knobs; this page composes the exact command and shows what each deviation from the production defaults does to the ledger — before you pay for it. It never runs anything: the run stays a deliberate act in your terminal, because every committed ledger line is supposed to be one.

Anatomy of a Gate Run — the mechanism Eval Ledger — fix a model, walk time Footprint Ladder — fix an era, walk models Run Inspector — drill into one run composes the command · never runs it

Compose

Defaults are the production configuration, and the command pins them explicitly — an ambient override in your shell cannot silently redirect a composed run. The model tag is validated, never shell-quoted: an invalid one is refused.

What to run
the desktop app’s bundled runtime picks its port per launch — Settings → Assistant shows the endpoint; only a 127.0.0.1 port is composable here
first K items only — the writer refuses the resulting partial run

The command

Run it from the repo root. Clicking the line selects all of it.


        
        
        

          
          
        

      

      
What this composition does to the ledger

      
Mechanism, not current values: the digests are computed from what is on disk when
      the run happens, so this page names which axis a knob moves and never claims to know today’s
      hashes.

      
    
    
    

    
    
      
    Before you run it
    
      
      
        
    • A live loopback runtime with the model present. Two ways: your own
        ollama serve on the default port (what this page composes unless told
        otherwise), or the desktop app’s bundled runtime — it picks its port per launch, so
        choose “another loopback port” above and copy the endpoint from Settings
        → Assistant. The harness never pulls; the model arrives through the app’s
        Model Manager or ollama pull on a user-managed daemon.
      • 
        
    • A model pulled through the app lives in the app-private store, so a plain user
        daemon will not see it: target the app’s runtime, or start your daemon against that
        store — Settings → Assistant shows the store path; export it as
        OLLAMA_MODELS before ollama serve.
      • 
        
    • Never CI. These are on-demand gates that need a live runtime and report numbers,
        not pass/fail.
      • 
        
    • The fixture seeds itself, isolated under the gitignored eval fixture store — a run
        never touches your real portfolio data.
      • 
        
    • A leftover FINBOARD_EVAL_SWEEP_ID in the shell stops a standalone gate
        run before it starts, by design — unset it if a previous sweep exported one.
      • 
        
    • A sweep evicts whatever model is resident before its first measurement; the
        app’s model reloads in a few seconds on its next assistant question. Nothing is lost.
      • 
      
    
    
    

    
    
      
    After it finishes
    
      
      
        
    • Two record tiers, one fork: the full detail lands in the gitignored local findings
        tier under a per-run stem, and one allowlisted line is appended to the committed ledger — or
        refused whole. Committing that line is your act, and a bad run stays on the record as
        measured, never re-rolled.
      • 
        
    • Read it with python3 evals/dashboard/serve.py — the
        Eval Ledger and Footprint
        Ladder read the committed line; the Run Inspector drills
        into the local detail under --local-details.
      • 
        
    • A narration run ends in a human pass: review_narration.py renders every
        generation, requires a verdict on each, and prints the attest command only then — it will not
        run it, and neither will this page.
      • 
      
    
    
    

  
    

  
    
    This page composes; it never executes and renders no measurement — the knob defaults shown
    are production configuration pinned to their sources by the test suite, not numbers from any
    run, and its one fetch is the loopback runtime’s model list, on your click. For anything measured, open the Eval Ledger,
    the Footprint Ladder, or
    the Run Inspector; for the mechanism these knobs move, see
    Anatomy of a Gate Run.
    Terms — docs/llm-assistant/glossary.md in the repo defines every one of these words
    once. Named, not linked: the glossary is outside the staging view this page is served from, so a
    link here would point at nothing.
    Authorities: evals/README.md (the documented invocations),
    docs/llm-assistant/eval-telemetry.md (spec 14) and LLM-ADR-005; the flags are the
    runners’ own argparse truth (run.py · run_multiturn.py ·
    run_narration.py · run_sweep.py).