# JevBench public split through `geistr decide` (14 October 2026)

Raw measurements for the article `research/jev_style_decisions/`. Do not edit
the files in `runs/`; derived numbers must point back to them.

## What was measured

JevBench's 231 public items (`easy` 48, `original` 72, `hard` 111) answered by
`geistr decide`: one prefill, the probabilities of the option-key tokens, no
generation, **no option rotation and no calibration** (raw readout). Every
metric is computed by JevBench's own runner and `jevbench.cli summarize`.

| Run | Model file (SHA-256) | Processor | Items |
|---|---|---|---|
| `bonsai2-27b-metal` | Ternary-Bonsai-2-27B-PQ2_0.gguf (`3907dc16…`) | Metal | 231/231 |
| `gemma4-e2b-metal` | gemma-4-E2B-it-Q4_K_M.gguf (`740185b2…`) | Metal | 231/231 |
| `gemma4-e2b-cpu` | gemma-4-E2B-it-Q4_K_M.gguf (`740185b2…`) | CPU (NEON) | 228/231, see limits |
| `gemma4-e2b-metal-chat` | gemma-4-E2B-it-Q4_K_M.gguf (`740185b2…`) | Metal, chat baseline | 231/231 |
| `bonsai2-27b-metal-chat` | Ternary-Bonsai-2-27B-PQ2_0.gguf (`3907dc16…`) | Metal, chat baseline | 231/231 |
| `gemma4-e2b-metal-ids` | gemma-4-E2B-it-Q4_K_M.gguf (`740185b2…`) | Metal, repeat of `gemma4-e2b-metal` recording the prompt token IDs | 231/231 |
| `gemma4-e2b-metal-llamacpp` | gemma-4-E2B-it-Q4_K_M.gguf (`740185b2…`) | Metal, llama.cpp `llama-server` on the same token IDs | 231/231 |
| `bonsai2-27b-metal-ids` | Ternary-Bonsai-2-27B-PQ2_0.gguf (`3907dc16…`) | Metal, repeat of `bonsai2-27b-metal` recording the prompt token IDs (15 Oct) | 231/231 |
| `bonsai2-27b-metal-prism` | Ternary-Bonsai-2-27B-PQ2_0.gguf (`3907dc16…`) | Metal, Prism's llama.cpp fork on the same token IDs (15 Oct) | 231/231 |

Results: [SUMMARY.md](SUMMARY.md); per run `summary.json` (JevBench's output),
`results.jsonl` (one record per item, with the full probability distribution),
`manifest.json` and `raw.tar.gz` (request and response per item, as written by
JevBench's runner).

## Setup

- Hardware: Apple M1 Max, 64 GB, macOS 27.0.1, mains power. Every run held the
  machine's benchmark lock (`with-mac-lock`), one run per lock.
- geist-runtime `b9de7a23d057023bf50663b32c4fa2d9b522f1d8` (2026-10-13), built
  with `make wheel DECISION=1`; Python package geistr 0.3.4; geistlib engine
  `3841fa68e277b8ce33211aa84e4728dd61c133de` (the Makefile pin).
- Decision profiles `gemma4-e2b-q4-v1` and `bonsai2-27b-pq2-v1`, dense mode,
  enabled in [`harness/decisions.json`](harness/decisions.json).
- Model context and decision prompt limit: 8192 tokens.
- JevBench `7625ecb1b075cfe4d09d34141d353325825eea1d` (2026-10-10), public
  split, dataset hash `dc3995d8ae1e2fc8e81ce38431add509eb8bb39b85aadfd0c7c32079382dde51`.
- Harness: [`harness/geistr_adapter.py`](harness/geistr_adapter.py) (JevBench
  adapter), [`harness/run.py`](harness/run.py), [`harness/all.sh`](harness/all.sh).

### How a JevBench item becomes a geistr decision

- question = the item's `instructions`; context = the item's `state` (JSON
  objects serialized with `json.dumps`).
- choice: options = the labels, description = `criteria[label]`.
- noul: options `no` / `yes`, descriptions `criteria.false` / `criteria.true`.
- score: options `0` … `n`, description = `criteria[i]`.
- The returned distribution is geistr's selection probabilities over these
  options (the model's own softmax restricted to the option keys).

### Chat baseline (`*-chat` runs)

The same models, files, machine and items, used as ordinary chat models
([`harness/geistr_chat_adapter.py`](harness/geistr_chat_adapter.py),
[`harness/run_chat.py`](harness/run_chat.py),
[`harness/all_chat.sh`](harness/all_chat.sh)): one fresh conversation per
item, a system message asking for exactly one label, the context, the question
and the allowed labels with their criteria; greedy decoding, at most 32 answer
tokens, thinking off for thinking models (the chat template's
`enable_thinking=false` rendering, `GEISTR_REASONING_THINK_OFF`, #212). The
answer is parsed into a label: strict (trimmed answer equals a label), else
lenient (exactly one label occurs as a whole word), else abstention. Gemma:
230 strict, 1 unparsed; Bonsai: 231 strict. JevBench scores these runs as
label-only (accuracy, no Brier/ECE). Each run was split into three parts
(items 0–119, 120–175, 176–230), each holding the lock on its own; one
`manifest-part-*.json` per part, `results.jsonl` is their concatenation.

### llama.cpp on the same tokens (`gemma4-e2b-metal-ids`, `gemma4-e2b-metal-llamacpp`)

[`harness/llama_cmp.sh`](harness/llama_cmp.sh) first repeats the geistr Metal
run with `GEISTR_IDS_OUT` set, so that the adapter also writes each item's exact
prompt and option-key token IDs (`runs/gemma4-e2b-metal-ids/ids.jsonl`; the
adapter in `harness/` contains this optional output, the earlier runs used it
without). Then it starts llama.cpp's `llama-server` 0.6.0 (build 11429, commit
`d81235049`, Homebrew; `-c 8192 -ngl 99 -np 1`, other settings default) and
sends the same token IDs ([`harness/llama_adapter.py`](harness/llama_adapter.py)):
`n_predict` 1, `temperature` -1 (plain softmax of the logits), `n_probs` 100,
`cache_prompt` false. The option probabilities are the softmax probabilities
of the key tokens, renormalized over the options; a key outside the top 100
counts as 0 (4 keys in all 231 items). Latency is JevBench's wall time per
item, which for llama.cpp includes a local HTTP request; the server's own
`timings` are in the raw responses. Each engine held the lock on its own. The
llama.cpp manifest was written after the run (the runner script failed on its
manifest step; the results are complete), see its `note`.

Differences that are not the engine: geistr's decisions use an F32 KV cache;
llama-server used its defaults (see `llama-server.log`).

### Bonsai against Prism's llama.cpp fork (`bonsai2-27b-metal-ids`, `bonsai2-27b-metal-prism`)

Upstream llama.cpp does not run the PQ2_0 format; Prism's fork does. The fork
(`01ae597e3f7d4742909e1e831abb12fe3d24b2cf`, 2026-09-21) was cloned from the
local checkout and built with CMake (`CMAKE_BUILD_TYPE=Release`, Metal backend
included; its Metal sources contain PQ2_0 kernels). The method is the one of
the Gemma comparison above: geistr recorded the token IDs of every decision
prompt (`runs/bonsai2-27b-metal-ids/ids.jsonl`), and the fork's `llama-server`
(`-c 8192 -ngl 99 -np 1`) read the same IDs with the same request settings.
Scripts: [`harness/bonsai_check/`](harness/bonsai_check/), comparison:
`runs/bonsai2-27b-metal-prism/compare.json` (from `compare.py`).

How the runs went (15 October 2026), each part under its own lock:

- geistr: items 0–159 in one part; a first attempt at items 160–230 stopped
  after 22 items when the job driving it stalled. That partial part was
  discarded, `ids.jsonl` cut back to the 160 complete items, and items
  160–230 run again in full.
- llama-server: items 0–150 in a first part, which was stopped by a two-hour
  job limit before it could write its manifest (so there is no
  `manifest-part-0.json`; its server log is kept); items 151–230 in a second
  part. `results.jsonl` is the concatenation of both parts.

### Unverified decisions from the chat template (`runs/unverified_check/`)

A follow-up change in geist-runtime (branch `feat/decide-any-model`, on top of
`feat/decide-default-on`) renders decisions for models without a verified
profile from the model's own chat template family. To check it, the same
models were run with an enabled policy *without* a profile (unverified path),
and each item's prompt token IDs, option-key token IDs and probabilities were
compared with the verified runs above ([`harness/unverified_check.py`](harness/unverified_check.py),
[`harness/uv_check.sh`](harness/uv_check.sh); 16 October 2026, Metal for Gemma and Bonsai, CPU for Qwen3).

| Model | Items | Same prompt IDs | Same key IDs | Same choice | Largest probability difference |
|---|---:|---:|---:|---:|---:|
| Gemma 4 E2B | 231 | 231 | 231 | 231 | 0.0 |
| Bonsai 2 27B | 50 (items 0–49) | 50 | 50 | 50 | 0.0 |
| Qwen3 0.6B (no verified profile) | 30 (items 0–29) | – | – | – | – (28 of 30 correct) |

### Option rotation (`gemma4-e2b-metal-rot`, `bonsai2-27b-metal-rot`, 16 October 2026)

Each item scored once per cyclic rotation of its options (n options, n
prefills), combined as the normalized mean of the rotations' log-probabilities
([`harness/geistr_rot_adapter.py`](harness/geistr_rot_adapter.py),
[`harness/run_rot.py`](harness/run_rot.py), [`harness/rot.sh`](harness/rot.sh)).
geistr from the `feat/decide-any-model` build, verified profiles, Metal. Gemma:
all 231 items in three parts; Bonsai: the 72 `original` items (48–119) only.
The first rotation reproduces the plain runs' choice in every item. Every
rotation's logits are in `raw.tar.gz`.

### Trust analysis (`runs/trust_base.json`, `runs/trust_rot.json`)

[`harness/trust.py`](harness/trust.py) computes, from the stored per-item
logits: the share of decisions that can be automated at ≤ 5 % error (most
confident first), share and accuracy at p ≥ 0.9 and p ≥ 0.99, and a
temperature fitted by 2-fold cross-validation (folds by a hash of the item ID)
with ECE and the automatable share on the held-out fold.

### Hidden-state probe (`runs/gemma4-e2b-hidden/`, 17 October 2026)

Last-token hidden state of Gemma 4 E2B's final layer (1536 values) for each
decision prompt, from Homebrew llama.cpp b11429 (`llama-server --embeddings
--pooling last --embd-normalize -1 -b 8192 -ub 8192`) fed geistr's recorded
prompt token IDs ([`harness/hidden_dump.py`](harness/hidden_dump.py),
[`harness/hidden.sh`](harness/hidden.sh)). Probe: PCA fitted on the training
fold, L2 logistic regression, 10 × 5-fold CV, eight settings reported
([`harness/probe.py`](harness/probe.py)); results in `probe_results.txt`.

### Readout variants without generation (`runs/gemma4-e2b-variants/`, 17 October 2026)

Gemma 4 E2B through Homebrew llama.cpp b11429 (same GGUF), next-token
distributions with `n_probs` 400 ([`harness/variants.py`](harness/variants.py),
[`harness/variants.sh`](harness/variants.sh)): key letters (control; same
choice as geistr in 230/231 items), key digits, label names (first token, and
the full label by teacher forcing: sum and per-token mean), and a reworded
prompt with letters. A key outside the top 400 gets the list's smallest
log-probability minus 1 (counted per item in `missing`). Evaluation:
[`harness/variants_eval.py`](harness/variants_eval.py) (`eval.txt`) and a
paired bootstrap against the letters ([`harness/variants_ci.py`](harness/variants_ci.py),
`bootstrap.txt`).

In 44 items (`label1_ambiguous`), two option IDs begin with the same token
(e.g. `coding` / `coding_agent`); the first-token label readout then splits the
probability between tied labels. The AUROC advantage of that readout on all
231 items (0.82 vs 0.74) comes from these ties and disappears on the 187
unambiguous items (0.78 vs 0.78). The article uses the 187 items:
[`harness/variants_unambiguous.py`](harness/variants_unambiguous.py), `unambiguous.txt`.

### Escalation with thinking (`runs/bonsai2-27b-metal-think/`, 17 October 2026)

The 85 items where Bonsai's `decide` readout had p < 0.9 (from
`runs/bonsai2-27b-metal/results.jsonl`) answered by Bonsai in chat mode with
thinking on (catalog `think_tags`), greedy, answer up to 4096 tokens, context
8192, same prompt and label parsing as the chat baseline
([`harness/run_think.py`](harness/run_think.py), [`harness/think.sh`](harness/think.sh)).
geistr from the `feat/decide-any-model` build, Metal. Resumable in parts of
about 30 minutes, each under its own lock; my driving job was stopped once by
a two-hour limit after 73 items and resumed. `results.jsonl.gz` holds per item
the label, the full thinking text and the time.

### Small model first (`runs/cascade.json`, `runs/cascade.txt`, 17 October 2026)

Gemma 4 E2B decides first; where a trigger fires, Bonsai 2 27B's decision
counts instead, optionally followed by Bonsai with thinking below p = 0.9.
Simulated from the stored runs, no new model calls
([`harness/cascade.py`](harness/cascade.py)). Triggers: Gemma's probability,
disagreement of its option rotations, of its chat answer, prompt length
(> 300 input tokens) and their combination. The signals and the 300-token
limit were chosen on the same 231 items, so the combined result is optimistic.
Time is the sum of the stored per-item latencies (M1 Max, Metal) of the Gemma
runs a trigger needs, plus Bonsai's readout for the escalated items.

### Notes first, then decide (`runs/gemma4-e2b-metal-facts/`, 17 October 2026)

Gemma 4 E2B first writes notes in chat mode (thinking off, greedy, at most 256
tokens): the facts and rules from the context that decide the question and
what follows from them, without naming a label. geistr decide then reads the
decision with the notes appended to the context. Same build, profile and
machine as `runs/gemma4-e2b-metal`; latency covers both steps
([`harness/geistr_facts_adapter.py`](harness/geistr_facts_adapter.py),
[`harness/run_facts.py`](harness/run_facts.py), [`harness/facts.sh`](harness/facts.sh)).
The notes are in `raw.tar.gz`. Evaluation and a cascade to Bonsai:
[`harness/facts_eval.py`](harness/facts_eval.py), `eval.txt`.

### Thinking on the confident items too (`runs/bonsai2-27b-metal-think-sure/`, 17 October 2026)

The other 146 items (Bonsai's readout p >= 0.9), answered the same way as
`runs/bonsai2-27b-metal-think/` (`SELECT=sure` in
[`harness/run_think.py`](harness/run_think.py),
[`harness/think_sure.sh`](harness/think_sure.sh)). Together the two runs give
thinking on all 231 items, to compare with thinking only below p = 0.9. The
second of two locked parts waited for the lock while another process held it;
waiting time is not in the per-item times.

### Where Bonsai's time goes (`runs/bonsai2-27b-metal-speed/`, 17 October 2026)

Three measurements on Bonsai 2 27B, Metal, each under the benchmark lock:

- `fixed.json` ([`harness/fixed_cost.py`](harness/fixed_cost.py)): the same
  short decision with filler contexts of 80 to 2406 prompt tokens, four times
  each, one model handle (geistr from the `feat/decide-any-model` build).
- `tiles.json` ([`harness/tile_sweep.py`](harness/tile_sweep.py)): prompt
  length raised one word at a time from 80 to 223 tokens, to test whether
  lengths that are a multiple of 32 are faster (geistlib's Metal PQ2 GEMM
  uses its interior kernel only for whole 32-row tiles).
- `shared.jsonl` ([`harness/shared_ctx.py`](harness/shared_ctx.py)): eight
  JevBench hard items of 808 to 2639 tokens, each item's question in its
  cyclic option rotations (2 to 4), asked through `geistr decide-batch
  --shared-context` once as a batch of one question and once as a batch of
  all rotations; the CLI's telemetry gives the context read (`context_ms`) and
  the questions (`scoring_ms`). geist-runtime `c9b00b2` built with
  `DECISION=1 DECISION_CONTEXT=1`, snapshot budget 8192 MiB, context 8192.

Scripts: [`harness/speed.sh`](harness/speed.sh), [`harness/speed2.sh`](harness/speed2.sh).

### After the engine update (`runs/engine-e78f9ee/`, 17–18 October 2026)

geist-runtime branch `perf/230-engine-ragged` (PR #234): geistlib pin `e78f9ee`
(geistlib#35, fast Metal GEMM and flash attention for any row count) instead of
`3841fa6`, built with `DECISION=1 DECISION_CONTEXT=1`. Real-model decision tests
passed first ([`harness/t230.sh`](harness/t230.sh)). Then, under the lock
([`harness/m230.sh`](harness/m230.sh)): `tiles.json`, `fixed.json`,
`shared.jsonl` (same scripts and items as `runs/bonsai2-27b-metal-speed/`),
and the 231 JevBench decisions for both models (`gemma4-e2b-metal-230/`,
`bonsai2-27b-metal-230/`, same adapter and config as the original runs). My
driving job was stopped once by its two-hour limit while it waited for the
lock; the finished steps were kept and the rest resumed.

### Merged main, stratified subset (`runs/main-8a468e5-subset/`, 18 October 2026)

geist-runtime `main` at `8a468e5` (PRs #228, #229 and #234 merged), wheel built
and tested with real models first ([`harness/tmain.sh`](harness/tmain.sh)).
Every 6th JevBench public item (39: easy 8, original 12, hard 19) through
`decide` and the label chat for both models, Metal
([`harness/sub.sh`](harness/sub.sh); `EVERY=6` in `harness/run.py` and
`harness/run_chat.py`).

### Where Bonsai's long-prompt time goes (`runs/engine-e78f9ee/attn_split_experiment.jsonl`)

A local, unpublished experiment build of geistlib `e78f9ee` with one extra
switch, `GEIST_SKIP_KVSTAGE`, which skips the f16 staging of an F32 KV cache.
With `GEIST_METAL_PROFILE=1`, the subtractive switches `GEIST_SKIP_<CAT>` drop
whole dispatch categories; outputs are garbage, only times count, and GPU
chains overlap, so deltas are not additive. The same decision at 664, 1245
and 2406 prompt tokens, 3 repetitions per configuration
([`harness/exp.sh`](harness/exp.sh), [`harness/attn_split.py`](harness/attn_split.py)).

### Merged main, all 231 items (`runs/main-8a468e5-full/`, 18 October 2026)

The same `decide` and label-chat runs as `runs/*-metal/` and `runs/*-metal-chat/`,
on geist-runtime `main` `8a468e5` (geistlib `e78f9ee`), Metal, in three locked
parts per run ([`harness/full.sh`](harness/full.sh); `--start/--stop` in
`harness/run.py`). Bonsai: same choice as the original run in 231/231 for both
modes. Gemma: chat 231/231; readout 230/231 (143 instead of 144 correct; the
changed item matches llama.cpp).

The same directory holds the Gemma 4 E2B rotation run (`gemma4-e2b-rot/`) and
the notes-first run (`gemma4-e2b-facts/`) on the current version
([`harness/cur.sh`](harness/cur.sh)), and `cascade.json` from
[`harness/cascade.py`](harness/cascade.py) over these runs. The article's
numbers come from these runs, except Bonsai's thinking answers (earlier
geistlib `3841fa6`), the llama.cpp-based key-format and probe experiments, and
the early-readout run (Prism's fork). The charts were regenerated from them
with [`harness/charts.py`](harness/charts.py).

### Reading the decision during thinking (`runs/bonsai2-27b-early-readout/`, 19–20 October 2026)

For Bonsai's uncertain items (decide p < 0.9) with a stored greedy thinking
answer (`runs/bonsai2-27b-metal-think/`), the thinking text is cut at 0, 10,
25, 50, 75 and 100 % of its characters (at a word boundary), the thinking block
is closed, and every allowed label is scored as a whole: the sum of the token
log-probabilities of the label followed by `<|im_end|>`, teacher-forced through
Prism's llama.cpp fork `01ae597e3` (`llama-server`, Metal, top-1000
next-token probabilities; a token outside them gets the floor minus 1, counted
in `missing`). [`harness/early_readout.py`](harness/early_readout.py),
[`harness/early.sh`](harness/early.sh), [`harness/early_cycle.sh`](harness/early_cycle.sh).
Run in short parts under the machine lock, each started only after the lock had
been free for a while ([`harness/early_queue.sh`](harness/early_queue.sh)).
Complete: all 85 uncertain items (`results.jsonl`; cuts 0/10/25/50/75/100 %,
and for the last 30 also 60/80/85/90/95 %), finer cuts 60/80/85/90/95 % for the
37 of the first 55 items whose readout was below 0.99 at 75 % (`fine.jsonl`,
`fine_ids.json`), and a stratified sample of 20 confident items from
`runs/bonsai2-27b-metal-think-sure/` (`sure.jsonl`, `sure_ids.json`).
Evaluation: [`harness/early_eval.py`](harness/early_eval.py), `eval.txt`. Items 52–54 were finished by a client
that had outlived its locked part (its server was orphaned by the part's time
limit); only probabilities are used from this run, no times.

## Limits

- **Public split only.** The sealed split can only be run by Benchmark Heaven.
  The numbers are therefore not a JevBench board score (that blends sealed
  items, chance correction, calibration, speed and cost).
- **The public items are public.** We did not train on them. Whether the
  models' training data contained them is unknown.
- **`gemma4-e2b-cpu` is incomplete:** the run was stopped by its 40-minute time
  limit after 228 items (14:23 to 15:02 local time); three `hard` items were
  not attempted. It has no `manifest.json`. The complete Gemma result is the
  Metal run; the CPU run shows the same model on the CPU.
- Latency is JevBench's wall time per item, including rendering and
  tokenization. Long `hard` items dominate the 95th percentile.
- Comparisons with published numbers of other systems (e.g. 86.6 % for
  Jev 1.13.0 on the same 231 items) compare different runs, hardware and dates.

## Failed attempts before these runs (not used)

1. A shell loop bug passed no `--processor`; no model was loaded.
2. JevBench's default cost reservation stopped every run after 50 items; set to
   0 for local runs.
3. The decision prompt limit defaulted to 512 tokens; three long `hard` items
   were refused and stopped the runs after 123 items. Raised to 8192; a refused
   item would now be recorded as status 422 (counted wrong, run continues). No
   item was refused in the final runs.