A person reads at about 250 words per minute. That is the bar I care about for a local assistant: if the model writes faster than you read, the hardware disappears from the experience. Nobody waits for anything.
So let’s see what happens when we put a ternary 2B model on an ordinary Raspberry Pi 5 and measure.
Goal
Find out whether a 2B-parameter language model can generate text on a Raspberry Pi 5, CPU-only, at or above human reading speed — and do it with numbers that anyone can reproduce, not with a feeling.
Reading speed, converted: 250 words per minute is about 4 words per second. At roughly 0.75 words per token, the threshold is somewhere around 6 tokens per second. That is the line to beat.
Hypothesis
Decode on a small CPU is limited by memory bandwidth, not arithmetic. A ternary model (weights in {-1, 0, +1}, about 1.58 bits each) moves an order of magnitude less data per token than an FP16 model of the same parameter count. If the bottleneck really is bandwidth, the Pi should comfortably pass 6 tokens per second — even though nobody designed this computer for language models.
Setup
- Hardware: Raspberry Pi 5 (8 GB), active cooler, stock clocks.
- Model: BitNet b1.58 2B-4T, i2_s quantization, 1.1 GB on disk, needs about 4 GB of RAM to run comfortably.
- Runtime: the geisten engine — one static binary, under 1 MB, no dependencies, hand-written ARM NEON kernels.
- Reference: bitnet.cpp on the same machine, same GGUF weights, same prompt.
We measure decode throughput: tokens per second after the prompt has been processed. Prefill is a separate story (see below).
geisten bench --model bitnet
Results
17.4 tokens per second.
Converted back: about 13 words per second, or roughly 780 words per minute. That is three times faster than a fast reader. Watching it is the interesting part — the text simply stays ahead of you. There is no moment where you are aware of waiting for the machine.
For a €80 computer running entirely offline, with the model weights sitting on the SD card next to your files, I find this result worth pausing on. Two years ago the standard answer was “you need a GPU for that.”
Benchmarks
CPU-only, identical GGUF weights, decode measured after prefill:
| System | Runtime | Decode |
|---|---|---|
| Raspberry Pi 5 | geisten | 17.4 tok/s |
| Raspberry Pi 5 | bitnet.cpp | 8.2 tok/s |
| AMD 9950X | geisten | 103.1 tok/s |
| AMD 9950X | bitnet.cpp | 54.3 tok/s |
The 2.1× over bitnet.cpp on the Pi comes almost entirely from the NEON kernels being shaped around the ternary weight layout instead of adapting a general-purpose path. The same pattern holds on the desktop CPU (1.9×), which suggests it is the memory layout doing the work, not a Pi-specific trick.
Hardware, model, quantization and measurement point are stated because a throughput number without them is an impression, not a result.
What went wrong
Three things, in honest order:
- My first speed claim was wrong. I initially compared tokens per second against words per minute without converting, which flattered the result by a third. Tokens are not words. The corrected math is in the Goal section — the conclusion survives, but barely a week goes by without this conversion biting someone.
- Prefill is not covered by this result. The reading-speed experience holds once generation starts. Feed the model a long document first and you wait: processing an 8 KB article on the Pi takes minutes, not seconds. “Faster than you can read” is a statement about decode, and it would be dishonest to let it imply more.
- “Efficient” is still an assertion, not a measurement. We claim energy efficiency but have not wired power measurement into the benchmark harness yet. Until there is a joules-per-token column, that part of the story is incomplete.
Next steps
- Add continuous power and CPU-utilization measurement to the harness, so the next log can report energy per token instead of adjectives.
- Look at prefill on the Pi. Decode is solved for this model class; time-to-first-token is now the honest bottleneck.
- Longer runs under sustained load: does the Pi throttle, and what does that do to the numbers after twenty minutes?
Everything needed to reproduce this — engine, weights format, benchmark method — is in the geisten repository. If you measure something different on your hardware, that is not a problem, that is a data point. Send it.