Esp32 Markov Mixer 973k

ESP32 Markov Mixer, 973k parameters

We wanted to see whether a language model made mostly from low-rank Markov lookups, two tiny recurrent heads, and a little nonlinear context mixing could fit on an ESP32 and still say recognizably different things. It can.

This repository is the exact model artifact from that experiment. It is a demonstration, not a useful general-purpose assistant or a supported model release.

The bare-metal Rust inference engine and serial TUI are published at arm64be/esp32-serial-tui.

What is here

File	Description
release-binding-allparam-r8-30ep.pt	PyTorch checkpoint with 973,471 total and trainable parameters
release-binding-allparam-r8-30ep.q8mm	Deployed fixed-point model, tokenizer included, 1,075,652 bytes
release-binding-allparam-r8-30ep.json	Integer format, tensor layout, quantization, and architecture manifest
tokenizer.json	Lowercasing 4,096-token byte-level BPE tokenizer
release-binding-run-notes.md	Training, validation, export, and release-gate notes

The architecture uses a 192-token context, eight direct Markov lags at rank 21, four hashed context features, separate 32-hidden-unit intent and grammar GRUs, a 64-unit context MLP, and a 48-unit gate. It has no positional embeddings, prompt-copy path, shared recurrent trunk, feature mixer, or rank normalization.

The integer export uses square-companded signed int8 lookup tables, per-row signed int8 matrices, Q10 int16 activations, and integer accumulators. In a 1,024-context oracle comparison against the float checkpoint, it reached 98.24% top-1 agreement; fixed and float greedy decoding matched on all release prompts.

What it says

Greedy decoding on the ESP32 produced:

tell me a story about a fish named bob
-> Bob was a little fish who was very happy. He was so happy to see the little bird.

what is the capital of france?
-> Hello, the little boy's mom and dad are glad.

hi
-> Hello. I am a little boy.

The capital answer is plainly wrong. The release gate was narrower: three different prompts had to yield distinct English instead of collapsing into the same tree-and-ball story. The final checkpoint was corrected for those three prompt families using 120 hand-authored examples and strong replay against the pre-correction checkpoint.

Training snapshot

The correction ran for 30 full epochs over 114 training records, with AdamW, peak learning rates of 2e-4 for context computation and 1e-4 for lookup and output parameters, 1e-4 weight decay, a 5% warmup, and replay KL weight 8.0. Its six-record correction validation result was CE 2.7134 / PPL 15.08; a broader 60-batch mixed validation was CE 2.7845 / PPL 16.19.

Prompt format

Text is normalized to lowercase and wrapped as:

<|user|>
{prompt}
<|assistant|>

Greedy decoding is intentional. Sampling experiments reduced some repetition but damaged prompt binding on this model.

Checksums

491047766597fdb8375fa7bde4fe3d069858a6305c9fcac9464d022bcc20e518  release-binding-allparam-r8-30ep.pt
c9b2d7102c72b3ba61d14d47fff85906d28e69446114f71ff1eba6c922df0b30  release-binding-allparam-r8-30ep.q8mm
ea17b665872be0d59d485b54d056f39b84b2cd7bfc18a86c408a8c219ed9bad1  release-binding-allparam-r8-30ep.json
4d06b372a8e7ba5f5b015c3901cbc8a2cdac8d248e3e7275e08b27899e76946b  tokenizer.json

Limitations

This model is tiny, narrow, repetitive, factually unreliable, and strongly shaped by TinyStories-like data. It should not be used for decisions, factual questions, or safety-critical tasks. The point is the constrained architecture and embedded execution, not model quality.