SIL v01

tinyshakespeare -- char tokenizer, n_embd=128

This is an educational project, not a practical language model. It is a from-scratch transformer built to learn the mechanics of language modeling end-to-end. It is not intended for any downstream or production use -- treat it as a worked example, not a tool.

Model architecture

Token + position embeddings -> n_layer stacked transformer blocks (pre-norm multi-head causal self-attention + feedforward, residual connections) -> final LayerNorm -> linear head to vocab logits. Full hyperparameters are in config.json; nothing about the architecture is hardcoded in the loader -- see load_model.py.

Parameters	812,609
n_embd	128
n_head	4
n_layer	4
block_size	32
Tokenizer	char-level, vocab_size=65

Training corpus

Trained on TinyShakespeare (~1.1M characters), fetched directly from https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt.

Results

Four runs, all on TinyShakespeare, same architecture family (n_layer=4, n_head=4, block_size=32, lr=1e-3, max_iters=8000, batch_size=32, seed=1337) — a controlled sweep over embedding width, plus one run swapping in a trained BPE subword tokenizer instead of character-level.

Raw cross-entropy (nats/token) is not comparable across tokenizers — a 1000-token BPE vocabulary and a 65-token char vocabulary have different random-guessing floors (ln(1000) ≈ 6.9 vs ln(65) ≈ 4.2), so a higher raw loss on BPE does not mean a worse model. Bits-per-character (BPC) normalizes both onto the same unit — bits of model surprise per character of the original text — and is the number to compare across the table.

Tokenizer	n_embd	Params	Train loss (nats)	Train BPC	Val loss (nats)	Val BPC	Train/val gap (BPC)	Wall clock
char	32	55,745	1.7159	2.4755	1.8849	2.7193	0.2438	4m48.8s
char	64	209,729	1.5247	2.1997	1.7220	2.4843	0.2846	7m19.4s
char	128	812,609	1.4043	2.0260	1.6257	2.3454	0.3194	16m37.6s
bpe (vocab=1000)	32	116,520	3.5683	2.1272	3.8843	2.4189	0.2917	6m29.5s

Published checkpoint: char, n_embd=128 (bolded row) — lowest val BPC of the char-level sweep.

Scaling observation: doubling n_embd costs a roughly constant ~3.8× parameters each step (32→64, 64→128), but the val-BPC improvement shrinks each time (0.235 → 0.139 in BPC terms), and the train/val gap widens — diminishing, saturating returns from embedding width alone once depth (n_layer=4) and context (block_size=32) are held fixed.

Tokenizer observation: char-level (BPC 2.35) still edges out this one BPE run (BPC 2.42) at comparable parameter count, but BPE achieves that with ~7x fewer parameters than the 812K char model and visibly more coherent generated chunks — a fair head-to-head at matched parameter count wasn't run here.

Files

File	Purpose
model.safetensors	state_dict() of the trained BigramLanguageModel, in safetensors format
config.json	Every hyperparameter needed to reconstruct the architecture and tokenizer
tokenizer_char.json	The char-level tokenizer's vocabulary
bigram.py	Model architecture source (vendored so this folder is self-sufficient)
tokenizer.py	Tokenizer source (vendored, same reason)
load_model.py	Reconstructs the model + tokenizer from config.json and generates a sample

Usage

pip install torch safetensors
python load_model.py

load_model.py reads config.json for every architectural parameter -- it does not assume or hardcode them -- loads model.safetensors into a freshly constructed model, and samples 400 characters to prove the checkpoint and its config agree.

Limitations

This is a ~65-char-vocabulary, 32-token-context toy model trained for 8,000 steps. It reproduces surface Shakespeare-ish texture but not coherent meaning, plot, or factual content. Do not use this for anything beyond studying how these mechanics fit together.