--- library_name: pytorch tags: - transformer - language-model - long-context - agillm - experimental --- # AGILLM-4 AGILLM-4 is the next training target after AGILLM-3. The current code is a production-oriented starting point, copied from the proven single-file trainer and extended for: - >1B parameter floor preset (`agillm4_floor`) and ~1.7B main preset (`agillm4_main`) with AR+SAT+NAT heads - 100 tokens per parameter target ratio, above the AGILLM-3 training ratio - longer block-size work on 24GB, B200, and B300 class GPUs - AR+SAT+NAT training, with sequential backward to reduce peak VRAM - SDPA and experimental sublinear local+landmark attention backends - exact M-fold expansion attention harvested from n1.py, with local verifier - fused QKV projection harvested from n1.py, with legacy checkpoint loading - profiling tools for memory, throughput, AR cost, SAT cost, and optimizer cost - synthetic long-context curriculum generation for recall and multi-hop tests Start with [AGILLM-4.md](AGILLM-4.md) for the training plan and command recipes. The current sublinear backend is intentionally experimental: profile it against SDPA before using it for a real run. On RTX 4090-class 24GB cards, `run_agillm4_4090_longblock.sh` now defaults to `agillm4_floor` instead of the AGILLM-3-sized `large` preset, starts at block `1280`, and backs off in smaller 20% steps if VRAM is too tight. For the current v47 seed, launch tmux with `/workspace/agillm-4/launch_agillm4_4090_floor_from_v47.sh`; it writes `/workspace/agillm4_floor_train.log`. Checkpoint upload policy is intentionally bounded for the public HF storage quota: status and log tails upload every 30 minutes, the latest multi-GB delta uploads at most daily, and full checkpoints upload at most weekly with only two current remote files retained. Local full saves default to daily and local retention is one full plus one delta, so the 64GB Vast disk does not slowly fill. Current harvest status from n1.py is tracked in [N1_HARVEST.md](N1_HARVEST.md).
