Albert

--- language: - en - de - fr - es - pt - it - nl - pl - multilingual license: lgpl-3.0 tags: - mixture-of-experts - ternary-weights - moe - edge-ai - research - candle - rust - federated-learning - low-precision - sprind - dual-stream - cord-surgery - net2net pipeline_tag: text-generation --- # Model Card — albert. (Albert-MoE-13) **Version:** v3.0 (ternary MoE) **Maintainer:** RFI-IRFOS, contact@ternlang.com **Repository:** https://github.com/rfi-irfos/ternary-intelligence-stack **License:** LGPL-3.0-or-later (model weights, training code, inference runtime). Platform infrastructure (API server, MCP tooling, HDL) is BSL-1.1. See [README §Licensing](README.md#licensing) for the full tier breakdown. **Last updated:** 2026-06-21 **Training status:** Active (ep~7592) on Modal — **33L dual-stream** · 20 depth surgeries + 1 cord surgery complete. S19 (31L→32L) fired ~ep6500, 2026-06-15. S20 (32L→33L) fired ~ep7000, 2026-06-21. Context window doubled 256→**512** — enabled by CTX-aware evolution gate (`CTX_LEVEL_TABLE` 64→1M, `recalibrate_ctx()` auto-scales mastery/plateau thresholds per CTX level). Best EP-AVG ATL **4.6842** (ep7588, 33L) and descending. Chip ATL **0.6116**. fib_index=8 · window=55 · Gen3 step1/6. Mastery surgery gate: 4.0 nats (recalibrated from 4.5 at 256CTX). --- ## Model Overview albert. is a research-grade language model trained from scratch using a **ternary weight representation** (-1, 0, +1) with a Mixture-of-Experts (MoE) architecture. It is developed by RFI-IRFOS as a demonstration that high-quality language modelling is achievable without 32-bit floating-point weights, targeting inference on edge hardware and low-power devices. | Property | Value | |----------|-------| | Architecture | **Dual-stream** Ternary MoE (Mixture of Experts) | | Streams | **2** (bifurcated via cord surgery ep4202, 2026-05-27) | | Layers | **33** per stream | | Hidden size | **2×256H** (256H per stream) | | Anastomosis gates | **6** — bidirectional F32 cross-stream fusion at Fibonacci layers [2,3,5,8,13,21] | | Experts | 12 per stream (Top-3 routing; shared FFN weights, independent routing gates) | | Context length | **512 tokens** (bumped 256→512 at ep7592, 2026-06-21) | | Vocabulary | 32,000 tokens (ByteLevel BPE — EN/DE/FR/ES/PT/IT/NL/PL) | | Weight representation | Ternary {-1, 0, +1} with STE training | | Gate linear | F32 | | Positional encoding | RoPE (rotate_half) | | Optimizer | AdamW, cosine LR decay, BATCH=1 (post-cord) | | Parameters (total) | **~255M** (approx, 33L — updated 2026-06-21) | | Safetensors | **~2,534 tensors · ~928 MB** (approx, 33L) | | Surgeries | **20 depth (S1–S20)** + **1 cord surgery** = 21 total surgical events | The central technical innovation is the **@sparseskip** primitive — a learned sparse-skip layer that dynamically bypasses computation paths based on token-level activation patterns, enabling sub-linear inference scaling without pruning. --- ## Intended Use **Intended uses:** - Research into ternary and low-precision neural network architectures - Benchmarking inference performance on CPU and edge GPU hardware - Academic study of Mixture-of-Experts routing dynamics - Demonstration platform for the SPRIND AI funding initiative (Germany) **Out-of-scope uses:** - Production deployment as a general-purpose assistant without further fine-tuning and safety evaluation - Safety-critical applications (medical, legal, financial decisions) - Any use requiring factual accuracy guarantees - Deployment to users without appropriate transparency disclosure --- ## Training Data See [DATA_PROVENANCE.md](DATA_PROVENANCE.md) for full source documentation and governance details. **Summary:** albert. is trained on a curated multilingual corpus composed of: | Tier | Content | Approximate Share | |------|---------|------------------| | Core | Project Gutenberg (public domain books, multilingual) | ~30% | | Core | Wikipedia (15 languages: EN, DE, FR, HU, ZH, AR, KO, SV, FI, NL, PL, RU, JA + more) | ~25% | | Core | OpenWebText (filtered Common Crawl) | ~15% | | Technical | GitHub issues, developer blogs, HN discussions | ~10% | | Chaos | Synthetic noise, adversarial patterns, mixed-language text | ~10% | | Structured | Code samples, structured data (JSON/YAML/TSV) | ~5% | | Multilingual | Additional EU language samples | ~5% | The **10% chaos layer** is a structural invariant enforced by the training pipeline (`train_tokenizer_v3.py`). It exists to prevent the model from over-fitting to clean text distributions and to improve robustness to noisy inputs. --- ## Evaluation **Primary metric:** Cross-entropy loss on a held-out WikiText-2 sample (`eval_sample.txt`, not seen during training). **Benchmark results (benchmark suite v2.0.0):** | Epoch | Loss (avg) | Epoch ATL | Batch ATL | Tok/s (T4 GPU) | |-------|-----------|-----------|-----------|----------------| | Ep54 | ~10.35 | 10.35 | — | 11.24 (CPU) | | Ep111 | ~10.36 | 10.36 | — | 18.52 | | Ep849 | ~10.22 | 10.2050 | — | pending | | Ep1177 | 10.2076 | 10.2059 (ep1158) | 10.1738 (ep1155) | pending | | Ep1390 | 10.1212 | 10.1212 (ep1390) | 10.0670 (ep1385) | pending | | Ep1435 | 10.1113 | 10.1113 (ep1435) | **10.0556** (ep1435) | pending | | Ep1438 | 10.1071 | 10.1071 (ep1438) | **10.0556** (ep1435) | pending | | Ep1441 | 10.1067 | 10.1067 (ep1441) | 10.0556 (ep1435) | pending | | Ep1455 | 10.1060 | **10.1060** (ep1455) | **10.0396** (ep1445) | pending | | Ep1474 | 10.0982 | **10.0982** (ep1474) | **10.0396** (ep1445) | pending | | Ep1553 | ~10.07 (est) | 10.0982 (ep1474) | **9.9948** (ep1553) ← first sub-10.0 batch | pending | | Ep2040 | ~9.82 (est) | 9.7976 (ep2084) | **9.6380** (ep1445) | 9.6–18.5 (CPU) | | Ep2084 | **9.7976** | **9.7976** ← epoch ATL | 9.6380 (ep1445) | pending (T4) | | Ep2104 | ~9.81 (est) | 9.7976 (ep2084) | 9.6380 (ep1445) | **9.9–21.3 (CPU)** | | Ep2109 | 9.7975 | **9.7975** (ep2109) | 9.6380 (ep1445) | pending | | Ep2114 | 9.7891 | 9.7891 (ep2114) | **9.6235** (ep2114) ← batch ATL | pending | | Ep2116 | **9.7884** | **9.7884** ← epoch ATL | **9.6235** (ep2114) | pending | | Ep2487 | S6 fired | 18L→19L surgery | — | 2026-05-20T21:33Z; Gen1 step1/6 | | Ep2922 | **9.4992** | **9.4992** ← first sub-9.50 | 9.1370 (chip) | 2026-05-22; LOG expert 0%→28% awakening | | Ep3263 | **9.3651** | ← epoch ATL | **9.0095** (chip) | Broke 139-epoch plateau | | Ep3325 | S7 fired | 18L→19L surgery | — | 2026-05-24T13:47Z; 1315 tensors | | Ep3326 | **9.3182** | ← epoch ATL (first 19L ep) | **8.9190** (chip) | +0.047 nat improvement over prior best | | Ep3383 | S8 fired | 19L→20L surgery | — | Only 58 epochs after S7 | | Ep3456 | **9.2847** | ← epoch ATL (20L) | **8.8540** (chip) | WALD ep3454 INT 91% cliff | | Ep~3470 | S9 fired | 20L→21L surgery | — | Largest post-surgery spike in history (+0.14 nat) | | Ep~3652 | S10 fired | 21L→22L surgery | — | Pre-surgery best 9.2933 | | Ep~4098 | S11 fired | 22L→23L surgery | — | 2026-05-27 morning | | Ep~4140 | S11b fired | 23L→24L surgery | — | Rapid plateau ~42 ep after S11 | | Ep4202 | S12 fired | 24L→25L surgery | — | 2026-05-27T16:43Z; Gen3 plateau | | **Ep4202** | **CORD surgery** | **25L → 2×25L dual-stream** | — | **2026-05-27T16:44Z — first ever autonomous single→dual-stream bifurcation** | | Ep~4203 | 9.3241 | ← first post-cord epoch avg | **8.7123** (chip, new ATL) | Dual-stream live | | Ep~4207 | S13 fired | 25L→26L surgery (both streams) | **8.6852** (chip, new ATL) | 2026-05-27T17:40Z; fib_index 6→7 | | ~ep4280 | S14 fired | 26L→27L (both streams) | — | 2026-05-29 | | ~ep4350 | S15 fired | 27L→28L (both streams) | — | 2026-05-29 | | ~ep4453 | backbone-trains cliff | EP-AVG **9.38 → 8.43** in <100 ep | — | DiffLayerNorm fix unfroze the transformer body — largest structural drop in the run | | ~ep4740 | S16 fired | 28L→29L (both streams) | — | 2026-05-31 (✓ checkpoint-mtime) | | ep5610 | S17 fired | 29L→30L (both streams) | — | 2026-06-06 21:08 (✓ checkpoint-mtime) | | ep6132 | EP-AVG ATL 6.4339 | 30L | 1.2637 (chip) | prior 30L best | | ep6339 | S18 fired | 30L→31L (both streams) | — | depth surgery; Fibonacci-gated | | ep6487 | EP-AVG ATL 5.8693 | 31L | 1.2637 (chip) | former all-time best | | ~ep6500 | S19 fired | 31L→32L (both streams) | — | 2026-06-15 · Fibonacci plateau gate | | ~ep7000 | S20 fired | 32L→33L (both streams) | — | 2026-06-21 · autonomous | | ep7588 | EP-AVG ATL **4.6842** | 33L — **all-time best** | **0.6116** (chip) | 512CTX · BATCH=1 · Modal · descending | The benchmark suite runs 5 fixed prompts covering English, German, multilingual, narrative, and technical domains. Results are reproducible via the open-source `moe-test` binary. **Surgery gate prediction (recorded 2026-05-16T18:40Z) — outcome update 2026-05-19:** A trendline fitted to the ep400–ep1459 loss curve predicted the surgery gate threshold (9.8 epoch-avg) at approximately **ep~2000**. **Prediction confirmed**: the loss gate was cleared at ep2080 (9.7997, 2026-05-19T10:40Z), within the predicted ep2000–2150 window. The gate fires when loss plateaus below 9.8 for a 144-epoch window with `myc_stable ≥ 5`. The loss gate was cleared at ep2080. Following that, albert. entered an **alternating descent phase**: five new epoch ATLs in seven epochs (ep2109–ep2116), dropping from 9.7976 → 9.7884 in under two hours of wall time. WALD sev=0.953; myc_L3 showed its first activity uptick (1.61→1.68 ×10−9) at ep2114. The plateau gate cannot fire during active descent — surgery timing is now conditioned on when the model settles into the next attractor, not on a fixed epoch countdown. **Milestone (2026-05-17T05:48Z):** First sub-10.0 batch loss in albert. history — **9.9948** at ep1553 batch 149/300. **Milestone (2026-05-19T10:40Z):** First sub-9.8 epoch average — **9.7997** at ep2080. Surgery loss gate cleared. **Milestone (2026-05-19T11:00Z):** New epoch ATL — **9.7976** at ep2084. **Milestone (2026-05-19T13:29Z):** New batch ATL — **9.6235** at ep2114 (prev 9.6282). **Milestone (2026-05-19T13:40Z):** Alternating descent confirmed — five new epoch ATLs in seven epochs; epoch ATL reaches **9.7884** at ep2116. **Known limitations:** - At current training depth (~ep6670, 31L dual-stream), output quality is pre-fluency: the model produces partially coherent text in familiar domains but lacks consistent grammatical structure across longer sequences. - Context window of 128 tokens is shorter than contemporary LLMs; cannot maintain coherence over longer passages. - Ternary quantization trades weight precision for size — at this scale, some representational capacity is lost relative to F32 equivalents. - No instruction-following fine-tuning has been applied. - No RLHF, Constitutional AI, or safety fine-tuning of any kind. - Bias evaluation is pending (see below). **Open research questions (scaling risks):** - **STE gradient approximation at scale:** Straight-Through Estimation is the training mechanism for ternary weights. Its stability and convergence properties are well-characterised at the current scale (~228.5M params). Whether STE remains stable through training runs at 500M–1B+ parameters is an open empirical question — no published work has demonstrated ternary STE convergence at frontier scale. - **@sparseskip speedup baseline:** The 83 tok/s inference figure is measured against albert.'s own F32-weight dense equivalent on the same hardware. It is not a direct comparison with INT4-quantized industry inference (TensorRT-LLM, llama.cpp Q4). The relevant claim is that ternary weights eliminate a quantization step entirely — the speedup over post-hoc INT4 quantization of a larger model is a separate, untested question. - **Net2net surgery stability at scale:** All five documented layer-addition surgeries were performed on a model in the 13M–58M parameter range. Whether the Fibonacci-gated surgery protocol remains stable when applied to models at 200M+ parameters has not been tested. The plateau-gate's **withhold behavior** is now validated across six...