--- library_name: transformers tags: - sentinel-prime - moe - sparse-mixture-of-experts - from-scratch - custom-architecture license: apache-2.0 language: - en pipeline_tag: text-generation model-index: - name: sentinel-prime-350m results: [] --- # Sentinel Prime 350M — Sparse MoE Language Model **Sentinel Prime 350M** is a from-scratch sparse Mixture of Experts (MoE) transformer built by [QubitPage Research](https://qubitpage.com). ## Architecture | Parameter | Value | |---|---| | Total Parameters | 471,231,488 | | Active Parameters | ~471,231,488 per token | | Hidden Dimension | 1024 | | Layers | 24 | | Attention Heads | 16 (Q) / 4 (KV) | | FFN Dimension | 2752 | | Experts | 1 total, top-1 active | | Vocab Size | 100,277 (tiktoken cl100k_base) | | Max Sequence Length | 2048 | | Position Encoding | RoPE (theta=500000.0) | | Normalization | RMSNorm | | FFN Type | SwiGLU | | Attention | Grouped Query Attention (GQA) | ## Key Features - **Sparse MoE**: Only 1/1 experts active per token - **GQA**: Memory-efficient grouped query attention - **SwiGLU**: LLaMA/Mistral-style feed-forward - **RoPE**: Rotary position embeddings for length generalization - **From Scratch**: No pretrained weights, trained from random initialization ## Training - **Data**: FineWeb-Edu (educational web text) - **Tokens Seen**: 3,113,287,680 - **Best Validation Loss**: 3.0578 - **Hardware**: NVIDIA GH200 96GB HBM3e - **Framework**: PyTorch 2.5.1 ## Usage ```python from transformers import AutoModelForCausalLM, AutoConfig # Register custom model from hf_model import SentinelBrainConfig, SentinelBrainForCausalLM from hf_tokenizer import SentinelBrainTokenizer model = SentinelBrainForCausalLM.from_pretrained("qubitpage/sentinel-prime-350m", trust_remote_code=True) tokenizer = SentinelBrainTokenizer() input_ids = tokenizer("The meaning of life is", return_tensors="pt")["input_ids"] output = model.generate(input_ids, max_new_tokens=50) print(tokenizer.decode(output[0])) ``` ## License Apache 2.0 ## Benchmarks Results from EleutherAI [`lm-evaluation-harness`](https://github.com/EleutherAI/lm-evaluation-harness) (latest) run on a single NVIDIA GH200 96GB. Full results, configs and per-sample logs are public in the companion dataset: **[qubitpage/sentinel-prime-350m-evals](https://huggingface.co/datasets/qubitpage/sentinel-prime-350m-evals)** | Model | Params | Train Tokens | arc_challenge | arc_easy | hellaswag | lambada_openai | openbookqa | piqa | sciq | winogrande | **Avg** | |---|---|---|---|---|---|---|---|---|---|---|---| | **Sentinel-Prime-350M (ours)** | 471M | 3.1B | 0.194 | 0.352 | 0.264 | 0.001 | 0.120 | 0.566 | 0.481 | 0.501 | **0.310** | | **Pythia-410M** | 410M | 300B | 0.240 | 0.520 | 0.400 | 0.510 | 0.300 | 0.670 | 0.810 | 0.530 | **0.498** | | **GPT-Neo-125M** | 125M | 300B | 0.190 | 0.430 | 0.300 | 0.370 | 0.260 | 0.630 | 0.760 | 0.520 | **0.433** | | **SmolLM-360M** | 360M | 600B | 0.340 | 0.660 | 0.520 | 0.460 | 0.370 | 0.720 | 0.910 | 0.570 | **0.569** | ### Training-Compute Context | Model | Hardware | Tokens Seen | Compute Multiplier vs ours | |---|---|---|---| | **Sentinel-Prime-350M** | 1x NVIDIA GH200 96GB | **3.1 B** | **1x (baseline)** | | Pythia-410M | TPU v4 cluster | 300 B | **97x** | | SmolLM-360M | 64x NVIDIA H100 | 600 B | **194x** | Sentinel-Prime-350M was trained on **3.1B tokens** — Pythia-410M on **300B** (97x more) and SmolLM-360M on **600B** (194x more). The current avg of **0.310** vs Pythia's **0.497** and SmolLM's **0.569** therefore reflects an early-checkpoint snapshot at <1% of typical training budget for this size class. Per Chinchilla (Hoffmann et al. 2022), a 471M dense model is compute-optimal at ~9.4B tokens (~20x params); we are at **0.33x compute-optimal**. Continued pretraining on the same architecture/data is expected to scale predictably toward the reference band. Reference scores from EleutherAI Pythia paper and HuggingFace SmolLM card. All evaluated 0-shot under identical prompt formats. Reproduce locally: ```bash pip install "lm_eval[hf]" lm_eval run --model hf \ --model_args pretrained=qubitpage/sentinel-prime-350m,trust_remote_code=True,dtype=float32 \ --tasks arc_challenge,arc_easy,hellaswag,lambada_openai,openbookqa,piqa,sciq,winogrande \ --device cuda:0 --batch_size auto:4 ``` ## Support This Project Sentinel-Prime is being trained on a single GH200 against models that used hundreds of GPUs. If these results interest you and you want to help us close the **97x–194x compute gap**, you can back the project here: **[Support Sentinel-Prime on Surge](https://surge.xyz/discovery/carphacom-robotised)** Every contribution funds more GH200 hours and brings the next checkpoint closer to (and past) the Pythia / SmolLM reference band.
