ModelVerificadoExterno

Sentinel Prime 350m

Sentinel Prime 350M is a from-scratch sparse Mixture of Experts (MoE) transformer built by QubitPage Research.

Fuentehuggingface.co/qubitpage/sentinel-prime-350m/tree/2e9eea70445ea47d6963114c965095073923ede7

TipoOtherVersiónmodelPublicador@qubitpageCcalidad

Fuente del modelo

Tipo: Other
Tamaño del archivo: 1,76 GB
Versión: vmodel
Actualizado: 1 oct 2026
Publicador: @qubitpage
Descargas: 16
Tareas: Generación de texto
Fuente: Hugging Face

Descripción de la fuente

Sentinel Prime 350M — Sparse MoE Language Model

Sentinel Prime 350M is a from-scratch sparse Mixture of Experts (MoE) transformer built by QubitPage Research.

Architecture

Parameter	Value
Total Parameters	471,231,488
Active Parameters	~471,231,488 per token
Hidden Dimension	1024
Layers	24
Attention Heads	16 (Q) / 4 (KV)
FFN Dimension	2752
Experts	1 total, top-1 active
Vocab Size	100,277 (tiktoken cl100k_base)
Max Sequence Length	2048
Position Encoding	RoPE (theta=500000.0)
Normalization	RMSNorm
FFN Type	SwiGLU
Attention	Grouped Query Attention (GQA)

Key Features

Sparse MoE: Only 1/1 experts active per token
GQA: Memory-efficient grouped query attention
SwiGLU: LLaMA/Mistral-style feed-forward
RoPE: Rotary position embeddings for length generalization
From Scratch: No pretrained weights, trained from random initialization

Training

Data: FineWeb-Edu (educational web text)
Tokens Seen: 3,113,287,680
Best Validation Loss: 3.0578
Hardware: NVIDIA GH200 96GB HBM3e
Framework: PyTorch 2.5.1

Usage

from transformers import AutoModelForCausalLM, AutoConfig
# Register custom model
from hf_model import SentinelBrainConfig, SentinelBrainForCausalLM
from hf_tokenizer import SentinelBrainTokenizer

model = SentinelBrainForCausalLM.from_pretrained("qubitpage/sentinel-prime-350m", trust_remote_code=True)
tokenizer = SentinelBrainTokenizer()

input_ids = tokenizer("The meaning of life is", return_tensors="pt")["input_ids"]
output = model.generate(input_ids, max_new_tokens=50)
print(tokenizer.decode(output[0]))

License

Apache 2.0

Benchmarks

Results from EleutherAI lm-evaluation-harness (latest) run on a single NVIDIA GH200 96GB. Full results, configs and per-sample logs are public in the companion dataset: qubitpage/sentinel-prime-350m-evals

Model	Params	Train Tokens	arc_challenge	arc_easy	hellaswag	lambada_openai	openbookqa	piqa	sciq	winogrande	Avg
Sentinel-Prime-350M (ours)	471M	3.1B	0.194	0.352	0.264	0.001	0.120	0.566	0.481	0.501	0.310
Pythia-410M	410M	300B	0.240	0.520	0.400	0.510	0.300	0.670	0.810	0.530	0.498
GPT-Neo-125M	125M	300B	0.190	0.430	0.300	0.370	0.260	0.630	0.760	0.520	0.433
SmolLM-360M	360M	600B	0.340	0.660	0.520	0.460	0.370	0.720	0.910	0.570	0.569

Training-Compute Context

Model	Hardware	Tokens Seen	Compute Multiplier vs ours
Sentinel-Prime-350M	1x NVIDIA GH200 96GB	3.1 B	1x (baseline)
Pythia-410M	TPU v4 cluster	300 B	97x
SmolLM-360M	64x NVIDIA H100	600 B	194x

Sentinel-Prime-350M was trained on 3.1B tokens — Pythia-410M on 300B (97x more) and SmolLM-360M on 600B (194x more). The current avg of 0.310 vs Pythia's 0.497 and SmolLM's 0.569 therefore reflects an early-checkpoint snapshot at <1% of typical training budget for this size class.

Per Chinchilla (Hoffmann et al. 2022), a 471M dense model is compute-optimal at ~9.4B tokens (~20x params); we are at 0.33x compute-optimal. Continued pretraining on the same architecture/data is expected to scale predictably toward the reference band.

Reference scores from EleutherAI Pythia paper and HuggingFace SmolLM card. All evaluated 0-shot under identical prompt formats.

Reproduce locally:

pip install "lm_eval[hf]"
lm_eval run --model hf \
  --model_args pretrained=qubitpage/sentinel-prime-350m,trust_remote_code=True,dtype=float32 \
  --tasks arc_challenge,arc_easy,hellaswag,lambada_openai,openbookqa,piqa,sciq,winogrande \
  --device cuda:0 --batch_size auto:4

Support This Project

Sentinel-Prime is being trained on a single GH200 against models that used hundreds of GPUs. If these results interest you and you want to help us close the 97x–194x compute gap, you can back the project here:

Support Sentinel-Prime on Surge

Every contribution funds more GH200 hours and brings the next checkpoint closer to (and past) the Pythia / SmolLM reference band.

Fuentes

1 fuente

Hugging FacePrincipalVersión 2e9eea70445ea47d6963114c965095073923ede7

huggingface.co/qubitpage/sentinel-prime-350m/tree/2e9eea70445ea47d6963114c965095073923ede7

Verificado 1 oct

Artefactos del modelo

1 artefacto

model.safetensors

safetensors · 1,76 GB · SHA-256 41c68d8682a9…0045 · Hugging Face

Descargar

Extractos de fuentes

3 extractos

Readme11 julhuggingface.co/qubitpage/sentinel-prime-350m

--- library_name: transformers tags: - sentinel-prime - moe - sparse-mixture-of-experts - from-scratch - custom-architecture license: apache-2.0 language: - en pipeline_tag: text-generation model-index: - name: sentinel-prime-350m results: [] --- # Sentinel Prime 350M — Sparse MoE Language Model **Sentinel Prime 350M** is a from-scratch sparse Mixture of Experts (MoE) transformer built by [QubitPage Research](https://qubitpage.com). ## Architecture | Parameter | Value | |---|---| | Total Parameters | 471,231,488 | | Active Parameters | ~471,231,488 per token | | Hidden Dimension | 1024 | | Layers | 24 | | Attention Heads | 16 (Q) / 4 (KV) | | FFN Dimension | 2752 | | Experts | 1 total, top-1 active | | Vocab Size | 100,277 (tiktoken cl100k_base) | | Max Sequence Length | 2048 | | Position Encoding | RoPE (theta=500000.0) | | Normalization | RMSNorm | | FFN Type | SwiGLU | | Attention | Grouped Query Attention (GQA) | ## Key Features - **Sparse MoE**: Only 1/1 experts active per token - **GQA**: Memory-efficient grouped query attention - **SwiGLU**: LLaMA/Mistral-style feed-forward - **RoPE**: Rotary position embeddings for length generalization - **From Scratch**: No pretrained weights, trained from random initialization ## Training - **Data**: FineWeb-Edu (educational web text) - **Tokens Seen**: 3,113,287,680 - **Best Validation Loss**: 3.0578 - **Hardware**: NVIDIA GH200 96GB HBM3e - **Framework**: PyTorch 2.5.1 ## Usage ```python from transformers import AutoModelForCausalLM, AutoConfig # Register custom model from hf_model import SentinelBrainConfig, SentinelBrainForCausalLM from hf_tokenizer import SentinelBrainTokenizer model = SentinelBrainForCausalLM.from_pretrained("qubitpage/sentinel-prime-350m", trust_remote_code=True) tokenizer = SentinelBrainTokenizer() input_ids = tokenizer("The meaning of life is", return_tensors="pt")["input_ids"] output = model.generate(input_ids, max_new_tokens=50) print(tokenizer.decode(output[0])) ``` ## License Apache 2.0 ## Benchmarks Results from EleutherAI [`lm-evaluation-harness`](https://github.com/EleutherAI/lm-evaluation-harness) (latest) run on a single NVIDIA GH200 96GB. Full results, configs and per-sample logs are public in the companion dataset: **[qubitpage/sentinel-prime-350m-evals](https://huggingface.co/datasets/qubitpage/sentinel-prime-350m-evals)** | Model | Params | Train Tokens |...

Note1 octhuggingface.co/qubitpage/sentinel-prime-350m

Source context: 16 downloads · 0 likes · Pipeline text-generation · Library transformers · Repo qubitpage/sentinel-prime-350m

Note16 julhuggingface.co/qubitpage/sentinel-prime-350m

Source context: 7 downloads · 0 likes · Pipeline text-generation · Library transformers · Repo qubitpage/sentinel-prime-350m