MiniMoE

--- language: - en tags: - mixture-of-experts - moe - gpt - language-model datasets: - HuggingFaceFW/fineweb-edu --- # miniMoE miniMoE is a sparse GPT-style language model I trained from scratch on 10B FineWeb-Edu tokens across 8x A100 GPUs with DDP. This repo holds the trained checkpoint. The code, training pipeline, and full write-up are on GitHub: https://github.com/maokner/miniMoE ## What's in here `minimoe_step_0019073.pt` is the final checkpoint, taken at step 19,073 after ~10B tokens. It is a full training checkpoint, so alongside the model weights it keeps the optimizer and RNG state. That means you can either sample from it or resume training where it left off. ## Model | Item | Value | |---|---:| | Context length | 1,024 | | Vocabulary | 50,304 GPT-2 BPE tokens | | Transformer blocks | 6 | | Hidden size | 768 | | Experts per block | 8 | | Active experts per token | 2 | | Total parameters | 280.4M | | Active parameters per token | 110.4M | Every dense feed-forward block is replaced with an 8-expert top-2 MoE layer. Attention, embeddings, layer norms, the router, and the tied output head are shared across tokens. ## Results These come from the completed training log. | Metric | miniMoE | |---|---:| | Train loss | 3.0869 | | Validation loss | 3.0409 | | Test loss | 3.0725 | On the full HellaSwag benchmark (10,042 examples, completion-style scoring), this checkpoint edges out GPT-2 124M while activating fewer parameters per token. | Model | Active params per token | HellaSwag | |---|---:|---:| | miniMoE (this checkpoint) | 110.4M | 31.2% | | GPT-2 124M | 124M | 29.6% | The instruction-tuned variant holds HellaSwag steady at 30.6%; see the GitHub repo for the full evaluation and routing ablations. ## Usage Clone the GitHub repo, drop the checkpoint next to the code, and sample from it: ```bash pip install -r requirements.txt python sample.py -c minimoe_step_0019073.pt -p "Sparse expert language models" ``` To pull just the checkpoint from here: ```python from huggingface_hub import hf_hub_download path = hf_hub_download("mokner123/miniMoE", "minimoe_step_0019073.pt") ```

Source context: 0 downloads · 0 likes · Repo mokner123/miniMoE