--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: text-generation tags: - pretraining - causal-lm - gpt-neox - pondering - latent-thoughts - icml-2026 datasets: - monology/pile-uncopyrighted --- # PonderLM-2-Pythia-410m Pythia-410m architecture pretrained with **PonderLM-2**, the method introduced in [*PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space*](https://arxiv.org/abs/2509.23184) (ICML 2026 Spotlight). > **TL;DR.** Chain-of-Thought scales test-time compute by generating extra > *tokens*. PonderLM-2 does the same at **pretraining** time, but in > **continuous space**: before predicting each next token the model first > emits a few **latent thoughts** — extra last-hidden-state vectors — and > feeds them back into itself. ``` vanilla: x1 ──► x2 ──► x3 ──► x4 PonderLM-2: x1 ──► z1 ──► x2 ──► z2 ──► x3 ──► z3 ──► x4 ──► z4 z_i = latent thought emitted before predicting x_{i+1} ``` - **Code**: [LUMIA-Group/PonderLM-2](https://github.com/LUMIA-Group/PonderLM-2) - **Paper**: [arXiv:2509.23184](https://arxiv.org/abs/2509.23184) - **Sibling checkpoint**: [`zeng123/PonderLM-2-Pythia-1.4b`](https://huggingface.co/zeng123/PonderLM-2-Pythia-1.4b) --- ## Usage The model ships with a custom `modeling_gpt_neox.py` that runs the pondering forward pass. Loading via `AutoModelForCausalLM` requires `trust_remote_code=True`: ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM ckpt = "zeng123/PonderLM-2-Pythia-410m" tok = AutoTokenizer.from_pretrained(ckpt) model = AutoModelForCausalLM.from_pretrained( ckpt, torch_dtype=torch.bfloat16, trust_remote_code=True, ).cuda() prompt = "The mitochondria is " out = model.generate( **tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=64, use_cache=True, ) print(tok.decode(out[0], skip_special_tokens=True)) ``` --- ## Model details | | | |---|---| | Architecture | GPT-NeoX (Pythia family) | | Parameters | 410 M | | Hidden size | 1024 | | Layers | 24 | | Attention heads | 16 | | Context length | 2048 | | Vocabulary | 50 304 | | Tokenizer | GPT-NeoX BPE (same as Pythia) | | Precision | BF16 | --- ## Citation ```bibtex @article{zeng2025ponderlm, title={Ponderlm-2: Pretraining llm with latent thoughts in continuous space}, author={Zeng, Boyi and Li, He and Song, Shixiang and Wang, Yixuan and Wang, Zitong and He, Ziwei and Wang, Xinbing and Lin, Zhouhan}, journal={arXiv preprint arXiv:2509.23184}, year={2025} } ``` ## Acknowledgements Built on top of the Pythia training stack and [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). The PonderLM baseline implementation is adapted from [LUMIA-Group/PonderingLM](https://github.com/LUMIA-Group/PonderingLM).
