Aller au contenu
ModelPrésentéExterne
Télécharger

Slm 125m Base

A 125M-parameter Llama-style base language model, pretrained from scratch on a legal/financial corpus (US case law + SEC filings + a web slice, 2.19B tokens).

Sourcehuggingface.co/thesreedath/slm-125m-base/tree/15e0afe1d7b78224f79287372c4868baab29df15
TypeText modelVersionmodelDClasse

Source du modèle

Type
Text model
Taille du fichier
240 MB
Version
vmodel
Mis à jour
19 oct. 2026
Téléchargements
275
Source
Hugging Face

Description de la source

slm-125m-base

A 125M-parameter Llama-style base language model, pretrained from scratch on a legal/financial corpus (US case law + SEC filings + a web slice, ~2.19B tokens).

  • Pretraining: 10 epochs, 8xH100. Held-out validation perplexity: 8.36.
  • This is a base COMPLETER, not a chat model. Prompt it with the start of a sentence and it continues in the legal register. It speaks the register fluently but does not know facts (knowledge is capped at ~2 bits/param at this size).
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("thesreedath/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("thesreedath/slm-125m-base")
ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True,
      temperature=0.8, min_new_tokens=40)[0]))

Full pipeline + fine-tuning code: https://github.com/Vizuara-AI-Lab/slm-125m-from-scratch

Sources

1 source
Hugging FacePrincipalVersion 15e0afe1d7b78224f79287372c4868baab29df15
huggingface.co/thesreedath/slm-125m-base/tree/15e0afe1d7b78224f79287372c4868baab29df15

Vérifié 8 août

Artefacts du modèle

1 artefact

model.safetensors

safetensors · 240 MB · SHA-256 cc31f3bff9c4…9359 · Hugging Face

Télécharger

Extraits de sources

3 extraits
Readme11 juil.huggingface.co/thesreedath/slm-125m-base

--- license: other language: en tags: [legal, from-scratch, llama, small-language-model] --- # slm-125m-base A **125M-parameter Llama-style base language model**, pretrained **from scratch** on a legal/financial corpus (US case law + SEC filings + a web slice, ~2.19B tokens). - **Pretraining:** 10 epochs, 8xH100. **Held-out validation perplexity: 8.36**. - **This is a base COMPLETER, not a chat model.** Prompt it with the start of a sentence and it continues in the legal register. It speaks the register fluently but does not know facts (knowledge is capped at ~2 bits/param at this size). ```python from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("thesreedath/slm-125m-base") model = AutoModelForCausalLM.from_pretrained("thesreedath/slm-125m-base") ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, min_new_tokens=40)[0])) ``` Full pipeline + fine-tuning code: https://github.com/Vizuara-AI-Lab/slm-125m-from-scratch

Note8 aoûthuggingface.co/thesreedath/slm-125m-base

Source context: 275 downloads · 1 likes · Repo thesreedath/slm-125m-base

Note16 juil.huggingface.co/thesreedath/slm-125m-base

Source context: 145 downloads · 1 likes · Repo thesreedath/slm-125m-base