Slm Rl Freeway

--- library_name: peft base_model: LiquidAI/LFM2.5-350M pipeline_tag: text-generation tags: - lora - peft - transformers - reinforcement-learning - atari - slm-rl - freeway license: apache-2.0 --- # BLANK/slm-rl-freeway PEFT LoRA adapter that warm-starts **Freeway** play for [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) in the [SLM-RL](https://github.com/CraftsMan-Labs/SLM-RL) workshop. | | | |---|---| | **Game** | `freeway` | | **Base model** | `LiquidAI/LFM2.5-350M` | | **Adapter layout** | `adapter/` (PEFT `adapter_config.json` + weights) | | **Training** | `reject_sft` on DQN teacher demos | | **Champion generation** | 1 | | **Promoted** | True (baked pack / SFT adopted as RL initialization) | | **Dataset pack** | [BLANK/slm-rl-freeway](https://huggingface.co/datasets/BLANK/slm-rl-freeway) | | **DQN teacher** | [BLANK/slm-rl-freeway-dqn](https://huggingface.co/BLANK/slm-rl-freeway-dqn) | Paste `BLANK/slm-rl-freeway` as the playground **adapter URL** (and usually the same id as the **dataset URL**). ## Install ```bash pip install "transformers>=4.46" peft accelerate torch ``` ## Load with transformers + PEFT Weights live under the `adapter/` subfolder — pass `subfolder="adapter"`. ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel BASE = "LiquidAI/LFM2.5-350M" ADAPTER = "BLANK/slm-rl-freeway" # this repo device = ( "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu" ) dtype = torch.bfloat16 if device != "cpu" else torch.float32 tokenizer = AutoTokenizer.from_pretrained(BASE) if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=dtype) model = PeftModel.from_pretrained(model, ADAPTER, subfolder="adapter") model.to(device).eval() messages = [ {"role": "system", "content": "You play Freeway. Reply with ACTION: <id>."}, {"role": "user", "content": "Legal actions: 1) NOOP 2) UP\nChoose."}, ] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=False, ) inputs = tokenizer(prompt, return_tensors="pt").to(device) with torch.inference_mode(): out = model.generate(**inputs, max_new_tokens=24, do_sample=False) print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))...