Gemma 4 E2b Nla L23 Ar v0 0 1

--- license: cc-by-4.0 language: - en base_model: google/gemma-4-E2B tags: - natural-language-autoencoder - nla - interpretability - mechanistic-interpretability - gemma - consumer-gpu - peft - lora library_name: peft pipeline_tag: text-generation --- # Gemma-4-E2B NLA AR (Activation Reconstructor) — v0.0.1 LoRA adapter (+ a 1536→1536 linear head) for `google/gemma-4-E2B` that takes a natural-language explanation produced by the matched [Activation Verbalizer](https://huggingface.co/Solshine/gemma-4-e2b-nla-L23-av-v0_0_1) and reconstructs a 1536-dimensional activation vector intended to round-trip (by cosine similarity) to the original L23 residual-stream activation. Trained end-to-end on a single 4 GB consumer GPU (NVIDIA GTX 1650 Ti Max-Q) following a **customized variation (see below)** of the methodology of Fraser-Taliente, Kantamneni, Ong et al. 2026 ([Transformer Circuits](https://transformer-circuits.pub/2026/nla/)). ### Customizations vs the source methodology A consumer-hardware *variation*, not a faithful reproduction: a **LoRA adapter (+ a 1536→1536 linear head) over a 4-bit NF4-quantized** frozen base rather than full-fine-tune bf16; a single 4 GB GPU with micro-batch 1 + gradient accumulation and a modest SFT-step budget; an **SFT-only** released pair (Phase-4 GRPO explored separately, not shipped); and added AV-side evaluations beyond round-trip cosine (see the matched [`MODEL_CARD_AV.md`](MODEL_CARD_AV.md)). Note round-trip cosine on this AR is largely a structural-projection metric, not per-row faithfulness — see Limitations. ## How to use ```python from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from peft import PeftModel from huggingface_hub import snapshot_download import torch import torch.nn as nn import numpy as np BASE = "google/gemma-4-E2B" AR_REPO = "Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1" AR_TRUNCATION = 18 # capture hidden state at layer (K+1-1) = 17 (0-indexed) D_MODEL = 1536 AR_TEMPLATE = "Summary of the following text: <text>{explanation}</text> <summary>" bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16, bnb_4bit_quant_type="nf4") tok = AutoTokenizer.from_pretrained(BASE) base = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto") ar_local = snapshot_download(repo_id=AR_REPO) ar =...

Source context: 3 downloads · 0 likes · Pipeline text-generation · Library peft · Repo Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1

Source context: 34 downloads · 0 likes · Pipeline text-generation · Library peft · Repo Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1