Jacobian Lens Mistral 7b Instruct v0.3
A fitted Jacobian lens for mistralai/Mistral-7B-Instruct-v0.3, from Anthropic's [Verbalizable Representations Form a Global Workspace in Language Models]( fitted with the reference implementation...
Model source
- Kind
- Other
- File size
- 992 MB
- Base model
- mistralai/Mistral-7B-Instruct-v0.3
- Version
- vRevision 95d95f9
- Updated
- Oct 4, 2026
- Downloads
- 0
- Source
- Hugging Face
Source description
Screenshot 2026-07-12 at 5.13.03 PM
Jacobian Lens for Mistral-7B-Instruct-v0.3
A fitted Jacobian lens for
mistralai/Mistral-7B-Instruct-v0.3,
from Anthropic's Verbalizable Representations Form a Global Workspace in Language
Models, fitted with the
reference implementation anthropics/jacobian-lens.
Contents
mistral7b_v03_lens.pt — a torch.save dict with keys J (dict layer -> [4096, 4096]
fp16 tensor), n_prompts, source_layers, d_model. Load it with
jlens.JacobianLens.load(...).
| field | value |
|---|---|
| base model | mistralai/Mistral-7B-Instruct-v0.3 (32 layers, d_model=4096) |
| source layers | 0–30 (all 31 layers below the final) |
| target layer | 31 (final) |
| Jacobian shape | [4096, 4096] per layer, fp16 on disk |
| file size | ~0.97 GiB |
| fitting corpus | 100 WikiText-103 prompts × 128 tokens (first 16 positions skipped as attention sinks) |
| estimator | dim_batch=32, mean over valid source positions |
| fit hardware | 1× NVIDIA H100 80GB, ~45 min |
Usage
import torch, transformers, jlens
from huggingface_hub import hf_hub_download
hf = transformers.AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-Instruct-v0.3", dtype=torch.bfloat16
).to("mps") # or "cuda"
tok = transformers.AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
model = jlens.from_hf(hf, tok)
path = hf_hub_download(
"dormantx/jacobian-lens-mistral-7b-instruct-v0.3", "mistral7b_v03_lens.pt"
)
lens = jlens.JacobianLens.load(path)
# per-layer lens logits at the last position, J-lens vs logit-lens baseline
lens_logits, model_logits, _ = lens.apply(
model, "Fact: The currency used in the country shaped like a boot is",
positions=[-1], layers=[14, 18, 21], use_jacobian=True,
)
Applying the lens is forward-only (no retained graph) and runs comfortably on a single consumer GPU or Apple Silicon (MPS).
License
Apache-2.0, matching the reference implementation. The base model
mistralai/Mistral-7B-Instruct-v0.3 is subject to its own license (Apache-2.0).
Sources
1 source
Sources
1 sourceVerified Oct 4
Model artifacts
1 artifact
Model artifacts
1 artifactSource excerpts
3 excerpts
Source excerpts
3 excerpts--- license: apache-2.0 base_model: mistralai/Mistral-7B-Instruct-v0.3 tags: - interpretability - jacobian-lens - logit-lens - mistral library_name: jlens ---  # Jacobian Lens for Mistral-7B-Instruct-v0.3 A fitted Jacobian lens for [`mistralai/Mistral-7B-Instruct-v0.3`](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3), from Anthropic's [*Verbalizable Representations Form a Global Workspace in Language Models*](https://transformer-circuits.pub/2026/workspace/index.html), fitted with the reference implementation [`anthropics/jacobian-lens`](https://github.com/anthropics/jacobian-lens). ## Contents `mistral7b_v03_lens.pt` — a `torch.save` dict with keys `J` (dict `layer -> [4096, 4096]` fp16 tensor), `n_prompts`, `source_layers`, `d_model`. Load it with `jlens.JacobianLens.load(...)`. | field | value | |---|---| | base model | `mistralai/Mistral-7B-Instruct-v0.3` (32 layers, `d_model=4096`) | | source layers | 0–30 (all 31 layers below the final) | | target layer | 31 (final) | | Jacobian shape | `[4096, 4096]` per layer, fp16 on disk | | file size | ~0.97 GiB | | fitting corpus | 100 WikiText-103 prompts × 128 tokens (first 16 positions skipped as attention sinks) | | estimator | `dim_batch=32`, mean over valid source positions | | fit hardware | 1× NVIDIA H100 80GB, ~45 min | ## Usage ```python import torch, transformers, jlens from huggingface_hub import hf_hub_download hf = transformers.AutoModelForCausalLM.from_pretrained( "mistralai/Mistral-7B-Instruct-v0.3", dtype=torch.bfloat16 ).to("mps") # or "cuda" tok = transformers.AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3") model = jlens.from_hf(hf, tok) path = hf_hub_download( "dormantx/jacobian-lens-mistral-7b-instruct-v0.3", "mistral7b_v03_lens.pt" ) lens = jlens.JacobianLens.load(path) # per-layer lens logits at the last position, J-lens vs logit-lens baseline lens_logits, model_logits, _ = lens.apply( model, "Fact: The currency used in the country shaped like a boot is", positions=[-1], layers=[14, 18, 21], use_jacobian=True, ) ``` Applying the lens is forward-only (no retained graph) and runs comfortably on a single consumer GPU or Apple Silicon (MPS). ## License Apache-2.0, matching the reference implementati...
Source context: 0 downloads · 0 likes · Library jlens · Repo andyx10/jacobian-lens-mistral-7b-instruct-v0.3
Source context: 0 downloads · 0 likes · Library jlens · Repo dormantx/jacobian-lens-mistral-7b-instruct-v0.3
