A fitted Jacobian lens for mistralai/Mistral-7B-Instruct-v0.3 , from Anthropic's Verbalizable Representations Form a Global Workspace in Language Models , fitted with the reference implementation anthropics/jacobian-lens . Contents
Model source
Source description
# Jacobian Lens for Mistral-7B-Instruct-v0.3 A fitted Jacobian lens for , from Anthropic's , fitted with the reference implementation . ## Contents — a dict with keys (dict fp16 tensor), , , . Load it with . | field | value | |---|---| | base model | (32 layers, ) | | source layers | 0–30 (all 31 layers below the final) | | target layer | 31 (final) | | Jacobian shape | per layer, fp16 on disk | | file size | ~0.97 GiB | | fitting corpus | 100 WikiText-103 prompts × 128 tokens (first 16 positions skipped as attention sinks) | | estimator | , mean over valid source positions | | fit hardware | 1× NVIDIA H100 80GB, ~45 min | ## Usage Applying the lens is forward-only (no retained graph) and runs comfortably on a single consumer GPU or Apple Silicon (MPS). ## License Apache-2.0, matching the reference implementation. The base model is subject to its own license (Apache-2.0).
Sources
1 sourceSource excerpts
3 excerptslicense: apache-2.0 basemodel: mistralai/Mistral-7B-Instruct-v0.3 tags: interpretability jacobian-lens logit-lens mistral libraryname: jlens
mistral7b_v03_lens.pttorch.saveJlayer -> [4096, 4096]n_promptssource_layersd_modeljlens.JacobianLens.load(...)mistralai/Mistral-7B-Instruct-v0.3d_model=4096[4096, 4096]dim_batch=32python import torch, transformers, jlens from huggingface_hub import hf_hub_download hf = transformers.AutoModelForCausalLM.from_pretrained( "mistralai/Mistral-7B-Instruct-v0.3", dtype=torch.bfloat16 ).to("mps") # or "cuda" tok = transformers.AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3") model = jlens.from_hf(hf, tok) path = hf_hub_download( "dormantx/jacobian-lens-mistral-7b-instruct-v0.3", "mistral7b_v03_lens.pt" ) lens = jlens.JacobianLens.load(path) # per-layer lens logits at the last position, J-lens vs logit-lens baseline lens_logits, model_logits, _ = lens.apply( model, "Fact: The currency used in the country shaped like a boot is", positions=[-1], layers=[14, 18, 21], use_jacobian=True, ) mistralai/Mistral-7B-Instruct-v0.3A fitted Jacobian lens for mistralai/Mistral-7B-Instruct-v0.3 , from Anthropic's Verbalizable Representations Form a Global Workspace in Language Models , fitted with the reference implementation anthropics/jacobian-lens . Contents
dormantx/jacobian-lens-mistral-7b-instruct-v0.3