SteerViT

--- base_model: - timm/vit_base_patch14_dinov2.lvd142m - timm/vit_base_patch16_224.mae - FacebookAI/roberta-large language: - en tags: - vision-transformer - image-feature-extraction - multimodal - visual-grounding - representation-learning pipeline_tag: image-feature-extraction license: apache-2.0 --- # SteerViT: Steerable Visual Representations [ECCV'26 Oral] [**Paper**](https://huggingface.co/papers/2604.02327) | [**Project Page**](https://jonaruthardt.github.io/project/SteerViT/) | [**GitHub**](https://github.com/manugaurdl/SteerViT) | [**Demo (HF Spaces)**](https://huggingface.co/spaces/JonaRuthardt/SteerViT) SteerViT equips pretrained Vision Transformers with **steerable visual representations**. Given an image and a natural-language prompt, it conditions the visual encoder through lightweight gated cross-attention to produce: - prompt-aware **global embeddings** - prompt-aware **dense patch features** - prompt-conditioned **heatmaps** This Hugging Face repository hosts the model checkpoints. ## Installation To use SteerViT, install the library directly from GitHub: ```bash python -m pip install "git+https://github.com/JonaRuthardt/SteerViT.git" ``` ## Quick start ```python import torch from PIL import Image from steervit import SteerViT device = "cuda" if torch.cuda.is_available() else "cpu" # Load the model (e.g., SteerDINOv2-Base) model = SteerViT.from_pretrained("steervit_dinov2_base.pth", device=device) transform = model.get_transforms() image = Image.open("path/to/image.jpg").convert("RGB") image_tensor = transform(image).unsqueeze(0) prompt = ["the red car"] global_features = model.get_global_features(image_tensor, texts=prompt) # pooled image embeddings dense_features = model.get_dense_features(image_tensor, texts=prompt) # patch-level visual features heatmaps = model.get_heatmaps(image_tensor, texts=prompt) # prompt-conditioned localization heatmaps attention_heatmaps = model.get_attention_heatmaps(image_tensor, texts=prompt) # attention-based heatmaps ``` If `texts=None`, SteerViT behaves like the underlying frozen ViT backbone and returns query-agnostic features. ## Available checkpoints > **🔔 New commercially usable checkpoints:** We now provide updated SteerViT checkpoints under **Apache-2.0**, which permits commercial use. These variants were adapted using only Visual Genome, COCO, and LVIS, with commercial-use image...

--- base_model: - timm/vit_base_patch14_dinov2.lvd142m - timm/vit_base_patch16_224.mae - FacebookAI/roberta-large language: - en tags: - vision-transformer - image-feature-extraction - multimodal - visual-grounding - representation-learning pipeline_tag: image-feature-extraction license: apache-2.0 --- # SteerViT: Steerable Visual Representations [**Paper**](https://huggingface.co/papers/2604.02327) | [**Project Page**](https://jonaruthardt.github.io/project/SteerViT/) | [**GitHub**](https://github.com/manugaurdl/SteerViT) | [**Demo (HF Spaces)**](https://huggingface.co/spaces/JonaRuthardt/SteerViT) SteerViT equips pretrained Vision Transformers with **steerable visual representations**. Given an image and a natural-language prompt, it conditions the visual encoder through lightweight gated cross-attention to produce: - prompt-aware **global embeddings** - prompt-aware **dense patch features** - prompt-conditioned **heatmaps** This Hugging Face repository hosts the model checkpoints. ## Installation To use SteerViT, install the library directly from GitHub: ```bash python -m pip install "git+https://github.com/JonaRuthardt/SteerViT.git" ``` ## Quick start ```python import torch from PIL import Image from steervit import SteerViT device = "cuda" if torch.cuda.is_available() else "cpu" # Load the model (e.g., SteerDINOv2-Base) model = SteerViT.from_pretrained("steervit_dinov2_base.pth", device=device) transform = model.get_transforms() image = Image.open("path/to/image.jpg").convert("RGB") image_tensor = transform(image).unsqueeze(0) prompt = ["the red car"] global_features = model.get_global_features(image_tensor, texts=prompt) # pooled image embeddings dense_features = model.get_dense_features(image_tensor, texts=prompt) # patch-level visual features heatmaps = model.get_heatmaps(image_tensor, texts=prompt) # prompt-conditioned localization heatmaps attention_heatmaps = model.get_attention_heatmaps(image_tensor, texts=prompt) # attention-based heatmaps ``` If `texts=None`, SteerViT behaves like the underlying frozen ViT backbone and returns query-agnostic features. ## Available checkpoints | Checkpoint | `from_pretrained(...)` identifier | Notes | | --- | --- | --- | | SteerDINOv2-Base | `steervit_dinov2_base.pth` | Primary model used for most experiments | | SteerMAE-Base | `steervit_mae_base.pth` | Alternative model based on MAE-backbo...

Source context: 0 downloads · 7 likes · Pipeline image-feature-extraction · Repo JonaRuthardt/SteerViT