Canvitb16 Add Vpe Pretrain G128px S512px In21k Dv3b16 2026 02 02

CanViT: Toward Active-Vision Foundation Models

CanViT (Canvas Vision Transformer) is a scalable recurrent architecture for fine-grained vision and the first Active-Vision Foundation Model (AVFM). It processes scenes through sequences of localized glimpses, integrating observations over time into a persistent scene-wide latent workspace — the canvas — via Canvas Attention, an efficient asymmetric cross-attention mechanism.

• Paper: CanViT: Toward Active-Vision Foundation Models
• Code: https://github.com/m2b3/CanViT-PyTorch

Model Description

CanViT decouples "thinking" (backbone-level) and "memory" (canvas-level), eliminating canvas-side self-attention to achieve low-latency sequential inference. The model was pretrained via policy-agnostic passive-to-active dense latent distillation, reconstructing scene-wide DINOv3 embeddings from sequences of randomized glimpses. It demonstrates strong performance on ADE20K semantic segmentation and ImageNet-1k classification, outperforming prior active vision models.

Sample Usage

We recommend installing via uv:

uv add canvit-pytorch

Or via pip:

pip install canvit-pytorch

Then, you can run inference using the following snippet:

from canvit_pytorch import CanViTForPretrainingHFHub, Viewpoint, sample_at_viewpoint
from canvit_pytorch.preprocess import preprocess
from PIL import Image
import torch

# CanViT is integrated with the HuggingFace Hub.
model = CanViTForPretrainingHFHub.from_pretrained(
    "canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02"
).eval()

# Replace with the image of your choice
image = Image.open("test_data/Cat03.jpg").convert("RGB")
image = preprocess(512)(image)
image = image.unsqueeze(0)  # [1, 3, 512, 512]

# CanViT is a recurrent model.
state = model.init_state(batch_size=1, canvas_grid_size=32)

# Let's process a first glimpse: centered, zoomed-out.
with torch.inference_mode():
    vp = Viewpoint.full_scene(batch_size=1, device=image.device)
    glimpse = sample_at_viewpoint(spatial=image, viewpoint=vp, glimpse_size_px=128)
    out = model(glimpse=glimpse, state=state, viewpoint=vp)

# Inspect the canvas structure
# The canvas contains the model's working understanding of the scene
canvas_spatial = model.get_spatial(out.state.canvas)  # [1, 1024, 1024]
canvas_spatial = canvas_spatial.unflatten(1, (32, 32))  # [1, 32, 32, 1024] — spatial feature map
print(out.state.recurrent_cls.shape)  # [1, 1, 768] — global CLS token

Citation

@article{berreby2026canvit,
  title={CanViT: Toward Active-Vision Foundation Models},
  author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
  year={2026},
  eprint={2603.22570},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2603.22570}
}