Canvitb16 Add Vpe Pretrain G128px S512px In1k Dv3b16 2026 06 22

--- datasets: - imagenet-1k library_name: canvit-pytorch license: mit pipeline_tag: image-feature-extraction tags: - canvit - active-vision - vision-transformer --- # CanViT-B, pretrained on ImageNet-1k CanViT-B pretrained on ImageNet-1k by dense latent distillation from [facebook/dinov3-vitb16-pretrain-lvd1689m](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m), as in the paper. CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas. [Paper (NeurIPS 2026)](https://arxiv.org/abs/2603.22570) · [Code](https://github.com/m2b3/CanViT) · [Project page](https://m2b3.github.io/CanViT/) · [All checkpoints](https://huggingface.co/canvit) ## Usage ```bash pip install "canvit-pytorch>=0.2" ``` ```python import torch from PIL import Image from canvit_pytorch import CanViTForPretraining from canvit_pytorch.episode import run_episode from canvit_pytorch.policies import make_policy from canvit_pytorch.preprocess import preprocess model = CanViTForPretraining.from_pretrained("canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in1k-dv3b16-2026-06-22").eval() scene = preprocess(512)(Image.open("scene.jpg").convert("RGB")).unsqueeze(0) # [1, 3, 512, 512] # Five glimpses of 128 px, coarse to fine: the full scene, then quadrants. policy = make_policy("coarse_to_fine", batch_size=1, device=scene.device, num_glimpses=5, canvas_grid_size=32) with torch.inference_mode(): steps = run_episode( canvit=model.canvit, images=scene, policy=policy, num_glimpses=5, glimpse_size_px=128, initial_state=model.init_state(batch_size=1, canvas_grid_size=32), ) canvas = steps[-1].state.canvas features = model.canvit.canvas_patch_grid(canvas) # [1, 32, 32, canvas_dim]: the scene-wide canvas teacher_patches = model.predict_teacher_patches(canvas) # the teacher's features of the whole scene, standardized ``` ## Details | | | |---|---| | Pretraining data | ImageNet-1k | | Teacher | [facebook/dinov3-vitb16-pretrain-lvd1689m](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m) | | Scenes | 512 px | | Glimpses | 128 px | | Canvas during pretraining | 32 × 32 (any size at inference) | | Training steps | 1,837,056 | ## Citation ```bibtex @article{berreby2026canvit, title={CanViT: Toward Active-Vision Foundation Models}, author={Berreby, Yoha{\"i}-Eliel...

--- license: mit pipeline_tag: image-feature-extraction --- # CanViT: Toward Active-Vision Foundation Models CanViT (Canvas Vision Transformer) is a scalable recurrent architecture for fine-grained vision and the first **Active-Vision Foundation Model (AVFM)**. It processes scenes through sequences of localized glimpses, integrating observations over time into a persistent scene-wide latent workspace — the **canvas** — via **Canvas Attention**, an efficient asymmetric cross-attention mechanism. - **Paper:** [CanViT: Toward Active-Vision Foundation Models](https://arxiv.org/abs/2603.22570) - **Code:** [https://github.com/m2b3/CanViT-PyTorch](https://github.com/m2b3/CanViT-PyTorch) ## Model Description CanViT decouples "thinking" (backbone-level) and "memory" (canvas-level), eliminating canvas-side self-attention to achieve low-latency sequential inference. The model was pretrained via policy-agnostic passive-to-active dense latent distillation, reconstructing scene-wide DINOv3 embeddings from sequences of randomized glimpses. It demonstrates strong performance on ADE20K semantic segmentation and ImageNet-1k classification, outperforming prior active vision models. This checkpoint was pretrained on **ImageNet-1k** for 1,837,056 steps, distilling a `dinov3_vitb16` teacher at 512px scene resolution with 128px glimpses onto a 32×32 canvas. ## Sample Usage We recommend installing via `uv`: ```bash uv add canvit-pytorch ``` Or via `pip`: ```bash pip install canvit-pytorch ``` Then, you can run inference using the following snippet: ```python from canvit_pytorch import CanViTForPretrainingHFHub, Viewpoint, sample_at_viewpoint from canvit_pytorch.preprocess import preprocess from PIL import Image import torch # CanViT is integrated with the HuggingFace Hub. model = CanViTForPretrainingHFHub.from_pretrained( "canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in1k-dv3b16-2026-06-22" ).eval() # Replace with the image of your choice image = Image.open("test_data/Cat03.jpg").convert("RGB") image = preprocess(512)(image) image = image.unsqueeze(0) # [1, 3, 512, 512] # CanViT is a recurrent model. state = model.init_state(batch_size=1, canvas_grid_size=32) # Let's process a first glimpse: centered, zoomed-out. with torch.inference_mode(): vp = Viewpoint.full_scene(batch_size=1, device=image.device) glimpse = sample_at_viewpoint(spatial=image, viewpoint=...

Source context: 33 downloads · 0 likes · Pipeline image-feature-extraction · Library canvit-pytorch · Repo canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in1k-dv3b16-2026-06-22