LEAP Distilled ViT

--- license: apache-2.0 pipeline_tag: image-classification --- # LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation [Paper](https://huggingface.co/papers/2606.19483) | [GitHub](https://github.com/KevinZ0217/LEAP) | [Project Page](https://kevinz0217.github.io/LEAP_page/) This repository contains the ViT-Tiny and ViT-S checkpoints (No Register) distilled from ViT-G DINOv2 on ImageNet-100 and ImageNet-1K. The knowledge distillation process follows the procedure proposed in the paper **"LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation"**. ### Introduction Vision Foundation Models (VFMs) with ViT backbones, such as DINOv2, are computationally demanding. LEAP (Layer-skipping Efficiency via Adaptive Progression) is a training curriculum for ViT feature-based knowledge distillation. Instead of supervising the student against a fixed teacher block, LEAP advances the supervisory target through the teacher's feature maps (shallow-to-deep) based on online CKA alignment. This allows the student to build a foundational representation before tackling higher-level abstractions. ### Use cases The ViT models output feature maps that can be used for a variety of downstream tasks, including: - Image Classification - Instance Retrieval - Semantic Segmentation ### Performance #### ImageNet-100:    #### ImageNet-1K:    ### Citation ```bibtex @article{leap2026, title={LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation}, author={Zhang, Jiaqi and Lee, Ashton and Wong, Anthony and Zou, John and BuGhanem, Sami and Balestriero, Randall}, journal={arXiv preprint arXiv:2...