Lejepa Vits16 Pretrain In1k

Method

This model is trained with LeJEPA: a heuristic free, negative free and teacher free SSL method. It combines SIGReg (Sketched Isotropic Gaussian Regularisation) promoting Gaussian distributed latent embeddings with an Invariance loss to preserve discriminative structure.

The backbone follows the ViT-v2 design from DINOv2 — without register tokens.

The main branch of this repository exposes the model trained for 300 epochs. please refer to other revisions for other models.

Training

The exact config used for this run is available at configs/{dino,ibot,lejepa}/vit_{s,b}16_ep{100,300}.json in the code repository.

Evaluation

Metrics are only available for the teacher weights.

Grouped by method, DINO first since you have results for it. Let me know if you'd prefer a different ordering (e.g. by architecture, or alphabetical by method).

Online probe results are logged during pre-training. Linear probe, k-NN, and downstream evaluations are coming soon.

Checkpoints

from transformers import AutoModel

# default checkpoint (teacher checkpoint after 300 epochs)
objective = "dino"  # lejepa, ibot
model_size = "s"  # b
pretrain_dataset = "in1k"  # currently only in1k is planned unless compute can be expanded.

hf_model_string = f"OK-AI/{objective}-vit{model_size}16-pretrain-{pretrain_dataset}"
model = AutoModel.from_pretrained(hf_model_string)

# alternate training checkpoints
epoch_variant = 100  # 300
state_dict_of = "student"  # `teacher` is default for dino and ibot, lejepa only has student.
model = AutoModel.from_pretrained(
    hf_model_string,
    revision=f"ep{epoch_variant}/{state_dict_of}",
)

Usage

import requests

import torch

from PIL import Image
from transformers import AutoModel, AutoImageProcessor

model_id = "OK-AI/dino-vits16-pretrain-in1k"

processor = AutoImageProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, torch_dtype=torch.bfloat16)
model.eval()

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# {
#     "latent": cls_tokens[:, 0],
#     "patch_latent": patch_tokens,
#     "raw_latent": x[:, 0],
#     "last_self_attention": attn,
#     "logits": self.head(cls_tokens[:, 0]),  # only exists for comptability, head is always identity in this case.
# }

# CLS token — use for classification, retrieval, k-NN
cls = outputs["latent"]        # (1, 384)

# Patch tokens — use for dense tasks (depth, segmentation)
patches = outputs["patch_latent"]   # (1, 196, 384)

Citation

@article{Sablayrolles2018Jun,
	author = {Sablayrolles, Alexandre and Douze, Matthijs and Schmid, Cordelia and J{\ifmmode\acute{e}\else\'{e}\fi}gou, Herv{\ifmmode\acute{e}\else\'{e}\fi}},
	title = {{Spreading vectors for similarity search}},
	year = {2018},
	month = jun,
	doi = {10.48550/arXiv.1806.03198}
}

@article{Caron2021Apr,
	author = {Caron, Mathilde and Touvron, Hugo and Misra, Ishan and J{\ifmmode\acute{e}\else\'{e}\fi}gou, Herv{\ifmmode\acute{e}\else\'{e}\fi} and Mairal, Julien and Bojanowski, Piotr and Joulin, Armand},
	title = {{Emerging Properties in Self-Supervised Vision Transformers}},
	year = {2021},
	month = apr,
	doi = {10.48550/arXiv.2104.14294}
}

@article{Zhou2021Nov,
	author = {Zhou, Jinghao and Wei, Chen and Wang, Huiyu and Shen, Wei and Xie, Cihang and Yuille, Alan and Kong, Tao},
	title = {{iBOT: Image BERT Pre-Training with Online Tokenizer}},
	year = {2021},
	month = nov,
	doi = {10.48550/arXiv.2111.07832}
}

@article{Dong2022Dec,
	author = {Dong, Xiaoyi and Bao, Jianmin and Zhang, Ting and Chen, Dongdong and Gu, Shuyang and Zhang, Weiming and Yuan, Lu and Chen, Dong and Wen, Fang and Yu, Nenghai},
	title = {{CLIP Itself is a Strong Fine-tuner: Achieving 85.7{\%} and 88.0{\%} Top-1 Accuracy with ViT-B and ViT-L on ImageNet}},
	year = {2022},
	month = dec,
	doi = {10.48550/arXiv.2212.06138}
}

@article{Oquab2023Apr,
	author = {Oquab, Maxime and Darcet, Timoth{\ifmmode\acute{e}\else\'{e}\fi}e and Moutakanni, Th{\ifmmode\acute{e}\else\'{e}\fi}o and Vo, Huy and Szafraniec, Marc and Khalidov, Vasil and Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and El-Nouby, Alaaeldin and Assran, Mahmoud and Ballas, Nicolas and Galuba, Wojciech and Howes, Russell and Huang, Po-Yao and Li, Shang-Wen and Misra, Ishan and Rabbat, Michael and Sharma, Vasu and Synnaeve, Gabriel and Xu, Hu and Jegou, Herv{\ifmmode\acute{e}\else\'{e}\fi} and Mairal, Julien and Labatut, Patrick and Joulin, Armand and Bojanowski, Piotr},
	title = {{DINOv2: Learning Robust Visual Features without Supervision}},
	year = {2023},
	month = apr,
	doi = {10.48550/arXiv.2304.07193}
}

@article{Balestriero2025Nov,
	author = {Balestriero, Randall and LeCun, Yann},
	title = {{LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics}},
	year = {2025},
	month = nov,
	doi = {10.48550/arXiv.2511.08544}
}