ClipL336 TTR

--- library_name: transformers license: mit pipeline_tag: image-feature-extraction tags: - clip --- # OpenCLIP ViT-L/14 with Test-Time Register Register tokens in ViTs were introduced as learnable tokens in [Vision Transformers Need Registers](https://arxiv.org/abs/2309.16588) to mitigate artifacts in intermediate feature maps. In [Vision Transformers Don't Need *Trained* Registers](https://arxiv.org/abs/2506.08010), we introduced a training-free method to create registers. These *test-time registers* serve a similar purpose as the original trained registers, but can be added post-hoc to any ViT to mitigate artifacts, enhance model interpretability, and modestly improve downstream performance in tasks such as segmentation, depth estimation, etc. ## Model description The base model is [OpenCLIP-ViT-L-14-laion2B-s32B-b82K](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K). With test-time registers, the model's internal representations are cleaner (see below). Using the environment from [here](https://github.com/nickjiang2378/test-time-registers/blob/main/environment.yml) and evaluating using bfloat16 leads to IN-1k zeroshot performance of 76.4 for both the original model and the variant with test-time registers. This model is intended to be used with this [repo](https://github.com/nickjiang2378/test-time-registers). Use transformers==4.45.1. The model can also be used for fine-tuning or other downstream tasks. <img src="https://huggingface.co/amildravid4292/clip-vitl14-test-time-registers/resolve/main/vitl14_attention.png" alt="drawing" width="600"/> <img src="https://huggingface.co/amildravid4292/clip-vitl14-test-time-registers/resolve/main/vitl14_patchnorms.png" alt="drawing" width="600"/> ## Quick Start ```python from transformers import AutoModel from PIL import Image import torch # Load the complete model with all components model = AutoModel.from_pretrained( "amildravid4292/clip-vitl14-test-time-registers", trust_remote_code=True ) # Check what was loaded print(f"Register tokens: {model.num_register_tokens}") print(f"Neuron dict: {model.neuron_dict}") print(f"Tokenizer available: {model.tokenizer is not None}") print(f"Preprocessor available: {model.preprocessor is not None}") print(f"Zero-shot classifier available: {model.zeroshot_classifier is not None}") ``` ## Usage Examples ### Image Processing ```python from PIL import I...

Source context: 14 downloads · 0 likes · Pipeline image-feature-extraction · Library transformers · Repo JH-C-k/clipL336_TTR