Rope Vit Reg4 B14 Capi

--- tags: - image-feature-extraction - birder - pytorch library_name: birder license: apache-2.0 --- # Model Card for rope_vit_reg4_b14_capi A RoPE ViT b14 image encoder pre-trained using CAPI. This model has *not* been fine-tuned for a specific classification task and is intended to be used as a general-purpose feature extractor or a backbone for downstream tasks like object detection, segmentation, or custom classification. ## Model Details - **Model Type:** Image classification and detection backbone - **Model Stats:** - Params (M): 85.7 - Input image size: 224 x 224 - **Dataset:** Trained on a diverse dataset of approximately 15M images, including: - iNaturalist 2021 (~2.6M) - imagenet-w21-webp-wds (~1.6M random subset) - WebVision-2.0 (~1.5M random subset) - GLDv2 (~820K random subset of 100 chunks) - SA-1B (~220K random subset of 20 chunks) - COCO (~120K) - NABirds (~48K) - Birdsnap v1.1 (~44K) - CUB-200 2011 (~6K) - The Birder dataset (~7M, private dataset) - **Papers:** - An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: <https://arxiv.org/abs/2010.11929> - Rotary Position Embedding for Vision Transformer: <https://arxiv.org/abs/2403.13298> - Vision Transformers Need Registers: <https://arxiv.org/abs/2309.16588> - Cluster and Predict Latent Patches for Improved Masked Image Modeling: <https://arxiv.org/abs/2502.08769> ## Model Usage ### Image Embeddings ```python import birder from birder.inference.classification import infer_image (net, model_info) = birder.load_pretrained_model("rope_vit_reg4_b14_capi", inference=True) # Get the image size the model was trained on size = birder.get_size_from_signature(model_info.signature) # Create an inference transform transform = birder.classification_transform(size, model_info.rgb_stats) image = "path/to/image.jpeg" # or a PIL image (out, embedding) = infer_image(net, image, transform, return_embedding=True) # embedding is a NumPy array with shape of (1, 768) ``` ### Detection Feature Map ```python from PIL import Image import birder (net, model_info) = birder.load_pretrained_model("rope_vit_reg4_b14_capi", inference=True) # Get the image size the model was trained on size = birder.get_size_from_signature(model_info.signature) # Create an inference transform transform = birder.classification_transform(size, model_info.rgb_stats) image = Image.open("...