Loading
Have a ComfyUI workflow? Get a readable Workflow Doc in your browser.Make a workflow readable.Try it now ↗
This model is a fine-tuned version of Salesforce/blip-image-captioning-base, adapted for captioning UI elements from macOS application screenshots.
Model source
Source description
This model is a fine-tuned version of Salesforce/blip-image-captioning-base, adapted for captioning UI elements from macOS application screenshots.
It is part of the Screen2AX research project focused on improving accessibility using vision-based deep learning.
Sources
1 sourceVerified Sep 17
Model artifacts
1 artifactSource excerpts
2 excerptsThe model takes an image of a UI icon or element and generates a natural language description (e.g., "Settings icon", "Play button", "Search field").
This helps build assistive technologies such as screen readers by providing textual labels for unlabeled visual components.
Salesforce/blip-image-captioning-baseimage-to-textfrom transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import requests
processor = BlipProcessor.from_pretrained("macpaw-research/blip-icon-captioning")
model = BlipForConditionalGeneration.from_pretrained("macpaw-research/blip-icon-captioning")
image = Image.open("path/to/ui_icon.png")
inputs = processor(images=image, return_tensors="pt")
output = model.generate(**inputs)
caption = processor.decode(output[0], skip_special_tokens=True)
print(caption)
# Example: "Settings icon"
This model is released under the MIT License.
If you use this model in your research, please cite the Screen2AX paper:
@misc{muryn2025screen2axvisionbasedapproachautomatic,
title={Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation},
author={Viktor Muryn and Marta Sumyk and Mariya Hirna and Sofiya Garkot and Maksym Shamrai},
year={2025},
eprint={2507.16704},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2507.16704},
}
Learn more at https://research.macpaw.com
--- language: - en base_model: - Salesforce/blip-image-captioning-base pipeline_tag: image-to-text tags: - blip - icon-description - image-captioning license: mit library_name: transformers --- # 🧠 BLIP — UI Elements Captioning This model is a fine-tuned version of [`Salesforce/blip-image-captioning-base`](https://huggingface.co/Salesforce/blip-image-captioning-base), adapted for **captioning UI elements** from macOS application screenshots. It is part of the **Screen2AX** research project focused on improving accessibility using vision-based deep learning. --- ## 🎯 Use Case The model takes an image of a **UI icon or element** and generates a **natural language description** (e.g., `"Settings icon"`, `"Play button"`, `"Search field"`). This helps build assistive technologies such as screen readers by providing textual labels for unlabeled visual components. --- ## 🏗 Model Architecture - Base model: [`Salesforce/blip-image-captioning-base`](https://huggingface.co/Salesforce/blip-image-captioning-base) - Architecture: **BLIP** (Bootstrapping Language-Image Pre-training) - Task: `image-to-text` --- ## 🖼 Example ```python from transformers import BlipProcessor, BlipForConditionalGeneration from PIL import Image import requests processor = BlipProcessor.from_pretrained("macpaw-research/blip-icon-captioning") model = BlipForConditionalGeneration.from_pretrained("macpaw-research/blip-icon-captioning") image = Image.open("path/to/ui_icon.png") inputs = processor(images=image, return_tensors="pt") output = model.generate(**inputs) caption = processor.decode(output[0], skip_special_tokens=True) print(caption) # Example: "Settings icon" ``` --- ## 📜 License This model is released under the **MIT License**. --- ## 🔗 Related Projects - [Screen2AX Project](https://github.com/MacPaw/Screen2AX) - [Screen2AX HuggingFace Collection](https://huggingface.co/collections/macpaw-research/screen2ax) --- ## ✍️ Citation If you use this model in your research, please cite the Screen2AX paper: ```bibtex @misc{muryn2025screen2axvisionbasedapproachautomatic, title={Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation}, author={Viktor Muryn and Marta Sumyk and Mariya Hirna and Sofiya Garkot and Maksym Shamrai}, year={2025}, eprint={2507.16704}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2507.167...
Source context: 67 downloads · 1 likes · Pipeline image-to-text · Library transformers · Repo macpaw-research/blip-icon-captioning