Florence2 Icon

--- language: - en license: mit library_name: transformers tags: - vision - image-to-text - custom_code - florence2 - icon-captioning base_model: microsoft/Florence-2-base --- # Florence-2 Icon Captioning `minhvn4/florence2-icon` is a fine-tuned version of [Microsoft's Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) specifically tailored for **Icon Captioning**. This model understands and generates descriptive captions for UI icons, symbols, and pictograms. Because it relies on custom code from the original Florence-2 implementation, you must use `trust_remote_code=True` when loading the model. ## Model Details - **Architecture**: Florence-2 (`AutoModelForCausalLM`) - **Base Model**: [microsoft/Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) - **Task**: Image to Text (Icon Captioning) - **License**: MIT - **Format**: Safetensors ## Usage Here's how to load and use the model for icon captioning in your Python code. Make sure to install the required dependencies (`transformers`, `torch`, `Pillow`, `einops`, `timm`). ```python import torch from PIL import Image from transformers import AutoProcessor, AutoModelForCausalLM # Set model ID model_id = "minhvn4/florence2-icon" # Load the processor and model processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval() # Move model to target device device = "cuda" if torch.cuda.is_available() else "cpu" model.to(device) def generate_caption(image_path, prompt="<CAPTION>"): image = Image.open(image_path).convert("RGB") # Process inputs inputs = processor(text=prompt, images=image, return_tensors="pt").to(device) # Generate text with torch.inference_mode(): generated_ids = model.generate( input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"], max_new_tokens=20, num_beams=1, do_sample=False ) # Decode output caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] return caption.strip() # Run inference on an icon image_path = "path/to/your/icon.png" caption = generate_caption(image_path) print(f"Generated caption: {caption}") ``` ## Intended Use This model is intended to be used for generating descriptive text for single UI icons. It can be integrated into UI parsing tools (like OmniParser), accessibility tools, or web/...

Source context: 8 downloads · 0 likes · Pipeline image-to-text · Library transformers · Repo minhvn4/florence2-icon