PRS Med

--- license: mit tags: - medical-imaging - segmentation - vision-language-model - llava - tinysam - lora datasets: - huyquoctrinh/PRS-Med language: - en pipeline_tag: image-segmentation --- <h1 align="center">PRS-Med: Position Reasoning Segmentation in Medical Imaging</h1> <p align="center"> <a href="https://arxiv.org/abs/2505.11872"><img src="https://img.shields.io/badge/arXiv-2505.11872-b31b1b.svg" alt="arXiv"></a> <a href="https://huyquoctrinh.github.io/prsmed/"><img src="https://img.shields.io/badge/🌐-Project%20Page-blue.svg" alt="Project Page"></a> <a href="https://github.com/huyquoctrinh/PRS-Med"><img src="https://img.shields.io/badge/GitHub-Code-black.svg" alt="GitHub"></a> <a href="https://huggingface.co/datasets/huyquoctrinh/PRS-Med"><img src="https://img.shields.io/badge/🤗-Dataset-yellow.svg" alt="HuggingFace Dataset"></a> </p> --- This repository contains the pretrained weights for **PRS-Med**, a modular framework for position reasoning segmentation in medical imaging. PRS-Med combines a vision-language model (LLaVA-Med) with a lightweight image encoder (TinySAM) and a custom mask decoder to perform context-aware medical image segmentation. **Accepted at CVPRW 2026.** ## Model Architecture PRS-Med consists of three main components: | Component | Description | Details | |-----------|-------------|---------| | **LLaVA-Med** | Vision-language model for semantic reasoning | Mistral-7B backbone, fine-tuned with LoRA (r=16, alpha=16) | | **TinySAM** | Lightweight image encoder | ViT-Tiny, extracts 256-dim features at 64x64 spatial resolution | | **Prompted Mask Decoder** | Cross-attention fusion + upsampling | Fuses LLM embeddings with image features to produce 1024x1024 masks | | **Classification Head** | 6-class medical modality classifier | Brain, Breast, Lung CT, Lung X-ray, Skin (ISIC), Other | ## Checkpoint Contents ``` checkpoint/ ├── lora_adapter/ # LoRA adapter weights for LLaVA-Med │ ├── adapter_config.json │ ├── adapter_model.safetensors │ ├── tokenizer.json │ ├── tokenizer.model │ ├── tokenizer_config.json │ └── special_tokens_map.json ├── image_encoder.pth # TinySAM image encoder weights ├── mask_decoder.pth # Prompted mask decoder weights └── cls.pth # 6-class classification head weights ``` ## Usage ### 1. Install Dependencies ```bash git clone https://github.com/huyquoctrinh/PRS-Med.git cd PRS-Med pip instal...