PMC Kvasir VQA X1 LoRA 250918 1352

--- library_name: transformers base_model: chaoyi-wu/PMC_LLAMA_7B license: apache-2.0 tags: - medical - vqa - multimodal - pmc-clip - pmc-llama - kvasir datasets: - SimulaMet/Kvasir-VQA-x1 metrics: - exact_match - f1_score pipeline_tag: visual-question-answering --- # PMC-VLM: Medical Visual Question Answering This model combines **PMC-CLIP** and **PMC-LLaMA** for medical visual question answering, specifically fine-tuned on the Kvasir-VQA-x1 dataset. ## Model Architecture - **Vision Encoder**: PMC-CLIP (frozen) - **Language Model**: PMC-LLaMA-7B with LoRA adapters - **Image Projector**: Linear projection with 4 soft prompt tokens - **Training**: QLoRA (4-bit quantization) fine-tuning ## Training Details - **Dataset**: Kvasir-VQA-x1 (Medical VQA) - **Learning Rate**: 0.0002 - **Batch Size**: 2 (with 4x accumulation) - **LoRA Rank**: 16 - **LoRA Alpha**: 32 - **Epochs**: 1 ## Usage Load the model and run inference for medical VQA tasks.

Source context: 0 downloads · 0 likes · Pipeline visual-question-answering · Library transformers · Repo JustATalentedGuy/PMC-Kvasir-VQA-x1-lora_250918-1352