Skip to content
ModelListedExternal
Browse files

QwigLip VQA

Custom Vision-Question-Answering Model built from scratch. Built upon earlier QwigLip-VLM with additional 5k data from, but with a custom MLP projector and LoRA fine-tuning for efficient training. Training data from...

Sourcehuggingface.co/teohyc/QwigLip-VQA/tree/d634614a3f3c64d686065aff5c64179fca80806a
KindOtherBase modelQwen/Qwen2-0.5B-InstructVersionProjectorCgrade

Model source

Kind
Other
File size
21.0–535 MB
Base model
Qwen/Qwen2-0.5B-Instruct
Version
vProjector
Updated
Oct 10, 2026
Source
Hugging Face

Source description

Qwiglip VQA (Qwen2 + SigLIP)

Custom Vision-Question-Answering Model built from scratch. Built upon earlier QwigLip-VLM with additional 5k data from, but with a custom MLP projector and LoRA fine-tuning for efficient training. Training data from https://huggingface.co/datasets/landersanmi/VQAv2 and https://huggingface.co/datasets/phiyodr/coco2017 Full repository at https://github.com/teohyc/qwiglip_vlm

Components

  • Base LLM: Qwen/Qwen2-0.5B-Instruct
  • Vision Encoder: SigLIP
  • LoRA fine-tuning
  • Custom MLP projector

Usage

***** CHECK OUT inference.py FOR DETAILED INFERENCE EXAMPLE *****

import torch
from PIL import Image
from transformers import AutoTokenizer, AutoProcessor, AutoModel, Qwen2ForCausalLM
from peft import PeftModel

from vlm_model import MLPProjector, SiglipQwenVLM

#configurations
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

LLM_NAME = "Qwen/Qwen2-0.5B-Instruct"
VISION_NAME = "google/siglip-base-patch16-224"

LORA_PATH = "vqa_lora_adapter"
PROJECTOR_PATH = "projector.pt"

NUM_IMAGE_TOKENS = 196

#refer to inference.py for full code

Sources

1 source
Hugging FacePrimaryVersion d634614a3f3c64d686065aff5c64179fca80806a
huggingface.co/teohyc/QwigLip-VQA/tree/d634614a3f3c64d686065aff5c64179fca80806a

Verified Oct 10

Download files

Model artifacts

2 artifacts

projector.pt

pt · 21.0 MB · SHA-256 e5d6f69d44b1…df2d · Hugging Face

Download file

vqa_lora_adapter/adapter_model.safetensors

safetensors · 535 MB · SHA-256 ceedf0cddc9d…2e89 · Hugging Face

Download file

Source excerpts

2 excerpts
ReadmeAug 2huggingface.co/teohyc/QwigLip-VQA

--- license: mit datasets: - phiyodr/coco2017 - landersanmi/VQAv2 language: - en metrics: - accuracy base_model: - Qwen/Qwen2-0.5B-Instruct - google/siglip-base-patch16-224 - teohyc/QwigLip-VLM library_name: transformers pipeline_tag: visual-question-answering --- # Qwiglip VQA (Qwen2 + SigLIP) Custom Vision-Question-Answering Model built from scratch. Built upon earlier QwigLip-VLM with additional 5k data from, but with a custom MLP projector and LoRA fine-tuning for efficient training. Training data from https://huggingface.co/datasets/landersanmi/VQAv2 and https://huggingface.co/datasets/phiyodr/coco2017 Full repository at https://github.com/teohyc/qwiglip_vlm ## Components - Base LLM: Qwen/Qwen2-0.5B-Instruct - Vision Encoder: SigLIP - LoRA fine-tuning - Custom MLP projector ## Usage ***** CHECK OUT inference.py FOR DETAILED INFERENCE EXAMPLE ***** ```python import torch from PIL import Image from transformers import AutoTokenizer, AutoProcessor, AutoModel, Qwen2ForCausalLM from peft import PeftModel from vlm_model import MLPProjector, SiglipQwenVLM #configurations DEVICE = "cuda" if torch.cuda.is_available() else "cpu" LLM_NAME = "Qwen/Qwen2-0.5B-Instruct" VISION_NAME = "google/siglip-base-patch16-224" LORA_PATH = "vqa_lora_adapter" PROJECTOR_PATH = "projector.pt" NUM_IMAGE_TOKENS = 196 #refer to inference.py for full code ```

NoteAug 2huggingface.co/teohyc/QwigLip-VQA

Source context: 0 downloads · 0 likes · Pipeline visual-question-answering · Library transformers · Repo teohyc/QwigLip-VQA