Ir para o conteúdo
ModelListadosExterno
Browse files

QwigLip VQA

Custom Vision-Question-Answering Model built from scratch. Built upon earlier QwigLip-VLM with additional 5k data from, but with a custom MLP projector and LoRA fine-tuning for efficient training. Training data from...

Fontehuggingface.co/teohyc/QwigLip-VQA/tree/d634614a3f3c64d686065aff5c64179fca80806a
TipoOtherModelo baseQwen/Qwen2-0.5B-InstructVersãoProjectorCnota

Fonte do modelo

Tipo
Other
Tamanho do arquivo
21,0–535 MB
Modelo base
Qwen/Qwen2-0.5B-Instruct
Versão
vProjector
Atualizado
10 de out. de 2026
Fonte
Hugging Face

Descrição da fonte

Qwiglip VQA (Qwen2 + SigLIP)

Custom Vision-Question-Answering Model built from scratch. Built upon earlier QwigLip-VLM with additional 5k data from, but with a custom MLP projector and LoRA fine-tuning for efficient training. Training data from https://huggingface.co/datasets/landersanmi/VQAv2 and https://huggingface.co/datasets/phiyodr/coco2017 Full repository at https://github.com/teohyc/qwiglip_vlm

Components

  • Base LLM: Qwen/Qwen2-0.5B-Instruct
  • Vision Encoder: SigLIP
  • LoRA fine-tuning
  • Custom MLP projector

Usage

***** CHECK OUT inference.py FOR DETAILED INFERENCE EXAMPLE *****

import torch
from PIL import Image
from transformers import AutoTokenizer, AutoProcessor, AutoModel, Qwen2ForCausalLM
from peft import PeftModel

from vlm_model import MLPProjector, SiglipQwenVLM

#configurations
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

LLM_NAME = "Qwen/Qwen2-0.5B-Instruct"
VISION_NAME = "google/siglip-base-patch16-224"

LORA_PATH = "vqa_lora_adapter"
PROJECTOR_PATH = "projector.pt"

NUM_IMAGE_TOKENS = 196

#refer to inference.py for full code

Fontes

1 fonte
Hugging FacePrimárioVersão d634614a3f3c64d686065aff5c64179fca80806a
huggingface.co/teohyc/QwigLip-VQA/tree/d634614a3f3c64d686065aff5c64179fca80806a

Verificado 10 de out.

Download files

Artefatos de modelo

2 artefatos

projector.pt

pt · 21,0 MB · SHA-256 e5d6f69d44b1…df2d · Hugging Face

Download file

vqa_lora_adapter/adapter_model.safetensors

safetensors · 535 MB · SHA-256 ceedf0cddc9d…2e89 · Hugging Face

Download file

Trechos de fonte

2 trechos
Readme2 de ago.huggingface.co/teohyc/QwigLip-VQA

--- license: mit datasets: - phiyodr/coco2017 - landersanmi/VQAv2 language: - en metrics: - accuracy base_model: - Qwen/Qwen2-0.5B-Instruct - google/siglip-base-patch16-224 - teohyc/QwigLip-VLM library_name: transformers pipeline_tag: visual-question-answering --- # Qwiglip VQA (Qwen2 + SigLIP) Custom Vision-Question-Answering Model built from scratch. Built upon earlier QwigLip-VLM with additional 5k data from, but with a custom MLP projector and LoRA fine-tuning for efficient training. Training data from https://huggingface.co/datasets/landersanmi/VQAv2 and https://huggingface.co/datasets/phiyodr/coco2017 Full repository at https://github.com/teohyc/qwiglip_vlm ## Components - Base LLM: Qwen/Qwen2-0.5B-Instruct - Vision Encoder: SigLIP - LoRA fine-tuning - Custom MLP projector ## Usage ***** CHECK OUT inference.py FOR DETAILED INFERENCE EXAMPLE ***** ```python import torch from PIL import Image from transformers import AutoTokenizer, AutoProcessor, AutoModel, Qwen2ForCausalLM from peft import PeftModel from vlm_model import MLPProjector, SiglipQwenVLM #configurations DEVICE = "cuda" if torch.cuda.is_available() else "cpu" LLM_NAME = "Qwen/Qwen2-0.5B-Instruct" VISION_NAME = "google/siglip-base-patch16-224" LORA_PATH = "vqa_lora_adapter" PROJECTOR_PATH = "projector.pt" NUM_IMAGE_TOKENS = 196 #refer to inference.py for full code ```

Note2 de ago.huggingface.co/teohyc/QwigLip-VQA

Source context: 0 downloads · 0 likes · Pipeline visual-question-answering · Library transformers · Repo teohyc/QwigLip-VQA