Llava v1.6 Mistral 7b 4bit RoVQA LoRA v1

llava-v1.6-mistral-7b-4bit-RoVQA-lora-v1

This is a parameter-efficient Low-Rank Adaptation (LoRA) adapter for LLaVA-v1.6-Mistral-7B fine-tuned on Romanian Visual Question Answering (VQA).

It was trained on the GRAI-UNSTPB/Flickr30K-RoQA-v1 dataset using Unsloth and Hugging Face's TRL library.

Model Details

• Base Model: unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit
• Adapter Parameters: LoRA modules injected into the language decoder block (~41M trainable parameters, representing 0.60% of total weights).

Inference & Usage

You can load and run inference on this model either using Unsloth (for 2x faster inference) or standard Hugging Face transformers and peft.

Installation

pip install torch transformers peft pillow datasets
# If you want to use Unsloth for faster inference, install it via:
# pip install unsloth

1. Using Unsloth (Recommended)

from unsloth import FastVisionModel
import torch
from datasets import load_dataset

# Load model and processor
model, processor = FastVisionModel.from_pretrained(
    model_name="GRAI-UNSTPB/llava-v1.6-mistral-7b-4bit-RoVQA-lora-v1",
    load_in_4bit=True
)
FastVisionModel.for_inference(model)

# Load a test sample directly from the Flickr30K-RoQA-v1 dataset
dataset = load_dataset("GRAI-UNSTPB/Flickr30K-RoQA-v1", split="test")
sample = dataset[0]
image = sample["image"]
question = sample["question"]

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": f"Răspunde la întrebare: {question}"}
        ]
    }
]

input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=input_text, return_tensors="pt").to("cuda")

# Generate response
outputs = model.generate(**inputs, max_new_tokens=64)
# Decode only the generated response (excluding the prompt)
prompt_len = inputs.input_ids.shape[1]
response = processor.decode(outputs[0][prompt_len:], skip_special_tokens=True)
print(response)

2. Using Transformers & PEFT

from transformers import LlavaNextForConditionalGeneration, LlavaNextProcessor
from peft import PeftModel
import torch
from PIL import Image

# Load base model and processor
base_model_id = "unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit"
processor = LlavaNextProcessor.from_pretrained(base_model_id)

model = LlavaNextForConditionalGeneration.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, "GRAI-UNSTPB/llava-v1.6-mistral-7b-4bit-RoVQA-lora-v1")

# Inference setup follows the same structure as above...

Evaluation Results

The model was evaluated on the Flickr30K-RoQA test set and on Romanian Image Captioning (zero-shot transfer task without captioning fine-tuning).

Romanian Visual Question Answering (VQA)

Model	BERTScore F1 (%)	ROUGE-L F1 (%)	BLEU
LLaVA-v1.6-Mistral-7B (Base)	69.96	38.30	8.25
LLaVA-v1.6-Mistral-7B + RoVQA LoRA (Ours)	71.43	52.82	26.05

Romanian Image Captioning (Zero-Shot Transfer)

Model	BERTScore F1 (%)	ROUGE-L F1 (%)	BLEU
LLaVA-v1.6-Mistral-7B (Base)	59.90	16.70	2.62
LLaVA-v1.6-Mistral-7B + RoVQA LoRA (Ours)	68.54	41.45	15.38

Citation

If you use this model or adapter in your research, please cite the following paper:

@article{dima2025parameter,
  title={Parameter-Efficient Multimodal Instruction Tuning for Romanian Vision--Language Models},
  author={Dima, George-Andrei and Smădu, Răzvan-Alexandru and Cercel, Dumitru-Clementin},
  journal={arXiv preprint arXiv:2512.14926},
  year={2025}
}