Vietnamese Traffic Sign Vqa Checkpoints

--- language: - vi license: apache-2.0 tags: - visual-question-answering - vietnamese - traffic-signs - lora - qwen2-vl - stable-diffusion --- # Vietnamese Traffic Sign VQA — Checkpoints Model checkpoints for the Deep Learning course final project. ## Contents | Path | Description | Size | |---|---|---| | `a1_lstm/best.pt` | A1: CLIP + PhoBERT + Co-Attention + **LSTM** decoder | ~2.4 GB | | `a2_transformer/best.pt` | A2: CLIP + PhoBERT + Co-Attention + **Transformer** decoder | ~2.5 GB | | `b2_sft_lora/` | B2-SFT: Qwen2.5-VL-3B **QLoRA** adapter (best checkpoint) | ~153 MB | | `b2_dpo_lora/` | B2-DPO: Qwen2.5-VL-3B DPO-aligned adapter (500 pairs, balanced) | ~153 MB | | `stable_diffusion_lora/` | SD LoRA: Stable Diffusion v1.5 LoRA on traffic sign images (12k steps) | ~6 MB | ## Results (Full Test Set) | Model | VQA Acc | BLEU-4 | ROUGE-L | BERTScore | Latency | |---|---:|---:|---:|---:|---:| | A1 LSTM | **0.9484** | **0.9602** | **0.9584** | 0.9631 | ~11 ms | | A2 Transformer | 0.9377 | 0.9476 | 0.9486 | **0.9713** | ~13 ms | | B1 Zero-shot | 0.1962 | 0.0350 | 0.2899 | 0.4753 | ~167 ms | | B2-SFT QLoRA | 0.9379 | 0.9494 | 0.9508 | 0.9111 | ~484 ms | Human evaluation (100 natural questions): B2-DPO **73%** > B2-SFT 70% > A1 61% ## Dataset See [`Anakonkai/vietnamese-traffic-sign-vqa`](https://huggingface.co/datasets/Anakonkai/vietnamese-traffic-sign-vqa) ## Code GitHub: https://github.com/Anakonkai/SDA ## Usage ### A1 / A2 ```python import torch from models.model_a import VQAModelA model = VQAModelA(decoder_type="lstm") # or "transformer" ckpt = torch.load("a1_lstm/best.pt", map_location="cpu") model.load_state_dict(ckpt["model_state_dict"]) ``` ### B2-SFT ```bash python evaluate/evaluate.py \ --model b2 --backend qwen25 --load-in-4bit \ --checkpoint b2_sft_lora/ \ --data data/processed/annotations ``` ### Stable Diffusion LoRA ```python from diffusers import StableDiffusionPipeline pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5") pipe.load_lora_weights("stable_diffusion_lora/") image = pipe("a Vietnamese street intersection with traffic signs").images[0] ```

Source context: 0 downloads · 0 likes · Pipeline visual-question-answering · Repo Anakonkai/vietnamese-traffic-sign-vqa-checkpoints