Diffvqa Qwen3 0.6b Multiobjective Regsteps

--- license: mit tags: - medical - chest-xray - visual-question-answering - difference-vqa - mimic-cxr library_name: pytorch --- # Diff-VQA — Qwen3-0.6B visual-prefix decoder (multiobjective-regsteps encoder) Online-trained **Difference Visual Question Answering** head for chest X-rays (MIMIC-CXR / Medical-Diff-VQA). A frozen ViT-L/14 vision encoder produces patch tokens for a current + reference image pair; a **Qwen3-0.6B** decoder, conditioned on those tokens as a visual prefix, generates the answer describing what changed. This is a **slimmed inference checkpoint**: decoder weights + the frozen vision encoder weights + architecture metadata (optimizer/scheduler state stripped). ## Contents (`*_best.pt`, `torch.load(..., weights_only=False)`) | key | description | |-----|-------------| | `state_dict` | `DiffVQAQwen3Head` weights (Qwen3-0.6B LLM + `vis_proj` + `frame_emb`) | | `encoder_state` | frozen ViT-L/14 vision encoder (`objective: multiobjective_caption+regioncontrastive`) | | `decoder` | `Qwen/Qwen3-0.6B` | | `tokenizer_name` | `Qwen/Qwen3-0.6B` | | `vis_dim` | 1024 | | `encoder_name` | `pretrain-multiobjective-regsteps` | | `epoch` / `step` / `best_val` | training position + best validation loss | ## Training - Data: Medical-Diff-VQA `difference` questions over MIMIC-CXR image pairs (~100k train / ~16k val / ~16k test). - Objective: next-token cross-entropy over the answer span (question tokens masked). - Vision encoder is **frozen**; only the decoder (LLM + visual adapter) is trained. ## Notes The bundled `encoder_state` is the exact frozen encoder this decoder was trained against — load both together so the visual features match the space `vis_proj` expects (a mismatched encoder produces degenerate output).

Source context: 0 downloads · 0 likes · Pipeline visual-question-answering · Library pytorch · Repo dmusingu/diffvqa-qwen3-0.6b-multiobjective-regsteps