Qwen2.5 VL 3B Instruct W4A16 Generic

--- base_model: Qwen/Qwen2.5-VL-3B-Instruct tags: - quantized - w4a16 - robotics - nova-robot pipeline_tag: image-text-to-text language: - en --- # Qwen2.5-VL-3B-Instruct-W4A16-generic Quantized with the NOVA quantization pipeline on 2026-04-27. Base model: [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) ## Quantization details | Parameter | Value | |---|---| | Method | `W4A16` | | Group size | 128 | | Calibration | `generic` | | Ignored modules | `re:.*lm_head, re:.*visual.*` | | Tool | `llm-compressor >= 0.4.2` | ## Benchmark results | Metric | Value | |---|---| | Perplexity (wikitext-2, 20 samples) | 20.466 | | OCR sanity check | ✅ PASS | | Tokens / second | 1.1 | | TTFT (exact, prefill only) | 1094.5 ms | | TPOT (exact, per output token) | 915.9 ms | | Inference VRAM | 9.15 GB | | Disk size | 3.41 GB | > TTFT and TPOT measured with `BaseStreamer` injection (prompt-skip corrected). ## Registry notes - Use llm-compressor==0.5.1 + transformers==4.51.3 for quantization. - Use vLLM>=0.7.2 for inference — Marlin kernels active on Ampere. - Projector (model.visual.merger) kept at FP32 — matched by visual.* regex. - OCR and bbox grounding regress 5x faster than MMMU under aggressive quant. - Keep merger at FP32, not BF16, for best bbox coordinate precision. - W8A8 requires A100 for calibration — activation statistics need >24GB VRAM. ## Usage ```python from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor import torch model = Qwen2_5_VLForConditionalGeneration.from_pretrained( "Mohaaxa/Qwen2.5-VL-3B-Instruct-W4A16-generic", torch_dtype=torch.bfloat16, device_map="auto", ) processor = AutoProcessor.from_pretrained("Mohaaxa/Qwen2.5-VL-3B-Instruct-W4A16-generic") ``` ## Citation If you use this model in research, please cite the NOVA project. Pipeline source: `Mohaaxa/nova-quant-pipeline`