--- license: apache-2.0 tags: - vision-language - video-to-text - image-to-text - qwen3-vl - awq - compressed-tensors - vllm pipeline_tag: video-text-to-text --- # JoyAI-VL-Interaction-Preview AWQ W4A16 (LLM 4bit, Vision BF16) This repository contains an AWQ W4A16 quantized version of `jdopensource/JoyAI-VL-Interaction-Preview`. Original model: - ModelScope/Hugging Face name: `jdopensource/JoyAI-VL-Interaction-Preview` - Architecture: `Qwen3VLForConditionalGeneration` - Model type: `qwen3_vl` - Project: https://github.com/jd-opensource/JoyAI-VL-Interaction - Paper: https://arxiv.org/abs/2606.14777 This quantized checkpoint is intended for local vLLM inference experiments. It is not an official release from JD. ## Quantization Quantization method: - Tool: `llmcompressor==0.12.0` - Runtime format: `compressed-tensors==0.17.1` - Scheme: `W4A16_ASYM` - Weight bits: 4 - Group size: 128 - Targets: language-model `Linear` layers - Ignored modules: `lm_head`, `model.visual.*` The visual encoder is kept in bf16 to reduce the risk of visual-understanding degradation. This means the checkpoint is not a fully quantized VLM; only the language-model linear layers are quantized. Observed local size: | Checkpoint | Size | |---|---:| | Original bf16 checkpoint | ~17 GB | | AWQ W4A16 checkpoint | ~6.8 GB | ## Verified Environment The checkpoint was tested with: ```text Python 3.10.20 vllm 0.23.0 llmcompressor 0.12.0 compressed-tensors 0.17.1 transformers 5.10.1 2 x RTX 3080 20GB ``` vLLM detected: ```text quantization=compressed-tensors Using MarlinLinearKernel for CompressedTensorsWNA16 ``` ## vLLM Usage Start the quantized model backend: ```bash vllm serve /path/to/JoyAI-VL-Interaction-Preview-AWQ-W4A16 \ --served-model-name JoyAI-VL-Interaction-Preview-AWQ-W4A16 \ --host 127.0.0.1 \ --port 8061 \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --enable-prefix-caching \ --gpu-memory-utilization 0.85 \ --limit-mm-per-prompt '{"image":64,"video":1}' ``` For lower memory usage, reduce context and multimodal limits: ```bash vllm serve /path/to/JoyAI-VL-Interaction-Preview-AWQ-W4A16 \ --served-model-name JoyAI-VL-Interaction-Preview-AWQ-W4A16 \ --host 127.0.0.1 \ --port 8061 \ --tensor-parallel-size 2 \ --max-model-len 4096 \ --max-num-seqs 1 \ --max-num-batched-tokens 2048 \ --enforce-eager \ --gpu-memory-utilization 0.50 \ --limit-mm-per-prompt '{"image":1,"video":1}' ``` ## vLLM-Omni Real-Time Interaction This model can be used as the main model backend for the JoyAI real-time interaction server from vLLM-Omni: ```bash PYTHONPATH=/path/to/vllm-omni python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \ --host 127.0.0.1 \ --port 8070 \ --main-backend-url http://127.0.0.1:8061/v1 \ --main-model JoyAI-VL-Interaction-Preview-AWQ-W4A16 \ --chunk-frames 16 ``` The interaction server exposes an OpenAI-compatible endpoint: ```text POST http://127.0.0.1:8070/v1/chat/completions ``` ## Chinese Output The model can output Chinese. For real-time video interaction, set the instruction explicitly, for example: ```text 请始终只用简体中文输出,实时简短描述视频中的重要变化;没有重要变化时保持沉默。 ``` Without an explicit Chinese instruction, the model may output English by default. ## Minimal Test ```bash curl http://127.0.0.1:8061/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "JoyAI-VL-Interaction-Preview-AWQ-W4A16", "messages": [ {"role": "user", "content": "用一句中文说明你已经启动成功。"} ], "max_tokens": 64, "temperature": 0 }' ``` Expected behavior: the model returns a short Chinese response. ## Limitations - This is an experimental quantized checkpoint. - The visual encoder is not quantized. - AWQ calibration used text prompts, so this release prioritizes preserving language-model behavior while keeping the vision tower unchanged. - Running on 8 GB edge devices such as Jetson Orin Nano Super is not guaranteed. Checkpoint size is not equal to runtime memory usage. - For real-time video, reduce frame sampling rate and image size according to available throughput. ## License This quantized checkpoint follows the license of the original model release: Apache-2.0. Please also follow the license and usage terms of the original JoyAI-VL-Interaction project.
