MiniCPM V 4.6

--- license: apache-2.0 pipeline_tag: image-text-to-text tags: - minicpm-v - multimodal - On-Device Model - lightweight library_name: transformers --- A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone [GitHub](https://github.com/OpenBMB/MiniCPM-o) | [CookBook](https://github.com/OpenSQZ/MiniCPM-V-CookBook) | [Demo](https://huggingface.co/spaces/openbmb/MiniCPM-V-4.6-Demo) | [Feishu (Lark)](https://raw.githubusercontent.com/openbmb/MiniCPM-V/main/assets/feishu_qrcode.png) ## News * [2026.05.17] ⭐️⭐️⭐️ We release the API service of MiniCPM-V 4.6, with a **public free API key** together! Try [it](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/api.md) now. ## MiniCPM-V 4.6 **MiniCPM-V 4.6** is our most edge-deployment-friendly model to date. The model is built based on SigLIP2-400M and the Qwen3.5-0.8B LLM. It inherits the strong single-image, multi-image, and video understanding capabilities of MiniCPM-V family, while significantly improving computation efficiency. It also introduces mixed 4x/16x visual token compression. Notable features of MiniCPM-V 4.6 include: - 🔥 **Leading Foundation Capability.** MiniCPM-V 4.6 scores 13 on the Artificial Analysis Intelligence Index benchmark, outperforming Qwen3.5-0.8B's score of 10 with 19x fewer token cost, and Qwen3.5-0.8B-Thinking's score of 11 with 43x fewer token cost. It also surpasses the larger Ministral 3 3B (score of 11). - 💪 **Strong Multimodal Capability.** MiniCPM-V 4.6 outperforms Qwen3.5-0.8B on most vision-language understanding tasks, and reaches Qwen3.5 2B-level capability on many benchmarks including OpenCompass, RefCOCO, HallusionBench, MUIRBench, and OCRBench. - 🚀 **Ultra-Efficient Architecture.** Based on the latest technique in [LLaVA-UHD v4](https://github.com/THUMAI-Lab/LLaVA-UHD-v4), MiniCPM-V 4.6 reduces the visual encoding computation FLOPs by more than 50%. It enables MiniCPM-V 4.6 to achieve better efficiency to even smaller models, achieving ~1.5x token throughput compared to Qwen3.5-0.8B. It also supports mixed 4x/16x visual token compression rate, allowing flexible switching between accuracy and speed. - 📱 **Broad Mobile Platform Coverage.** MiniCPM-V 4.6 can be deployed across all three mainstream mobile platforms — iOS, Android, and HarmonyOS. With every edge adaptation code open-sourced, developers can reproduce the on-device experience in [just a few steps](#deploy-minicpm-v-46-on-ios-android-and-harmonyos-platforms). - 🛠️ **Developer Friendly.** MiniCPM-V 4.6 is adapted to [inference frameworks](#inference-and-training) such as vLLM, SGLang, llama.cpp, Ollama, and supports [fine-tuning ecosystems](#inference-and-training) such as SWIFT and LLaMA-Factory. Developers can quickly customize models for new domains and tasks on consumer-grade GPUs. We provide multiple quantized variants across GGUF, BNB, AWQ, and GPTQ formats. ### Evaluation **Overall Performance (Instruct)** Click to view MiniCPM-V 4.6-Thinking performance. **High-Concurrency Throughput** **Single Request TTFT (ms)** ### Examples #### Overall MiniCPM-V 4.6 can be deployed across three mainstream end-side platforms — **iOS, Android and HarmonyOS**. The clips below are raw screen recordings on phone devices without edition. iPhone iPhone 17 Pro Max Android Redmi K70 HarmonyOS HUAWEI nova 14 ### Usages #### Inference with Transformers ##### Installation ```bash pip install "transformers[torch]>=5.7.0" torchvision torchcodec ``` > **Note on CUDA compatibility:** `torchcodec` (used for video decoding) may have compatibility issues with certain CUDA versions. For example, `torch>=2.11` bundles CUDA 13.1 by default, while environments with CUDA 12.x may encounter errors such as `RuntimeError: Could not load libtorchcodec`. Two workarounds: > > 1. **Replace `torchcodec` with `PyAV`** — supports both image and video inference without CUDA version constraints: > ```bash > pip install "transformers[torch]>=5.7.0" torchvision av > ``` > 2. **Pin the CUDA version** when installing torch to match your environment (e.g. CUDA 12.8): > ```bash > pip install "transformers>=5.7.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128 > ``` ##### Load Model ```python from transformers import AutoModelForImageTextToText, AutoProcessor model_id = "openbmb/MiniCPM-V-4.6" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForImageTextToText.from_pretrained( model_id, torch_dtype="auto", device_map="auto" ) # Flash Attention 2 is recommended for better acceleration and memory saving, # especially in multi-image and video scenarios. # model = AutoModelForImageTextToText.from_pretrained( # model_id, # torch_dtype=torch.bfloat16, # attn_implementation="flash_attention_2", # device_map="auto", # ) ``` ##### Image Inference ```python messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}, {"type": "text", "text": "What causes this phenomenon?"}, ], } ] downsample_mode = "16x" # Using `downsample_mode="4x"` for Finer Detail inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", downsample_mode=downsample_mode, max_slice_nums=36, ).to(model.device) generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=512) generated_ids_trimmed = [ out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) ] output_text = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False ) print(output_text[0]) ``` ##### Video Inference ```python messages = [ { "role": "user", "content": [ {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"}, {"type": "text", "text": "Describe this video in detail. Follow the timeline and focus on on-screen text, interface changes, main actions, and scene changes."}, ], } ] downsample_mode = "16x" # Using `downsample_mode="4x"` for Finer Detail inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", downsample_mode=downsample_mode, max_num_frames=128, stack_frames=1, max_slice_nums=1, use_image_id=False, ).to(model.device) generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048) generated_ids_trimmed = [ out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) ] output_text = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False ) print(output_text[0]) ``` ##### Advanced Parameters You can customize image/video processing by passing additional parameters to `apply_chat_template`: | Parameter | Default | Applies to | Description | |-----------|---------|------------|-------------| | `downsample_mode` | `"16x"` | Image & Video | Visual token downsampling. `"16x"` merges tokens for efficiency; `"4x"` keeps 4× more tokens for finer detail. Must also be passed to `generate()`. | | `max_slice_nums` | `9` | Image & Video | Maximum number of slices when splitting a high-resolution image. Higher values preserve more detail for large images. Recommended: `36` for image, `1` for video. | | `max_num_frames` | `128` | Video only | The `max_num_frames` parameter dynamically controls the temporal context length and prevents VRAM overflow: **Short Videos** (duration ≤ `max_num_frames` sec): The processor defaults to **1 FPS**, capturing second-by-second details without hitting the upper limit. **Long Videos** (duration > `max_num_frames` sec): The processor automatically switches to **uniform sampling**, selecting exactly `max_num_frames` evenly spaced across the entire timeline. | | `stack_frames` | `1` | Video only | Total sample points per second. `1` = main frame only (no stacking). `N` (N>1) = 1 main frame + N−1 sub-frames per second; the sub-frames are composited into a grid image and interleaved with main frames. Recommended setting is `1` for short videos, and `3` or `5` for long videos. | | `use_image_id` | `True` | Image & Video | Whether to prepend ` N ` tags before each image/frame placeholder. Set `True` for image, `False` for video. | > **Note:** `downsample_mode` must be passed to **both** `apply_chat_template` (for correct placeholder count) and `generate` (for the vision encoder). All other parameters only need to be passed to `apply_chat_template`. ##### Serving with `transformers serve` Hugging Face Transformers includes a lightweight OpenAI-compatible server for quick testing and moderate-load deployment. ```bash pip install "transformers[serving]>=5.7.0" ``` Start the server: ```bash transformers serve openbmb/MiniCPM-V-4.6 --port 8000 --host 0.0.0.0 --continuous-batching ``` Send a request: ```bash curl -s http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "openbmb/MiniCPM-V-4.6", "messages": [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}}, {"type": "text", "text": "What causes this phenomenon?"} ] }] }' ``` Tool calling example: ```bash curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "openbmb/MiniCPM-V-4.6", "messages": [{"role": "user", "content": [ {"type": "text", "text": "the weather of Beijing"} ]}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a given location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"} }, "required": ["location"] } } }] }' ``` The model returns a natural-language explanation followed by a structured block embedded in the content field. Note that a dedicated tool call parser for this format has not yet been added to the transformers library, so the tool calls need to be extracted manually via regex for now. ``` { "id": "f4f09c7d-8045-4cb1-ade9-07aa5dee637d", "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "I need to check the current weather for Beijing, so I will call the get_weather function.\n\n \n \n \nBeijing\n \n \n ", "role": "assistant" } } ], "created": 1778748859, "model": "openbmb/MiniCPM-V-4.6@main", "object": "chat.completion", "usage": { "completion_tokens": 47, "prompt_tokens": 283, "total_tokens": 330 } } ``` #### Handling Escaped Newlines in Model Outputs In some cases, the model might output escaped newline characters `\n` as string literals instead of actual newlines. To render the text correctly, especially in UI layers, you can use the following utility function. This function carefully replaces literal `\n` with real newlines while protecting scenarios where `\n` has specific semantic meaning. **Utility Function:** ```python import re _PATTERN = re.compile( r'(```[\s\S]*?```' # fenced code blocks r'|`[^`]+`' # inline code r'|\$\$[\s\S]*?\$\$' # display math r'|\$[^$]+\$' # inline math r'|\\\([\s\S]*?\\\)' # \(...\) r'|\\\[[\s\S]*?\\\]' # \[...\] r')' r'|(? str: """ Lightweight post-processing: Converts literal '\\n' to actual newlines, while protecting code blocks, inline code, and LaTeX commands. """ if not isinstance(text, str) or "\\" not in text: return text return _PATTERN.sub(lambda m: m.group(1) or '\n', text) ``` #### Deploy MiniCPM-V 4.6 on iOS, Android, and HarmonyOS Platforms We have adapted MiniCPM-V 4.6 for deployment on **iOS, Android, and HarmonyOS** platforms, with **all edge adaptation code fully open-sourced**. Developers can reproduce the on-device experience in just a few steps. Visit our [edge deployment repository](https://github.com/OpenBMB/MiniCPM-V-edge-demo) for platform-specific build guides, or go to the [download...