Deepseek Ai DeepSeek V4 Flash GGUF YMMV

--- base_model: deepseek-ai/DeepSeek-V4-Flash base_model_relation: quantized library_name: llama.cpp license: mit pipeline_tag: text-generation tags: - gguf - llama.cpp - deepseek_v4 - text-generation - conversational - en - license:mit - endpoints_compatible - region:us - base_model:deepseek-ai/DeepSeek-V4-Flash - base_model:quantized:deepseek-ai/DeepSeek-V4-Flash - endpoints_compatible - region:us - ds4 --- > 🌤️ **Update: 2026/05/09 - Core functionality good - dogfooding/testing/refining. ** Focused on real, long running sessions now. Uploaded fresh GGUF. See punch list below for progress. Bugs w/ agent diagnoses welcome. ** ## 🧪 Experimental llama.cpp fork and GGUFs for DeepSeek-V4-Flash A stopgap to experiment with DeepSeek-V4-Flash with CUDA and ROCm locally while the tools ecosystem catches up. Expect rough edges. Priority is validating for text and coding coherence. GGUF files for [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash). ### ⚠️ You need the custom fork These GGUFs **require** a DeepSeek-V4-capable fork of llama.cpp. Vanilla llama.cpp doesn't support this architecture yet. - **llama.cpp fork:** [ssweens/llama.cpp-deepseek-v4](https://github.com/ssweens/llama.cpp-deepseek-v4) - **Backends:** Tested on CPU, CUDA, ROCm and Vulkan. - **Compatability:** Also compatible with Antirez's ggufs [antirez/deepseek-v4-gguf](https://huggingface.co/antirez/deepseek-v4-gguf) ### Example: ``` llama-server -ngl 99 --no-mmap -fa on -np 1 --reasoning-format auto --jinja --threads 3 -ts 4,4,3 -dev CUDA0,CUDA1,CUDA2 \ -m /mnt/supmodels/gguf/deepseek-ai__DeepSeek-V4-Flash/deepseek-ai__DeepSeek-V4-Flash-Q4_K_M.gguf -c 65536 -b 2048 -ub 512 -ctk q8_0 -ctv q8_0 --chat-template-kwargs '{"reasoning_effort": "high"}' --reasoning on --reasoning-budget 1024 --reasoning-budget-message "... thinking budget exceeded, let's answer now." ```` ## Performance #### Basic Coding coherence (humaneval_instruct, n=30, thinking "high" capped at 1024) | Model | pass@1 | | ----- | ------ | | IQ1_M | 1.000±0.000 | | IQ2_XXS | 1.000±0.000 | | IQ2_XXS (Antirez) | 1.000±0.000 | | BF16ish | 1.000±0.000 | #### Deeper Coding (LiveCodeBench sample, n=30, thinking "high" capped at 1024) | Model | Bench | pass@1 | Easy | Medium | Hard | |---|---|---|---|---|---| | IQ2_XXS | `v5` | 76.7% | 100% | 60% | 70% | | IQ2_XXS | `v6 only` | 73.3% | 100% | 70% | 50% | #### Speed (llama-benchy, defaults, thinking "high" capped at 1024) >Note: models up to IQ2_XS are CUDA, pipeline parallel mix of consumer RTX. >IQ2_antirez and bigger are CUDA+ROCm, pipeline parallel mix with Strix Halo mixed in. | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:--------------------------------------|-------:|--------------:|-------------:|----------------:|----------------:|----------------:| | deepseek-ai/DeepSeek-V4-Flash-IQ2_XXS | pp2048 | 358.44 ± 2.05 | | 5714.56 ± 32.61 | 5713.91 ± 32.61 | 5714.56 ± 32.61 | | deepseek-ai/DeepSeek-V4-Flash-IQ2_XXS | tg32 | 30.62 ± 0.28 | 31.00 ± 0.00 | | | | | deepseek-ai/DeepSeek-V4-Flash-IQ2_XXS | pp32768 | 249.63 ± 0.79 | | 131244.94 ± 413.90 | 131244.26 ± 413.90 | 131244.94 ± 413.90 | | deepseek-ai/DeepSeek-V4-Flash-IQ2_XXS | tg32 | 24.54 ± 0.23 | 25.00 ± 0.00 | | | | ##### ROCm Only | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-------------------------------------------|-------:|-------------:|------------:|------------------:|------------------:|------------------:| | deepseek-ai/DeepSeek-V4-Flash-IQ2_XXS-rocm | pp2048 | 66.96 ± 1.38 | | 30640.40 ± 637.11 | 30597.24 ± 637.11 | 30640.40 ± 637.11 | | deepseek-ai/DeepSeek-V4-Flash-IQ2_XXS-rocm | tg32 | 8.72 ± 0.09 | 9.00 ± 0.00 | | | | ## Punch List | Status | Feature | |:------:|---------| | X | Simple chat | | X | Basic quants | | X | iMatrix quants | | X | Chat template | | X | Tool calling | | X | Decent context | | X | Pipeline parallelism | | ? | Tensor parallelism | | X | Prompt caching | | X | CPU | | X | CUDA | | X | ROCm | | ? | Vulkan | | X | Cross-platform GPU | | X | antirez/ds4 compat | | X | Prefill optimization | | ?? | DSv4 Pro compat | | N/A | MTP support (believe this to be for training purposes only)| ## Original model - [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) ## Thanks - [antirez](https://github.com/antirez) — llama.cpp fork for Metal and CUDA in [llama.cpp-deepseek-v4-flash](https://github.com/antirez/llama.cpp-deepseek-v4-flash) and [DS4](https://github.com/antirez/ds4) - [ml-explore/mlx-lm #1192](https://github.com/ml-explore/mlx-lm/pull/1192) — MLX DSV4 - [DeepSeek](https://github.com/deepseek-ai) — open inference code and the [technical report](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf) - [nisparks et al](https://github.com/ggml-org/llama.cpp/issues/22319) - some early implementation efforts and discussion - [llama.cpp](https://github.com/ggml-org/llama.cpp) — the project that makes local LLM inference possible