--- base_model: llmfan46/Ornith-1.0-35B-uncensored-heretic library_name: transformers pipeline_tag: text-generation license: other tags: - vllm - compressed-tensors - nvfp4 - fp8 - dgx-spark - dgx-spark-optimized - optimized-quant - uncensored-heretic - gb10 - qwen3.5 - moe --- # Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10 This is a DGX Spark optimized quant for vLLM: a GB10-oriented compressed-tensors quantization of [llmfan46/Ornith-1.0-35B-uncensored-heretic](https://huggingface.co/llmfan46/Ornith-1.0-35B-uncensored-heretic) for vLLM inference. The source model is [llmfan46/Ornith-1.0-35B-uncensored-heretic](https://huggingface.co/llmfan46/Ornith-1.0-35B-uncensored-heretic). ## Quantization - Dense attention and linear-attention projections: FP8 W8A8. - Routed MoE experts and shared expert projections: NVFP4 W4A4. - Preserved in BF16: embeddings, `lm_head`, router gates, visual modules, norms, Conv1D, `A_log`, `dt_bias`, and non-target tensors. - Visual encoder: copied from the source checkpoint and preserved in BF16 for multimodal/image support. - Calibration data: `HuggingFaceH4/ultrachat_200k`, `train_sft`. - Calibration samples: 512. - Calibration sequence length: 2048. - Pipeline: sequential, CPU offload, all MoE experts calibrated. The quantization groups use mutually exclusive explicit target regexes. ## Output - Format: `mixed-precision`. - Quantization method: `compressed-tensors`. - Config groups: `group_0, group_1`. - Weight layout: sharded safetensors with model.safetensors.index.json. - Safetensors shards: 5. - Indexed tensors: 124376. - Max position embeddings: 262144. - MTP weights: not present in the inspected source checkpoint. ## DGX Spark vLLM This model is intended for the GB10 patched vLLM path used by [demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10](https://huggingface.co/demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10). Example: ```bash docker run --gpus all -p 8000:8000 --ipc host \ -v /opt/vllm-cache:/root/.cache/huggingface \ -e CUBLASLT_WORKSPACE_SIZE=33554432 \ -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \ -e RUNAI_STREAMER_MEMORY_LIMIT=4294967296 \ vllm/vllm-openai:v0.22.1 \ --model thanet-s/Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10 \ --served-model-name Ornith-1.0-35B-uncensored-heretic \ --kernel-config '{"moe_backend": "flashinfer_b12x"}' \ --load-format runai_streamer \ --gpu-memory-utilization 0.50 \ --kv-cache-dtype fp8 \ --enable-prefix-caching \ --enable-chunked-prefill \ --max-num-batched-tokens 4176 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 ``` On DGX Spark / GB10, the vLLM image may still need the same CUTLASS DSL and flashinfer SM12x patch files described in the reference model card above. ## DGX Spark benchmark Measured on NVIDIA DGX Spark / GB10 with `vllm/vllm-openai:v0.22.1`, the GB10 CUTLASS DSL and flashinfer SM12x patches, `--gpu-memory-utilization 0.50`, `--kv-cache-dtype fp8`, `--kernel-config {"moe_backend": "flashinfer_b12x"}`, and `--max-num-batched-tokens 4176`. - vLLM log throughput during a warm single request: `Avg generation throughput: 60.8 tokens/s`. - Client-side measurement for the same warm request: 1024 completion tokens in 16.905s, about `60.58 tok/s`. - First measured request after server readiness: 768 completion tokens in 13.676s, about `56.15 tok/s`; vLLM log showed `Avg generation throughput: 53.5 tokens/s`. - Model load memory reported by vLLM: 21.04 GiB. - Max model length reported by vLLM: 262144 tokens. - GPU KV cache size at `--gpu-memory-utilization 0.50`: 3,344,321 tokens. - Maximum concurrency reported by vLLM for 262144-token requests: 12.76x. These figures are from one local DGX Spark run and may change with prompt shape, sampling settings, vLLM version, patch versions, and CUDA graph/cache warmup state. ## Status Local DGX Spark conversion validation: - Source: `llmfan46/Ornith-1.0-35B-uncensored-heretic`. - Conversion completed with the same NVFP4-FP8Dense GB10 recipe used for the non-uncensored Ornith 35B build. - `verify-output.py` passed after vLLM config patching and visual tensor merge. - Visual tensors preserved: 333 `model.visual.*` tensors copied from the source checkpoint. - vLLM serving smoke test passed on DGX Spark / GB10 after upload.
