LocateAnything 3B ONNX WebGPU INT4

--- license: apache-2.0 base_model: nvidia/LocateAnything-3B tags: - onnx - onnxruntime-web - webgpu - int4 - quantization - object-detection - visual-grounding library_name: onnxruntime pipeline_tag: object-detection --- # LocateAnything-3B — ONNX WebGPU (INT4 + 4-bit embeddings) In-browser ([onnxruntime-web](https://onnxruntime.ai/docs/tutorials/web/) / WebGPU) build of [`nvidia/LocateAnything-3B`](https://huggingface.co/nvidia/LocateAnything-3B), a visual-grounding / open-vocabulary detector. The language tower is weight-only **INT4** and the **embedding table is true group-wise 4-bit**. ## Why this repo exists The naive "4-bit ONNX" of this model was ~3GB because the model has **tied word embeddings** (`vocab 152681 × hidden 2048 = 1.25GB` in fp32). ORT's `MatMulNBits` INT4 quantizer compresses the tied `lm_head` MatMul but **leaves the input-embedding `Gather` at full fp32** — so 1.25GB of fp32 embeddings stayed in the package. This build fixes that with a **custom quantized embedding gather**: 1. The language graph was surgically rewired to consume `inputs_embeds` directly (the fp32 embedding `Gather` and its 1.25GB initializer are removed). It still takes `input_ids` (used only by the SDLM block-mask `==` comparisons, not a Gather) and `visual_features` (spliced at the image token). 2. The embedding table ships as a **group-wise symmetric INT4 blob** (`(q-8)·scale`, block size 32): `embed_tokens_int4_packed.bin` (uint8 nibble-packed) + `embed_tokens_int4_scales.bin` (fp16). 3. The browser does the **gather + dequant in JS** to build `inputs_embeds`, then runs the INT4 language graph. ## Files (browser-facing) | File | Size | Notes | |------|------|-------| | `onnx/vision_mlp.onnx` (+`.data`) | ~1.73 GB | MoonViT + projector, fp32 (see note below) | | `onnx/language_tail_int4.onnx` (+`.data`) | ~1.69 GB | Qwen2 language tower + tied lm_head, weight-only INT4 (block 128) | | `onnx/embed_tokens_int4_packed.bin` | ~156 MB | INT4 embedding table, uint8 nibble-packed `[152681, 1024]` | | `onnx/embed_tokens_int4_scales.bin` | ~19.5 MB | fp16 group scales `[152681, 64]` | | `onnx/embed_tokens_int4_meta.json` | — | layout / dequant scheme | | `web_config.json` | — | runtime wiring, token ids, tail size | **Total browser payload ≈ 3.6 GB.** The big win here is the language side: the embedding table dropped from **1.25 GB fp32 → 176 MB INT4** and...