Sam3 Browser Int8

--- license: apache-2.0 tags: - sam3 - segment-anything - onnx - int8 - browser - webgpu - computer-vision - segmentation pipeline_tag: image-segmentation --- # SAM3 Browser INT8 — Quantized ONNX Models INT8-quantized ONNX models for running [SAM3 (Segment Anything Model 3)](https://github.com/facebookresearch/sam3) entirely in the browser via [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/). ## Files | File | Size | Description | |---|---|---| | `sam3_image_encoder.onnx` | 466 MB | ViT backbone — encodes the input image into feature maps | | `sam3_language_encoder.onnx` | 387 MB | CLIP text encoder — converts text prompts into embeddings | | `sam3_decoder.onnx` | 35 MB | DETR-style decoder — produces boxes, scores, and pixel masks | | `clip_tokenizer.json` | 1.5 MB | CLIP BPE tokenizer vocabulary (encoder + merge table + byte encoder) | | **Total** | **~889 MB** | | ## Tokenizer `clip_tokenizer.json` contains the full CLIP BPE tokenizer data needed to tokenize text prompts for the language encoder. It includes: - **`encoder`**: BPE token → integer ID mapping (~49,408 entries) - **`merges`**: BPE merge rules (~48,894 pairs) - **`byte_encoder`**: byte-to-unicode mapping for UTF-8 handling This is extracted from OpenAI's CLIP `bpe_simple_vocab_16e6.txt.gz` and packaged as JSON for browser use. Load it at runtime (~500 KB gzipped by CDN) and use it to tokenize prompts into `int64` token sequences of length 32, padded with `[START=49406, ...tokens..., END=49407, 0, 0, ...]`. ## Quantization Dynamic INT8 quantization via `onnxruntime.quantization.quantize_dynamic` with `QUInt8` weights. Original FP32 models totaled ~3.5 GB. Quality is preserved: the quantized pipeline scores **0.9495** on a test image (vs 0.9471 for FP32). ## Usage These models are designed for browser inference. Load them with ONNX Runtime Web: ### Pipeline 1. **Tokenizer**: text prompt → `clip_tokenizer.json` → `int64[1, 32]` (CLIP BPE tokens) 2. **Image encoder**: input `image` (uint8, shape `[3, 1008, 1008]`) → 6 output tensors (vision pos encodings + backbone FPN features) 3. **Language encoder**: input `tokens` (int64, shape `[1, 32]`) → `text_attention_mask`, `text_memory`, `text_embeds` 4. **Decoder**: combines encoder outputs + prompt tensors → `boxes`, `scores`, `masks` ### Decoder inputs | Input | Type | Shape | Source | |---|---|---|---| | `or...