Docling Layout Heron Int8 Onnx

--- license: apache-2.0 base_model: docling-project/docling-layout-heron-onnx pipeline_tag: object-detection library_name: onnxruntime tags: - document-layout-analysis - object-detection - onnx - int8 - quantized - rt-detr - docling --- # Docling Heron Layout — backbone-only INT8 (ONNX) Static **INT8** ONNX quantization of [`docling-project/docling-layout-heron-onnx`](https://huggingface.co/docling-project/docling-layout-heron-onnx) (Docling Heron, RT-DETRv2, 17-class document-layout detection at 640×640). Produced for [RailReaderCore](https://github.com/sjvrensburg/RailReaderCore). ## What this is QDQ static quantization of **only the CNN backbone `Conv` ops**; the RT-DETR transformer **decoder is left in FP32**. ## Why backbone-only Quantizing the decoder drives two quantization scales (`model.decoder.anchors_scale`, `add_2355_scale`) to `Inf`, collapsing all detection scores → zero detections. Leaving the decoder FP32 avoids this, and the backbone is where essentially all of the convolutional compute (and the speedup) lives. Quantizing any MatMul (encoder or decoder) was measured to either break the model or reduce recall with no speed gain. ## Validation Versus the FP32 base on **88 held-out** real academic pages (detection agreement, FP32 as reference): | metric | value | |---|---| | recall | 0.990 | | precision | 0.988 | | mean IoU (matched) | 0.984 | | pages with <90% recall | 1 / 88 | | inference speedup (AVX-VNNI CPU, i7-12700H) | ~2.6–3× | Visually faithful on dense and math-heavy pages. > **Caveat:** agreement is measured against the FP32 Heron model, not labelled > ground truth — i.e. this is *near-lossless versus the model you would > otherwise ship*, not an independent mAP claim. ## I/O contract (identical to the base export) - **Inputs:** `images` uint8 `[1,3,640,640]` NCHW; `orig_target_sizes` int64 `[1,2]` = `[W, H]` (note: `[W, H]`, not `[H, W]`). - **Outputs:** `labels` `[1,300]`, `boxes` `[1,300,4]` (xyxy in pixel space), `scores` `[1,300]`. 17 classes: caption, footnote, formula, list_item, page_footer, page_header, picture, section_header, table, text, title, document_index, code, checkbox_selected, checkbox_unselected, form, key_value_region. ## Recipe ```python from onnxruntime.quantization import quantize_static, QuantFormat, QuantType, CalibrationMethod quantize_static( pre_processed_fp32, out, calibration_reader,...