Loading
Have a ComfyUI workflow? Get a readable Workflow Doc in your browser.Make a workflow readable.Try it now ↗
Static INT8 ONNX quantization of docling-project/docling-layout-heron-onnx (Docling Heron, RT-DETRv2, 17-class document-layout detection at 640×640). Produced for RailReaderCore.
Model source
Source description
Static INT8 ONNX quantization of
docling-project/docling-layout-heron-onnx (Docling Heron, RT-DETRv2, 17-class
document-layout detection at 640×640). Produced for
RailReaderCore.
Sources
1 sourceVerified Sep 16
Model artifacts
1 artifactSource excerpts
2 excerptsQDQ static quantization of only the CNN backbone Conv ops; the RT-DETR
transformer decoder is left in FP32.
Quantizing the decoder drives two quantization scales
(model.decoder.anchors_scale, add_2355_scale) to Inf, collapsing all
detection scores → zero detections. Leaving the decoder FP32 avoids this, and
the backbone is where essentially all of the convolutional compute (and the
speedup) lives. Quantizing any MatMul (encoder or decoder) was measured to
either break the model or reduce recall with no speed gain.
Versus the FP32 base on 88 held-out real academic pages (detection agreement, FP32 as reference):
| metric | value |
|---|---|
| recall | 0.990 |
| precision | 0.988 |
| mean IoU (matched) | 0.984 |
| pages with <90% recall | 1 / 88 |
| inference speedup (AVX-VNNI CPU, i7-12700H) | ~2.6–3× |
Visually faithful on dense and math-heavy pages.
Caveat: agreement is measured against the FP32 Heron model, not labelled ground truth — i.e. this is near-lossless versus the model you would otherwise ship, not an independent mAP claim.
images uint8 [1,3,640,640] NCHW; orig_target_sizes int64
[1,2] = [W, H] (note: [W, H], not [H, W]).labels [1,300], boxes [1,300,4] (xyxy in pixel space),
scores [1,300].17 classes: caption, footnote, formula, list_item, page_footer, page_header, picture, section_header, table, text, title, document_index, code, checkbox_selected, checkbox_unselected, form, key_value_region.
from onnxruntime.quantization import quantize_static, QuantFormat, QuantType, CalibrationMethod
quantize_static(
pre_processed_fp32, out, calibration_reader,
quant_format=QuantFormat.QDQ, per_channel=True,
activation_type=QuantType.QUInt8, weight_type=QuantType.QInt8,
calibrate_method=CalibrationMethod.MinMax,
op_types_to_quantize=["Conv"],
)
Calibration: real rasterized academic pages. Full reproducible tooling:
tools/quant-probe/ in RailReaderCore.
Inherits the license of the base model
docling-project/docling-layout-heron-onnx.
--- license: apache-2.0 base_model: docling-project/docling-layout-heron-onnx pipeline_tag: object-detection library_name: onnxruntime tags: - document-layout-analysis - object-detection - onnx - int8 - quantized - rt-detr - docling --- # Docling Heron Layout — backbone-only INT8 (ONNX) Static **INT8** ONNX quantization of [`docling-project/docling-layout-heron-onnx`](https://huggingface.co/docling-project/docling-layout-heron-onnx) (Docling Heron, RT-DETRv2, 17-class document-layout detection at 640×640). Produced for [RailReaderCore](https://github.com/sjvrensburg/RailReaderCore). ## What this is QDQ static quantization of **only the CNN backbone `Conv` ops**; the RT-DETR transformer **decoder is left in FP32**. ## Why backbone-only Quantizing the decoder drives two quantization scales (`model.decoder.anchors_scale`, `add_2355_scale`) to `Inf`, collapsing all detection scores → zero detections. Leaving the decoder FP32 avoids this, and the backbone is where essentially all of the convolutional compute (and the speedup) lives. Quantizing any MatMul (encoder or decoder) was measured to either break the model or reduce recall with no speed gain. ## Validation Versus the FP32 base on **88 held-out** real academic pages (detection agreement, FP32 as reference): | metric | value | |---|---| | recall | 0.990 | | precision | 0.988 | | mean IoU (matched) | 0.984 | | pages with <90% recall | 1 / 88 | | inference speedup (AVX-VNNI CPU, i7-12700H) | ~2.6–3× | Visually faithful on dense and math-heavy pages. > **Caveat:** agreement is measured against the FP32 Heron model, not labelled > ground truth — i.e. this is *near-lossless versus the model you would > otherwise ship*, not an independent mAP claim. ## I/O contract (identical to the base export) - **Inputs:** `images` uint8 `[1,3,640,640]` NCHW; `orig_target_sizes` int64 `[1,2]` = `[W, H]` (note: `[W, H]`, not `[H, W]`). - **Outputs:** `labels` `[1,300]`, `boxes` `[1,300,4]` (xyxy in pixel space), `scores` `[1,300]`. 17 classes: caption, footnote, formula, list_item, page_footer, page_header, picture, section_header, table, text, title, document_index, code, checkbox_selected, checkbox_unselected, form, key_value_region. ## Recipe ```python from onnxruntime.quantization import quantize_static, QuantFormat, QuantType, CalibrationMethod quantize_static( pre_processed_fp32, out, calibration_reader,...
Source context: 0 downloads · 1 likes · Pipeline object-detection · Library onnxruntime · Repo stefanj0/docling-layout-heron-int8-onnx