Docling Layout Heron 101 Mlx BF16

--- license: apache-2.0 library_name: mlx pipeline_tag: object-detection base_model: docling-project/docling-layout-heron-101 base_model_relation: quantized tags: - mlx - rt_detr_v2 - object-detection - layout-analysis - docling - document-understanding --- # docling-layout-heron-101 — MLX (bfloat16) MLX-converted weights of [`docling-project/docling-layout-heron-101`](https://huggingface.co/docling-project/docling-layout-heron-101), the larger (ResNet-101-vd backbone) variant of the Docling layout model. Apache-2.0, same as upstream. The architecture is RT-DETRv2 with a **ResNet-101-vd** backbone (`depths=[3, 4, 23, 3]`), 300 queries, and 17 layout classes. The smaller ResNet-50 sibling is at [`mlx-community/docling-layout-heron-mlx-bf16`](https://huggingface.co/mlx-community/docling-layout-heron-mlx-bf16). ## Inference Requires the RT-DETRv2 MLX port in [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm). ```python from pathlib import Path from PIL import Image from huggingface_hub import snapshot_download from transformers import AutoProcessor from mlx_vlm.utils import load_model from mlx_vlm.models.rt_detr_v2.generate import RTDetrV2Predictor import mlx_vlm.models.rt_detr_v2 # registers the processor with AutoProcessor path = Path(snapshot_download("mlx-community/docling-layout-heron-101-mlx-bf16")) model = load_model(path) processor = AutoProcessor.from_pretrained(path) predictor = RTDetrV2Predictor(model, processor, threshold=0.3) result = predictor.predict(Image.open("page.png")) for name, score, box in zip(result.class_names, result.scores, result.boxes): print(f"{name:20s} {score:.3f} {box.tolist()}") ``` `result` is a `DetectionResult` with vectorized fields: `boxes` (N, 4) xyxy in original-image pixels, `scores` (N,), `labels` (N,) integer class ids, and `class_names`. ## Conversion Produced with: ```bash python -m mlx_vlm.models.rt_detr_v2.convert \ --hf-path docling-project/docling-layout-heron-101 \ --output ./docling-layout-heron-101-mlx-bf16 \ --dtype bfloat16 ``` Numerical validation against `transformers.RTDetrV2ForObjectDetection` on real document inputs: max abs error ~2e-5 on logits, sub-pixel on bboxes. ## License and citation Apache-2.0. The original work is described in *"Advanced Layout Analysis Models for Docling"* by Livathinos et al. ([arXiv:2509.11720](https://arxiv.org/abs/2509.11720)); please cite the up...