Docling Layout Heron Mlx BF16

--- license: apache-2.0 library_name: mlx pipeline_tag: object-detection base_model: docling-project/docling-layout-heron base_model_relation: quantized tags: - mlx - rt_detr_v2 - object-detection - layout-analysis - docling - document-understanding --- # docling-layout-heron — MLX (bfloat16) MLX-converted weights of [`docling-project/docling-layout-heron`](https://huggingface.co/docling-project/docling-layout-heron), the default document-layout model of the [Docling](https://github.com/docling-project/docling) project. Apache-2.0, same as upstream. The architecture is RT-DETRv2 with a ResNet-50-vd backbone, 300 queries, and 17 layout classes (`caption`, `footnote`, `formula`, `list_item`, `page_footer`, `page_header`, `picture`, `section_header`, `table`, `text`, `title`, `document_index`, `code`, `checkbox_selected`, `checkbox_unselected`, `form`, `key_value_region`). ## Inference Requires the RT-DETRv2 MLX port in [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm). ```python from pathlib import Path from PIL import Image from huggingface_hub import snapshot_download from transformers import AutoProcessor from mlx_vlm.utils import load_model from mlx_vlm.models.rt_detr_v2.generate import RTDetrV2Predictor import mlx_vlm.models.rt_detr_v2 # registers the processor with AutoProcessor path = Path(snapshot_download("mlx-community/docling-layout-heron-mlx-bf16")) model = load_model(path) processor = AutoProcessor.from_pretrained(path) predictor = RTDetrV2Predictor(model, processor, threshold=0.3) result = predictor.predict(Image.open("page.png")) for name, score, box in zip(result.class_names, result.scores, result.boxes): print(f"{name:20s} {score:.3f} {box.tolist()}") ``` `result` is a `DetectionResult` with vectorized fields: `boxes` (N, 4) xyxy in original-image pixels, `scores` (N,), `labels` (N,) integer class ids, and `class_names`. ## Conversion Produced with: ```bash python -m mlx_vlm.models.rt_detr_v2.convert \ --hf-path docling-project/docling-layout-heron \ --output ./docling-layout-heron-mlx-bf16 \ --dtype bfloat16 ``` Numerical validation against `transformers.RTDetrV2ForObjectDetection` on real document inputs: max abs error ~2e-5 on logits, sub-pixel on bboxes. ## License and citation Apache-2.0. The original work is described in *"Advanced Layout Analysis Models for Docling"* by Livathinos et al. ([arXiv:2509.11720](htt...