Visualheist Base

--- license: mit license_link: https://huggingface.co/microsoft/Florence-2-base-ft/resolve/main/LICENSE pipeline_tag: image-text-to-text tags: - vision - ocr - segmentation --- # VisualHeist - figure, scheme and table segmentation from PDFs (with captions, headers & footnotes) ## Model Summary VisualHeist is an object detection model finetuned to extract tables and figures from PDFs. VisualHeist has two versions: - visualheist-base[[HF]](https://huggingface.co/shixuanleong/visualheist-base) (0.23B) - visualheist-large[[HF]](https://huggingface.co/shixuanleong/visualheist-large) (0.77B) **The base model is recommended if you are running it on low-RAM systems** The models are finetuned from [microsoft/Florence-2](https://huggingface.co/microsoft/Florence-2-large-ft) checkpoints. VisualHeist is inspired by and adapted from [yifeihu/TF-ID](https://huggingface.co/yifeihu/TF-ID-large) - The models were finetuned with 3435 figures and 1716 tables from 110 PDF articles across various publishers. All bounding boxes are manually annotated using [CoCo Annotator](https://github.com/jsbroks/coco-annotator). - TF-ID models take an image of a single paper page as the input, and return image files for all figures, schemes and tables in the given page. ## Training Code and Dataset - Dataset: [Zenodo repository](https://doi.org/10.5281/zenodo.14917752) - Code: [github.com/aspuru-guzik-group/MERMaid](https://github.com/aspuru-guzik-group/MERMaid) ## Benchmarks We manually curated a diverse evaluation dataset consisting of 121 literature articles covering a range of topics, including organic and inorganic chemistry, atmospheric science, batteries, materials science, metal-organic frameworks (MOFs), biology, and science education. These PDFs, published between 1949 and 2025, include both main articles and supplementary materials. We also additionally curated another collection of 98 literature articles (MERMaid-100) reporting novel reaction methodologies that spans three distinct chemical domains: organic electrosynthesis, photocatalysis, and organic synthesis. Additional performance discussion can be found from our [preprint article](XXXXXXX) The full DOI lists can be downloaded from our[Zenodo repository](https://doi.org/10.5281/zenodo.14917752). The evaluation results for visualheist-large are: | | Total Images | F1 score | |------------------------------------...