Finetune Donut Cord v3

--- license: openrail datasets: - naver-clova-ix/cord-v2 metrics: - accuracy library_name: transformers pipeline_tag: image-to-text language: - en tags: - donut - document-understanding - ocr-free - receipt-processing - vision-language-model - financial-documents --- # Donut Base Fine-tuned on CORD v2 Dataset ## Model Description This model is a fine-tuned version of [naver-clova-ix/donut-base](https://huggingface.co/naver-clova-ix/donut-base) on the [CORD v2 (Consolidated Receipt Dataset)](https://huggingface.co/datasets/naver-clova-ix/cord-v2) for OCR-free receipt understanding and structured information extraction. **Donut** (Document Understanding Transformer) is an end-to-end vision-language model that processes document images directly to extract structured information without requiring an intermediate OCR step. This approach eliminates error propagation from OCR mistakes and provides a more streamlined solution for document understanding tasks. ## Model Details - **Base Model:** naver-clova-ix/donut-base (201M parameters) - **Task:** Image-to-text generation for structured receipt information extraction - **Training Approach:** Full fine-tuning (all 201M parameters updated) - **Hardware:** RTX 5090 GPU with 32GB VRAM - **Training Framework:** PyTorch Lightning with mixed precision (BF16) - **Dataset:** CORD v2 - 800 training images, 100 validation images - **Image Resolution:** 1280×960 pixels - **Max Sequence Length:** 768 tokens ## Performance ### Test Set Results (100 held-out receipt images) - **Mean Accuracy:** 90.73% - **Accuracy Range:** 27% - 100% across samples - **Performance by Complexity:** - Simple receipts (3-5 items): 95-100% accuracy - Complex receipts (15+ items): 75-80% accuracy ### Evaluation Metric Accuracy is calculated using the JSONParseEvaluator from the Donut framework, which performs field-level comparison of extracted information: ``` Accuracy = (Number of Correctly Extracted Fields) / (Total Number of Fields) ``` The evaluator compares predicted and ground truth JSON objects by matching key-value pairs across multiple fields including: - Menu items (name, count, unit price) - Subtotals - Tax amounts - Service charges - Total amounts **Note:** The 90.73% accuracy means that on average, approximately 9 out of every 10 information fields are correctly extracted from unseen receipts. ## Training Details ###...