Fundus 9model Benchmark

Two result tables appear in this README. They measure different things and must not be mixed:

Always cite § 5.1 in the paper. The single-run baseline (§ 5.2) is retained for completeness — CLIP underperforms there (86.25%) due to a single unlucky initialisation versus 90.15% over 5 folds.

1. Abstract

Automatic interpretation of colour fundus photographs is a foundational task for screening prevalent blinding diseases such as diabetic retinopathy, glaucoma and age-related macular degeneration. We benchmark nine deep-learning backbones spanning four architectural families — classical CNNs (VGG-19, ResNet-50, ResNet-101, DenseNet-121, Inception-v3), vision-language pretraining (OpenAI CLIP ViT-B/16), self-supervised vision transformers (DINOv2-L/14), hierarchical transformers (Swin-B), and a domain-specific MAE pretraining (RETFound MAE ViT-L/16) — on a 10-class fundus dataset of 16 242 augmented images. To suppress augmentation-induced label leakage we construct a group-aware (perceptual-hash) stratified 5-fold split and report mean accuracy, F1, ROC-AUC, ECE, Cohen's κ, Brier score, bootstrap 95% confidence intervals, Bonferroni-corrected McNemar tests, and 90% Mondrian conformal sets.

Headline result (5-fold CV). Classical CNNs and CLIP ViT-B/16 are statistically indistinguishable at the top, with Inception-v3 leading at 90.18% accuracy (F1 = 92.54%, κ = 0.884, ROC-AUC = 0.9930). Contrary to expectations, foundation models pretrained on general vision (DINOv2-L: 89.61%, Swin-B: 87.00%) or fundus images (RETFound: 83.35%) do not outperform classical CNNs on this moderate-sized dataset. A soft-vote ensemble of all nine models reaches ROC-AUC = 0.9941 and accuracy = 89.68%.

2. Motivation & Model Selection

Modern fundus screening pipelines are increasingly built on pre-trained image backbones, but the question "which backbone family is best for fundus disease classification on a moderately-sized, imbalanced dataset?" has no consensus answer. We deliberately chose backbones that exercise four distinct inductive biases / pretraining regimes:

This grid isolates three confounders: (i) scale (ResNet-50 vs ResNet-101; ViT-B vs ViT-L); (ii) pretraining modality (ImageNet supervised vs CLIP language-supervised vs DINOv2 self-supervised vs RETFound domain-MAE); and (iii) architecture class (CNN vs ViT vs hierarchical).

3. Dataset

• Source. [Mendeley Data][6] (10 classes; 5 335 original images).
• Augmentation. Class-balancing augmentation expanded the pool to 16 242 images (rotation, horizontal flip, brightness/contrast jitter, Gaussian blur). Each augmented image carries its source's diagnostic label.
• Companion dataset: DoB24/fundus-10class-augmented.

3.1 Class distribution

3.2 Group-aware splitting (data-leakage prevention)

Because the augmented set contains visually near-duplicate copies of each original image, a naïve train_test_split over the augmented pool would let models memorise patient-level identities. We prevent this by:

1. Computing a 64-bit perceptual hash (pHash) on every image.
2. Linking each augmented image to its nearest original at Hamming distance ≤ 8 → defines a group_id.
3. Running scikit-learn StratifiedGroupKFold(n_splits=5) so that all augmented children of a given original sit in exactly one fold.

The held-out test set is fixed across all 5 folds: 3 208 images (≈ 19.8% of the augmented pool). Exact manifest: splits/holdout_split_augmented.json (3.2 MB).

4. Training Protocol

4.1 CNN / CLIP backbones

4.2 Foundation model backbones (DINOv2-L, Swin-B, RETFound)

Two-stage schedule per fold:

Batch size: 24. Early stopping patience: 8 epochs on val F1.

4.3 Ensemble

F1-weighted soft-vote across all 9 models, using each model's validation F1 as the weight.

5. Results

All metrics are on the fixed 3 208-image holdout test set.

5.1 Five-fold cross-validation — authoritative results (use in paper)

Acc, F1, 95% CI, ROC-AUC, and ECE are means over 5 independent training runs. κ and Brier for CNN/CLIP are from 5-fold pooled predictions. For foundation models (†), κ and Brier are from single-run inference on the same holdout (fold-level predictions not stored); acc/F1/ROC/ECE are still 5-fold averages.

† κ and Brier for DINOv2-L, Swin-B, RETFound from single-run holdout inference. All other foundation model metrics are 5-fold CV averages.

Key findings:

• Top-4 models (Inception-v3, CLIP, VGG-19, ResNet-101) are statistically indistinguishable: all 6 pairwise McNemar tests p > 0.05 after Bonferroni correction (see kfold/cnn_clip/mcnemar.json).
• DINOv2-L (89.61%) matches DenseNet-121 (89.65%) within noise despite having 7× more parameters.
• RETFound, pretrained on 1.6 M fundus images, ranks last — the LP+FT protocol with patience=8 may be insufficient on this dataset size.
• The ensemble gains ROC-AUC parity with the best individual model.

Machine-readable full table: kfold/kfold_v2_summary.csv

5.2 Single-run baseline (§ 5.2 — for reference only, do not cite in paper)

Retained to document the original preliminary experiment (one run, fold-0 split). CLIP's 86.25% here vs 90.15% in § 5.1 is single-run variance, not an architectural effect.

The ensemble row is identical in both tables — it is computed once on the fixed holdout and does not depend on the training run.

5.3 Statistical significance

All 15 pairwise Bonferroni-corrected McNemar tests among the 6 CNN/CLIP models: p > 0.05 — no statistically significant pairwise differences. Full χ2 and p-value matrix: kfold/cnn_clip/mcnemar.json.

5.4 Calibration (ECE)

weights/dinov2_l_v2.pth

pth · 1.13 GB · SHA-256 68d3215857fb…4734 · Hugging Face