Medsiglip Modality Ft v3

MedSigLIP Radiology Modality Classifier (v3) — with OOD Gate

Fine-tuned from google/medsiglip-448 for zero-shot image classification across 17 classes — 11 radiology modalities plus 6 non-radiology categories. v3 adds a binary radiology gate that rejects non-radiological images before classification, achieving AUROC 0.910 for out-of-distribution detection.

Model Description

This is a contrastive fine-tune of MedSigLIP — both vision and text encoders were trained jointly with SigLIP loss on image+caption pairs. Unlike classification-head approaches that freeze the vision encoder, this preserves and strengthens MedSigLIP's vision-language alignment.

v3 extends v2 by adding 6 non-radiology classes (Clinical Photograph, Gross Pathology Specimen, Pathology Histology, Schematic/Illustration, DEXA, PET-MRI) to the 11 radiology classes from v2. This enables the model to not only classify radiology modalities but also reject non-radiological images via a dedicated binary gate.

Classes (17 total)

Radiology (11): Angiography, CT, Fluoroscopy, Mammography, MRI, PET, PET-CT, Scintigraphy, SPECT, Ultrasound, X-ray

Non-Radiology (6): Clinical Photograph, DEXA, Gross Pathology Specimen, Pathology Histology, PET-MRI, Schematic / Illustration

Key Innovation: Binary Radiology Gate

v3 includes a lightweight binary classifier gate (nn.Linear(1152, 1) + sigmoid) trained on top of the frozen vision encoder's pooler_output. The gate learns a hyperplane that separates radiology visual features from non-radiology ones:

Metric	Value
Gate AUROC	0.910
Threshold	0.868 (chosen for 95% TPR)
Non-rad rejection rate	82.2%
Gate parameters	1,153 (weight + bias)

Why the gate works (and other OOD methods failed):

• Energy-based OOD (AUROC 0.581): Contrastive loss produces similar logit magnitudes for all classes — no separation.
• Nearest-centroid OOD (AUROC 0.522): All 17 class centroids are within ~0.98 cosine similarity on the hypersphere; every image finds a nearby centroid.
• Linear gate (AUROC 0.910): Learns a dedicated hyperplane that separates radiology visual features (grids, grayscale, anatomical structure) from non-radiology ones — without relying on inter-class separation.

The gate is stored as a standalone JSON file (radiology-gate.json, ~30KB) with weights, bias, threshold, and class metadata — no PyTorch dependency needed for loading.

Pipeline

Image → Vision Encoder → pooler_output (1152-dim)
                              │
                              ├─→ Binary Gate: sigmoid(W·x + b) → score
                              │       score < 0.868 → REJECT (non-radiology)
                              │       score ≥ 0.868 → ACCEPT
                              │
                              └─→ L2 normalize → cosine similarity with text prompts
                                      → softmax → top-k modality prediction

Training

Training Data

~5,000+ images across 17 classes (11 radiology + 6 non-radiology), each paired with a modality-specific text prompt.

Class prompts:

Class	Prompt
Angiography	"an angiography image"
CT	"a CT scan"
Clinical Photograph	"a clinical photograph"
DEXA	"a DEXA bone density scan"
Fluoroscopy	"a fluoroscopy image"
Gross Pathology Specimen	"a gross pathology specimen"
Mammography	"a mammogram"
MRI	"an MRI scan"
PET	"a PET scan"
PET-CT	"a PET-CT scan"
PET-MRI	"a PET-MRI scan"
Pathology Histology	"a photomicrograph or histology slide"
Schematic / Illustration	"a schematic or diagram"
Scintigraphy	"a scintigraphy or nuclear medicine scan"
SPECT	"a SPECT scan"
Ultrasound	"an ultrasound image"
X-ray	"an X-ray radiograph"

Training Procedure

• Loss: SigLIP contrastive loss (image-text alignment within batch)
• Epochs: 10
• Batch size: 32
• Learning rate: 1e-4 with cosine schedule
• Optimizer: AdamW (weight decay 0.01)
• Precision: FP16 mixed (on GPU)
• Best checkpoint: 1755
• Weighted sampling: Inverse class-frequency oversampling

Hardware

Trained on a single RTX Pro 6000.

How to Use

Basic Zero-Shot Classification

from transformers import AutoModel, AutoProcessor
from PIL import Image
import torch

model = AutoModel.from_pretrained("kashol/medsiglip-modality-ft-v3")
processor = AutoProcessor.from_pretrained("kashol/medsiglip-modality-ft-v3")

MODALITY_PROMPTS = {
    "Angiography":              "an angiography image",
    "CT":                       "a CT scan",
    "Clinical Photograph":      "a clinical photograph",
    "DEXA":                     "a DEXA bone density scan",
    "Fluoroscopy":              "a fluoroscopy image",
    "Gross Pathology Specimen": "a gross pathology specimen",
    "Mammography":              "a mammogram",
    "MRI":                      "an MRI scan",
    "PET":                      "a PET scan",
    "PET-CT":                   "a PET-CT scan",
    "PET-MRI":                  "a PET-MRI scan",
    "Pathology Histology":      "a photomicrograph or histology slide",
    "Schematic / Illustration": "a schematic or diagram",
    "Scintigraphy":             "a scintigraphy or nuclear medicine scan",
    "SPECT":                    "a SPECT scan",
    "Ultrasound":               "an ultrasound image",
    "X-ray":                    "an X-ray radiograph",
}

modalities = list(MODALITY_PROMPTS.keys())
prompts = list(MODALITY_PROMPTS.values())

image = Image.open("chest_ct.jpg").convert("RGB")
text_inputs = processor(text=prompts, padding="max_length",
                        truncation=True, max_length=64, return_tensors="pt")
img_inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(
        pixel_values=img_inputs["pixel_values"],
        input_ids=text_inputs["input_ids"],
        attention_mask=(text_inputs["input_ids"] != 0).long(),
    )
    pred_idx = outputs.logits_per_image.argmax(dim=-1).item()

print(f"Predicted: {modalities[pred_idx]}")

With Binary Radiology Gate

For production use where non-radiology images must be rejected, load the gate alongside the model:

import json
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoProcessor
from PIL import Image

# Load gate
with open("models/radiology-gate.json") as f:
    gate = json.load(f)

gate_weight = torch.tensor(gate["weights"]).float()       # (1152,)
gate_bias = torch.tensor(gate["bias"]).float()
gate_threshold = gate["threshold"]  # 0.868

# Load model
model = AutoModel.from_pretrained("kashol/medsiglip-modality-ft-v3")
processor = AutoProcessor.from_pretrained("kashol/medsiglip-modality-ft-v3")

image = Image.open("some_image.jpg").convert("RGB")
img_inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    # Get raw pooler_output (NOT L2-normalized)
    vision_outputs = model.vision_model(
        pixel_values=img_inputs["pixel_values"]
    )
    pooler = vision_outputs.pooler_output  # (1, 1152)

    # Gate check
    gate_score = torch.sigmoid(gate_weight @ pooler[0] + gate_bias).item()

    if gate_score < gate_threshold:
        print(f"REJECTED — not a radiology image (gate score: {gate_score:.3f})")
    else:
        # Proceed with zero-shot classification
        text_inputs = processor(text=MODALITY_PROMPTS.values(), ...)
        outputs = model(
            pixel_values=img_inputs["pixel_values"],
            input_ids=text_inputs["input_ids"],
            attention_mask=(text_inputs["input_ids"] != 0).long(),
        )
        pred = modalities[outputs.logits_per_image.argmax(dim=-1).item()]
        print(f"ACCEPTED (gate: {gate_score:.3f}) → {pred}")

classify_server.py (FastAPI Microservice)

A production-ready FastAPI server is included in the repository:

python classify_server.py \
  --model kashol/medsiglip-modality-ft-v3 \
  --gate models/radiology-gate.json \
  --host 0.0.0.0 --port 8765

Endpoints:

• POST /classify — Full pipeline: image → gate check → zero-shot classification
• POST /embed — Return raw 1152-dim pooler_output embedding
• GET /classify/health — Gate status, AUROC, threshold, class list

Example response from /classify:

{
  "is_radiology": true,
  "gate_score": 0.952,
  "gate_threshold": 0.868,
  "gate_auroc": 0.910,
  "modality": "CT",
  "all_classes": [
    {"label": "CT", "score": 0.987},
    {"label": "MRI", "score": 0.008},
    {"label": "PET-CT", "score": 0.003}
  ],
  "note": "Gate passed. Top-3 zero-shot classification results."
}

Rejected non-radiology image:

{
  "is_radiology": false,
  "gate_score": 0.321,
  "gate_threshold": 0.868,
  "gate_auroc": 0.910,
  "modality": null,
  "all_classes": [],
  "note": "Image rejected by radiology gate (score 0.321 < threshold 0.868)."
}

OOD Detection: Three Methods Compared

Three out-of-distribution detection methods were evaluated on the same test set. Only the binary gate proved effective:

Method	AUROC	Effective?
Binary Gate	0.910	✅ Yes — 82.2% non-rad rejection
Energy-based (MaxLogit)	0.581	❌ No — barely above random
Nearest-centroid cosine	0.522	❌ No — worse than random

Why contrastive embedding space fails for centroid/energy OOD: All 17 class centroids are within ~0.98 cosine similarity of each other. The embedding space is a tight ball on the 1152-dim hypersphere — there is no "empty region" for unknown images to fall into. The linear gate works because it learns a hyperplane rather than relying on distance to centroids.

Limitations

• Schematics & Histology false positives: Schematics and pathology histology images score at P=1.000 on the gate (17.8% of non-radiology images pass). These share visual features with radiology (grids, grayscale, structured content). Mitigation: more training data or multi-label heads.
• Nuclear medicine confusion: SPECT, Scintigraphy, and PET share visual characteristics and are the primary source of confusion among radiology classes.
• Single-modality assumption: The model classifies each image as exactly one class.
• Not for diagnosis: This model identifies imaging modality, NOT pathology. It is a triage, search, and organization tool — not a diagnostic aid.

Files

File	Description
models/medsiglip-modality-ft-v3/checkpoint-1755/	Fine-tuned model weights (best checkpoint)
models/radiology-gate.json	Binary gate weights, bias, threshold (30KB JSON)
classify_server.py	FastAPI production server with gate integration
Dockerfile	CUDA Docker container for classify_server

Citation

@misc{medsiglip-modality-ft-v3-2026,
  author = {kashol},
  title = {MedSigLIP Radiology Modality Classifier v3 — with OOD Gate},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/kashol/medsiglip-modality-ft-v3}},
}

Base model: google/medsiglip-448