ModelPrésentéExterne

Medfuse Seg

Sourcehuggingface.co/biodatlab/medfuse-seg/tree/947b67a609366eeb616d321ae15374fae2aa41bf

TypeOtherModèle de basegoogle/medgemma-4b-itVersionBf16 zero pp rank 0 mp rank 00 optim statesCClasse

Source du modèle

Type: Other
Taille du fichier: 2,34–15,4 GB
Modèle de base: google/medgemma-4b-it
Version: vBf16 zero pp rank 0 mp rank 00 optim states
Mis à jour: 1 nov. 2026
Source: Hugging Face

Description de la source

MedFuse-Seg: Multi-Level Visual and Semantic Context Fusion for Segmentation-Based Medical Reasoning (MICCAI 2026)

Dataset Code

MedFuse-Seg bridges the semantic-spatial gap in language-driven medical image analysis by combining multi-level visual feature injection with LLM-guided mask decoding. Built on MedGemma-4B, MedSigLIP, and MedSAM, the model allows clinicians to obtain both diagnostic reasoning and precise anatomical segmentation through natural language prompts.

For full training details and hyperparameters, see the project repository and Med-ReasonSeg dataset.

Model Performance

MedFuse-Seg outperforms zero-shot BiomedParse by 13.49% DSC and 54.04 px HD95, and fine-tuned LISA-7B (same training setup) by 4.89% DSC and 15.29 px HD95.

Method	DSC (Ref)	DSC (Sem)	DSC (Avg)	HD95 (Ref)	HD95 (Sem)	HD95 (Avg)
SAM 3 (zero-shot)	0.1425	0.1167	0.1296	373.52	370.50	372.01
BiomedParse (zero-shot)	0.6703	0.6344	0.6524	105.23	115.97	110.60
LISA-7B (fine-tuned)	0.7398	0.7370	0.7384	71.55	72.13	71.84
MedFuse-Seg (Ours)	0.7879	0.7867	0.7873	56.46	56.65	56.55

Evaluated on the Med-ReasonSeg test set. LISA-7B was retrained with identical training setup for a fair comparison.

Download & Use

1. Install dependencies

git clone https://github.com/biodatlab/medfuse-seg.git
cd medfuse-seg
pip install -r requirements.txt

2. Download MedSAM checkpoint (required)

Download from the original MedSAM paper's repository:

gdown "https://drive.google.com/uc?id=1UAmWL88roYR7wKlnApw5Bcuzf2iQgk6_"

Place medsam_vit_b.pth in the repository root.

3. Download checkpoint and run inference

from huggingface_hub import hf_hub_download
from medfuseseg import MedFuseSegPipeline

hf_hub_download(repo_id="biodatlab/medfuse-seg", local_dir="ckpts", repo_type="model")

pipe = MedFuseSegPipeline(checkpoint="ckpts")

result = pipe(
    image="chest_xray.png",  # filepath, URL, PIL Image, or numpy array
    prompt="Segment the pneumonia region"
)

print(result.text)        # "The affected lung parenchyma is [SEG]..."
result.save_mask("mask.png")
result.save_overlay("vis.png")

MedGemma-4B-IT will be downloaded automatically from HuggingFace Hub on first run.

Citation

@inproceedings{LimKee_MedFuseSeg_MICCAI2026,
  title={MedFuse-Seg: Multi-Level Visual and Semantic Context Fusion for Segmentation-Based Medical Reasoning},
  author={Limaroon, Keetawan and Chiewhawan, Monrada and Timklaypachara, Watcharapong and Vateekul, Peerapon and Achakulvisut, Titipat},
  booktitle = {Proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
  year={2026}
}

License

Apache-2.0

Sources

1 source

Hugging FacePrincipalVersion 947b67a609366eeb616d321ae15374fae2aa41bf

huggingface.co/biodatlab/medfuse-seg/tree/947b67a609366eeb616d321ae15374fae2aa41bf

Vérifié 1 nov.

Download files

Artefacts du modèle

5 artefacts

global_step66855/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt

pt · BF16 (unverified) · 2,34 GB · SHA-256 8271d5a0f2a2…8422 · Hugging Face

Download file

global_step66855/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt

pt · BF16 (unverified) · 2,34 GB · SHA-256 6f4f45084175…c5b7 · Hugging Face

Download file

global_step66855/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt

pt · BF16 (unverified) · 2,34 GB · SHA-256 b3c155f3f9fa…52be · Hugging Face

Download file

global_step66855/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt

pt · BF16 (unverified) · 2,34 GB · SHA-256 2e0d0e3f327f…09a6 · Hugging Face

Download file

global_step66855/mp_rank_00_model_states.pt

pt · 15,4 GB · SHA-256 9e86a1e50ced…5d2e · Hugging Face

Download file

Extraits de sources

2 extraits

Readme17 aoûthuggingface.co/biodatlab/medfuse-seg

--- language: en license: apache-2.0 libraries: - deepspeed - transformers - peft pipeline_tag: image-segmentation base_model: - google/medgemma-4b-it tags: - medical-image-segmentation - multimodal-llm - reasoning-segmentation - miccai-2026 datasets: - biodatlab/Med-ReasonSeg --- # MedFuse-Seg: Multi-Level Visual and Semantic Context Fusion for Segmentation-Based Medical Reasoning (MICCAI 2026) [![Dataset](https://img.shields.io/badge/🤗%20Dataset-Med--ReasonSeg-yellow)](https://huggingface.co/datasets/biodatlab/Med-ReasonSeg) [![Code](https://img.shields.io/badge/💻%20Code-GitHub-blue)](https://github.com/biodatlab/medfuse-seg) MedFuse-Seg bridges the semantic-spatial gap in language-driven medical image analysis by combining multi-level visual feature injection with LLM-guided mask decoding. Built on MedGemma-4B, MedSigLIP, and MedSAM, the model allows clinicians to obtain both diagnostic reasoning and precise anatomical segmentation through natural language prompts. For full training details and hyperparameters, see the [project repository](https://github.com/biodatlab/medfuse-seg) and [Med-ReasonSeg dataset](https://huggingface.co/datasets/biodatlab/Med-ReasonSeg). ## Model Performance MedFuse-Seg outperforms zero-shot BiomedParse by 13.49% DSC and 54.04 px HD95, and fine-tuned LISA-7B (same training setup) by 4.89% DSC and 15.29 px HD95. | Method | DSC (Ref) | DSC (Sem) | DSC (Avg) | HD95 (Ref) | HD95 (Sem) | HD95 (Avg) | |--------|-----------|-----------|-----------|------------|------------|------------| | SAM 3 (zero-shot) | 0.1425 | 0.1167 | 0.1296 | 373.52 | 370.50 | 372.01 | | BiomedParse (zero-shot) | 0.6703 | 0.6344 | 0.6524 | 105.23 | 115.97 | 110.60 | | LISA-7B (fine-tuned) | 0.7398 | 0.7370 | 0.7384 | 71.55 | 72.13 | 71.84 | | **MedFuse-Seg (Ours)** | **0.7879** | **0.7867** | **0.7873** | **56.46** | **56.65** | **56.55** | > Evaluated on the Med-ReasonSeg test set. LISA-7B was retrained with identical training setup for a fair comparison. ## Download & Use ### 1. Install dependencies ```bash git clone https://github.com/biodatlab/medfuse-seg.git cd medfuse-seg pip install -r requirements.txt ``` ### 2. Download MedSAM checkpoint (required) Download from the [original MedSAM paper's repository](https://github.com/bowang-lab/MedSAM): ```bash gdown "https://drive.google.com/uc?id=1UAmWL88roYR7wKlnApw5Bcuzf2iQgk6_" ``` Place `m...

Note17 aoûthuggingface.co/biodatlab/medfuse-seg

Source context: 0 downloads · 1 likes · Pipeline image-segmentation · Repo biodatlab/medfuse-seg