PanoSAMic

What is in this repository

Only the trainable PanoSAMic components are hosted here:

• Feature fusion blocks (MCBAM) — spatio-modal cross-attention applied to the branch features extracted by the frozen encoder
• Semantic decoder — convolutional decoder with spherical attention and dual-view fusion head

The full model state dict has two parts:

The frozen SAM ViT backbone is NOT hosted here. It is downloaded separately from Meta's official release (Apache-2.0) and combined at load time. This keeps each checkpoint small and avoids redistributing the SAM weights.

Available checkpoints

Each variant lives in its own subfolder of dfki-av/PanoSAMic (e.g. stanford2d3ds-vith-rgbdn-fold1/model.safetensors). 3-fold checkpoints are published per fold so each can be evaluated on its held-out split.

Reported results

Stanford2D3DS (3-fold validation), main table:

Encoder-size study (Stanford2D3DS, 3-fold, RGB-D-N):

Matterport3D (BEV360 splits):

ToF-360 (zero-shot, fold 1, instance-refined):

No checkpoint is trained on ToF-360 (only 179 real-world samples across 4 scenes — too little to train on). Instead, the Stanford2D3DS ViT-H fold-1 checkpoints are evaluated directly against it (single split, not averaged across folds), with ToF-360's own label ontology remapped to Stanford2D3DS's 13 classes at load time and SAM instance masks used to refine the semantic predictions. Adding depth/normals hurts zero-shot transfer here, likely due to the domain gap between ToF-360's real sensor depth and Stanford2D3DS's.

How to reproduce

1. Environment

• Python 3.11+
• Install with uv sync from the GitHub repo (pyproject.toml pins dependencies)
• 1× GPU with ≥16 GB VRAM for ViT-H inference (≥24 GB for training)

2. Get the frozen SAM backbone

Download the official SAM weights from Meta and place them in sam_weights/:

• sam_vit_h_4b8939.pth
• sam_vit_l_0b3195.pth
• sam_vit_b_01ec64.pth

(See https://github.com/facebookresearch/segment-anything#model-checkpoints)

3. Load a checkpoint

from panosamic.model import PanoSAMic

model = PanoSAMic.from_pretrained_panosamic(
    "dfki-av/PanoSAMic",
    subfolder="stanford2d3ds-vith-rgbdn-fold1",
    config_path="config/config_stanford2d3ds_dv.json",
    vit_model="vit_h",
    modalities=("image", "depth", "normals"),
    num_classes=13,
    sam_weights_path="./sam_weights",  # omit to auto-download from Meta's servers
)

from_pretrained_panosamic loads only the trainable weights from the Hub, initialises the frozen SAM backbone from the local sam_weights/ directory (auto-downloaded if not present), and returns the model in eval() mode.

4. Run inference

import torch
from panosamic.model.instance_semantic_fusion import refine_semantic_with_instances

# batched_input: list of dicts, one per image.
# Each dict maps modality name → float tensor (3, H, W), values in [0, 255].
# Image must be equirectangular 2:1 (e.g. 512 × 1024).
batched_input = [{"image": image_tensor, "depth": depth_tensor, "normals": normals_tensor}]

with torch.no_grad():
    outputs = model(batched_input)

sem_preds = outputs[0]["sem_preds"]        # (num_classes, H, W) — logits
instance_masks = outputs[0]["instance_masks"]

# Instance-guided refinement: each SAM mask is assigned the majority
# semantic class within it, sharpening boundaries.
if instance_masks:
    sem_preds = refine_semantic_with_instances(sem_preds, instance_masks)

seg_map = sem_preds.argmax(dim=0)  # (H, W) — integer class indices

5. Prepare the data

Use the exact splits reported in the paper:

• Stanford2D3DS: the authors' 3-fold cross-validation splits. Source: https://github.com/alexsax/2D-3D-Semantics . Preprocess with panosamic/data_preparation/ into the processed structure documented in the repo README.
• Matterport3D: the BEV360 pre-processed data and splits (20-class subset) for a fair comparison. Source: https://github.com/InSAI-Lab/360BEV .
• ToF-360: used for zero-shot evaluation only, no training split. Source: https://huggingface.co/datasets/COLE-Ricoh/ToF-360 .

6. Run evaluation

From a released Hub checkpoint (trainable weights only, SAM loaded separately):

python panosamic/evaluation/evaluate.py \
    --dataset_path /path/to/processed/dataset \
    --config_path config/config_stanford2d3ds_dv.json \
    --checkpoint dfki-av/PanoSAMic \
    --subfolder stanford2d3ds-vith-rgbdn-fold1 \
    --sam_weights_path ./sam_weights \
    --dataset stanford2d3ds \
    --fold 1 \
    --vit_model vit_h \
    --modalities image,depth,normals \
    --num_gpus 1

From a local training run (full checkpoint including frozen backbone):

python panosamic/evaluation/evaluate.py \
    --dataset_path /path/to/processed/dataset \
    --config_path config/config_stanford2d3ds_dv.json \
    --experiments_path ./experiments \
    --dataset stanford2d3ds \
    --fold 1 \
    --vit_model vit_h \
    --modalities image,depth,normals \
    --num_gpus 1

Repeat for folds 1–3 and average for the 3-fold numbers. For Matterport3D use config/config_matterport3d_dv.json, --dataset matterport3d, and the modalities for that row.

Zero-shot on ToF-360 (no fine-tuning, fold-1 Stanford2D3DS checkpoint):

python panosamic/evaluation/evaluate.py \
    --dataset_path /path/to/processed/tof360 \
    --config_path config/config_tof360_dv.json \
    --checkpoint dfki-av/PanoSAMic \
    --subfolder stanford2d3ds-vith-rgb-fold1 \
    --sam_weights_path ./sam_weights \
    --dataset tof360 \
    --fold 1 \
    --vit_model vit_h \
    --modalities image \
    --num_gpus 1

Swap --subfolder/--modalities (stanford2d3ds-vith-rgbd-fold1 + image,depth, stanford2d3ds-vith-rgbdn-fold1 + image,depth,normals) to reproduce the other rows above. Unlike the Stanford2D3DS/Matterport3D tables, ToF-360 numbers are fold-1 only, not averaged across folds.

7. Key configuration (matches the paper)

• Frozen SAM ViT-H, encoder depth 32, global attention at blocks [8, 16, 24, 32]
• Batch size 8, 50 epochs, Ranger21 optimizer
• Max LR 0.0005 (Stanford2D3DS) / 0.001 (Matterport3D)
• Input resized to 512 × 1024
• MCBAM window 8×8, stride 4; spherical attention kernel 7×7, stride 1
• Dual-view shift s = W/2
• Loss: Jaccard (Stanford2D3DS); alternating Cross-Entropy/Jaccard schedule (Matterport3D)
• Depth preprocessed to pseudo-disparity (threshold = 99.5th percentile of train depths, rounded to nearest 10 cm), replicated to 3 channels

Intended use and limitations

Indoor panoramic semantic segmentation with RGB / RGB-D / RGB-D-N input. Evaluated only on indoor datasets; outdoor generalization is not guaranteed.

License and access terms

• This model card and the released trainable weights: CC BY-NC-SA 4.0 (Attribution–NonCommercial–ShareAlike). Use is restricted to non-commercial purposes.
• The frozen SAM backbone (downloaded separately) remains under its original Apache-2.0 license from Meta AI.

Citation

@article{chamseddine2026panosamic,
  title   = {PanoSAMic: Panoramic Image Segmentation from SAM Feature Encoding and Dual View Fusion},
  author  = {Chamseddine, Mahdi and Stricker, Didier and Rambach, Jason},
  journal = {arXiv preprint arXiv:2601.07447},
  year    = {2026}
}

Acknowledgement

Funded by the European Union as part of the projects HumanTech (Grant Agreement 101058236) and ShieldBOT (Grant Agreement 101235093).

stanford2d3ds-vitb-rgbdn-fold1/model.safetensors

safetensors · 791 MB · SHA-256 aae73775e3a7…873e · Hugging Face