Loading
Vous avez un workflow ComfyUI ? Obtenez une documentation lisible directement dans votre navigateur.Rendre un workflow lisible.Essayer maintenant ↗
This repository hosts checkpoints fine-tuned with Semantic Generative Tuning (SGT) — a training paradigm that couples visual understanding and generation in Unified Multimodal Models (UMMs) by using image...
Source du modèle
Description de la source
This repository hosts checkpoints fine-tuned with Semantic Generative Tuning (SGT) — a training paradigm that couples visual understanding and generation in Unified Multimodal Models (UMMs) by using image segmentation as a generative proxy.
Sources
1 sourceVérifié 15 sept.
Artefacts du modèle
6 artefactstransformer/text_encoder/model-00001-of-00004.safetensors
safetensors · 4,63 GB · SHA-256 6c7479219372…adc3 · Hugging Face
Téléchargertransformer/text_encoder/model-00002-of-00004.safetensors
safetensors · 4,59 GB · SHA-256 33bc88f9cfaf…6f86 · Hugging Face
TéléchargerExtraits de sources
2 extraitsUnified multimodal models typically optimize understanding and generation with misaligned objectives (sparse text tokens vs. dense pixel targets), which isolates the two capabilities. SGT introduces segmentation — a high-level semantic task — as a unified generative objective that aligns the two branches, improves feature linear separability, and optimizes visual-textual attention allocation.
SGT reformulates classical visual tasks as generative proxies and establishes a hierarchical taxonomy (low-/mid-/high-level). Extensive experiments show that high-level semantic tasks (e.g. image segmentation) are the optimal proxy, outperforming depth, edge, reconstruction and MAE/inpainting for synergizing understanding and generation.
Key findings:
| Repo | Type | Base Model | Content |
|---|---|---|---|
Two-hot/SGT-BAGEL | model | BAGEL-7B-MoT | SGT fine-tuned BAGEL checkpoint |
Two-hot/SGT-Gen2 | model | OmniGen2 | SGT fine-tuned OmniGen2 checkpoint (transformer/ only) |
Two-hot/SAM-SGT | dataset | — | Segmentation training data (tar-sharded) used by SGT |
See Two-hot/SAM-SGT for the data
layout and the extraction instructions.
Apache-2.0. Base models remain under their original licenses: BAGEL (Apache-2.0, based on Qwen2.5-7B + SigLIP + FLUX VAE) and OmniGen2 (based on Qwen2.5-VL + diffusion transformer).
If you find this work useful, please cite our paper:
@article{sgt2026,
title = {Semantic Generative Tuning for Unified Multimodal Models},
author = {Songsong Yu, Yuxin Chen, Ying Shan, and Yanwei Li},
journal = {arxiv},
year = {2026}
}
transformer/text_encoder/model-00003-of-00004.safetensors
safetensors · 4,59 GB · SHA-256 e9c67b8457f2…bf4c · Hugging Face
Téléchargertransformer/text_encoder/model-00004-of-00004.safetensors
safetensors · 172 MB · SHA-256 eb49a2f415de…a9fe · Hugging Face
Téléchargertransformer/transformer/diffusion_pytorch_model-00001-of-00002.safetensors
safetensors · 9,31 GB · SHA-256 96bde5280c80…f0d6 · Hugging Face
Téléchargertransformer/transformer/diffusion_pytorch_model-00002-of-00002.safetensors
safetensors · 5,47 GB · SHA-256 e95073a15f55…031b · Hugging Face
Télécharger--- license: apache-2.0 pipeline_tag: any-to-any library_name: bagel-mot tags: - sgt - semantic-generative-tuning - unified-multimodal - image-segmentation - visual-understanding - visual-generation --- # SGT: Semantic Generative Tuning for Unified Multimodal Models This repository hosts checkpoints fine-tuned with **Semantic Generative Tuning (SGT)** — a training paradigm that couples visual *understanding* and *generation* in Unified Multimodal Models (UMMs) by using **image segmentation as a generative proxy**. > Unified multimodal models typically optimize understanding and generation with *misaligned* > objectives (sparse text tokens vs. dense pixel targets), which isolates the two capabilities. > SGT introduces segmentation — a **high-level semantic task** — as a unified generative objective > that aligns the two branches, improves feature linear separability, and optimizes visual-textual > attention allocation. ## 🧠 Method Overview SGT reformulates classical visual tasks as generative proxies and establishes a **hierarchical taxonomy** (low-/mid-/high-level). Extensive experiments show that **high-level semantic tasks (e.g. image segmentation) are the optimal proxy**, outperforming depth, edge, reconstruction and MAE/inpainting for synergizing understanding and generation. Key findings: 1. **High-level > low-level**: segmentation gives larger gains in visual understanding than depth / edge / pixel reconstruction. 2. **Perception, not reasoning**: visual supervision mainly strengthens perception (spatial, hallucination, vision-centric, general VQA), rather than abstract reasoning (e.g. math, chart) 3. **Architecture-agnostic**: the gains hold for both **BAGEL** and **OmniGen2**. ## 📦 Released Artifacts | Repo | Type | Base Model | Content | |---|---|---|---| | [`Two-hot/SGT-BAGEL`](https://huggingface.co/Two-hot/SGT-BAGEL) | model | BAGEL-7B-MoT | SGT fine-tuned BAGEL checkpoint | | [`Two-hot/SGT-Gen2`](https://huggingface.co/Two-hot/SGT-Gen2) | model | OmniGen2 | SGT fine-tuned OmniGen2 checkpoint (transformer/ only) | | [`Two-hot/SAM-SGT`](https://huggingface.co/datasets/Two-hot/SAM-SGT) | dataset | — | Segmentation training data (tar-sharded) used by SGT | ### Use the SAM-SGT dataset See [`Two-hot/SAM-SGT`](https://huggingface.co/datasets/Two-hot/SAM-SGT) for the data layout and the extraction instructions. ## 📊 Highlights...
Source context: 0 downloads · 0 likes · Pipeline any-to-any · Library bagel-mot · Repo Two-hot/SGT-Gen2