ModelPublicadoExterno

Fine Grained Soundscape Control

Pretrained TSE (Target Sound Extraction) models from MobiSys 2026 paper.

Fuentehuggingface.co/ooshyun/fine_grained_soundscape_control/tree/5197565959392e3a56376908cfb4a984c0f7c768

TipoOtherVersiónRevision 5197565Ccalidad

Fuente del modelo

Tipo: Other
Tamaño del archivo: 6,64–21,7 MB
Versión: vRevision 5197565
Actualizado: 11 oct 2026
Descargas: 0
Fuente: Hugging Face

Descripción de la fuente

Fine-Grained Soundscape Control for Augmented Hearing

Pretrained TSE (Target Sound Extraction) models from MobiSys 2026 paper.

These models extract target sounds from binaural mixtures in real-time (~10ms latency), conditioned on sound class labels via FiLM (Feature-wise Linear Modulation).

Paper

Fine-Grained Soundscape Control for Augmented Hearing arXiv:2603.00395 Proceedings of the 24th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys '26), 2026.

Models

Table 1: TSE Model Comparison (1ch output, single source)

Model	Architecture	D	H	B	Params	SNRi (dB)	SI-SDRi (dB)
Orange Pi	TFGridNet	32	64	6	1.9M	12.26	10.16
Raspberry Pi	TFGridNet	16	64	3	0.5M	10.81	8.89
NeuralAids	TFMLPNet	32	32	6	0.7M	11.57	9.44
Waveformer	Waveformer	-	-	-	1.7M	8.32	6.51

Table 2: Multi-output Models (Orange Pi, 5 sources)

Outputs	SNRi (dB)	SI-SDRi (dB)
5-out	12.26	10.16
20-out	11.89	9.72

Table 3: FiLM Ablation

Model	FiLM	SNRi (dB)	SI-SDRi (dB)
Orange Pi	First	11.43	9.28
Orange Pi	All	12.31	10.18
Orange Pi	All-except-first	12.26	10.16
NeuralAids	First	10.62	8.58
NeuralAids	All	11.62	9.51
NeuralAids	All-except-first	11.57	9.44

Usage

from huggingface_hub import hf_hub_download
import torch, json

# Download model + config
repo_id = "ooshyun/fine_grained_soundscape_control"
model_name = "orange_pi"  # or: raspberry_pi, neuralaid, orange_pi_film_all, etc.

# Using the evaluation script (recommended):
# pip install -r requirements.txt
# python -m src.tse.eval --pretrained ooshyun/fine_grained_soundscape_control --model orange_pi --data_dir /path/to/data_dir

Model Names

Short Name	Description	HF Directory
`orange_pi`	Table 1 default (1ch, film=all-ex-1st)	`tfgridnet_large_..._1ch_..._film_all_except_first_...`
`raspberry_pi`	Table 1 small (1ch)	`tfgridnet_small_..._1ch_...`
`neuralaid`	Table 1 TFMLPNet (1ch)	`tfmlpnet_..._1ch_...`
`waveformer`	Table 1 baseline (1ch)	`waveformer_...`
`orange_pi_5out`	Table 2 (5ch, 5out)	`tfgridnet_large_..._5ch_5spk_5out_..._film_all_except_first_...`
`orange_pi_20out`	Table 2 (20ch, 20out)	`tfgridnet_large_..._20ch_5spk_20out_...`
`orange_pi_film_first`	Table 3 FiLM=first	`tfgridnet_large_..._5ch_..._film_first_...`
`orange_pi_film_all`	Table 3 FiLM=all	`tfgridnet_large_..._5ch_..._film_all_...`
`orange_pi_film_all_except_first`	Table 3 FiLM=all-ex-1st	`tfgridnet_large_..._5ch_..._film_all_except_first_...`
`neuralaid_film_first`	Table 3 NeuralAids first	`tfmlpnet_..._film_first_...`
`neuralaid_film_all`	Table 3 NeuralAids all	`tfmlpnet_..._film_all_layers_6_...`
`neuralaid_film_all_except_first`	Table 3 NeuralAids all-ex-1st	`tfmlpnet_..._film_all_except_first_...`

STFT Configuration

All models use causal STFT with:

Sample rate: 16 kHz
Chunk size: 96 samples (6 ms)
Lookahead: 64 samples (4 ms)
Lookback: 96 samples (6 ms)
FFT size: 256
Algorithmic latency: 10 ms

Training Data

Models are trained on BinauralCuratedDataset — on-the-fly binaural synthesis from 6 public datasets (FSD50K, ESC-50, musdb18, DISCO, TAU-2019, CIPIC HRTF).

Code

github.com/ooshyun/fine_grained_soundscape_control

Citation

@inproceedings{oh2026fine,
  title     = {Fine-Grained Soundscape Control for Augmented Hearing},
  booktitle = {Proceedings of the 24th ACM International Conference on
               Mobile Systems, Applications, and Services (MobiSys '26)},
  year      = {2026},
}

Fuentes

1 fuente

Hugging FacePrincipalVersión 5197565959392e3a56376908cfb4a984c0f7c768

huggingface.co/ooshyun/fine_grained_soundscape_control/tree/5197565959392e3a56376908cfb4a984c0f7c768

Verificado 14 ago

Artefactos del modelo

22 artefactos

tfgridnet_large_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt

pt · 8,97 MB · SHA-256 da4df626b2fe…88ef · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt

pt · 10,0 MB · SHA-256 f31c74014305…1e87 · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_20ch_5spk_20out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt

pt · 9,14 MB · SHA-256 91e7ba3929aa…47db · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_20ch_5spk_20out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt

pt · 10,9 MB · SHA-256 dc98e391b0f4…e8b3 · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt

pt · 10,7 MB · SHA-256 70a046bb1d7e…e300 · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt

pt · 10,8 MB · SHA-256 59545e7cfb12…a342 · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_onflight/checkpoints/best.pt

pt · 9,50 MB · SHA-256 ef09d011e0b9…6b93 · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_onflight/checkpoints/last.pt

pt · 10,8 MB · SHA-256 bee69817fb28…bf5e · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_first_onflight/checkpoints/best.pt

pt · 8,97 MB · SHA-256 33ad786cd9c9…72e4 · Hugging Face

Descargar

tfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_first_onflight/checkpoints/last.pt

pt · 9,22 MB · SHA-256 3b70f9b074da…facb · Hugging Face

Descargar

tfgridnet_small_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt

pt · 6,64 MB · SHA-256 a708413daf8a…3d16 · Hugging Face

Descargar

tfgridnet_small_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt

pt · 9,90 MB · SHA-256 38e9c07a39d9…455f · Hugging Face

Descargar

tfmlpnet_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt

pt · 7,42 MB · SHA-256 8789cbe35f8a…5279 · Hugging Face

Descargar

tfmlpnet_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt

pt · 10,9 MB · SHA-256 57a0ccb109fd…6b95 · Hugging Face

Descargar

tfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt

pt · 8,61 MB · SHA-256 09a67b923ce5…a024 · Hugging Face

Descargar

tfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt

pt · 8,80 MB · SHA-256 5963768150f7…fe6e · Hugging Face

Descargar

tfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_layers_6_onflight/checkpoints/best.pt

pt · 7,87 MB · SHA-256 26d84752a561…5322 · Hugging Face

Descargar

tfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_layers_6_onflight/checkpoints/last.pt

pt · 10,2 MB · SHA-256 b444126cb9d0…9a0d · Hugging Face

Descargar

tfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_first_onflight/checkpoints/best.pt

pt · 8,95 MB · SHA-256 395d003c7e5e…e240 · Hugging Face

Descargar

tfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_first_onflight/checkpoints/last.pt

pt · 9,61 MB · SHA-256 191b55ec946b…d1d0 · Hugging Face

Descargar

waveformer_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_256chunk_film_all_onflight/checkpoints/best.pt

pt · 21,6 MB · SHA-256 dcab2d523ccb…df60 · Hugging Face

Descargar

waveformer_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_256chunk_film_all_onflight/checkpoints/last.pt

pt · 21,7 MB · SHA-256 d1064524a91c…b1df · Hugging Face

Descargar

Extractos de fuentes

2 extractos

Readme14 agohuggingface.co/ooshyun/fine_grained_soundscape_control

--- license: mit pipeline_tag: audio-to-audio tags: - audio - target-sound-extraction - source-separation - tfgridnet - real-time - binaural - augmented-hearing datasets: - custom language: - en --- # Fine-Grained Soundscape Control for Augmented Hearing Pretrained TSE (Target Sound Extraction) models from **MobiSys 2026** paper. These models extract target sounds from binaural mixtures in real-time (~10ms latency), conditioned on sound class labels via FiLM (Feature-wise Linear Modulation). ## Paper > **Fine-Grained Soundscape Control for Augmented Hearing** > [arXiv:2603.00395](https://arxiv.org/abs/2603.00395) > Proceedings of the 24th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys '26), 2026. ## Models ### Table 1: TSE Model Comparison (1ch output, single source) | Model | Architecture | D | H | B | Params | SNRi (dB) | SI-SDRi (dB) | |-------|-------------|---|---|---|--------|-----------|-------------| | Orange Pi | TFGridNet | 32 | 64 | 6 | 1.9M | 12.26 | 10.16 | | Raspberry Pi | TFGridNet | 16 | 64 | 3 | 0.5M | 10.81 | 8.89 | | NeuralAids | TFMLPNet | 32 | 32 | 6 | 0.7M | 11.57 | 9.44 | | Waveformer | Waveformer | - | - | - | 1.7M | 8.32 | 6.51 | ### Table 2: Multi-output Models (Orange Pi, 5 sources) | Outputs | SNRi (dB) | SI-SDRi (dB) | |---------|-----------|-------------| | 5-out | 12.26 | 10.16 | | 20-out | 11.89 | 9.72 | ### Table 3: FiLM Ablation | Model | FiLM | SNRi (dB) | SI-SDRi (dB) | |-------|------|-----------|-------------| | Orange Pi | First | 11.43 | 9.28 | | Orange Pi | All | 12.31 | 10.18 | | Orange Pi | All-except-first | 12.26 | 10.16 | | NeuralAids | First | 10.62 | 8.58 | | NeuralAids | All | 11.62 | 9.51 | | NeuralAids | All-except-first | 11.57 | 9.44 | ## Usage ```python from huggingface_hub import hf_hub_download import torch, json # Download model + config repo_id = "ooshyun/fine_grained_soundscape_control" model_name = "orange_pi" # or: raspberry_pi, neuralaid, orange_pi_film_all, etc. # Using the evaluation script (recommended): # pip install -r requirements.txt # python -m src.tse.eval --pretrained ooshyun/fine_grained_soundscape_control --model orange_pi --data_dir /path/to/data_dir ``` ## Model Names | Short Name | Description | HF Directory | |------------|-------------|--------------| | `orange_pi` | Table 1 default (1ch, film=all-ex-1st) | `tfgrid...

Note14 agohuggingface.co/ooshyun/fine_grained_soundscape_control

Source context: 0 downloads · 0 likes · Pipeline audio-to-audio · Repo ooshyun/fine_grained_soundscape_control