Fine Grained Soundscape Control
Pretrained TSE (Target Sound Extraction) models from MobiSys 2026 paper.
Modellquelle
- Art
- Other
- Dateigröße
- 6,64–21,7 MB
- Version
- vRevision 5197565
- Aktualisiert
- 11. Okt. 2026
- Downloads
- 0
- Quelle
- Hugging Face
Quellenbeschreibung
Fine-Grained Soundscape Control for Augmented Hearing
Pretrained TSE (Target Sound Extraction) models from MobiSys 2026 paper.
These models extract target sounds from binaural mixtures in real-time (~10ms latency), conditioned on sound class labels via FiLM (Feature-wise Linear Modulation).
Paper
Fine-Grained Soundscape Control for Augmented Hearing arXiv:2603.00395 Proceedings of the 24th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys '26), 2026.
Models
Table 1: TSE Model Comparison (1ch output, single source)
| Model | Architecture | D | H | B | Params | SNRi (dB) | SI-SDRi (dB) |
|---|---|---|---|---|---|---|---|
| Orange Pi | TFGridNet | 32 | 64 | 6 | 1.9M | 12.26 | 10.16 |
| Raspberry Pi | TFGridNet | 16 | 64 | 3 | 0.5M | 10.81 | 8.89 |
| NeuralAids | TFMLPNet | 32 | 32 | 6 | 0.7M | 11.57 | 9.44 |
| Waveformer | Waveformer | - | - | - | 1.7M | 8.32 | 6.51 |
Table 2: Multi-output Models (Orange Pi, 5 sources)
| Outputs | SNRi (dB) | SI-SDRi (dB) |
|---|---|---|
| 5-out | 12.26 | 10.16 |
| 20-out | 11.89 | 9.72 |
Table 3: FiLM Ablation
| Model | FiLM | SNRi (dB) | SI-SDRi (dB) |
|---|---|---|---|
| Orange Pi | First | 11.43 | 9.28 |
| Orange Pi | All | 12.31 | 10.18 |
| Orange Pi | All-except-first | 12.26 | 10.16 |
| NeuralAids | First | 10.62 | 8.58 |
| NeuralAids | All | 11.62 | 9.51 |
| NeuralAids | All-except-first | 11.57 | 9.44 |
Usage
from huggingface_hub import hf_hub_download
import torch, json
# Download model + config
repo_id = "ooshyun/fine_grained_soundscape_control"
model_name = "orange_pi" # or: raspberry_pi, neuralaid, orange_pi_film_all, etc.
# Using the evaluation script (recommended):
# pip install -r requirements.txt
# python -m src.tse.eval --pretrained ooshyun/fine_grained_soundscape_control --model orange_pi --data_dir /path/to/data_dir
Model Names
| Short Name | Description | HF Directory |
|---|---|---|
orange_pi | Table 1 default (1ch, film=all-ex-1st) | tfgridnet_large_..._1ch_..._film_all_except_first_... |
raspberry_pi | Table 1 small (1ch) | tfgridnet_small_..._1ch_... |
neuralaid | Table 1 TFMLPNet (1ch) | tfmlpnet_..._1ch_... |
waveformer | Table 1 baseline (1ch) | waveformer_... |
orange_pi_5out | Table 2 (5ch, 5out) | tfgridnet_large_..._5ch_5spk_5out_..._film_all_except_first_... |
orange_pi_20out | Table 2 (20ch, 20out) | tfgridnet_large_..._20ch_5spk_20out_... |
orange_pi_film_first | Table 3 FiLM=first | tfgridnet_large_..._5ch_..._film_first_... |
orange_pi_film_all | Table 3 FiLM=all | tfgridnet_large_..._5ch_..._film_all_... |
orange_pi_film_all_except_first | Table 3 FiLM=all-ex-1st | tfgridnet_large_..._5ch_..._film_all_except_first_... |
neuralaid_film_first | Table 3 NeuralAids first | tfmlpnet_..._film_first_... |
neuralaid_film_all | Table 3 NeuralAids all | tfmlpnet_..._film_all_layers_6_... |
neuralaid_film_all_except_first | Table 3 NeuralAids all-ex-1st | tfmlpnet_..._film_all_except_first_... |
STFT Configuration
All models use causal STFT with:
- Sample rate: 16 kHz
- Chunk size: 96 samples (6 ms)
- Lookahead: 64 samples (4 ms)
- Lookback: 96 samples (6 ms)
- FFT size: 256
- Algorithmic latency: 10 ms
Training Data
Models are trained on BinauralCuratedDataset — on-the-fly binaural synthesis from 6 public datasets (FSD50K, ESC-50, musdb18, DISCO, TAU-2019, CIPIC HRTF).
Code
github.com/ooshyun/fine_grained_soundscape_control
Citation
@inproceedings{oh2026fine,
title = {Fine-Grained Soundscape Control for Augmented Hearing},
booktitle = {Proceedings of the 24th ACM International Conference on
Mobile Systems, Applications, and Services (MobiSys '26)},
year = {2026},
}
Quellen
1 Quelle
Quellen
1 QuelleVerifiziert 14. Aug.
Modellartefakte
22 Artefakte
Modellartefakte
22 Artefaktetfgridnet_large_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt
pt · 8,97 MB · SHA-256 da4df626b2fe…88ef · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt
pt · 10,0 MB · SHA-256 f31c74014305…1e87 · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_20ch_5spk_20out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt
pt · 9,14 MB · SHA-256 91e7ba3929aa…47db · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_20ch_5spk_20out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt
pt · 10,9 MB · SHA-256 dc98e391b0f4…e8b3 · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt
pt · 10,7 MB · SHA-256 70a046bb1d7e…e300 · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt
pt · 10,8 MB · SHA-256 59545e7cfb12…a342 · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_onflight/checkpoints/best.pt
pt · 9,50 MB · SHA-256 ef09d011e0b9…6b93 · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_onflight/checkpoints/last.pt
pt · 10,8 MB · SHA-256 bee69817fb28…bf5e · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_first_onflight/checkpoints/best.pt
pt · 8,97 MB · SHA-256 33ad786cd9c9…72e4 · Hugging Face
Herunterladentfgridnet_large_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_first_onflight/checkpoints/last.pt
pt · 9,22 MB · SHA-256 3b70f9b074da…facb · Hugging Face
Herunterladentfgridnet_small_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt
pt · 6,64 MB · SHA-256 a708413daf8a…3d16 · Hugging Face
Herunterladentfgridnet_small_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt
pt · 9,90 MB · SHA-256 38e9c07a39d9…455f · Hugging Face
Herunterladentfmlpnet_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt
pt · 7,42 MB · SHA-256 8789cbe35f8a…5279 · Hugging Face
Herunterladentfmlpnet_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt
pt · 10,9 MB · SHA-256 57a0ccb109fd…6b95 · Hugging Face
Herunterladentfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/best.pt
pt · 8,61 MB · SHA-256 09a67b923ce5…a024 · Hugging Face
Herunterladentfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_except_first_onflight/checkpoints/last.pt
pt · 8,80 MB · SHA-256 5963768150f7…fe6e · Hugging Face
Herunterladentfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_layers_6_onflight/checkpoints/best.pt
pt · 7,87 MB · SHA-256 26d84752a561…5322 · Hugging Face
Herunterladentfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_all_layers_6_onflight/checkpoints/last.pt
pt · 10,2 MB · SHA-256 b444126cb9d0…9a0d · Hugging Face
Herunterladentfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_first_onflight/checkpoints/best.pt
pt · 8,95 MB · SHA-256 395d003c7e5e…e240 · Hugging Face
Herunterladentfmlpnet_snr_ctl_v2_5ch_5spk_5out_20000samples_20sounds_16000sr_96chunk_film_first_onflight/checkpoints/last.pt
pt · 9,61 MB · SHA-256 191b55ec946b…d1d0 · Hugging Face
Herunterladenwaveformer_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_256chunk_film_all_onflight/checkpoints/best.pt
pt · 21,6 MB · SHA-256 dcab2d523ccb…df60 · Hugging Face
Herunterladenwaveformer_snr_ctl_v2_1ch_1spk_1out_20000samples_20sounds_16000sr_256chunk_film_all_onflight/checkpoints/last.pt
pt · 21,7 MB · SHA-256 d1064524a91c…b1df · Hugging Face
HerunterladenQuellenauszüge
2 Auszüge
Quellenauszüge
2 Auszüge--- license: mit pipeline_tag: audio-to-audio tags: - audio - target-sound-extraction - source-separation - tfgridnet - real-time - binaural - augmented-hearing datasets: - custom language: - en --- # Fine-Grained Soundscape Control for Augmented Hearing Pretrained TSE (Target Sound Extraction) models from **MobiSys 2026** paper. These models extract target sounds from binaural mixtures in real-time (~10ms latency), conditioned on sound class labels via FiLM (Feature-wise Linear Modulation). ## Paper > **Fine-Grained Soundscape Control for Augmented Hearing** > [arXiv:2603.00395](https://arxiv.org/abs/2603.00395) > Proceedings of the 24th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys '26), 2026. ## Models ### Table 1: TSE Model Comparison (1ch output, single source) | Model | Architecture | D | H | B | Params | SNRi (dB) | SI-SDRi (dB) | |-------|-------------|---|---|---|--------|-----------|-------------| | Orange Pi | TFGridNet | 32 | 64 | 6 | 1.9M | 12.26 | 10.16 | | Raspberry Pi | TFGridNet | 16 | 64 | 3 | 0.5M | 10.81 | 8.89 | | NeuralAids | TFMLPNet | 32 | 32 | 6 | 0.7M | 11.57 | 9.44 | | Waveformer | Waveformer | - | - | - | 1.7M | 8.32 | 6.51 | ### Table 2: Multi-output Models (Orange Pi, 5 sources) | Outputs | SNRi (dB) | SI-SDRi (dB) | |---------|-----------|-------------| | 5-out | 12.26 | 10.16 | | 20-out | 11.89 | 9.72 | ### Table 3: FiLM Ablation | Model | FiLM | SNRi (dB) | SI-SDRi (dB) | |-------|------|-----------|-------------| | Orange Pi | First | 11.43 | 9.28 | | Orange Pi | All | 12.31 | 10.18 | | Orange Pi | All-except-first | 12.26 | 10.16 | | NeuralAids | First | 10.62 | 8.58 | | NeuralAids | All | 11.62 | 9.51 | | NeuralAids | All-except-first | 11.57 | 9.44 | ## Usage ```python from huggingface_hub import hf_hub_download import torch, json # Download model + config repo_id = "ooshyun/fine_grained_soundscape_control" model_name = "orange_pi" # or: raspberry_pi, neuralaid, orange_pi_film_all, etc. # Using the evaluation script (recommended): # pip install -r requirements.txt # python -m src.tse.eval --pretrained ooshyun/fine_grained_soundscape_control --model orange_pi --data_dir /path/to/data_dir ``` ## Model Names | Short Name | Description | HF Directory | |------------|-------------|--------------| | `orange_pi` | Table 1 default (1ch, film=all-ex-1st) | `tfgrid...
Source context: 0 downloads · 0 likes · Pipeline audio-to-audio · Repo ooshyun/fine_grained_soundscape_control
