Omura Embed Audio
A small linear adapter head trained on top of frozen laion/largerclapgeneral (CLAP) audio embeddings, used by Omura for natural-language audio search over the Walrus protocol's audio corpus.
Fuente del modelo
- Tipo
- Other
- Tamaño del archivo
- 1,00 MB
- Modelo base
- laion/larger_clap_general
- Versión
- vOmura clap head
- Actualizado
- 19 oct 2026
- Fuente
- Hugging Face
Descripción de la fuente
omura-embed-audio
A small linear adapter head trained on top of frozen laion/larger_clap_general (CLAP) audio embeddings, used by Omura for natural-language audio search over the Walrus protocol's audio corpus.
Important — what this repo is and isn't:
- This repo contains only the trained adapter head (
omura_clap_head.pt, ~1 MB — a single residual linear layer). It is not a re-upload of CLAP's weights, and the base CLAP model is unmodified. - All credit for the CLAP architecture and pretraining belongs to
LAION-AI. Use this adapter only
together with the original
laion/larger_clap_generalcheckpoint.
What was trained
A single Linear(512, 512) residual adapter (out = normalize(x + W x)) applied to
CLAP's frozen audio embeddings, trained with a cross-entropy contrastive objective
against CLAP's frozen text-class embeddings.
Training data: ESC-50 folds 1-4 (1,600 clips). Fold 5 (400 clips) was never seen during training and is used exclusively for evaluation below, so these numbers reflect genuine held-out generalization, not memorization of the training set.
Results (ESC-50, held-out fold 5 only)
| Accuracy | |
|---|---|
| CLAP zero-shot (no adapter) | 85.25% |
| + omura-embed-audio adapter | 95.75% |
Full reproduction script: benchmarks/eval/clap/finetune_esc50_head.py in the
omura-backend repo; see also
BENCHMARK_REPRODUCTION.md there.
(For reference, the full-dataset zero-shot number Omura also reports elsewhere — 86.65% — evaluates CLAP across all 2000 ESC-50 clips including the folds used to train this adapter, so it isn't directly comparable to the 85.25%/95.75% held-out figures above; both are honestly reported here for transparency.)
Usage
import torch, torch.nn.functional as F
from transformers import ClapModel, ClapProcessor
base = ClapModel.from_pretrained("laion/larger_clap_general")
processor = ClapProcessor.from_pretrained("laion/larger_clap_general")
ckpt = torch.load("omura_clap_head.pt", map_location="cpu")
head_w = ckpt["state_dict"] # {"proj.weight": ..., "proj.bias": ...}
def apply_head(x, w, b):
return F.normalize(x + x @ w.T + b, dim=-1)
# audio_emb = base.get_audio_features(**inputs) # frozen CLAP embedding
# adapted = apply_head(audio_emb, head_w["proj.weight"], head_w["proj.bias"])
License
Apache 2.0 for this adapter. The base CLAP model is subject to LAION's own license terms.
Attribution
Base model: laion/larger_clap_general by LAION-AI. This repo is a small trained adapter on top of that frozen model — no CLAP weights are included or modified here.
Fuentes
1 fuente
Fuentes
1 fuenteVerificado 19 oct
Artefactos del modelo
1 artefacto
Artefactos del modelo
1 artefactoExtractos de fuentes
2 extractos
Extractos de fuentes
2 extractos--- license: apache-2.0 base_model: laion/larger_clap_general tags: - omura-embed - audio-text-retrieval - clap - audio-embeddings pipeline_tag: audio-classification --- # omura-embed-audio A small linear adapter head trained on top of **frozen** [laion/larger_clap_general](https://huggingface.co/laion/larger_clap_general) (CLAP) audio embeddings, used by Omura for natural-language audio search over the Walrus protocol's audio corpus. **Important — what this repo is and isn't:** - This repo contains **only the trained adapter head** (`omura_clap_head.pt`, ~1 MB — a single residual linear layer). It is **not** a re-upload of CLAP's weights, and the base CLAP model is unmodified. - All credit for the CLAP architecture and pretraining belongs to [LAION-AI](https://huggingface.co/laion/larger_clap_general). Use this adapter only together with the original `laion/larger_clap_general` checkpoint. ## What was trained A single `Linear(512, 512)` residual adapter (`out = normalize(x + W x)`) applied to CLAP's frozen audio embeddings, trained with a cross-entropy contrastive objective against CLAP's frozen text-class embeddings. **Training data:** ESC-50 folds 1-4 (1,600 clips). **Fold 5 (400 clips) was never seen during training** and is used exclusively for evaluation below, so these numbers reflect genuine held-out generalization, not memorization of the training set. ## Results (ESC-50, held-out fold 5 only) | | Accuracy | |---|---| | CLAP zero-shot (no adapter) | 85.25% | | **+ omura-embed-audio adapter** | **95.75%** | Full reproduction script: `benchmarks/eval/clap/finetune_esc50_head.py` in the [omura-backend repo](https://github.com/OmuraHQ/omura-backend); see also `BENCHMARK_REPRODUCTION.md` there. (For reference, the full-dataset zero-shot number Omura also reports elsewhere — 86.65% — evaluates CLAP across all 2000 ESC-50 clips including the folds used to train this adapter, so it isn't directly comparable to the 85.25%/95.75% held-out figures above; both are honestly reported here for transparency.) ## Usage ```python import torch, torch.nn.functional as F from transformers import ClapModel, ClapProcessor base = ClapModel.from_pretrained("laion/larger_clap_general") processor = ClapProcessor.from_pretrained("laion/larger_clap_general") ckpt = torch.load("omura_clap_head.pt", map_location="cpu") head_w = ckpt["state_dict"] # {"proj...
Source context: 0 downloads · 0 likes · Pipeline audio-classification · Repo immortaltatsu/omura-embed-audio
