Clearspeech Unet

• 🧹 AI-Powered Noise Reduction: Custom U-Net model trained to remove background noise
• 📝 Automatic Transcription: Whisper integration for accurate speech-to-text
• ⚡ Fast Processing: Optimized pipeline with GPU support
• 🌐 REST API: Easy-to-use FastAPI backend
• 🎯 High Quality: Val loss of 0.031
• 🔧 Flexible: Enhancement-only, transcription-only, or both

📋 Table of Contents

• Installation
• Quick Start
• API Documentation
• Project Structure
• Contributing

🚀 Installation

Prerequisites

• Python 3.8+

• pip

• Optional CUDA GPU

Step 1: Clone Repository

git clone https://github.com/yourusername/ClearSpeech.git
cd ClearSpeech

Step 2: Create Virtual Environment

# Create environment 
python3.10 -m venv venv 
# Activate (macOS/Linux) 
source venv/bin/activate 
# Activate (Windows) 
venv\Scripts\activate

Step 3: Install Dependencies

# Install dependencies
pip install -r requirements.txt

Step 4: Download Pretrained Model

# Download model
python -c "
from huggingface_hub import hf_hub_download
hf_hub_download(
    repo_id='thecodeworm/clearspeech-unet',
    filename='best_model.pt',
    local_dir='enhancement_model/checkpoints/'
)
"

Step 5: Generate Noisy Samples

1. Make your own WAV sample
2. Run the generate_noisy_samples.py file on the sample to make the audio noisier to test the model

# Generate all noise types at multiple SNR levvels
python generate_noisy_samples.py \
 --input my_clean_voice.wav \
 --output test_samples/

⚡ Quick Start

Method 1: Using the API (Recommended)

Start the server:

python -m backend.app

Server starts at http://localhost:8000

Start the server:

cd frontend
python -m http.server 3000

Frontend starts at http://localhost:3000

Process audio:

# Full pipeline (enhance + transcribe)
curl -X POST "http://localhost:8000/process" \
  -F "file=@your_audio.wav" \
  | jq .

# Enhance only
curl -X POST "http://localhost:8000/enhance" \
  -F "file=@your_audio.wav" \
  -o enhanced_output.wav

# Transcribe only
curl -X POST "http://localhost:8000/transcribe" \
  -F "file=@your_audio.wav" \
  -F "enhance=true" \
  | jq .

Method 2: Using Python

from backend.inference_pipeline import EnhancementPipeline

# Initialize pipeline
pipeline = EnhancementPipeline(
    cnn_checkpoint_path="enhancement_model/checkpoints/best_model.pt",
    whisper_model_name="base",
    device="cpu"  # or "cuda" or "mps"
)

# Process audio
result = pipeline.process("path/to/noisy_audio.wav")

print(f"Transcript: {result['transcript']}")
print(f"Duration: {result['duration']:.2f}s")

# Save enhanced audio
import soundfile as sf
sf.write("enhanced.wav", result['enhanced_audio'], result['sample_rate'])

Method 3: Command Line

# Enhance audio file
python enhancement_model/infer.py \
  --checkpoint enhancement_model/checkpoints/best_model.pt \
  --input noisy_audio.wav \
  --output enhanced_audio.wav \
  --comparison  # Creates stereo comparison file

📚 API Documentation

Interactive Docs

Once the server is running, visit:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Endpoints

POST /process

Process audio with enhancement and transcription.

Request:

  -F "[email protected]" \
  -F "language=en" \
  -F "skip_enhancement=false"

Response:

{
"success": true,
"transcript": "Transcribed text here",
"duration": 3.5,
"language": "en",
"enhanced_audio_url": "/download/enhanced_123.wav",
"segments": [...],
"processing_time": 2.3
}

POST /enhance

Enhance audio only (no transcription).

Request:

curl -X POST "http://localhost:8000/enhance" \
  -F "[email protected]" \
  -o enhanced.wav

Response: Enhanced audio file (WAV)

POST /transcribe

Transcribe audio with optional enhancement.

Request:

curl -X POST "http://localhost:8000/transcribe" \
  -F "[email protected]" \
  -F "language=en" \
  -F "enhance=true"

Response:

{
"success": true,
"transcript": "Transcribed text",
"duration": 3.5,
"language": "en",
"segments": [...]
}

GET /download/{filename}

Download enhanced audio file.

GET /health

Health check endpoint.

📁 Project Structure

ClearSpeech/
├── backend/                    # FastAPI backend
│   ├── app.py                 # Main API server
│   ├── inference_pipeline.py  # Processing pipeline
│   └── requirements.txt
├── enhancement_model/          # U-Net model
│   ├── model.py               # U-Net architecture
│   ├── dataset.py             # PyTorch dataset
│   ├── train.py               # Training script
│   ├── infer.py               # Inference script
│   ├── checkpoints/           # Trained models
│   │   └── best_model.pt
│   └── requirements.txt
├── data/                       # Training/test data
│   ├── audio_clean/           # Clean audio
│   ├── audio_raw/             # Noisy audio
│   ├── metadata/
│   │   └── metadata.json      # Dataset metadata
│   └── spectrograms/          # Mel-spectrograms
│       ├── clean/
│       └── noisy/
├── frontend/                   # Web interface (optional)
│   ├── index.html
│   └── script.js
├── tests/                      # Test files
│   └── test_backend.py
├── README.md
└── requirements.txt

🤝 Contributing

We welcome contributions! Here's how:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open a Pull Request

Development Setup

# Install dev dependencies
pip install -r requirements-dev.txt

# Run tests before committing
python -m pytest tests/

# Format code
black backend/ enhancement_model/

🙏 Acknowledgments

• U-Net Architecture: Inspired by Ronneberger et al.
• Whisper: OpenAI Whisper
• Training Data: LibriSpeech, MS-SNSD

📧 Contact

Project Maintainers: Aditya Chanda, Josh Pal, Advik Kumar Singh

Project Link: https://github.com/thecodeworm/ClearSpeech

⭐ Show Your Support

Give a ⭐️ if this project helped you!

Built with ❤️ using PyTorch and FastAPI