CosyVoice2 KO SFT v6 Epoch3

CosyVoice2 Korean SFT v6 - Epoch 3

CosyVoice2 Korean SFT

한국어 억양과 감정·발화 스타일을 강화하기 위해 CosyVoice2-0.5B의 LLM 부분을 supervised fine-tuning한 community checkpoint입니다. Claude와 Codex에 연결하는 로컬 MCP는 cosyvoice2-ko-sft-mcp 저장소에서 제공합니다.

Highlights

• Local privacy: 문장과 reference audio를 외부 TTS API로 전송하지 않고 사용자 PC에서 합성
• Korean prosody: 한국어 감정·발화 스타일 데이터 기반 SFT
• Accessible GPU target: 0.5B급 LLM 코어, 12GB VRAM급 RTX 3060에서 사용 가능, RTX 5060 Ti 16GB에서는 더 여유 있는 구동
• Unlimited local retries: 다운로드 후 호출당 과금·API rate limit 없이 반복 생성
• Agent integration: Claude Code, Claude Desktop, Codex에서 공통 MCP tool로 호출
• Offline inference: 최초 다운로드가 끝나면 Hugging Face offline mode 지원

0.5B는 언어모델 코어의 규모를 뜻합니다. 전체 TTS 실행에는 flow, vocoder, tokenizer, speaker encoder 및 ONNX 구성요소가 추가로 필요합니다.

Verified Hardware

GPU	Runtime	Result
RTX 5060 Ti 16GB	PyTorch 2.8.0, CUDA 12.8, FP16	7.04s WAV, RTF 1.31, peak allocated 3.38GiB, peak reserved 4.06GiB
RTX 3090 24GB	PyTorch 2.3.1, CUDA 12.1	7.76s WAV, RTF 1.07
RTX 3060 12GB	12GB-class target	Supported with memory headroom based on the measured 4.06GiB peak reserve

Exact throughput depends on driver, PyTorch build, prompt length, and whether the model cache is warm.

Model Details

Item	Value
Base model	FunAudioLLM/CosyVoice2-0.5B
Fine-tuned component	CosyVoice2 LLM
Selected checkpoint	SFT v6 Epoch 3
Training precision	BF16 autocast
Saved tensor dtype	FP32
Gradient accumulation	32
Learning rate	1e-5
Checkpoint metadata	epoch 3, step 344
State-dict tensor entries	295
State-dict parameter entries	641,938,468
Sample rate	24 kHz

Epoch 3 was selected after per-epoch validation and listening comparison. The recorded validation snapshot reports loss 3.0528906387 and accuracy 0.2699536899. These training metrics are not a standalone measure of perceived TTS quality; broader CER, speaker-similarity, RTF, and loudness benchmarks will be added in a later model-card revision.

Required Base Files

This repository intentionally contains only the fine-tuned LLM artifact. Download the remaining components from the pinned FunAudioLLM/CosyVoice2-0.5B base model:

• cosyvoice2.yaml
• CosyVoice-BlankEN/
• flow.pt
• hift.pt
• campplus.onnx
• speech_tokenizer_v2.onnx

The companion MCP installer assembles the base snapshot and overlays llm.safetensors automatically.

Local MCP Usage

git clone --recursive https://github.com/feedingstick321-maker/cosyvoice2-ko-sft-mcp.git
cd cosyvoice2-ko-sft-mcp
powershell -ExecutionPolicy Bypass -File .\scripts\install.ps1
.\.venv\Scripts\cosyvoice-ko-prepare.exe

Then register the executable with an MCP client:

codex mcp add cosyvoice-ko -- "$PWD\.venv\Scripts\cosyvoice-ko-mcp.exe"
claude mcp add --transport stdio --scope user cosyvoice-ko -- "$PWD\.venv\Scripts\cosyvoice-ko-mcp.exe"

Training Data

The model was fine-tuned using AI Hub's 감성 및 발화 스타일별 음성합성 데이터. The source audio, labels, parquet files, speech tokens, embeddings, and production audio are not included in this repository.

본 모델은 과학기술정보통신부의 재원으로 한국지능정보사회진흥원의 지원을 받아 구축된 AI Hub의 '감성 및 발화 스타일별 음성합성 데이터'를 활용하여 학습되었습니다.

Dataset page: https://www.aihub.or.kr/aihubdata/data/view.do?dataSetSn=466

Intended Use

• Korean narration, podcast drafts, local agent workflows, accessibility prototypes
• Private/local synthesis where text and reference audio should remain on-device
• Research and evaluation of Korean prosody and voice-conditioned TTS

Limitations

• Output quality depends strongly on reference-audio quality and matching prompt text.
• Long text should be segmented into sentence or utterance units.
• This checkpoint does not include a distributable fixed-speaker spk2info.pt; use a reference voice you have permission to use.
• It may produce pronunciation, prosody, loudness, or stability errors and requires listening review for production use.
• RTX 3060/5060 Ti support refers to local inference. Throughput depends on driver, PyTorch build, text length, and selected precision.

Responsible Use

Do not impersonate people, evade consent, or distribute generated audio deceptively. Users are responsible for the rights to every registered reference voice and for clearly disclosing synthetic audio where required.

License and Attribution

Code and base model lineage are Apache-2.0. See LICENSE and NOTICE. This is a community model and is not an official FunAudioLLM, Alibaba, NIA, Anthropic, or OpenAI release.