Korean Sign Word Classifier Mediapipe

Korean Sign Word Classifier with MediaPipe Holistic

This repository contains an end-to-end inference wrapper for isolated Korean sign-language word clips:

mp4 video
-> MediaPipe Holistic keypoint extraction
-> [T, 115, 4] keypoints
-> WordKeypointClassifier
-> Korean word prediction

The neural classifier itself is trained on keypoints, not raw RGB frames. The included wrapper connects the raw mp4 input to the keypoint extractor and then to the trained PyTorch checkpoint.

Latest Update

• best.pt was updated on 2026-06-06 with a continued-training checkpoint.
• Latest checkpoint commit: f0b27cffccae4b3676db65d32ff84fab0c686c1a
• Checkpoint epoch: 41
• Validation accuracy: 89.62%
• best.pt SHA-256: c1ee1138ac2f2cde81b9f1ce258f6f13d041f0c01050181c7a76e855f6ba9eb6
• The continued-training and Test-100 evaluation workflow was prepared through the TechieMoon/realtime-ksl-captioning project environment.

Additional local Test-100 evaluation on 100 isolated-word videos:

• Top-1 correct: 60 / 100
• Top-5 contains expected: 71 / 100
• Inference errors: 0
• Label normalization used for the filename typo 꺠끗하다 -> 깨끗하다.

Files

• best.pt: best validation checkpoint, epoch 41.
• label_to_id.json: word label vocabulary.
• training_config.json: base training/inference configuration reference. The current model config is also stored in best.pt.
• model/predict_word_classifier.py: mp4/keypoint inference CLI.
• model/assets/holistic_landmarker.task: MediaPipe HolisticLandmarker task asset used by the wrapper.
• model/*.py: model architecture, preprocessing, and keypoint extraction code needed for inference.

Current Checkpoint

• Epoch: 41
• Validation accuracy: 89.62%
• Number of word classes: 2,946
• Max frames: 64
• Keypoint layout: 115 points
• Raw keypoint values per point: x, y, z, confidence
• Classifier input features per keypoint after preprocessing: 16

Continued-training setup for the current checkpoint:

• Batch size: 32
• Learning rate: 0.0003
• Optimizer: AdamW
• d_model: 256
• Spatial Transformer layers: 2
• Temporal Transformer layers: 4
• Attention heads: 4
• Dropout: 0.1

Installation

Create a Python environment and install dependencies:

pip install -r requirements.txt

If this repository is private, authenticate with a Hugging Face access token before downloading:

hf auth login
# or set HF_TOKEN in your environment

Create a token at https://huggingface.co/settings/tokens. A read token is enough to download and run this private model repository.

Inference from MP4

Run:

python -m model.predict_word_classifier path/to/sample.mp4 \
  --checkpoint best.pt \
  --device cpu \
  --top-k 5

For JSON output:

python -m model.predict_word_classifier path/to/sample.mp4 \
  --checkpoint best.pt \
  --device cpu \
  --top-k 5 \
  --json

CUDA can be used for the classifier if available:

python -m model.predict_word_classifier path/to/sample.mp4 \
  --checkpoint best.pt \
  --device cuda

MediaPipe extraction itself may still use CPU depending on the local MediaPipe installation.

Inference from Pre-Extracted Keypoints

If you already have a .npy keypoint cache:

python -m model.predict_word_classifier \
  --keypoints path/to/keypoints.npy \
  --checkpoint best.pt \
  --device cpu \
  --top-k 5

The expected keypoint shape is:

[frames, 115, 4]

where the last dimension is:

[x, y, z, confidence]

Data Source

This model was trained using Korean sign-language video data from AIHub.

The original AIHub video dataset is not included in this repository. To reproduce training or access the raw source videos, obtain the dataset separately from AIHub according to AIHub's access process, license, and terms of use. Depending on the AIHub download method, a separate AIHub account/API key/token may be required.

MediaPipe Holistic Notes

The inference wrapper uses MediaPipe HolisticLandmarker to convert mp4 video into model-ready keypoints.

The included task asset path is:

model/assets/holistic_landmarker.task

MediaPipe itself does not require an AIHub token. If this Hugging Face repository is private, however, a Hugging Face token is required to download the repository files, including the MediaPipe task asset and checkpoint.

If replacing the task asset, keep the extracted keypoint layout compatible with the training pipeline:

0..12: upper-body pose
13..33: left hand
34..54: right hand
55..114: selected face points

Limitations

• This is an isolated-word classifier, not a continuous sentence-level sign-language recognizer.
• Accuracy is reported on the current validation manifest and may differ on new domains, camera angles, signers, lighting, or recording setups.
• The classifier relies on MediaPipe keypoint quality. Failed or noisy keypoint extraction can reduce prediction quality.
• The model card does not redistribute AIHub source data.

Example Output

prediction=지도 label_id=2422 probability=0.1032
checkpoint_epoch=41 checkpoint_val_acc=0.8962213765938981
frames=64 effective_frames=64 device=cpu