--- license: cc0-1.0 language: - bo base_model: jhu-clsp/mmBERT-base pipeline_tag: token-classification tags: - tibetan - boundary-detection - modernbert - token-classification --- # Bo-Boundary-mmBert A **ModernBERT** model fine-tuned on Tibetan text for **text-boundary detection** (token classification). Each token is classified as either a **boundary token (B)** or a **non-boundary token (O)**. ## Model Details | Property | Value | |---|---| | Base model | [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) | | Architecture | `ModernBertForTokenClassification` | | Hidden size | 768 | | Layers | 22 | | Attention heads | 12 | | Max sequence length | 8,192 | | Vocab size | 256,000 | | Labels | `O` (non-boundary), `B` (boundary) | ## Training ### Data | Split | Windows | |---|---| | Train (raw) | 62,416 | | Train (after negative sampling @ 0.1) | 38,610 | | Validation | 5,874 | The boundary label is extremely rare — **B : O ratio is 1 : 638** — so negative window sampling (ratio 0.1) and focal loss were used to handle the class imbalance. ### Hyperparameters | Parameter | Value | |---|---| | Epochs | 5 | | Batch size (per GPU) | 1 | | Gradient accumulation steps | 16 | | Effective batch size | 16 | | Learning rate | 2 × 10 -5 | | Warmup steps | 1,206 (10 %) | | Total optimisation steps | 12,065 | | Loss | Focal Loss (gamma = 1.5, alpha = [O: 0.1, B: 0.9]) | | Early stopping patience | 15 evals | | Mixed precision | bfloat16 | | torch.compile | enabled | ### Infrastructure - **GPU:** 1 × NVIDIA GeForce RTX 4090 (23.5 GB) - **Training time:** ~24 h 52 min ### Training Curve The model was evaluated every 200 optimisation steps. Key milestones: | Step | Epoch | Train Loss | Val F1 | Val F2 | Val Precision | Val Recall | |---|---|---|---|---|---|---| | 200 | 1 | 0.0278 | 0.069 | 0.129 | 0.039 | 0.308 | | 1,200 | 1 | 0.0051 | 0.370 | 0.521 | 0.249 | 0.717 | | 2,600 | 2 | 0.0002 | 0.391 | 0.574 | 0.255 | 0.834 | | 3,600 | 2 | 0.0002 | 0.396 | 0.581 | 0.258 | 0.846 | | 5,400 | 3 | 0.0002 | 0.428 | 0.610 | 0.286 | 0.850 | | 7,200 | 3 | 0.0002 | 0.445 | 0.627 | 0.300 | 0.862 | | 9,200 | 4 | 0.0001 | 0.495 | 0.648 | 0.355 | 0.817 | | **9,800** | **5** | **0.0001** | **0.496** | **0.651** | **0.355** | **0.823** | | 12,065 | 5 | 0.0001 | 0.479 | 0.637 | 0.340 | 0.815 | **Best checkpoint** selected by F2 score: **0.6510** at step 9,800 (epoch 5). ## Evaluation on Benchmark The best checkpoint was evaluated on 30 held-out benchmark documents with tolerance = 25 and threshold = 0.75. ### Aggregate Metrics | Metric | Value | |---|---| | Micro Precision | 0.644 | | Micro Recall | 0.854 | | Micro F1 | **0.734** | | Macro F1 | 0.711 | | True Positives | 543 | | False Positives | 300 | | False Negatives | 93 | | Total Predicted | 843 | | Total True | 636 | ### Per-Document Highlights | Document | Precision | Recall | F1 | |---|---|---|---| | W8LS76156 (google_books) | 0.931 | 1.000 | **0.964** | | W1KG16597 (google_books) | 0.893 | 1.000 | 0.943 | | W1KG22443 (ocrv1) | 0.886 | 0.987 | 0.934 | | W8LS31006 (ocrv1) | 1.000 | 0.875 | 0.933 | | W3CN3089 (ocrv1) | 0.864 | 1.000 | 0.927 | | W3KG466 (ocrv1) | 0.882 | 0.938 | 0.909 | | W3KG439 (ocrv1) | 0.767 | 0.959 | 0.853 | | IE3CN3396 (tei) | 0.778 | 0.875 | 0.824 | The model performs best on clean sources (google_books, ocrv1) and can struggle on noisier OCR or pages with no true boundaries (false-positive predictions on zero-boundary pages). ## Limitations - The model was trained on a specific corpus of Tibetan texts; generalisation to unseen document styles or OCR engines may vary. - High recall (0.854) but moderate precision (0.644) means the model tends to over-predict boundaries — downstream consumers should apply confidence thresholding. - Pages with zero true boundaries can produce false-positive boundary predictions. ## Credits This model was trained by [Dharmaduta](https://dharmaduta.org) from specifications provided by the [Buddhist Digital Resource Center](https://www.bdrc.io) (BDRC) for the BDRC Etext Corpus, with funding from the [Khyentse Foundation](https://khyentsefoundation.org).
