# Naga Languages Cosine Similarity Analysis ## What This Is A computational study of **how similar Naga languages are to each other**, measured using parallel Bible translations as a controlled corpus. 36 Naga and Naga-adjacent languages plus English — compared using character-level patterns to find shared vocabulary, cognates, and structural connections. The goal is not to rank languages or tribes. It is to find the common threads — the shared words, the similar sounds, the patterns that connect communities across mountains and state borders. ## Key Findings ### 1. Six similarity methods evaluated; three are primary | Method | Cohen's d (Full) | ARI (Full) | Cohen's d (NT) | ARI (NT) | Status | |--------|-----------------|-----------|----------------|---------|--------| | **Jaccard (top-1000 trigrams)** | **1.64** | **0.65** | **1.57** | 0.55 | ✅ **Best clustering; most robust across coverage** | | **TF-IDF (2-4 char n-grams)** | **1.74** | 0.47 | **1.94** | **0.66** | ✅ **Best pair ranking; improves on NT-only** | | **Subword Vocabulary JSD** | 1.27 | 0.44 | 1.37 | 0.32 | ✅ Respectable complement | | Character Trigram Frequency | 1.11 | 0.36 | 1.12 | 0.35 | ⚠️ Marginal (most stable across coverage) | | Character Frequency (unigrams) | 0.66 | 0.30 | 0.63 | 0.29 | ❌ Fails for same-script languages | | Glot500-m Embeddings | 0.27 | 0.15 | 0.27 | 0.15 | ❌ Fails for parallel texts | **Jaccard** uses binary presence/absence of the top-1000 most frequent trigrams, computing overlap proportion (`|intersection| / |union|`). It outperforms TF-IDF for clustering on full Bibles (ARI 0.65 vs. 0.47) because TF-IDF's IDF weighting can accidentally suppress genuine branch-internal vocabulary if those n-grams also appear in a few other languages. Jaccard also eliminates corpus-size bias between full-Bible and NT-only translations via top-K truncation. **TF-IDF** achieves the highest Cohen's d on both corpora and *improves* on NT-only (1.74 → 1.94) because uniform corpus sizes let IDF weighting work correctly (no longer distorted by languages with 5× more text). **Subword JSD** provides a complementary, fully automated signal. The three primary methods (Jaccard, TF-IDF, Subword JSD) produce consistent branch-level clusters; the other three serve as negative controls confirming what does NOT work. ### 2. Known language families cluster correctly The TF-IDF, Jaccard, and Subword JSD dendrograms, t-SNE, and PCA all show: - **Ao (Central Naga)** and **South Patkaian** languages cluster together — Ao, Sangtam, Yimchunger near Chang, Phom, Konyak, Khiamniungan, Wancho - **Angami-Pochuri** forms a distinct cluster — Angami, Chokri, Pochuri, Poumei, Rengma N, Rengma S, Sumi - **Monsang-Moyon** pair is the most similar in the dataset — confirmed close relatives - **English** is correctly the most distant language from all Naga languages ### 3. Data coverage: 24 full Bibles, 10 NT-only, 2 partial, all 36 included | Coverage | Count | Languages | |----------|-------|-----------| | Full Bible (66 books, 1189 chapters) | 24 | ASII Lakdap (Tangsa), Anal, Angami, Ao, Chang, English, Hawa, Khiamniungan, Kom, Konyak, Liangmai, Maring, Phom, Pochuri, Poumei, Rengma N, Rengma S, Rongmei, Sangtam, Sumi, Tangkhul, Wancho, Yimchunger, Zeme | | NT only (27 books, 260 chapters) | 10 | Chokri, Chothe, Kharam, Maram, Monsang, Moyon, Muklom, Nagamese, Tarao, Tutsa | | Partial | 2 | Lamkang (29 books, 441 chapters), Nocte (46 books, 1,018 chapters) | | Data disparity (included) | 1 | **Chuyo (Tangsa)** — 1 book, 16 chapters; included in all analyses with explicit low-volume flag | **Data disparity note:** Chuyo (Tangsa) and Nocte are no longer used as coverage thresholds. The fixed **27 New Testament books** (260 chapters) serve as the uniform corpus threshold for all analyses. Chuyo contributes only Mark (16 chapters) — its low volume is explicitly flagged in every downstream analysis and report, but the language is retained in the dataset rather than excluded. Nocte, despite being partial (46 books), contributes substantial NT content and is fully included. See `documentation/phase_06_word_extraction_methodology.md` §10.4 and `scripts/02g_extract_common_books.py` for implementation. ### 4. NT-only experiment: controlled corpus-size comparison (modular pipeline) A dedicated **NT-only analysis pipeline** controls for the corpus-size bias between full-Bible (24 languages, ~4–6M chars) and NT-only (10 languages, ~1–2M chars) translations. The pipeline uses three modular scripts mirroring the full-Bible workflow: ```bash # 1. Extract NT + clean (reads data/bibles/, writes data/bibles_cleaned_nt/) python scripts/02e_clean_bible_nt.py # 2. Compute all 6 similarity methods (reads data/bibles_cleaned_nt/, writes results/nt_only/*.tsv) python scripts/03_similarity_methods_nt.py # 3. Generate visualizations (reads results/nt_only/*.tsv, writes results/nt_only/visualizations/*.png) python scripts/04_visualization_nt.py ``` **Key design:** - **02e_clean_bible_nt.py** — Extracts only NT content (MAT–REV, 260 chapters) from raw Bible files via chapter-marker detection (`### MAT 1` ... `### REV 22`), then applies the **identical** cleaning pipeline from 02d_clean_bible_texts.py. Outputs to `data/bibles_cleaned_nt/` (local only, `.gitignore`'d). - **03_similarity_methods_nt.py** — Loads cleaned NT texts, computes all 6 methods (TF-IDF, CharFreq, Embeddings optional, Trigram, Jaccard, Subword JSD optional), evaluates each with Cohen's d + ARI against Glottolog classifications, and saves similarity matrices to `results/nt_only/`. - **04_visualization_nt.py** — Reads NT-only similarity matrices and generates 4 chart types per method (dendrogram, heatmap, t-SNE, PCA) to `results/nt_only/visualizations/`, color-coded by linguistic branch. **Key NT-only findings:** - **TF-IDF improves most** (Cohen's d 1.74 → 1.94, ARI 0.47 → 0.66) — uniform corpus sizes eliminate IDF distortion - **Jaccard degrades slightly** but remains robust (Cohen's d 1.64 → 1.57, ARI 0.65 → 0.55) - **Trigram is most stable** (barely changes) — raw distributional counts are less affected by corpus size - **Methods D/E/F (Trigram, Jaccard, Subword JSD)** are the most robust across coverage levels, as designed ### 5. Character frequency fails for same-script languages Character frequency (single letter counts) cannot distinguish Naga languages from each other or from English — all Latin-script languages look nearly identical at the individual character level. Cohen's d = 0.66 (small), ARI = 0.30 (poor). Published literature confirms: unigrams fail for same-script languages; bigrams and trigrams are needed (Singh 2006, Cavnar & Trenkle 1994). ### 6. Sentence embeddings fail for parallel texts Glot500-m embeddings (`cis-lmu/glot500-base`, layer 8) produce similarity > 0.92 for every pair — including English vs. Naga languages. Three factors combine: (a) multilingual models are trained to align parallel content, so same Bible verses in different languages map to the same point; (b) most Naga languages are not in Glot500's 511-language training set, causing the model to fall back to character-level tokenization that treats all Latin-script text similarly; (c) the inter-language variation among 34 closely related Tibeto-Burman languages is too small for the model's embedding space to resolve. Cohen's d = 0.27 (small), ARI = 0.15 (poor). ### 7. Five classification errors were corrected Initial branch assignments were verified against Glottolog 5.x, correcting five errors: - **Chang, Phom**: moved from Ao to South Patkaian (Glottolog: Chang-Phom-Konyak clade) - **Maram**: moved from Angami-Pochuri to Zemeic (Glottolog: mara1379) - **Nocte, Tutsa**: moved from Konyak to North Patkaian (Morey 2019: Tangsa-Nocte) - **Kom**: moved from standalone to Pakan group (Glottolog: NW Kuki-Chin, Kolhrengic) See [documentation/classification_verification.md](documentation/classification_verification.md) for full details with data validation. ### 8. Text cleaning improved results significantly (cleaned vs. uncleaned) A systematic cleaning pass removed chapter markers, verse numbers, cross-references, inline footnotes, and English book titles from all raw Bible texts. The cleaning **improved classification quality measurably**: | Metric | First Run (Uncleaned, Wrong Labels) | Cleaned + Corrected Labels | Interpretation | |--------|-------------------------------------|---------------------------|----------------| | Cohen's d | 1.03 | **1.74** | Better same-branch vs. different-branch separation | | Adjusted Rand Index | 0.53 | **0.47** | Clustering matches known linguistics | | Within-group avg similarity | 0.3290 | **0.3445** | Same-branch pairs are genuinely more similar | Cleaning removed a "background similarity" of ~0.05–0.10 that was uniform across all pairs (shared cross-references, verse numbers, book abbreviations). This increased the effective dynamic range and sharpened the true linguistic signal. ### 9. Annotation density varies dramatically — Wancho is an extreme outlier (30.6%) The cleaning script now breaks removed content into six categories: chapter markers, hash footnotes, parenthetical cross-references, English titles, digits, and `annot_other` (non-standard inline annotations not matching typical patterns). Key findings: | Language | Total Removed | Density | annot_other | % other | Source | |----------|-------------|---------|-------------|---------|--------| | **Wancho** | **1,475,983** | **30.6%** | **1,344,834** | **91%** | **Inline translator notes, Hebrew glosses, alternative readings** | | Maring | 1,082,002 | 18.2% | 948,767 | 88% | Extensive non-standard markup | | English (NRSVUE) | 931,472 | 19.6% | 609,947 | 65% | Section headings, footnote text | | Rengma (N) | 498,424 | 11.9% | 377,783 | 76% | Non-standard markup | | Liangmai | 495,347 | 11.0% | 373,302 | 75% | Non-standard markup | | **Median (all Naga)** | — | **~3.1%** | — | ~40% | Typical range 2–4% | Wancho at 30.6% is an extreme outlier — 10× the median. Crucially, **91% of Wancho's removed content is `annot_other`** — inline translator notes explaining Hebrew words, alternative renderings, geographic commentary, and other metalinguistic content. The cleaning script catches these via whitespace collapse and line stripping. Wancho's cleaned corpus (530,803 words, 3.3 MB) remains substantial and fully usable. ### 10. Anal–Kom investigation: ~0.10 similarity was artifactual (shared digitization pipeline) **Anal–Kom (both Pakan)** dropped from 0.454 to 0.353 after cleaning — a 0.101 decrease, the largest within-branch drop. An annotation pattern investigation (`scripts/investigate_anal_kom.py`) found: | Finding | Value | Meaning | |---------|-------|---------| | **Footnote signature Jaccard** | **1.000** | Identical footnote/cross-reference patterns across all 1,189 chapters | | Annotation word overlap | 0.481 | 13 shared English book abbreviations | | Hash line density | 0.891 | Near-identical footnote density | **Verdict: ~0.10 of the uncleaned Anal–Kom similarity was artifactual**, caused by shared annotation formatting from the same bible.com digitization pipeline. The cleaned score of **0.353** is the accurate linguistic similarity. ### 11. Word extraction with PMI and cognate detection validates core vocabulary The `scripts/05_extract_similar_words.py` script produces two complementary outputs: 1. **PMI translation table** (~1200 English concepts × 36 Naga languages, v5.0): Maps English words to their best Naga equivalents using Pointwise Mutual Information on **real chapter alignment** (`MAT_1` ↔ `MAT_1` across all languages), not coarse pseudo-chapters. **v5.0 improvements:** (a) **Top-3 PMI candidates** per English word with `select_best_candidate()` heuristic that prefers **shorter base forms** over grammatical clitics — eliminating false positives like "johnwui" (possessive "John's") for "dismissed" and "khorumshimli" (locative "at the temple") for...
