--- language: en license: other library_name: transformers pipeline_tag: text-classification base_model: microsoft/deberta-v3-xsmall base_model_relation: finetune tags: - fiction - nonfiction - binary-classification - text-classification - deberta-v3 - calibrated-threshold --- # V2.19 fiction / nonfiction classifier ## Model summary Approved V2.19 release model for binary fiction/nonfiction classification. - promoted model: DeBERTa-v3-xsmall V2.19 release candidate - task: binary text classification (`fiction` vs `nonfiction`) - threshold mode: `temperature_scaled` - fitted temperature on validation logits: `2.1562` - selected threshold: `0.35` - threshold source: guarded calibrated validation sweep within `0.005` macro-F1 of the best threshold This release also includes a fallback model: - fallback model: TinyBERT 6L V2.19 release candidate - threshold mode: `guarded_raw` - selected threshold: `0.55` ## Intended use Recommended inference granularity is paragraph/chunk/window classification rather than whole-book single-pass inference. Use this model to classify 512-token text windows into `fiction` or `nonfiction`. Operational definition: - fiction = actual story prose only, including excerpts and narrative scenes - nonfiction = metatext, glossary/reference material, legal/copyright pages, author bios, reviews, explanatory text, and other non-story prose - exclude = junk, mixed, or uncertain material used only in review/auditing, not as a runtime label Not intended for: - factuality detection - authorship attribution - literary quality assessment - truthfulness classification ## Anti-leakage protocol The evaluation protocol includes: - canonical-work split isolation - near-duplicate filtering - Jaccard similarity leakage audits - hard negative mining - disagreement-based relabel review ## Training data / evaluation summary Training used the final V2.19 cleaned dataset after: - approved patch manifest application - two finite disagreement-review smoothing rounds Applied dataset changes: - 53 relabels - 1 exclusion Evaluation summary for the promoted model: - test macro-F1: `0.9338` - OOD macro-F1: `0.9327` - fresh stress macro-F1: `0.9718` - validated metatext recall: `1.0000` on the maintained 21-row V2.19 acceptance slice - structural nonfiction recall: `0.9952` - fresh story-prose false-nonfiction rate: `0.0316` - nonfiction precision: `0.9542` The fallback TinyBERT 6L model also passed release gates, with: - test macro-F1: `0.9339` - OOD macro-F1: `0.9332` - fresh stress macro-F1: `0.9533` - validated metatext recall: `0.9000` on the maintained V2.19 acceptance slice - structural nonfiction recall: `0.9857` - fresh story-prose false-nonfiction rate: `0.0484` - nonfiction precision: `0.9604` ## Thresholds and calibration note The promoted model was calibrated with temperature scaling and uses `temperature_scaled` operating mode. Calibration was acceptable but not perfect: - test ECE: `0.0166` - OOD ECE: `0.0196` - fresh-stress ECE: `0.0425` - fresh-stress Brier: `0.0222` ## Recommended Operating Points The model was calibrated using temperature scaling (`temperature = 2.1561811131629827`). Different thresholds trade off: - fiction protection - nonfiction recall - overall macro-F1 | Mode | Threshold | Best For | OOD Macro-F1 | Fresh Stress Macro-F1 | Story-Prose False Nonfiction | | --- | ---: | --- | ---: | ---: | ---: | | Conservative Fiction Protection | `0.50` | Avoiding story false positives | `0.9275` | `0.9739` | `0.0274` | | Balanced Release Default | `0.35` | General deployment | `0.9327` | `0.9718` | `0.0316` | | Aggressive Nonfiction Recall | `0.10` | Maximum nonfiction detection | `0.9438` | `0.9523` | `0.0842` | Recommended default: - use `0.35` after temperature scaling unless your application strongly prioritizes minimizing story-prose false nonfiction errors. Validation best-threshold reference: - best validation macro-F1: `0.9384` at threshold `0.10` - selected validation macro-F1: `0.9335` at threshold `0.35` Do not assume the default `0.5` threshold reproduces the reported operating point. ## Limitations - This is a binary classifier only. - The validated metatext slice is small (`21` rows), so acceptance confidence is narrower than on the main test/OOD splits. - The structural nonfiction acceptance slice is also small (`422` rows). - Nonfiction precision is above threshold but with a limited margin (`0.9542` vs `0.9500`). - Fresh story-prose acceptance still contains some contamination/non-story material. - Performance is reported for the V2.19 release distribution and should not be treated as a guarantee on unrelated domains. ## Fallback path If the promoted model is unavailable, use: - TinyBERT 6L V2.19 release candidate That fallback is the released TinyBERT 6L candidate and remains the closer substitute within this repo. ## Supporting artifacts Supporting evaluation artifacts are included in the release bundle. ## Risk notes - No unresolved medium/high leakage concern was found in the release audit. - Final dataset build reported canonical overlaps `0`, rows with Jaccard `>= 0.80` `0`, and max train/eval Jaccard `0.5200`. - The model is intended for release-consistent use on similar text classification inputs, not as a general literary or factuality detector. ## License Released for research/evaluation use only. See `LICENSE`.
