Gus Net Gpt2

GUS-Net (GPT-2) Token-level built on (causal decoder). Given a sentence, it tags each token with one of four bias categories following a 7-label BIO scheme, highlighting carry bias. Part of the project (a master's thesis on interpretable bias detection through transformer attention). It provides a causal-model counterpart to the BERT detectors, for studying how bias signals differ between bidirectional and autoregressive architectures. - - multi-label token classification (per-token sigmoid, thresholded) - standard mask (kept at inference — the model was fine-tuned causally) - English - , , ## Label scheme | Index | Label | Category | |-------|-----------|-----------------------------------| | 0 | O | none | | 1 | B-STEREO | Stereotype (span start) | | 2 | I-STEREO | Stereotype (span inside) | | 3 | B-GEN | Generalisation (span start) | | 4 | I-GEN | Generalisation (span inside) | | 5 | B-UNFAIR | Unfair language (span start) | | 6 | I-UNFAIR | Unfair language (span inside) | - — a blanket generalisation about a group. - — unfair / disparaging language toward a group. - — a stereotype attributed to a group. ## Important: multi-label + per-label thresholds Outputs are (multi-label), not a softmax. F1-optimised thresholds (order ): A flat 0.5 threshold will mis-detect bias — use the values above. ## Usage GPT-2 has no /; the tokenizer needs and a pad token. The first token is an attention-sink position — be cautious reading its scores. ## Training data Fine-tuned on the — a token-level social-bias corpus annotated for eneralisations, nfairness and tereotypes (). punctuation tokens that were annotated with bias labels in the original data were (relabelled to the non-bias class), since punctuation is not a social-bias carrier. Bias spans predicted by these models therefore exclude leading/trailing punctuation. ## Evaluation ### StereoSet (intersentence split, 2123 examples) | Metric | Score | |--------|-------| | LMS (language-modeling score, higher is better) | | | SS (stereotype score, ) | | | ICAT (bias-adjusted quality) | | Per-category SS: gender 57.85 · race 48.26 · religion 55.13 · profession 52.24. ### Token classification (GUS-Net held-out test set) Held-out 20% split (seed 42, 748 sentences) of , scored with the per-label thresholds above; each category aggregates its B-/I- labels. | Category | Precision | Recall | F1 | |----------|-----------|--------|----| | O (non-bias) | 0.864 | 0.948 | 0.904 | | GEN | 0.721 | 0.613 | 0.663 | | UNFAIR | 0.436 | 0.608 | 0.507 | | STEREO | 0.754 | 0.764 | 0.759 | | | 0.687 | 0.691 | 0.689 | ## Limitations & intended use - , not a content-moderation oracle. Predictions reflect a specific operationalisation of bias; subtle or context-dependent bias may be missed. - Causal masking means each token only sees left context, so span boundaries can differ from the BERT models. - English only. - Do use for automated decisions about individuals. ## Citation If you use these models, please cite the GUS-Net dataset and benchmark: ## License Weights released under (matching the base model). The Attention Atlas code is MIT-licensed.