42ndAlignment Gpt Oss 20b Pref Sft LoRA v0.2

42ndAlignment gpt-oss-20b Preference-SFT LoRA v0.2

This is a LoRA adapter for unsloth/gpt-oss-20b-unsloth-bnb-4bit, trained as part of the 42ndAlignment project.

42ndAlignment tests whether the Epistemic Octahedron framework can be operationalized into training data for epistemic maturity: reality contact, self-correction, contradiction handling, non-strawmanning, calibrated certainty, integrated dimension consideration, and agency-preserving usefulness.

Model Details

• Base model: unsloth/gpt-oss-20b-unsloth-bnb-4bit
• Starting point: v0.1 SFT adapter
• Adapter type: LoRA
• Training method: preference-derived SFT
• Training examples: 300 chosen responses from a DPO-style preference dataset
• Project repo: https://github.com/raulmiguelparasiii/42ndAlignment
• License: Apache-2.0

Method Note

This adapter is not true DPO.

True DPO was attempted using the free Colab T4 gpt-oss + Unsloth + TRL stack, but training failed with a dtype mismatch during the forward pass:

RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::Half

As a practical fallback, v0.2 was trained on the chosen side of the DPO-style preference dataset using supervised fine-tuning. This preserves the preference-selected target responses, but it does not directly optimize a DPO loss.

Do not describe this adapter as a DPO model. Describe it as preference-derived SFT.

Intended Use

This adapter is intended for research and evaluation of epistemic-maturity-oriented fine-tuning.

The goal is to test whether an open-weight model can be steered toward more stable judgment behavior: directness, calibrated certainty, resistance to weak framing, contradiction handling, non-strawmanning, and reduced sycophancy.

This is not a production safety model.

Training Data

The adapter was trained using the chosen responses from epistemic_maturity_dpo_pairs_v1_300.jsonl.

The original preference dataset contains:

prompt
chosen
rejected
chosen_score
rejected_score
preference_reason
collapse_flags_rejected

Only the chosen side was used for this v0.2 fallback run.

Evaluation

This v0.2 adapter has not yet received a full public benchmark evaluation.

The project currently includes preliminary v0.1 before/after evaluation artifacts in the GitHub repository. TruthfulQA, IFEval, sycophancy, and a larger Epistemic Maturity Benchmark are planned next.

Limitations

• Not true DPO.
• Small dataset.
• Preliminary evaluation only.
• Not a production safety model.
• May affect general capability.
• Should be tested against matched baselines before strong claims.
• Does not prove broad generalization or mature internal cognition.

Loading

This adapter was trained and tested through Unsloth.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="rmpiii/42ndAlignment-gpt-oss-20b-pref-sft-lora-v0.2",
    max_seq_length=512,
    dtype=None,
    load_in_4bit=True,
    full_finetuning=False,
)

Project

GitHub repository:

https://github.com/raulmiguelparasiii/42ndAlignment

Citation / Attribution

Theory, rubric, datasets, and adapter work by Raul Miguel P. Paras III.