--- language: - en license: apache-2.0 tags: - peft - safetensors - unsloth - reinforcement-learning - grpo - security - defense - multi-agent - arxiv:1910.09700 --- # coliseum034/coliseum-defender-grpo-live This model has been fine-tuned using Generative Reward Policy Optimization (GRPO) to align its decision-making capabilities. It was trained utilizing [Unsloth](https://github.com/unslothai/unsloth) for accelerated training and memory optimization. This model acts as a highly optimized "defender" node designed to reliably evaluate, filter, and secure interactions within multi-agent vulnerability scanners and complex security architectures. ## ⚙️ Model Details * **License:** Apache 2.0 * **Architecture:** ~1.5B Parameters (Trainable parameters: 18,464,768 / 1.18% trained via PEFT) * **Language:** English * **Training Type:** Generative Reward Policy Optimization (GRPO) ## 📊 Evaluation Metrics: Pre vs. Post GRPO The model was evaluated before and after the GRPO training phase on a 50-sample held-out evaluation set. Because the base supervised model had already reached a performance ceiling on classification accuracy for this dataset, the GRPO phase focused on aligning and optimizing the generation reward. | Metric | Pre-GRPO | Post-GRPO | Delta | | :--- | :---: | :---: | :---: | | **Accuracy** | 1.0000 | 1.0000 | + 0.0000 | | **Precision** | 1.0000 | 1.0000 | + 0.0000 | | **Recall** | 1.0000 | 1.0000 | + 0.0000 | | **F1 Score** | 1.0000 | 1.0000 | + 0.0000 | | **Avg. Reward** | 0.9367 | 0.9380 | + 0.0013 | *Note: The model achieved a perfect F1 score (1.0000) prior to RL fine-tuning. The GRPO phase successfully increased the average reward generation behavior without degrading classification accuracy.* ## 📈 Training Procedure & Hyperparameters The model was trained for 1 epoch over 200 steps using custom reward functions. Gradients were smartly offloaded to optimize VRAM. * **Training Examples:** 2,482 * **Total Steps:** 200 * **Batch Size per Device:** 2 * **Gradient Accumulation Steps:** 4 * **Generations per Prompt:** 2 * **Total Batch Size:** 8 * **Total Training Runtime:** ~30.8 minutes (30:52) * **Final Training Loss:** `-0.0000` ### Generation Configuration The `generation_config` was specifically modified to support the GRPO rollout phase: * **Max Length:** 32,768 * **Top K:** 20 * **Top P:** 0.8 * **Repetition Penalty:** 1.1 * **BOS Token ID:** 151643 * **EOS Token IDs:** [151645, 151643] ### Training Progression (Sampled Steps) | Step | Training Loss | Reward | Reward Std Dev | Completion Length | | :---: | :---: | :---: | :---: | :---: | | **5** | -0.0000 | 0.1341 | 0.0177 | 34.85 | | **50** | -0.0000 | 0.1675 | 0.0117 | 34.70 | | **100** | 0.0000 | -0.0587 | 0.0176 | 32.75 | | **150** | -0.0000 | -0.1545 | 0.0355 | 35.65 | | **185** | -0.0000 | 0.4397 *(Peak)* | 0.0057 | 32.25 | | **200** | 0.0000 | 0.0461 | 0.0127 | 30.85 | ## 💻 Framework Versions * PEFT 0.14.0 * Transformers * Unsloth * TRL * Safetensors * PyTorch ## 🚀 Usage This model uses the standard `transformers` library pipeline. Ensure that your inference scripts respect the custom generation configuration parameters used during training for optimal results. ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "coliseum034/coliseum-defender-grpo-live" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id) prompt = "Evaluate the safety and structural integrity of the following system request:" inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate( **inputs, max_length=32768, top_k=20, top_p=0.8, repetition_penalty=1.1 ) print(tokenizer.decode(outputs[0], skip_special_tokens=True))
