Loading
Have a ComfyUI workflow? Get a readable Workflow Doc in your browser.Make a workflow readable.Try it now ↗
The RL warm-start checkpoint for multi-turn tool-agent RL on τ2-bench's airline domain.
Model source
Source description
The RL warm-start checkpoint for multi-turn tool-agent RL on
τ2-bench's airline domain.
Behavior-cloned from teacher trajectories, collected by running the teacher as the through the exact same τ2-bench harness used for training and evaluation.
Sources
1 sourceVerified Sep 22
Model artifacts
1 artifactSource excerpts
2 excerptsThe collection script requests
deepseek-chat, which is DeepSeek's alias for their current chat model. Every raw API response in the training data reportsmodel: deepseek-v4-flash(1156/1156 assistant turns, 671/671 user-simulator turns), so the teacher is V4 Flash, not V3. Stated here from the data rather than from the script.
GRPO on τ2-bench-airline from raw Qwen2.5-7B-Instruct is null — and not because of a bug.
Base success is ~20%, so most rollout groups come back all-fail → zero intra-group variance → the
group-relative advantage is ~0 → no gradient. RL sharpens what a policy already does sometimes;
it cannot invent a skill. Worse, the base policy's own successful rollouts never invoke the
write-tools (update_reservation_*) that held-out tasks require, so self-sampling can't bootstrap
them either.
This adapter is the floor that makes the RL step work.
This adapter, merged into the base, is the step 0 policy of the
tau2_airline verl recipe. Measured under
that recipe's protocol (held-out BINARY mean@4, 20 held-out airline tasks, VAL_TEMP=0.5,
llama-3.3-70b user simulator):
| val@0 (this adapter) | val@20 (after GRPO) | |
|---|---|---|
| seed 42 | 0.275 | 0.5625 |
| seed 123 | 0.375 | 0.55 |
GRPO endpoint across 2 seeds: 0.556 ± 0.01. Measured evaluation noise on this same checkpoint (6 repeats of val@0) is 0.30 ± 0.05, so the two val@0 values above are a single checkpoint sampled twice from that band, not a difference.
Scope note. A separate hand-written pipeline reported
base 0.20 → distilled SFT 0.405 → GRPO 0.545for this approach, but under a different eval protocol (pass^1 over 10 trials with a self-hosted 7B user simulator, n=200). Those numbers are not directly comparable to the table above — different user simulator, different metric — and I have not re-verified that this exact file is the checkpoint they were measured on. The table above is what this file scores.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "yuyu0529nya/qwen2.5-7b-tau2-airline-sft-lora")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
To use it as an RL starting point, merge first (model.merge_and_unload()) and point the trainer at
the merged weights — that is exactly how the recipe above consumes it.
This adapter only makes sense inside the τ2-bench airline harness — same system prompt, same 14 tool schemas, same dialogue loop. It is a policy for that environment, not a general chat model.
| base | Qwen/Qwen2.5-7B-Instruct |
| method | behavior-cloning SFT (QLoRA 4-bit) on teacher trajectories |
| teacher | DeepSeek V4 Flash (deepseek-v4-flash, requested via the deepseek-chat alias) — agent and user-simulator, temp 0.7 |
| data | yuyu0529nya/tau2-airline-deepseek-distill (ds_114 config) — 114 successful teacher trajectories (reward = 1), deduped by action signature, capped 4/task |
| LoRA | r=16, alpha=32, dropout=0.05, targets = q,k,v,o,gate,up,down_proj |
| loss | assistant-token-only (render-twice-diff mask) |
Teacher success on these tasks was ~76% vs base ~20%, and 51% of teacher trajectories invoke the write-tools the base policy never reaches on its own. That skill gap is why self-RFT from base cannot work here, and why distillation — rather than more RL — was the fix.
Qwen2.5-7B-Instruct, Apache-2.0 → derivatives unrestricted.@example.com, RFC 2606 reserved). No real PII.llama-3.3-70b. Not controlled for.airline domain only. Not evaluated on other τ2-bench domains. Not a general-purpose model.--- license: apache-2.0 base_model: Qwen/Qwen2.5-7B-Instruct library_name: peft pipeline_tag: text-generation datasets: - yuyu0529nya/tau2-airline-deepseek-distill tags: - lora - peft - agent - tool-use - function-calling - reinforcement-learning - grpo - tau2-bench - distillation language: - en --- # Qwen2.5-7B-Instruct · τ²-bench airline · distilled SFT LoRA The **RL warm-start checkpoint** for multi-turn tool-agent RL on [τ²-bench](https://github.com/sierra-research/tau2-bench)'s `airline` domain. Behavior-cloned from **DeepSeek V4 Flash** teacher trajectories, collected by running the teacher as the *agent* through the exact same τ²-bench harness used for training and evaluation. > The collection script requests `deepseek-chat`, which is DeepSeek's *alias for their current chat > model*. Every raw API response in the training data reports `model: deepseek-v4-flash` (1156/1156 > assistant turns, 671/671 user-simulator turns), so the teacher is V4 Flash, not V3. Stated here > from the data rather than from the script. ## Why this exists GRPO on τ²-bench-airline from raw `Qwen2.5-7B-Instruct` is **null** — and not because of a bug. Base success is ~20%, so most rollout groups come back all-fail → zero intra-group variance → the group-relative advantage is ~0 → **no gradient**. RL sharpens what a policy already does sometimes; it cannot invent a skill. Worse, the base policy's *own* successful rollouts never invoke the write-tools (`update_reservation_*`) that held-out tasks require, so self-sampling can't bootstrap them either. This adapter is the floor that makes the RL step work. ## Results This adapter, merged into the base, is the `step 0` policy of the [`tau2_airline` verl recipe](https://github.com/verl-project/verl-recipe/pull/121). Measured under **that recipe's protocol** (held-out `BINARY mean@4`, 20 held-out airline tasks, `VAL_TEMP=0.5`, `llama-3.3-70b` user simulator): | | val@0 (this adapter) | val@20 (after GRPO) | |---|---|---| | seed 42 | 0.275 | **0.5625** | | seed 123 | 0.375 | **0.55** | GRPO endpoint across 2 seeds: **0.556 ± 0.01**. Measured evaluation noise on this same checkpoint (6 repeats of val@0) is **0.30 ± 0.05**, so the two val@0 values above are a single checkpoint sampled twice from that band, not a difference. > **Scope note.** A separate hand-written pipeline reported `base 0.20 → distilled SFT 0....
Source context: 12 downloads · 0 likes · Pipeline text-generation · Library peft · Repo yuyu0529nya/qwen2.5-7b-tau2-airline-sft-lora