--- tags: - ml-intern --- # Agent Cost Optimizer (ACO) Lightweight cost control layer for AI agent harnesses. Reduces agent run cost while preserving or improving task quality. **Status (verified 2026-06-15):** v1 DistilBERT specialists are the production models. The v2 ModernBERT experiment **did not improve** results and **regressed** the tool-gater (−19.6pp F1) — **use v1**. See [AUDIT.md](AUDIT.md) for the independent verification, and [TRUTH.md](TRUTH.md) for the honest project state. The headline "56% cost reduction" is a **trace-based simulation** result, not a live Docker test-verified result (see caveats below). [](https://huggingface.co/narcolepticchicken/agent-cost-optimizer) ## What ACO Does | Module | Problem | Solution | Impact | |--------|---------|----------|--------| | **Model Cascade Router** | Overusing frontier models | Routes to cheapest capable model | ~56% cost reduction (simulated, iso-quality) | | **Tool-Use Cost Gate** | Unnecessary tool calls | Predicts whether tool call is worth cost | gate F1=0.92 (verified) | | **Verifier Budgeter** | Verifying everything | Selectively gating verification | gate F1=0.65 (verified, weak) | | **Context Compressor** | Growing context windows | ACON-pattern state-preserving compression | 30-54% token reduction (literature) | | **Cache-Aware Layout** | Cache-unfriendly prompts | Static-at-prefix optimization | 50-90% input cost reduction (literature) | | **Intervention Threshold** | Verifier-caused regressions | Only verify when safe (d/(d+r) threshold) | Prevents collapse (literature) | | **Token Budget Estimator** | Verbose outputs | Per-query optimal length prediction | 15-25% output token reduction (literature) | | **Doom Detector** | Continuing doomed runs | Terminate/rescue on failure signals | Prevents wasted cost | | **Meta-Tool Miner** | Repeated workflows | Compress successful traces into macros | LLM calls saved on repeats | > **Evidence levels:** "verified" = independently re-evaluated on held-out data in this repo. > "simulated" = derived from trace simulation (`aco/benchmark.py`, SWE-Router traces), not live > test execution. "literature" = reported by cited papers, not independently measured here. ## Verified vs. Simulated — what is actually proven | Claim | Status | Evidence | |-------|--------|----------| | Tool-gater F1 = 0.92, acc = 0.98 | ✅ **Verified** | Independent re-eval, N=1695 held-out | | Tier-router F1 = 0.67, acc = 0.70 | ✅ **Verified** | Independent re-eval, N=1260 held-out | | Verifier-gater F1 = 0.65, acc = 0.65 | ✅ **Verified** (weak model) | Independent re-eval, N=863 held-out | | v2 ModernBERT beats v1 | ❌ **False** | v2 regressed tool-gater −19.6pp F1; flat elsewhere | | Cascade ≈ 56% cost reduction at iso-quality | ⚠️ **Simulated only** | Trace simulation; frontier-retry actually solves +4 instances | | "4/4 Django instances solved by T1" | ⚠️ **Misleading** | Patches passed `git apply --check` only — **not** test-verified | | Live SWE-bench resolution by cascade agent | ❌ **Not demonstrated** | No Docker/test-verified results file exists in repo | See [AUDIT.md](AUDIT.md) for full detail. ## Architecture ``` Agent Request │ ▼ ┌─────────────────────────────────────┐ │ ACO Control Layer │ │ │ │ ┌─────────┐ ┌───────────┐ │ │ │Classifier│─▶│Tier Router│──┐ │ │ └─────────┘ └───────────┘ │ │ │ ▼ │ │ ┌──────────┐ ┌───────────┐ │ │ │Compressor│ │Cache Layout│ │ │ └──────────┘ └───────────┘ │ │ │ │ │ │ └─────┬──────┘ │ │ ▼ │ │ ┌──────────┐ ┌───────────┐ │ │ │Tool Gate │ │Ver. Budget│ │ │ └──────────┘ └───────────┘ │ │ │ │ │ │ ▼ ▼ │ │ ┌──────────────────────────┐ │ │ │ Retry + Recovery + Doom │ │ │ └──────────────────────────┘ │ │ │ │ └─────────────┼────────────────────┘ ▼ Agent API Call ``` ## Quick Start ```python from aco import ACOPipeline # v1 DistilBERT specialists are the production models (v2 regressed — do not use). aco = ACOPipeline(use_v2=False) result = aco.optimize( request="Fix the connection pool timeout in database.py and run the tests", available_tools=["edit_file", "run_tests", "git_diff"], context={"system_prompt": "...", "tool_definitions": "..."} ) print(result.routing) # Which model to use print(result.tool_decisions) # Which tools to call print(result.needs_verify) # Whether to verify output print(result.context_budget) # How much context to include print(result.cost_estimate) # Expected cost ``` ## Install ```bash pip install transformers torch datasets git clone https://huggingface.co/narcolepticchicken/agent-cost-optimizer cd agent-cost-optimizer pip install -e . ``` ## Specialist Models (v1 — production) Metrics below are **independently verified** on held-out test splits (not training logs): | Model | Base | Params | Accuracy | F1 Macro | Test N | |-------|------|--------|----------|----------|--------| | [Tier Router](https://huggingface.co/narcolepticchicken/aco-specialists-tier-router) | DistilBERT | 67M | 0.70 | 0.67 | 1260 | | [Tool Gater](https://huggingface.co/narcolepticchicken/aco-specialists-tool-gater) | DistilBERT | 67M | 0.98 | 0.92 | 1695 | | [Verifier Gater](https://huggingface.co/narcolepticchicken/aco-specialists-verifier-gater) | DistilBERT | 67M | 0.65 | 0.65 | 863 | **v2 models (ModernBERT-base, 149M):** trained but **not recommended**. The architecture upgrade + focal loss did not help: tier-router flat (+0.7pp), verifier-gater flat (+1.3pp), and tool-gater **regressed −19.6pp F1** (ModernBERT overfits the 5.6% minority class). Raw comparison in `v2_verification_results.json`. Use the v1 models above. ## Key Papers | Finding | Paper | |---------|-------| | BERT classifier routing → 2x cost reduction | RouteLLM (2406.18665) | | Guideline compression → 26-54% token reduction | ACON (2510.00615) | | Critics cause collapse when overused | Intervention Paradox | | Systematic component optimization → lower cost | Efficient Agents (2508.02694) | | Cascade + self-consistency → cheaper | In-Context Distillation (2512.02543) | ## Documentation - [AUDIT.md](AUDIT.md) — independent verification of all metric & cost claims - [TRUTH.md](TRUTH.md) — honest state of the project - [CORRECTED_REPORT.md](CORRECTED_REPORT.md) — corrected cascade cost analysis - [Literature Review](docs/literature_review.md) - [Context Compression](aco/context_compression.py) — ACON + TALE + cache layout ## License MIT ## Generated by ML Intern This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub. - Try ML Intern: https://smolagents-ml-intern.hf.space - Source code: https://github.com/huggingface/ml-intern
