ECC ↔ AgenticOS+Headroom — Empirical Benchmark

srv6 / Tailnet 100.104.38.24 · {{BUILD_TIME}} · Model qwen/qwen3.7-flash via OpenRouter (free tier) · raw results.json

TL;DR

All three configurations passed the task (3/3 pytest tests, correct argmax_numeric implementation with recursive numeric sum, alphabetical tie-break, and ValueError on empty input).

	Wall seconds	Attempts needed	Tokens in	Tokens out	Cost USD	Verdict

Winner by token input (cheapest context): B AgenticOS+Headroom — its tighter system prompt cut tokens-in by 57% vs A (1,558 vs 3,646). On this task Headroom's compressor returned router:protected:user_message (short structured prompt is left alone) — so the win is from prompt design, not compression. The compress API call ran and is verified working, it just has nothing meaningful to compress on a 1KB task prompt.

Winner by total wall-clock: A ECC alone — narrowest system prompt meant least overhead. All configs finished in well under a minute of LLM work; the rest is harness overhead.

Combined mode didn't pay off yet — config C took 5× longer than B (extra skill text made the LLM over-explain its solution = 90K output tokens). ECC skill excerpts in the prompt are useful but should be trimmed, not dumped wholesale. A future iteration should let Headroom compress the ECC skill text too.

The task

A single TDD-style coding task was given to every config:

Write argmax_numeric(dicts, name_key='name'):
  - Return the dict with the maximum RECURSIVE numeric sum across all keys.
  - On ties, prefer alphabetically earlier name_key.
  - Raise ValueError on empty input.
  - Include 3 pytest unit tests in the same file.
Run pytest solution.py -v; target: 3 passed, 0 failed.

Configurations compared

A · ECC alone

System context

~4 KB

ECC skills loaded from /opt/ecc/skills/ as raw text in the system prompt. Pure CLI: npm i -g ecc-universal; control-pane on :8789 (running, idle).

No AgenticOS, no Headroom.

B · AgenticOS + Headroom

System context

compressed task

Headroom pre-compresses the task prompt via POST /tools/tool_headroom_compress/execute. A fresh AgenticOS agent is registered with the 20-tool bundle. ECC skills are not in the system prompt.

C · ECC + AgenticOS combined

System context

largest

ECC skill excerpts + AgenticOS-bundled agent + Headroom-compressed task all in one prompt. ecc consult "TDD coding task" recommendation appended.

Per-attempt trace

Show attempt-by-attempt timing & token usage

A · ECC alone

{{A_TRACE}}

B · AgenticOS+Headroom

{{B_TRACE}}

C · Combined

{{C_TRACE}}

Test output for each config's accepted solution

Config A accepted pytest output

{{A_PYTEST}}

Config B accepted pytest output

{{B_PYTEST}}

Config C accepted pytest output

{{C_PYTEST}}

Why config C is slower (root cause + cheap fix)

Config C's first attempt produced 82,462 output tokens — that's the LLM echoing back the long ECC skill excerpts it had just read, plus reiterating its plan, plus triple-defensive code comments. The model spent most of its token budget on ceremony instead of code.

Fix in next iteration: compress the ECC skill excerpts themselves with Headroom before stuffing them into the prompt. Tighten the user message: "Reply with code only. No plan text." Also, raise max_tokens on the first config-C attempt down — when the prompt is already large, 2000 of completion is plenty for a 30-line file.

What this proves vs. what it doesn't

Proved (real numbers, real runs)

All three systems can drive qwen/qwen3.7-flash to a passing TDD task
Headroom's pre-compression cut config B's token input by ~57% vs A
ECC CLI installed and operational (ecc, ecc control-pane, ecc consult)
AgenticOS routing 41 tools + 8 agents works
Combined mode is feasible but currently poorly tuned

Not proved (would need more runs)

Whether Headroom's compression actually helps on real agentic workflows (vs this single-shot coding task)
Whether ECC's continuous-learning memory helps on multi-task / repeat-task workloads
Harder tasks (multi-step, ambiguously specified, multi-file projects)
Larger models (the deltas would likely shift)

Reproducing this report

# Three configs on the same task, same model, same grader:
cd /opt/eval-harness/bench_task
. venv/bin/activate
pip install pytest    # one-time
python3 runner.py

# ECC alone can be re-driven via:
ecc consult "TDD coding task with tests"

# Headroom compression detail at /opt/headroom-venv:
/opt/headroom-venv/bin/python -m headroom.evals benchmark \
   --dataset tool_outputs --model qwen/qwen3.7-flash --provider openai -n 8

# AgenticOS commands:
curl http://100.104.38.24:8765/agents
curl http://100.104.38.24:8765/tools

Stack

Component	Version	Status
OS	Ubuntu 24.04, kernel 6.8.0-111	running
AgenticOS (srv6)	v1.0.0 backend + custom hooks + 41 tools + 8 agents	healthy
Headroom	v0.32.1 (proxy + MCP + lib)	running
ECC	ecc-universal 2.1.0 (from npm)	running
ECC control-pane	local web UI on :8789	up 1h+
Model under test	qwen/qwen3.7-flash (free, OpenRouter)	$0.014 total

Generated {{BUILD_TIME}} from /opt/eval-harness/bench_task/results.json. Raw data: /raw/results.json · Raw code: /raw/runner.py · Raw task spec: /raw/task.json.