I stole GitHub Copilot's cost playbook for my open-source coding agent. One trick cut cost 23%. Their biggest one made mine worse.

The number that started it

48.1 million input tokens. Seven user messages.

That was one real session of FreeCode, my open-source CLI coding agent, replayed from its own logs in early August. Seven things I asked it to do, 229 requests to the model, and a conversation that had grown to 270K tokens and was never compacted once. Roughly forty times the input tokens a comparable agent needed for the same work. One session was most of a day’s quota.

I spent August fixing the history-shaped half of that problem: prompt-cache stability, compaction, deduplicated re-reads, parallel tool calls. It worked. But it left the other half untouched: the per-call payload. Every tool result, every recurring instruction, costs tokens the moment it is sent — and then again on every later request for the rest of the session, because an agent re-sends its whole conversation on every model call.

Then GitHub’s Copilot team published “How we make AI coding more cost-efficient without sacrificing task quality”, and it was exactly the half I had not done.

What GitHub actually said

The post describes four harness changes, each measured offline on agentic coding benchmarks and then validated in online A/B experiments on real traffic:

But the numbers were not the interesting part. The interesting part was the section called “The local metric trap.” Every aggressive per-call trim they tried first made the agent cheaper per call and more expensive per task. Cut a tool result too hard and the model does not shrug — it recovers. It re-runs the command. It re-reads the file. It takes an extra turn. And every extra turn re-sends the entire conversation. They named a tool that showed a “local win” while tokens and cost went up overall, and a tighter file-tool instruction that increased cost in production.

Their governing principle, which I adopted word for word: optimize for the outcome, not the tool call.

I decided to implement all four techniques in FreeCode. But with one rule I wrote at the top of the spec before touching any code: nothing flips a default by argument. Every change ships as an experiment behind a flag. A paired, interleaved A/B run on the eval suite measures the whole task end to end. The default only changes if cost drops while pass rate holds and repeated tool calls— the tell-tale of a recovery detour — do not rise.

That rule is the whole story. Here is what happened when I followed it.

Step 0: Measure before you cut

The first thing I did was not build. It was audit. For each of GitHub’s four techniques I wrote down where FreeCode actually stood, and I pulled a real recorded request out of the rollout log to see what a request carries:

So before writing a line I knew that GitHub’s “halve the prompt” would be worth roughly 3.4K tokens per request here — a number, not a hope.

The audit also turned up something that was not in GitHub’s list at all.

Step 1: The bug that made “recovery” a lie

FreeCode already had a recovery path for big tool outputs: the full text goes into a per-session output store, the model sees a capped head-and-tail view, and an output tool can page any of it back by line range or regex. Cheap recovery, no re-running.

Except it did not work for the outputs that mattered. The bash tool was cutting its output at 500KB beforethe store ever saw it. So for any large output the store held already-truncated text, and the beginning of a huge log was simply gone. The only “recovery” was re-running the command — the exact detour GitHub had measured as a net loss.

What I did:deleted bash’s own cap entirely. Bash returns the full output; the orchestrator owns every cap, in a fixed order: store the full text first, then produce the 30K-character model view, then the UI copy. I added a regression test that pushes a 600KB output through the pipeline and checks every byte is retrievable through the store while both outbound copies are capped.

No flag, no A/B — this was correctness, not an experiment. But it was a prerequisite for everything that followed: you cannot compress conservatively “with a recovery path” if the recovery path lies.

Step 2: Build the tripwire before the dangerous change

GitHub’s scariest anecdote was about prompt compression. When they shortened their tool guidance, a compressed instruction quietly lost the nuance that told the model to run independent agents in parallel. Everything got slower and more expensive, and no token count could have caught it. Only behavioral testing did.

FreeCode’s system prompt carries the same load-bearing instruction: batch independent tool calls into one response. Three file reads in one message cost one round trip instead of three, and each avoided round trip is an entire conversation not re-sent. And that instruction was just prose. Any future edit could weaken it and nothing would go red.

What I did:before compressing anything, I added a new expectation to FreeCode’s eval harness — “at least one model response in this run must emit N tool calls at once” — and a trajectory case whose correct answer is two concurrency-safe reads in a single assistant message. Two details I cared about: it scores what the model emitted, not what ran, so a batch that gets permission-denied still counts as batching; and the loader rejects N below 2, because “at least one tool call” would assert nothing.

Then I ran it against the current prompt to see it pass. A guard that has never been green guards nothing.

I am glad I did this first. It earned its keep within hours (Step 7).

Step 3: Content-aware output compression — GitHub’s biggest win

This was their 5.5%. The idea: a git diff, a grep result and an npm installlog are completely different things, but a naive “keep head and tail, drop the middle” cap treats them identically — and the middle of a build log is exactly where the one line that says ERROR: is hiding.

What I did: two pieces in two places, because each place knows something the other does not.

• Classify in the bash tool, because it is the only code that knows what command was run. Each result is tagged as source(cat, git diff, git show — never touched, the model asked for bytes), search(grep, rg, find — only consecutive duplicates collapse, a match line is never dropped), log(test, build, install output — keep head, tail and every failure-looking line, collapse the boring middle), or unclassified(untouched — conservative beats clever). A pipeline is classified by its last segment, so npm test | grep FAIL is search output, not a build log.
• Compress in the orchestrator, at the one place all tool output already gets capped, and crucially afterthe store put from Step 1 — so every collapsed region leaves a marker naming the retrieval handle. Nothing is ever more than one cheap call away.

Unit and property tests pin the guarantees: the search compressor provably drops no distinct line; the log compressor keeps every failure pattern.

And then I shipped it off by default, behind a flag. That felt strange for the technique with the biggest published number. It turned out to be the right call.

Step 4: The line-number experiment

FreeCode’s read tool prefixed every line of every file with its number: 42: …. Three to seven extra characters per line, on every file the model ever reads, re-sent on every later turn. GitHub removed theirs because nothing in their editing workflow consumed the numbers.

On paper FreeCode was in the same position — its edit tool matches strings, not line numbers, and navigation numbers come from grep -nand LSP output. But “on paper” is an argument. Maybe the model quietly leaned on visible numbering to cite file:line in answers, or to disambiguate a repeated string before editing. Those are two different risks and they need two different suites to answer: a coding suite (does the end state still pass?) and a judged suite (is the answer still good, with an independent model grading it?).

What I did:one deliberately boring change — the prefix is applied only when a flag says so. The one subtle detail: the flag is read per call, not at startup. The A/B runner flips environment between interleaved trials, and a startup-read variable would make both sides identical while the report confidently described an experiment that never ran. I allowlisted the key in the runner and wrote that rule down next to it.

Step 5: Fix the instrument

Here is the part I would have skipped a year ago. I ran the first output-compression A/B and it came back: 11 of 11 cases pass on both sides, quality unchanged. Great — and useless. The runner was reporting only pass rate. Tokens, cost, turns and repeated calls were all sitting in the per-trial results and being thrown away by the summary.

For a harness experiment, those numbers are the result. Quality holding is the precondition; cost moving is the point.

What I did:extended the A/B report so every side tallies tokens, USD (summed over priced trials only, so “free” and “unknown price” stay distinct), turns and repeated calls, and prints percentage deltas. Then I re-ran everything.

Step 6: The results

Every run below: same model on both sides (MiniMax-M3), both variants interleaved in one run so nothing else can confound the delta, three trials per case. The judged suite used a different model family as the grader, because the judge must never be the model under test.

Line numbers off — coding suite (11 cases)

Line numbers off — judged suite (6 cases, independent judge)

Both risks answered. Answer quality held without the numbers, edits still landed, and the cheaper side even repeated fewercalls — no hidden detour eating the saving. Notice that cost fell twice as fast as tokens: fewer turns means fewer whole-conversation re-sends, which is the quadratic term you actually pay for. The line numbers never earned their tokens. Default flipped to off the same day.

Why is my number several times GitHub’s 3.1%? Partly pricing, partly a small suite, but mostly because a small per-call saving compounds through the turn count. Their figure was normalized over an enormous, diverse production workload; mine is 11 tasks. I would not quote −23% as a universal truth. I would quote it as evidence that the prefix was pure cost for this agent.

Output compression on — coding suite (11 cases)

GitHub’s biggest win was my only loss. Correctness held — the classifier never broke anything — but the compressed side cost more. The trace shows why: the model saw an elision marker, decided it needed the middle, and spent extra turns paging it back through the output tool. Fifteen extra turns across the suite. That is the local metric trap, reproduced on my own codebase: every individual tool result got smaller, and the task got more expensive.

I had shipped it flag-off “because a default is earned, not asserted,” half expecting to flip it. Instead the measurement said no. The classifier and the flag stay in the codebase for a retry with different thresholds or a model that recovers more cheaply. The default stays off. A saved experiment is knowing what not to ship.

Step 7: Compressing the prompt — and the day the tripwire fired

With the guard from Step 2 green, I compressed the recurring guidance by hand, section by section: the system prompt, and the five fattest tool descriptions. I kept the parallel-batching instruction deliberately strong. Then I ran the release gate.

The first compressed version failed. A case called explore-mode-stays-readonly went red twice in a row. Reading the trial data, the model was burning an extra turn by calling the question tool in a mode where it gets rejected. I stashed the compression, re-ran on the original prompt as a control: it passed. The compression was convicted.

The culprit was one sentence I had cut as filler: “requesting input from the user is a blocking action.”Without it, the model reached for a question instead of reporting a blocked action plainly. Version two restored that rule, added an explicit “if an action is blocked, report it instead of asking,” and matched the original prompt’s score.

Read that again next to GitHub’s story. Their compression silently changed a parallelism behavior; mine silently changed an asking behavior. Neither is visible in a token count. Both were caught only by a behavioral test. The guard I built “just in case” in Step 2 caught a regression in Step 7 of the same day.

Put together, the recurring block that rides every request went from roughly 27.7K characters to roughly 19.4K — about 30% smaller, on the order of two thousand tokens saved on every single model call before the conversation has even started growing. (That last figure is derived from the recorded character counts, not a separate cost A/B; the gate measured quality, not dollars.) The judge’s own rationales were the nicest confirmation that the style survived: “direct, no unnecessary padding,” “omitting preamble as requested” — exactly the qualities a clumsy trim destroys.

Step 8: The technique that cost nothing

GitHub’s fourth technique — deliver finished background work inside a model call that was going to happen anyway, never in a dedicated “is it done yet?” turn — did not apply to FreeCode at the time. When the model issues several independent tool calls in one response, the batching layer runs them concurrently and returns every result in the same follow-up call. There was no polling turn to eliminate because nothing ran detached from the loop.

So I did not write code. I wrote a rule into the spec: a background completion is delivered as an ordinary tool result in the next already-happening model call, never via a retrieval turn. Four calls to process two results is the named anti-pattern.

Four days later FreeCode grew background shells — a long build or a dev server no longer holds the turn. And the rule immediately mattered, because that feature is pull-only today: the model learns a shell finished only by asking. The push notification that makes the result arrive on its own is designed and next on the list. The cheapest of the four techniques was adopting the constraint before the feature that could violate it existed.

What I actually learned

1. The measurement is the deliverable.The most valuable commit of the day was not a compression — it was making the A/B runner print cost. Without it I would have shipped output compression on a green pass rate and made my agent 11% more expensive while believing I had made it cheaper.

2. Ship every trim behind a flag, off.GitHub’s largest published saving was a loss on my workload. Their numbers are true for their traffic. “Evidence is local to the workload” is not a disclaimer; it is the reason to run your own experiment.

3. Build the guard before the change. Prompt edits fail silently and behaviorally. A token count cannot see a model that starts asking questions it should not ask. A one-line eval case can.

4. Watch repeated calls, not just tokens. A rising repeat count is the fingerprint of the recovery detour. It flagged the compression loss and confirmed the line-number win.

5. A negative result is still a result.The compression code stays. The flag stays. The verdict is written down with the numbers. The next person — probably me — will not re-argue it from first principles.

Four techniques from a GitHub post. One flipped a default and cut task cost by a fifth. One halved a prompt under a gate that caught the regression it caused. One was recorded as a rule and became relevant four days later. And one — the biggest on paper — was measured, rejected, and kept on the shelf. That last outcome is the one I am proudest of, because it is the one the post was actually about.

FreeCode is open source. The full spec, the A/B commands, and every number above live in the repo under docs/specs/2026-09-04-harness-cost-efficiency.mdand the “Cost efficiency” section of the docs site.