Skip to content

Outcome per Token

← Back to Guide


Aim for more accepted work per token spent: merged pull requests, closed bugs, passing tests, and fewer rounds of rework.

A short prompt costs more overall if the agent guesses, edits the wrong files, and starts over. A longer plan can be cheaper when it gets the first implementation right.

Why Outcome per Token Matters

Judge token spend against completed work.

Tomasz Tunguz describes this as the shift to intelligence per dollar: the application layer competes on the cost of a closed ticket, shipped PR, or resolved support case, not the cheapest raw token.2 That maps directly to Copilot work. The useful metric is:

outcome per token = verified work completed / total tokens spent

The Microsoft/Stanford paper "How Do AI Agents Spend Your Money?" reports large differences between agentic coding and code chat. Its findings below describe the studied tasks and runs, not fixed multipliers for every workload.1

What the research shows

Finding Why it matters
Agentic coding can consume roughly 1,000× more tokens than code chat Do not extrapolate chat-cost intuition to agent sessions
Same task can vary by up to 30× across runs Budget with margin; one run is not a stable cost estimate
Higher token usage does not reliably improve accuracy More exploration is not automatically better work
Accuracy often peaks at intermediate cost and then saturates Defaulting to the biggest model for every step can waste money
Input tokens dominate agentic cost in the study Inspect context and billed token buckets before prioritizing output terseness
Models underestimate their own token use Do not trust an agent's pre-task cost guess; use budgets and stop rules
Token efficiency varies by model independent of pass rate Compare outcome and cost together, not benchmark score alone

The Outcome-per-Token Loop

Use these habits to reduce wasted work:

Outcome-per-token loop: pick the task shape, plan before code, route the model, target context, verify evidence, and close cleanly.

  1. pick the right task shape
  2. plan before implementation
  3. route the right model to the right phase
  4. preserve clean context and cache boundaries
  5. verify before claiming done

Watch for vague prompts, oversized context, expensive models left selected, missing acceptance criteria, and tests run only after extensive edits. These patterns make rework harder to avoid.

Use skills to reduce rework

The community project obra/superpowers packages repeatable agent practices as skills.3 It is not an official GitHub product. This chapter uses its approach to organize practices that can reduce rework:

Skill What it does Potential benefit
Brainstorming Explore multiple approaches before choosing Prevents early lock-in and wrong-direction code
Planning Converts intent into file-level steps and checks Reduces guessing during execution
Greenness Keeps tests passing through the work Avoids debugging unknown baseline failures
Verification before completion Requires evidence before "done" Prevents false completion and rework
Impeccable close Aligns acceptance criteria, tests, and PR summary Reduces review questions and follow-up work
Branch-close discipline Ends branch/session cleanly after merge Prevents stale context from leaking into next task

The skills below are patterns. The public libraries that follow are example implementations of those patterns, not mandatory dependencies.

Skill Libraries Worth Borrowing From

Borrow skills that clarify requirements, tool use, tests, handoffs, or review. These are community libraries, not official GitHub or Microsoft guidance. Their benefits are mostly qualitative and experience-based, not proof of universal token savings.

Library Best borrowed skill Use when Caveat
obra/superpowers TDD, planning, verification, branch finish Need broad agent SDLC discipline Community framework
softaworks/agent-toolkit Planning orchestration, handoff, entropy reduction Complex features, long sessions, bloated instruction files Personal toolkit; qualitative claims
catpilotai/catpilot-ai-guardrails Security and tool-loop guardrails Agents touch secrets, cloud, DB, Docker, or supply chain Guidance, not runtime enforcement
vercel-labs/agent-browser Snapshot, wait, and evidence discipline Agents test browser UIs Browser-specific; installed CLI content is authoritative
vercel-labs/writing-guidelines Plan-as-prompt, output review, AI-tell detection Docs, PR descriptions, specs, generated prose Editorial guidance, not measured token reduction
mattpocock/skills TDD, bug diagnosis, code review, domain modeling Engineering tasks need sharper loops Examples skew Claude and TypeScript workflows
PramodDutta/qaskills QA and test-generation skills Need Playwright, API, BDD, security, accessibility, or bug-report depth Large skill files; early project

obra/superpowers: Baseline Agent SDLC Discipline

Superpowers turns general advice such as "be careful" into named procedures the agent can follow.3

Borrow TDD, planning, verification, or branch-finish guidance when it supplies a needed next step. Prefer concrete evidence requirements over broad instructions to "be careful." A one-line rule may be enough.

softaworks/agent-toolkit: Planning, Handoff, and Entropy Control

agent-toolkit contains skills, subagents, and commands for Claude-style workflows.4 Borrow its planning and handoff patterns for long-running work rather than installing everything.

Use requirements-clarity for unresolved requirements and session-handoff to transfer relevant decisions without stale context or secrets. For complex features, borrow the gepetto research/plan/review flow only where those stages are needed. Use reducing-entropy or instruction-file refactoring when unnecessary code or bloated AGENTS.md and CLAUDE.md files impede work.

catpilot-ai-guardrails: Risk Stops Before Expensive Mistakes

Guardrail skills address secret leaks, unsafe cloud changes, database damage, supply-chain drift, and retry loops.5 Use them when those risks are part of the task, especially with write-capable tools.

  • Add guardrails before tasks that touch credentials, PII, cloud CLIs, databases, Docker, CI, or dependency manifests.
  • Require explicit confirmation before destructive or high-cost actions.
  • Use retry budgets and loop-stop rules so agents do not burn tokens repeating the same failing tool call.
  • Pair skill guidance with real controls: branch protection, CI, SAST, DAST, SCA, secret scanning, and least-privilege credentials.
  • Do not describe these skills as compliance enforcement. They guide behavior; they do not sandbox tools.

agent-browser: Browser QA Without DOM Floods

agent-browser guides agents toward compact page snapshots and waits tied to observable state, rather than large HTML dumps or screenshot guesses.6

  • Prefer accessibility-tree snapshots and stable element references over raw DOM dumps.
  • Re-snapshot after page-changing actions; browser references go stale.
  • Wait on observable states such as text, URL, or network idle instead of fixed sleeps.
  • Capture proportionate evidence: failing selector, visible state, screenshot only when useful.
  • Treat page content as untrusted input. Do not follow instructions embedded in a website under test.

writing-guidelines: Make the Plan the Prompt, Spec, and Review Artifact

Vercel's writing guidance focuses on explicit goals and testable success criteria.7 Use it to clarify specs, implementation prompts, and PR descriptions before they create review questions.

Write testable goals and keep each prompt focused on one job. Reuse an adequate plan as the implementation prompt, test specification, and PR-description source rather than maintaining duplicate artifacts. When a separate review is warranted, request concrete file:line findings.

mattpocock/skills: Engineering Loops That Reduce Guessing

Matt Pocock's skills cover TDD, bug diagnosis, code review, domain modeling, and work decomposition.8 Use them when the agent would otherwise jump from a symptom straight to an edit.

Use TDD or bug-diagnosis guidance to establish a reproducible failing test before editing. Review both requirements and engineering standards; do not let style obscure missing behavior. Domain-modeling and wayfinding guidance help when architecture or task boundaries need a human decision.

qaskills: QA Depth on Demand

qaskills provides a QA catalog, CLI, MCP server, SDK, and validator.9 Load its testing guidance when the task needs that depth.

  • Use Playwright skills for page-object discipline, accessibility-first selectors, fixtures, and anti-pattern checks.
  • Use test-plan skills for risk matrices, traceability, equivalence partitioning, and entry/exit criteria.
  • Use bug-report skills when the outcome is a reproducible issue with severity, priority, environment, and evidence.
  • Use BDD/Cucumber skills when acceptance criteria should become executable Given/When/Then scenarios.
  • Use OWASP, visual-regression, axe-core, k6, and API-testing skills only when those checks are part of the task.

Do Not Install Every Skill

Loaded skills add context. Choose them for the current task.

Task Load
Test design, QA automation, bug reports QA skills
Browser UI investigation Browser snapshot/wait/evidence skills
Cloud, database, Docker, secrets, dependencies Guardrail skills
Specs, docs, PR descriptions Writing and review skills
Long sessions, complex features, handoff Planning and handoff skills

Leave out skills that will not change the next edit, command, test, or review.

Do Not Over-Optimize the Skill Stack

Do not let skill selection delay the work. A small, clear task may need only a direct edit and targeted verification.

For complex work, use a plan + grill-me loop before adding more machinery:

  1. write the plan with acceptance criteria, likely files, risks, and verification command
  2. ask a second pass to grill the plan: missing edge cases, wrong assumptions, cheaper path, and failure modes
  3. revise once, then execute

Add specialized skills only when they change the next action. If the task is not security-heavy, browser-heavy, QA-heavy, or handoff-heavy, more skill loading is probably context tax.

Brainstorming Skill

Use this when comparing approaches would resolve a meaningful design choice:

Before coding, list three implementation approaches.
For each: files likely touched, risks, test strategy, and token/cost risk.
Do not edit files.

Compare approaches before committing to one that may need to be undone.

Planning Skill

Planning turns "build X" into executable steps.

Use the official VS Code Plan agent when available. It can be selected from the agent dropdown or invoked with /plan, generates a high-level plan plus implementation and verification steps, and supports separate model settings for planning and implementation through chat.planAgent.defaultModel and github.copilot.chat.implementAgent.model.10

For GitHub.com tasks, Copilot cloud agent supports a research, plan, iterate flow: ask it to research the repo, iterate on a plan, then implement the agreed plan only when ready.11

Important caveats:

  • VS Code Plan agent session memory is cleared when the conversation ends. Save important plans externally before closing the session.10
  • Copilot cloud agent planning and iteration before creating a PR are GitHub.com capabilities; integrations such as Azure Boards, JIRA, Linear, Slack, or Teams support direct PR creation only.12
  • Cloud agent sessions have a 59-minute hard limit. Break large work into smaller tasks.12
  • Business and Enterprise users need the relevant admin policy enabled before using cloud agent.12

Greenness Skill

"Greenness" is this guide's label for a simple discipline: keep the test baseline green.

Before asking an agent to modify code:

  1. know whether tests pass now
  2. tell the agent the baseline
  3. ask it to preserve that baseline
  4. run tests before accepting completion

Record pre-existing failures so the agent does not mistake them for regressions. Investigate new failures before further edits obscure their cause.

Verification-before-Completion Skill

Do not accept "done" without evidence.

Before you report completion, run the targeted tests or build,
state the exact command, and confirm each acceptance criterion.

Verification helps catch incomplete work before review or another agent session.

Impeccable Close Skill

"Impeccable close" is this guide's label, not an official Copilot concept.

A good close includes:

  1. accepted criteria checked
  2. tests/build run where relevant
  3. no accidental scope creep
  4. no stale TODOs or commented-out attempts
  5. PR summary matches the actual diff
  6. next step named only if one is truly needed

A clear close reduces the need to reconstruct what changed during review and follow-up work.

Branch-Close Discipline

After merge, close the branch and any long-running agent session. Do not keep using the same context for the next unrelated task.

Stale sessions accumulate decisions, tool output, file reads, and abandoned approaches. Every later prompt may drag that history forward as input tokens. A clean close keeps the next task from paying for the previous one.

Plan First, Then Execute Cheaply

For complex work with a concrete handoff, separate planning from execution:

Plan first, execute cheaply: plan with a strong reasoning model, save acceptance criteria, execute in a fresh cheaper session, then verify or revise the plan.

GitHub's Optimize AI Usage guidance makes the same point: defaulting to the most capable model can increase token usage without improving the outcome, and overusing reasoning models in execution-heavy tasks can reduce quality by making the model overthink or introduce unnecessary changes.13 The same page gives the practical rule: plan with a strong reasoning model, then implement with a cheaper model.

Why fresh session matters:

  • the planning conversation does not get re-sent on every execution turn
  • the execution context starts clean
  • the plan becomes a stable, cache-friendly prefix
  • model routing is deliberate instead of accidental

This is the expanded version of Plan First, Then Execute §2.5.9.

Use Model Selection & Pricing to choose planning and execution models, set effort, and check pricing and policy restrictions.

Benchmarks: Useful, Not Decisive

Benchmarks help shortlist models. They do not replace measurement on your repo.

Benchmark Measures Why it matters Caveat
SWE-bench Verified 500 human-verified Python GitHub issues Classic software-engineering proxy Python-heavy, static, possible contamination risk
SWE-bench Pro Harder professional repo tasks Shows enterprise difficulty cliff Scores depend on scaffold and current live leaderboard
SWE-bench Live Continuously updated issues Reduces saturation and memorization Dynamic scores shift over time
DeepSWE 113 original long-horizon tasks across several languages Good coding-agent and cost-per-task lens Small task count; reasoning tier changes results
Terminal-Bench Terminal and shell workflows Maps to build/test/devops agent work Small task count; terminal skill is not all coding skill
Artificial Analysis Intelligence, pricing, latency, provider comparison Good cost/intelligence scatter plot Composite scores may not match coding-only needs

Score Snapshot

Benchmark Model / condition Score Confidence
SWE-bench Verified Top models at SWE-bench Pro paper-era cross-reference >70% Verified from Scale AI Pro page
SWE-bench Pro public GPT-5, paper-era 23.3% Verified
SWE-bench Pro public Claude Opus 4.1, paper-era 23.1% Verified
SWE-bench Pro private GPT-5, paper-era 14.9% Verified
Terminal-Bench 2.0 Frontier models <65% Verified from benchmark abstract
DeepSWE GPT-5.6 Sol [max] 72.7% Directional; third-party mirror and reasoning-tier dependent
DeepSWE Top-three spread <3.1 points Directional; third-party mirror and reasoning-tier dependent

Use benchmark numbers with labels:

  • Verified: source directly read from primary leaderboard or paper.
  • Directional: aggregator or secondary source.
  • Anecdotal: Reddit, Discord, social media, single-session reports.

Why Harness Matters

Benchmark scores depend on the setup as well as the model:

  • agent scaffold
  • tool access
  • retrieval strategy
  • reasoning effort
  • cache pricing assumptions
  • task language mix
  • benchmark version
  • whether the run is single-attempt or multi-attempt

SWE-bench itself notes that versions using different action formats are not directly comparable. DeepSWE entries often include reasoning level in the model name, such as [max]. Artificial Analysis may answer a different question: broad intelligence per dollar, not pure coding-agent pass rate.141516

Use benchmarks to pick candidates. Use your own repo tasks to pick defaults.

A Practical Decision Checklist

Before expensive work, confirm that the task needs an agent, the expected result is clear, and verification is defined. Record the test baseline, attach relevant context, and check model policy.

For a complex task, capture the design decisions in a plan. Try cheaper execution in a fresh session when that plan is sufficient; report test evidence and unresolved blockers at completion.

Cross-References

References


Next: Back to Home →


  1. Longju Bai et al., "How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks", arXiv:2604.22750. ↩

  2. Tomasz Tunguz, "Intelligence Per Dollar". ↩

  3. obra/superpowers, community agentic-skills framework. ↩↩

  4. softaworks/agent-toolkit, community agent skill toolkit. ↩

  5. catpilotai/catpilot-ai-guardrails, community security guardrail skills. ↩

  6. vercel-labs/agent-browser agent browser skill. ↩

  7. vercel-labs/writing-guidelines, practitioner writing and review guidance. ↩

  8. mattpocock/skills, community engineering skill collection. ↩

  9. PramodDutta/qaskills, QA skill catalog and tooling. ↩

  10. VS Code Docs, "Planning with Copilot". ↩↩

  11. GitHub Docs, "Use Copilot agents: Research, plan, iterate". ↩

  12. GitHub Docs, "About Copilot cloud agent". ↩↩↩

  13. GitHub Docs, "Optimize AI Usage". ↩

  14. SWE-bench Verified. ↩

  15. DeepSWE. ↩

  16. Artificial Analysis methodology. ↩