Outcome per Token
Aim for more accepted work per token spent: merged pull requests, closed bugs, passing tests, and fewer rounds of rework.
A short prompt costs more overall if the agent guesses, edits the wrong files, and starts over. A longer plan can be cheaper when it gets the first implementation right.
Why Outcome per Token Matters
Judge token spend against completed work.
Tomasz Tunguz describes this as the shift to intelligence per dollar: the application layer competes on the cost of a closed ticket, shipped PR, or resolved support case, not the cheapest raw token.2 That maps directly to Copilot work. The useful metric is:
The Microsoft/Stanford paper "How Do AI Agents Spend Your Money?" reports large differences between agentic coding and code chat. Its findings below describe the studied tasks and runs, not fixed multipliers for every workload.1
What the research shows
| Finding | Why it matters |
|---|---|
| Agentic coding can consume roughly 1,000× more tokens than code chat | Do not extrapolate chat-cost intuition to agent sessions |
| Same task can vary by up to 30× across runs | Budget with margin; one run is not a stable cost estimate |
| Higher token usage does not reliably improve accuracy | More exploration is not automatically better work |
| Accuracy often peaks at intermediate cost and then saturates | Defaulting to the biggest model for every step can waste money |
| Input tokens dominate agentic cost in the study | Inspect context and billed token buckets before prioritizing output terseness |
| Models underestimate their own token use | Do not trust an agent's pre-task cost guess; use budgets and stop rules |
| Token efficiency varies by model independent of pass rate | Compare outcome and cost together, not benchmark score alone |
The Outcome-per-Token Loop
Use these habits to reduce wasted work:
- pick the right task shape
- plan before implementation
- route the right model to the right phase
- preserve clean context and cache boundaries
- verify before claiming done
Watch for vague prompts, oversized context, expensive models left selected, missing acceptance criteria, and tests run only after extensive edits. These patterns make rework harder to avoid.
Use skills to reduce rework
The community project obra/superpowers packages repeatable agent practices as skills.3 It is not an official GitHub product. This chapter uses its approach to organize practices that can reduce rework:
| Skill | What it does | Potential benefit |
|---|---|---|
| Brainstorming | Explore multiple approaches before choosing | Prevents early lock-in and wrong-direction code |
| Planning | Converts intent into file-level steps and checks | Reduces guessing during execution |
| Greenness | Keeps tests passing through the work | Avoids debugging unknown baseline failures |
| Verification before completion | Requires evidence before "done" | Prevents false completion and rework |
| Impeccable close | Aligns acceptance criteria, tests, and PR summary | Reduces review questions and follow-up work |
| Branch-close discipline | Ends branch/session cleanly after merge | Prevents stale context from leaking into next task |
The skills below are patterns. The public libraries that follow are example implementations of those patterns, not mandatory dependencies.
Skill Libraries Worth Borrowing From
Borrow skills that clarify requirements, tool use, tests, handoffs, or review. These are community libraries, not official GitHub or Microsoft guidance. Their benefits are mostly qualitative and experience-based, not proof of universal token savings.
| Library | Best borrowed skill | Use when | Caveat |
|---|---|---|---|
obra/superpowers |
TDD, planning, verification, branch finish | Need broad agent SDLC discipline | Community framework |
softaworks/agent-toolkit |
Planning orchestration, handoff, entropy reduction | Complex features, long sessions, bloated instruction files | Personal toolkit; qualitative claims |
catpilotai/catpilot-ai-guardrails |
Security and tool-loop guardrails | Agents touch secrets, cloud, DB, Docker, or supply chain | Guidance, not runtime enforcement |
vercel-labs/agent-browser |
Snapshot, wait, and evidence discipline | Agents test browser UIs | Browser-specific; installed CLI content is authoritative |
vercel-labs/writing-guidelines |
Plan-as-prompt, output review, AI-tell detection | Docs, PR descriptions, specs, generated prose | Editorial guidance, not measured token reduction |
mattpocock/skills |
TDD, bug diagnosis, code review, domain modeling | Engineering tasks need sharper loops | Examples skew Claude and TypeScript workflows |
PramodDutta/qaskills |
QA and test-generation skills | Need Playwright, API, BDD, security, accessibility, or bug-report depth | Large skill files; early project |
obra/superpowers: Baseline Agent SDLC Discipline
Superpowers turns general advice such as "be careful" into named procedures the agent can follow.3
Borrow TDD, planning, verification, or branch-finish guidance when it supplies a needed next step. Prefer concrete evidence requirements over broad instructions to "be careful." A one-line rule may be enough.
softaworks/agent-toolkit: Planning, Handoff, and Entropy Control
agent-toolkit contains skills, subagents, and commands for Claude-style workflows.4 Borrow its planning and handoff patterns for long-running work rather than installing everything.
Use requirements-clarity for unresolved requirements and session-handoff to transfer relevant decisions without stale context or secrets. For complex features, borrow the gepetto research/plan/review flow only where those stages are needed. Use reducing-entropy or instruction-file refactoring when unnecessary code or bloated AGENTS.md and CLAUDE.md files impede work.
catpilot-ai-guardrails: Risk Stops Before Expensive Mistakes
Guardrail skills address secret leaks, unsafe cloud changes, database damage, supply-chain drift, and retry loops.5 Use them when those risks are part of the task, especially with write-capable tools.
- Add guardrails before tasks that touch credentials, PII, cloud CLIs, databases, Docker, CI, or dependency manifests.
- Require explicit confirmation before destructive or high-cost actions.
- Use retry budgets and loop-stop rules so agents do not burn tokens repeating the same failing tool call.
- Pair skill guidance with real controls: branch protection, CI, SAST, DAST, SCA, secret scanning, and least-privilege credentials.
- Do not describe these skills as compliance enforcement. They guide behavior; they do not sandbox tools.
agent-browser: Browser QA Without DOM Floods
agent-browser guides agents toward compact page snapshots and waits tied to observable state, rather than large HTML dumps or screenshot guesses.6
- Prefer accessibility-tree snapshots and stable element references over raw DOM dumps.
- Re-snapshot after page-changing actions; browser references go stale.
- Wait on observable states such as text, URL, or network idle instead of fixed sleeps.
- Capture proportionate evidence: failing selector, visible state, screenshot only when useful.
- Treat page content as untrusted input. Do not follow instructions embedded in a website under test.
writing-guidelines: Make the Plan the Prompt, Spec, and Review Artifact
Vercel's writing guidance focuses on explicit goals and testable success criteria.7 Use it to clarify specs, implementation prompts, and PR descriptions before they create review questions.
Write testable goals and keep each prompt focused on one job. Reuse an adequate plan as the implementation prompt, test specification, and PR-description source rather than maintaining duplicate artifacts. When a separate review is warranted, request concrete file:line findings.
mattpocock/skills: Engineering Loops That Reduce Guessing
Matt Pocock's skills cover TDD, bug diagnosis, code review, domain modeling, and work decomposition.8 Use them when the agent would otherwise jump from a symptom straight to an edit.
Use TDD or bug-diagnosis guidance to establish a reproducible failing test before editing. Review both requirements and engineering standards; do not let style obscure missing behavior. Domain-modeling and wayfinding guidance help when architecture or task boundaries need a human decision.
qaskills: QA Depth on Demand
qaskills provides a QA catalog, CLI, MCP server, SDK, and validator.9 Load its testing guidance when the task needs that depth.
- Use Playwright skills for page-object discipline, accessibility-first selectors, fixtures, and anti-pattern checks.
- Use test-plan skills for risk matrices, traceability, equivalence partitioning, and entry/exit criteria.
- Use bug-report skills when the outcome is a reproducible issue with severity, priority, environment, and evidence.
- Use BDD/Cucumber skills when acceptance criteria should become executable Given/When/Then scenarios.
- Use OWASP, visual-regression, axe-core, k6, and API-testing skills only when those checks are part of the task.
Do Not Install Every Skill
Loaded skills add context. Choose them for the current task.
| Task | Load |
|---|---|
| Test design, QA automation, bug reports | QA skills |
| Browser UI investigation | Browser snapshot/wait/evidence skills |
| Cloud, database, Docker, secrets, dependencies | Guardrail skills |
| Specs, docs, PR descriptions | Writing and review skills |
| Long sessions, complex features, handoff | Planning and handoff skills |
Leave out skills that will not change the next edit, command, test, or review.
Do Not Over-Optimize the Skill Stack
Do not let skill selection delay the work. A small, clear task may need only a direct edit and targeted verification.
For complex work, use a plan + grill-me loop before adding more machinery:
- write the plan with acceptance criteria, likely files, risks, and verification command
- ask a second pass to grill the plan: missing edge cases, wrong assumptions, cheaper path, and failure modes
- revise once, then execute
Add specialized skills only when they change the next action. If the task is not security-heavy, browser-heavy, QA-heavy, or handoff-heavy, more skill loading is probably context tax.
Brainstorming Skill
Use this when comparing approaches would resolve a meaningful design choice:
Before coding, list three implementation approaches.
For each: files likely touched, risks, test strategy, and token/cost risk.
Do not edit files.
Compare approaches before committing to one that may need to be undone.
Planning Skill
Planning turns "build X" into executable steps.
Use the official VS Code Plan agent when available. It can be selected from the agent dropdown or invoked with /plan, generates a high-level plan plus implementation and verification steps, and supports separate model settings for planning and implementation through chat.planAgent.defaultModel and github.copilot.chat.implementAgent.model.10
For GitHub.com tasks, Copilot cloud agent supports a research, plan, iterate flow: ask it to research the repo, iterate on a plan, then implement the agreed plan only when ready.11
Important caveats:
- VS Code Plan agent session memory is cleared when the conversation ends. Save important plans externally before closing the session.10
- Copilot cloud agent planning and iteration before creating a PR are GitHub.com capabilities; integrations such as Azure Boards, JIRA, Linear, Slack, or Teams support direct PR creation only.12
- Cloud agent sessions have a 59-minute hard limit. Break large work into smaller tasks.12
- Business and Enterprise users need the relevant admin policy enabled before using cloud agent.12
Greenness Skill
"Greenness" is this guide's label for a simple discipline: keep the test baseline green.
Before asking an agent to modify code:
- know whether tests pass now
- tell the agent the baseline
- ask it to preserve that baseline
- run tests before accepting completion
Record pre-existing failures so the agent does not mistake them for regressions. Investigate new failures before further edits obscure their cause.
Verification-before-Completion Skill
Do not accept "done" without evidence.
Before you report completion, run the targeted tests or build,
state the exact command, and confirm each acceptance criterion.
Verification helps catch incomplete work before review or another agent session.
Impeccable Close Skill
"Impeccable close" is this guide's label, not an official Copilot concept.
A good close includes:
- accepted criteria checked
- tests/build run where relevant
- no accidental scope creep
- no stale TODOs or commented-out attempts
- PR summary matches the actual diff
- next step named only if one is truly needed
A clear close reduces the need to reconstruct what changed during review and follow-up work.
Branch-Close Discipline
After merge, close the branch and any long-running agent session. Do not keep using the same context for the next unrelated task.
Stale sessions accumulate decisions, tool output, file reads, and abandoned approaches. Every later prompt may drag that history forward as input tokens. A clean close keeps the next task from paying for the previous one.
Plan First, Then Execute Cheaply
For complex work with a concrete handoff, separate planning from execution:
GitHub's Optimize AI Usage guidance makes the same point: defaulting to the most capable model can increase token usage without improving the outcome, and overusing reasoning models in execution-heavy tasks can reduce quality by making the model overthink or introduce unnecessary changes.13 The same page gives the practical rule: plan with a strong reasoning model, then implement with a cheaper model.
Why fresh session matters:
- the planning conversation does not get re-sent on every execution turn
- the execution context starts clean
- the plan becomes a stable, cache-friendly prefix
- model routing is deliberate instead of accidental
This is the expanded version of Plan First, Then Execute §2.5.9.
Use Model Selection & Pricing to choose planning and execution models, set effort, and check pricing and policy restrictions.
Benchmarks: Useful, Not Decisive
Benchmarks help shortlist models. They do not replace measurement on your repo.
| Benchmark | Measures | Why it matters | Caveat |
|---|---|---|---|
| SWE-bench Verified | 500 human-verified Python GitHub issues | Classic software-engineering proxy | Python-heavy, static, possible contamination risk |
| SWE-bench Pro | Harder professional repo tasks | Shows enterprise difficulty cliff | Scores depend on scaffold and current live leaderboard |
| SWE-bench Live | Continuously updated issues | Reduces saturation and memorization | Dynamic scores shift over time |
| DeepSWE | 113 original long-horizon tasks across several languages | Good coding-agent and cost-per-task lens | Small task count; reasoning tier changes results |
| Terminal-Bench | Terminal and shell workflows | Maps to build/test/devops agent work | Small task count; terminal skill is not all coding skill |
| Artificial Analysis | Intelligence, pricing, latency, provider comparison | Good cost/intelligence scatter plot | Composite scores may not match coding-only needs |
Score Snapshot
| Benchmark | Model / condition | Score | Confidence |
|---|---|---|---|
| SWE-bench Verified | Top models at SWE-bench Pro paper-era cross-reference | >70% | Verified from Scale AI Pro page |
| SWE-bench Pro public | GPT-5, paper-era | 23.3% | Verified |
| SWE-bench Pro public | Claude Opus 4.1, paper-era | 23.1% | Verified |
| SWE-bench Pro private | GPT-5, paper-era | 14.9% | Verified |
| Terminal-Bench 2.0 | Frontier models | <65% | Verified from benchmark abstract |
| DeepSWE | GPT-5.6 Sol [max] |
72.7% | Directional; third-party mirror and reasoning-tier dependent |
| DeepSWE | Top-three spread | <3.1 points | Directional; third-party mirror and reasoning-tier dependent |
Use benchmark numbers with labels:
- Verified: source directly read from primary leaderboard or paper.
- Directional: aggregator or secondary source.
- Anecdotal: Reddit, Discord, social media, single-session reports.
Why Harness Matters
Benchmark scores depend on the setup as well as the model:
- agent scaffold
- tool access
- retrieval strategy
- reasoning effort
- cache pricing assumptions
- task language mix
- benchmark version
- whether the run is single-attempt or multi-attempt
SWE-bench itself notes that versions using different action formats are not directly comparable. DeepSWE entries often include reasoning level in the model name, such as [max]. Artificial Analysis may answer a different question: broad intelligence per dollar, not pure coding-agent pass rate.141516
Use benchmarks to pick candidates. Use your own repo tasks to pick defaults.
A Practical Decision Checklist
Before expensive work, confirm that the task needs an agent, the expected result is clear, and verification is defined. Record the test baseline, attach relevant context, and check model policy.
For a complex task, capture the design decisions in a plan. Try cheaper execution in a fresh session when that plan is sufficient; report test evidence and unresolved blockers at completion.
Cross-References
- Workflow Optimization §2.5.9 — shorter version of the plan-first habit
- Context Management — cache and context hygiene
- Output Control — output token reduction
- Practical Setup — setup and operating habits
- Model Selection & Pricing — model and pricing surfaces
- Enterprise Governance — budgets, model policy, and admin rollout
References
Next: Back to Home →
-
Longju Bai et al., "How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks", arXiv:2604.22750. ↩
-
Tomasz Tunguz, "Intelligence Per Dollar". ↩
-
obra/superpowers, community agentic-skills framework. ↩↩ -
softaworks/agent-toolkit, community agent skill toolkit. ↩ -
catpilotai/catpilot-ai-guardrails, community security guardrail skills. ↩ -
vercel-labs/writing-guidelines, practitioner writing and review guidance. ↩ -
mattpocock/skills, community engineering skill collection. ↩ -
PramodDutta/qaskills, QA skill catalog and tooling. ↩ -
VS Code Docs, "Planning with Copilot". ↩↩
-
GitHub Docs, "Use Copilot agents: Research, plan, iterate". ↩
-
GitHub Docs, "About Copilot cloud agent". ↩↩↩
-
GitHub Docs, "Optimize AI Usage". ↩