I built an open-source toolkit for finding the minimum token budget an AI agent needs to complete tasks successfully without reducing task quality.

AI agents are getting more capable, but they are also getting increasingly expensive in a way that is easy to overlook.

A single agent run may repeatedly pay for a system prompt, conversation history, retrieved context, memory, tool schemas, tool outputs, intermediate reasoning, and the final response. When the workflow becomes multi-step, token consumption compounds quickly.

Most benchmarks ask a binary question: did the agent complete the task?

I wanted to ask a second one:

How many tokens did it actually need to succeed?

That question led me to build Suffice, an open-source toolkit for measuring the minimum token budget an AI agent needs to complete a task while preserving a required level of quality.

The Problem: Agent Quality Is Only Half of the Equation

When we optimize an agent, it is tempting to focus on reducing prompt length, trimming history, lowering retrieval top_k, compressing tool descriptions, or limiting the final answer.

But optimizing token usage in isolation is dangerous.

A 200-token run that fails is not better than a 1,000-token run that succeeds.

That is the core rule behind Suffice:

Never optimize token usage independently of task success.

The objective is simple:

minimize tokens
subject to task success >= required quality threshold

Instead of asking for the smallest possible prompt or context window, Suffice looks for the smallest configuration that still satisfies the quality constraint you define.

What Suffice Measures

Suffice tracks token usage across the different parts of an agent workflow rather than treating every token as one undifferentiated number.

A trace can include categories such as:

• system
• user
• history
• context
• memory
• retrieval
• tools
• tool_output
• assistant_intermediate
• assistant_output
• cached_input
• reasoning

This makes it much easier to see where an agent is spending its token budget.

For example, two agents can have the same total token usage but completely different efficiency profiles: one may spend most of its budget on retrieved context, while another may spend it on tool schemas or a long final answer.

That distinction matters because each source requires a different optimization strategy.

The Two Metrics I Care About Most

Suffice calculates several metrics, but two are especially useful.

Minimum Successful Token Budget

This is the smallest observed token budget that reaches the required success rate.

If an experiment tests budgets of:

16, 32, 64, 128, 256

and 128 is the first budget that satisfies your quality requirement, then 128 becomes the Minimum Successful Token Budget for that experiment.

The important part is that the number is not meaningful without the success constraint beside it.

Token Efficiency Frontier

Suffice can also build a Token Efficiency Frontier from observed (budget, success, tokens) points.

That lets you inspect the trade-off between consuming more tokens and obtaining more reliable task performance.

For agent systems, this is often more useful than a single score. Real deployments usually involve a trade-off between quality, latency, cost, and context size rather than one universally optimal configuration.

A Candidate Must Preserve Quality

Suffice compares a baseline against candidate configurations and only accepts a candidate when it satisfies three conditions:

1. It reduces token usage.
2. It still reaches the configured minimum success rate.
3. Its success-rate drop stays within the maximum loss you allow.

A configuration can look dramatically cheaper and still be rejected if it degrades task quality too much.

That constraint is what makes the optimization useful instead of cosmetic.

What You Can Experiment With

The toolkit is framework-neutral, so the idea is not tied to one particular agent stack.

Some experiments I had in mind while building it include:

• shortening system prompts;
• trimming conversation history;
• reducing retrieval top_k;
• compacting tool schemas;
• compressing tool outputs;
• budgeting memory;
• comparing models under the same tasks;
• limiting final-answer size;
• testing different context-window configurations.

The interesting question is not simply “can I remove this context?”

It is:

Can I remove this context and still pass the task-quality threshold?

How an Experiment Looks

A Suffice experiment is configured in YAML.

name: prompt-budget-study
tasks: ../benchmarks/agent_efficiency.jsonl
seed: 42
system_prompt: Answer correctly and concisely.

model:
  provider: mock
  model: deterministic
  temperature: 0
  max_tokens: 128

token_budgets: [16, 32, 64, 128, 256]

optimization:
  minimum_success_rate: 0.95
  maximum_success_drop: 0.01

history_tail: 4
retrieval_top_k: 3
tool_description_mode: compact
tool_output_mode: structured
output_directory: ../runs

Then you can validate and run it from the CLI:

suffice validate examples/baseline.yaml
suffice run examples/baseline.yaml
suffice frontier examples/frontier.yaml

Each saved run contains reproducible artifacts including the configuration, manifest, per-case results, metrics, and a static HTML report.

runs/<run-id>/
├── config.yaml
├── manifest.json
├── cases.jsonl
├── metrics.json
└── report.html

You can also compare two runs directly:

suffice compare runs/<baseline-id> runs/<candidate-id>

Deterministic Evaluation Matters

One problem with evaluating agent optimization is that it is easy to introduce another model as the judge and end up measuring one probabilistic system with another probabilistic system.

Suffice currently supports deterministic evaluation for exact, contains, structured, and numeric outputs.

That is intentionally conservative.

For tasks where correctness can be determined mechanically, deterministic evaluation makes it much easier to know whether a token reduction actually preserved the required behavior.

For more subjective tasks, the benchmark and evaluator should be designed for that specific domain rather than pretending there is a universal definition of “good enough.”

Token Counts Need Provenance Too

Another detail I did not want to hide is that token accounting is not always equally precise across providers.

Suffice therefore records how a count was obtained. Token counts can be marked as:

provider_reported
exact_tokenizer
estimated
unavailable

Provider-reported usage is preferred. When exact counts are not available, the fallback can estimate them, but the result stays explicitly labeled as an estimate.

That sounds like a small implementation detail, but it matters if the goal is to compare experiments honestly.

Architecture

The core architecture is intentionally small:

JSONL tasks + YAML config
          │
          ▼
 AgentAdapter ── Mock / OpenAI-compatible
          │
          ▼
 AgentRunResult + TokenTrace
          │
          ├── deterministic Evaluator
          ├── Metrics / Comparison / Frontier
          └── JSON artifacts + static HTML

Adapters isolate model providers. Evaluators isolate correctness. Token traces isolate accounting.

That separation means the experiment harness does not need to be coupled to a specific agent framework, vector database, or model vendor.

The core runtime only depends on PyYAML, and the bundled deterministic mock adapter means the project can be exercised offline without an API key, internet connection, or GPU.

Using an OpenAI-Compatible Endpoint

Suffice also supports generic OpenAI-compatible chat-completions endpoints.

A configuration can specify:

model:
  provider: openai_compatible
  base_url: https://your-endpoint.example/v1
  api_key_env: MY_API_KEY

The configuration accepts the name of an environment variable rather than embedding the secret itself.

HTTP support is optional:

pip install -e ".[openai]"

What the Included Benchmark Does — and Does Not — Prove

The repository includes a small offline smoke benchmark covering factual QA, extraction, calculations, tool selection/results, multi-step work, explanation, comparison, and distractor context.

Its purpose is to exercise and validate the harness.

It is not evidence that a particular model universally needs a certain number of tokens, and I do not think such a universal number exists.

Every real application has different tasks, tools, context sources, failure costs, and definitions of success. A useful Minimum Successful Token Budget therefore has to be measured against a benchmark that represents the actual use case.

That is why I think of Suffice primarily as an experimentation framework rather than a leaderboard.

The Direction I Want to Explore Next

The current version focuses on measuring and comparing configurations. The natural next step is an Agent Token Optimizer that can automatically search for cheaper configurations while continually checking that quality remains acceptable.

Conceptually:

baseline
   ↓
benchmark
   ↓
locate the largest token sources
   ↓
propose smaller configurations
   ↓
evaluate
   ↓
keep only candidates that preserve quality

Future experiments could include prompt compression, semantic context pruning, adaptive retrieval, memory compression, tool-output compression, dynamic token budgets, cache-aware analysis, routing, latency and energy measurements, and multi-objective Pareto analysis.

The question I ultimately want the toolkit to answer is:

What is the cheapest token configuration that still achieves the success rate my application requires?

Try It

Suffice is open source under the Mozilla Public License 2.0.

git clone https://github.com/edujbarrios/suffice.git
cd suffice
python -m venv .venv

# macOS / Linux
source .venv/bin/activate

# Windows
# .venv\Scripts\activate

python -m pip install -e ".[dev]"
suffice validate examples/frontier.yaml
suffice frontier examples/frontier.yaml
suffice run examples/baseline.yaml

You can find the project here:

github.com/edujbarrios/suffice

If you are building AI agents and already have a representative task suite, I would be especially interested in experiments around system prompts, retrieval, history, tools, and memory. Those are exactly the places where agents can quietly spend far more context than they actually need.

The goal is simple: do the task, preserve the quality, use fewer tokens.