A more sophisticated AI agent is not necessarily a more profitable one. That is the conclusion, documented by two studies published in October 2026, that should concern any small or mid-sized business deploying, or planning to deploy, AI agents to automate customer support, sales qualification, or software development. Researchers compared several "harnesses" (the program that drives a language model) and several memory systems on the same tasks, with the same model. The result: the bill can vary by up to five times for nearly identical accuracy.
In short
- A study from Shandong University, reported by VentureBeat on October 23, 2026, shows that the cost of an AI agent can vary by up to 3x depending on the harness used (Claude Code, OpenCode, mini-SWE-agent), for near-identical accuracy on the SWE-bench Verified benchmark.
- An earlier study from the UC Berkeley Sky Computing Lab, cited in that same article, had measured a cost gap of up to 5x between harnesses, across seven different models, without a comparable gain in success rate.
- A second study, "MemoryTax," led by Shanghai Jiao Tong University and reported by VentureBeat on October 25, 2026, shows that the most powerful models (such as Claude Opus 5.5) often don't need costly structured memory: they achieve the same accuracy with raw history.
- On the same model (Claude Opus 5.5), answering 152 questions costs $6.95 with a minimalist harness and $28.41 with Claude Code, for accuracy of 99% and 97% respectively.
- For an SMB, the lesson is not to avoid AI agents, but to measure cost per task alongside accuracy, before choosing a tool or a vendor.
What is an AI agent "harness"?
An AI agent harness is the program that wraps around a language model: it gives it tools (read a file, run a command, search the web), manages conversation memory, and decides how much information to send to the model at each step. Claude Code, Codex, Cursor, or a simple in-house script are all harnesses. Two harnesses using the same model can produce very different results in cost, simply because they structure context differently.
That is exactly the point two separate studies tested in 2026, with converging conclusions.
A more sophisticated harness does not guarantee a better score
The Shandong University study (published on arXiv, reported by VentureBeat) ran three harnesses, Claude Code, OpenCode, and mini-SWE-agent (a minimalist harness of about a hundred lines of code), on 492 tasks from the SWE-bench Verified benchmark, with five different models, including Claude Opus 5, DeepSeek-V4-Flash, and two Qwen models.
The finding is clear: on the larger task pool, Claude Code and mini-SWE-agent scored within 2 percentage points of each other. But the cost per task varied by up to 3x depending on the harness chosen, for the same model. With the GLM-5.3-Flash model, for example, a Claude Code trial costs almost twice as much as mini-SWE-agent, and more than three times as much as OpenCode.
The researchers also point to an earlier study from the UC Berkeley Sky Computing Lab, nicknamed "HarnessTax," which tested seven models across three harnesses (Claude Code, Codex CLI, Pi): success rates varied by only a few points, while inference costs climbed up to 5x.
Sophisticated harness (e.g. Claude Code)
Large starting context, over 16,000 tokens on the first call alone, more tools, more steps. Higher cost per task, with no guaranteed accuracy gain.
Minimalist harness (e.g. mini-SWE-agent, Pi)
Smaller starting context, under 1,000 tokens in the Shandong study, fewer tools. Statistically comparable accuracy, notably lower cost per task.
Agent memory also carries a hidden cost
The second study, called MemoryTax and led by Shanghai Jiao Tong University, looked at another architecture choice: should you build structured memory (summaries, a knowledge graph) for an agent that needs to recall a long conversation, or let it access the raw history directly?
The researchers tested seven memory approaches (full context, keyword search, consolidation tools like Mem0 or Zep, agent-directed search) combined with three models: GPT-4o-mini, GPT-6 Luna, and Claude Opus 5.5. The result: structured memory mainly helps the weakest model, GPT-4o-mini. With Claude Opus 5.5, the strongest model tested, raw context and structured memory score the same, 95%: building a sophisticated memory layer then adds no accuracy gain, only an extra build cost.
The detail most useful to an SMB concerns the Pi harness, a minimalist tool that lets the model search the raw history itself instead of reloading everything for every question:
Cost to answer 152 questions with the Claude Opus 5.5 model, per the MemoryTax study (Shanghai Jiao Tong University, reported by VentureBeat on October 25, 2026). The percentage shown is the share of correct answers.
In this example, the cheapest harness is also the most accurate. The gap doesn't come from the model, identical in all four cases, but from how each harness structures and reuses information.
Key takeaway
The price of an AI agent doesn't depend only on the model chosen. The harness and memory management often weigh more on the bill than the model choice itself. Always ask a vendor for cost per task, not just the name of the model used.
What an SMB should check before choosing or building an AI agent
Both studies converge on the same method, applicable to any automation project: customer support, sales qualification, document processing.
Start simple
Add retrieval if needed
Build memory only if it will be reused
Compare the whole stack, not just the model
These four checks take a few hours. They prevent paying, month after month, for an engineering layer that delivers no measurable benefit, a risk made more real by the fact that many vendors bill their AI agents by tokens consumed: the chattier the harness, the higher the bill, regardless of result quality.
FAQ
Is a more expensive AI agent harness always more accurate?
No. According to the MemoryTax study (Shanghai Jiao Tong University, 2026), the most expensive harness tested, Claude Code at $28.41 for 152 questions, scored lower than the cheapest harness, Pi at $6.95, which reached 99% correct answers.
Should you always build structured memory for an AI agent?
No. The study shows structured memory, like Mem0 or a knowledge graph, mainly helps less powerful models. With a top-tier model like Claude Opus 5.5, raw context achieves identical accuracy, without the cost of building memory.
How can an SMB compare two AI agents before choosing one?
By asking for cost per task and accuracy on a real sample of its own records, not a vendor demo. The Shandong University study shows cost can vary by up to 5x between two harnesses using the same model, without a comparable gain in success rate.
Do these studies apply only to software development agents?
Both studies tested coding tasks (SWE-bench) and long conversations (customer support). The principle itself is generic: any AI agent handling documents, tickets, or long conversations faces the same trade-off between harness cost, memory cost, and accuracy.
Conclusion
The complexity of an AI agent shows up in the bill before it shows up in the results. For an SMB, the right move in 2026 is not to pick the trendiest tool or the most powerful model, but to demand a measured number, cost per task and accuracy, before integrating an AI agent into a business process. That's also the question at the heart of choosing between building your own agent or buying a turnkey solution: see our build vs. buy guide for AI agents in SMBs and our customer success stories to go further.
