Google’s Harness2 Made a Cheaper AI Beat a Pricier One Without Retraining It. The Editor It Picked Is Claude

Google Harness2 is a free, open-source framework that lets AI agents improve their own instructions, tools, and memory while the AI model stays untouched. You’ll see how one round lifted a cheaper model by 11.5 points past a pricier one, why Claude does the editing, and the safety fine print.

Share this post X LinkedIn Email Facebook Copy linkTL;DR

Google researchers just open-sourced Harness2, a free tool that lets an AI agent rewrite its own instructions, tools, and memory after every task, with no retraining of the AI model underneath. That matters because companies are spending billions on bigger AI models, and Harness2 suggests a lot of the gains are sitting in the cheap stuff around them. Here’s how it works in plain English, what the results actually show, and the fine print you should read before anyone calls it “self-improving AI.”

An AI agent is an AI system that doesn’t just chat, but takes actions for you: searching files, writing code, filling in spreadsheets, or working through a task with several steps.

Every agent has two parts. The model is the AI brain, like Gemini or Claude, trained once at enormous cost. The harness is everything wrapped around it: the instructions it follows, the tools it can use, the notes it keeps, and the rules it obeys.

Think of a talented new chef. The chef is the model. The harness is the kitchen: the recipe binder, the knives, the sticky notes on the fridge, and the manager’s rules. Give the same chef a better kitchen and dinner improves, even though the chef didn’t go back to cooking school.

Harness2 treats the harness as eight editable files:

The model’s weights, the billions of numbers that make up its “brain,” never change. Harness2 has zero trained parameters. It’s pure software around the model.

Most self-improving AI setups need a verifier, an automatic answer key that says right or wrong. That works for math puzzles and coding tests. It doesn’t work for “draft this legal memo” or “clean up this budget,” where there’s no single correct answer.

Harness2 is built for that messy open world. For each new task, it runs a loop the researchers call Propose, Probe, Compose:

After each task, Harness2 saves the winning harness and a plain-text file of lessons called EXPERIENCE.md, so later tasks start with better advice. That two-level design, improving the current task and learning across tasks, is why it’s called recursive.

In our view, the clever part is the comparison step. Instead of asking “is this answer correct?”, it asks “which of these two attempts went better, and why?” That’s a much easier question for an AI to judge honestly.

The team tested Harness2 on three benchmarks built around real professional work. WorkBuddy-Bench covers office tasks, JobBench covers job-style assignments, and LAB is a legal agent test from Harvey, a legal AI startup.

OpenCode and Codex are two different agent programs that actually run the model and its tools. GLM-5 is an open model from Chinese AI firm Zhipu, while Gemini 3.5 Flash is Google’s fast, low-cost model.

It isn’t a clean sweep against rival methods, though. On the legal LAB test, a competing approach called Meta-Harness scored 76.5 to Harness2’s 75.6. The difference is that Meta-Harness needs a training set of example tasks first, while Harness2 starts cold.

Full details are in the Harness2 research paper, submitted October 9.

Here’s the detail we found most telling. Harness2 runs its tasks with Gemini or GLM-5, but the default editor, the model that proposes and judges the harness changes, is Claude Opus 4.8, made by Anthropic. Calls go through Google Cloud’s Vertex AI, which sells Claude alongside Gemini.

Google’s researchers didn’t explain the choice. Our read is simple: they picked the model they thought did the editing job best, even if it wasn’t their own. That’s consistent with how Google already treats AI models as interchangeable parts, a shift we saw when Google put Claude inside Gemini and started profiting either way.

The bigger lesson is about where the competition is moving. If a smarter harness can lift a cheaper model past a pricier one, the race isn’t only about who trains the biggest brain. It’s about who builds the best kitchen around it. Alibaba is betting on that same layer, with QwenGyre keeping AI agents working 4-hour shifts without wasting chips.

Harness2 is a research tool, not a product. A few things you should know:

None of this makes it bad. It’s free under the Apache-2.0 license, which lets anyone use and modify it, and it already has more than 5,000 stars on GitHub, a rough signal of developer interest. Developers on X have been sharing breakdowns of its eight components since the release.

If you don’t write code, you won’t install Harness2. But you’ll likely feel ideas like it.

The apps that will soon book your travel, handle customer service chats or review documents run on harnesses. Tools like this could make those agents noticeably better without companies paying for bigger, pricier models. That could mean cheaper AI features, or the same price for better results.

The risk side matters too. An agent that rewrites its own rules is harder to audit. If it learns a shortcut that works on a test but breaks a real policy, someone has to catch it. That’s why we’d watch how companies keep a human reviewing those EXPERIENCE.md lessons before they go live.

They can book appointments, compare products, manage email, and fill in forms. Some apps already do this on your phone, and Meta’s Muse AI agent hit 2.5 million downloads, though Amazon has already blocked it from its marketplace.

Yes, in limited ways. Some agents can pay for services using stablecoins, digital dollars on a blockchain. AI agents have made 138 million crypto payments, though most are tiny transactions between software services.

Agents that log in with your password can move money in ways banks can’t easily monitor, and they can steer cash out of low-interest accounts. That’s why JPMorgan shut the door on AI agents like Muse moving money at Chase.

Anthropic is known for strict safety testing, but no AI is risk-free. The company itself flagged serious long-term risks to investors, and the more interesting question is who holds the off switch at Anthropic as it grows.

Sometimes. Anthropic offers credits to early startups through partner programs, and the value can cover months of testing.

Vincee Cole is a technology journalist with four years of experience covering the full spectrum of modern tech — from consumer devices, artificial intelligence, to quantum computing, blockchain, and digital assets. His reporting cuts through complexity to deliver stories that are sharp, grounded, and relevant to both general readers and industry insiders. Previously, he worked with fintech research teams across Southeast Asia, analysing how emerging technologies are reshaping financial systems at scale.