For the same development task and the same model, different Coding Agents will produce completely different results because of their different execution strategies. Some agents can quickly locate the problem and complete modifications; some agents will repeatedly search for files, try various repair methods, and spend a large amount of tokens to finally complete the same task.
Therefore, how to make the Agent perform tasks more efficiently has become a concern of Harness optimization research. Last week, researchers from Tsinghua University, Zhipu and other institutions published a paper titled “[HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs]HarnessPro proposed in “HarnessPro” explores exactly this issue.
By analyzing the execution trajectory of Harness, continuously iterating and optimizing Harness, and combining the generated tests with Pareto optimization to screen candidate solutions, Harness Pro attempts to find a more appropriate balance between task success rate and execution cost.
Cost issues in Harness self-optimization
RHO (Retrospective Harness Optimization, retrospective Harness optimization, an Agent Harness self-optimization method proposed by Microsoft Research Asia) has actually tried to use historical execution trajectories to optimize Harness, but the improvement in task success rate is also accompanied by higher operating costs. For example, on SWE-Bench Pro, RHO increased the success rate of GPT-5.5 + Codex from 59.0% to 80.0%, and the average single task cost also increased from US$1.88 to US$2.68.
This also reflects a problem faced by Harness optimization: adding exploration, reasoning and verification steps can help improve the task success rate, but it may also produce a large number of redundant operations; if the reasoning intensity or the number of tool calls is excessively restricted, the Agent may not be able to complete the task due to insufficient information.
In response to this problem, HarnessPro requires that the verification success rate of the candidate Harness is no less than that of the initial version, and retains candidate solutions with different performance and cost combinations to adapt to different execution budgets.
HarnessPro’s recursive optimization process
HarnessPro’s optimization process is divided into three stages: execution trajectory diagnosis, optimization plan and modification, and candidate evaluation and screening.
The system will first let the initial Harness perform a set of representative tasks, analyze the failure reasons and redundant overhead, and generate different modification plans accordingly. After evaluation and screening, Harness with good performance will enter the next round of optimization.
The entire process does not change the model weight, but mainly adjusts components such as instructions, tools, skills, hooks and verification rules in Harness.
Figure 1: Overall architecture of HarnessPro
Locate problems from execution traces
HarnessPro will first screen training tasks through DPP (Determinantal Point Process), taking into account task difficulty and diversity, while controlling the cost of collection and analysis of execution trajectories. In the paper’s experiments, 10 training tasks were selected for each benchmark test, and each Harness candidate independently executed each task three times.
The system will record the complete execution process, including model response, tool invocation, environment feedback and final product, and statistics of task results, token consumption and execution delay. Based on this information, the Trajectory Analyzer (Trajectory Analyzer of the architecture diagram) analyzes problems that affect the task success rate or increase the execution cost.
Trajectory Analyzer compares the results of multiple executions of the same task, identifies recurring problems, and distinguishes occasional environmental anomalies. For example, if the Agent fails at the same step multiple times, it may mean that Harness lacks necessary execution guidance; if the task has been completed, but the Agent is still repeatedly reading irrelevant files or running the same test, there may be redundant operations.
The analysis results will be organized into structured diagnostic records, including problem description, impact on performance and cost, and corresponding execution evidence. Each piece of evidence is associated with a specific trajectory file and event location to facilitate subsequent inspection.
These diagnostic records will be provided to the subsequent optimization planner (Evolution Planner of the architecture diagram). In order to reduce context overhead, the system will give priority to providing aggregated diagnostic results. When it is necessary to further understand the problem, the system will check the original trajectory based on the path in the record. If the diagnostic content exceeds 64K Tokens, HarnessPro will summarize the content to avoid analysis results taking up too much context.
Generate optimization plans from diagnostic results
The optimization planner receives the complete implementation of the current Harness, the diagnostic results of the trajectory analyzer, and the execution indicators and code differences of each candidate Harness. By comparing the modification content of different candidates with actual performance, the optimization planner can determine which adjustments may be effective and determine the direction of the next round of optimization.
For example, one candidate reduces costs by compressing the tool output, but the task success rate decreases; another candidate retains the full log, although the cost is higher, but more tasks can be completed. The optimization planner can combine the code differences and execution results of the two to analyze the effects of different strategies and find room for further improvement.
For each Harness candidate, the optimization planner will generate three plans, focusing on reducing costs, balancing performance and cost, and improving mission success rate.. Taking too many exploration steps as an example, a cost-prioritized solution may tighten the tool call budget, while a performance-prioritized solution may retain more exploration space and reduce repeated checks.
Each plan needs to clearly define what changes should be made, how they will be implemented, and what the expected effects will be. The researchers also required that the optimization strategy be somewhat general and avoid adding hard-coded rules for individual tasks in the training set.
The optimization planner can view the execution trace of the training task, but cannot access the information of the validation set and test set. Among them, the verification set is used for candidate screening, and the test set is only used after the optimization is completed to prevent Harness from directional optimization for the evaluation task.
Implement the modification plan to Harness
The Harness Editor (Harness Editor of the architecture diagram) modifies the Harness files and running components according to the optimization plan, including prompt words, configuration files, Skills, tool scripts, and Runtime Hooks.
For example, if the optimization planner finds that the Agent repeatedly executes the same command during environment check, the Harness editor can add a unified environment detection tool; if the tool output is too long, it can also adjust the processing method of the returned results to reduce unnecessary context consumption.
Each optimization plan will be implemented in an independent workspace, and modification records will be saved through Git submission. This not only allows multiple candidate Harnesses to evolve in parallel, but also facilitates the optimization planner to compare the code differences of different solutions in the next round.
For each parent Harness, the optimization planner will propose three optimization directions, and the Harness editor will generate three new candidates accordingly. Assuming that three different parents are currently retained, nine new candidates are generated. Including the original three parents, a total of twelve Harnesses participated in the evaluation.
The original parent will also be retained in the candidate set to avoid losing the previously better-performing configuration when the new solution is not effective.
HarnessPro How to modify Harness
The appendix of the paper provides real execution diagnostics and optimization plans. From these records, you can see more specifically how HarnessPro analyzes execution behavior and which components the modifications are ultimately implemented to.
Mailman: redundant overhead in successful tasks
In the Mailman task of Terminal-Bench 2, the Agent needs to configure Mailman and Postfix to implement the mailing list joining, exiting and mail delivery functions. The same Harness was executed independently three times, and all passed the automatically generated tests, but the trajectory analyzer still found obvious redundant overhead.
First, after confirming the main configuration scope, the Agent continues to read a large number of configuration files and source codes of Mailman and Postfix. The total amount of input tokens in the three executions reached approximately 1.06 million, 1.76 million, and 2.52 million respectively, and some of the source code inspections did not provide significant help to the final task results.
Secondly, after the Agent completes the email delivery test, it runs the verification program multiple times that has been covered by the test. These operations involve creating Unix users, processing mail queues, and waiting for retries, which increases Token consumption and execution time.
The trace analyzer also found some sporadic anomalies. For example, in one execution, mail was bounced due to mailbox file permission issues, so the Agent additionally checked the logs and retested; another time, because the service startup command did not end normally, a process status check step was added.
The trace analyzer records these problems separately and associates them with specific execution traces. Although all three executions completed the task, the redundant exploration, repeated verification and additional environment processing overhead still provide a basis for Harness optimization.
Evolution Planner: A real optimization plan
The appendix of the paper also shows a Medium optimization plan in Terminal-Bench 2, explaining how the optimization planner implements the improvement plan into specific Harness components. This plan and the previous Mailman diagnostic record are two separate examples.
Task execution instructions: Modify README.md to require the Agent to clarify its goals and delivery results before starting the task, prioritize checking relevant files and environmental information, and conduct verification around the actual modified content to reduce unnecessary repeated checks.
Inference budget routing: Add the running configuration in runtime.json and select the inference intensity according to the task description through router_prompt.txt. Simple tasks can use a lower budget, while complex code modification and debugging tasks are allocated more inference resources.
Tool output controls: Added guarded-run tool to set timeout for command execution and save complete log to file. The tool only returns execution status, time consumption, log path and partial output to the Agent, reducing the occupation of context by long logs.
Environmental inspection: A new quick-probe tool is added to obtain the working directory, Git status and availability of common development tools at one time, reducing repeated environment check operations.
The optimization plan also takes into account potential risks. For example, inference budget routing may misjudge task difficulty, resulting in insufficient resource allocation; compression tool output may also miss important error information, so it is necessary to retain the method of reading the complete log.
These modifications affect task execution, inference resource allocation, tool output, and environment inspection respectively. HarnessPro can adjust the operation mode of the Agent by combining different strategies, and then judge the effect of the modification through subsequent evaluation.
Figure 2: Real diagnosis and optimization plan
Automatic test generation and candidate selection
After completing the Harness modification, HarnessPro still needs to evaluate the actual effects of each candidate solution. However, in the absence of official tests and standard answers, how to accurately judge whether the Agent has completed the task has become a difficult problem for candidate evaluation.
Test Synthesizer: Generate canned tests for tasks
Previous RHO mainly relied on LLM to determine the execution trajectory, but researchers found that this method may overestimate the task success rate. For example, the Agent reports that the task is over after completing the code modification, and there are no obvious errors during execution. However, the actual function may still have defects, but LLM judges it as a success.
In order to solve this problem, HarnessPro introduces a test generator (Test Synthesizer) to generate tests based on the task description and the initial Harness execution trace to check the final product of the task. These tests are generated once before optimization begins and remain unchanged in subsequent iterations, ensuring that different candidate harnesses use the same evaluation criteria.
HarnessPro also partitions the training set, validation set, and test set. The training set is used to collect execution traces and provide a basis for Harness modification; the verification set is used to evaluate and screen candidate solutions, and the results will not be provided to the optimization planner; the test set is only used after the optimization is completed, and the final effect is evaluated through official tests.
During each round of evaluation, each candidate Harness performs verification tasks respectively, and then uses fixed generation test check results to calculate the task success rate and execution cost.
Figure 3: Comparison of LLM judgment and generated tests
The researchers also compared the two evaluation methods on some training tasks of Terminal-Bench 2 and SWE-Bench Pro. The results show that the recall rate of failure recognition directly judged by LLM is 41.7%, which increases to 83.3% after using generated testing.
This suggests that, in this set of experiments, the generated tests were able to identify more tasks that actually failed. Although insufficient test coverage may still exist, fixed test conditions can improve the consistency of evaluation and reduce reliance on the subjective judgment of LLM.
Pareto filtering: retaining candidates with different performance and cost
After the candidate evaluation is completed, HarnessPro will obtain the verification success rate and average execution cost of each Harness. The system first eliminates candidates with a success rate lower than the initial Harness to avoid sacrificing task completion capabilities in order to reduce costs.
Next, the system further filters through the Pareto dominance relationship. For example, candidate A has a success rate of 80% and costs $1.50; candidate B has a success rate of 78% and costs $1.80. Since A is better than B on both indicators, B will be eliminated.
If candidate C has a success rate of 83% and a cost of $2, then candidate A has a lower cost and C has a higher success rate. Both have their own advantages and both can remain on the Pareto front.
Considering that the Pareto front may contain more candidates, HarnessPro will further screen. The system first normalizes the success rate and cost, then calculates the weighted square distance between each candidate and the ideal success rate and lowest cost, and selects the solution that best meets different optimization goals.
Figure 4: Recursive updating and screening of Harness, showing the updating, screening and formation process of three configurations of candidates in multiple rounds of iterations
Specifically, Lite prioritizes cost, Heavy focuses more on success rate, and Medium considers both indicators. The three modes use different weights, and each selects the candidate with the smallest weighted distance as the starting point for the next round of optimization.
Since the three modes are selected independently and the same Harness may be selected, the final number of different candidates retained in each round may be less than three. This approach not only preserves different performance and cost configurations, but also controls the number of candidates that need to be evaluated in subsequent iterations.
What mechanisms are used by the evolved Harness?
The researchers analyzed the Harness finally generated by HarnessPro and found that the system mainly reduces operating overhead by dynamically allocating inference resources, controlling tool calls, and compressing contexts.
One of the optimizations is to dynamically adjust the inference intensity and tool call budget based on the task description. For simple tasks, Harness can reduce reasoning and exploration overhead; for complex tasks, more reasoning and verification resources are reserved. Different operating modes also have their own task judgment rules.
Part of Harness also controls the execution process through Runtime Hook. For example, in the Medium mode of Terminal-Bench 2, when the system selects the standard execution configuration, Hook will take different measures based on the number of tool calls:
-
Reached 14 times: Agents are reminded to avoid duplicate checks and to consider invoking tools in parallel.
-
Reached 26 times: Remind the Agent to focus on completing the task and final verification.
-
Reached 44 times: Prevent the Agent from continuing to call tools.
These thresholds result from the evolution of a specific Harness and are not applicable to all tasks. In this way, the Agent can conduct necessary exploration in the early stage of the task, gradually converge when it approaches the budget upper limit, and concentrate on completing the remaining work. Relevant reminders will be appended to the results returned by the tool. Try to keep the prompt word prefix stable to facilitate reuse of the prompt cache.
In terms of context management, part of Harness refers to RTK’s approach to compress tool output. For example, delete duplicate content in logs, shorten overly long output, and streamline recurring instructions in Developer Instructions to reduce Token consumption.
But compression and restriction also need to be balanced. If the Agent stops exploring prematurely, it may miss the information needed to solve the problem; if the tool output is compressed too much, important error information may also be deleted. The comparative experiment of the paper found that after uniformly reducing the inference intensity of RHO, although the execution cost dropped significantly, the task success rates dropped to 62.7% and 59.0% on Terminal-Bench 2 and SWE-Bench Pro respectively.
This also shows that using a certain cost optimization strategy alone may affect the task completion effect. HarnessPro therefore needs to combine the candidate evaluation results to find a more suitable strategy combination.
Experimental results and analysis
The researchers evaluated HarnessPro in three benchmark tests: Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, which respectively involved terminal operations, code warehouse problem repair, and program reconstruction tasks.
The experiment uses two combinations of GPT-5.5 + Codex and GLM-5.2 + Claude Code, and compares them with the initial Harness, RHO and other methods.
Taking GPT-5.5 + Codex as an example, the results of HarnessPro Heavy and RHO are as follows:
Judging from the results, Heavy mode achieved a higher success rate in all three benchmarks while reducing the cost of single task execution.
For scenarios where cost is more of a concern, the Medium model also achieves good results. For example, on SWE-Bench Pro, the success rate reaches 79.00%, which is close to RHO’s 80.00%, while the average single task cost drops from US$2.68 to US$1.26.
Multiple rounds of evolution and component effects
The paper also records the performance of HarnessPro in different iteration rounds. On Terminal-Bench 2, the success rate of Heavy mode increased from 74.6% in the first round to 80.8% in the third round, and the average single task cost dropped from US$0.91 to US$0.72.
However, success rates and costs do not improve simultaneously every round. For example, on SWE-Bench Pro, from the first round to the second round in Medium mode, the success rate increased from 70.0% to 77.0%, and the cost also increased from US$1.57 to US$1.66. By the third round, the success rate further increased to 79.0%, and the cost dropped to $1.26.
This shows that HarnessPro may experience staged performance and cost trade-offs during the iteration process, and improve the performance of candidate solutions through subsequent optimization.
Figure 5: Performance and cost changes over multiple iterations
Ablation experiments further examined the role of each component. On SWE-Bench Pro, after removing the optimization planner, the success rate of Medium mode dropped from 79.0% to 73.0%, and the average single task cost increased from US$1.26 to US$1.42. After removing the test generator, the task success rates in different modes also decreased, indicating that both components have a certain impact on the optimization results.
It should be noted that the execution cost in the paper is mainly calculated based on the model Token usage and corresponding price, and has not been fully included in the calculation and operation overhead incurred by HarnessPro itself in multiple rounds of optimization.
Conclusion
HarnessPro allows the Agent to continuously optimize the operation strategy based on historical execution feedback through execution trajectory diagnosis, automatic iteration of Harness, test generation and multi-objective screening.
Judging from the actual evolution results, strategies such as dynamic budgeting, tool call control, and output compression have a certain basis in engineering practice. HarnessPro further explores how to automatically discover, combine and evaluate these strategies and select a Harness configuration that better balances performance and cost.
Currently, relevant experiments are mainly focused on the Coding Agent scenario. The reliability of the generated tests, the ability to generalize across tasks, and the computational cost of the recursive optimization itself remain to be further verified.
