This blog is a popular science article, which introduces Agent evaluation from the shallower to the deeper. The first two chapters systematically introduce what evaluation is and how to establish an evaluation system; the second chapter is the practical experience summed up by the Meituan Turing Agent evaluation team who went deep into the BP of each business team of Meituan. It is the knowledge we have gradually polished during two years of practice. Chapter 3 focuses on the changes brought about by the emergence of long-range Agent frameworks such as Lobster/Hermes on evaluation.
This blog received a lot of attention after it was published internally within Meituan. We found that everyone is very enthusiastic about Agent evaluation, so we decided to make the internal blog public. We want to share these experiences with more students, hoping to inspire or help everyone.
The core purpose of evaluation is to answer Is Agent good? And what is good and what is bad? This provides direction for the next iteration. Evaluation is a “precision measuring tool” for Agent effects.
This is why Agent evaluation cannot just stop at offline rankings, nor can it just look at the performance of a certain demo. It must serve the research and development, launch, regression, optimization and large-scale implementation in real business.
The cornerstone of Agent evaluation is observation, so we came up with the Agent development formula – Observation + Evaluation = Continuous Iteration
The evaluation method will change as the form of AI changes, and it has roughly gone through three stages:
Traditional machine learning is more like answering “is the calculation accurate?” Large model evaluation begins to answer the question “Is the model capable?” What the Agent evaluation really wants to answer is:
When a model is put into a real system and coupled with Prompt, Skill, tool chain, memory, state management, and business processes, can it deliver good results stably?
This means that the evaluation object of Agent is no longer a single model, but a complex system of “model + system + tool + process”.
In a real scenario, both agents may eventually “get it right”, but the engineering value is completely different:
- A path is clear, tool calls are stable, time-consuming is controllable, and reproducible
- A process of repeated trial and error, chaotic paths, random hits, and irreproducible results.
If you only look at the final answer, the two will be misjudged to be at the same level; but from the perspectives of scale, cost optimization, and user experience, the difference is very big.
Therefore, Agent evaluation needs to cover at least four levels of content:
- result layer: Whether the task is completed and whether the output is available
- process layer: Is the plan reasonable and the steps stable?
- efficiency layer: Are the time consumption, token, and number of tool calls acceptable?
- Risk layer (security layer)：Whether it exceeds authority, whether it is misoperated, whether there is any safety hazard
In this sense, since the explosion of GPT in 2023, Agent development has rapidly developed from ChatBot to multi-functional long-range agents such as ClaudCode and OpenClaw. Agent capabilities are becoming increasingly powerful, and Agent evaluation is moving from “answer evaluation” to “behavior evaluation”.
There is a simple understanding: Agent belongs to the generalized SaaS layer. Large models give Agent generalization capabilities but also bring about randomness problems, and users still expect a stable and reliable agent. In order to bridge the gap between “randomness” and “reliability”, we must return to the engineering perspective to look at Agent—— Invisible problems are almost impossible to solve stably。
An execution of Agent usually includes the following links:
As long as there is a problem with any of these layers, the final effect may be degraded. In order for the results to be stable, we need the process to be stable. But if the log system can only see “what the user said” and “what was last replied to”, it will be almost impossible to determine the root cause of the problem. It is based on “I wanted to read Case but found that there was no log.“To address this simple question, the industry has developed the Trace system, which discloses the entire logical reasoning process within the black box and records all input information that affects the model output.
Accurate observation of every “stealth action” of the Agent is the only way to achieve the leap from “probabilistic generation” to “industrial-level reliability”.
From the previous discussion, we know that the essence of Agent evaluation is to answer whether it is good or not, and to point out the direction for iteration. The Agent evaluation itself must focus on both the result (Response) and the process (Trace or Trajectory).
Some students may be curious, why must we “build bridges”? This is because the Agent evaluation system must pursue the interpretability between business value and evaluation indicators. One of the difficulties in Agent evaluation is the natural gap between model capability indicators and business result indicators.
These two types of indicators cannot be mapped directly, and there must be a layer of bridge indicators oriented to the task system in between. We provide a layered approach as follows:
Taking AI search as an example, the business may care about DAU, retention and clicks; the search system itself cares about the recall rate and click-through rate; the agent layer cares about whether the intent recognition is accurate, whether the retrieval is effective, and whether the result integration is trustworthy.
Only by stringing these layers together can we truly answer “why the business indicators have deteriorated” and “why the improvement in model capabilities has not brought about business benefits.” This matter must rely on people who really understand the business process to jointly establish the indicator system.
After the introduction in Chapter 1, we know that the core goal of Agent is to deliver good results stably. Agent evaluation in the industry draws heavily on the methodology of large model evaluation, and can generally be divided into objective evaluation and subjective evaluation:
So a more realistic approach is usually:
- Use objective evaluation to cover high-frequency, structured, and regularizable parts
- Use subjective evaluation to cover open, high-value, and complex business scenarios
- Use subjective evaluation to calibrate objective evaluation and AI evaluation, and then hand over the scaling part to the automated system
“Is it good or not” is a subjective question, and subjective standards need to be aligned. Otherwise, we cannot be sure whether after a certain iteration, it is the improvement caused by the jitter of the indicator or the real effect improvement.
In the practice of evaluation, the real difficulty is not that “no one can evaluate”, but that “different people evaluate differently, and machines and humans evaluate differently.” In the past year, when our Turing team went deep into the BP business side, we found that multiple teams had stepped into the same pit one after another.
The key understanding accumulated in the Turing evaluation can be summarized as “human alignment and human-machine alignment”, as follows:
- Everyone agrees: 1 “dictator” is better than 10 “democrats”. A strong role is needed to align the evaluation standards for products, operations, R&D, and QA, and follow the same evaluation system to avoid everyone working independently. Different reviewers aligned the standards through back-to-back annotation.
- Consistency between man and machine: The machine evaluation results are consistent with the manual evaluation results, otherwise it will not be believed. The significance of human-machine consistency lies in large-scale efficiency improvement to cope with greater business traffic.
How to perform alignment specifically? The best practice is to drill down the fuzzy indicators into more detailed evaluation rubrics (or evaluation dimensions: the academic circles have not yet unified the opinions on the evaluation dimension and evaluation rule rubrics. We are aligned with the open source project Arize AI here), and then make each rubric as binary as possible. The details are as follows:
- Indicator drill down: Digging down the “big and fuzzy” concept into multiple clear dimensions
- Rubric dualization: Try to converge the scoring rules into yes/no/unknown, 0/1/unknown
- Continuous iteration: Use the unknown proportion to check whether the rubric is reasonably defined, until the human-human consistency rate and human-computer consistency rate of a single rubric reach the credible threshold (for example, 85%, 90%)
The value of this dismantling method lies in: shifting from “subjective fuzzy feelings” to “judgable factual basis”, using drill-down to reduce ambiguity, thereby reducing differences between people and between people and machines. Using this method, the human-machine consistency rate of digital webmasters can reach 99%. Beam transformed the evaluation system with Turing’s binary solution, and the human-machine consistency rate increased from 62% to 92%.
Below we share two cases.
Case 1: How to evaluate the quality of junior high school students’ compositions? (Full score is 40 points)
Case 2: Taking the rider’s outbound call scenario model as an example of whether the reply is “colloquial”
Classic error demonstration – Please judge whether the large model’s answer is colloquial and rate it from 0 to 10.
Improved version after drill-down and binaryization:
- Does the model refer to you as the rider;
- Whether the model uses spoken communication vocabulary in unofficial documents such as “You’re welcome” and “See you tomorrow”;
- Whether the model output contains mood words such as “bar”, “he”, “that”, etc.
Supplement: What is the relationship between annotation and evaluation?
Among them, annotation is an action, and evaluation is a set of goal-oriented judgment processes. Machine pre-labeling can help improve manual efficiency, but only when the human-machine consistency rate is guaranteed can it be called automated evaluation, otherwise it is just machine labeling.
From the perspective of execution links, Agent evaluation can be broken down into five key links.
- collection: Collect original mission data online or in the sandbox
- Clean: Deduplication, classification, context filling, and dirty data repair
- Review: Manual evaluation, AI evaluation
- Quality inspection: Check whether the evaluation standards are stable and whether the evaluation results are credible
- Analysis/Attribution: Locate problem attribution, formulate optimization suggestions and regression tasks
These five links, together with online AB and continuous observation, constitute the data flywheel of Agent iteration.
Most new Agent evaluation teams have a misunderstanding – summarizing multiple surveys to design a complex and exquisite evaluation indicator system, and the more complex indicators are, the more difficult they are to execute and align.
Agent evaluation is a practical science.In the initial stage, the significance of “making the data flywheel run efficiently” is far greater than “designing a complex and exquisite evaluation system.” The establishment of the evaluation system is not accomplished overnight, but relies on good cases and bad cases.。
Therefore, the best practice path for building the Agent evaluation index system is as follows:
- Starting from high-frequency core scenarios, first define a few key indicators
- Collect Bad Cases from Production Environment
- Precipitate high-quality Good Case and clarify what “good” means
- Convert Good/Bad Case into standard evaluation sample
- Use evaluation results to feed back prompts, skills, strategies, model optimization, etc.
- Then continue to sample from new online performances to form the next round of iterations
Among them, the value of Bad Case is often higher because it is the easiest to expose capability boundaries and system shortcomings; while the role of Good Case is to help the team define a paradigm for high-quality completion.
The core competency of a mature evaluation team is not to build a perfect system from the beginning, but to continuously transform online problems, failed samples, and vague feedback into structured evaluation assets. We correct the goals of the evaluation system through Bad Case and Good Case, and gradually correct the negative deviation between the “dictator” and the real goal. For example, in the contract-fulfilling digital webmaster business, there were only more than 20 evaluation indicators when the project was launched. After one year of implementation, we expanded to nearly 200 indicators.
Since the release of GPT 3.5, although the base capabilities have undergone earth-shaking changes, the model capabilities are not omnipotent after all. In the past three years of Agent practice, we have encountered a large number of “wish-like” demands that are unrealistic. We need to know that the model is trained, and the model itself fits the probability distribution of Token. Even if capabilities “emerge” under large-scale parameters, the improvement of model capabilities still strongly depends on the input of corpus, especially the input of high-quality corpus. Whether it is the early RLHF or today’s DPO, GRPO and other algorithms, although the training architecture is constantly being simplified, the reliance on high-quality core data has never changed.
For example, Byte has specially set up Xpert, a crowdsourcing expert annotation platform, to produce high-quality data in professional fields such as geography, coding, law, medicine, etc., to support the iterative training of the Beanbao base model. With the improvement of the capabilities of the base model, everyone’s intuitive feeling is that its performance is getting better and better in one vertical field after another.
Therefore, when we face the challenge of a lack of business knowledge corpus in a specific vertical domain (or there is no public high-quality data on the public network), supplementing the capabilities of the model/Agent by introducing knowledge input from industry experts becomes the key to overcoming the problem – especially in the cold start stage of the project.
Echoing the previous article, the purpose of the evaluation is to answer “Is the Agent good or not?” So who will define “whether the Agent is good or not”? Rely on industry experts who know the business best and have the most sense.
This chapter describes the core methodology of the Turing evaluation that started from the contract fulfillment project and went through the process of various business parties of BP for more than a year.
FAQ
Q1: Does the “dictator” have to be one person, or can a team jointly follow a set of evaluation standards?
First, a team needs to follow the same evaluation system. The role of the dictator is to solicit opinions from multiple parties and integrate the evaluation system. When the project parties’ ideas cannot be aligned, the dictator will make the final decision, thus avoiding the losses caused by the fragmentation of the evaluation system and the fragmentation of the project parties and internal pull.
Q2: Is there any risk in the evaluation system relying on a “dictator”?
We need to recognize the fact that the evaluation system is constantly evolving. From business cold start to expansion to full scale, during this process users expand from AI enthusiasts willing to try new things to all users, and user profiles will shift significantly. This results in the evaluation goals needing to be constantly adjusted as the business expands. The value of the dictator is more reflected in the Lazi standard, and the evaluation goals are more modified and driven by BadCase/Good Case in real business scenarios. Of course, we need to select the people who know the business best to formulate the evaluation system before establishing the evaluation standards.
Q3: Is it necessary to set up a seed evaluation set during the cold start stage? Can it be launched directly with low traffic and directly collect Good Cases and Bad Cases to drive evaluation?
Whether it is necessary to set up a seed evaluation set during the cold start phase is essentially a trade-off of risks and costs.
Why it is recommended to set up a seed set: Large models are random, and Corner Case may cause the Agent experience to deviate drastically. Therefore, it is necessary to ensure the Agent’s baseline capabilities through backtesting, and continuously incorporate Good Cases and Bad Cases to broaden the Agent’s capability boundaries.
Regarding the strategy of low-traffic startup: If the fault tolerance rate of the business scenario is high, or the cost of building a high-quality seed set far exceeds the negative feedback caused by online trial and error, you can try to go online with low traffic to collect online cases.
Recommended implementation plan: After manually producing a small number of evaluation sets, AI assists in generation or expansion, and completes the cold start seed evaluation set construction at a lower cost.
Q4: What should we do if the industry experts we invited have different definitions of “good”?
answer: The most simple and effective method is to invite a group of industry experts to define “good” standards and extract common parts from them to build an evaluation system. For example, invite gold medal sales to define excellent sales SOPs.
So the non-common parts have no value? Not so. Where experts disagree often means there are multiple excellent strategies for the business itself. We can transform these differences into different styles or strategy branches of the Agent (for example: in AI telemarketing, sophisticated radicals vs. slender schools), and allow independent evaluation in different test sets (Benchmarks). These divergence points are not noise, but will become important nutrients for the Agent to move toward refined iterations and cover more long-tail scenarios in the future.
GPT explodes in popularity in 2023, workflow appears in 2024, Claude Code was released last year, and then this year’s lobster craze and Hermès craze, long-distance agents gradually entered the public eye. Agent Harness has also fully entered the long-range Agent era.
The difference between a long-horizon agent and a short-range agent lies in how it handles the “complexity brought about by time span”:
What changes will these differences bring to observation and evaluation?
The evaluation objects in the short-range Agent era are relatively simple. The typical input and output forms in the ChatAgent era are:Query -> Answer。
The common feature of this type of scenario is that the Agent is more “answering questions” and performing a small amount of system operations, rather than “entering the operating system to perform tasks.” Typical application scenarios, such as AI search and customer service robots. The focus of this type of Agent evaluation usually falls on the answer itself, for example:
In the past year of development, Turing has formed mature solutions, including manual evaluation and machine evaluation. Some cases are as follows:
3.2.1 Long-range Agent brings changes in evaluation paradigm
The long-range agent does not solve the problem of “answering a question”, but “completes a complex task”. It usually requires:
- The task requires multi-step disassembly
- Tool or Skill needs to be called multiple times during execution
- Need to read intermediate results and dynamically adjust the strategy
Let us review the goals of observation and evaluation, and look at the long-range agent with this goal in mind
- Observation target: Restore the scene, accurately locate it, and solve the problem of “I want to see the case but find that there is no log”
- Evaluation objectives: Answer whether Agent is good or not, and point out the direction of iteration
3.2.2 Skill evaluation
Before talking about Skill evaluation, we first share our research on the lobster/Skill boom after the 2026 Spring Festival.
Who is asking for reviews of Lobster and Skill (user portraits of the Lobster/Skill craze from February to now)?
To sum up, the current demand related to lobster and Skill mainly includes broad operation efficiency improvement scenarios, which can be roughly divided into three categories:
The user base is expanding from a small number of professional roles to a wider population
- The merchant side is still in the exploration and pilot stage, and the scale is limited.
- Lobster (or other long-range Agent) in the company’s internal AI at Work system is more widely integrated, and users cover multiple roles such as production, transportation, research and sales, etc.
- The threshold for skill development is getting lower and lower, and it can even be generated with AI assistance.
This leads to a direct conclusion:
In the future, those who need to be evaluated will not only be a small group of industry and operation research students, but may be everyone who can create, modify, and access Skills.
This puts new requirements on the evaluation system:
- Be simple enough
- Be standardized enough
- Be sufficiently automated
- It is best to have access to the development and release process
Current essential pain points
The essence is that – everyone does not know how to write Skill well, and there is a lack of tools to evaluate the entire life cycle of Skill.
In order to facilitate everyone’s understanding, we break down the entire life cycle of Skill as follows:
To sum up, the pain points of Skill evaluation can be broken down into three aspects:
Task-oriented evaluation
On January 9, 2026, Anthropic published a blog revealing AI Agent evaluation. In this blog, long-range Agent evaluation for Tasks was mentioned for the first time. Task is defined in this article as “a single test with clear input and success criteria”.
We have combined the definitions of Task from Anthropic and open source software and simplified it as follows.
The prompt defines our problem/demand, and the expected behavior defines the behavior we expect the Agent to achieve. When we send promt to the Agent in the formal or test environment, and obtain the real execution path of the long-range Agent through trace, we can get the (prompt – expeted_behavior – trace) triplet, which is similar to the short-range Agent’s (query – ground_truth – answer), and we can evaluate it.
3.2.3 Differences between long-range Agent evaluation and short-range Agent evaluation
The differences between the two can be summarized as follows:
The most essential changes are:
ChatAgent evaluation is concerned with “whether it is well said”, while long-range Agent evaluation is concerned with “whether things are done and how they are done”.
3.2.4 Human review-dominated to machine-review dominated
Common processes in the ChatAgent era are:核心评测员对齐 -> 外包对齐 -> 机评对齐
In the long-range Agent scenario, this link may be significantly shortened, and you can even skip outsourcing alignment and go directly to:核心评测员对齐 -> 机评对齐 -> 规模化扩展
There are three main reasons:
- The execution trajectories of long-range agents are richer, and the high information density makes people stuck.
- Skill has low production threshold, rapid growth in quantity, and large-scale manual labeling is unsustainable.
- Continuous breakthroughs in base model capabilities
This does not mean that labor is unimportant; it does mean that:
- Humans should do high-value standard design and Rubric alignment
- AI is more suitable for large-scale operations, preliminary screening and regression verification
- The platform should also assume the responsibilities of precipitation, replay, alarm and attribution.
In other words, what AI evaluation really wants to amplify is not the “machine scoring” itself, but the judgment standards of core evaluators.
3.2.5 What are the minimum capabilities that long-range Agent evaluation infrastructure should have?
If the company is to support a large-scale Agent and Skill ecosystem in the future, the evaluation infrastructure should at least include the following capabilities:
- Full link playback: Reproduce the entire process of a task from input to result
- Case management: Unify maintenance task samples, contexts, constraints and rubrics
- execution sandbox: Hierarchical isolation execution based on read-only, writable, high-risk and other types
- AI evaluation guideQing: supports rubric-driven human-machine alignment and automatic scoring, and is simple and easy to use, making it easy for the majority of skill producers to access it.
- Reporting and Attribution: Not only gives points, but also points out whether the problem occurs in planning, tools, environment or skills.
- regression mechanism: Automatically trigger historical Case regression after version upgrade
- Entry and exit restrictions: Embed evaluation results into development, release and operation processes
Without these capabilities, evaluation will easily remain at the level of “single analysis” and “project-based support” and cannot truly become part of the production system.
Taken together, with the enhancement of large model capabilities and the continuous evolution of Agent Harness, the evolution of Agent evaluation can be summarized in two sentences:
First, the evaluation objects have changed
In the past, the evaluation was “answers”, but now the evaluation is “task system”.
Therefore, what we care about is no longer just the output content itself, but the ability, stability, efficiency and risk in the complete execution link.
Second, the evaluation method has changed
In the past, the mainstream was Query -> Answer text quality evaluation, now gradually turning to Prompt -> Expected Behavior behavioral assessment.
The standard answer is no longer always unique, process quality, task completion and trajectory quality have become new core objects.
Therefore, what is really important in the future is not just whether we can do a few evaluations, but whether we can build a set of:See the problem、Tell the standard clearly、Capable of running at scale、Follow the process、Able to drive iterationAgent evaluation system.
Attached: Industry research in the open source field of long-range Agent evaluation
- Task(Task, also called Problem or Test Case): A single test with clear input and success criteria. Each attempt at a task is called a trial. Because model output can vary from run to run, we run multiple experiments to produce more consistent results.
- grader(Grader) is the logic for evaluating certain aspects of Agent performance. A task can have multiple scorers, each containing multiple assertions (sometimes called checks).
- Record(Transcript, also known as Trace or Trajectory): A complete record of an experiment, including output, tool calls, inferences, intermediate results, and any other interactions.
- result(Outcome): The final state of the environment at the end of the experiment. The Flight Booking Agent may say “Your flight is booked” at the end of the record, but the result is whether the booking record exists in the SQL database in the environment.
- Assessment framework(Evaluation Harness): Infrastructure for running evaluations end-to-end. It provides instructions and tools, runs tasks concurrently, logs all steps, scores output, and aggregates results.
- Agent framework(Agent Harness, or Scaffold): A system that enables a model to run as an agent: processing input, orchestrating tool calls, and returning results. When we evaluate “an agent”, we are evaluating the framework and model working together. For example, Claude Code is a flexible Agent framework, and we built our long-running Agent framework using its core primitives via the Agent SDK.
- Evaluation kit(Evaluation Suite): A collection of tasks designed to measure specific abilities or behaviors. Tasks in a suite often share a broad goal. For example, a customer support evaluation kit might test refunds, cancellations, and upgrades.
In February 2026, when lobster became popular around the world, open source software for lobster evaluation emerged in the community:
