Two trends have been occurring simultaneously over the past three years.
One isThe capabilities of base models continue to evolvethe result it brings is that the requirements for users are declining. In the early days, in order to stably output the model according to the format and correctly select a tool, it was necessary to repeatedly polish the prompts and design various backends. Nowadays, more and more of these capabilities are borne by the model itself. Long context, native tool calls, stronger instruction following and multi-step reasoning have filled in a large number of areas that originally required engineering work.
The second is The functions of the Agent framework continue to be enriched. From the early days when you had to implement planning, memory, tool registration, and status management yourself, now the framework and protocol layer abstract these into standard components, and add a tool and a skill, and the workload has been significantly reduced.
The two curves are superimposed, and the conclusion is straightforward: the threshold for building an Agent is decreasing year by year. The direct change brought about by this is that there are more and more Agent landing scenarios and they are becoming more and more divergent.
Since the release of ChatGPT (based on GPT-3.5) at the end of 2022, wave after wave of Agent projects have appeared on the market. However, there are only a handful of projects that have actually gone through cold start, completed gray scale expansion, and been pushed to full scale. This results in a shortage of people who master Agent evaluation methods.
Most of the Agent projects have died in the past three years. The reasons for their disappearance have some commonalities:
- Stop at Demo: The demonstration effect was very good, but as soon as real traffic was received, it was found that it could not be covered. User questions, data was dirty, and the scene boundaries were far beyond expectations. The team was tired of patching, and finally gave up.
- Stuck in expansion: It’s okay for a small-scale trial, but all kinds of problems are exposed when the volume is increased. The team couldn’t tell which layer the problem was, and they didn’t dare to make major changes, because they didn’t know if the original correct parts would be damaged after the changes were made, and the project swayed repeatedly in place.
- Can’t tell the business value: After several rounds of model changes, and dozens of Prompt iterations, the team felt it had gotten better, but could not provide evidence. When it comes time to ask for resources and continue to invest, we cannot answer “What exactly does the project bring?” and eventually stop investing.
These three ways of death look different. If you follow them further, you will find the commonalities:The team lacks a reliable judgment mechanism. I don’t know if the current version will work, I don’t know which layer the problem lies in, I don’t know if it’s really getting better after the change or if something went wrong in another place, and I don’t know how to prove the value of this matter to the outside world. Decision-making relies on feelings and individual cases, and feelings are unreliable in the face of complex systems.
If you look at the two things together, you will get a slightly fragmented picture:The threshold for using an agent is rapidly lowering, but the knowledge about how to use an agent well is still very scarce.. Moreover, the limited evaluation knowledge is also inadequate when faced with divergent scenarios. The experience of making a ChatBot cannot be seamlessly generalized to making a long-range Agent. To this end, we hope to share with you the evaluation knowledge that the team has honed in many business parties over the past two years in a systematic form, hoping to help you avoid some detours.
Not long ago, we published a blog “Agent Evaluation Talk”, which introduced the core methodology of evaluation and answered the question “What is evaluation and why is it done”. After the article was published, we received a lot of feedback, among which the most frequent category was: I have read the methodology, but when it comes to my own project, I still don’t know where to start.
To this end, we plan to write a series of blogs called “Agent Evaluation White Paper” to systematically explain to everyone the implementation guide for evaluation – what to do first and then what to do, what the output of each step looks like, what level of achievement is considered qualified, and how to deal with common pitfalls. The plan is divided into four parts:
This series of blogs is intended for those in charge of Agent products, R&D, algorithms, operations, and evaluation. If you are building an Agent from scratch, or the project in your hand is stuck at a certain stage and cannot be pushed forward, I hope these articles can provide something you can use directly.
It also needs to be emphasized here:Assessment is a practical sciencethe practices given in this series come from the practice of our evaluation team in multiple businesses of Meituan. They have been verified to be effective in their respective scenarios, but when migrating to new scenarios, they still need to be adjusted based on business characteristics. Rather than copying the steps, what deserves more attention is the problem to be solved behind each approach. Once you understand the problem, you can naturally adapt your approach to local conditions.
The purpose of the first “Evaluation Overview” is to let everyone systematically understand the composition of evaluation. We use a unified language to align concepts that are still divergent and entangled in the industry, and use a set of narrative logic to open up cold start, expansion and self-evolution. After having an overview, everyone can start with the end in mind to determine where they are now and which piece to work on next. A complete evaluation system can be summarized asFour modules, three capabilities, two loops, and a set of assets。
The core purpose of the evaluation is to answer two questions:Is Agent good? What’s good and what’s bad? The former gives judgment and the latter points out direction. A complete evaluation system must answer these two questions at the same time.
In the article “Agent Evaluation Talk”, we believe that evaluation is a set of “precision measuring tools”. This analogy actually contains two metaphors:
- Measuring tools must be accurate (only when the variance is small can you be confident): For the same sample, different people, different times, people and machines, the measured results must be consistent. This corresponds to the “everyone is consistent, man and machine are consistent” in the blog methodology.
- Measuring tools need to be calibrated regularly: After business expansion, user portraits change, Agent capability boundaries expand, and business goals change, the original evaluation set and Rubric will gradually lose their representativeness. This corresponds to the iteration capability of the evaluation system itself.
The second point is a key means to keep the evaluation close to real business scenarios and eliminate dictator bias, but it is the most easily ignored.
The ideal evaluation system is a continuously operating system. It consists of two Loops, which share the same batch of online real samples and interlock with each other through the hub of Case mining and attribution.
Article 1: Evaluation system iteration loop (evaluation evolution)
After Agent went online, online evaluation and online monitoring began to continuously generate abnormal signals. The online real samples behind these signals are mined into the Case pool, and then the nature of the problem is determined through attribution. If the attribution conclusion is “the evaluation set does not cover this type of problem”, then this sample should be added to the evaluation set; if the attribution conclusion is “the evaluation standard is wrong”, then the iterative evaluation is the Metrics and Rubrics themselves. This loop allows the evaluation set to be manually built from cold start and gradually approach the feedback of real users, gradually eliminating the dictator bias mentioned in the previous blog.
Article 2: Agent iteration Loop (Agent evolution)
For the same batch of online real samples, after case mining and attribution, the problem point of the Agent itself was located – it may be that the prompt is not clearly written enough, the description of a certain Skill leads to routing errors, context management loses key information, or the model capability is indeed insufficient. R&D makes changes based on this. After the changes are made, the evaluation set is used for backtesting to confirm that no degradation is introduced, and then AB is launched to observe the real effect. This Loop allows the Agent’s capabilities to continue to improve, while ensuring that each improvement does not come at the expense of existing capabilities.
In addition to the obvious bite point of Case mining and attribution, there is actually a second bite point – review assets (review sets). The output of the iterative loop of the evaluation system is a more complete evaluation set, and the higher the access control value of the backtest; the stricter the access control of the backtest, the fewer new problems will be exposed after the Agent goes online. This creates positive feedback.
Looking at the two loops together, a complete evaluation system must have three capabilities at the same time and accumulate evaluation assets (evaluation sets).
Evaluation assets (evaluation set)：
After these three capabilities are put into operation, reusable assets will be accumulated. They are the real long-term value of the evaluation system.
- Evaluation criteria(Metrics & Rubric): Make the team’s consensus on “good” explicit and quantifiable, eliminating the ambiguity of evaluation.
- Evaluation sample: Structured precipitation of real problems. The golden set defines the “lower limit of good”, the wrong question set records the “pits that have been stepped on”, and the challenge set demarcates the “boundary of ability”. Its value lies in its reversibility – every mistake once made should not be made again.
However, the consensus of “what is good” and the record of “what pitfalls have been stepped on” will continue to expand with the expansion of Agent. In this sense, the essence of evaluation system construction isTransform the team’s implicit understanding of business quality into explicit assets that are quantifiable, reusable, transferable, and automatically executable。
From the perspective of the evaluation system, four modules can be disassembled. The first three modules answer different questions respectively, and the fourth module is the foundation of the first three.
Notes on module division
The definition of offline evaluation in the industry and open source fields is very clear, but there are fuzzy areas among the modules of online evaluation, online monitoring, observation, and case mining and attribution. For example, Arize AI, LangFuse, and Braintrust have different definitions of these concepts. For example, some products combine “Online Evaluation” and “Monitoring” into one concept, and some classify Trace collection into Observability and Trace analysis into Evaluation.
Since there is no unified concept division in the industry, in order to facilitate concept alignment in subsequent discussions and use a set of frameworks to run through cold start, expansion and self-evolution, we disassembled the above four major modules. In the process of practice, readers can re-divide it according to the organization method of their own team, so as not to miss any responsibility.
An example throughout the text
In order to make the following discussion more realistic, we first agree on a scenario that runs throughout the entire article – Merchant Business Analysis Agent (corresponding to the internal Merchant Claw scenario).
The characteristics of this scenario are typical long-range agents: multiple steps, multiple Skill calls, intermediate products, frequent interactions with the environment, and no single correct answer. Every time we talk about a module later, we will return to this example to see what it looks like.
The core function of offline evaluation is the backtest after the Agent version is changed, which can be used as a gate control before going online. It relies on: a fixed evaluation set (fixed samples and fixed evaluation standards), a fixed execution environment that is as close to production as possible.
When any aspect of Agent Harness changes – model upgrade, prompt adjustment, knowledge base update, Skill added or offline – the effect can be verified through offline evaluation to avoid immature changes that lead to deterioration of Agent capabilities.
Why emphasize “fixed”
The essence of backtesting is to control variable comparison: only the Agent version is changed, and everything else remains unchanged, so that the difference in scores can be attributed to the version change.
The most vulnerable thing to be broken in practice is the execution environment. For example, during evaluation, the downstream interface of the test environment is called, but the data in the test environment is inconsistent with that in production; or the user’s history file is not preset in the sandbox, resulting in the Agent not being able to obtain the necessary context. Such differences in environment will systematically deviate from offline scores and online performance, ultimately causing teams to no longer trust offline evaluations.
A practical test method: take the same batch of samples and run them in the offline environment and online shadow traffic to see if the conclusions on both sides are consistent. If the difference is too large, it indicates that the fidelity of the environment is insufficient.
The generalized evaluation set consists of (Questions, reference answers, evaluation criteria) triples, whereQuestion-reference answerTogether they are called evaluation samples. This concept first came from the field of large model evaluation, and is now borrowed into the field of Agent evaluation.
However, as the Agent form evolves from ChatBot to long-range Agent, the specific meaning of these triples is also changing:
Behind this migration is a fundamental change: Agent evaluation has moved from “answer evaluation” to “behavior evaluation”. In the short-range scenario, the reference answer is a string that can be compared verbatim; in the long-range scenario, the reference answer becomes a description of the execution process. The evaluation depends on whether the path traveled by the agent meets expectations and whether the final environmental state achieves the goal.
Borrowing the definition of long-range Agent evaluation task from Anthropic’s “Demystifying evals for AI agents”: it is “a single test with clear input and success criteria.” This concept can also be generalized to the field of short-range Agents.
Borrowing this concept, each row of triples in the evaluation set can be regarded as a Task.
Specifically, the Task must have fixed inputs, such as Query/answer, or prompt/expected behavior; the evaluation part consists of Metrics and Rubric, where Rubric is the smallest granular evaluation standard.
A complete Task roughly looks like this:
The binary Rubric idea emphasized in “Agent Evaluation Talk” is used here: to drill down the fuzzy judgment of “Is this PPT good or not” into a set of specific check items that can answer yes/no, so as to quantify the fuzzy questions as much as possible and eliminate differences. The improvements in the human-to-human consistency rate and the human-machine consistency rate essentially come from this kind of drill-down.
The evaluation set can generally be divided into two categories:End-to-end evaluation set and process evaluation set. Each evaluation set is composed of the triples above.
For both end-to-end evaluation and process evaluation, we can establish what we often call the golden set and the must-pass set. These two concepts are independent of each other and do not conflict with each other.
Why are both categories needed?? The end-to-end evaluation set looks at the final results, focusing on answering “Has the matter been accomplished?“; Looking at the execution process in the process evaluation set, the answer is “Which step in the middle went wrong?“. Back to the example of merchant business analysis. The end-to-end evaluation set tells you that “68 out of 100 tasks delivered usable PPTs.” The end-to-end delivery rate is the number that the business side is most concerned about. But when this number drops from 68 to 55, the end-to-end evaluation set cannot tell you the reason. The process evaluation set can: If the “success rate of delivering data Skill call” drops from 95% to 70%, the problem has been located; if Skill The success rate has not changed, but the “data merging accuracy rate” has decreased. The problem lies in another place. If the process evaluation set indicators do not fluctuate, but the end-to-end delivery rate decreases, it means that an unknown anomaly has occurred, and we need to further dig and attribute it to iterate the evaluation set.
a practical experience: The end-to-end evaluation set answers “Should I sound the alarm?” and the process evaluation set answers “Who should I call if the alarm goes off?” With only end-to-end, you will fall into “knowing that something is broken but not knowing which part is broken”; with only process, you will fall into “every module meets the standards but users are just not satisfied”. The two complement each other.
Again, the end-to-end evaluation set is built from the user’s perspective. That is to say, as Agent functions gradually increase, different end-to-end evaluation sets need to be established according to the end-to-end functional modules. For example, Merchant Agent, store management is a core functional module, composed of several skills; store encyclopedia is another core functional module, composed of enumeration skills and store knowledge base. In this case, end-to-end evaluation sets should be established separately.
The process evaluation set can be further broken down. For example, questions related to the knowledge base can be separately deposited into a knowledge base evaluation set; questions related to specific skill calls can be deposited into a skill evaluation set. If new sub-scenarios develop later in the business, you can continue to dismantle them.
The evaluation set should be continuously refined with the development of the Agent’s functions and the emergence of problems.. Taking business operation analysis as an example, it can be broken down into a tree like this:
An additional benefit brought by dismantling isClear responsibilities. When the process evaluation set is aligned with the organizational division of labor, and the score at a certain level drops, the corresponding person in charge can be found directly without having to hold a meeting to determine “whose problem it is.”
Boundaries of dismantling: Don’t dismantle too finely too early
Layered disassembly is the result of evolution, not the starting point of design. The “Agent Evaluation Talk” blog clearly mentioned that evaluation is a practical science. Instead of designing an exquisite evaluation system, it is better to grasp the core scenarios and start running.
The correct sequence is: first build the end-to-end evaluation set and the process evaluation set of the core module and run them. If you encounter the situation of “knowing that something is broken but not knowing where it is broken”, then further enrich it. Let the teardown be driven by real problems, not completeness anxiety. This point will be expanded in detail in “Cold Start”.
When the Agent changes, the evaluation set acts as a gate control. Through backtesting, we can ensure the consistency of core links and core indicators before and after changes. However, there is still a distance between “running the evaluation set” and “establishing an effective backtesting mechanism”. Several key engineering considerations:
- Multiple trials (Trial) to obtain stable conclusions. The output of large models is naturally random, and the results of running the same Task twice may be different. In practice, multiple tests are usually run for each Task, which is what we often call Pass@k, and the pass rate (Pass Rate) is used as the conclusion instead of a single pass or failure.
- Hierarchical access control. Not all rubrics are equally important. The security and data accuracy check items should be “vetoed”, and any one of them will block the release; for the experience type and bonus items, thresholds can be set.
- Access to the R&D process is binding. What is really effective is to embed backtesting into the CI/CD or release process and make it an automated level.Access control without access process is essentially just a suggestion。
Inherent limitations of offline evaluation
We must clearly understand the boundaries of offline evaluation capabilities. It can only find known problems – scenarios that are not in the evaluation set and can never be detected by backtesting.
For example, in the scenario of merchant business analysis: if the evaluation set is full of “making PPT” tasks, then when users start to put forward a large number of new demands such as “help me compare the data of three stores”, the fixed offline evaluation set will obviously not be able to cover the new cases.
This is why a complete evaluation system must include an online part – offline is responsible for guarding the known, and online is responsible for discovering the unknown.
No matter how rich and diverse the offline evaluation is, it cannot exhaust all scenarios. Online evaluation can make up for this shortcoming by accepting real-world score feedback.
Online evaluation and online monitoring deal with quality observation during real online operation. The two focus on different levels, but there is an important intersection – inspection.
- Online reviewIncluding AB, shadow mode and inspection, used to observe the effect of Agent in a real online environment.
- Online monitoringThen focus on Skill/Tool success rate, failure rate, Token consumption, abnormal fluctuations and other operating indicators.
An intuitive distinction: Online review answers “Does the system work well?“, online monitoring answer”Is the system working properly?“. The former is from a quality perspective, and the latter is from an operation and maintenance perspective. The Skill call success rate is 100%, but wrong data is returned every time. The monitoring is green and the evaluation is red.
The complementary relationship between the three: The shadow mode uses real traffic before going online, AB confirms the business value after going online, and inspections continue to keep an eye on the basic market on a daily basis.
The inspection office is at the intersection of online evaluation and online monitoring, and its core role is to continuously monitor the stability of online Agent effects. Inspection is a common means in business to turn “online quality observation” into a “continuously traceable process”. The most common approach is to extract a portion of representative Tasks from the offline evaluation set to periodically check the performance of the online Agent. As the evaluation becomes more refined, a separate inspection evaluation set can also be maintained. Inspection has two attributes at the same time:
- Review properties: See whether a certain type of capability is sustained and stable, such as whether the success rate of key tasks is lower than expected. This is making a quality judgment.
- Monitor properties: Precipitate score changes, failed samples and abnormal trends as input for subsequent case mining and attribution. This is doing signal output.
Going back to the example of merchant business analysis, a reasonable set of inspections can be designed like this:
Online monitoring focuses on operational indicators, and a large number of stability indicators common in projects are within the scope of online monitoring. Common monitoring dimensions include:
- usability class:Skill/Tool call success rate, failure rate, timeout rate, exception type distribution. This is the most basic and most effective layer.
- Cost category: Token consumption, average number of steps per task, and average time consumption.
- Behavior monitoring: Task type distribution, Skill call frequency distribution, average dialogue rounds. The value of this layer lies in discovering changes in user structure. When the proportion of a certain type of new tasks increases rapidly, it often means that the evaluation set needs to be expanded.
Stability indicators and behavior monitoring indicators must be looked at together to draw meaningful conclusions. For example, if the task completion rate drops by 5 points and the failure rate of a certain Skill increases from 2% to 15%, then the conclusion is that it is most likely an engineering problem; if the Skills are all normal, but the proportion of “data comparison” tasks increases from 10% to 35%, then the conclusion is that the user structure has changed and the Agent is insufficient in new scenarios – the response to these two situations is completely different. This is the scope of Case mining and attribution.
Case mining and attribution mainly accept problem clues exposed in online evaluation and online monitoring. It also receives online feedback, real sample sampling and samples screened by business rules.
Division of responsibilities:Case mining is responsible for collecting samples into the Case pool, and attribution is responsible for continuing to locate and determine the core problem points in the Agent process.。
This module is the hub of the entire system and the core entry point of the current hot “Agent self-evolution”.
The four types of sources need to be used together because they have different “blind areas”. Relying solely on online feedback, you can only see those problems that users are willing to complain about; relying solely on monitoring, you can only see problems that can be characterized by indicators; relying solely on rule mining, you can only see the risks you have already thought of.Random sampling is the only method that can discover “problems you didn’t expect at all”. Although it is the most expensive and has the lowest hit rate, it cannot be omitted.。
After the excavated samples enter the Case pool, they can be further divided into Good Case and Bad Case, which are deposited into different types of assets:
Wrong question sets are the type of asset with the most direct returns. Its logic is simple: every mistake you have ever made should not be made a second time.
The value of the golden set is easily underestimated. Bad Case tells you what’s wrong, but it can’t tell you “to what extent it’s OK.” Without the golden set, the team can only talk about their own opinions when discussing quality; with the golden set, “we must achieve this level” becomes a specific sample that can be pointed at.
The goal of attribution is to locate the core problem points in the Agent process. A Bad Case can only explain “something went wrong here” but not “why something went wrong”.
Attribution must continue to determine which link the problem occurs: a common approach is to spread out the entire Agent execution process and locate problem points one by one through Trace logs. This process is highly similar to looking at logs to find bugs in the software problem troubleshooting process. Because of this, attribution relies heavily on the observation infrastructure Trace. This process can be done manually or with machine assistance.
An actionable attribution layer looks something like this:
After the attribution is clear, positive cases can generally be classified into the golden set and challenge set in the evaluation set; negative cases will enter two different paths:
- Enter Agent effect iteration – repair, backtest, and go online. This is the Agent iteration loop.
- Verify the Rubric. If the existing Rubric does not accurately describe the risk, you need to iterate the Rubric and flow the Case into the wrong question set. This is the iteration loop of the evaluation system.
Many teams only take the first path – defaulting to “the evaluation says it doesn’t work, it means the Agent can’t work”. This default assumption creates a hidden problem:Agent will be optimized to “meet evaluation standards” rather than “solve user problems”. When the evaluation standards themselves are biased, the harder you try to optimize, the further you deviate from the real needs. Regularly checking “whether the evaluation judgment is wrong” is a necessary action for self-calibration of the evaluation system.
There is a simple statement in “Agent Review Talk”:Invisible problems are almost impossible to solve stably. The core value of observation is to white-box the Agent link. This is also the Agent R&D formula”Observation + Evaluation = Continuous Iteration”, the reason why observation is given top priority.
An execution of Agent is a long link: user input → intention understanding → task planning → tool/skill invocation → environment interaction → intermediate result processing → policy adjustment → final output. As long as there is a problem with any of these layers, the final effect may be degraded. But if the system can only see “what the user said” and “what was last replied to”, it will be almost impossible to determine the root cause of the problem. This is the reason why the industry developed the Trace system – to disclose the entire reasoning process inside the black box and record all input information that affects the model output. The significance of observation for evaluation has been fully discussed in the previous blog and will not be repeated here.
From Trace SDK to Trace Plugin integrated in OpenClaw and Claude Code. The observation infrastructure is also rapidly iterated with Agent Harness.
Based on the previous discussion, a set of observation infrastructure that supports a complete evaluation system should at least have the following capabilities:
Because Agent Harness iterates too fast, even in the open source community, the related infrastructure lags behind the demand. Observation is the foundation, but this does not mean that a complete system must be established at the beginning of the project. In the actual implementation process, observation indicators and observation infrastructure also need to be reverse designed based on needs. The judgment standard is very simple: when you find that “you want to automatically judge something, but it is not in the data”, it is time to fill in the gaps.
A pragmatic construction idea is:First ensure “minimum attribution”——At least you can get the complete model call input and output and Skill call records, and you can use Trace to reproduce what happened in a Bad Case. This is the bottom line of the cold start stage, and it is also what “Cold Start” will focus on.
In practice, we have observed the three most common “partially complete” forms of each business party’s evaluation system. Each of them can solve part of the problem, but there are clear capability gaps.
The third form deserves the most vigilance because it seems to be the most “complete” – it has both a review set and a monitoring board, and the reporting data is very complete. However, due to the lack of an intermediate hub, the two systems are separated: problems found online cannot be returned to the evaluation set, and the conclusions of the evaluation set cannot explain online phenomena. In this form, a lot of investment is often made but the results are limited. The team can easily fall into the confusion of “We have done everything we should do, why is this still the case?”
Finally, the previous discussion is condensed into a self-check sheet that can be used directly. It is recommended to compare the scores and find the board where your business is the shortest.
How to use this table？
Don’t pursue reaching L3 in all dimensions. Different business stages correspond to different reasonable water levels: in the cold start stage, observation L1 + evaluation set L1 + backtest access control L1 is enough to support the minimum closed loop; in the expansion stage, the focus is on pushing the evaluation standards and online evaluation to L2, and a small number of indicators to L3; only in the full operation stage, the automation and scale of L3 need to be considered.
A more valuable use is to look at “shortcomings” and “mismatches”. If your evaluation standards have reached L3 (large-scale machine evaluation), but case mining is still at L0 (relying on user complaints), then no matter how sophisticated the machine evaluation is, it is just repeating the evaluation of known samples – investment should be diverted to supplement the mining capabilities. The upper limit of the system’s capabilities is determined by the shortest board.
Back to the question at the beginning of this article: What does a complete evaluation system look like? can be summarized asFour modules, three capabilities, two loops, and a set of assets：
- four modules: Offline evaluation is responsible for guarding the known, online evaluation and monitoring are responsible for discovering the unknown, case mining and attribution are responsible for positioning and allocation, and observation infrastructure is the foundation of everything.
- three abilities: Discover problems, locate problems, and drive evolution. If one item is missing, the system will get stuck in the corresponding link.
- Two loops: The Agent’s iterative loop enables continuous improvement of capabilities, and the evaluation system’s iterative loop allows continuous calibration of measuring tools. The two share entrances at Case mining and attribution, and share assets at the evaluation set.
- a set of assets: An end-to-end and process evaluation set covering each functional module of the Agent. It is the gatekeeper of the Agent’s effect, the real long-term value of the evaluation system, the anchor point for the stability of the Agent’s effect, and the foundation for the future self-evolution of the Agent.
It needs to be emphasized that this panorama is the “end” rather than the “starting point”. No team should build it from start to finish according to this diagram – it is unrealistic and not cost-effective in terms of investment and output.
There is a sentence in “Agent Review” that is worth repeating here:Agent evaluation is a practical science. In the initial stage, the significance of “making the data flywheel run efficiently” is far greater than “designing a complex and exquisite evaluation system”. The value of a panoramic picture is that it allows you to know clearly at every stage: where am I now, which piece of work will benefit the most next, and what is the cost of not doing a certain piece for the time being.
So where to start? In the cold start stage where there is nothing and no evaluation set, how to achieve the minimum closed loop of the evaluation flywheel? This is exactly the question that the next article “Cold Start” will answer.
Q: Is the Agent offline evaluation set only used for backtesting? Why can’t it be released according to the same iteration as the large model, and the score must be higher than the last time?
This question involves the core difference between Agent evaluation and large model evaluation, and needs to be answered from two aspects.
1. Why doesn’t Agent rank like a base model? The base model provides general capabilities and can be disassembled into a series of general capabilities. Let me remind everyone that large model evaluation itself is also an extremely professional field in the industry. The evaluation index system for the capabilities of large model bases that everyone sees now was compiled from numerous papers in the past three years. The meaning of the base model benchmark itself has also undergone great changes in the past three years. The Agent serves the vertical business field, and the best Agent indicator is the business indicator. Therefore, for the Agent, going online AB to see changes in business indicators is the gold standard for evaluating its capabilities.
2. The evaluation set can be divided into a must-pass set and a challenge set. Obviously, in the vertical business field, the construction of challenge sets is extremely difficult, and it is very easy to fail to keep up with the iteration of Agent version functions. The must-pass sample belongs to the core function of the Agent, and known problems must be solved as much as possible before they can go online. Therefore, in daily work, it often happens that the required questions (must-pass sets) are all mastered, but the difficult questions (challenge sets) are not seen.
To sum up, Agent offline evaluation is mostly used for backtesting. The real effect can be directly measured through the online AB to see the business indicators.
Supplement: If a challenge set is set up, the distribution of the real scene on the fitting line must be updated in time, otherwise the evaluation set will not be trusted.
