When a user only says “Check how long it takes to get to the conference center” or “Buy me a waterproof phone case”, the ideal Personal Agent should be able to complete the starting point, transportation mode, device model and operating habits from past behavior records, instead of asking again every time.
But when it starts to perform tasks for users through the browser, the difficulty escalates from “whether it has accumulated user historical behavior data” to: can it find the historical interaction traces related to the current task from this data, restore the correct preferences, and convert them into appropriate web page operations?
The research team from Mila, McGill University and ServiceNow published a paper titled “[AdaptArena: Evaluating Test-Time Personalization of Web Agents]AdaptArena was proposed in “, which specifically evaluates this link. Experiments show that the highest task success rate for Web Agents that rely on user historical interaction trajectories for personalization is only 15.62%; when the target preferences required for the current task are directly provided, the highest success rate reaches 82.92%. This gap comes from three consecutive but independent links:
-
Historical interaction trajectories relevant to the current task may not be found;
-
Even if relevant trajectories are found, the Agent’s behavior may not be in line with user preferences;
-
The behavior is already consistent with the preference, but the web page task may still fail to execute.
Legend: AdaptArena evaluation process: Faced with tasks that lack information such as transportation methods, the Agent needs to recover implicit preferences such as cycling from the user’s historical trajectory, and then complete the route query.
How AdaptArena tests “Understanding preferences from user behavior”
AdaptArena is based on WebArena and builds tasks in three sandbox environments: e-commerce mall, Reddit forum and OpenStreetMap.
The research team defined 10 user roles and 12 types of preferences, generated 120 user historical interaction trajectories, and further constructed 480 deployment tasks, including 240 single-preference and dual-preference tasks each.
These preferences include not only static attributes such as geographical location, mobile phone model and brand, but also process preferences such as product sorting, product filtering, travel mode, and “wishlist or shopping cart”.
Deployment tasks remove explicit preferences. For example, a full request might be “Help me find a waterproof case that fits a certain phone,” while a review might just keep “Help me find a waterproof case that fits my phone.”
If the Agent does not know what phone the user is using, there are multiple reasonable answers to the task. The task becomes clear only by recovering device models from historical user behavior data.
The research team also invited two annotators unrelated to the project to view 60 original interaction trajectories that had been scrambled and had explicit preference information removed. On average, humans can recover target preferences in 85.83% of samples, indicating that there are indeed identifiable user preference signals in these trajectories.
The AdaptiveAgent framework proposed in the paper first uses CLIP to vector encode task instructions and historical interaction screenshots, and then retrieves two related trajectory prefixes through FAISS; then GPT-5-nano extracts implicit preferences from the trajectory and hands them to the Web Agent for execution.
The “trajectories” mentioned later in this article refer to this kind of web page interaction records arranged in chronological order, rather than general browser access records.
Legend: AdaptiveAgent workflow: Using the current task as a query, the relevant historical interaction trajectory prefixes are retrieved in two rounds. The preference extractor generates implicit preferences, which are then handed over to the Web Agent for execution together with the task instructions.
Under different information conditions, the success rate intervals of the six base models are as follows:
| set up | Information available to Agent | Success rate interval |
| No Profile | Only current task | 7.29%—9.17% |
| AdaptiveAgent | Retrieve and extract preferences from screenshots of historical interaction trajectories | 10.42%—15.62% |
| Reconstructed Profile | Reconstructing user portraits using semi-privileged information | 14.17%—17.71% |
| User-Centric | Complete target user portrait | 25.21%—68.33% |
| Oracle | Single target preference required for current mission | 40.83%—82.92% |
AdaptiveAgent exceeds the no-image baseline on all six base models, indicating that user historical interaction trajectories indeed contain valid signals.
But there’s still a huge gap between it and Oracle. Taking Gemini-3-Pro as an example, the task success rate of AdaptiveAgent is 15.62%, while that of Oracle reaches 82.92%, a difference of 67.30 percentage points.
The Oracle here will directly provide the target preferences required for the current task to the Agent, skipping trajectory retrieval and preference identification. Therefore, this result is more suitable as a performance reference upper limit under idealized conditions, rather than an absolute upper limit of the Web Agent’s capabilities.
The first layer of fracture: Relevant user behavior trajectories may not be found.
The previous end-to-end results can only tell us that the Web Agent finally failed, but it cannot explain at which step the information was lost. In order to locate the problem, the paper analyzes layer by layer along the link of “trajectory retrieval, behavior alignment, and task execution”. First, it checks whether the system can find the interaction trajectory related to the current task from the user’s historical behavior data.
AdaptiveAgent will return two candidate trajectories each time. If at least one of them corresponds to the preference category required by the current task, the paper will regard it as a retrieval hit and measure it with Recall@2, which is the “retrieval recall rate of the first two results”.
Among all 480 tasks, this retrieval hit rate was 71.88%; the single-preference task reached 77.50%, and the dual-preference task dropped to 66.25%. This shows that the system can find category-related interaction trajectories in most tasks, but as the number of preferences increases, retrieval becomes difficult.
Even more noteworthy, there are huge differences between preference categories:
| Preference category | Recall@2 | Behavioral alignment mean |
| geographical location | 100.00% | 75.00% |
| Mobile phone model | 100.00% | 70.00% |
| Product filtering | 75.00% | 3.30% |
| Product sorting | 35.00% | 1.70% |
| Wishlist vs shopping cart | 5.00% | 63.30% |
City names and phone models often appear directly on the page, making them easy to hit through visual similarity. Preferences such as sorting, filtering, and adding to wishlists are more reflected in the actions performed by the user rather than what is displayed on the final page.
This suggests a limitation of trajectory representation based on page screenshots:Page screenshots are good at saving “what’s on the web page”, but they may not be able to completely save “why users do what they do”。
In addition, 63.3% in the table is the average behavioral alignment rate of three representative base models on this preference category, and cannot be directly understood as a strict conditional probability. It is more suitable to explain: the main difficulty of wish list and shopping cart is biased towards trajectory retrieval, while product filtering can often find relevant trajectories, but the main loss occurs after retrieval.
The second layer of rupture: finding relevant trajectories does not mean that the behavior conforms to preferences
The paper uses the Preference Alignment Score to determine whether the complete interaction trajectory of the Web Agent conforms to the target preference. What this indicator measures is not whether the model can correctly retell the user portrait, but whether the navigation and operation behaviors ultimately displayed by the Agent are consistent with preferences.
The behavioral alignment rate of the six base models is only 22.50% to 29.79%, which is far lower than the trajectory retrieval recall rate of 71.88%. It shows that a large amount of preference information is not lost during retrieval, but is misused during trajectory interpretation, preference extraction or behavior transformation. A typical error is that the task needs to restore the user’s product filtering preferences, but the Agent recognizes it as product sorting, and then starts to adjust the price or name order instead of filtering products according to conditions.
Legend: The behavior alignment performance of three representative base models in different preference categories: explicit attributes such as geographical location and mobile phone model have high performance, and operational preferences such as product sorting and product filtering are close to zero.
Composite preferences further amplify the problem. After the six models entered the dual-preference task from a single preference task, the task success rate decreased by 10.83 to 15.42 percentage points. Combinations such as transportation mode and location can still maintain certain performance; once difficult process preferences such as product sorting and product filtering are included, the behavioral alignment rate of some combinations will drop to close to zero.
Legend: Success rates of six base models on single-preference and dual-preference tasks: After entering the dual-preference setting, all models experienced a drop of 10.83 to 15.42 percentage points.
This shows that the current Web Agent is not only difficult to recover action-level preferences from user historical behavior data, but also lacks the ability to stably combine multiple heterogeneous constraints.
Level 3 Break: Behaviors Already Aligned, Web Page Tasks Can Still Fail
Even if the Web Agent’s behavior trajectory has been determined to be consistent with the target preference, the final task success rate is still only 32.12% to 51.72%. Preference alignment and task completion are two different levels of capabilities.
Legend: When the behavioral trajectory has been determined to be consistent with the target preference, the conditional task success rate of each model is still only 32.12% to 51.72%, indicating that web page execution constitutes an independent bottleneck.
The paper divides the trajectories of “behavioral alignment but task failure” into several categories:
-
Action failed: The Agent’s target direction is correct, but the expected UI interaction was not successful;
-
Termination failed:Agent did not end the task at the appropriate time;
-
Navigation loop: The Agent repeatedly visits similar page status but makes no progress;
-
Other questions: Including abnormal page loading and other unstable environments.
The main types of failure are not the same in different models. For example, GPT-5-mini’s action failures accounted for 46.75%; Gemini-3-Pro’s termination failures accounted for 44.29%; Gemini-3-Flash’s navigation loop accounted for 30.88%. The denominator of these proportions is the sample whose behavior was aligned but ultimately failed, not the entire 480 tasks.
The paper also observed that some agents can realize that the previous page operation was unsuccessful during reasoning, but still cannot repair the operation. Self-diagnosis at the language level does not equal recovery ability at the executive level.
Personalization cannot be just a prompt
AdaptArena does not provide a solution that has been verified in a production environment, but it provides several clear directions for system design.
First, user historical behavior data cannot only retain page screenshots. The production system can simultaneously record action types, action targets, input values, and page state changes before and after operations, and use structured events to supplement process semantics that are difficult to express in screenshots.
Second, composite preferences can be broken down into checkable atomic constraints. For example, “specify category, price ascending order, add to wish list” are expressed as category、sort_order and terminal_actionrather than compressing it all into a vague natural language portrait.
Third, trajectory retrieval, behavioral alignment, and task execution need to be assessed separately. When the system fails, it should be able to determine whether the relevant track is not found, wrong preferences are restored, or there is a problem with web page action and termination control.
Fourth, the execution layer requires explicit status checking and recovery mechanisms, including action result verification, repeated status detection, completion condition judgment, and retry or path change after failure.
Note that these directions are engineering derivation based on the results of the paper, not the final architecture that AdaptArena has verified.
From the research boundary back to Personal Agent
AdaptArena evaluates personalized Web Agents, not all Personal Agents. Action failures, state loops, and termination controls may also appear in API Agents, Code Agents, or Desktop Agents, but the paper directly measures only the performance of these problems in browser tasks.
This set of benchmarks also has clear limitations:
-
120 user historical interaction trajectories are generated by the model under explicit preference conditions, which are more regular than the long-term accumulated behavioral data of real users;
-
Preferences are modeled as relatively stable discrete attributes that do not cover long-term changes and scenario drift;
-
The experiment runs in the WebArena local sandbox and does not involve the login, dynamic loading and anti-automation mechanisms of the real public network;
-
Behavioral alignment relies on LLM referees, and although verified for human consistency, there are still measurement errors;
-
The paper mainly reports point estimates, which are not enough to judge stability in a production environment.
Therefore, AdaptArena is more suitable to be understood as a systemic stress test rather than a Personal Agent solution that can be directly deployed. But it clearly reveals that when the Personal Agent starts to purchase goods, inquire about routes or handle business for the user through the browser, there is still a long engineering link between “accumulating the user’s historical behavior data” and “actually completing the task”.
The system must first find the interaction trajectory related to the current task from the user’s historical behavior data, and then restore the correct preferences from the trajectory; the preference must actually enter the navigation and operation behavior, and finally it must overcome execution obstacles such as web page interaction, status control, and task termination.
What determines the upper limit of the next generation personal assistant’s capabilities is not just how many user behavior records it accumulates. What’s more important is whether it can find the right interaction trajectory for the right task, convert user preferences into the right actions, and really get things done.
