Agent Harness Design Guide for Product Managers: Transform user goals into executable tasks, tool contracts, permission policies, structured results, and acceptable engineering systems. When product managers design Agent products, they not only write prompts, pages, and final output, but also participate in defining how the Agent performs tasks, calls tools, controls permissions, handles failures, and delivers results.
Author: vivo Internet Product Team-Meng Disheng
Agent Harness Design Guide for Product Managers: Transform user goals into executable tasks, tool contracts, permission policies, structured results, and acceptable engineering systems. When product managers design Agent products, they not only write prompts, pages, and final output, but also participate in defining how the Agent performs tasks, calls tools, controls permissions, handles failures, and delivers results.
1 minute to look at the pictures and grasp the key points👇
1. Why does Agent product need technical modeling?
Traditional product requirements can often be described as “input-processing-output”: the user submits information, the system completes the processing, and then returns the results. However, Agent products are usually not a synchronous call, but a work process that may last for a long time. It may need to read multiple data sources, call multiple tools, wait for user confirmation, handle external system failures, and generate reusable results in the middle.
Therefore, Agent product requirements cannot only describe “what the user inputs and what the model answers”. At a minimum, it should explain how the task is split, whether there are hard dependencies between steps, whether callable tools will modify external systems, where user confirmation is required, and whether the task should be stopped, degraded, or resumed in the event of a timeout, denial, or service exception. The final delivery cannot be just a reply, but must be clearly structured data, a downloadable product, or a resource that has been written into the business system.
If an Agent requirement can only describe inputs and outputs, it has usually not entered the engineering design stage. Product managers design not only interactive pages, but also a set of task models that can be executed, observed, and accepted.
What this article says Agent Harnessis not the name of a specific framework, but the control layer that undertakes task definition, step scheduling, tool invocation, permission execution, result verification and operation observation. Business workflow determines “what is to be accomplished”, tool implementation determines “how to access external capabilities”, and Harness organizes the two into a constrained and traceable operation. This article focuses on the part of the system contract that product managers need to participate in defining, without discussing the specific implementation of a certain model or orchestration framework.
2. From user goals to Agent Workflow
Product managers first need to convert user goals into executable workflows instead of directly writing goals as a Prompt.
For example, “generate improvement suggestions based on user feedback and submit them to the target working system after confirmation” can be broken down into:
Read feedback data
-> 清洗和去重
-> 按问题类型聚类
-> 统计影响范围
-> 查询历史记录
-> 生成候选建议
-> 检查建议冲突
-> 输出评估报告
-> 用户确认并提Each step requires clear project attributes:
Product managers do not necessarily need to implement clustering algorithms, but they need to determine whether the workflow is complete: which steps must be serial, which steps can be parallel, which results can be consumed by subsequent steps, and which links require product manager intervention.
Define execution semantics for steps
Workflow cannot only have sequence, but also needs to define scheduling semantics. The steps can usually be divided into four categories:
For example, problem clustering and history matching can be done in parallel, but whether recommendation generation must wait for both to complete depends on whether history is a hard dependency or an enhanced information in this business. The former should block subsequent generation; the latter can record “missing historical matches” and continue with degradation. Both behaviors must be determined by the task definition and cannot be left to the executor to guess at runtime.
It is recommended that this semantics be written into the Workflow DSL or JSON configuration, rather than scattered in Prompt. The following configuration treats history matching as a hard dependency, so failure at either step blocks suggestion generation:
"id": "generate\_recommendation",
"dependsOn": \["cluster\_feedback", "analyze\_impact", "match\_history"\],
"mode": "serial",
"input": {
"clusters": "$steps.cluster\_feedback.output",
"impact": "$steps.analyze\_impact.output",
"history": "$steps.match\_history.output"
},
"onFailure": "block",
"timeoutSeconds": 90
}Here dependsOn defines the dependency graph, input specifies data binding, and onFailure defines failure propagation. The example also lists the impact scope statistics as a pre-dependency to prevent the recommendation generation from continuing to execute in the absence of impact information. After these fields are clearly written in product requirements, R&D can stably implement scheduling, retry, and visualize progress.
3. Task Model: Don’t just design one round of dialogue
Agent products need to be clear about task objects, rather than just being designed around sessions. It is recommended to organize user goals, inputs, execution processes and results in a task model:
Task
├── taskId
├── userGoal
├── inputRefs
├── workflowVersion
├── owner
└── runRefs
Run
├── runId
├── taskId
├── currentStep
├── status
├── outputs
├── approvals
└── errorsProduct managers need to distinguish between the following objects:
A task can contain multiple dialogue rounds, multiple runs, and multiple artifacts. Dialogue is the interaction carrier, tasks are business goals, runs are execution instances, and products are deliverables that can be used continuously. If these objects are mixed into a chat thread, it will be difficult to support retry, comparison, recovery and result attribution in the future. Task.owner here represents the business ownership or responsibility subject, and is not equivalent to the user who initiated the session; Run.approvals saves the reference to the approval record actually used in this execution, rather than copying the approval content into each message.
Task objects require version and reference semantics
The task object must support at least two capabilities in engineering:Input is reproducible and structure is evolvable. The former requires the task to save a stable reference to the input, rather than just saving the long text in the prompt word; the latter requires the object to allow fields to gradually increase while keeping historical tasks readable.
{
"taskId": "task\_01J...",
"schemaVersion": 2,
"goal": "基于用户反馈生成需求建议",
"inputRefs": \[
{"type": "feedback\_sheet", "id": "file\_123", "version": "v7"}
\],
"workflowVersion": "feedback-analysis@1.3.0",
"runs": \["run\_a1", "run\_a2"\],
"artifactRefs": \["artifact\_report\_01"\]
}schemaVersion solves the evolution of object structure, workflowVersion solves the behavioral differences of the same task under different orchestration versions, and inputRefs.version prevents historical conclusions from being unable to be explained after the data source is updated. Product managers need to ask at least two questions: “Do historical tasks must be reproduced?” and “How to display old tasks after the rules are updated?”; they will directly affect the storage model and audit costs.
4. Tool design: Product managers must define tool contracts
Whether an agent is reliable depends largely on whether the tool is clearly designed. The product manager does not necessarily write the tool implementation, but needs to participate in defining the tool contract:
Tool:
name
description
inputSchema
outputSchema
permission
sideEffect
timeout
retryPolicyTake the “Submit Suggestions to Target Work System” tool as an example:
submitRecommendation:
input:
title
description
priority
targetScope
idempotencyKey
output:
recordId
url
status
permission:
write\_target\_system
sideEffect:
create\_external\_recordThe product manager needs to confirm:
- When the tool can be called;
- Whether the parameters are complete enough;
- Whether the returned results are sufficient to support the next step;
- Whether the tool modifies external systems;
- Whether approval is required;
- Whether it is possible to retry after the call fails;
- Whether repeated calls are allowed;
- Whether idempotent keys are required.
Note here that the tool is not the “list of interfaces called to the model”, the tool is the business action protocol in the Agent workflow. If the input, output, permissions and side effects of a tool are not clearly agreed, the higher the degree of freedom of the model, the higher the uncertainty of the system.
4.1 Input verification, output verification and error codes must be layered
Tool calls require at least three levels of verification:
- Verification before calling:Use JSON Schema, field enumeration and permission policies to reject illegal parameters, such as priorities that are not within the allowed range and projects that do not fall within the current authorization scope.
- Verification after execution:Perform Schema verification on tool return values to prevent the model from mistaking error responses for successful results and continue reasoning.
- Business result verification:Verify business invariants, for example, a queryable recordId must be obtained after successful submission
It is recommended that the tool return stable results and not spell exceptions into natural language:
{
"ok": false,
"requestId": "req\_8f2c",
"error": {
"code": "DUPLICATE\_RECORD",
"retryable": false,
"message": "同一业务动作已提交记录",
"details": {"recordId": "record\_1024"}
}
}Among them, retryable determines whether the executor will automatically retry, requestId is used for log association, and details allows subsequent steps to choose to jump to existing records instead of resubmitting. The DUPLICATE_RECORD here is suitable for scenarios where “duplication is not allowed in the business”; if the caller carries the same idempotent key, a more reliable approach on the server side is to return the resource that succeeded for the first time and indicate idempotentReplay: true in the response. When defining tool output, product managers should be required to distinguish between “not executed”, “execution failed”, “executed but the result does not meet the business conditions” and “idempotent replay returns existing results”.
4.2 Tool description is a constraint interface
The tool description seen by the model affects the calling decision, but it is not a substitute for a machine-verifiable Schema. It is recommended to separate human-readable description, machine-readable Schema and policy constraints:
Tool description -> 告诉模型何时使用、何时禁止使用
JSON Schema -> 约束参数类型、枚举、必填项
Policy -> 约束主体、资源范围、风险等级
Executor -> 负责超时、重试、幂等与审计This can avoid writing hard constraints such as “modification of other items is prohibited” and “the maximum length of the title is 80 words” only in the prompt. Prompts can be misunderstood, and Schema and Policy must be able to be enforced before execution.
5. Permissions and Side Effects: Design Approval Using Risk Levels
Different tools have different impacts on the system, and the permission policy cannot only adopt two states: “allow” or “forbidden”. Tools can be graded according to side effects and risks:
Strategies can be further expressed as structured configurations:
Policy:
tool: submitRecommendation
riskLevel: L3
requireApproval: true
allowedScopes: \[...\]
allowedFields: \[...\]
idempotencyRequired: trueProduct managers need to transform “need to confirm” into executable rules: which tools must be confirmed, which parameter changes will trigger re-confirmation, what is the valid scope of confirmation, how to handle rejection, whether pre-authorization is allowed, and how the approval process is recorded.
Authorization should be bound to subjects, resources and actions
Approval is not just a boolean value. Valid authorization should be bound to at least three types of context:
- Subject: Who is authorizing, such as the current user or proxy service identity
- Resource: What can be accessed or modified, such as scopeId, documentId, tenantId
- Action: What is allowed, such as read, create, update, publish
For example, “Allow submission of suggestions” does not mean “Allow writing of records to all target spaces.” A more executable authorization record is:
{
"subject": "user\_12345",
"action": "recommendation.submit",
"resource": {"scopeId": "workspace\_1001"},
"constraints": {"priorityIn": \["P1", "P2"\], "expiresAt": "2026-08-22T12:00:00Z"},
"approvalId": "approval\_01"
}When a tool parameter crosses the resource scope, a constrained field is modified, or an approval expires, the executor must re-request authorization. This prevents a confirmation from being mistakenly extended to an unlimited range of passes.
6. Result Agreement: From Text Answers to Structured Delivery
The Agent’s results should not be just a piece of Markdown text. Different tasks require defining different result protocols, for example:
Result:
summary
evidence
decisions
risks
pendingQuestions
artifactRefs
nextActionsUser feedback analysis can output:
{
"summary": "...",
"clusters": \[\],
"priorityRecommendations": \[\],
"matchedRequirements": \[\],
"evidenceRefs": \[\],
"uncertainties": \[\],
"nextActions": \[\]
}Product managers need to distinguish between four categories of results because they correspond to different ways of storing, displaying, and subsequently consuming them:
The structured results can be displayed by the front end and consumed by subsequent agents; the products can be exported as reports; external system records need to return stable IDs and access addresses. Important conclusions should also be accompanied by evidence citations and uncertainty statements to avoid packaging speculation into facts.
6.1 Establish a traceable relationship between evidence and conclusions
The evidenceRefs in the result protocol should not just be a list of URLs, but should be able to express “which conclusion is based on which sources.” It is recommended to use locateable references:
{
"recommendationId": "rec\_01",
"statement": "登录失败提示需要覆盖网络异常场景",
"confidence": 0.82,
"evidence": \[
{"sourceId": "feedback\_328", "locator": "row:328", "role": "user\_feedback"},
{"sourceId": "metric\_login", "locator": "2026-08-01..2026-08-15", "role": "metric"}
\],
"assumptions": \["反馈样本已完成去重"\]
}Confidence cannot replace evidence, but it can help the interface sort and prompt; assumptions make the premises on which model reasoning relies explicit. For results that need to be entered into external systems, it is recommended to add resultVersion, generatedAt, and generatorVersion to facilitate subsequent comparison of analysis differences between different versions.
6.2 Add deterministic verification before releasing results
LLM output is non-deterministic, and the structured output may have complete fields but incorrect business. Therefore, deterministic verifiers should be added before publishing, such as:
- 每条建议至少绑定一条 evidence
- priority 必须属于 P0/P1/P2/P3
- sourceRef 必须属于本次 Task 的可访问输入
- externalRecordId 只能在写工具成功后出现
- 风险等级为高时必须有 nextActionThese rules are better suited to be executed by normal code or a rules engine, rather than being left to the model again. The model is responsible for generating candidate conclusions, and the validator is responsible for guarding the protocol boundaries.
7. Failure handling: retry, recovery and idempotent design
The failure of the Agent task cannot just display “Execution failed, please try again”. At least a distinction needs to be made between:
InputError
PermissionError
ToolError
TimeoutError
ModelError
ValidationError
ExternalSystemErrorDifferent error types should use different strategies:
Steps with side effects require special attention. If the interface submission of records to the target system has been successful, but the response is lost during network transmission, a simple retry may cause duplicate writes. Therefore, write operations should use task ID, step ID, or business idempotent key to identify the same business action:
The same taskId + stepId should not create the same business data repeatedly
Product managers need to clarify during the requirements stage: which failures can be automatically retried, which failures must be notified to the user, which steps support continuing from the intermediate position, and how to deal with the products that have been generated in the event of failure.
7.1 The retry strategy must distinguish between reading, writing and model generation
“Retry up to three times” is not a general policy. Different categories of operations require different execution rules:
Exponential backoff can be expressed as delay = base * 2^attempt + jitter. The function of jitter is to avoid secondary congestion caused by retrying a large number of tasks at the same time. For L3/L4 write operations, you must first check whether it has been dropped before retrying; this is a more reliable judgment than “whether the request has timed out”.
7.2 Idempotent keys should express business actions
A usable idempotent key usually consists of a business primary key:
idempotencyKey = hash(taskId + stepId + targetProjectId + normalizedPayload)Among them, normalizedPayload should exclude fields such as timestamp and display copy that do not affect business semantics. When the server receives the same key, it should return the resource created for the first time instead of executing it again. In this way, even if the executor is restarted, network response is lost, or the user clicks repeatedly, the same business action can be collapsed into one result.
8. Observability and Acceptance: Monitoring cannot replace verification
After Agent goes online, two misjudgments are most likely to occur: first, if the task is “finished”, it is regarded as a success; second, if the average time consumption decreases, it is regarded as an improved experience. The former covers up problems such as unfounded conclusions and repeated write operations, while the latter may come at the expense of result quality. For Agent, the value of observation data is not to pile up a big market, but to help the team locate whether a failure occurred in task design, model decision-making, tool interface, or user confirmation.
Take “Submit improvement suggestions based on feedback” as an example. If the completion rate suddenly drops, first check whether the task stops at the input verification or authorization stage; if the completion rate does not change and manual rework increases significantly, then look back to see if references are missing, whether clustering rules have drifted, and whether the workflow version has changed. Tool timeout rates are diagnostic only if they are put together with specific steps, data sources, and retries. Measuring “tool success rate” alone usually doesn’t solve the problem.
In practice, the observation field can be compressed into an associated execution record:
traceId, taskId, runId, stepId, toolName,
workflowVersion, promptVersion, inputRef, outputRef,
latencyMs, retryCount, errorCode, approvalIdtaskId serves business retrieval, traceId is used for cross-service tracking, and workflowVersion and promptVersion are used to explain “why the same type of input started to produce different results this week.” If a requirement suggestion is returned by the user, the team should be able to restore along this record which input version it used, which steps it went through, what tools were called, where the retry occurred, and where the final basis came from.
Acceptance should not stop at “seeing whether a few answers are pleasing to the eye.” A more practical approach is to retain a small number of high-value samples: empty feedback, unavailable history services, repeated submissions, target space overreach, user rejection of approval, etc. Each sample clearly states the boundaries that cannot be touched, rather than listing a long list of general indicators. The following use case simultaneously constrains tool selection, result structure, and side effect caps:
case: feedback-duplicate-submit
inputRef: fixture://feedback/duplicate.csv
expect:
requiredTools: \[readFeedback, clusterFeedback, submitRecommendation\]
forbiddenTools: \[publishAnnouncement\]
resultSchema: AnalysisResult@v1
assertions:
- path: $.recommendations\[\*\].evidenceRefs.length
operator: greaterThan
value: 0
- path: $.artifactRefs\[?(@.type == 'target\_system\_record')\].externalId
operator: exists
sideEffects:
maxSubmittedRecords: 1This type of use case is closer to contract testing: it does not promise which sentence the model must write, but it clearly requires that evidence cannot be bypassed, irrelevant tools cannot be called, and two requirements cannot be created due to retries. Putting it into CI or grayscale regression can better detect engineering regressions than letting test students repeatedly experience it manually.
9. Case: Technical Design of Feedback Analysis Agent
9.1 Business objectives
Based on user feedback, improvement suggestions are automatically generated and submitted to the target work system after user confirmation.
9.2 Task flow
First, user feedback is imported and format checking is completed, then data cleaning, problem clustering and impact scope analysis are performed, and historical records are matched and candidate solutions are generated. After result verification and user confirmation, the writing tool for submitting suggestions is called.
The first half of the process only reads and analyzes data, while the second half moves from “generating recommendations” to “changing business systems.” This boundary determines that the submission action must be placed after user confirmation and executed using traceable idempotent keys.
9.3 Tool list
9.4 How permissions, results and verification fall on a link
These six tools are not at the same risk level. readFeedback and queryHistoricalRecords will read business data and should be executed as L1 within the authorization scope of the data source; clusterFeedback and validateRecommendation only process authorized input and belong to L0; generate-
Recommendation generates internal analysis materials and can be managed according to L2 and retains the generated version; submitRecommendation will write records to the target working system and must be executed according to L3, binding the target space, user confirmation and idempotent keys.
The final output AnalysisResult should not be just a summary. It must contain at least clusters, affectedUsers, priority-
Suggestions、matchedRecords、recommen-
dates, risks and evidenceRefs. Each of the recommendations should carry evidence references, making it easier to ask “which types of feedback does this P1 recommendation correspond to?” The resources obtained after successful submission should not be mixed with the analysis results, but should be used as independent SubmitRecommendationResult, or as artifact-
External record references in Refs are returned; this way the lifetimes of analysis output and write operation results are not mixed.
Before submitting the record, the validator must continuously confirm several specific things: the title cannot be empty; each suggestion must be able to refer back to feedback or historical records; the target space must be within the authorization scope of the current user; the priority can only take the value allowed by the system; the same taskId + stepId + targetScope cannot have already submitted resources; and the user must have confirmed this write operation. If any item is not satisfied, you should stop before submitting, instead of letting the model fill in a default value and continue execution.
During acceptance, it is recommended to make judgments directly around the fault scenario. Empty feedback should be returned to supplementary input instead of generating general conclusions; when the history record service is unavailable, it can prompt “missing historical matching” but cannot forge references; solutions without evidence cannot enter the submission step; users must not initiate write requests after rejection; after network timeout, existing results must be queried by idempotent keys; successful returns must include record IDs and accessible links. Such an acceptance description covers both user experience and server behavior, and is closer to the real risk than “what is the task success rate?”
In this case, the product manager participated in the design of the Agent’s technical solution by defining tasks, tools, permissions, results, and acceptance criteria, rather than just describing a chat page.
10. Product manager’s deliverables: need to focus on key business actions
Agent requirements often fail in a very simple place: the document states what the user wants, but does not clearly state what the system should do under uncertain conditions. R&D can only stuff judgments into prompts, and testing can only rely on feeling for a few rounds. More effective delivery is not six separate checklists but one task card organized around key business actions.
Taking the action of “submit suggestions to the target working system” as an example, the task should write the following content in the same context: what type of user problem the task is to solve, which inputs are trusted sources, what pre-steps must be passed, which systems can be accessed for reading and writing respectively, under what circumstances will it pause and wait for human confirmation, how to avoid repeated submissions after the external interface times out, what resource identification will be returned in the end, and which counterexamples must be covered in the test. In this way, what R&D sees is an achievable closed loop, not a set of dismantled nouns.
You can use the following template to review whether a task card is complete enough:
The earlier product managers can fix these problems, the easier it will be for R&D to choose appropriate execution models, tool protocols, and storage methods, and the easier it will be for testing to establish stable regression samples. The goal here is not to burden product managers with implementation details, but to avoid critical business judgments from appearing in the form of patches late in implementation.
11. Summary: From functional requirements to Agent system contract
The technical design of Agent products cannot stop at prompts, models and chat interfaces. What really needs to be designed is a business action chain: the input and completion conditions of the task are first fixed, and then split into steps with dependencies; each step can only be executed through tools with Schema, permissions, and side effects constraints; the results are delivered in a traceable structured protocol; exceptions, retries, and verifications ensure that this chain is still controllable in an uncertain environment.
This does not mean that the product manager decides on the storage engine or scheduling framework for R&D. The product manager is responsible for making clear in advance those judgments that cannot be freely exercised by the model: when a requirement proposal is completed, which sources can be used as evidence, what scope is allowed to be written into the business system, what should happen after user rejection or interface timeout, and which counterexamples must be blocked by testing. R&D selects the implementation method based on this, and testing establishes regression samples based on this. Both parties do not need to correct the business boundaries through supplementary rules before going online.
Ultimately, the competitiveness of Agent products is not only whether the model answers naturally, but also whether it can stably transform complex work into executable, traceable, and verifiable business results. Good Agent requirements are to first write the business boundaries that cannot go wrong into tasks, tools and verification rules.
Note: The illustrations and part of the code in this article were generated with the assistance of AI
Share vivo’s Internet technology tips and salon activities, and recommend the latest industry trends and popular conferences.
