Session 已经持久化了，为什么还需要 Durable Execution？ – 付威的网络博客

The pi-rust project address involved in this article:

Suppose you ask the Agent to go CHANGELOG.md Append a record. The model issues a tool call, the tool changes the file, but the process exits before saving the tool results.

After restarting, the chat history is still there, and the tool calls proposed by the model are still there, but there are no corresponding results. Should it be done again now?

If executed again, there may be one more duplicate record in the file. Without execution, you cannot directly tell the model “it has been successful” because the session record does not have this evidence.

The Session has been persisted, which does not mean that this execution has a reliable recovery method. What Durable Execution needs to make up for is the period from “the record is still there” to “knowing how to continue”.

Let’s expand the task a little bit and see how it is completed without interruption:

Give CHANGELOG.md Append a line “Fix login timeout” and run pytest -qand finally report the results.

This hypothetical task is used throughout the text below. For ease of understanding, let’s assume that only one tool call is executed, with no parallelism, retries, or context compression.append_note It is the tool name in the teaching example, not a built-in tool of rpi; rpi can pass bash Complete the same file append action.

There are four roles here:

The model generates responses based on the input and can also make tool calls. It will not modify the disk file itself just because it generates a call parameter.
Executors are programs that call models, dispatch tools, and receive results. The so-called Agent Loop is mainly driven by it.
The tool performs actual operations, such as modifying files, running tests, and returns the results to the executor.
Session saves the session history so that the executor can retrieve these messages later.

A normal execution can be divided into several steps. The executor first saves the user task and sends the task and existing history to the model. Assume that the model first requires appending records, the executor saves this tool call, and then calls append_note。

After the tool modifies the file, it returns “Add successfully”. The executor saves this result as a tool message and requests the model again. This time, the model can see the original task, the call it made earlier, and the result of the call simultaneously. Only then does it have a basis to continue to propose “running tests”.

After the test result is returned, the executor hands it to the model. The model generates the final report, and the executor saves the response and ends the run.

Simplified into teaching pseudocode, it is:

messages = session.load_messages()
messages.append(user_task)
session.save(user_task)while True:
    reply = model.generate(messages)
    session.save(reply)
    messages.append(reply)
    if not reply.tool_calls:
        break
    for call in reply.tool_calls:
        result = tools.execute(call)
        message = tool_result(call.id, result)
        session.save(message)
        messages.append(message)

This program has saved user messages, model replies, tool calls, and tool results. The “message persistence only” mentioned in this article refers to this range of capabilities, not a simple chat recorder that only saves the final answer.

The round of driving from receiving tasks to generating reports is called a run, in which the model can be requested and tools can be called multiple times. Session retains the session and can also receive subsequent new tasks. Being able to open this history after restarting does not mean that the previous round of execution will automatically continue.

After the tool results are saved, the executor also puts the updated message into the next model request. The model does not automatically continue from the last requested “think position”.

Let’s first look at the situation where the results have been saved, and put aside the problem of “lost results” for the time being. Suppose the append is successful and the tool results are saved, but the process exits before the next request is issued.

There are three business messages in the session at this time. The following is presented in an easy-to-read format and does not correspond to any vendor’s API protocol:

消息 1：用户任务
  追加“修复登录超时”到 CHANGELOG.md，
  然后运行 pytest -q，最后汇报结果。消息 2：模型提出工具调用
  call_id: call-9
  tool: append_note
  args:
    path: CHANGELOG.md
    line: 修复登录超时
消息 3：工具结果
  call_id: call-9
  result: 已追加一行

on disk CHANGELOG.md There is already one more line. It and the above conversation message are two different pieces of data: one is the external effect generated by the operation, and the other is the result record saved by the executor.

What disappears is the process memory:while Where does the loop execute and the current messages Variables and asynchronous tasks waiting for the next model request disappear when the process exits. The new process will not automatically go back to the next line of the old loop.

If the new program only does this after starting:

messages = session.load_messages()
show_history(messages)
wait_for_user_input()

Well the message does get restored but the task is stuck there. There is no program to initiate the next model request, and the test will certainly not start by itself.

This is not to say that all Session products can only wait for user input, but this implementation does not include recovery scheduling. Replacing the database, or keeping the history more complete, will not automatically fill in the missing call.

Now we can also answer a very direct question: since the last one is the tool result, wouldn’t it be enough to continue to request the model when starting?

For this simple example, this could be a recovery rule. But once an application adopts it, it has already begun to achieve execution recovery. We also have to deal with the situation where the tool requests to end the run, the user explicitly cancels, the task is suspended, etc. All history ending with the tool result cannot be regarded as to-do tasks.

Messages can of course carry these states. The key is not to build an extra one runs Instead, the program must have a clear basis, distinguishing between “the past that can be viewed” and “work that still needs to be performed once.”

Next, we add a set of running records to the program just now to see what judgments the new process can make based on this. The fields are still teaching instructions. There is no need to compare the actual storage format of rpi for the time being:

run_started
  run_id: run-17
  task: 消息 1工具调用消息：消息 2，call-9
tool_started
  run_id: run-17
  call_id: call-9
  effective_args: 消息 2 中实际执行的参数
  replay_policy: never
工具结果消息：消息 3，call-9，已追加一行
# 进程在这里退出，没有 run_finished(run-17)

run_started Indicates that the system starts processing this task, and there will be a corresponding message when it ends normally. run_finished. This case leaves run_started(run-17)but there is no corresponding end record, so the new process needs to check this unfinished run. Whether to continue should also check the tool results and status such as cancellation and termination.

call_id Then correspond the call and the result. Confirmation required when restoring”call-9 There is a result”, not just “A certain success message appeared in the history.” The same session may have modified the file many times, and they cannot mix the results.

tool_started Save execution parameters and rerun strategies. Here’s never Indicates that old calls cannot be automatically rerun due to missing results. Appending is not an action that can be repeated casually: executing the same call twice will append two lines. This is also called a non-idempotent operation.

In this window where the results have been saved, the recovery steps are not complicated:

The host starts, opens the existing Session, and calls the recovery program.
The recovery program found no end record run-17。
It checks the message by call ID and finds call-9 The result has been obtained and this addition will not be executed again.
It confirms that the current message is allowed to continue requesting the model, registering this run as a job to be continued.
The host drives the work to be continued, and the executor loads the message and initiates a new model request.

These steps are procedural actions within the application. The “restorer” is the part of the code that checks records, handles legacy state, and schedules follow-up work, not another model that guesses at progress.

Only draw this successful continuation branch, the timing is as follows. Focus on the first arrow after the process exits: the new process reads the record first and then decides what request to initiate.

Figure: Teaching concept timing, not the measured trajectory of rpi. The executors of the first half and the second half belong to the old process and the new process respectively; record writing is assumed to be successful. The figure only draws the “result saved” branch, and the situation that has not been saved will be expanded in the next section. Click on the image to enlarge it, or open the original image or view it interactively.

What exactly should be included in the next model request?

The recovery process has decided to continue, what should be sent to the model next? Let’s expand the input of this request. The request must retain the original message relationship. It cannot only have an isolated “continue” sentence, nor can it treat the original user task as a new task and start it over again:

system:
  当前系统指令user:
  追加“修复登录超时”到 CHANGELOG.md，
  然后运行 pytest -q，最后汇报结果。
assistant / tool_call:
  id: call-9
  name: append_note
  args:
    path: CHANGELOG.md
    line: 修复登录超时
tool / tool_result:
  call_id: call-9
  content: 已追加一行
# 请求还携带当前可用工具的定义，例如运行命令的工具。

Listed here is the complete sequence of business messages in this example, omitting the timestamp, usage and other fields in the real protocol. Practical applications may also have earlier histories, compression summaries, and recovery hints.

This request already contains “Append Success”, so the executor does not need to call it again append_note to get results. This is all about reusing existing results: reading out a previously saved tool message and putting it into the model input.

Suppose the model next returns:

调用 call-10：运行 pytest -q

The executor returns to the normal loop: Save call-10execute the test, save the test results, and then request the final report. After this branch is completed normally, the record will continue to grow to:

assistant：提出 call-10，运行 pytest -q
tool：call-10 的实际测试结果
assistant：依据实际结果生成的最终汇报
run_finished：run-17，completed

“Test passed” is not pre-populated here. The executor has to wait for the tool to actually return before the model can report accordingly. The subsequent response is also a teaching hypothesis and does not guarantee that the model will necessarily be selected for testing; it may still need to be clarified, or a different call made.

Put the above actions into the code, and the control flow is roughly as follows:

# 教学伪代码：单运行、顺序工具；保存成功才进入后续步骤。
# 省略并发接管、分支、取消、预算和存储错误处理。
def recover_on_startup(session):
    for run in session.find_unfinished_runs():
        messages = session.load_messages_for(run)        for call in find_calls_without_results(messages):
            # 这里才处理结果缺失：重跑，或者补“结果未知”。
            resolve_missing_result(session, run, call)
        messages = session.load_messages_for(run)
        if (not run.cancelled
                and all_calls_have_results(messages)
                and last_message_is_tool_result(messages)
                and not tool_requests_stop(messages)):
            scheduler.enqueue_continuation(run.id)
def drive_continuation(run_id):
    messages = session.load_messages_for(run_id)
    reply = model.generate(messages, tools=current_tools)
    # 保存新回复，执行它提出的新调用，再进入正常循环。
    agent_loop.accept_reply(run_id, reply)

In the snapshot just now,find_calls_without_results Returns an empty collection. Recovery procedure confirmation call-9 After the results are available, only the next model request is scheduled and the old tool is not executed. The pseudocode only covers the tool result continuation branch of this article, and is not a general recovery algorithm for any Agent state.

The combination of such records and driving logic constitutes Durable Execution: it not only saves the progress and intentions required for recovery, but also advances unfinished work according to rules in the new process.

Now let’s go back to the more difficult window at the beginning and move the interruption position one step forward:CHANGELOG.md has been modified,call-9 The results were not saved in time.

At this time, the messages only have user tasks and tool calls. The recovery program can recognize call-9 There is no result, but you cannot judge whether the file has been modified based on this gap alone.

Putting several adjacent interrupt positions together, you can see the difference:

Interrupt position	Saved basis	Judgments that the new process can make
`tool_started` Saved, tool not called yet	Call, parameters, strategy, no results	External effects cannot be confirmed solely by relying on these records
The file has been modified, but the result has not been saved.	Call, parameters, strategy, no results	You may see the same record as the previous line, but the effect is still unknown.
The results have been saved and the model has not been requested again	Call and corresponding results, the operation has not ended yet	Results can be reused to schedule the next model request

The first two lines may be different in the real world, but may appear the same to the recovery program. The new process does not see the last machine instructions of the old process, only the records that have been committed.

why save `tool_started` not enough

Let’s sort out “save execution records” and “modify files” to see why writing one more record is still not enough:

先写 tool_started → 再追加文件

If the process exits just after writing the record, the file has not been changed. If you exit after appending the file, the file has been changed. In both cases it is possible to leave only tool_started。

On the contrary, changing the file first and then writing the record will not work. The process may exit between steps, leaving an unrecorded file modification.

The problem is: the submission of session records is different from CHANGELOG.md The modifications are not in the same atomic transaction. The so-called atom refers to several things that either happen together or not at all. Here, both sides write separately, and there may be interruption in the middle. Transactions are only used within the Session database, and external file modifications cannot be automatically included.

so,The lack of results only means that the results were not successfully recorded, but does not prove that the tool is not effective. Persistence execution must deal with this unknown head-on and cannot treat it as a normal failure and try again.

It is not safe to rerun: first tell the unknown to the next round

For file appending, the recovery program can give call-9 Add a clear result:

tool / tool_result:
  call_id: call-9
  is_error: true
  content:
    上次执行在结果保存前中断。
    外部效果未知，可能已追加，也可能没有。
    恢复程序没有自动重跑该调用。

This is not a fake “append failed”.is_error It means that this call did not leave a confirmable result normally, and the text content still clearly distinguishes between failure and unknown.

Completing results has two purposes. It allows the tool call in the next request to have a corresponding reply, and also allows the model to see the uncertainty of this step, instead of quietly treating the gap as a success.

Subsequently, the application can require the model to be read first. CHANGELOG.mdcheck the target record, and then decide whether to make up for it. If the file may have been modified by others, or the same text may have already existed, simply searching for a line may not be enough to prove that the original call was successful. The verification rules are determined by the task itself, and can be forwarded to the user if it cannot be confirmed.

Refusing to automatically rerun old calls does not prevent the model from reproposing the same write operation in the next round. The application still needs to constrain subsequent new calls; it cannot just add an unknown message and claim that the entire system will not be modified repeatedly.

Safe to rerun: re-execute and save new results

If the missing result is a read file, and the tool states that safe reruns are allowed, the restorer can re-execute the read, save the return content as the result of the original call, and continue requesting the model.

The tool is actually executed here, unlike “Use Saved Results”. Moreover, the current file is re-read, which may not be equal to the content read before the interruption. The meaning of safety is to allow the action to be repeated, but not to guarantee that the same value in the past will be returned.

For external operations such as sending messages and creating orders, the downstream may also need to provide idempotent keys so that repeated submissions can be recognized as the same business operation; or the results of the original request can be checked through the query interface. The executor cannot guarantee exactly one execution for another system based on its own log.

The previous records and pseudocode are to illustrate the mechanism. With the two paths of “result saved” and “result unknown”, let’s compare the actual implementation of rpi, the Rust Agent project. The verified version is 0.15.0,submit fd47e29the corresponding relationship between the teaching fields and the real rpi records will be explained below.

During operation, which recovery basis should be saved first?

rpi use operation_started、operation_finished Indicates the beginning and end of an operation. The normal prompt path will submit the first user input together with the run start record, so that there is a basis for searching for unfinished operations during recovery.

After the model proposes a tool call, the executor will try to submit the assistant message carrying the call in advance without waiting for the completion of the entire round of tasks. When you are ready to execute the tool, try saving again. tool_startedincluding the associated run, call ID, parameters carried by the call start event, reserved result entry ID, and the replay strategy at that time.

The parameters here cannot be used as final execution parameters.ToolExecutionStart Emitted before parameter preparation, checksum execution and pre-hook;start_tool What is saved is the parameters in the event, and subsequent processing may still overwrite it.

When the tool returns, the executor will attempt to commit the results to the reserved entry. In this way, if the append results have been saved but the process exits before subsequent model requests, the new process can read the complete call and results without appending the file again.

These commit boundaries can be found in settle.rs of settle_assistant、start_tool、settle_tool_result Check in. But “try to save” cannot be read as “no matter how the storage fails, the state will not be lost.” Partial write failure paths only record warnings and do not prevent tool execution. The recovery judgment in this article is based on the successful submission of the corresponding record, and the guarantee when the disk is powered off or damaged is not verified.

When restarting, who initiates the next model request?

The host is constructed in a mode that allows resumption of existing sessions AgentHarness。create It will search for unfinished operations, repair the status left by the last interruption, and then determine whether the current message is suitable for continuation.

For the position of “result saved” in this example, the key basis is: there is an unfinished run before, the current branch has reached the tool result, and this result does not require the termination of the run. When the conditions are met, Harness first records “waiting for host connection” in the memory. This mark is called PendingResume. It is not a scheduled to-do that has been saved to the Session.

The registration itself does not yet request the model. Host call resume_pending Only then does Harness actually enter the continuation path. The interactive startup code of rpi and other running modes have this call entry, which does not require the user to re-enter the original task.

Continuation also has protection conditions: when the number of times budget is exhausted, or the message is in a position that does not allow continuation, it will not be started directly. A run that has ended normally or been explicitly canceled does not have any unfinished operations and should not be re-executed by opening the history.

rpi also has an implementation detail that is easily covered up by “restoring the original task”: it will first restore the old operation to Aborted / interrupted Finally, create a new operation for the continuation. New record passed resumedFrom、resumeAttempt The association and number of continuations are retained.

From the user’s perspective, the task continues to be completed; from the running record, this is a new execution process taking over the old context. The original process’s Rust call stack, old HTTP connection, and memory variables are not revived.

What rules are followed when results are missing?

rpi’s resolve_tool_call Responsible for handling calls with missing results. For ordinary local tool calls, it requires two conditions to be met at the same time before it will automatically rerun: the strategy saved at the time of the call is Safethe currently registered tool is still declared as Safe。

The historical policy states whether reruns were allowed then, and the current statement states whether they are still allowed now; without one of these, you cannot guess based on the tool name alone.

In the CLI default tool,read、docs Explicitly declared as Safe。bash、write、edit The default is Never. Therefore, even if a certain bash It seems that it just reads the file, and it will not automatically be qualified for safe re-run; the executor is processed according to the registration policy and does not rely on analyzing the shell string to prove safety.

use bash If the record is appended and then interrupted, the restorer will not execute the old command again if the result is missing. It will supplement tool results that account for unknown external effects, allowing subsequent model requests to have clear factual boundaries. Even if the calling message has been saved, the process has not had time to write it. tool_started Just exit, and you cannot automatically conclude that “the tool must not have been executed” based on this.

Safe declarations are the responsibility of the tool registrant. It is a security promise, not Harness automatically proves that the tool implementation has no side effects.

While the Agent was performing append and test tasks, the user sent another sentence, “Check the format after the test is completed.” If the application has acknowledged receipt but only placed the instruction in the memory queue, the work will disappear after the process exits. Just because you can see this sentence on the chat interface doesn’t mean that the executor has remembered it and hasn’t processed it yet.

rpi saves the enqueue record before posting the input to the in-memory queue. During recovery, the input that is still pending is reconstructed based on this, rather than re-executing every user message in the session.

After failing to request a model, the executor may also leave itself a to-do: wait for a while and try again. rpi will try to save before entering wait retry_pendinginstead of just leaving this arrangement in the memory timer.

During recovery, it will also be checked whether the corresponding results have been saved and whether subsequent generation has started. Having this record in history does not mean that you still owe a retry. If the retry attempt fails, only an alarm will be recorded, so this judgment is still based on the successful saving of the record.

These records are first used to arrange work for the executor, and do not have to be turned into model chat content. Saving conversations and saving to-dos can share Session storage, but are read by different codes and produce different actions.

If you just need to open the history and let the user decide when to continue, message persistence can meet this product goal. If the system promises to take over the accepted task after restarting, we can check our implementation in the following three places along with a tool call:

The results are saved, and the next model request is exited before it is issued: who recognizes that the run has not ended, and who actually initiates the new request?
The external operation may be completed and the results have not been saved before exiting: on what basis is rerun allowed, and how to expose the unknown when rerun is not possible?
Restarting after a run has ended normally or been canceled by the user: what prevents it from being considered a to-do again?

These three questions do not require buying a product named Durable, nor do they require moving all states out of the Session. They require applications to write clear logging and recovery rules.

Back to appending records: if the result has been saved, hand it over to the next model request; if the result has not been saved, deal with the uncertainty of this step first. Only by being able to complete these actions in the new process can we move from “history is still there” to “someone will take over the work.”

Original link:

Source link