原文：Your Game Is A Harness
Author: Kyle Kukshtel (producer of the independent game “Cantata”, author of the Depot game data editor, 15-year senior game engineer)
Preface: From the smoothness of the Web to the wall of the game
I want to talkHow to use large language models in real game development. I’m not talking about those “one-shot” demos that look cool on social media but only generate toy projects with one click, butTrue full-time R&Dthe kind of engineering environment where you work five days a week and write solid code for eight hours a day.
Before getting into the topic, I think it is necessary to reach a consensus on the understanding of LLM, or at least clarify some basic common sense, so that the subsequent content will not appear abrupt. For many developers who are already familiar with this field, this part may just be a refresher; if you just want to go straight to the practical stuff, you can scroll down to the next chapter.
My journey of introducing LLM into my daily development work began with a mood board website I made called mood.site. At that time, there was no so-called “intelligent agent” concept in the industry. I just wanted to test whether ChatGPT at that time could write decent code. I learned about GPT at the beginning. At that time, it was very primitive and clumsy to use web-side coding. Although the experience was painful, the results were eye-catching for me. Although I still needed to manually check a lot of logic at the time, that experience was enough to completely transform me into a believer in AI programming. Without LLM, I would never have been able to complete it in such a short period of time (it took about 2 to 3 weeks and a total of about 20 hours in the evening after get off work). If I had made it from scratch by myself, I might still be tinkering with the first version to this day.
Since then,mood It became my testing ground in the evolution of LLM technology and inspired me to start many new projects using similar technology stacks. In addition, with the birth of Claude Code and subsequent top-notch Claude models, LLM’s engineering capabilities have experienced a qualitative leap. From my personal experience, I think that when faced with a modern web technology stack, LLM There is almost nothing that cannot be done。
Seeing this, people can easily conclude that “LLM is omnipotent”. After all, most of today’s software engineering is Web software. However, whenever I try to apply LLM to building any non-Webgame developmentscene, things become different:
Big language models are not good at making games at all
Every time I try to get LLM to handle game-related systems, it almost always crashes.
This may contain my personal professional bias. I’ve been a game programmer for nearly 15 years, and I think I know the underlying logic of a game so well that I can smell bad game code from a mile away. But if it is a Web application written by Svelte (Svelte is a new generation of Web front-end framework), my involvement is obviously not that deep, so I am more likely to think that the Web code written by LLM is “pretty good.” Maybe it is actually a piece of garbage in the eyes of senior front-end developers? But we do have good reason to believe that the quality of the Svelte and Web full-stack code produced by LLM is objectively higher. This is largely due to the existence of the open source world.Massive high-quality Web open source projectsfor model training. In comparison, the amount of high-quality game source code publicly available for training is several orders of magnitude smaller.
Not only that, the game also contains a vast number ofnon-code content. Games are much more than just placing static images and text into a page layout. There are a lot of complex assets in the game world that simply cannot be represented simply by code.What’s more deadly ison top of this, the game still needs to handle all the things similar to Web layout, but this is just the tip of the iceberg of more than 100 independent subsystems that the game has to deal with.
This is one of the reasons why many developers are keen to use Three.js to make AI demonstration games: they can easily abstract all assets into basic geometric blocks, thus skipping asset generation, format adaptation and pipeline pre-processing that are usually indispensable in game development. The price is that many such works are full of “industrial garbage” flavor, because apart from the mediocre style of the LLM pre-training set, they do not have any real authorial expression.
However, this line of thinking is not entirely without merit.
The core logic of the One-shot Three.js workflow (and LLM’s superb proficiency in building web applications) is:everything is code. When everything exists as plain text code, LLM can understand context, make deductions, and perform modifications extremely easily.
However, for traditional game development, this is simply not the case. In other words, code is far from the whole story. Of course the game has code, but it also contains a lot ofnon-code assets. This means that there are a lot of problems in the game for LLMcompletely opaquekey components. If your game’s core logic relies on manipulating and transforming these black-box assets that are invisible to large models, then LLM is bound to collapse on the spot when trying to modify your game.
I think a lot of game developers have tried to use LLM a while ago and have hit this brick wall without exception. The public opinion cry “LLM sucks/has no use at all in the game” came into being.
in short:From the underlying structure, LLM cannot read the game at all by default.
Large language model minimalist common sense and context management
Let’s take a closer look at the root causes and how exactly we can reverse this situation.
A prompt word call initiated to any LLM service can essentially be broken down into three basic components: system prompt word, assistant prompt word, and user prompt word.
- System prompt word: A common command common to all users. It is usually used to define the basic personality and rules of the model itself, such as “You are Claude, an intelligent large language model trained by Anthropic…”. This is not unimportant (it’s very important), but it has little relevance to the specific scenario we’re discussing today. You can view cases of system prompt words leaked by major manufacturers here.
- Assistant prompt word: You can think of it as a sidecar structure used to guide how the AI responds. It represents the historical response vector given by the model based on certain inputs. Normally you don’t need to worry about it directly.
- User prompt words：This is the most important thing. Almost everything you input into any AI tool will fall here. All chat, accompanying code, error logs, etc. are dumped into this area. When people talk about “context”, most of the time they are referring to “the number of tokens accumulated in the user and assistant prompt words.”
It’s easy to misunderstand that every chat in a dialog box is a separate “new” version. But the actual operating mechanism is: when you collaborate with the model in a session (different tools have different definitions of session boundaries, such as a single terminal Session before closing Claude Code, or an independent Chat in the sidebar of a web page),Every historical message you have sent before will be completely resubmitted to the model when you send a new message.。
It’s like the game of Solitaire you played as a kid on the playground: when it’s your turn to speak, you have to add what everyone else has said before you.Retell it from beginning to endand then add your own new content at the end.
Context and context pollution
When you repeat this process over and over again, the “context” in the current session expands dramatically.
This sounds logical, but please be aware of a fatal trap:What you can really directly intervene is always the “latest” message in the context.. Everything from previous messages is mechanically submitted to the model over and over again.
Therefore, you must do your best to ensure that all that remains in context is “good information.” Otherwise, you will end up in extremely bad degradation scenarios:Poor-quality information generated in the early stage (erroneous attempts, hallucinatory codes, invalid logs) will form continuous “context pollution” and continuously mislead all subsequent inferences of the model.。
It is under this physical constraint that managing context has become the decisive skill to control LLM, and a new engineering practice has been born in the industry:context engineering. The core soul of contextual engineering can be summarized as one question:“How do we inject the most accurate and effective high-quality information into context with the minimum amount of contamination?”
Daniel Breunig once wrote a very enlightening and in-depth article “Typical Patterns of Long Context Failure and How to Repair”, in which he extracted a very penetrating conclusion:
Contextual engineering itself is an extremely ambitious topic and is beyond the scope of this article. However, we will draw on its core principles in depth to solve the proposition of “how to make LLM show its talents in game development”.
in other words:Making LLM competent in game development is essentially a matter of “contextual engineering”, and it is not because of any irreconcilable mutually exclusive flaws in the game or LLM’s genes!
However, the complication here is:The game is so special. Or to put it more bluntly: humans have never had a standardized template for how to make a game. Although we have commercial game engines such as Unity and Unreal, there is almost no standardization at the specific tool flow level: each game will evolve a completely different architecture based on its own mechanical requirements.
What is “scaffolding”?
This lack of standardized tooling is particularly acute in gaming, but it’s also common in other cutting-edge technology fields. Because of this, a brand new concept emerged to command intelligent agents to efficiently manipulate complex engineering products. This isscaffold。
The core concept of the so-called “Harness (scaffolding/restraint/wire harness)” refers to:A set of standard action patterns and interaction contracts pre-set for the agent so that it can respond accurately to clear (or broad) input instructions.
The Claude Code and Cursor you use every day are themselves a kind of general Harness; but they can be completely specific to what you are engaged in.Specific business areasExpand, distill and customize. Recently, the industry has been making frequent moves in this field, and a large number of customized harnesses for different software development cycles have emerged:
These excellent Harnesses have only one thing in common:“How can the Agent understand the object it is building more clearly and accurately guide it throughout the entire development process?” A modern Harness can integrate browser automated verification, telemetry data capture, performance analysis, and even throughout the entire software development life cycle, such as automatically creating work orders, updating design documents, etc.
However, all existing general-purpose harnesses have a fatal shortcoming: they simply cannot talk to the core of your game.
Your game is like a tightly welded black box to them. The only thing LLM can see is the text characters in the code editor. But as we discussed earlier, for games, plain text code simply cannot represent the actual running state of the game. Universal Harness itself cannot automatically solve the core pain point of “games are more than just code”.
The real problems that need to be solved are:We must find a way to transform all the dark matter in the game (status, resources, physics, performance) into a concrete carrier that LLM can understand and manipulate. We must pry open the black box of the game with our own hands, making the internal running details of the game transparent and addressable to LLM.
During the speech that day, I put up a very humorous slide, claiming that the arrow on the screen was the entire meaning of the speech. Although the effect of this baggage has been discounted in the blog post, I decided to keep it here as it is:
Now, let’s roll up our sleeves and get into real engineering practice.
The following is a set of highly concrete and immediately implementable game engineering architecture principles that I have summarized, aiming to make your game transparent, easy to read, and LLM-friendly.Stable operation。
By the way: even if you leave AI aside entirely, most of the following suggestions are excellent engineering practices for game development itself. With the support of AI, they will explode ten times more productivity; but even if you develop purely by hand, following these principles will make your code structure cleaner and more elegant.
1. Fixed randomness
The core theme throughout this article and the entire speech iscertainty。
in short:To what extent and at what cost can you 100% restore an extremely specific state in the game?
Our ultimate goal is that state reproduction should be extremely easy and flawless. If you fix this when the project is almost complete, it will be extremely painful, and you may even have to reconstruct the entire underlying structure.
On the contrary, if you establish this constraint as an unshakable iron law before typing the first line of code, you can completely say goodbye to the nightmare of logically confusing code throughout the entire development cycle.
The first step to achieve this is very straightforward:At the beginning of the game, establish a global single random seed source. It is mandatory that any calls involving randomness in the code go through this unified seed generator.
This means that your game has the ability to reproduce almost any game situation under deterministic conditions. Even if your logic is called in 30 different places Random.Float To determine the behavior of the monster, as long as the seed is fixed, the value of the 30th call must be exactly the same every time it is run. Certainty is thus established.
2. Visual screenshot ability and image difference
Why is deterministic recurring state so critical?
Imagine that the Agent is going through dozens of rounds of automated iterations, trying to fine-tune the layout of a certain game interface. If the Agent cannot repeatedly and accurately restore the game scene from scratch, but is frequently hindered by failed state reproduction, its iteration will completely stagnate.
And once the Agent successfully enters a specific state and makes changes, how should it evaluate whether its changes are right or wrong?
The most intuitive way is naturallyscreenshot. Most commercial engines provide ready-made screenshot APIs:
It is strongly recommended to add to the engineFreely switch between UI layer and gameplay layer rendering switchesability. In this way, you can turn off the dynamic 3D rendering of the background when verifying the UI style, completely eliminating the interference of background animation on visual evaluation.
Letting the Agent capture the “before modification” and “after modification” pictures before and after modification seems to be an excellent way for it to intuitively understand the changes in the world.
…Is this really the case?
The fact is extremely cruel: when you directly throw the two large pictures before and after modification to a large multi-modal model, the model oftenExtremely difficult to accurately identify subtle changeslet alone quantitatively assess whether this change is approaching the final goal.
This kind of interference from massive raw pixels is itself a kind of hidden “context pollution”:Blindly feeding high-resolution screenshots into the model will seriously blur its perception of the progress of the task, eventually leading to confusion for both the agent and the human.
Let’s look at a vivid real-life case: Suppose you take a screenshot of the official homepage of this developer conference:
Your command is: “Turn the skeleton in the picture into a pirate.” Even if the model has local annotation capabilities, except for your core modification area,A large number of irrelevant pixels in the green areaThey will all be stuffed into the context:
All pixel data within the green area is essentially purecontextual noise. LLM is not only forced to analyze the changes you requested, but it is also passively processing the messy information in this entire picture.
You have to force LLM Only focus on the parts that have changed. How to do it?
The answer is:Image difference!
Just like when we compare code Diff in Git, if you perform a mathematical difference on the two images and perform precise local cropping, LLM can visually see the changes with a very high degree of confidence, and even obtain aQuantitative similarity index。
Fortunately, you don’t need to write the algorithm by hand from scratch at all. Now the open source community has a very excellent, free and command-line-supported image difference artifact: odiff.
Just give it two pictures,odiff You can instantly calculate the difference between the two and directly output a Diff image with the difference highlighted:
Even more wonderful is,odiff Ability to print quantified similarity values directly to standard output (e.g. 85% similar). LLM is extremely sensitive to such plain text values! In successive Agent rounds, it can use this percentage as a solid baseline evaluation function to autonomously and steadily converge toward the goal of “reduce the difference/increase the similarity”.
But that’s not the full picture. Games are much more than static visuals. Under visual verification, there is another more criticalinternal state advance loop：
This inner loop is the battlefield where the Agent spends the most time: before finally triggering screenshot comparison, it will autonomously run many rounds of purely logical iterations in an environment without visual rendering.
Our core principles are:Ensure that every change is the smallest change required to achieve the goal and can be verified in milliseconds. You need an extremely stable, almost “boring” code baseline that can perform deductions in a nearly completely isolated vacuum environment to prevent the spread of technical debt and spaghetti code.
This brings us to the core engineering solution to deterministic states.
3. Deterministic state and single Fat Struct architecture
There is an extremely simple litmus test to evaluate how close a game architecture is to a “deterministic state”:
How easy is it to fully save and load exactly a specific state in your game?
Beginners often regard archiving/reading files as an ordinary “function” that is patched later in development; but senior engineers know that if the archiving/reading mechanism is integrated into the entire architecture design from the first day of development, it will become an engineering superpower throughout the entire life cycle.
Let’s look at a revolutionary design philosophy:
Most game programmers who have been immersed in traditional object-oriented hierarchies for a long time will be painfully aware of how true this sentence on the screen is towards the end of the project:“Trees are dogs”。
When you look at the specific logic, you will find that these two entities, which are very different in reality and have very different inheritance chains in the OOP system, are actually the same thing from the perspective of the underlying game code:
- Tree: has Mesh (grid) and Collider (collision body).
- Dog: Has Mesh, Collider, plus movement ability.
“Movement” is not a unique property of dogs (other things in the game also move). Therefore, trees and dogs are nothing more than different combinations of several components. From the perspective of the underlying code, they are exactly the same thing.
This insight may be counterintuitive and even feel “uncomfortable” when first heard. But once you break this obsession, it suddenly becomes clear that you can finally break free of the suffocating wall of abstraction you’ve built to maintain a complex OOP class hierarchy.
Embrace Thing-driven development
If you implement this concept to the end, you can make a bold assumption at the beginning of the game project:Everything in the game is essentially the same structure(We might as well define it directly as Thing structure):
In this structure, it contains some common basic attribute values, as well as isAlive Logo expressing life cycle concept. As for the specific category of the entity, it is purely composed of an enumeration field Type To identify; the specific game logic code performs branch processing based on the enumeration value.
And it turns out that AI simply loves this structure!
In this architecture, your object model hasthe only source of truth. Looking back at the traditional OOP architecture: if you define 100 inheritance classes for 100 types of objects, in order to complete a modification, AI must read all 100 classes and their implicit inheritance relationships into the context. This will instantly fill up the context window and put the AI into severe decision-making anxiety:“Which base class should I inherit? Or refactor the parent class method? Or create a new derived class?”
Under a single large structure, there is no ambiguity. AI inevery timeDuring the changes, everyone clearly knows where to strike.
Not only that, the single structure makes archiving and loading extremely simple and robust:
When there is only one data type, serialization and deserialization become a one-way highway, with only one standard pipe in and out. If you switch to dozens of derived classes, you will run into a race condition caused by the object dependency tree during deserialization, and you will be forced to fight against the frustrating “polymorphic deserialization” in Microsoft documents. In game development, there are deadly pits out there.
Under the Thing architecture, the entire game is nothing more than a huge List list. Saving is to export this list at once, and loading is a completely symmetrical reverse analysis. The enumeration branch judgment occurs purely in the logic layer after deserialization, which is extremely refreshing.
4. Addressable state and command mode
Once you establish the above foundation, the complexity of other modules of the game will fall apart.
If there is only one object type in the system, capturing and tracking state changes becomes extremely simple, because the gameany modificationsthe goal is always clearly pointed at that unique structure.
On this basis, we can very naturally introduce the classiccommand mode：
The core idea of the command pattern is:Instead of directly calling naked functions that modify object fields, each change is encapsulated into a (ideally serializable) command object. These commands are queued and sent to a unified processor, which centrally implements changes to the object.
It seems like an extra layer of indirect reference, but its power is amazing:
Suppose the player performs a series of operations in the game and triggers a vicious bug. In the traditional mode, you have to find ways to contact the player and try to reproduce it on your computer based on vague dictation.
But in command mode,You only need to get the initial state snapshot, and then “play back” the serialized command flow intact to reproduce the bug with 100% accuracy!
More importantly, this mechanism can not only be used by humans,AI Agent can also automatically play back this command sequencein the process of independently trying to repair the code, it can verify whether the status is as expected at any time.
Taking it one step further: you don’t even need to manually generate these commands in the game client at all. You (or the AI) can manually compile a set of JSON commands in any text editor, and then inject them directly into the running game instance for execution.No need to compile any code in the whole process!
Command mode provides a strongly typed operating interface for AI, making the gameControllable like a puppet。
Stay for a moment on the compilation-free mechanism: The more things in your game pipeline that can be modified without recompiling, the more powerful the overall architecture will be. This is not only related to human development comfort, but also decisive for agent circulation:Agent does not need to wait for long C++/C# compilation and linking times, but can quickly verify code assumptions in a very narrow loop of several seconds.
5. Externalized data management
This naturally leads to one of my core concerns:Data management. Let’s review the previous single large structure code again:
By “data”, I mean the configuration data used by the game to describe the underlying static properties of entities. In the above example,Type、attackDamage、walkSpeed、boatSpeed All are static data. They form part of the game Thing The basic skeleton of existence.
Many people are used to hard-coding these values in C# / C++ class or structure definitions when writing games. Once you want to modify the base movement speed of a certain unit, you must modify the code, wait for compilation, and restart the game.
In the Agent’s workflow, such changes that require recompilation will cause huge friction losses.
A better approach would be:Completely externalize all static data into independent data assets (such as JSON, configuration tables) and completely decouple it from the logic code.
When the game starts, you only need to deserialize the external data and pour it into the object. Modifying values becomes a pure data file modification, and hot reloading occurs in seconds while the game is running. This architecture even allows any collaborator (planning, testing, or even outsourcing) who does not have an engine editor environment to directly take the released final game client for high-speed tuning.
This was our standard development pipeline when we were making the indie strategy game Cantata. The 7 non-programmer members of the team all use daily builds of the game, along with a custom data editor I developed. Depot Iterate on content. This workflow is so efficient that I am now building Depot into an independent commercial product for the industry:
When team members modify data in the Depot, the game build version is deserialized in real time and takes effect. The entire process does not require me to repackage frequently.
Likewise,AI Agents can also easily learn to use these external data tools. It can directly adjust data files and perform hot reload verification in the game like a senior numerical planner in the team, completely removing the productivity bottleneck of a long compilation cycle.
6. Standalone headless simulation black box
Now enter the core principles of the deepest water area of the entire architecture:Black box game architecture。
In traditional game development, core logic is often tightly intertwined with visual rendering, UI callbacks, and sound effects playback. The character launches an attack, and the script deducts the blood volume, plays the attack animation, updates the health bar UI and triggers sound effects in the same frame.
The consequences of this strong coupling are:In order to test any tiny branch of logic, you have to run the entire massive game: All 3D models must be loaded, materials must be compiled, and level scenes must be initialized. When a bug occurs, you can’t quickly tell whether there is a problem with the simulation logic, a damaged model mesh, or a null pointer exception in the UI binding.
A far better architecture is to completely separate the “simulation” logic of the game from the graphics engine and use it as an independent, headless pure code dynamic link library.
The engine itself does not “own” the game simulation core, it is just a dumb client that consumes simulation calculation results. The connection between the engine and the simulation core is completely through what we introduced earlier Command object For communication, the pattern is exactly the same as the client/server architecture in modern web development:
- The engine converts player input into Commands and sends them to the independent simulation core.
- The simulation kernel processes the state calculation and returns the updated state object (or the State Diff difference package from the previous frame) to the engine.
- The engine is only responsible for performing mindless visual rendering and animation playback based on the received status differences, without any core decision logic.
Give a specific example of physical displacement:
The character moves from point A to point B, and the presentation layer includes character walking animation, skeletal rotation, footsteps, and vision fog updates. But in the eyes of the simulation kernel, the displacementIt is just a numerical change of coordinates from A to B。
The engine submits a displacement instruction and the simulation kernel determines whether the path is legal; if it is legal, the kernel updates the entity coordinates and returns the latest status to the engine. The engine sees the state change and smoothly interpolates the model across the screen. The simulation kernel is the absolute only source of truth.
When your simulation kernel exists completely independently of the graphics engine, a huge bonus comes:Any entity can send commands to your core seamlessly!
- The network module can send instructions (network multiplayer online development becomes extremely simple).
- Unit test scripts can send instructions.
- LLM/AI Agent can also inject instructions directly into pure logic cores!
Traditional game unit testing often struggles due to heavy environment dependencies. But under the independent simulation core architecture,You can completely abandon the heavy game engine and use the purest standard C#/C++ command line tool chain to run thousands of rounds of automated unit tests in seconds in pure memory with extremely high concurrency.. This is exactly the operating environment that AI is best at and likes the most!
When you leverage this architecture, it becomes a superpower for game development. You can treat the game as a purely technical deduction object, completely free of heavy audio-visual resource overhead. Run simulations, hypothesis testing, and automated testing in a vacuum environment:
At this point, we have reached a critical point near the end of our entire flywheel operation diagram: we have deterministic input parameters, a structured and compilation-free state modification path, and can verify our changes at the millisecond level in a completely isolated vacuum environment.
7. Impact assessment and plain text performance analysis
After completing deterministic input, compilation-free control and isolation testing, we come to the last link of the flywheel:How do we confirm that the modifications we made have the expected positive impact on overall machine performance?
Traditional performance analysis relies heavily on manual experience. Developers need to manually look through the call stack and flame graph in Unity Profiler, Unreal Insights or JetBrains dotMemory:
These graphics tools are designed specifically for the “human eye”. AI hates this kind of non-interactive graphical interface(Although Unity is working on making its analyzer more readable for Agents).
What AI is best at dealing with isstructured plain text. If performance analysis tools can directly output sampling snapshots as detailed console text reports, the Agent’s iteration efficiency will soar exponentially.
Here, I strongly recommend a free, modern and command-line-supported performance analysis tool: sample:
samply You can easily mount it into your local project and dump the performance capture results directly to the terminal. AI Agent is familiar with it samply Source code and documentation, it can independently perform sampling, analyze performance hot line numbers, modify optimization codes, and resample for verification.Implement a fully autonomous performance tuning inner loop.
8. Closed-loop puzzle: Agent’s high-speed autonomous iteration flywheel
When we connect all the above tools, architectures and methodologies together, the ultimate picture of the entire AI game engineering emerges:
In this set withData-oriented, binary-level isolation, full-link compilation-freeIn the core engineering architecture, an Agent iteration round that originally took several minutes or even tens of minutes in a traditional engine has been replaced byRuthlessly compressed into a few seconds！
It essentially eliminates all the contextual blind spots and compilation waits that make large models flounder. It is precisely because the entire closed loop runs so quickly and accurately that this set of processes can cover all aspects of the development process without distinction.every small changeinstead of waiting until it is too late to do an expensive integration test.
Re-examining the whole picture, you will find: This is the ultimate Harness tailored for games.
Spoilers:This isn’t some external mystery tool, it’s your game itself! Any game has the potential to carry all of this from the beginning. Or as we said at the beginning:
Thank you for reading!
