Warcraft III is a real-time strategy game from 2002: you gather gold and lumber, build a base, research technology, raise heroes and an army, and destroy your opponent while they do the same. More than twenty years on, people still play it competitively, and streamers such as former pro player Grubby and the casters of Back2Warcraft bring it to large audiences.
This post is about two AIs. The first plays Warcraft III: the whole game, on the classic ladder map Echo Isles, under the game's own rules, against the strongest opponent the game ships with, the built-in AI on insane. It now wins about nine games in ten. The second built the first. Claude Opus 5.5, working as a coding agent in Claude Code, wrote the code and ran the research: it launched every experiment, watched it, judged it and chose the next one. My part was mostly to say where to go.
Loops like this are catching on. Andrej Karpathy's autoresearch lets an agent tune a small language model's training on its own, about a hundred five-minute experiments overnight on one GPU. NVIDIA's Eureka had a language model write and refine the reward functions that train robots in simulation. The AutoLab benchmark gives agents long research and engineering tasks, such as speeding up CUDA kernels, and found that the agents that succeed are the ones that keep measuring and trying again; Claude Opus 4.6 came out on top. This project runs the same kind of loop on a slower problem: weeks of reinforcement learning in a real game.
Why this is hard
A game lasts 10 to 25 minutes: thousands of decisions for dozens of units at once, four races with their own units, buildings and spells, and a fog of war that hides the opponent. The one real reward, a win or a loss, comes at the very end. DeepMind's AlphaStar (StarCraft II) and OpenAI Five (Dota 2) mastered games like this in 2019 and beat top human players, with the game makers' help and a datacenter: AlphaStar trained 12 agents for 44 days on 32 TPUs each. We had one desktop, a game that was never built for any of this, and an AI that runs experiments.
An AI runs the lab
Claude worked in a loop. Every experiment was a new training run that started from the best agent so far, with one change: a reward weight, a new input, a different mix of opponents. Every half hour it checked on the run: is it alive, how fast is it, any errors, and how does the new version play against the one it started from? Once each had played about 300 test games, Claude called the result, wrote it into the experiment log and started the next run, day and night.
The loop only works with a yardstick the agent can trust. Every run plays test games against the insane AI at full strength on the real map, and the new version and the one it started from play exactly the same matchups, so they can be compared directly. A laptop next to the desktop plays about 1,300 of these games an hour, so a verdict takes hours, not days. The dashboard below is what Claude reads at every check.
In about three and a half weeks, Claude co-wrote 496 of the project's 497 commits: some 30,000 lines of Python, plus the C of a DLL that runs inside the game, the game-script harness and a C++ inference loop. It ran 86 whole-game experiments and kept a lab notebook of them, along with notes it reads back at the start of every new session. Most experiments fail. Of the 21 on the real map so far, five were gains; the rest were level or worse, and the log keeps every one with its numbers.
It also kept the lab running. When a Warcraft III window popped up on my desktop in the middle of training, it traced it to a game process that had inherited my screen, and fixed it. When the laptop dropped out every twelve hours, it found that the laptop's remote login wanted a person to approve it again. That one it could not fix alone: I had to turn the check off.
My part was the direction: beat insane on the real map, try AlphaStar's ideas, find out what limits learning, keep the experiments bold. And now and then a correction, when it changed more in a run than I had asked for.
Running the real game, fast
Warcraft III has no API for bots, no headless mode and no faster-than-real-time option. It is a 32-bit Windows program, and its rules, pathfinding, spells and AI are decades of details that no reimplementation would get right. So we run the real thing, under Wine on Linux, about fifty copies at once, and drive it from Python.
The harness, written in the game's own map-scripting language, reports what the player can see every half second of game time and carries out the agent's orders. The shim, a small DLL inside the game process, replaces the game's clocks with a virtual one: the game runs as fast as the CPU allows and waits at every step for Python's orders. Games are deterministic, so a replay reproduces a game step for step. That is how every video here was made.
Self-play speed, agent steps per second
An agent step is half a second of game time for one player (two steps per game second, and two players in a self-play game). Each bar adds one optimization to the ones above it.
Most of the speed came from removing waste, not adding hardware: games draw nothing, five share one load of the map, the per-unit bookkeeping moved from the game's script into C, and the network runs as CUDA graphs from a C++ loop. Today the desktop plays about 48 games at once, each about 15 times faster than real time: roughly 730 hours of Warcraft III every hour.
The agent
The agent sees only what its player sees, through the fog of war, and every half second it gives each of its units an order: train this, build that there, attack that unit, move here. It has no screen and no mouse. It starts as a copy of the built-in AI, trained to imitate the AI's orders in thousands of its games. Then it learns from playing, with reinforcement learning (PPO, as in OpenAI Five), against a league of opponents: itself, its past selves and the built-in AI.
Watching it learn
Starting on the full game would mean about 3,000 decisions per side before the one reward arrives, and an agent that loses every game has nothing to learn from. Even a copy of the built-in AI, trained on the real map, lost all 40 of its test games. So the agent climbed a ladder of smaller games, each a step closer to the real thing.
Micro fights came first: small battles of a hero and a few units a side, over in seconds. They are where the pieces were worked out: units as tokens, targets as pointers, starting from a copy of a scripted player, and a league of past selves.
The opponent needed a curriculum too. The first self-play run won none of its last 75 games against the built-in AI. Giving the AI a late start backfired: the agent learned to rush the idle AI. What worked was a tax: the AI loses a share of everything it gathers, more after each of the agent's losses and less after each win, so the agent wins about half of those games and always has something to learn from. The tax fell towards zero as the agent improved. The test games never have a tax: they are always against the AI at full strength.
The teacher looked over the student's shoulder. A copy of the AI only knows the situations the AI gets itself into. So, as in on-policy distillation, the built-in AI supervised the agent in its own games: it labelled the orders the agent should give, in the states the agent itself reached. With this advice in half its games on the six-minute map, the agent went from 1 win in 98 test games to 13 in 50 within fifty minutes.
Moving to the bigger map needed one change in how the agent sees positions: measured from its own base, at the scale it already knew. After that, the agent trained on the fast rules won 49% of its test games under the real rules straight away.
What moved the needle
The real game on Echo Isles: games won against the insane AI
Each bar is the best agent after one change, measured in test games it never trained on.
Every change was one experiment, judged on test games against the insane AI. These helped, in the order they came:
Credit that skips mistakes 41% → 73%
AlphaStar's UPGO: a good action's credit follows the game only while the actions after it did at least as well as expected. A later mistake no longer drags down the good moves before it.
Train the network that plays 55% → 81%
The network played its games without dropout but was trained with it, so every update measured a slightly different network than the one that played. Without dropout in training too: one of the biggest jumps on that map.
Stop listening to the teacher 51% → 63%
Once the agent beat its teacher, the built-in AI's advice (above) only held it back. Switching it off was a gain, and training ran faster without the AI in the games.
Look further ahead 55% → 59%
How much a reward counts, by how far ahead it comes. With the old discount (0.999) a win eight minutes away counted about a third. Now that takes 28 minutes, the length of a long game.
Find room to build 57% → 62%
Building spots the agent learned on the flat duel map often missed Echo Isles' trees, cliffs and blight, and the game refused three of four undead buildings. The harness now looks for a free spot three times farther out.
Half the hint 62% → 72%
Besides +1 for a win and −1 for a loss, the agent gets small rewards as its material lead grows or shrinks (potential-based shaping, which leaves the best play unchanged). At half the weight, the win counts for more.
A looser anchor 74% → 82%
Two forces train the agent: the reward pushes it towards better play, and a KL term pulls it back towards an earlier version of itself, the anchor, for stability. Per update, the reward's push was about 1% signal and 99% noise, the anchor's pull was clean, and experiments kept ending level. A quarter of the pull let the agent move.
Give it a plan 84% → 89%
As in AlphaStar, half of the training games come with a target build order and tech path, drawn from 8,072 seen in games of the AI and of people. The agent follows the targets only loosely, yet it became a better player.
And some that didn't help, each one a run in the log: rewarding hero levels, twice the learning rate, a bigger batch, more games for the weakest races, less of the built-in AI among the opponents, mixing imitation back in, an exploiter trained to beat the agent, more reward for following the target, annealing the material reward away, an anchor that moves along with the agent, and capping workers in the material reward.
Here is the agent today, with each of the four races.
What it can't do yet
It plays one strategy. It wins with an early rush of cheap units, half its wins inside ten minutes. When the rush fails the game goes long, and almost all of its losses are long games. In the best agent's 183 long losses, 27 minutes on average, it built a big base with 35 workers to the AI's 16, but stayed with two or three kinds of cheap unit to the AI's nine, and three upgrades to the AI's seventeen.
It neglects heroes: about one a game, where the built-in AI raises one and a half or more. In the clip at the top it has none at all.
It has only met the built-in AI. Good human players beat the insane AI every time; they are far stronger than anything the agent has faced. A human would likely see the rush coming and hold it off, and the long game that follows is where the agent is weakest.
What's next
The agent plays one way, and so far it has only learned to beat a script. Next, we want it to:
- Play more diverse strategies, not just one early rush.
- Keep getting better through self-play, well past the built-in AI.
- Play more maps. Beyond the ladder maps, Warcraft III has thousands of custom maps, the birthplace of whole genres: DotA, which became Dota 2 and inspired League of Legends, or the tower-defense map Legion TD, now the standalone Legion TD 2.
- Scale up compute. Everything here ran on one desktop.
All of this runs on patch 1.29 from 2018. Newer versions of the game, from Warcraft III: Reforged in 2020 to today's 3.0 with its Definitive Edition graphics, run only through Battle.net, Blizzard's online launcher and copy protection, so we didn't attempt them.
The research continues, and there will be follow-up posts on how it goes. Until then, the details are in the project's documentation, including Claude's experiment log with every run.