Warcraft III is a real-time strategy game from 2002. You gather gold and lumber, build a base, research technology, raise heroes and an army, and destroy your opponent while they do the same. More than twenty years on, people still play it competitively, and streamers such as former pro player Grubby and the casters of Back2Warcraft bring it to large audiences.
This post is about an agent that plays the whole game of Warcraft III on the classic ladder map Echo Isles, under the game’s own rules, against the strongest opponent the game ships with, the built-in AI on insane, the hardest of its levels (easy, normal and insane). It now wins about nine games in ten. Here is how it got there.
Why this is hard
Most games last 10 to 25 minutes, with thousands of decisions for dozens of units at once. There are four races, human, orc, undead and night elf, each with its own units, buildings and spells, and a fog of war hides the opponent. The one real reward, a win or a loss, comes at the very end. DeepMind’s AlphaStar (StarCraft II) and OpenAI Five (Dota 2) mastered games like this in 2019 and beat top human players, with the game makers’ help and a datacenter. AlphaStar trained 12 agents for 44 days on 32 TPUs each. I had one desktop and a game that was never built for any of this.
Running the real game, fast
Warcraft III has no API for bots, no headless mode and no faster-than-real-time option. It is a 32-bit Windows program, and its rules, pathfinding, spells and AI hold so many details that reimplementing them faithfully would be very hard. So I run the real thing, under Wine on Linux, about 50 copies at once, and drive it from Python.
The harness, written in the game’s own map-scripting language, reports what the player can see every half second of game time and carries out the agent’s orders. The shim, a small DLL inside the game process, replaces the game’s clocks with a virtual one, so the game runs as fast as the CPU allows and waits at every step for Python’s orders. Games are deterministic, so a replay reproduces a game step for step. That is how every video here was made.
Self-play speed, agent steps per second
An agent step is one decision by one player, made every half second of game time. Each bar adds one optimization to the ones above it.
Most of the speed came from removing waste, not from adding hardware. The games skip drawing, five games share one map load, the per-unit bookkeeping moved from the game’s script into C, and the network runs as CUDA graphs from a C++ loop. Today the desktop plays about 50 games at once, each about 15 times faster than real time, which adds up to roughly 730 hours of Warcraft III every hour.
The agent
Swipe sideways to see the whole diagram →
The agent sees only what its player sees, and every half second it decides for every one of its units at once. The network has four main parts:
- The order head picks each unit’s order, such as train, build, attack, move, cast or nothing.
- The target heads pick what that order is aimed at, either a unit in view or a point on the map. A point is picked as x and then y, so both coordinates belong to the same spot.
- The memory carries a state from step to step, so the agent can remember what it saw before it went out of view.
- The value head predicts how the game will end. It reads hidden information and is only used in training.
The whole network is small, about 3.2 million weights. One decision for 60 units, with 120 in view, takes about 13 milliseconds on four cores of an ordinary CPU, far less than the half second it has.
A head start from the built-in AI. Learning by playing from scratch is too hard: an agent that gives random orders almost never wins, so it has nothing to learn from. To give it a good start, I recorded thousands of games of the built-in AI playing itself and trained the agent to copy the orders the AI gave. This is called behavioral cloning, and it leaves the agent playing roughly like the AI.
Learning from playing. Once the copying stops improving, the agent switches to self-play. It plays against a league of opponents (itself, its past selves and the built-in AI) and learns from the reward. The reward is +1 for a win and −1 for a loss, plus small rewards along the way for gaining material (more on those later). After each batch of games, every order the agent gave is judged by the reward that followed it, using policy gradients. If the reward turned out better than the value head expected at that moment, the order becomes a little more likely in that situation; if worse, a little less. The gap between the reward and what was expected is called the advantage.
Keeping it steady. A second term, a KL divergence, pulls the agent back towards an earlier version of itself, the anchor, so one noisy batch cannot undo what already works.
Measuring progress
Training numbers can’t tell whether a change made the agent better. The reward includes the small material rewards, not just wins and losses, and the training opponents change all the time, from past versions of the agent to a built-in AI that is made easier or harder as the agent improves. So every training run also plays test games that it never learns from. They are games against the built-in AI at the hardest level each stage aims at, without any tax or other help for the agent. On the real map that is insane. Each experiment is a new run that starts from the best agent so far, with one change. The new version and the one it started from play exactly the same matchups, with the same races and sides, so their win rates can be compared directly. A few hundred games each are enough to see clear differences. Every experiment, with its numbers, is in the experiment log.
How it learned
The problem. A real game often lasts 25 minutes, and the agent decides twice a second, for all of its units at once. That is 25 × 60 × 2, about 3,000 decisions per side before the one reward, a win or a loss, arrives. With a single result at the end it is hard to tell which of those decisions mattered, and an agent that loses every game gets nothing to learn from. Even an agent trained by behavioral cloning on the built-in AI’s games on the real map lost all 40 of its test games.
A curriculum. So the training follows a curriculum: the agent starts with easier problems and moves on to harder ones as it gets better. It began with small fights, then whole games on a small map with sped-up rules, then longer games, then the real map, first with fast rules and then with the game’s own. At each stage the test games showed a problem, experiments went after it, and most of them failed. These are the ones that worked, in the order they came.
The first stage was small fights: a hero and a few units on each side, over in seconds. This is where the basics were worked out, such as units as tokens, targets as pointers, starting from behavioral cloning of a scripted player, and a league of past selves.
27 September – 2 OctoberTwo-minute games
The first whole games were played on a small flat duel map with two bases laid out like Echo Isles’ main bases, with the rules sped up seven times, so a game took two minutes. The agent began with behavioral cloning of the built-in AI’s orders. On its first day of self-play it lost game after game, even against the easy AI.
A fair opponent
What was wrong. The first self-play run lost all of its last 75 games against the AI, so it had nothing to learn from. Two ways of making the AI easier failed, because the agent exploited both. With extra hit points for its units, it never learned to spend its money. Against an AI that started late, it simply rushed the AI before it could do anything.
The change. A tax on the AI. It plays normally, but loses a share of the gold and lumber it gathers. The share grows after each of the agent’s losses and shrinks after each win, so the agent wins about half of those games. The test games never have a tax.
The outcome. As the agent improved, the tax fell from about 75% to almost zero, averaged over all matchups. By 2 October it won 79% of its test games against the AI.
2 – 14 OctoberSix-minute games, against insane
The next stage had slower rules and a harder opponent, the built-in AI on insane. Against the untaxed AI, games took about six minutes. The agent again started from behavioral cloning, this time of the AI’s games under these rules. The tax kept the training games balanced, but they didn’t teach much. Most of the taxed training games dragged on and ended in a tie at the 20-minute time limit. The agent built more than the taxed AI, but sat on about 4,000 gold and lumber it never spent, and hardly attacked. The tax also hid how slowly the agent started. One minute in, the taxed AI had 7 supply against the agent’s 15, but the untaxed AI had 21 and won the test games in five to seven minutes.
On-policy distillation from the built-in AI
What was wrong. After hours of self-play the agent had won only 1 of 98 test games. Behavioral cloning only shows the agent situations that the built-in AI itself gets into. The built-in AI always keeps its buildings busy, so its games almost never show an idle barracks with money to spend, and the agent never learned what to do there.
The change. Let the built-in AI coach the agent in half of its training games. The built-in AI watches the agent’s game. In every situation the agent gets into, the built-in AI notes the order it would give, and training nudges the agent’s orders towards those notes. This is on-policy distillation.
Swipe sideways to see the whole diagram →
The outcome. Within fifty minutes of training the agent went from 1 win in 98 test games to 13 in 50, and the training games stopped ending in ties.
Over the next week and a half, most experiments made no difference. Two did.
Credit that skips mistakes
What was wrong. A game’s result is spread back over every action before it, so a good move followed by a lost fight was blamed for the fight.
The change. Switch to AlphaStar’s UPGO. With it, a move gets credit from what actually happened next only until the first later move that went worse than expected. After that point, it is judged by what the value head expected instead.
The outcome. Over two runs, this was the largest gain on the six-minute map.
Stop on-policy distillation
What was wrong. The agent now won about 60% of its games against insane, so it had become better than the built-in AI coaching it. Yet copying the built-in AI’s orders still made up most of each training update.
The change. Stop on-policy distillation.
The outcome. It won more often against insane, and training ran faster without the built-in AI coaching in the games.
15 – 17 OctoberThe real map
By 14 October the agent won 80% of its games against insane on the six-minute map, so it moved on to the real map, Echo Isles. The real map is far bigger and busier than the duel map, and a few things had to change for it:
- Where things are. Positions are now measured from the agent’s own base, at the scale it knew from the duel map, and the stretch between the bases is squeezed so that the enemy base is where it expects it. Without that, its buildings landed in the trees and its attacks went to the middle of the map, where the enemy base used to be.
- Where it can point. The grid of map points it can target grew from 128 to 352 cells a side, to cover the bigger map.
- The rules, in two steps. Training first used Echo Isles with the fast rules of the six-minute games, then the game’s own rules.
Swipe sideways to see the whole diagram →
The agent was trained on Echo Isles with the fast rules only. Tested under the game’s real rules, which it had never played, it won 49% straight away. From there it kept improving:
Games won against the insane AI on Echo Isles
Each bar is the best agent after one change, measured in test games. The outcomes below use the same numbers.
Look further ahead
What was wrong. Real games often last 25 minutes or more, against six on the small map, but the agent only looked about eight minutes ahead. Anything that paid off later hardly counted.
The change. The agent values a reward less the further in the future it comes. At every step, half a second, a future reward is multiplied by a number just below 1, the discount. With a discount of 0.999, a win eight minutes away (about 1,000 steps) counts only about a third as much as a win right now, since 0.9991000 ≈ 0.37. Raising the discount to 0.9997 stretches that to about 28 minutes, so the agent also cares about what happens late in a long game. The chart shows how much a win counts, depending on how far ahead it is.
The outcome. Its win rate rose from 49% to 59%, and the games got shorter. Half of the test games were over within 13 minutes, against 18 before, and fewer reached the 40-minute limit.
Find room to build
What was wrong. Building spots learned on the flat duel map often landed on Echo Isles’ trees, cliffs or blight (the undead build only on their own corrupted ground), and the game refused three of four undead buildings.
The change. Make the harness search three times farther for a free spot whenever the one the agent asked for is blocked.
The outcome. Once the agent had learned to use it, its win rate rose from 59% to 62%.
Halve the material rewards
What was wrong. Winning or losing is the only reward that really matters, but it comes only at the end of a long game. To give the agent feedback sooner, it also gets small rewards for material, the value of its units and buildings compared with the opponent’s.
Rewards like this have to be designed carefully. If the agent were rewarded for every unit it trains, it could train a footman, let it die and train another, and collect reward forever without getting any closer to a win. So the material reward is paid only for changes in the material lead (this is called potential-based shaping). Training a footman worth 135 gold adds 135 to the lead, and losing it takes the 135 away again. Going round that loop earns nothing, so the only way to gain is to come out ahead, and the best way to play stays the same.
Even so, the material rewards could pull the agent towards trading armies instead of winning. And now that the agent looked further ahead, the win itself already counted from early in the game, so it needed less help.
The change. Halve the weight of the material rewards.
The outcome. Its win rate rose from 62% to 72%, most of all for the weakest races.
A looser anchor
What was wrong. Three experiments in a row made no difference. Two forces train the agent. The reward pushes it towards better play, and a KL term pulls it back towards an earlier version of itself, the anchor, for stability. Measured per update, the push was about 1% signal and 99% noise, and the pull was clean.
The change. Cut the pull to a quarter.
The outcome. No difference for the first ninety minutes of training, then it pulled ahead, from 72% to 82%.
Give it a plan
What was wrong. The agent played one strategy, an early rush, and its losses were long games in which it never moved up to stronger units.
The change. Give the agent a plan, as in AlphaStar. Half of its training games now come with a target build order: which buildings, units, heroes and upgrades to make, and when. Each one is the build order of one player in a real game, one of 8,072 taken from the built-in AI’s own games and from replays of games between people. The target is an input to the network, and the agent gets a small reward for following it.
The outcome. Its win rate rose from 82% to 89%, even though it follows the targets only loosely.
Many more experiments didn’t help, and each one is in the log. They include rewarding hero levels, twice the learning rate, a bigger batch, more games for the weakest races, less of the built-in AI among the opponents, mixing behavioral cloning back in, an exploiter trained to beat the agent, more reward for following the target, phasing out the material reward, an anchor that moves along with the agent, and capping workers in the material reward.
Here is the agent today, playing each of the four races.
What it can’t do yet
It plays one strategy. It wins with an early rush of cheap units, half of its wins within ten minutes. When the rush fails the game goes long, and almost all of its losses are long games. Across the best agent’s 183 long losses, which lasted 27 minutes on average, it built a big base with 35 workers against the AI’s 16. But it stuck to two or three kinds of cheap unit against the AI’s nine, and researched three upgrades against the AI’s seventeen.
It neglects heroes. It raises about one a game, where the built-in AI raises one and a half or more.
So far, the target was the built-in AI. Apart from itself and its past selves, the insane AI is the only opponent it has played, and beating it was the whole goal. Good human players beat the insane AI reliably; they are far stronger than anything the agent has faced. A human would likely see the rush coming and hold it off, and the long game that follows is where the agent is weakest.
What’s next
The agent plays one way, and so far it has only learned to beat a script. Next, I want it to:
- Play more diverse strategies, not just one early rush, and use more of the game. It hardly creeps, fighting the neutral camps for experience and items. It rarely expands to a second gold mine or builds a shop, and its heroes don’t buy or use items. It never hires heroes at the tavern or mercenaries at their camps. Giving it a target build order was a first step, but it still follows those targets only loosely. Even when the target is enforced, so that it can only make the target’s items, it makes only about a third of the target’s buildings, units and upgrades.
- Keep getting better through self-play, well past the built-in AI. Playing against itself and its past selves can go in circles, with the agent forgetting how to beat strategies it no longer sees. AlphaStar countered this with a league that also has exploiters: agents trained only to find and punish the main agent’s weaknesses, so it has to fix them.
- Play more maps. Echo Isles is only one of the game’s ladder maps. Patch 1.29 alone ships with many more two-player maps, such as Terenas Stand, Tirisfal Glades, Melting Valley, Secret Valley and The Two Rivers, plus four-player classics like Lost Temple, Twisted Meadows and Turtle Rock. Beyond the ladder maps, Warcraft III has thousands of custom maps, the birthplace of whole genres, such as DotA, which became Dota 2 and inspired League of Legends, or the tower-defense map Legion TD, now the standalone Legion TD 2.
- Scale up compute. Everything here ran on one desktop, which played about 11 years of Warcraft III in all, two years a day at full speed. OpenAI Five played about 45,000 years of Dota 2 in ten months, some 250 years a day, on 800 petaflop/s-days of compute. AlphaStar trained 12 agents for 44 days on 32 TPUs each. And reinforcement learning, like language models, tends to improve predictably with more experience and bigger networks (scaling laws).
All of this runs on patch 1.29 from 2018. Newer versions of the game, from Warcraft III: Reforged in 2020 to today’s 3.0 with its Definitive Edition graphics, run only through Battle.net, Blizzard’s online launcher and copy protection, so I didn’t attempt them.
Beating the insane AI was the first milestone. The next ones are more strategic variety, and beating good human players.
The research continues, and I will write follow-up posts as it goes. The project’s documentation has the details, and the experiment log has every run.
Updates
18 October 2026. As pointed out on Reddit, the built-in AI on insane gathers gold and lumber twice as fast as a normal player. The post didn’t mention this. I measured it: with the same number of workers, insane gathered twice as much as the normal AI. Every test game here was against this insane AI, with its double income.
Citation
If you use this work, please cite the project:
@software{kahlow2026warcraftsim,
author = {Kahlow, Robin},
title = {{warcraftsim} - {Self-play} reinforcement learning in {Warcraft III}},
year = {2026},
url = {https://github.com/RobinKa/warcraftsim}
}Contact
Have questions or ideas, or want to work together? Get in touch, or join the Warcraft 3 AI server on Discord.
tora@warlock.ai LinkedIn github.com/RobinKa Discord: Warcraft 3 AI