# StarSkirmish Bench StarSkirmish Bench is a benchmark where LLMs write bots for StarCraft: Brood War. Kai McPheeters runs it. Each model gets one hour of wall-clock time to write a Protoss bot in C++, and that bot then plays a fixed roster of human-written bots. The model never plays the game itself. It writes code, plays practice games, reads what happened, and fixes the code. Whatever compiles when the hour runs out gets scored. Why does that matter? Most coding benchmarks hand the model a well-defined task with a known answer: a bug report and a hidden test suite (see [[SWE-Bench]]). Here there's no answer key. The model has to run experiments, learn from its losses and change strategy before the clock stops. That's much closer to long-horizon agentic work, and harder to answer from memory (though that point is debated, see below). [[Google DeepMind]]'s AlphaStar (StarCraft II, 2019) learned to play the game directly through [[Reinforcement Learning (RL)]]. StarSkirmish tests something else: whether a coding agent can improve a program over an hour of practice and feedback. There are two versions so far: v0.1 (26 September 2026, ten models) and v0.2 (15 October 2026, nine flagship models from [[OpenAI]], [[Anthropic]], Google and Meta). ## How it works - **The task**: write a Protoss bot in C++ against BWAPI 4.4.0 (the Brood War API, around since 2009), played on OpenBW, an open-source Brood War engine. Protoss vs Protoss only, on three two-player ladder maps: Heartbreak Ridge, Benzene and Destination - **The starting point**: a skeleton bot that only mines, the source code of three demo bots, and two standard bot libraries: FAP (a combat simulator) and BWEM (a map analyzer) - **The tools**: compile the bot, play practice games, and read each game's summary, transcript and the bot's own log. Practice opponents come in tiers: D (the three demo bots), C and B (three mid-ranked human-written bots each), A (BananaBrain, Locutus) and S (Stardust, PurpleWave). The model can practice against the exact bots it's graded against, but it never sees their source code (McPheeters on [[Hacker News]]) - **No internet**: web search and web fetch are off. There's no submit tool either. The harness grabs the bot folder when the hour ends, so the model has to keep its bot compiling the whole time - **The games**: a game ends when one side's buildings are all destroyed. Games that reach the 60-minute cap (86,400 frames) go to the higher kills + razings score, as on the BASIL ladder (the 24/7 Brood War bot ladder) - **The tournament (v0.2)**: five runs per model. Each run's final bot plays the 13 roster bots (10 competitive human-written bots and 3 demo bots) 6 times on each map. That's 18 games per pair, 234 per bot and 10,530 games in total - **The score (v0.2)**: a bot's win rate from 0 to 100. A model's score is the average over its five bots Why Protoss? McPheeters says Protoss has the most mature human-written bot scene (much of the BASIL top 10), and mirror matches keep the problem small enough for one hour. ### What changed from v0.1 to v0.2 | | v0.1 (26 September 2026) | v0.2 (15 October 2026) | |---|---|---| | Models | 10 from 8 labs, including GLM 5.3, Grok 4.7, MiMo V2.6 Pro and [[DeepSeek V4.1 Flash]] | 9 flagships from OpenAI, Anthropic, Google and Meta | | Harness | One neutral third-party harness for every model: Inspect's ReAct deepagent (bash, a text editor, a memories tool, research subagents), models called through OpenRouter with first-party inference only | Each lab's own CLI, at its mid-October 2026 version: [[Claude Code]], [[Codex CLI\|Codex]], [[Antigravity CLI]], Muse Code | | Reasoning effort | High for every model | Max, except Gemini 4 Argon at high (the highest it supports) | | Tournament | Full round robin of 62 entrants (50 LLM bots, 9 human-written bots, 3 demo bots); LLM bots play each other | Each LLM bot plays only the fixed roster of 13; LLM bots never meet | | Scoring | Elo ratings fitted from all games at once, scaled so Stardust scores 100 and Four Gate Dragoon (the weakest demo bot) scores 0 | Plain win rate against the roster | | Tier S | Stardust | Stardust and PurpleWave | | Cost basis | OpenRouter list prices, assuming 70% of input tokens are cache reads | Each lab's API list prices | Every v0.2 harness compacts the context at about 250K tokens. Codex, for example, gives the GPT models a 272K token window and compresses at 90%. The fixed roster is the most useful change. A bot's games no longer depend on which other LLMs are in the field, so a new model can be scored without replaying everyone, and scores from different model generations stay comparable. Elo fitted inside one field can't do that. The harness change cuts the other way: v0.2 ranks a model and its CLI together. For the record, v0.1 had [[GPT-6 Astra]] (Elo 1992) and [[Claude Opus 5.5]] (1979) functionally tied at the top, GPT-6 Sol (1912) third, then a big gap to GLM 5.3 (1430) and Gemini 3.8 Flash (1404). The bottom five: [[GPT-6 Luna]] (1312), Grok 4.7 (1301), Muse Spark 1.3 (1193), MiMo V2.6 Pro (1160) and DeepSeek V4.1 Flash (897). ## v0.2 results Tournament of 14 October 2026: | Rank | Model | CLI | Score (0 to 100) | Average API cost per 1-hour run | |---|---|---|---|---| | 1 | [[GPT-6 Astra]] | Codex | 57.3 | $18.07 | | 2 | [[GPT-6.1 Sol]] | Codex | 45.8 | $2.02 | | 3 | [[Gemini 4 Argon]] | Antigravity CLI | 39.3 | $3.63 | | 4 | [[Claude Opus 5.5]] | Claude Code | 38.6 | $5.12 | | 5 | [[Claude Fable 5.1]] | Claude Code | 37.2 | $7.78 | | 6 | [[Claude Sonnet 5.5]] | Claude Code | 25.8 | $2.38 | | 7 | Muse Wonder Full (Meta) | Muse Code | 21.4 | $5.87 | | 8 | [[Claude Haiku 5.5]] | Claude Code | 21.0 | $0.98 | | 9 | [[GPT-6 Luna]] | Codex | 13.1 | $0.57 | Muse Wonder Full's cost uses Muse Wonder's API pricing. - Astra is a clear step ahead of Sol. Argon, Opus 5.5 and Fable 5.1 are "functionally tied" behind them, in McPheeters' words - Sol is the value pick: second place at just over a tenth of Astra's cost per run. It's the only model in the chart's "attractive quadrant" (cheaper than the median model and scoring 40 or more) - Argon and Muse Wonder Full are new. They replace v0.1's Gemini 3.8 Flash and Muse Spark 1.3 ### Still far from the top bots The best human-written bots are well ahead of every model. Here's each model's record against the three strongest, as wins out of games over its five bots: | Model | Stardust | PurpleWave | BananaBrain | |---|---|---|---| | GPT-6 Astra | 6/90 | 9/90 | 4/90 | | GPT-6.1 Sol | 0/90 | 4/90 | 4/90 | | Gemini 4 Argon | 0/90 | 5/88 | 0/90 | | Claude Opus 5.5 | 0/90 | 1/90 | 0/90 | | Claude Fable 5.1 | 0/90 | 1/90 | 0/90 | | Claude Sonnet 5.5 | 0/90 | 4/90 | 0/90 | | Muse Wonder Full | 0/90 | 2/87 | 0/90 | | Claude Haiku 5.5 | 0/90 | 4/90 | 2/90 | | GPT-6 Luna | 0/90 | 0/90 | 0/90 | Astra, the best model, wins about 7% of those games. And McPheeters points out that real bot tournaments are harder than this setup, which uses only three two-player maps. No sign of saturation yet. ## Practice and run-to-run variance The agent traces show how each run spent its hour. According to McPheeters, the strongest predictor of a model's score is how much it practiced against the 7 strongest roster bots (TscmooP2, Steamhammer, McRave, BananaBrain, Locutus, Stardust and PurpleWave). | Model | Median practice games vs the 7 strongest bots | Score | Five runs, lowest to highest | Spread | |---|---|---|---|---| | GPT-6 Astra | 159 | 57.3 | 52.6 to 62.0 | 9.4 | | GPT-6.1 Sol | 87.5 | 45.8 | 38.5 to 51.3 | 12.8 | | Gemini 4 Argon | 54 | 39.3 | 32.5 to 50.0 | 17.5 | | Claude Opus 5.5 | 18 | 38.6 | 35.0 to 42.7 | 7.7 | | Claude Fable 5.1 | 0 | 37.2 | 33.8 to 41.5 | 7.7 | | Claude Sonnet 5.5 | 22 | 25.8 | 12.8 to 32.9 | 20.1 | | Muse Wonder Full | 2 | 21.4 | 18.2 to 26.1 | 7.9 | | Claude Haiku 5.5 | 0 | 21.0 | 16.2 to 31.6 | 15.4 | | GPT-6 Luna | 0 | 13.1 | 6.4 to 19.2 | 12.8 | The spread column is my own arithmetic on the published run data. - Astra, Sol and Argon start playing the strong bots early in the hour, while their own bots still lose to them. Muse Wonder Full, Haiku 5.5 and Luna rarely or never do - Fable 5.1 is the outlier. It scores 37.2 while practicing almost entirely against weaker bots (median of 0 games against the strong ones) - Argon varies far more than the other labs' flagships: 18 points between its best and worst run, against 9 for Astra and 8 for Opus 5.5. Sonnet 5.5 spreads even wider (20 points), though the write-up doesn't call it out - Argon preferred writing its bots in one shot: its five runs had the fewest builds and code versions of all the traced runs. 41% of its practice games were against high-tier opponents (second only to Astra), yet its bots won only 7% of their games against those tiers, against 28% for Astra - Argon's bots won 5 of their 268 games against Stardust, PurpleWave and BananaBrain, all five against PurpleWave. Its typical strategy: Dragoons and Zealots held at home behind cannons for one big push. Its second-best bot builds Reavers in over 70% of its games - Muse Wonder Full practiced almost only against the easiest opponents (1% of its practice matches at tier B or harder), just like Muse Spark 1.3 in v0.1 Five runs per model isn't a lot. With spreads of 8 to 20 points, the 0.7-point gap between Argon and Opus 5.5 means nothing, which is why "functionally tied" is the right reading. ## Benchmark correlations Spearman ρ (a rank correlation, from -1 to 1) between the StarSkirmish score and public benchmarks, over the models each benchmark scores: | Benchmark | v0.1: ρ (models) | v0.2: ρ (models) | |---|---|---| | [[Artificial Analysis]] Intelligence Index | not reported | 0.26 (9) | | Terminal-Bench 4.0 (Artificial Analysis) | 0.68 (9) | 0.49 (9) | | Terminal-Bench (Vals) | 0.71 (10, version 2.1) | 0.42 (9, version 4.0) | | Vals IOI | 0.90 (10) | 0.95 (9) | | WeirdML v3 | 0.94 (6) | 1.0 (7) | - In v0.2, StarSkirmish barely moves with the Artificial Analysis Intelligence Index. Astra, the top model here, only ties for 4th of these nine models on the Index (53, level with Fable 5.1 and Argon; Opus 5.5 leads with 58) - It drifted away from Terminal-Bench 4.0, which it tracked closely in v0.1. Sonnet 5.5 and Muse Wonder Full rank near the top of Terminal-Bench 4.0 and near the bottom here - It still tracks Vals IOI (olympiad programming problems solved in C++) and WeirdML v3 (an independent benchmark of unusual machine learning coding tasks). Argon and Astra both score 100 on Vals IOI. WeirdML v3 hasn't scored Argon or Muse Wonder Full yet - McPheeters' reading: the most heavily reported evals have become targets that models get tuned for, while less targeted tests of long, difficult coding tasks still move together. That's [[Goodhart's Law]] applied to AI benchmarks. Back in v0.1 he already described WeirdML as "thought to be less benchmaxed than most" - An independent echo from the HN thread: GodelNumbering noted that the v0.2 order for six models (Astra > Sol > Argon > Opus 5.5 > Fable 5.1 > Sonnet 5.5) matches a friend's GoBench ranking exactly (9x9 Go, with KataGo opponents as Elo anchors) ## Hacker News critiques McPheeters (HN user `__cayenne__`) posted v0.2 on 16 October 2026 as "Show HN: Gemini 4 Argon plays StarCraft". The thread had 254 points and 111 comments when I checked on 21 October. The title oversells it: Argon is one new entrant in a nine-model refresh, and no model plays the game live. - **Training-data recall.** gritzko argued that StarCraft "probably has hundreds of bot implementations lying on GitHub", which makes the bench "a recall-from-the-training-set competition". chis said performance depends on how closely the models memorized the best open-source bots. liuliu added that if the human-written bots are in the training data, you'd expect at least 50% against them, so the win rates look miscalibrated - **The pushback on recall.** boxed: a 7% win rate against bots they should be copying from memory means the models are "failing extremely hard", because "a copy paste command should do 50%!". dgant (Dan Gant, author of PurpleWave) said top bots are around 50K lines of code, too much for any LLM to memorize, and the fine-tuning lost in compression is most of the work anyway. McPheeters said the Brood War AI community raised the same concern, that the models write code that doesn't resemble bots in the training data, and pointed to the open-sourced starskirmish-hillclimb repo as evidence - **What does it measure?** iugtmkbdfil834 said too many factors are mixed together (code generation speed, logic, knowledge of the game platform, strategy) to tell what it measures. chis called the one-hour limit "a bit incoherent". robotresearcher replied that end-to-end measures work like that by design (company revenue, fuel efficiency, life expectancy), and iugtmkbdfil834 withdrew the objection - **The harness.** djoldman: "harness is HUGE". The same model can change dramatically across harnesses, allowed tools (with no web access, how does the model look up documentation?) and thinking levels. krzyk asked for a comparison of first-party CLIs against pi or opencode - **Reasoning effort.** TomGarden remembered Opus 5.5 doing worse at max on many benchmarks and asked for a run one notch down. oh_no said later numbers smoothed that out, but that under a clock "lower reasoning = more iteration = better code" is plausible - **The format rewards variance.** dgant, writing about McPheeters' longer hillclimb runs: unlimited attempts at each opponent tier reward variance-maximizing strategies like a proxy 2-Gateway Zealot rush. Turn each game into a coin flip, take enough flips, and you clear a tier with a poor overall win rate. He thinks that's "worth addressing in the future". He also noted that the benchmark bots mostly can't store data between games, so Stardust ran one conservative strategy almost every game, which weakens it. McPheeters himself caveated that the format advantaged the LLM bots compared to traditional tournament rules. The v0.2 score comes from a fixed tournament, but the practice hour still allows unlimited retries - **Real-time play.** MichaelNolan pointed to BroodBench, where LLMs play Brood War directly. LarsDu88's summary: "they suck and many default to doing probe rushes". devmor added that frontier models are too slow for a game that needs several informed decisions per second - **Generalization.** carychang wondered how much of the progress comes from better coding versus better experiments, and how the strategies would hold up on unseen maps and opponents - **Counter-anecdotes.** t1amat found Opus 5.5 far better than Sol and Astra at building bots for a ModernUO server (pulled the GPT models after a week, Opus rebuilt it in a night). dudeinhawaii finds Astra more adaptive than Opus 5.5 and OpenAI models "particularly adept at finding the meta game and exploiting it" ## The hillclimb runs Before v0.2, McPheeters ran StarSkirmish Hillclimb: a longer contest (started 2 October 2026) where GPT-6 Astra and Claude Opus 5.5 each tried to clear the opponent tiers one by one, over about two days instead of one hour. Per the repo: - Astra cleared all five tiers, tier S on its 29th submission at that tier, 40.8 hours in (McPheeters said 43 hours on HN) - Opus 5.5 cleared D, C and B, but not A: its best tier A submission won 27 of 60 games, 47.5 hours in - Afterwards, Astra went 0 for 1,000 against Pluto, a bot trained only by self-play [[Reinforcement Learning (RL)]], on a livestream with up to 30 games running at once Gant also described Astra's path in detail. Astra settled on one-base Dragoon + Observer play, switched to a proxy Zealot rush when the A-tier bots out-executed it, then used Dark Templar to secure an expansion and a two-base Dragoon + High Templar push. It turned out both PurpleWave and Stardust had a broken Psionic Storm dodge at the same time, for unrelated reasons, and Astra's storms exploited it. His conclusion: Astra won by simplifying the game, and "winning StarCraft has always been more about bland fundamentals than fancy tricks". ## My take - I like the design. It scores the loop I actually care about in a coding agent: try, measure, fix, under a deadline. The fixed roster and plain win rates make it easy to read, and the exact numbers are published in a data file behind the charts - The v0.2 harness switch matches how these models get used in practice, but it means the ranking can't separate the model from its CLI. The drift away from Terminal-Bench also happened at the same time as the harness, reasoning and model-set changes. So I read the Goodhart argument as plausible, not proven. With nine models (seven for WeirdML), a single rank swap moves ρ a lot - Treat gaps under about 5 points as ties. Five runs per model with spreads up to 20 points can't do better - The practice finding is the lesson I'd keep: the models that went looking for losses early won. Fable 5.1 is the one exception, and the write-up doesn't explain it - For [[Gemini 4 Argon]], it's the second independent result after Artificial Analysis: third, tied with Opus 5.5 and Fable 5.1, cheaper per run than both, and the least consistent of the flagships ## References - StarSkirmish Bench v0.2 (Kai McPheeters, 15 October 2026): https://starskirmish.com/bench/v0-2/ - StarSkirmish Bench v0.2 data file (scores, costs, practice, runs, correlations): https://starskirmish.com/bench/v0-2/data.js - StarSkirmish Bench v0.1 (Kai McPheeters, 26 September 2026): https://starskirmish.com/bench/v0-1/ - StarSkirmish hillclimb repo (tier-clearing bot versions from GPT-6 Astra and Claude Opus 5.5): https://github.com/llmskirmish/starskirmish-hillclimb - Hacker News discussion (Show HN: Gemini 4 Argon plays StarCraft): https://news.ycombinator.com/item?id=50118688 - Dan Gant on the hillclimb run (X): https://x.com/dgant/status/2106880240907620731 - GoBench (Roland Gao): https://rolandgao.com/blog/gobench/ - BroodBench: https://broodbench.com/ - BWAPI: https://github.com/bwapi/bwapi - BASIL ladder: https://www.basil-ladder.net/ ## Related - [[Gemini 4 Argon]] - [[GPT-6 Astra]] - [[GPT-6.1 Sol]] - [[GPT-6 Luna]] - [[Claude Opus 5.5]] - [[Claude Fable 5.1]] - [[Claude Sonnet 5.5]] - [[Claude Haiku 5.5]] - [[DeepSeek V4.1 Flash]] - [[Claude Code]] - [[Codex CLI]] - [[Antigravity CLI]] - [[AI Agent Harness]] - [[AI Evaluation]] - [[SWE-Bench]] - [[Artificial Analysis]] - [[Goodhart's Law]] - [[Reinforcement Learning (RL)]] - [[Hacker News]]