Rendered at 23:24:29 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
pelagicAustral 3 hours ago [-]
Unrelated to the benchmark...
I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...
I miss those days so much.
Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.
nemo1618 3 hours ago [-]
Even at Burning Man, in the middle of the desert, there is a camp that hosts a StarCraft tournament every year (on the dustiest setups you've ever seen!) :)
jpgvm 1 hours ago [-]
I hope it's called Mar Sara.
oddsockmachine 1 hours ago [-]
That would be cool. It's Captain Pump's Raiders
PorciiVorbesc 2 hours ago [-]
So ... Zerging Man?
benswerd 3 hours ago [-]
So dope
snicky 2 hours ago [-]
Oh boy, I've spent more hours playing it than I dare to admit. I won over 10000 battle.net games ... on just one of my several accounts :) When the SC2 beta came out, I played about 50-100 games and never bought the full game, because I knew it would be like heroin to me, and I was already an adult that had to take care of himself.
layoric 1 hours ago [-]
Same and Brood War is starting to have a bit of a resurgence. Not anything huge. When Battle.net servers are actually working, tend to play on the ladder a few times a week. Get crushed, but still one of the greatest competitive games ever made.
benswerd 3 hours ago [-]
I agree.
My first time playing StarCraft was at summer camp around a decade after it came out.
All the smartest people played it so I wanted to too. Great decision, I have been continually impressed with the people who StarCraft introduced me to.
gary17the 2 hours ago [-]
My best greetings to all Starcraft Elders clanners of yesteryear :).
stavros 1 hours ago [-]
Yes! Being young is great, and it always seems like those days were the best.
sidewndr46 3 hours ago [-]
Later we even listened to Eminem and played violent video games. Most of have never even been charged with a crime, much less abused someone.
mcteamster 17 minutes ago [-]
I love this. Funnily enough StarCraft has influenced how I approach AI at a meta level
Protoss: powerful and expensive frontier coding agents you directly micromanage for the toughest tasks
Terran: versatile team comps of dedicated agent roles you can delegate well-defined tasks to
Zerg: massive swarms of specialist custom agents inside your apps that you evolve and optimise for speed and cost
Knowing every faction has its strengths and weaknesses helps me decide which tools to use for the job.
AntiRush 2 hours ago [-]
Back in 2010, during the early days of bwapi, there was a Brood War AI tournament held by the Expressive Intelligence Studio at UC Santa Cruz. It's interesting to see how different the approaches were back then, vs this or Deepmind's SC2 work.
A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on
A better benchmark might be asking the LLMs to write GO ai and then comparing that-- the issue is that there will be a HUGE difference in performance that depends purely on this game being in the LLM's training... but training a general LLM to directly play these games would be a waste of capacity and shouldn't be encouraged for benchmaxxing sake.
Programming an engine OTOH is a skill that is more general and they should all have.
Might be useful to have the target of the engine be some specific virtual machine that gets a strict cycle budget-- e.g. execution runs so many cycles, and result is read out of a specific memory address at the end (or when it terminates early).
ForHackernews 1 hours ago [-]
Excuse me? Is this thing supposed to be a borderline SGI or not? We already know LLMs are good at spitting out code.
nullc 1 hours ago [-]
Playing these games autoregressively isn't even the right way to use the LLM for this task (unless it was trained to do so...). It's somewhat like having a creative writing bechmark but requiring that all the input/output be base64 encoded. It can do it-- but no guarantees on the results!
And it's also just bencmaxxing bait: you can get a huge improvement on the task by RLing on it, but make no improvement on anything else. Doing so would just waste model capacity.
If you could tell that every LLM was equally not being exposed to the task then you could justify it as a test of abstract reasoning, but you can't. So it ends up on how much go transcripts ended up in the training, which is ... not a very interesting metric.
faeyanpiraat 3 hours ago [-]
There is currently a bot beating everyone on the ladder. Just watched it today on Artosiscasts yt channel.
If you read the TL thread on this, the bot is technically running at 240 APM, but its commands get split off into separate commands for each unit, so the APM looks higher.
I've watched several replays on this. It using map hacks is lame but it also doesn't always take advantage of them. It sometimes does respect its own fog of war. It mostly wins with incredible micro (kinda has to, it's macro kinda sucks).
Most of its few losses come from drops (it never makes turrets and doesn't know how to handle them), bad macro (blocking its own ramp) or just incredibly unconventional play from its opponent (which is certainly not in its training set).
chrishare 24 minutes ago [-]
Specifically, it can micro across multiple screens of engagement in a way that we can't as easily.
readams 1 hours ago [-]
Bots have been unable to beat even pretty good amateur humans despite those advantages so it's still pretty interesting.
iammjm 2 hours ago [-]
It is in fact APM limited, and its creator claims it was trained without the map hack so it should do just as well without it. I think the bot is VERY impressive and has already managed to beat several top Korean pros which is really godlike
y-curious 1 hours ago [-]
I’m a little bummed it did have map hack on. Would love to see this play “legit”. It really reminds me of my pastime, chess. When you play bots, if you don’t mess up early, it can often feel like you’re winning in the early game even though you’re just hanging yourself slowly with the rope it’s giving you.
cindyllm 32 minutes ago [-]
[dead]
tweakimp 2 hours ago [-]
If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net/
gadtfly 3 hours ago [-]
Did it play by looking at screenshots and sending clicks, or was there other mediation/symbolization?
It sounds like it might have been actually played in real time, which would be very important to distinguish.
I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into something turn-based https://www.youtube.com/watch?v=ruuGXFAmiOE
callmekit 23 minutes ago [-]
It plays from a special API. Some things that are impossible with normal UI are possible with API, like selecting a unit under other units. Invisible units are also reported via the API.
loeg 2 hours ago [-]
There's a bot data stream already; it's probably hooked up to that rather than screencap. Yes, I believe these were playing in real time.
therealdrag0 24 minutes ago [-]
Would Jev be good for this?
aetherspawn 22 minutes ago [-]
It can play doom in realtime so maybe it can play this.
minimal_action 2 hours ago [-]
I think we're on the early days of games you connect with your agent to. Human + AI units one versus the other. Like knights with their horses. Not sure which is the horse..
WillMorr 2 hours ago [-]
I've been running these with friends recently, it's very fun. I ran irl bot tournaments for board games a couple times but the agent era opens up a huge amount of possibilities.
Must recently I built out a MMORPG puzzle box thing, I wrote a general game architecture doc but left the specific puzzle design up to Fable. Nobody is actively playing rn but I left it up at bot.willmorrison.net.
malfist 3 hours ago [-]
This is a great idea for a benchmark. Something all the benchmarks seem to be missing is strategy, tactical solutions in most of the benchmarks are all thats required but here requires actual long term thinking and tactical thinking, balancing and orchestration.
Any details about the harness the agents were given? I am curious what representation of the screen and world state was provided to the agents and what tools they had available.
benswerd 3 hours ago [-]
Oh sorry I should be more clear on that. Will add to report.
For agent harness I did Claude Code, Codex, Grok Build. This was primarily a cost driven decision — I have a lot of free tokens and I didn't want to pay API prices for this.
For game harness I used minimal BW-API issue command and get observation apis as tools. I felt this was the most fair way to do it on my small scale.
In the future I would like to integrate code mode and multiple games/I think if it was a best of 5 where each agent could learn from its past games and build its own automations over time that would be much more interesting.
moomin 1 hours ago [-]
Wondering what it would look like if you allowed them to write scripts. You could throttle the number of clicks to make it interesting.
mococa 1 hours ago [-]
+1 because StarCraft
c7b 2 hours ago [-]
At last something that feels properly orthogonal to pelicans on bicycles.
benswerd 9 hours ago [-]
+ Playable Agent driven Starcraft
conorcleary 3 hours ago [-]
Dibs on the fly brain
bee_rider 3 hours ago [-]
A ton of conversations about the game must be in the training set. I wonder, is there any way just from watching how they play, of telling if they tend to pick strategies that people complain or meme about online?
tekla 2 hours ago [-]
At high levels of play Zerg is generally considered significantly stronger than the other races. So I generally will assume AI will tend to pick Zerg
rrr_oh_man 2 hours ago [-]
I only played the StarCraft demo two decades ago. Why is that so? Rushing?
xmcp123 2 hours ago [-]
Unless you Protoss and mind control the Zerg, and then are Zerg+Protoss.
duskwuff 16 minutes ago [-]
This is not a viable strategy in competitive play. It's a huge resource/time investment which is unlikely to pay off.
stackghost 3 hours ago [-]
I wonder if there is a library to decipher brood war replay files. Perhaps an agent could learn by watching.
frutiger 3 hours ago [-]
I haven’t checked for SC:BW but Blizzard has official parsers/replayers for SC2 on GitHub.
benswerd 3 hours ago [-]
Not hard to build. I was shocked at how fast/easy this was to pull together.
winwang 3 hours ago [-]
Would be interesting if you could team a fast and slow agent together -- slow model can either act directly or maybe just communicate to the fast model.
benswerd 3 hours ago [-]
I might open this up to a tournament if enough people want. Any interest?
3 hours ago [-]
chaostheory 2 hours ago [-]
Gemini wasn’t included, but I’m guessing its performance would have been similar to Grok’s performance despite having roots in DeepMind.
dschuessler 3 hours ago [-]
Somewhat related: In 2018, Google DeepMind had already created AIs that were capable of beating professional gamers in StarCraft 2 (the sequel to Brood War): https://www.youtube.com/watch?v=cUTMhmVh1qs
qerghnui 2 hours ago [-]
AlphaStar beat one retired professional by cheating.
AlphaStar won a showmatch against TLO. TLO was never one of the strongest players in the world. He had been retired for over three years by the time of the match. Google set the rule that their system would have human-like mechanics, but it played several times faster than any human, never issued a wasted action, had an inhumanly fast reaction time, issued commands with perfect accuracy using an API, and could see the entire map at once.
It was later released to the open ladder with more human-level mechanics. Even strong amateurs regularly beat it. I have beaten it myself. It was strong, but not even close to the level of the strongest human players. It had obvious and easily-exploitable deficiencies in strategy and building placement.
I think even the cheater version would have lost handily to Serral or any of the strongest players.
(It apparently beat MaNa as well as TLO, but those matches were never released to my knowledge. I see no reason to assume Google cheated less flagrantly in private than they did in public.)
benswerd 3 hours ago [-]
I predict LLMs will reach superhuman level and beat even that model in the next 12 months
orbital-decay 2 hours ago [-]
Starcraft is APM-dependent. Unless the latency will improve greatly in frontier reasoning LLMs (which is unlikely), it will remain a bit like knitting with an excavator.
benswerd 2 hours ago [-]
I predict latency will improve greatly in the next 12 months to more than 4x speed on current frontier tasks
loeg 2 hours ago [-]
Yeah but Starcraft needs, like, 10-20x the APM these agents are doing.
benswerd 2 hours ago [-]
I’m not convinced a lot of it can’t be solved with code mode.
Marine staggering for example seems like an ideal code mode task.
loeg 2 hours ago [-]
Yeah. Some of it may just be "thinking" less rather than faster token generation.
I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...
I miss those days so much.
Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.
My first time playing StarCraft was at summer camp around a decade after it came out.
All the smartest people played it so I wanted to too. Great decision, I have been continually impressed with the people who StarCraft introduced me to.
Protoss: powerful and expensive frontier coding agents you directly micromanage for the toughest tasks
Terran: versatile team comps of dedicated agent roles you can delegate well-defined tasks to
Zerg: massive swarms of specialist custom agents inside your apps that you evolve and optimise for speed and cost
Knowing every faction has its strengths and weaknesses helps me decide which tools to use for the job.
https://web.archive.org/web/20091124210529/http://eis.ucsc.e...
There's a great contemporary Ars Technica piece by a competitor:
https://arstechnica.com/gaming/2011/01/skynet-meets-the-swar...
As an undergrad I did a project using genetic programming. It was not very successful, but it was a lot of fun.
https://tomisin.space/archive/starcraft-genetic-programming/
[1] https://rolandgao.com/blog/gobench/
[2] https://rolandgao.com/gobench.pdf
Programming an engine OTOH is a skill that is more general and they should all have.
Might be useful to have the target of the engine be some specific virtual machine that gets a strict cycle budget-- e.g. execution runs so many cycles, and result is read out of a specific memory address at the end (or when it terminates early).
And it's also just bencmaxxing bait: you can get a huge improvement on the task by RLing on it, but make no improvement on anything else. Doing so would just waste model capacity.
If you could tell that every LLM was equally not being exposed to the task then you could justify it as a test of abstract reasoning, but you can't. So it ends up on how much go transcripts ended up in the training, which is ... not a very interesting metric.
I've watched several replays on this. It using map hacks is lame but it also doesn't always take advantage of them. It sometimes does respect its own fog of war. It mostly wins with incredible micro (kinda has to, it's macro kinda sucks).
Most of its few losses come from drops (it never makes turrets and doesn't know how to handle them), bad macro (blocking its own ramp) or just incredibly unconventional play from its opponent (which is certainly not in its training set).
It sounds like it might have been actually played in real time, which would be very important to distinguish.
I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into something turn-based https://www.youtube.com/watch?v=ruuGXFAmiOE
Must recently I built out a MMORPG puzzle box thing, I wrote a general game architecture doc but left the specific puzzle design up to Fable. Nobody is actively playing rn but I left it up at bot.willmorrison.net.
For agent harness I did Claude Code, Codex, Grok Build. This was primarily a cost driven decision — I have a lot of free tokens and I didn't want to pay API prices for this.
For game harness I used minimal BW-API issue command and get observation apis as tools. I felt this was the most fair way to do it on my small scale.
In the future I would like to integrate code mode and multiple games/I think if it was a best of 5 where each agent could learn from its past games and build its own automations over time that would be much more interesting.
AlphaStar won a showmatch against TLO. TLO was never one of the strongest players in the world. He had been retired for over three years by the time of the match. Google set the rule that their system would have human-like mechanics, but it played several times faster than any human, never issued a wasted action, had an inhumanly fast reaction time, issued commands with perfect accuracy using an API, and could see the entire map at once.
It was later released to the open ladder with more human-level mechanics. Even strong amateurs regularly beat it. I have beaten it myself. It was strong, but not even close to the level of the strongest human players. It had obvious and easily-exploitable deficiencies in strategy and building placement.
I think even the cheater version would have lost handily to Serral or any of the strongest players.
(It apparently beat MaNa as well as TLO, but those matches were never released to my knowledge. I see no reason to assume Google cheated less flagrantly in private than they did in public.)
Marine staggering for example seems like an ideal code mode task.