I’ve found various recent complex-strategy-game AI efforts very interesting, but always have one key complaint: They don’t properly ground the mechanical execution of their AI to realistic human levels for comparison. And if you give the AI an unfair mechanical advantage, your model isn't going to have to learn nearly as good of strategy. I will say however - this is the closest to realistic I have seen - but is still lacking.
The two main measurable parameters of performance are:
1 - reaction time
2 - rate/volume of actions (i.e. Actions Per Minute)
And I would argue some there should be an additional consideration of some form of:
3 - mouse-click accuracy
I read through the details of the implementation, and they did decent at 1, 2 but overall need to do better.
Their reaction times end up as a random draw between 170-270ms. I think raw, simple visual reaction times for a pro gamer could be ~200ms, BUT that’s just for a simple “click this button when the light changes” type of tests. There are “complex reaction time” tests where you sometimes click, but other times don’t (eg a red or green light), and reaction times in that case are around ~400ms. I think if a pro is in a game situation where they anticipate their opponent will take some action and are ready to immediately respond, 200ms is a fair reaction time. But that’s not the usual state through a game, and the bot effectively has that perfect anticipation mindset at all times. So not crazy, superhuman reactions, but definitely not completely realistic/fair either.
In regard to action rate, they allow the model to take 1 action every 7.5 ms - which translates to 450 APM. The very best pro gamers are in the 300-350 APM range. And i think a humans actions include various thoughtless click spamming (which AI doesn’t need to do), as well as visual map movement/unit examination that an AI would not need as much of with a direct, comprehensive feed available information. So the sustained 450 APM seems pretty superhuman to me - BUT dota 2 is much less of a APM intensive game, and certainly sustained APM isn’t as important. And humans get get higher APM in important burst moments whereas this AI is at an exact fixed rate of 450 APM. So all-in-all, the APM is maybe fair (at least close to fair)
The mouse click accuracy piece, however is pretty unfair if the ai can make precise clicks across the screen with no affect to reaction time. This factor isn’t considered at all by the AI team. I feel they should either add in some randomization to simulate inaccuracy, or cause delayed reaction time based on how far the mouse would have to move.
With all these factors combined - I still feel this is not quite a fair test. But it’s closer than other’s I’ve seen, and it’s still a very impressive overall achievement! I’d love to see them go the small extra distance of constraining these mechanical performance parameters just a bit more. I feel that would make a BIG difference in the level of strategy required to beat the best humans. They’re SOOO close to amazing me!
The two main measurable parameters of performance are: 1 - reaction time 2 - rate/volume of actions (i.e. Actions Per Minute) And I would argue some there should be an additional consideration of some form of: 3 - mouse-click accuracy
I read through the details of the implementation, and they did decent at 1, 2 but overall need to do better.
Their reaction times end up as a random draw between 170-270ms. I think raw, simple visual reaction times for a pro gamer could be ~200ms, BUT that’s just for a simple “click this button when the light changes” type of tests. There are “complex reaction time” tests where you sometimes click, but other times don’t (eg a red or green light), and reaction times in that case are around ~400ms. I think if a pro is in a game situation where they anticipate their opponent will take some action and are ready to immediately respond, 200ms is a fair reaction time. But that’s not the usual state through a game, and the bot effectively has that perfect anticipation mindset at all times. So not crazy, superhuman reactions, but definitely not completely realistic/fair either.
In regard to action rate, they allow the model to take 1 action every 7.5 ms - which translates to 450 APM. The very best pro gamers are in the 300-350 APM range. And i think a humans actions include various thoughtless click spamming (which AI doesn’t need to do), as well as visual map movement/unit examination that an AI would not need as much of with a direct, comprehensive feed available information. So the sustained 450 APM seems pretty superhuman to me - BUT dota 2 is much less of a APM intensive game, and certainly sustained APM isn’t as important. And humans get get higher APM in important burst moments whereas this AI is at an exact fixed rate of 450 APM. So all-in-all, the APM is maybe fair (at least close to fair)
The mouse click accuracy piece, however is pretty unfair if the ai can make precise clicks across the screen with no affect to reaction time. This factor isn’t considered at all by the AI team. I feel they should either add in some randomization to simulate inaccuracy, or cause delayed reaction time based on how far the mouse would have to move.
With all these factors combined - I still feel this is not quite a fair test. But it’s closer than other’s I’ve seen, and it’s still a very impressive overall achievement! I’d love to see them go the small extra distance of constraining these mechanical performance parameters just a bit more. I feel that would make a BIG difference in the level of strategy required to beat the best humans. They’re SOOO close to amazing me!