Bottom line: After handily defeating Dota 2 pros last year, OpenAI decided to allow amateur gamers to face off against its Dota playing bots (called ‘agents’) before retiring the team. The results were predictably lopsided.
In total, the AI team, called OpenAI Five, played 7,257 matches and only lost 42 for a win ratio of 99.4 percent. The agents won 4,075 games outright. Human players quit, admitting defeat, 3,140 times. OpenAI Five’s dominance against amateur players is not surprising.
Last summer the bots were pitted against a team of professional Dota players and defeated them 2-1 in a best of three series. Even though the AI did lose one match, it was constrained to a choice of only 18 of the 100 heroes and had its reaction time increased to 200ms from its usual 80ms.
More recently, OpenAI’s agents were tested not only against professionals but with The International 2018 champion, team OG. In the best-of-three match, OpenAI Five swept OG in the first two games making a third round unnecessary.
The Verge notes that at the time of the pairing between OpenAI Five and OG, the agents had already had the equivalent of 45,000 years worth of training.
“OpenAI Five is powered by deep reinforce learning, which means we didn’t code it how to play. We coded it how to learn,” said Greg Brockman, OpenAI’s CTO. “In its 10 months of existence, it’s already played 45,000 years of Dota 2 gameplay. That’s a lot — it hasn’t grown bored yet.”
The OpenAI bots can also play in cooperative mode with humans. Logically, the amateur players enjoyed playing with the agents, more than playing against them. The Verge notes that the cooperative version of the AI received far more attention with more than 35,000 games played.
For now, OpenAI Five is retired. The team will review the data accumulated from the matches and “will use Dota 2 as a research platform for further development of powerful AI systems.”