TL;DR: A couple of programmers wanted to see if they could train a generative adversarial network (GAN) to create its own version of Grand Theft Auto 5. The result was a somewhat blurry acid trip but is instantly recognizable as GTA5.

Harrison Kinsley, who goes by Sentdex on YouTube, and his partner trained a fork of Nvidia's GameGAN neural network using a black car on a short section of highway in Grand Theft Auto 5. Nvidia loaned them its DGX Station, which is equipped with four A100 80-gigabyte cards to help with the processing. Kinsley explains what they did and demos the results in the video above.

Initial models were very pixelated, but Kinsley improved this with AI-assisted supersampling. While it's not pretty, it's important to keep in mind that this is not GAN-generated footage. It is a real-time interactive demo. Kinsley is driving an AI-created car in a fully AI-created environment.


Before and after upsampling.

Kinsley said that since the DGX Station was on loan, his training time was limited. Although the model does try to compute obstacle clipping in some instances, he would have liked to run more collision samples. Kinsley also wanted to see how much of the GTA5 map the GAN could process. However, that would have required hours more training as he incrementally increased the driving distance, which he did not have time to do.

The video shows the neural network did a pretty decent job recreating some unexpected details. For efficiency's sake, one might think that the GAN would ignore shadows and the sun reflecting off the car. To Kinsley's surprise, it didn't. Shadows, lighting, and reflections move more or less as expected. The GAN also created its own elementary physics system after training it by running into things. An example is how another car skews to the right when tapped on the left rear quarter panel. Head-on collisions are not handled as well.

Kinsley has posted the playable demo dubbed "GAN Theft Auto" to GitHub for those interested in giving it a spin. Again, it's just a tiny portion of the GTA map, and it's not like playing a real game. It's more tech demo than anything else, but it is interesting to watch what the model does in untrained situations. That's when things get a little psychedelic.