Self-driving AI is teaching itself to drive in GTA V live on Twitch

Cal Jeffrey

Posts: 2,587   +600
Staff member

As autonomous vehicles continue to become a “thing,” software developers will continue to program the AI that goes into such vehicles. The problem with working on AI driving software is having a practical way of testing it. While some are seeking open road testing, some developers opt to test in virtual environments.

One such developer is Harrison Kinsley. Known as Sentdex on Twitch, he has programmed (in Python) a self-driving AI and has unleashed it in Grand Theft Auto V. Kinsley is not the first developer to use the game for this purpose, but he is the first to stream the training session live 24/7 on Twitch (see below).

The AI is named Charles, and if you did not know what you were watching, you would swear that it was a homicidal 13-year-old behind the wheel. Charles drives like a player hellbent on causing as much destruction and chaos as possible, which is not an unusual play style for GTA V. Pedestrians are not an obstacle to Charles; they are a target. The AI is also fond of plowing into police cars, driving on the sidewalk, repeatedly smashing into unbreakable walls, and flinging the car into the ocean.

According to Digital Trends, the software “uses pixel-level-analysis of on-screen information to make decisions about how to drive.”

Unfortunately, Charles was programmed without a memory, so not much learning is going on at the moment. Kinsley eventually plans to give Charles a memory. At that point, the AI will use what Sentdex calls “deep learning” to improve its driving skills.

The way it works is, Charles looks at pixels on a frame-by-frame basis. At each frame it decides what to do; accelerate, brake, turn, etc. With no memory, the AI has no other information to base decisions on other than what it sees on the screen for that single frame. From a single frame, Charles is unable to tell things like speed, the direction of travel, and other factors that play into the decision-making process while driving.

Sentdex maintains that even with a short-term memory of only a few frames programmed in, Charles would be able to “know” if it was going forward or backward and at what speed. Long-term memory (again, once programmed) would “remind” Charles how to respond to what the short-term memory is telling it.

It is an engaging and sometimes funny stream to watch as Charles navigates through Los Santos, but it should become even more engrossing as the software blossoms and becomes more advanced.

Permalink to story.



I can't wait for them to put this AI in a real car and have it lead police on a high speed chase only to vanish without a trace by parking behind some bushes and waiting for three minutes.
  • Like
Reactions: mbrowne5061


Posts: 1,639   +928
Wouldn't be too hard for R* to tell them where to find those vectors in RAM.
Then you'd have simulated sensor input.
for a second I thought you were talking about R, the statistics/mathematics/AI/ML computer language, and was confused/interested how R would him find specific vectors inside of RAM.


Posts: 14   +1
How great it would be if NASCAR got involved with driverless racing. There is no more controlled driving environment than a NASCAR high speed tri-oval and the parameters are well understood. An AI system tied into the car's telemetry system making sub-millisecond decisions could be safe as well as fast and help advance the state of the art. It would be so cool it the automated drivers could beat the meat drivers.