WTF?! Even as billions are poured into AI and its use becomes increasingly widespread, we're still seeing very worrying instances of the technology getting things wrong. A prime example happened at a high school when a student was swarmed by police after an AI thought the Doritos bag he was holding was a gun.
Taki Allen, a 16-year-old high school student at Kenwood High School in Baltimore County, Maryland, got an unwelcome surprise as he sat outside with friends after football practice while waiting to be picked up last week.
About 20 minutes after he finished the bag of Doritos he had been eating, around eight police cars showed up. Officers climbed out of the cars and, with their weapons pointing at Allen, told him to put his hands on his head and walk toward them. The student said he was then told to get on his knees and was placed in handcuffs. Baltimore County Police Department later confirmed that Allen was handcuffed but not arrested.
Unfortunately for Allen, the artificial intelligence that reviews the school's surveillance camera footage mistook the crumpled Doritos bag for a firearm and alerted the police.
"Do you have a gun on you?" one officer asked the boy as the others searched his friends.
After reviewing the footage that the AI had identified as a gun, police looked in a nearby trash can and found the bag of chips.
"I guess just the way you were eating chips … Doritos, whatever …. it picked it up as a gun," one officer said. So, remember: never eat chips the wrong way. Another cop added that "AI is not the best," which is an understatement.
School Superintendent Dr. Myriam Rogers told reporters that the system worked how it was meant to. "The program is based on human verification and in this case the program did what it was supposed to do which was to signal an alert and for humans to take a look to find out if there was cause for concern in that moment," Rogers said.
Understandably, Allen didn't share Rogers' point of view. "I don't think no chip bag should be mistaken for a gun at all."
School principal Kate Smith sent a letter to parents following the incident emphasizing the importance of student safety, but Allen said Smith never spoke to him until three days after it happened.
The mistake has led to calls from local politicians for the school's use of AI-based surveillance technology to be reviewed.
Allen said he now waits inside after football practice, as he does not think it is "safe enough to go outside, especially eating a bag of chips or drinking something."
Omnilert, the developer of the AI system used by the school, told BBC News, "We regret this incident occurred and wish to convey our concern to the student and the wider community affected by the events that followed."
"While the object was later determined not to be a firearm, the process functioned as intended: to prioritise safety and awareness through rapid human verification," it said.
