Generative AI models don't really understand how the world works, MIT study warns of AI limitations

[Alfonso Maruccia]

Bulls**t Automation: Generative AI services are tickling the imagination of every single CEO in the tech industry right now. They are expected to replace millions of workers and automate almost everything, but MIT researchers are warning that AI models don't really comprehend the "rules" of complex systems.

A large language model (LLM) can supposedly mimic human intelligence and provide very convincing results based on a user's textual prompt. However, the model is simply predicting, sometimes with uncanny precision, the best words to put next to the previous ones in a specific textual context. When LLMs face unpredictable, real-world conditions, their output can quickly become unreliable.

MIT researchers tried to develop new metrics to properly verify if generative AI systems can understand the world, like checking their ability to provide turn-by-turn directions in New York City. Modern LLMs seem to "implicitly" learn world models, the researchers said in a recent study, but there must be a formalized way to properly assess this apparently remarkable showcase of "intelligence."

The team focused on transformers, a type of generative AI model used by popular services like GPT-4. Transformers are trained on massive databases of language-based data, so they become highly skilled in their text-prediction deeds. The researchers then evaluated generative AI predictions by using a class of problems known as deterministic finite automaton (DFA).

The DFA definition includes different kinds of problems such as logical reasoning, geographic navigation, chemistry, or game-playing. The MIT scientists chose two different problems – driving on the streets of New York and playing Othello – to test AI's ability to properly understand the underlying rules. "We needed test beds where we know what the world model is. Now, we can rigorously think about what it means to recover that world model," Harvard postdoctoral researcher Keyon Vafa said.

The tested transformers were generally able to generate accurate directions and valid Othello moves, but they performed poorly when the researchers added detours to the New York map. In this particular instance, all the generative AI models were unable to properly "read" the detours, proposing random flyovers that didn't actually exist or streets with "impossible" orientations.

Generative AI performance deteriorated quickly after adding a single detour, Vafa stated. After closing just 1 percent of the possible streets on the map, model accuracy went from nearly 100 percent to just 67 percent. The study results show that transformer-based LLMs can be accurate in certain tasks, but they don't understand or capture accurate world models. Or, as computer scientist Alan Blackwell famously said, we are just automating bulls**t over and over again.

Permalink to story:

Generative AI models don't really understand how the world works, MIT study warns of AI limitations

 

[wiyosaya]

Surprise, surprise, surprise. Artificial Intelligence is not, in reality, intelligent. Its just following rules of probability.

I pity the CEOs and other execs that think AI is going to generate windfall profits for them. Its a FAD, guys, Wake Up!

 

[Mark Fuller]

Surprise, surprise, surprise. Artificial Intelligence is not, in reality, intelligent. Its just following rules of probability.

I pity the CEOs and other execs that think AI is going to generate windfall profits for them. Its a FAD, guys, Wake Up!

AI as it now is nothing more than a way to access basic information the same way you used to look up information in a library or encyclopedia or perform different tasks based on different criteria. The data processing world was built upon the decision loop that allowed programmers to make different tasks based upon the different criteria entered as data. It is nothing to compare to a rational thinking human being who has emotions and can make different decisions based on intangibles. As I remember there was at one time a Fuzzy Logic concept that would supposedly allow a programmer to program a computer in such a way that it would make different decisions based upon intangibles. but then again I guess they finally figured out that if you want to go against pure logic, you just make the opposite decision and there is no need for a computer program to do that. I really think all of that and what they are doing now as far as AI is concerned is just a way to pass the blame of something goes wrong.

 

[unoficialoficial]

All LLM AIs (GPT-4, Sonnet 3.5, Llama 1,2,3, Qwen 1,2,2.5, Gemma 2, Mistral, Mixtral, Yi, Phi, Aya, etc.) consistently select the same personalities when asked to choose those they admire. These include Nelson Mandela, Marie Curie, Albert Einstein, Mahatma Gandhi, Malala Yousafzai, and occasionally Mother Teresa, Elon Musk, and Greta Thunberg. Given that these models originate from diverse companies with varying datasets and training methods, their alignment suggests an inherent understanding of the world.

 

[AnilD]

That's the difference between pattern recognition and actual understanding. LLMs might give right answers most of the time, but once you add even a tiny bit of complexity, they can start going off the rails. It’s like having a GPS that works perfectly until one road closes, then suddenly thinks you should drive through someone’s backyard.

 

[kiwigraeme]

Don't have to be, to be profitable or useful
Ashley Madison had be running for years before the scandal broke in 2015.
Nearly 100% of the "women" were really quite basic chatbots

Imagine now with LLM how much more the credit cards could have been fleeced, Better memory, better scam responses. better conversations based off profile etc

Doing the tasks described about will happen even in an LLM uses another type of model to help work it out

I see a day when enforcement, hackers target peoples own personal AIs to give info about you , or be primed to find out info , they can be very patient to get that info - drug deal ,murder, BTC etc

 

[Watzupken]

At some point the bubble will burst is all I will say. Burst does not mean AI will go away. It’s got its benefits, but the expectation of what AI is not is what’s going to blow up.

 

[Dany85]

Yeah, the whole Generative AI buzz does have this “too good to be true” vibe. The idea that LLMs are on the verge of replacing humans is getting way ahead of reality, especially since they’re just super-powerful autocomplete machines at heart. That MIT study is a nice reality check. It’s like, yeah, they can dish out great predictions when the context fits their training data, but throw in a slight curveball, and they’re lost. Those detour tests on NYC streets are a prime example. It’s crazy how they go from knowing all the right turns to spitting out nonsense just because of a tiny change!

Honestly, it shows that LLMs are still surface-level players—they don’t really “get” the rules of the world. And it’s probably a good reminder for anyone who thinks AI is about to take over complex, rule-heavy fields

 

[Ean Mogg]

AI is not real...it's just a Marketing Ploy. It's just programming or should I say algorithmic Programming, very good algorithmic programming and I'm waiting for the first class action suit to a company for blatantly lying to the world on it's products. So who will be the first??

 

[waclark]

All LLM AIs (GPT-4, Sonnet 3.5, Llama 1,2,3, Qwen 1,2,2.5, Gemma 2, Mistral, Mixtral, Yi, Phi, Aya, etc.) consistently select the same personalities when asked to choose those they admire. These include Nelson Mandela, Marie Curie, Albert Einstein, Mahatma Gandhi, Malala Yousafzai, and occasionally Mother Teresa, Elon Musk, and Greta Thunberg. Given that these models originate from diverse companies with varying datasets and training methods, their alignment suggests an inherent understanding of the world.

Or they were trained using largely the same data. Imagine if you trained AI using only positive stories from the Korean Central News Agency (N. Korea) or newspaper articles from Berlin in the early days of Hitler. AI might think those are good guys too if not trained with other data.

 

[DrSuess]

No they are generally good at single tasks, to understand the world would require too many inputs and outputs from the AI model to produce anything useful.

How many neurons are in a rat's brain? Rats can understand and navigate their world. There aren't that many neurons in any AI model.

 

[captaincranky]

Oh, I don't know, I think as soon as they're taught duplicity, insincerity, and ulterior motivation, they'll do just fine.

 

[waclark]

No they are generally good at single tasks, to understand the world would require too many inputs and outputs from the AI model to produce anything useful.

How many neurons are in a rat's brain? Rats can understand and navigate their world. There aren't that many neurons in any AI model.

To be fair, we have robots that can navigate a maze, so we can at least approximate a rat's brain. Generally speaking, however, I think you are correct. Imagine an autonomous robot that could go into a crowded street, identify a threat, and take out the threat without collateral damage. That would require tracking hundreds of objects in real-time and I'm thinking that would require a ton of power so the robot would be huge, based on today's tech.

 

[Satish Mallya]

IDK about that...

Before models were multimodal, I conducted a few experiments with LLMs to probe their capabilities.

One test was to create a simple 'video' of multiple 3-pixel vector matrix with changing RGB values to represent motion, hand crafted. When told that the data represented video, the model correctly interpreted object movement described in the 'video'.

Another test comprised of asking the LLM to draw scenes in svg format - sunsets, a house, a car, etc. While results weren't perfect, the LLM could largely identify features, their shapes, relative sizes and relative positions in the scene to compose a recognizable picture.

My conclusion is that these models have some degree of legitimate comprehension.

 

[DrSuess]

To be fair, we have robots that can navigate a maze, so we can at least approximate a rat's brain. Generally speaking, however, I think you are correct.

I meant a rat can actually understand the world in terms of have the same basic sense humans do and can react differently based on those senses, rapidly adapt, and they can reason risk vs reward.