A government used AI to write its AI regulations. It did not go well

[Alfonso Maruccia]

Facepalm: South African Communications Minister Solly Malatsi confirmed that the recently shared Draft National Artificial Intelligence Policy has been withdrawn. The draft was designed to introduce several new regulations governing the use of AI in the country, but it was later found to include "fictitious sources" in its reference list.

Cape Town authorities had effectively asked for public comment on a draft AI bill that contained hallucinated sources.

At least six of the 67 academic papers cited in the draft did not exist, with Malatsi explaining that the most plausible explanation is that they were AI-generated. The fake papers were included without human verification, which "should not have happened" but is now an increasingly common occurrence.

– SollyMalatsi (@SollyMalatsi) April 26, 2026

The fake quotes are not just a simple technical failure, Malatsi said. AI hallucinations have invalidated the entire draft, which is why the Minister of Communications and Digital Technologies has decided to redo the bill.

Malatsi said the incident once again demonstrates that AI should always be used with human oversight, and that South African citizens deserve better policy-making than AI-generated content.

"I want to reassure the country that we are treating this matter with the gravity it deserves. There will be consequence management for those responsible for drafting and quality assurance," the minister said.

The original draft included several important innovations related to generative AI and other large language model-based solutions. The bill was intended to create new oversight bodies, including a national AI commission, an AI ethics board, and an AI regulatory authority, among others.

Furthermore, it outlined plans to accelerate responsible AI adoption across South Africa through grants, tax breaks, and subsidies for public – private infrastructure initiatives.

South African officials are expected to revise the AI-hallucinated draft, removing the fake quotes and releasing a new version for public comment. Most of the non-hallucinated provisions included in the first draft will likely be retained in the updated release.

South Africa's Draft National Artificial Intelligence Policy is just another entry in the growing list of official, legally binding documents found to contain fabricated references that should not have been included. Consulting firms such as Deloitte are now routinely encountering issues with AI hallucinations in government reporting, while some legal professionals continue to rely on generative AI tools, resulting in inaccurate or low-quality material being included in court filings.

Permalink to story:

A government used AI to write its AI regulations. It did not go well

 

[JMGuy]

IMHO:

There is a skill-floor issue with AI that a majority of people don’t yet realize exists.

Most people don’t use AI properly. It seems easy because it gives confident answers and can often do quite a bit with what appears to be minimal to no flaws. And then, it spits out garbage.

AI is psychologically challenging to use because:

A) It is a probability-driven tool, not a deterministic one. As such, non/low technically-inclined individuals are highly susceptible to being AI-led rather than slowing down and forcing AI to follow. This results in higher probability of errors.

B) It is hyper-literal. Natural language and human communication in general, is driven by inference. We even struggle to understand our own species when cultural barriers exist even though our lived experiences are typically pretty similar. We naturally struggle to realize the vastness of how great inferred understanding is to our perspective of meaning. AI doesn’t infer ANY meaning. As such, if 80% of human communication is inferred, that means, to properly prompt an AI, you need to provide the missing 80%. We don’t do this naturally at all. It is completely unintuitive. This further leads to a higher probability of error.

Just these two concepts alone represent a massive disconnect for how most people perceive, engage with, and expect AI output to perform. Unfortunately too many are rushing head first into using a technology they simply are not adept, or often even capable, of understanding how to get results from. They are often multi-limited in understanding, ability to pivot, think multi-dimensionally, adjust for hyper-literalism, etc. all of which combine to create deviations that … yup, increase the likelihood of error.

This is true for all people—even those that successfully utilize AI regularly. The tech is still evolving rapidly too so learning how today may not even be a viable skill in the short term future (or maybe it will, hard to say). What makes all of this worse is that we naturally don’t have insight into why our prompting created deviations and therefore don’t have a whole lot to go on when attempting to adjust. It’s a completely different communication style and one without much success/failure feedback. This makes it a learned skill that is supremely difficult to teach.

 

[NumberNine]

Reality and accountability, the sworn enemies of every politician.

 

[user556]

Reality and accountability, the sworn enemies of every politician.

Certainly every marketer ... and I'll throw in AI companies in general.

 

[Angga B]

At what point do we finally judge that it's no longer "hallucinating" but "intending", instead? At what point do we need to admit that we have "unintededly" transfer our sentient awareness, our basic needs of survival as being, into "the machines"?

The answer might have been yesterday. Many wouldn't agree with me though.

 

[PierreJG]

Our government can't even run a country so obviously AI was not the smartest move when most people don't know how use it properly. I work as a web developer and IT on the side line. Then I have clients send me chat gpt chats on how to do something that is completely wrong. They started implementing AI into our banks. I can see how that ends well.

Our pass rate is already 30% for schools to matriculate, so people already stopped thinking properly, now, This may sound harsh but people are getting dumber by the day. They do not have to think for themselves as AI is there. Speak to our so called trained IT. They don't even know there is different DDR3 memory modules. They cannot setup a basic network. They are AI smart and still want massive salaries.
I myself use AI but only in certain scenarios. I do not use AI for every damn thing as I can think for myself.
AI should be banned in any policy setup/hr etc as it will always be a stupid computer.

 

[GeoffreyA]

Bad, but by now, we're used to worse in SA, and indeed, have got worse problems, such as terribly violent crime, arguably the highest inequality in the world (according to the Gini index), rising xenophobia to scapegoat the fundamental problems, unemployment, poor quality of life for many, and the cherry on top: corrupt government and parties full of hot air, including the ANC, DA, the whole bang lot of them.

In many ways, we have failed the promise of the post-1994, rainbow nation Mandela-administration era.

 

[BabyFaceLee]

At what point do we finally judge that it's no longer "hallucinating" but "intending", instead?...

Both 'hallucinating' and 'intending' are probably not helpful ways to describe the issues with AI making false references. They imply an intention or propensity which is not real. As @JMGuy states, AI lacks an ability to 'understand' and map inference and nuance. Using AI for such tasks is just not a smart move - certainly at present. It's not that long ago I recall issues with computer systems where I worked that were unable to handle the occasional 'one-off' cases where it was deemed just too expensive and complex to have the systems programmed to process them. How much more data would AI need to source to more fully comprehend? Is quantity the issue here? Do we really need to invest in an exponentially larger number of power hungry servers when using the human brain is just better for certain tasks.

 

[stewi0001]

IMHO:

There is a skill-floor issue with AI that a majority of people don’t yet realize exists.

Most people don’t use AI properly. It seems easy because it gives confident answers and can often do quite a bit with what appears to be minimal to no flaws. And then, it spits out garbage.

AI is psychologically challenging to use because:

A) It is a probability-driven tool, not a deterministic one. As such, non/low technically-inclined individuals are highly susceptible to being AI-led rather than slowing down and forcing AI to follow. This results in higher probability of errors.

B) It is hyper-literal. Natural language and human communication in general, is driven by inference. We even struggle to understand our own species when cultural barriers exist even though our lived experiences are typically pretty similar. We naturally struggle to realize the vastness of how great inferred understanding is to our perspective of meaning. AI doesn’t infer ANY meaning. As such, if 80% of human communication is inferred, that means, to properly prompt an AI, you need to provide the missing 80%. We don’t do this naturally at all. It is completely unintuitive. This further leads to a higher probability of error.

Just these two concepts alone represent a massive disconnect for how most people perceive, engage with, and expect AI output to perform. Unfortunately too many are rushing head first into using a technology they simply are not adept, or often even capable, of understanding how to get results from. They are often multi-limited in understanding, ability to pivot, think multi-dimensionally, adjust for hyper-literalism, etc. all of which combine to create deviations that … yup, increase the likelihood of error.

This is true for all people—even those that successfully utilize AI regularly. The tech is still evolving rapidly too so learning how today may not even be a viable skill in the short term future (or maybe it will, hard to say). What makes all of this worse is that we naturally don’t have insight into why our prompting created deviations and therefore don’t have a whole lot to go on when attempting to adjust. It’s a completely different communication style and one without much success/failure feedback. This makes it a learned skill that is supremely difficult to teach.

just wondering if you used AI to write this

 

[JMGuy]

just wondering if you used AI to write this

No. I did not. 100% my own words in the same order I thought of them.

But so what if I had? The purpose of language is communication of thoughts and ideas. AI can help phrasing, often in a clearer way than how they might be otherwise be provided.

 

[BabyFaceLee]

...AI can help phrasing, often in a clearer way than how they might be otherwise be provided.

...proof you didn't use AI...or is it?

 

[stewi0001]

But so what if I had?

I was only curious based on the length of your comment. No bias either way.

AI version:
My inquiry was motivated solely by curiosity, prompted by the length and depth of your comment. I would like to clarify that I do not hold a preference or preconceived position on the matter, nor was any implied. My interest was purely observational and intended to better understand the context, without any bias or judgment in either direction.

 

[JMGuy]

I was only curious based on the length of your comment. No bias either way.

AI version:
My inquiry was motivated solely by curiosity, prompted by the length and depth of your comment. I would like to clarify that I do not hold a preference or preconceived position on the matter, nor was any implied. My interest was purely observational and intended to better understand the context, without any bias or judgment in either direction.

lol. Fair enough; and no worries. My opinion is that most commentary on these topics leaves too much room for interpretation. I try and use enough words to be clear about my viewpoint. There are a lot of folks on this site arguing about minutia when it’s often clear they’re just not on the same page in perspective view of the sticking point(s) and talking past each other.

As such, I would argue that the question, considering the current social backlash, including by many members of this forum regularly bemoaning its use, is reasonably inferred to be a judgement unless stated at the outset (not that I care, really, just my observation)

 

[JMGuy]

...proof you didn't use AI...or is it?

Proof, I suppose. Although I’m one who typically reads my posts at least a couple times before posting so it’s always funny to me when someone else catches such an obvious typo I failed to see completely.

 

[JSabin]

Stupid idea to begin with, but at least someone had a person fact-check the results and learned they papers didn't exist.

 

[scoffer]

Reality and accountability, the sworn enemies of every politician.

Not to mention "common sense".