AI-assisted code can be inherently insecure, study finds

Alfonso Maruccia

Posts: 269   +131
Forward-looking: Machine learning algorithms are all the rage now, as they are used to generate any kind of "original" content after being trained on enormous pre-existing datasets. Code-generating AIs, however, could pose a real issue for software security in the future.

AI systems like GitHub Copilot promise to make programmers' lives easier by creating entire chunks of "new" code based on natural-language textual inputs and pre-existing context. But code-generating algorithms can also bring an insecurity factor to the table, as a new study involving several developers has recently found.

Looking specifically at Codex, the AI platform developed by OpenAI, which is also the code-making engine of the aforementioned GitHub Copilot, the study recruited 47 different developers. Ranging from undergraduate students to experienced professionals, said developers were tasked with using Codex to solve security-related problems in Python, JavaScript, C, and other high-level programming languages.

The researchers said that when the programmers had access to the Codex AI, the resulting code was more likely incorrect or insecure compared to the "hand-made" solutions conceived by the control group. Furthermore, the programmers with AI-assisted solutions were more likely to say that their insecure code was secure compared to the aforementioned control group.

Neil Perry, a PhD candidate at Stanford and the study lead co-author, said that "code-generating systems are currently not a replacement for human developers." Said developers could be using AI-assisted tools to complete tasks outside their own areas of expertise, or to speed up a programming task they are already skilled in. They should be both concerned, the study author said, and they should always double-check the generated code.

According to Megha Srivastava, a postgraduate Stanford student and the second co-author of the study, Codex is anything but useless: despite the shortcomings of the "stupid" AI, code-generating systems can be useful when employed for low-risk tasks. Furthermore, the programmers involved in the study didn't have a particular expertise in security matters, which could have helped in spotting vulnerable or insecure code, Srivastava said.

AI algorithms could also be fine-tuned to improve their coding suggestions, and companies that develop their own systems can get better solutions with a model generating code more in-line with their own security practices. Code-generating technology is an "exciting" development with many people eager to use it, the study authors said. It's just that there is still a lot of work to be done on finding proper solutions to AI shortcomings.

Permalink to story.



Posts: 61   +56
There is at least as much chances to produce insecure code with AI as doing it yourself, AI is not perfect yet. If you use AI for anything you should check and fix the results before submitting your code but it will still make you save a lot of time. It makes coding on new languages a lot easier and it can guess your code from comments and previous code.


Posts: 247   +145
Well, these AIs are trained off piles of existing code on code sites... you've seen the quality of that code yourself I'm sure. Some of it's just plain wrong and crap. Some is correct but has 0 error handling (including no effort to be secure, no validation of inputs etc.).. sometimes because that programmer writes code that is insecure and not robust; sometimes because it's a bare minimum example and they assume you will take care of error handling and securing the code. So given what the AI trained off of, I would not expect secure code.

Unfortunately, many people are not going to realize how these AIs work and will assume the system is writing robust, secure code.


Posts: 39   +30
Aside of security problems, I would really like to know level of optimization of that AI generated crap....


Posts: 1,198   +1,740
Aside of security problems, I would really like to know level of optimization of that AI generated crap....
For starters, NV aims to generate AI-programmed optimizations into their drivers, with some 30% boost expected. Will see.