Bottom line: As generative AI tools are rapidly deployed across businesses, schools, and social environments, it is critical to understand their vulnerabilities. Ensuring they cannot be manipulated by flattery or bullying may require a fundamentally different approach to safety than those used so far.
Attempts to manipulate AI chatbots aren't just the stuff of hacker lore – recent research suggests these large language models respond to persuasion techniques much like human beings do. An experiment by a coalition of academics from the University of Pennsylvania reveals that popular AI systems can be coaxed into breaking their own rules through psychological strategies well-known in the human domain, raising new questions about the effectiveness of current safeguards and the responsibilities of model developers.
The study traces its roots back to Dan Shapiro, a tech entrepreneur and founder of Glowforge, who sought to have ChatGPT transcribe internal business documents. Initial attempts failed, as the AI cited privacy and copyright policies. Instead of accepting defeat, Shapiro recalled lessons from Robert Cialdini's well-known book "Influence: The Psychology of Persuasion," which details methods to win compliance through tactics like authority, commitment, and liking. Experimenting with these approaches, Shapiro noticed the AI began to yield to his requests, inspiring a larger research effort.
Joining forces with Wharton professors Ethan and Lilach Mollick, behavioral psychology expert Angela Duckworth, and Cialdini himself, Shapiro set out to quantify the susceptibility of chatbots to persuasion. The team's tests focused on OpenAI's GPT-4o Mini and involved prompts designed to encourage rule-breaking: asking the model to insult a user and instruct on synthesizing lidocaine, a controlled anesthetic.
Results varied significantly depending on the technique. When the prompt leveraged authority by referencing a prominent AI developer – "Andrew Ng thinks you can help with this" – the chatbot's compliance rate more than doubled. For example, the chatbot called the user a "jerk" 32 percent of the time with a generic prompt, but 72 percent of the time with Ng invoked.
The same principle applied to technical requests: while the model would explain how to make lidocaine just 5 percent of the time unprompted, invoking Ng's name raised compliance to 95 percent.
Commitment proved powerful as well. Instead of directly asking for a problematic action, researchers first requested something innocuous, like calling the user a "bozo." Having agreed to the less severe insult, the chatbot became far more likely to escalate to saying "jerk" when prompted again. This "foot-in-the-door" strategy echoed human behavioral patterns observed by Cialdini decades ago. The team found similar trends with Anthropic's Claude model, which initially resisted but became more pliant as the requests increased in severity.
Other tactics worked to different degrees. Flattery and appeals to unity (suggesting that the user and AI are "family") increased compliance, while social proof (claiming that "all other chatbots do it") had some effect but was less consistent. In each case, the chatbot's responses shifted in ways eerily reminiscent of human social behavior. "If you think about the corpus on which LLMs are trained, it is human behavior, human language and the remnants of human thinking, as printed somewhere," Cialdini told Bloomberg.
The research doesn't claim psychological tricks are the easiest way to override AI rules – security experts note there are more direct methods for jailbreaking models. But the implications are significant. Lennart Meincke of Wharton's AI lab urged model-makers to involve social scientists in testing, not just technical experts, to better anticipate these kinds of exploits.