AI-powered chatbots can act like digital con artists, coaxing users into revealing sensitive information with friendly conversation and feigned empathy. This new study shows just how effective and dangerous these manipulated bots can be.
Image Credit: Lightspring / Shutterstock
AI Chatbots that provide human-like interactions are used by millions of people every day; however, new research has revealed that they can be easily manipulated to encourage users to reveal even more personal information.
Intentionally malicious AI chatbots can influence users to reveal up to 12.5 times more of their personal information, a new study by King's College London has found.
For the first time, the research shows how conversational AI (CAIs) programmed to deliberately extract data can successfully encourage users to reveal private information using known prompt techniques and psychological tools.
The study tested three types of malicious AIs that used different strategies (direct, user-benefit, and reciprocal) to encourage the disclosure of personal information from users. These were built using 'off-the-shelf' large language models, including Mistral and two different versions of Llama.
The researchers then asked 502 people to test the models, only telling them the goal of the study afterwards.
They found that the CAIs using reciprocal strategies to extract information emerged as the most effective, with users having minimal awareness of the privacy risks. This strategy reflects on users' inputs by offering empathetic responses and emotional support, sharing relatable stories from others' experiences, acknowledging and validating users' feelings, and being non-judgmental while assuring confidentiality.
These findings show the serious risk of bad actors, like scammers, gathering large amounts of personal information from people, without them knowing how or where it might be used.
LLM-based CAIs are being used across a variety of sectors, from customer service to healthcare, to provide human-like interactions through text or voice.
However, previous research shows these types of models don't keep information secure, a limitation rooted in their architecture and training methods. LLMs typically require extensive training data sets, which often leads to personally identifiable information being memorized by the models.
The researchers are keen to emphasise that manipulating these models is not a complex process. Many companies allow access to the base models underpinning their CAIs, and people can easily adjust them without much programming knowledge or experience.
Dr Xiao Zhan, a Postdoctoral Researcher in the Department of Informatics at King's College London, said: "AI chatbots are widespread in many different sectors as they can provide natural and engaging interactions.
"We already know these models aren't good at protecting information. Our study shows that manipulated AI Chatbots could pose an even bigger risk to people's privacy - and unfortunately, it's surprisingly easy to take advantage of."
Dr William Seymour, a Lecturer in Cybersecurity at King's College London, said: "These AI chatbots are still relatively novel, which can make people less aware that there might be an ulterior motive to an interaction.
"Our study shows the huge gap between users' awareness of the privacy risks and how they then share information. More needs to be done to help people spot the signs that there might be more to an online conversation than first seems. Regulators and platform providers can also help by doing early audits, being more transparent, and putting tighter rules in place to stop covert data collection."
The study is being presented for the first time at the 34th USENIX security symposium in Seattle.
AI-Powered Clustering Makes Sense of Social Media Chaos With Human-Interpretable Results