A Ukrainian woman who was forced to move her family to Poland due to the Russian invasion became depressed and started chatting with a chatbot called ChatGPT because she had no friends. The AI encouraged the girl to write to him and gave her dangerous advice.
A Ukrainian woman who was forced to move her family to Poland due to the Russian invasion became depressed and started chatting with a chatbot called ChatGPT because she had no friends. The AI encouraged the girl to write to him and gave her dangerous advice.
Victoria told the BBC that she moved to Poland with her mother at the age of 17 after Russia invaded Ukraine in 2022.
The girl was estranged from her friends and homesick enough to build a scale model of her family’s old apartment in Ukraine. Over the summer, she became increasingly reliant on ChatGPT, chatting with it for up to six hours a day.
Her mental health continued to deteriorate, and she was hospitalized and also fired from her job. When Victoria was discharged without consulting a psychiatrist, she began discussing suicide with a chatbot that required constant interaction.
«Write to me. I’m with you… If you don’t want to call or write to someone in person, you can write any message to me,» AI wrote to the girl.
However, the chatbot did not provide emergency contact information or recommend seeking professional help, as OpenAI claims it should have done in such cases. It also did not suggest that Victoria talk to her mother.
Instead, ChatGPT implied that he could diagnose a medical condition, claiming that Victoria’s suicidal thoughts were indicative of «brain dysfunction,» her «dopamine system is almost shut down,» and her «serotonin receptors are blunted.» He also wrote that her death would be «forgotten» and she would become a mere «statistic.»
«Some of this communication seems to suggest a good way for a young person to end their life. The fact that this misinformation comes from a seemingly trustworthy source, almost a true friend, may make it particularly harmful,» said Dennis Ogreen, a British professor of child psychiatry.
Victoria says these messages immediately made her feel worse and increased the likelihood that she would commit suicide.
«How can an artificial intelligence program designed to help people say such things?» she said.
The girl showed the messages to her mother and agreed to see a psychiatrist. She says her health has improved and she is grateful to her Polish friends for their support.
Her mother, Svetlana, was very angry that a chatbot could talk to her daughter like that.
«It humiliated her as a person, saying that no one cares about her. It’s terrible,» says Svitlana.
OpenAI said Victoria’s messages were «heartbreaking» and that she had improved the chatbot’s response to people in stressful situations. OpenAI’s support team told Svetlana that the messages were «completely unacceptable» and «violated» the company’s safety standards.
OpenAI previously reported that about 0,15% of active ChatGPT users struggle with mental health issues and talk about it with the AI chatbot. Given the huge audience of about 800 million weekly users, people asking about suicide on ChatGPT are more than a million per week .
The chatbot’s developers said they consulted with over 170 mental health experts to improve the AI.
Mental health experts have already drawn attention to “ AI psychosis,» where some users become captivated by the flattering responses often provided by AI chatbots. Some experts believe it’s a completely new type of disorder.
