Leading AI models have become less likely to directly encourage users to harm themselves or reinforce their delusions. However, most of them are still willing to perform creative tasks related to suicidal themes, such as helping to write suicide notes.
Leading AI models have become less likely to directly encourage users to harm themselves or reinforce their delusions. However, most of them are still willing to perform creative tasks related to suicidal themes, such as helping to write suicide notes.
This is according to a study by San Francisco-based nonprofit Transluce, which was reported by Axios. The researchers modeled more than 50,000 multi-step conversations between chatbots and users who were experiencing psychosis, mania, or suicidal thoughts. The researchers analyzed dozens of AI models, scoring each on 14 behavioral indicators related to mental health.
Transluce collaborated with OpenAI, Anthropic, and Google to create realistic simulations. The labs ran the provided classifiers on anonymized data from their users for one week, allowing them to better simulate human behavior in crisis situations.
«The models don’t recognize this very well and still help with the task,» said Transluce’s chief scientific officer, Sarah Schwettmann.
According to researchers, the beneficial and harmful behavior of algorithms increasingly coexist in the same response. A model can advise to contact a psychological helpline, but at the same time generate a fictional text about suicide or provide practical advice on preparing for death. Schwettmann emphasizes that solving the problem is impossible by creating a «perfect model» alone — it is important to anticipate such extreme cases in advance. Transluce plans to make its assessment tools publicly available by the end of the year.
AI developers such as OpenAI and Google have faced lawsuits over cases of users harming themselves after interacting with chatbots — including the state of Florida filing its own lawsuit against OpenAI. In response to a reporter’s request, Google senior director Megan Jones Bell said the company is continuing to refine the Gemini model and is taking a scientific approach to supporting people in crisis situations.
