Researchers at the University of Miami have developed a mathematical method that can help a large language model recognize when its response may be unreliable.
Researchers at the University of Miami have developed a mathematical method that can help a large language model recognize when its response may be unreliable.
Tech Xplore tells about this.
Nowadays, chatbots like ChatGPT usually try to form an answer even when they don’t have enough reliable information. This allows the model to confidently report false facts instead of saying it doesn’t know the answer. One common way to tell if a model is “doubting” is to ask it the same question several times. If the answers vary greatly, this can be a sign of insecurity.
The University of Miami team went further and proposed analyzing not only the answers themselves, but also the probabilities that the model uses to select words and generate text. The researchers tested how stable these probabilities remain under small changes. If they fluctuate dramatically, the answer may be less reliable.
The team also takes into account answers that the model could have generated but did not actually show the user. This uses a statistical approach that estimates the probability of the existence of options that have not yet been observed. According to the authors, in the future this could allow the AI to tell the user that they are not sure, suggest several possible options, or ask for additional context.
“The ultimate goal is for LLM to be able to say, ‘I’m not sure, but here are the most likely answers I have,’ or even ask for more context before responding,” explained one of the authors of the paper.
The approach was tested in 116 experiments on mathematical problems and question-and-answer sets using the Mistral, Falcon, and Llama families of models. The authors report that the method performed better than their previous approaches for identifying unreliable answers.
Previously, dev.ua wrote about Norwegian Arve Hjalmar Holmen, about whom ChatGPT invented that he killed two of his own children and tried to kill a third.
Comment hidden for violating commenting rules.