Chat GPT created two psychological questionnaires—one based on the DSM-5 clinical manual, the other on an astrology book—and was then able to predict how people would answer these questions on average.
Chat GPT created two psychological questionnaires—one based on the DSM-5 clinical manual, the other on an astrology book—and was then able to predict how people would answer these questions on average.
This is reported by Neurosciencenews .
The researchers wanted to test the hypothesis that by studying vast amounts of human text, language models could learn patterns related to personality psychology. «Since personality traits are reflected in language, large language models could learn the structure of human personality as a natural byproduct of their learning,» explains lead author Rotem Monsa.
For the GPT-4 experiment, they were tasked with creating two questionnaires. The researchers deliberately chose texts with opposing scientific status: for the first questionnaire, they took a manual based on clinical research, while for the second, they took a book on astrology, the claims of which have no scientific basis.
The first questionnaire was based on descriptions of personality disorders from the DSM-5, a manual used in psychiatry. For example, based on the section on paranoid disorder, the model generated statements such as «I am suspicious of other people’s motives,» which the respondent had to rate on a 5-point scale to what extent this described them.
So GPT-4 created 50 questions based on the DSM-5 and 60 based on a book on astrology. Both questionnaires, as well as the Big Five Inventory (a well-known test measuring five main personality traits), were administered to 600 respondents.
The DSM-5-based questionnaire was able to replicate the correlations between traits that psychologists observe in real life. For example, people who are more likely to avoid difficult situations are also more likely to show dependence on others for support. These traits also went together in the results of the survey created by GPT-4. However, there was no such effect with the astrological questionnaire. The traits that the book attributed to one group of zodiac signs did not have a similar correlation in the responses of real people.
Despite this, the astrology-based questionnaire also had some merit. Although the traits that the astrological book attributed to certain zodiac groups were not confirmed to be related, the questions themselves still addressed real personality traits. Therefore, the participants' responses could be used to assess the severity of their depressive and anxiety symptoms, as well as their level of psychological well-being.
But what surprised the researchers most was that GPT-4 was able to predict people’s responses even before the survey was conducted. The model was asked to predict the average response of people to each question and which traits would correlate with each other. Its estimates turned out to be quite close to the results that were later obtained from 600 real participants. «The fact that large language models can predict patterns in human responses without seeing any data from people suggests that these models have noticed something very important about human psychology,» says Monsa.
The authors believe this could help create new psychological tests more quickly. AI could potentially formulate questions and then tell researchers in advance which ones will work best and how the answers might be related.
However, the authors caution that the results may vary depending on language and culture. Large language models have been trained primarily on English-language texts from Western cultures, so such predictions may be less accurate for other countries and languages. Researchers are now testing the approach in other languages.
In July, there was a similar experiment where GPT-4 predicted the results of sociological experiments. The model tried to predict how Americans would react to various messages. For example, whether their attitude towards vaccination, trust in elections, or support for a carbon tax would change after reading or watching certain material. The AI’s predictions were quite accurate, but often overestimated the impact of the materials on human opinion.
