What they found
The authors gave SI language models a user's social-media posts and asked them to fill in the BFI-2-60 Big Five personality questionnaire as that user. They tested GPT-4, Gemma, Llama 3.1 and Mistral on the MyPersonality dataset. User text did not consistently improve results compared with answers made without any user text or with values based on general population tendencies. Models did better for people near the middle of a trait range. For people with low or high scores, they often stayed near central values, or did well at one end and poorly at the other. When given only an explicit Big Five profile, GPT-4 most clearly turned it into consistent answers, especially for Extraversion and Neuroticism. Adding more text did not steadily cut error: open-source models fluctuated more, while GPT-4 stayed fairly stable.
Why it matters
Many people want to use SI models as digital twins, meaning stand-ins that act like a real person, for surveys or social research. This paper tests that idea with a careful check against real questionnaire scores. It suggests that models may look plausible while mostly giving average answers. That matters for anyone who would rely on such stand-ins to represent individuals, especially people at the extremes.
How it changes the race
This does not change how frontier SI is built. It does set a more careful bar for claims about simulating individual people, and it shows that gains from adding more context are not guaranteed.
Safety and control
The results warn against trusting SI models to represent specific individuals, for example in research or decisions about people. The abstract does not discuss misuse or control directly.
The abstract, by the authors
This paper investigates whether Large Language Models can answer a personality questionnaire as a specific individual would. Given a user's social-media posts, the model completes the BFI-2-60 Big Five questionnaire as that user. The predicted responses are converted into trait scores and compared with the user's actual scores. We evaluate GPT-4, Gemma, Llama 3.1, and Mistral on the MyPersonality dataset. To determine whether predictions reflect information from the user's writing, we compare them with answers produced without user text and with values derived from general population tendencies. We measure prediction error across all users and separately for individuals with low, intermediate, and high trait scores. We also test whether an LLM can express an explicit personality profile when given only the target Big Five scores, without user text, and examine how predictions change as more text is provided. The results show that user text does not consistently improve performance. Models perform better near the middle of the trait distribution, while at the lower and upper ends they often remain close to central values or perform well at one end but poorly at the other. When given an explicit profile, GPT-4 most clearly translates trait information into consistent questionnaire responses, particularly for Extraversion and Neuroticism. Increasing context does not systematically reduce error: open-source models fluctuate more, whereas GPT-4 remains comparatively stable. Overall, the evaluated models do not consistently reproduce the questionnaire behavior of specific individuals.
Quoted from arXiv:2611.16586. The research and its words belong to its authors.
