Number 10 of 12 · November 14, 2026

Large Language Models as Digital Twins: a Distribution-Aware Questionnaire-Based Evaluation

The authors found that today's language models do not reliably answer a personality questionnaire the way a specific person would, even when given that person's social-media posts.

What they found

The authors gave SI language models a user's social-media posts and asked them to fill in the BFI-2-60 Big Five personality questionnaire as that user. They tested GPT-4, Gemma, Llama 3.1 and Mistral on the MyPersonality dataset. User text did not consistently improve results compared with answers made without any user text or with values based on general population tendencies. Models did better for people near the middle of a trait range. For people with low or high scores, they often stayed near central values, or did well at one end and poorly at the other. When given only an explicit Big Five profile, GPT-4 most clearly turned it into consistent answers, especially for Extraversion and Neuroticism. Adding more text did not steadily cut error: open-source models fluctuated more, while GPT-4 stayed fairly stable.

Why it matters

Many people want to use SI models as digital twins, meaning stand-ins that act like a real person, for surveys or social research. This paper tests that idea with a careful check against real questionnaire scores. It suggests that models may look plausible while mostly giving average answers. That matters for anyone who would rely on such stand-ins to represent individuals, especially people at the extremes.

How it changes the race

This does not change how frontier SI is built. It does set a more careful bar for claims about simulating individual people, and it shows that gains from adding more context are not guaranteed.

Safety and control

The results warn against trusting SI models to represent specific individuals, for example in research or decisions about people. The abstract does not discuss misuse or control directly.

The abstract, by the authors

This paper investigates whether Large Language Models can answer a personality questionnaire as a specific individual would. Given a user's social-media posts, the model completes the BFI-2-60 Big Five questionnaire as that user. The predicted responses are converted into trait scores and compared with the user's actual scores. We evaluate GPT-4, Gemma, Llama 3.1, and Mistral on the MyPersonality dataset. To determine whether predictions reflect information from the user's writing, we compare them with answers produced without user text and with values derived from general population tendencies. We measure prediction error across all users and separately for individuals with low, intermediate, and high trait scores. We also test whether an LLM can express an explicit personality profile when given only the target Big Five scores, without user text, and examine how predictions change as more text is provided. The results show that user text does not consistently improve performance. Models perform better near the middle of the trait distribution, while at the lower and upper ends they often remain close to central values or perform well at one end but poorly at the other. When given an explicit profile, GPT-4 most clearly translates trait information into consistent questionnaire responses, particularly for Extraversion and Neuroticism. Increasing context does not systematically reduce error: open-source models fluctuate more, whereas GPT-4 remains comparatively stable. Overall, the evaluated models do not consistently reproduce the questionnaire behavior of specific individuals.

Quoted from arXiv:2611.16586. The research and its words belong to its authors.

The authors

Francesco Di Cursi, Chiara Boldrini, Andrea Passarella, Marco Conti

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.