What they found
The authors built AfriNORM, a set of multiple-choice questions based on short scenarios. It is meant to measure how well language models reason about norms in African social settings. They add two scores: the Universal Commonsense Bias Score (UCBS), which measures commonsense bias, and the Distractor Bias Score (DBS), which measures limited understanding of African cultural norms. In a case study on Yorùbá, they tested Gemini 2.0 Flash, GPT-5 and Llama 3.1 8B Instruct. Accuracy in English was high, but accuracy in Yorùbá was much lower. The authors say this comes from cultural confusion rather than simple bias, and that the models show semantic confusion when reasoning about norms in Yorùbá. The abstract gives no exact scores or dataset size.
Why it matters
Many tests of SI models check grammar and translation, not whether a model knows what is acceptable in a local culture. A model that sounds fluent but breaks local norms could cause harm or offense when used in African settings. This benchmark gives builders a way to find and diagnose those gaps.
How it changes the race
This is a new evaluation tool, not a new capability. If it holds up, it could push developers to test cultural competence and low-resource languages, not only fluency, before deployment.
Safety and control
The work bears on the reliability of SI systems across cultures. It shows that models can give wrong answers about social norms in a language like Yorùbá even when they do well in English. That is a risk for deployment.
The abstract, by the authors
Large language models (LLMs) are increasingly deployed across various sectors. However, current evaluation benchmarks are insufficient to assess the applicability of these models in local African contexts, as most focus on linguistic fluency (grammar/translation) rather than cultural competence. The few existing benchmarks that measure general knowledge or social conventions do not capture the African deontic modality, i.e., the strict norms and taboos that govern diverse social interactions. In this paper, we introduce AfriNORM, a scenario-based, multiple-choice question dataset designed to measure and diagnose the normative reasoning limitations of LLMs when applied to African sociocultural contexts. We introduce two metrics: a Universal Commonsense Bias Score (UCBS) to quantify commonsense bias in LLM and a Distractor Bias Score (DBS) to quantify an LLM's limited understanding of African cultural norms. We also conduct a case study on the Yorùbá language, identifying confusion gaps in two closed-source LLMs, "Gemini 2.0 Flash" and "GPT-5", and one open-source LLM, "Llama 3.1 8B Instruct" when prompted in English versus Yorùbá. While their accuracy in English is high, their performance in Yorùbá is much lower, attributable to cultural confusion rather than simple bias, indicating that the models exhibit semantic confusion when reasoning about norms in Yorùbá.
Quoted from arXiv:2611.22580. The research and its words belong to its authors.
