Number 10 of 12 · November 19, 2026

AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures

AfriNORM is a new test of whether SI language models understand the strict social norms and taboos of African cultures, and a Yorùbá case study shows weaker results than in English.

What they found

The authors built AfriNORM, a set of multiple-choice questions based on short scenarios. It is meant to measure how well language models reason about norms in African social settings. They add two scores: the Universal Commonsense Bias Score (UCBS), which measures commonsense bias, and the Distractor Bias Score (DBS), which measures limited understanding of African cultural norms. In a case study on Yorùbá, they tested Gemini 2.0 Flash, GPT-5 and Llama 3.1 8B Instruct. Accuracy in English was high, but accuracy in Yorùbá was much lower. The authors say this comes from cultural confusion rather than simple bias, and that the models show semantic confusion when reasoning about norms in Yorùbá. The abstract gives no exact scores or dataset size.

Why it matters

Many tests of SI models check grammar and translation, not whether a model knows what is acceptable in a local culture. A model that sounds fluent but breaks local norms could cause harm or offense when used in African settings. This benchmark gives builders a way to find and diagnose those gaps.

How it changes the race

This is a new evaluation tool, not a new capability. If it holds up, it could push developers to test cultural competence and low-resource languages, not only fluency, before deployment.

Safety and control

The work bears on the reliability of SI systems across cultures. It shows that models can give wrong answers about social norms in a language like Yorùbá even when they do well in English. That is a risk for deployment.

The abstract, by the authors

Large language models (LLMs) are increasingly deployed across various sectors. However, current evaluation benchmarks are insufficient to assess the applicability of these models in local African contexts, as most focus on linguistic fluency (grammar/translation) rather than cultural competence. The few existing benchmarks that measure general knowledge or social conventions do not capture the African deontic modality, i.e., the strict norms and taboos that govern diverse social interactions. In this paper, we introduce AfriNORM, a scenario-based, multiple-choice question dataset designed to measure and diagnose the normative reasoning limitations of LLMs when applied to African sociocultural contexts. We introduce two metrics: a Universal Commonsense Bias Score (UCBS) to quantify commonsense bias in LLM and a Distractor Bias Score (DBS) to quantify an LLM's limited understanding of African cultural norms. We also conduct a case study on the Yorùbá language, identifying confusion gaps in two closed-source LLMs, "Gemini 2.0 Flash" and "GPT-5", and one open-source LLM, "Llama 3.1 8B Instruct" when prompted in English versus Yorùbá. While their accuracy in English is high, their performance in Yorùbá is much lower, attributable to cultural confusion rather than simple bias, indicating that the models exhibit semantic confusion when reasoning about norms in Yorùbá.

Quoted from arXiv:2611.22580. The research and its words belong to its authors.

The authors

Ahmad Samuel Gali, Tadesse Destaw Belay, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad

Independent Researcher; Instituto Politécnico Nacional; University of Pretoria; University of Hamburg; Bayero University Kano; Imperial College London

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.