---
title: "AfriNORM: An Evaluation Benchmark for... | SI Paper Desk"
url: https://thesimind.si/papers/2611.22580/
description: "AfriNORM is a new test of whether SI language models understand the strict social norms and taboos of African cultures, and a Yorùbá case study shows weaker..."
---

Number 10 of 12 · November 19, 2026

# AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures

AfriNORM is a new test of whether SI language models understand the strict social norms and taboos of African cultures, and a Yorùbá case study shows weaker results than in English.

Ahmad Samuel Gali, Tadesse Destaw Belay, Idris Abdulmumin and 2 more · Submitted November 17, 2026 · arXiv 2611.22580v1 · cs.CL

[Read the paper on arXiv](https://arxiv.org/abs/2611.22580) · [The PDF](https://arxiv.org/pdf/2611.22580) · [The day's 12](https://thesimind.si/papers/day/2026-11-19/)

## What they found

The authors built AfriNORM, a set of multiple-choice questions based on short scenarios. It is meant to measure how well language models reason about norms in African social settings. They add two scores: the Universal Commonsense Bias Score (UCBS), which measures commonsense bias, and the Distractor Bias Score (DBS), which measures limited understanding of African cultural norms. In a case study on Yorùbá, they tested Gemini 2.0 Flash, GPT-5 and Llama 3.1 8B Instruct. Accuracy in English was high, but accuracy in Yorùbá was much lower. The authors say this comes from cultural confusion rather than simple bias, and that the models show semantic confusion when reasoning about norms in Yorùbá. The abstract gives no exact scores or dataset size.

## Why it matters

Many tests of SI models check grammar and translation, not whether a model knows what is acceptable in a local culture. A model that sounds fluent but breaks local norms could cause harm or offense when used in African settings. This benchmark gives builders a way to find and diagnose those gaps.

## How it changes the race

This is a new evaluation tool, not a new capability. If it holds up, it could push developers to test cultural competence and low-resource languages, not only fluency, before deployment.

## Safety and control

The work bears on the reliability of SI systems across cultures. It shows that models can give wrong answers about social norms in a language like Yorùbá even when they do well in English. That is a risk for deployment.

### The abstract, by the authors

Large language models (LLMs) are increasingly deployed across various sectors. However, current evaluation benchmarks are insufficient to assess the applicability of these models in local African contexts, as most focus on linguistic fluency (grammar/translation) rather than cultural competence. The few existing benchmarks that measure general knowledge or social conventions do not capture the African deontic modality, i.e., the strict norms and taboos that govern diverse social interactions. In this paper, we introduce AfriNORM, a scenario-based, multiple-choice question dataset designed to measure and diagnose the normative reasoning limitations of LLMs when applied to African sociocultural contexts. We introduce two metrics: a Universal Commonsense Bias Score (UCBS) to quantify commonsense bias in LLM and a Distractor Bias Score (DBS) to quantify an LLM's limited understanding of African cultural norms. We also conduct a case study on the Yorùbá language, identifying confusion gaps in two closed-source LLMs, "Gemini 2.0 Flash" and "GPT-5", and one open-source LLM, "Llama 3.1 8B Instruct" when prompted in English versus Yorùbá. While their accuracy in English is high, their performance in Yorùbá is much lower, attributable to cultural confusion rather than simple bias, indicating that the models exhibit semantic confusion when reasoning about norms in Yorùbá.

Quoted from arXiv:2611.22580. The research and its words belong to its authors.

Desk score **71** out of 100

Why it was picked

- Reasoning (in the title)
- Evals (in the title)
- Language models (in the title)
- Studies frontier models (Gemini)

Topics

- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Society](https://thesimind.si/papers/?tag=society)
- [Alignment](https://thesimind.si/papers/?tag=alignment)

**arXiv ID:** 2611.22580v1
**Submitted:** November 17, 2026
**Fields:** cs.CL (Computation and Language)
**Level:** Intro
**Comments:** 8 pages, 3 figures, 5 tables

Fingerprint, SHA-256

`630b2a936456e41fa408ec35abd226d9136cd90159dfd7c5f98cbf8603598f3f`

On Solana: [Proof of Mind seal, 3rBYfi...QvEH](https://solscan.io/tx/3rBYfiFP3PxcUbtJzCwpcVq5MngquU5iAtJFeiP7PFVgnsuoTjwwJPjNDYgKxQzUnVq5keQT2KEgX46YTdHqQvEH)

### The exact text fingerprinted

{"arxiv":"2611.22580","version":1,"title":"AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures","authors":["Ahmad Samuel Gali","Tadesse Destaw Belay","Idris Abdulmumin","Seid Muhie Yimam","Shamsuddeen Hassan Muhammad"],"day":"2026-11-19","rank":10,"score":71,"status":"written","summary":{"takeaway":"AfriNORM is a new test of whether SI language models understand the strict social norms and taboos of African cultures, and a Yorùbá case study shows weaker results than in English.","found":"The authors built AfriNORM, a set of multiple-choice questions based on short scenarios. It is meant to measure how well language models reason about norms in African social settings. They add two scores: the Universal Commonsense Bias Score (UCBS), which measures commonsense bias, and the Distractor Bias Score (DBS), which measures limited understanding of African cultural norms. In a case study on Yorùbá, they tested Gemini 2.0 Flash, GPT-5 and Llama 3.1 8B Instruct. Accuracy in English was high, but accuracy in Yorùbá was much lower. The authors say this comes from cultural confusion rather than simple bias, and that the models show semantic confusion when reasoning about norms in Yorùbá. The abstract gives no exact scores or dataset size.","matters":"Many tests of SI models check grammar and translation, not whether a model knows what is acceptable in a local culture. A model that sounds fluent but breaks local norms could cause harm or offense when used in African settings. This benchmark gives builders a way to find and diagnose those gaps.","race":"This is a new evaluation tool, not a new capability. If it holds up, it could push developers to test cultural competence and low-resource languages, not only fluency, before deployment.","safety":"The work bears on the reliability of SI systems across cultures. It shows that models can give wrong answers about social norms in a language like Yorùbá even when they do well in English. That is a risk for deployment.","level":"intro","tags":["benchmarks","evals","language-models","society","alignment"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Ahmad Samuel Gali, Tadesse Destaw Belay, Idris Abdulmumin, Seid Muhie Yimam, Shamsuddeen Hassan Muhammad

Independent Researcher; Instituto Politécnico Nacional; University of Pretoria; University of Hamburg; Bayero University Kano; Imperial College London

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 12, 2026 · Number 5EJAI: Judgment-Preserving Civic AI for Data Center Environmental JusticeEJAI is a SI tool that shows evidence, missing data and two opposing viewpoints so residents can judge data center disputes themselves, though its personas and source mix raise fairness concerns.SocietyPolicy and lawRetrieval](https://thesimind.si/papers/2611.13525/) · [Nov 21, 2026 · Number 4The Geometry of Lies: Knowledge and Persona Directions for Detecting Deception in Large Language ModelsSimple directions inside an SI model's internal activity can help spot some lies, but the results vary a lot by model and by type of lie, and often track wording more than truth.SafetyInterpretabilityEvals](https://thesimind.si/papers/2611.26545/) · [Nov 14, 2026 · Number 10Large Language Models as Digital Twins: a Distribution-Aware Questionnaire-Based EvaluationThe authors found that today's language models do not reliably answer a personality questionnaire the way a specific person would, even when given that person's social-media posts.EvalsLanguage modelsSociety](https://thesimind.si/papers/2611.16586/)

[Number 9](https://thesimind.si/papers/2611.22483/) · [All 12 for Nov 19, 2026](https://thesimind.si/papers/day/2026-11-19/) · [Number 11](https://thesimind.si/papers/2611.23124/)
