- Published on
Global PIQA: Testing AI with Local Culture
- Authors
- Name
- Alfred Kondoro
- X
- Name
- Cynthia Amol
- X
- Name
- Sharon Ibejih
- X
- Name
- Adeyemi Praise
- X
- Name
- Okechukwu God'spraise
- X
- Name
- Deborah Popoola
- X
Global PIQA is a groundbreaking participatory commonsense reasoning benchmark, carefully created by a global network of 335 researchers from 65 countries. It is a shared task designed to evaluate LLMs' understanding of everyday physical knowledge. This is the kind of knowledge that is deeply embedded in local contexts, customs, and traditions.
The benchmark covers over 100 language varieties, across five continents, 14 language families, and 23 writing systems. What makes this work unique is its cultural depth. In the non-parallel split of Global PIQA, over 50% of examples reference local foods, customs, traditions, or other culturally-specific elements. This ensures that models are tested on knowledge that is truly local, moving beyond translations of Western-centric concepts.
The initial findings from the project identified that while state-of-the-art LLMs perform well in aggregate, they exhibit significant weaknesses in lower-resource languages, with accuracy gaps as large as 37% (compared to a random chance of 50%). Furthermore, open-source models generally lag behind proprietary ones. This highlights a crucial point: everyday, culturally-embedded knowledge remains a major area for improvement in LLMs, especially in less-resourced communities.
As part of this massive collaborative effort, we are happy to have contributed to the curated data samples for Nigerian Pidgin and Yoruba. Our team, which was composed of native speakers and linguists, hand-crafted commonsense reasoning questions that are specific to Nigerian culture, customs, and environment. For example, a question might reference how a specific Nigerian food is prepared, or common traditional practices. This dedication ensures that the resulting data genuinely tests an LLM's comprehension of the Nigerian context. Our participation in Global PIQA aligns directly with our overarching mission, which is to significantly increase data resources for African languages. By contributing high-quality, culturally-rich data to a global resource like Global PIQA, we are helping to ensure that future LLMs evaluations are more inclusive, and accurate.