We’ve updated our Global-MMLU paper to include Global-MMLU Lite ✨—a faster, more efficient eval set. Plus, we’ve added to the eval harness!
It covers a balanced subset of languages with an equal mix of Culturally Sensitive (CS) & Culturally Agnostic (CA) questions per language
Is MMLU Western-centric? 🤔
As part of our cross-institutional work:
🥢 We conduct a large-scale cultural bias study on MMLU
🔍 Examine how cultural sensitivity impacts multilingual evaluations
🌍 Release Global-MMLU: a benchmark with MMLU translations in 42 languages