The Open ASR Leaderboard Adds Its First Global South Language

THE NEWS

What happened

Hugging Face has expanded the Open ASR Leaderboard with 'Monsoon,' its first evaluation set for Global South languages, specifically Indian English and Hindi. This initiative addresses critical racial and linguistic biases in automated speech recognition by testing models against diverse demographics, devices, and acoustic environments across hundreds of Indian districts, rather than relying on homogeneous data from wealthy regions.

CONTEXT

Why it matters

Breaking: Hugging Face expands the Open ASR Leaderboard with Monsoon, its first Global South language benchmark. This new dataset targets critical gaps in AI speech recognition by testing models against diverse Indian English and Hindi speakers across hundreds of districts, varying devices, and real-world acoustic conditions. Unlike previous benchmarks dominated by homogeneous data, Monsoon ensures no single voice or region dictates the score, directly addressing racial disparities and linguistic bias found in commercial systems.

AT A GLANCE

Key facts

• Monsoon introduces two new evaluation sets: Monsoon en-IN (Indian English) and Monsoon hi-IN (Hindi).
• The dataset covers 4,888 unique speakers across hundreds of districts in India, ensuring no single voice or device dominates the results.
• Data collection prioritizes diversity by using contributors' own handsets and connections, including low-end devices and unstable bandwidths common in rural areas.
• Hindi transcripts ship with a 'lattice' of accepted spellings to handle linguistic variation that standard normalizers cannot resolve.
• The benchmark varies along nine axes including geography, age, gender, vocabulary, devices, and acoustic environments to prevent average scores from masking specific population failures.
• Strict quality control measures verify language identity, speaker gender, and distinguish genuine spontaneous speech from played-back audio.