# FFASR Leaderboard The Far-Field ASR Leaderboard is an open benchmark from Treble Technologies and Hugging Face, measuring speech recognition under realistic acoustic conditions rather than clean studio audio. It is the first open far-field ASR benchmark maintained as a continuously updated leaderboard. The headline finding is blunt: **far-field word error rate at low signal-to-noise is consistently several times higher than near-field WER on the same speech.** Same words, same speakers, a room instead of a booth, and the numbers fall apart. ## What it measures Nine conditions, four of them used for ranking: - **Near-field (dry)**: clean anechoic speech, the condition every other benchmark uses - **Far-field, high SNR**: above 14 dB - **Far-field, mid SNR**: 8 to 12 dB - **Far-field, low SNR**: below 6 dB Plus sim-to-real validation (lab-measured against simulated) and moving-source splits where the speaker does not stay put. Metrics are WER and RTFx (real-time factor on an NVIDIA L4), with a Pareto front view of the accuracy-versus-speed trade-off. The held-out test set is never exposed to submitters. ## How the data is made This is the part that makes the benchmark credible. Treble's hybrid simulation engine solves the wave equation at low frequencies and switches to geometrical acoustics at high frequencies, which captures diffraction, scattering, and modal behaviour that simpler room simulations miss. Two thousand anechoic speech samples, placed into **fourteen furnished rooms** from 20 to 470 cubic metres: bathrooms, living rooms with hallways, offices, classrooms, restaurants. One target speaker plus up to three noise sources, transient and continuous, at three SNR tiers. Roughly eight hours per condition, with Whisper-style text normalization applied consistently. It builds on the Treble10 dataset, whose room impulse responses are available for research. ## Why it matters **It separates accuracy from robustness, and those are different things.** A model can top every clean-speech leaderboard and fall over the moment there is a fridge humming and the speaker is three metres away. Until now there was no standard way to tell the difference, so the field optimized for the condition it could measure. This is Goodhart's Law wearing a lab coat: the benchmark was clean audio, so models got very good at clean audio. **The gap it exposes is the gap that matters commercially.** Nobody deploys ASR in an anechoic chamber. Meeting rooms, kitchens, cars, and classrooms are the deployment reality, and this is the first public number describing how much of the advertised accuracy survives the trip. **Simulation is what makes it maintainable.** Recording thousands of hours across fourteen real rooms at three noise levels is not a thing you redo when a new model ships. Simulating it is, which is why this can be a live leaderboard instead of a one-off challenge paper like CHiME or URGENT. ## Caveats - The rankings are only as good as the simulation. The sim-to-real track exists precisely because that question is open, and it is the first thing to check before trusting a number - Language coverage is not specified in the announcement - Multi-talker scenarios, microphone arrays, and echo cancellation are planned, not present. Those are three of the most common real deployment conditions - WER against a Whisper-style normalizer measures agreement with one normalization convention, which quietly advantages models in that lineage ## References - Announcement — https://huggingface.co/blog/ffasr-leaderboard ## Related - [[Whisper]] - [[AI Evaluation]] - [[Large Language Models (LLMs)]] - [[AI Open Weight Models]] - [[Goodhart's Law]] - [[Mistral AI]]