No single expert wins

Each corpus in the study comes with three pretrained recognizers, the experts. None of them is best on every clip. Choosing the best of the three for each clip in hindsight, the per-clip oracle, gives a much lower word error rate than the best single expert (WER in %):

AMI-SDM

best single18.42

per-clip oracle11.68

Earnings-22

best single12.79

per-clip oracle9.00

People's Speech

best single17.39

per-clip oracle12.74

AfriSpeech-200

best single25.82

per-clip oracle17.51

Reaching that headroom without running every recognizer is the routing problem. It is hard for three reasons. The cues that decide which expert will do well, such as a burst of overlapping speech, a change of microphone or a stretch of accented speech, sit somewhere inside the clip. The router has to compare the experts with one another, not score each one in isolation. And transcribing with every expert to pick the best output means paying for every decoder.

Why averaging over time is not enough

A router that averages each expert's encoder states over the clip sees one vector per clip. Two clips that hold the same conditions in opposite order give it the same vector, even when they need different experts. A router that reads the frames sees which condition comes first.

Two clips with the same two conditions in opposite order: their frames in order, and each clip's average over time.

The idea

• Read the framesHIT-ASR keeps every expert's encoder states frame by frame, at each expert's own frame rate and width, instead of a pooled summary.
• Summarise, then compareA first transformer stage summarises each expert's stream over time; a second stage compares the experts' summaries and scores them jointly.
• All encoders, one decoderEvery encoder runs once per clip; the router picks one expert and only that expert's decoder runs.

The method page walks through the router step by step, with the training objective and the cost.

Results at a glance

HIT-ASR has a lower word error rate than every baseline on all four corpora. It closes 18.2 to 34.1 % of the gap between the best single expert and the per-clip oracle; MLP-pool, the strongest other router, closes less on every corpus.

How much of the gap between the best single expert (left end) and the per-clip oracle (right end) each router closes on the test folds; the ends give the two WERs in %.

MethodThe problem, the router step by step, the training objective, what runs at inference. ResultsFour corpora, the synthetic regime switch, routing behaviour, ablation, cost. ReproduceEvery table from the stored results on a CPU, every model retrained on a GPU, the experts rebuilt from audio. DataThe audio, the experts' transcripts and encoder states, and the stored results, on the Hugging Face Hub.

Citation

@article{karaca2026hitasr,
  title  = {{HIT-ASR}: Hierarchical Transformer Routing for Adaptive {ASR} Expert Selection},
  author = {Karaca, Huseyin and Namli, A. Samil and Kozat, Suleyman S.},
  year   = {2026},
  note   = {Under review at IEEE Transactions on Audio, Speech and Language Processing}
}