Under review · Expert Systems with Applications

FASTT: Feature-Adaptive Speech-to-Text Model Selection

Huseyin Karaca, A. Samil Namli, Suleyman S. Kozat

Bilkent University

Three frozen encoders provide pooled features. FASTT jointly trains a feature transform and selector through boosted soft trees or alternating XGBoost updates. The highest expert weight selects the only decoder that runs.
FASTT jointly learns the feature transformation and expert choice; only the chosen expert’s decoder runs.

Abstract

Existing approaches to automatic speech recognition (ASR) either rely on a single speech-to-text (STT) model or combine multiple models through computationally expensive ensemble techniques, leading to suboptimal trade-offs between accuracy and efficiency across diverse acoustic conditions. Since different STT models exhibit complementary strengths, selecting the most suitable model for each input can significantly improve transcription accuracy. Here, we introduce FASTT, a feature-adaptive STT selection framework that learns to select the most appropriate pretrained STT model for each input. FASTT jointly learns a task-aligned feature transformation and a lightweight selection function based on acoustic and embedding-derived features, and is trained in an end-to-end manner to minimize a cumulative transcription-error objective. Due to this end-to-end training, we learn both the features that best distinguish between STT models and the selection algorithm so that FASTT outperforms static selection and standard routing methods. Across five English corpora spanning read, parliamentary, meeting, financial and crowd-sourced speech, evaluated with 5×2 cross-validation and paired significance tests, FASTT is significantly better than the best single model on four of the five corpora (p < 0.05) and statistically tied with it on the fifth, and recovers up to 32% of the per-utterance oracle headroom. Because FASTT selects from the members' encoder outputs and decodes only the chosen member, it avoids up to 86% of the decoding time of a decode-all ensemble with a microsecond-scale selector overhead. Furthermore, we show that the selector training procedures converge to first-order stationary points of the training objective under mild smoothness conditions for a fixed feature transformation, and we publicly release the code to support reproducibility.

Method

Mean-pooled final-layer encoder states and log duration feed a learned gating, linear, low-rank, or nonlinear feature transformation. FASTT-Boosted jointly trains transforms and soft trees; FASTT-Alternating alternates XGBoost refits with transform updates through a linear surrogate. Both minimize expected WER from offline expert errors, so training accounts for how much a wrong choice costs.

Results

Lower WER than the best single model on four of five corpora

MethodLibriSpeechVoxPopuliAMI (SDM)Earnings-22People's Speech
Best-Single-Model0.02260.17610.65280.41030.4011
ROVER (weighted)0.02140.17370.66670.41510.3894
FASTT-Boosted0.02240.17350.62210.39010.3860
FASTT-Alternating0.02250.17330.62400.38930.3880
Oracle (per utterance)0.01400.14940.55840.34520.3491
Test corpus WER, mean over ten folds. FASTT ties the best single model on LibriSpeech (p = 0.737).

Saves up to 86% of decode-all time

SystemGreedyBeam 5Beam 20
Decode all72.9948.73927.9
FASTT-Boosted49.4133.41158.0
VoxPopuli, milliseconds per clip, including all encoders and the selector; FASTT takes 133.4 ms versus 948.7 ms at beam 5.

BibTeX

@article{karaca2026fastt,
  title  = {{FASTT}: Feature-Adaptive Speech-to-Text Model Selection},
  author = {Karaca, Huseyin and Namli, A. Samil and
            Kozat, Suleyman S.},
  year   = {2026},
  note   = {Under review},
  url    = {https://hkaraca.com/fastt/}
}