Under review · Expert Systems with Applications
FASTT: Feature-Adaptive Speech-to-Text Model Selection
Bilkent University
Abstract
Existing approaches to automatic speech recognition (ASR) either rely on a single speech-to-text (STT) model or combine multiple models through computationally expensive ensemble techniques, leading to suboptimal trade-offs between accuracy and efficiency across diverse acoustic conditions. Since different STT models exhibit complementary strengths, selecting the most suitable model for each input can significantly improve transcription accuracy. Here, we introduce FASTT, a feature-adaptive STT selection framework that learns to select the most appropriate pretrained STT model for each input. FASTT jointly learns a task-aligned feature transformation and a lightweight selection function based on acoustic and embedding-derived features, and is trained in an end-to-end manner to minimize a cumulative transcription-error objective. Due to this end-to-end training, we learn both the features that best distinguish between STT models and the selection algorithm so that FASTT outperforms static selection and standard routing methods. Across five English corpora spanning read, parliamentary, meeting, financial and crowd-sourced speech, evaluated with 5×2 cross-validation and paired significance tests, FASTT is significantly better than the best single model on four of the five corpora (p < 0.05) and statistically tied with it on the fifth, and recovers up to 32% of the per-utterance oracle headroom. Because FASTT selects from the members' encoder outputs and decodes only the chosen member, it avoids up to 86% of the decoding time of a decode-all ensemble with a microsecond-scale selector overhead. Furthermore, we show that the selector training procedures converge to first-order stationary points of the training objective under mild smoothness conditions for a fixed feature transformation, and we publicly release the code to support reproducibility.
Method
Mean-pooled final-layer encoder states and log duration feed a learned gating, linear, low-rank, or nonlinear feature transformation. FASTT-Boosted jointly trains transforms and soft trees; FASTT-Alternating alternates XGBoost refits with transform updates through a linear surrogate. Both minimize expected WER from offline expert errors, so training accounts for how much a wrong choice costs.
Results
Lower WER than the best single model on four of five corpora
| Method | LibriSpeech | VoxPopuli | AMI (SDM) | Earnings-22 | People's Speech |
|---|---|---|---|---|---|
| Best-Single-Model | 0.0226 | 0.1761 | 0.6528 | 0.4103 | 0.4011 |
| ROVER (weighted) | 0.0214 | 0.1737 | 0.6667 | 0.4151 | 0.3894 |
| FASTT-Boosted | 0.0224 | 0.1735 | 0.6221 | 0.3901 | 0.3860 |
| FASTT-Alternating | 0.0225 | 0.1733 | 0.6240 | 0.3893 | 0.3880 |
| Oracle (per utterance) | 0.0140 | 0.1494 | 0.5584 | 0.3452 | 0.3491 |
Saves up to 86% of decode-all time
| System | Greedy | Beam 5 | Beam 20 |
|---|---|---|---|
| Decode all | 72.9 | 948.7 | 3927.9 |
| FASTT-Boosted | 49.4 | 133.4 | 1158.0 |
BibTeX
@article{karaca2026fastt,
title = {{FASTT}: Feature-Adaptive Speech-to-Text Model Selection},
author = {Karaca, Huseyin and Namli, A. Samil and
Kozat, Suleyman S.},
year = {2026},
note = {Under review},
url = {https://hkaraca.com/fastt/}
}