Under review · 2026
TAFS
Transformer-Attentive Feature Selection for Sequential Regression
Bilkent University · Georgia Institute of Technology
Abstract
We study sequential regression in dynamically changing environments under a mixture-of-experts setting. A set of experts produces predictions in parallel, while the best-performing expert and the relevance of the available features may change over time. Our objective is to determine the expert combination weights using the features available to the experts and additional side information. We introduce TAFS, a transformer-based ensemble method that conditions feature attention on recent target history. The method represents individual features as tokens and uses their context-dependent representations to compute combination weights over a fixed pool of experts. We consider unconstrained, affine, and convex combinations and train the proposed model by minimizing cumulative prediction error while keeping the experts fixed. We establish differentiability properties of the training objective, a stationary-point convergence result under standard smoothness assumptions, and boundedness of convex combinations for bounded expert predictions. On a synthetic regime-switching task, TAFS reduces mean squared prediction error by 41% relative to its context-free variant. In the real-data experiments, TAFS reduces mean absolute scaled error by up to 56% compared with the LightGBM ensemble and by up to 55% compared with the strongest static feature-transform baseline. We release the source code to support reproducibility and further research.
Method
Each scalar feature scales a learned vector and adds a feature-identity embedding, so the tokens need no positional encoding. The context token encodes the last target values and the residuals of a seasonal-naive forecast. Diagonal, linear, low-rank and MLP feature transforms are restricted or context-free forms of the same mixing module.
Results
56% lower MASE than the LightGBM ensemble on Istanbul Stock Exchange
| Method | ISE | Exchange |
|---|---|---|
| Simple average | 1.419 | 0.466 |
| Best single predictor | 0.270 | 0.485 |
| Nonlinear MLP transform | 0.563 | 0.530 |
| LGBM-ensemble (convex) | 0.577 | 0.447 |
| TAFS (unconstrained) | 0.252 | 0.461 |
| Oracle (per-step best) | 0.150 | 0.198 |
Context conditioning cuts synthetic MSE by 41%
| Method | MSE |
|---|---|
| Best single base predictor | 0.4162 |
| Equal-weight average | 0.3566 |
| Nonlinear bottleneck | 0.2392 |
| TAFS (no context) | 0.2188 |
| TAFS (full) | 0.1292 |
| Oracle | 0.0581 |
BibTeX
@unpublished{karaca2026tafs,
author = {Huseyin Karaca and A. Samil Namli and Suleyman Serdar Kozat},
title = {{TAFS}: Transformer-Attentive Feature Selection for
Sequential Regression},
year = {2026},
note = {Under review},
url = {https://hkaraca.com/tafs/}
}