Under review · 2026

TAFS

Transformer-Attentive Feature Selection for Sequential Regression

Huseyin Karaca, A. Samil Namli, Suleyman S. Kozat

Bilkent University · Georgia Institute of Technology

Recent target values feed a context token c_t, and each engineered feature feeds its own token t_1 to t_p. In a feature-axis transformer, the CLS token attends to the context token and to every feature token, with arcs of different widths. The CLS readout goes to a head with three options, unconstrained, affine and convex, which outputs weights w_t. The weights combine the cached predictions of frozen experts f_1 to f_K into a prediction, which goes to the loss against y_t. A dashed gradient from the loss returns to the head and the transformer only.
Attention over feature tokens, conditioned on recent targets, sets the weights on frozen experts; one loss (dashed) trains the rest.

Abstract

We study sequential regression in dynamically changing environments under a mixture-of-experts setting. A set of experts produces predictions in parallel, while the best-performing expert and the relevance of the available features may change over time. Our objective is to determine the expert combination weights using the features available to the experts and additional side information. We introduce TAFS, a transformer-based ensemble method that conditions feature attention on recent target history. The method represents individual features as tokens and uses their context-dependent representations to compute combination weights over a fixed pool of experts. We consider unconstrained, affine, and convex combinations and train the proposed model by minimizing cumulative prediction error while keeping the experts fixed. We establish differentiability properties of the training objective, a stationary-point convergence result under standard smoothness assumptions, and boundedness of convex combinations for bounded expert predictions. On a synthetic regime-switching task, TAFS reduces mean squared prediction error by 41% relative to its context-free variant. In the real-data experiments, TAFS reduces mean absolute scaled error by up to 56% compared with the LightGBM ensemble and by up to 55% compared with the strongest static feature-transform baseline. We release the source code to support reproducibility and further research.

Method

Each scalar feature scales a learned vector and adds a feature-identity embedding, so the tokens need no positional encoding. The context token encodes the last target values and the residuals of a seasonal-naive forecast. Diagonal, linear, low-rank and MLP feature transforms are restricted or context-free forms of the same mixing module.

Results

56% lower MASE than the LightGBM ensemble on Istanbul Stock Exchange

MethodISEExchange
Simple average1.4190.466
Best single predictor0.2700.485
Nonlinear MLP transform0.5630.530
LGBM-ensemble (convex)0.5770.447
TAFS (unconstrained)0.2520.461
Oracle (per-step best)0.1500.198
One-step MASE, mean of three seeds, same seven frozen experts throughout. On near-random-walk Exchange-Rate returns, the LightGBM ensemble is lower.

Context conditioning cuts synthetic MSE by 41%

MethodMSE
Best single base predictor0.4162
Equal-weight average0.3566
Nonlinear bottleneck0.2392
TAFS (no context)0.2188
TAFS (full)0.1292
Oracle0.0581
A hidden change point switches which features matter; 28 of the 32 features are noise. Mean of three seeds.

BibTeX

@unpublished{karaca2026tafs,
  author = {Huseyin Karaca and A. Samil Namli and Suleyman Serdar Kozat},
  title  = {{TAFS}: Transformer-Attentive Feature Selection for
            Sequential Regression},
  year   = {2026},
  note   = {Under review},
  url    = {https://hkaraca.com/tafs/}
}