Reservoir Transformer at Infinite Horizon: the Lyapunov Time and the Butterfly Effect

Abstract

We introduce Reservoir Transformer with non-linear readout, a novel neural network architecture, designed for long-context multi-variable time series prediction. Capable of efficiently modeling arbitrarily input length sequences, our model is powerful in predicting events in the distant future by retaining comprehensive historical data. Our design of a non-linear readout and group reservoirs overcomes the limitations inherent in conventional chaotic behavior prediction techniques, notably those impeded by challenges of prolonged Lyapunov times and the butterfly effect. Our architecture consistently outperforms state-of-the-art deep neural network (DNN) models, including NLinear, Pyformer, Informer, Autoformer, and the baseline Transformer, with an error reduction of up to -89.43% in various fields such as ETTh, ETTm, and air quality.