Feature-Based Echo-State Networks: A Step Towards Interpretability and Minimalism in Reservoir Computer

INTRODUCTION

This paper proposes a novel architecture for echo-state networks (ESNs), a type of recurrent neural network used for model-free prediction of dynamical systems. The proposed "feature-based echo-state network" (Feat-ESN) uses smaller, parallel reservoirs, each processing a different set of input features. This allows the model to be more interpretable, as the relative contribution of each feature to the output prediction can be determined. The Feat-ESN approach also significantly reduces the number of reservoir nodes required for effective prediction, which is particularly useful for high-dimensional systems like traffic volume forecasting.

The paper evaluates the Feat-ESN approach on two synthetic chaotic datasets and a real-world traffic volume dataset. It also includes an ablation study examining the impact of reservoir size. Overall, the Feat-ESN architecture provides improved interpretability and efficiency compared to standard ESNs, making it a useful technique for modeling complex dynamical systems.

Echo-State Networks for Dynamical Systems Prediction

Echo-state networks (ESNs) are a type of recurrent neural network with a large, randomly connected reservoir of neurons. The network takes input signals and processes them nonlinearly through the reservoir, with the output then linearly combined from the reservoir states. The performance of an ESN depends heavily on the richness of the reservoir dynamics.

The ESN consists of an input layer, a recurrent reservoir layer, and an output layer. The reservoir evolves nonlinearly over time, with the output linearly read out from the reservoir states. The input weights and reservoir weights are randomly initialized and held fixed, while the output weights are trained using linear regression with regularization.

During training, the ESN is driven by an input sequence, which generates a sequence of reservoir states. These reservoir states are stored in a matrix, and the corresponding output sequence is also stored. The output weights are then computed using a linear regression procedure that incorporates Tikhonov regularization to ensure numerical stability.

The provided text discusses the echo state property (ESP) of echo state networks (ESNs). It states that ESNs are universal approximators, meaning they can accurately represent any nonlinear operator with bounded memory if they satisfy the ESP. The ESP requires that the reservoir, represented by the weight matrix W, asymptotically washes out any information from the initial conditions. For the hyperbolic tangent activation function, it is empirically observed that the ESP holds when the spectral radius of W is less than one. To ensure this condition, W is normalized by its spectral radius.

Feature-Based Echo-State Network: A Bite Sized Approach

The paper presents a new architecture called Feat-ESN for echo state networks (ESNs) that aims to reduce the overall size of the reservoir while maintaining the predictive performance. The key components of Feat-ESN are:

Input map: Features are selected from the input vector, where a feature is any combination of the input components. A feature matrix is constructed to feed each feature separately into smaller individual reservoirs.

Reservoirs: Each feature is fed into a separate smaller reservoir. The total number of reservoir nodes is the product of the number of features and the block size of the individual reservoirs. The reservoirs evolve linearly, with each smaller reservoir depending only on its corresponding feature.

Readout: A nonlinear readout network is used to maintain the expressivity of the ESN. The readout is a combination of the reservoir, its nonlinear map, and a bias. The output is then linearly read out from this expanded reservoir representation.

The paper shows that Feat-ESN satisfies the universal approximation property under certain conditions on the feature matrix and the readout functions. Additionally, the relative magnitudes of the output weights can be used to prune the reservoir by removing less important features.

V Numerical Examples

This section presents the performance and ablation study of the Feat-ESN algorithm on three time series datasets: two generated by chaotic dynamical systems (Lorenz and Rössler systems) and one from real-time traffic flow data.

For the Lorenz and Rössler systems, Feat-ESN achieves better prediction accuracy compared to a regular ESN, using a much smaller number of reservoir nodes. The normalized root mean square error (NRMSE) is lower for Feat-ESN across different block sizes. The contribution of the nonlinear features to the output is found to be higher than the linear features.

For the traffic flow data, Feat-ESN is applied to predict the time series of traffic volume recorded by a sensor. To handle the partial observation from a single sensor, a delay embedding is used in the input layer. Feat-ESN outperforms the regular ESN, showing lower NRMSE and higher Pearson correlation between the predicted and true traffic volumes. The contribution of the features (delay combinations) decreases as the delay increases, as expected.

Overall, the results demonstrate the improved performance and reduced complexity of the Feat-ESN algorithm compared to a regular ESN for time series prediction tasks.

Conclusion

The paper proposes a feature-based approach to generate the reservoir for an echo-state network (ESN). The algorithm, called Feat-ESN, uses parallel smaller linear neuronal reservoirs driven by different input combinations, called features. This significantly reduces the computational complexity of the ESN while maintaining the same predictive performance as a much larger reservoir. The proposed approach demonstrates improved prediction performance with fewer reservoir nodes compared to a regular ESN for chaotic time-series. The method is then applied to real traffic data from the University of Maryland campus to predict traffic volume at various intersections.

Acknowledgement

The section acknowledges support from Dr. Derek A. Paley and the University of Maryland Department of Transportation for providing Numina sensor data, as well as Dr. Artur Wolek for preprocessing the data.