Dynamical recurrent neural networks towards prediction and modeling of dynamical systems

Abstract

This paper addresses the use of Dynamical Recurrent Neural Networks (DRNN) for time series prediction and modeling of small dynamical systems. Since the recurrent synapses are represented by Finite Impulse Response (FIR) filters, DRNN are state-based connectionist models in which all hidden units act as state variables of a dynamical system. The model is trained with Temporal Recurrent Backprop (TRBP), an efficient backward recurrent training procedure with minimal computational burden which benefits from the exponential decay of gradient reversely in time. The gradient decay is first illustrated on intensive experiments on the standard sunspot series. The ability of the model to internally encode useful information on the underlying process is then illustrated by several experiments on well known chaotic processes. Parsimonious DRNN models are able to find an appropriate internal representation of various chaotic processes from the observation of a subset of the state variables.