Modified Kalman filter based method for training state-recurrent multilayer perceptrons

D. Erdogmus, Justin C. Sanchez, J. C. Principe

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Kalman filter based training algorithms for recurrent neural networks provide a clever alternative to the standard backpropagation in time. However, these algorithms do not take into account the optimization of the hidden state variables of the recurrent network. In addition, their formulation requires Jacobian evaluations over the entire network, adding to their computational complexity. We propose a spatial-temporal extended Kalman filter algorithm for training recurrent neural network weights and internal states. This new formulation also reduces the computational complexity of Jacobian evaluations drastically by decoupling the gradients of each layer. Monte Carlo comparisons with backpropagation through time point out the robust and fast convergence of the algorithm.

Original languageEnglish (US)
Title of host publicationNeural Networks for Signal Processing - Proceedings of the IEEE Workshop
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages219-228
Number of pages10
Volume2002-January
ISBN (Print)0780376161
DOIs
StatePublished - 2002
Externally publishedYes
Event12th IEEE Workshop on Neural Networks for Signal Processing, NNSP 2002 - Martigny, Switzerland
Duration: Sep 6 2002 → …

Other

Other12th IEEE Workshop on Neural Networks for Signal Processing, NNSP 2002
CountrySwitzerland
CityMartigny
Period9/6/02 → …

Fingerprint

Multilayer neural networks
Kalman filters
Recurrent neural networks
Backpropagation
Computational complexity
Extended Kalman filters

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Artificial Intelligence
  • Software
  • Computer Networks and Communications
  • Signal Processing

Cite this

Erdogmus, D., Sanchez, J. C., & Principe, J. C. (2002). Modified Kalman filter based method for training state-recurrent multilayer perceptrons. In Neural Networks for Signal Processing - Proceedings of the IEEE Workshop (Vol. 2002-January, pp. 219-228). [1030033] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/NNSP.2002.1030033

Modified Kalman filter based method for training state-recurrent multilayer perceptrons. / Erdogmus, D.; Sanchez, Justin C.; Principe, J. C.

Neural Networks for Signal Processing - Proceedings of the IEEE Workshop. Vol. 2002-January Institute of Electrical and Electronics Engineers Inc., 2002. p. 219-228 1030033.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Erdogmus, D, Sanchez, JC & Principe, JC 2002, Modified Kalman filter based method for training state-recurrent multilayer perceptrons. in Neural Networks for Signal Processing - Proceedings of the IEEE Workshop. vol. 2002-January, 1030033, Institute of Electrical and Electronics Engineers Inc., pp. 219-228, 12th IEEE Workshop on Neural Networks for Signal Processing, NNSP 2002, Martigny, Switzerland, 9/6/02. https://doi.org/10.1109/NNSP.2002.1030033
Erdogmus D, Sanchez JC, Principe JC. Modified Kalman filter based method for training state-recurrent multilayer perceptrons. In Neural Networks for Signal Processing - Proceedings of the IEEE Workshop. Vol. 2002-January. Institute of Electrical and Electronics Engineers Inc. 2002. p. 219-228. 1030033 https://doi.org/10.1109/NNSP.2002.1030033
Erdogmus, D. ; Sanchez, Justin C. ; Principe, J. C. / Modified Kalman filter based method for training state-recurrent multilayer perceptrons. Neural Networks for Signal Processing - Proceedings of the IEEE Workshop. Vol. 2002-January Institute of Electrical and Electronics Engineers Inc., 2002. pp. 219-228
@inproceedings{48b971817f95456c8b01a00732b80e77,
title = "Modified Kalman filter based method for training state-recurrent multilayer perceptrons",
abstract = "Kalman filter based training algorithms for recurrent neural networks provide a clever alternative to the standard backpropagation in time. However, these algorithms do not take into account the optimization of the hidden state variables of the recurrent network. In addition, their formulation requires Jacobian evaluations over the entire network, adding to their computational complexity. We propose a spatial-temporal extended Kalman filter algorithm for training recurrent neural network weights and internal states. This new formulation also reduces the computational complexity of Jacobian evaluations drastically by decoupling the gradients of each layer. Monte Carlo comparisons with backpropagation through time point out the robust and fast convergence of the algorithm.",
author = "D. Erdogmus and Sanchez, {Justin C.} and Principe, {J. C.}",
year = "2002",
doi = "10.1109/NNSP.2002.1030033",
language = "English (US)",
isbn = "0780376161",
volume = "2002-January",
pages = "219--228",
booktitle = "Neural Networks for Signal Processing - Proceedings of the IEEE Workshop",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Modified Kalman filter based method for training state-recurrent multilayer perceptrons

AU - Erdogmus, D.

AU - Sanchez, Justin C.

AU - Principe, J. C.

PY - 2002

Y1 - 2002

N2 - Kalman filter based training algorithms for recurrent neural networks provide a clever alternative to the standard backpropagation in time. However, these algorithms do not take into account the optimization of the hidden state variables of the recurrent network. In addition, their formulation requires Jacobian evaluations over the entire network, adding to their computational complexity. We propose a spatial-temporal extended Kalman filter algorithm for training recurrent neural network weights and internal states. This new formulation also reduces the computational complexity of Jacobian evaluations drastically by decoupling the gradients of each layer. Monte Carlo comparisons with backpropagation through time point out the robust and fast convergence of the algorithm.

AB - Kalman filter based training algorithms for recurrent neural networks provide a clever alternative to the standard backpropagation in time. However, these algorithms do not take into account the optimization of the hidden state variables of the recurrent network. In addition, their formulation requires Jacobian evaluations over the entire network, adding to their computational complexity. We propose a spatial-temporal extended Kalman filter algorithm for training recurrent neural network weights and internal states. This new formulation also reduces the computational complexity of Jacobian evaluations drastically by decoupling the gradients of each layer. Monte Carlo comparisons with backpropagation through time point out the robust and fast convergence of the algorithm.

UR - http://www.scopus.com/inward/record.url?scp=84953731767&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84953731767&partnerID=8YFLogxK

U2 - 10.1109/NNSP.2002.1030033

DO - 10.1109/NNSP.2002.1030033

M3 - Conference contribution

SN - 0780376161

VL - 2002-January

SP - 219

EP - 228

BT - Neural Networks for Signal Processing - Proceedings of the IEEE Workshop

PB - Institute of Electrical and Electronics Engineers Inc.

ER -