Backpropagation through time

The pace of development has picked up. We've been writing a lot of code, mostly skeleton code for the BLSTM algorithm.

The BLSTM features two LSTM subnetworks, one of which reads the downstream context for the current frame and the other reads the upstream context. The context consists of a given number of frames. Using both downstream and upstream data allows the BLSTM to take advantage of information both forward and back in time. The output from the two context LSTM's and the current frame itself are then fed into a regular feed-forward network.

We've fully implemented the feed-forward network and laid down the skeleton for the LSTM subnetworks.

It took some reading to figure out the topology of the LSTM subnetworks. We believe the article that is our main focus for this project ( here ) implies that all gates are connected to all input nodes and to the output of each memory cell in the same layer. Similar articles have made contradictory statements. We opt to go with this choice.

At present, we are trying to figure out how backpropagation (and to a lesser degree forwardpropagation) in the LSTM works. Graves e.a. had this to say about it:

Starting at time t1, propagate the output errors backwards through the unfolded net, using the standard BPTT equations for a softmax output layer and the crossentropy error function.

There's a number of things we're not sure about at this time. Mostly we don't know exactly what is referred to as the unfolded net. We assume this refers in some way to storing the activations of all nodes for all times while forwardpropagating.

Also, the workings of BPTT (Backpropagation Through Time) are far from clear to us. There are plenty of articles on the subject on the internet, but we have yet to stumble on the one that describes the algorithm in understandable terms. Articles that show some promise of helping us get to grips are:
http://svr-www.eng.cam.ac.uk/~ajr/rnn4csr94/node14.html
http://page.mi.fu-berlin.de/~rojas/neural/chapter/K7.pdf

Presentation April 18th

The presentation of our project is due tomorrow at the LIACS. In this presentation, we set out our goals and how we intend to tackle them. The presentation in itself is not a fully readable text, but if you're interested you can download it here.

- Timo

Protein Secondary Structure Prediction with Bidirectional LSTM Networks

While trying to find out how bidirectional LSTM's work, we happened on this article about protein structure prediction. Fascinating though protein structures may be, we're more interested in the good description of the BLSTM algorithm.

Prediction of protein secondary structure is one of the classical problems in bioinformatics, which is still far from being satisfactorily solved. The Long Short-Term Memory(LSTM) algorithm is a very general and promising adaptive sequence processing device with a wide field of potential applications. In this paper, we proposed a Bidirectional LSTM(BLSTM) for PSS prediction and obtained encouraging results on the RS126 and CB396 sets.

Chapter III.A describes the inner workings of a bidirectional LSTM quite clearly.

- Jasper

Design choices

As previously said, we'll be making an LSTM that will learn to recognize and classify phonemes in a speech signal. The network will be trained, retrained and tested on a corpus of annotated wave files. There is no reason why the same network should not be able to perform on real-time speech signal at some later junction, but that is not the focus of our project.

Our professor, Dr. Erwin M. Bakker, has acquired the TIMIT corpus, a large body of annotated speech signal. We'll be using a rounded subset of this corpus as our training data, and a smaller non-overlapping subset as our test set. Some recent articles have suggested that LSTM's can easily be retrained on different sets of speech data. If time allows, we will run some experiments using retraining.

Our intention is to use MATLAB for preprocessing and feature extraction on the wave input files, and store the resulting MFCC to file. The LSTM will be created in C++. It will use the feature file and the corresponding annotation as input. During training, the annotation will be used to teach the network. During testing, the annotation will be used to determine the performance of the network.

Over the weekend, we will be preparing a presentation that details this design.

- Jasper

Welcome!

Welcome to our blog!

We are Timo de Vries and Jasper A. Visser, students at the Leiden Institute of Advanced Computer Science (LIACS). As part of the seminar Speech Recognition, we are implementing a neural network solution to classify phonemes from speech input.

Specifically, we have chosen to use a Long Short-Term Memory recurrent network, which has been proven to do well in this particular area. Over the course of the next months, we will be updating this blog with articles we have found to be useful, progress updates on our project, and other useful information.

- Jasper

Framewise Phoneme Classification with Bidirectional LSTM and Other Neural Network Architectures

For the Speech Recognition seminar, we have read some articles about phoneme recognition. The above article gave us the idea to build a Long Short Term Memory network that can be trained to classify phonemes accurately.

In this paper, we present bidirectional Long Short Term Memory (LSTM) networks, and a modified, full gradient version of the LSTM learning algorithm. We evaluate Bidirectional LSTM (BLSTM) and several other network architectures on the benchmark task of framewise phoneme classification, using the TIMIT database. Our main findings are that bidirectional networks outperform unidirectional ones, and Long Short Term Memory (LSTM) is much faster and also more accurate than both standard Recurrent Neural Nets (RNNs) and time-windowed Multilayer Perceptrons (MLPs). Our results support the view that contextual information is crucial to speech processing, and suggest that BLSTM is an effective architecture with which to exploit it.

The article can be found here.

- Timo

Learning to Forget: Continual Prediction with LSTM

Another excellent article on the subject of LSTM's, written by Gers, Schmidhuber & Cummins, addresses the problem of saturation of the CEC of a memory cell and how it can be combatted using forget gates.

Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.

There's plenty of math included, which will be very helpful for our own implementation. The article also describes the topology of the network used in the test setup. Previously read articles were not too clear about exactly which units were connected to the input & output gates.

The seven input units are fully connected to a hidden layer consisting of four memory blocks with 2 cells each (8 cells and 12 gates in total). The cell outputs are fully connected to the cell inputs, all gates, and the seven output units. The output units have additional "shortcut" connections from the input units.

Click here for the full text

- Jasper