Moderate Cheers

We got the hang of the backpropagation formulas, with a little help of one of our professors, dr. W.A. Kosters, and were able to implement most of them. The one part we haven't implemented yet, updating of weights, we have a firm understanding of what we're supposed to put down.

Of course, our code is as yet mostly untested, and we won't be able to test it until the whole network is complete.

We've also selected a subset of 380 utterances from the TIMIT corpus and identified 341 of them which aren't corrupted after conversion.

Remaining tasks:

• Select files to use from TIMIT
  
• Convert WAV files
  
• Matlab code for extracting features from WAV files
• Matlab code for writing features into feature files
• C++ code for reading feature files
• C++ code for reading TIMIT annotation files
  
• C++ code object skeleton for BLSTM
• C++ code for forward propagation
• C++ code for back propagation
  
• C++ code for BLSTM trainer
• C++ code for BLSTM tester
• C++ code for serializing BLSTM
• Training the network on selected files
• Testing the network on selected files
• Write report

Serialization

Short status update: serialization of the network has been implemented and tested.

Remaining tasks:

• Select files to use from TIMIT
  
• Convert WAV files
  
• Matlab code for extracting features from WAV files
• Matlab code for writing features into feature files
• C++ code for reading feature files
• C++ code for reading TIMIT annotation files
  
• C++ code object skeleton for BLSTM
• C++ code for forward propagation
• C++ code for back propagation
  
• C++ code for BLSTM trainer
• C++ code for BLSTM tester
• C++ code for serializing BLSTM
• Training the network on selected files
• Testing the network on selected files
• Write report

Lower Case Sigma

Still, our main obstacle in the project is implementing back propagation. We've identified a few articles that describe the process best and provide the right formulas we need to reproduce it in our code. Currently, we're working with Long Short-Term Memory (Hochreiter & Schmidhueber, 1997). The article is extremely long, covers a lot of design choices for the LSTM, and most importantly, the appendix features all the formula's we'd need in a clear package.

We've come to the point that we understand how back propagation through time unrolls the network, and how each time step affects the time step before it. We were able to implement back propagation for the regular neurons in the network, the CEC, and the output gates. Unfortunately, we do not understand the formula's used for calculating the error/delta for the input gates and the forget gates (A.23 through A.26).

These formula's feature a symbol, the lower case sigma, that we're unsure how to read. As far as we're able to determine, the lower case sigma is only used in math to denote the standard deviation. Standard deviation does not seem appropriate in this formula.

Task Division

About time for a status update on the project. My apologies for not posting regularly. We've been doing a lot of coding, and the small victories/defeats you encounter while coding don't usually seem worth posting about.

We've divided the project into the following tasks:

• Select files to use from TIMIT
  
• Convert WAV files
  
• Matlab code for extracting features from WAV files
• Matlab code for writing features into feature files
• C++ code for reading feature files
• C++ code for reading TIMIT annotation files
  
• C++ code object skeleton for BLSTM
• C++ code for forward propagation
• C++ code for back propagation
  
• C++ code for BLSTM trainer
• C++ code for BLSTM tester
• C++ code for serializing BLSTM
• Training the network on selected files
• Testing the network on selected files
• Write report

I've indicated the status of each task by color. Red tasks are those with which we've encountered a problem that we currently do not know how to solve. Green tasks are complete, barring potential cross-task bugs. Yellow tasks are complete, but cannot be tested because they depend on unfinished tasks. All other tasks are open, usually sketched out on paper, but not yet (fully) implemented.

The main problem areas at the moment are back propagation and selecting the WAV files to be used.

You may wonder what could possibly be so difficult about selecting files. Well, the TIMIT corpus consists of a non-regular type of WAV files, the NIST format type. Our Matlab code was tuned to use PCM WAV files. We've been able to find a good tool that converts the NIST files into usable PCM files, called sox. Conversion seems to work fine. The majority of converted files sound like we expect them to sound. A few, seemingly randomly determined, files are corrupt after conversion, garbled noise. We can't be sure whether this is a problem caused by the conversion or some problem in the file that is converted, since we are unable to listen to the original.

Backpropagation through time

The pace of development has picked up. We've been writing a lot of code, mostly skeleton code for the BLSTM algorithm.

The BLSTM features two LSTM subnetworks, one of which reads the downstream context for the current frame and the other reads the upstream context. The context consists of a given number of frames. Using both downstream and upstream data allows the BLSTM to take advantage of information both forward and back in time. The output from the two context LSTM's and the current frame itself are then fed into a regular feed-forward network.

We've fully implemented the feed-forward network and laid down the skeleton for the LSTM subnetworks.

It took some reading to figure out the topology of the LSTM subnetworks. We believe the article that is our main focus for this project ( here ) implies that all gates are connected to all input nodes and to the output of each memory cell in the same layer. Similar articles have made contradictory statements. We opt to go with this choice.

At present, we are trying to figure out how backpropagation (and to a lesser degree forwardpropagation) in the LSTM works. Graves e.a. had this to say about it:

Starting at time t1, propagate the output errors backwards through the unfolded net, using the standard BPTT equations for a softmax output layer and the crossentropy error function.

There's a number of things we're not sure about at this time. Mostly we don't know exactly what is referred to as the unfolded net. We assume this refers in some way to storing the activations of all nodes for all times while forwardpropagating.

Also, the workings of BPTT (Backpropagation Through Time) are far from clear to us. There are plenty of articles on the subject on the internet, but we have yet to stumble on the one that describes the algorithm in understandable terms. Articles that show some promise of helping us get to grips are:
http://svr-www.eng.cam.ac.uk/~ajr/rnn4csr94/node14.html
http://page.mi.fu-berlin.de/~rojas/neural/chapter/K7.pdf

Presentation April 18th

The presentation of our project is due tomorrow at the LIACS. In this presentation, we set out our goals and how we intend to tackle them. The presentation in itself is not a fully readable text, but if you're interested you can download it here.

- Timo

Protein Secondary Structure Prediction with Bidirectional LSTM Networks

While trying to find out how bidirectional LSTM's work, we happened on this article about protein structure prediction. Fascinating though protein structures may be, we're more interested in the good description of the BLSTM algorithm.

Prediction of protein secondary structure is one of the classical problems in bioinformatics, which is still far from being satisfactorily solved. The Long Short-Term Memory(LSTM) algorithm is a very general and promising adaptive sequence processing device with a wide field of potential applications. In this paper, we proposed a Bidirectional LSTM(BLSTM) for PSS prediction and obtained encouraging results on the RS126 and CB396 sets.

Chapter III.A describes the inner workings of a bidirectional LSTM quite clearly.

- Jasper