Non recurrent Neural Structure for Long Long-Term Term Dependence
Abstract: In this paper, we propose a novel neural network structure, namely feedforward sequential memory networks (FSMN), to model long long-term term dependence in time series without using recurrent feedback. The proposed FSMN is a standard fully connected feedforward neural network equipped with some learnable memory blocks in its hidden layers. TThe memory blocks use a tapped-delay delay line structure to encode the long context information into a fixed fixed-size size representation as shortshort term memory mechanism which are somehow similar to the time-delay time neural networks layers. We have evaluated the FSMNs in sev several eral standard benchmark tasks, including speech recognition and language modeling. Experimental results have shown that FSMNs outperform the conventional recurrent neural networks (RNN) while can be learned much more reliably and faster in modeling sequential sequent signals like speech or language. Moreover, we also propose a compact feedforward sequential memory networks (cFSMN) by combining FSMN with lowlow rank matrix factorization and make a slight modification to the encoding method used in FSMNs in order to fur further ther simplify the network architecture. On the speech recognition Switchboard task, the proposed cFSMN structures can reduce the model size by 60% and speed up the learning by more than seven times while the model can still significantly outperform the pop popular ular bidirectional LSTMs for both frame-level cross-entropy entropy criterion criterion-based training and MMI-based based sequence training.