LSTM and GRU

Contents

LSTM

The avoid the problem of vanishing gradient and exploding gradient in vanilla RNN, LSTM was published, which can remember information for longer periods of time.

Here is the structure of LSTM:

The calculate procedure are:

ft=σ(Wf⋅[ht−1,xt]+bf)it=σ(Wi⋅[ht−1,xt]+bi)ot=σ(Wo⋅[ht−1,xt]+bo)Ct~=tanh(WC⋅[ht−1,xt]+bC)Ct=ft∗Ct−1+it∗Ct~ht=ot∗tanh(Ct)\begin{aligned} f_t&=\sigma(W_f\cdot[h_{t-1},x_t]+b_f)\\ i_t&=\sigma(W_i\cdot[h_{t-1},x_t]+b_i)\\ o_t&=\sigma(W_o\cdot[h_{t-1},x_t]+b_o)\\ \tilde{C_t}&=tanh(W_C\cdot[h_{t-1},x_t]+b_C)\\ C_t&=f_t\ast C_{t-1}+i_t\ast \tilde{C_t}\\ h_t&=o_t \ast tanh(C_t) \end{aligned}

ftf_t,iti_t,oto_t are forget gate, input gate and output gate respectively. Ct~\tilde{C_t} is the new memory content. CtC_t is cell state. hth_t is the output.

Use ftf_t and iti_t to update CtC_t, use oto_t to decide which part of hidden state should be outputted.

GRU

zt=σ(Wz⋅[ht−1,xt])rt=σ(Wr⋅[ht−1,xt])ht~=tanh(W⋅[rt∗ht−1,xt])ht=(1−zt)∗ht−1+zt∗ht~\begin{aligned} z_t&=\sigma(W_z\cdot[h_{t-1},x_t])\\ r_t&=\sigma(W_r\cdot[h_{t-1},x_t])\\ \tilde{h_t}&=tanh(W\cdot[r_t \ast h_{t-1},x_t])\\ h_t&=(1-z_t)\ast h_{t-1}+z_t \ast \tilde{h_t} \end{aligned}

ztz_t is update gate, rtr_t is reset gate, ht~\tilde{h_t} is candidate activation, hth_t is activation.

Compare with LSTM, GRU merge cell state and hidden state to one hidden state, and use ztz_t to decide how to update the state rather than ftf_t and iti_t.

Ref

  1. Understanding LSTM Networks