Recurrent Neural Networks (RNNs): How Sequence Memory Works, Plus LSTM and GRU

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

A recurrent neural network (RNN) processes ordered data one step at a time while carrying a learned hidden state from the previous step. That state lets earlier inputs influence later outputs. Vanilla RNNs provide the basic idea, while LSTM and GRU cells add gates to make long-range dependencies easier to learn.

What is a recurrent neural network?

A recurrent neural network is a neural-network family designed for data in which order matters. Text, speech, sensor readings, financial time series and event logs are sequences: changing the order changes the meaning or the prediction problem. Instead of treating every item as independent, an RNN updates an internal state as it reads each item.

#	Preview	Product	Price
1		Recurrent Neural Networks: From Simple to Gated Architectures (Engineering)	$39.99	Buy on Amazon
2		Convolutional Neural Networks and Recurrent Neural Networks: Convolutional Neural Networks and...	$1.99	Buy on Amazon
3		Neural Networks and Deep Learning: A Textbook	$61.95	Buy on Amazon
4		Neural Network Design (2nd Edition)	$36.50	Buy on Amazon
5		Neural Networks with R: Smart models using CNN, RNN, deep learning, and artificial intelligence...	$45.99	Buy on Amazon

At time step t, the model receives the current input x_t and the previous hidden state h_(t-1). A learned transition produces the new state:

h_t = f(x_t, h_(t-1))

An output can then be calculated from h_t, for example a probability distribution over the next word or a class label for the sequence. The same transition parameters are reused at every time step. This parameter sharing is what makes the network recurrent and allows it to process sequences of different lengths.

#1 Best Overall

Sale

Recurrent Neural Networks: From Simple to Gated Architectures (Engineering)

$39.99

Buy on Amazon

The hidden state is a compact, imperfect summary

The hidden state is not a searchable archive containing every previous input. It is a fixed-size, learned representation that preserves whatever context helps the training objective. Information can be compressed, overwritten or lost, especially when a sequence is long. Saying that an RNN “remembers” an earlier word therefore means that traces of that word can influence the current state, not that the model stores an exact copy indefinitely.

Why order changes the computation

Suppose a model reads the tokens “the animal did not cross the road because it was …”. The interpretation of a later token depends on the preceding sequence. A feed-forward model that sees only the current token has no built-in path for that context. An RNN carries a state from “the” through each subsequent step, giving later computations access to a learned summary of what came before.

How does an RNN remember previous inputs?

During inference, the recurrence is applied from left to right (or in the chosen sequence direction):

1. Initialize a hidden state, commonly with zeros or a learned initial value.
2. Read the first input and combine it with the initial state to obtain h_1.
3. Feed the next input and h_1 into the same cell to obtain h_2.
4. Continue until the sequence ends; use one state, every state, or a separate output head according to the task.

For next-step language modeling, the output at step t can predict item t+1. For sequence classification, the final state may feed a classifier. For sequence labeling, each state can produce a label for its corresponding input, such as a part-of-speech tag.

Unrolling through time

It is useful to draw one recurrent cell repeatedly across the sequence. The unrolled diagram has one copy per time step, but all copies share weights. This view explains both the efficiency of parameter sharing and the difficulty of training: an early state influences many later states through a long chain of operations.

Rank #2

Convolutional Neural Networks and Recurrent Neural Networks: Convolutional Neural Networks and Recurrent Neural Networks

$1.99

Buy on Amazon

Backpropagation through time

Training calculates a loss from the model’s outputs and propagates gradients backward through the unrolled steps. This procedure is called backpropagation through time. The gradient reaching an early input is a product of many derivatives and recurrent weight transformations. Repeated multiplication can make the signal shrink toward zero or grow very large, depending on the values encountered along the path.

Why do RNNs have a vanishing-gradient problem?

A vanishing gradient becomes extremely small as it travels through many recurrent steps. Parameters connected to distant history then receive little useful learning signal, so the model struggles to discover relationships separated by many time steps. An exploding gradient is the opposite: repeated multiplication makes updates enormous, which can produce unstable training or numerical overflow.

These are optimization problems, not a statement that an RNN cannot produce a long output. A vanilla cell may run over thousands of steps, yet still fail to learn that an event near the beginning should affect a decision near the end. Stanford’s recurrent-network teaching materials identify vanishing gradients as a central issue in recurrent models.

What the symptoms look like

• Training improves on short sequences but performance falls as the relevant context moves farther back.
• The model behaves as if it relies mostly on recent inputs, even when earlier information is important.
• Loss or gradient norms can become erratic when gradients explode.
• Increasing the hidden size does not necessarily solve the underlying long-range credit-assignment problem.

Common engineering responses include choosing a gated cell, limiting the length of backpropagation windows, and clipping unusually large gradients. These measures change the optimization behavior; they do not turn the hidden state into perfect long-term memory.

What is a vanilla RNN cell?

A vanilla RNN is the simplest recurrent design. A typical cell applies an affine transformation to the current input and previous hidden state, then a nonlinear activation such as tanh:

Rank #3

Sale

Neural Networks and Deep Learning: A Textbook

$61.95

Buy on Amazon

h_t = tanh(W_x x_t + W_h h_(t-1) + b)

An output layer maps the state to the required prediction. This architecture is valuable as a teaching baseline because the state update is easy to inspect and the parameter count is modest. Its weakness is that every step must carry useful information through the same recurrent pathway, leaving it exposed to vanishing and exploding gradients over long intervals.

How does an LSTM help?

Long short-term memory (LSTM) adds a separate cell state and learned gates that regulate information flow. The cell state provides a path intended to let information and gradients travel across many steps, while gates decide what to discard, write and expose.

The three main gates

• Forget gate: chooses which parts of the previous cell state to retain.
• Input gate: controls how much newly computed candidate information is written into the cell state.
• Output gate: controls how much of the updated cell state becomes the hidden state visible to the next layer or output head.

In common notation, the gates use sigmoid functions and the candidate content uses a hyperbolic tangent. A simplified sequence of equations is:

f_t = sigmoid(W_f [h_(t-1), x_t] + b_f)
i_t = sigmoid(W_i [h_(t-1), x_t] + b_i)
c~_t = tanh(W_c [h_(t-1), x_t] + b_c)
c_t = f_t ⊙ c_(t-1) + i_t ⊙ c~_t
o_t = sigmoid(W_o [h_(t-1), x_t] + b_o)
h_t = o_t ⊙ tanh(c_t)

Notation and parameter layouts vary by framework, but the concept is consistent: the cell state c_t is updated additively, and gates learn when information should persist or change.

Rank #4

Neural Network Design (2nd Edition)

$36.50

Buy on Amazon

What the original LSTM evidence shows

Sepp Hochreiter and Jürgen Schmidhuber introduced LSTM to address decaying error flow in recurrent backpropagation. Their 1997 paper states: “Learning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow.” On artificial data, the proposed method learned minimal time lags in excess of 1,000 discrete-time steps. That is an experimental result under the paper’s setup, not a universal guarantee that every LSTM application can preserve arbitrary-length information.

What is a GRU?

A gated recurrent unit (GRU) is another recurrent cell designed to control information flow with fewer distinct state components than an LSTM. In the documented PyTorch implementation, a reset gate and an update gate are computed from the current input and previous hidden state.

A representative form of the PyTorch equations is:

r_t = sigmoid(W_ir x_t + b_ir + W_hr h_(t-1) + b_hr)
z_t = sigmoid(W_iz x_t + b_iz + W_hz h_(t-1) + b_hz)
n_t = tanh(W_in x_t + b_in + r_t ⊙ (W_hn h_(t-1) + b_hn))
h_t = (1 − z_t) ⊙ n_t + z_t ⊙ h_(t-1)

• Reset gate r_t: controls how much of the previous state contributes to the candidate state.
• Update gate z_t: blends the candidate state with the old hidden state.

PyTorch notes that its new-gate calculation differs subtly from the original GRU paper and from some other frameworks. When reproducing equations, parameter counts or checkpoints, identify the implementation rather than assuming that every library orders the operations identically.

RNN vs. LSTM vs. GRU: what is the difference?

There is no source-supported rule that GRU is always faster or that LSTM is always more accurate. Compare candidate cells on your own validation splits, using the same preprocessing, sequence windows, hidden width, training budget and deployment hardware. Include latency, memory, parameter count and quality in the decision, not just a single test score.

Best Value

Neural Networks with R: Smart models using CNN, RNN, deep learning, and artificial intelligence principles

$45.99

Buy on Amazon

What are bidirectional and stacked RNNs?

Bidirectional RNNs

A bidirectional RNN runs one recurrent network from the beginning of the sequence and another from the end. Their states are combined so a prediction at a position can use both left and right context. This is useful for offline tasks such as labeling a complete transcript or document.

The trade-off is information availability. A bidirectional model needs future inputs. It is therefore unsuitable for next-token generation, live speech recognition with strict real-time constraints, or any prediction made before the rest of the sequence arrives.

Stacked or deep RNNs

A stacked RNN places recurrent layers on top of one another. The first layer can capture local temporal patterns while higher layers form more abstract sequence features. Additional layers increase representational capacity, but also add computation, memory use and optimization complexity. Validate whether the extra layer improves held-out performance rather than assuming depth will help.

How do RNN encoder-decoder models work?

In an encoder-decoder sequence-to-sequence model, one recurrent network reads the input symbols and compresses them into a fixed-length vector. A second recurrent network, the decoder, uses that representation to generate the target sequence one symbol at a time.

1. The encoder consumes each source item and updates its hidden state.
2. At the end of the source sequence, the encoder state (or states) initializes or conditions the decoder.
3. The decoder predicts a target item, feeds the appropriate next input, and repeats until an end marker or length limit.

Cho and colleagues introduced this formulation for statistical machine translation. Their paper reported that phrase-pair conditional probabilities from the encoder-decoder improved an existing log-linear translation model when added as a feature. That is a historical, paper-specific result; it is not evidence that an RNN encoder-decoder is the best modern machine-translation architecture.

When should you use an LSTM or GRU?

Start with the task constraints rather than the name of a cell.

Choose a vanilla RNN when simplicity is the priority

• You need a transparent baseline for an experiment or course project.
• The meaningful context is short enough that long-range credit assignment is unlikely to dominate.
• You want to isolate data or training issues before adding gating.

Test an LSTM when persistent context is important

• Evidence suggests that events separated by many steps affect the target.
• You can afford the extra state and computation.
• You want explicit separation between a long-lived cell state and the exposed hidden state.

Test a GRU when a gated alternative fits the budget

• You want gating without adopting the LSTM’s separate cell-state structure.
• Memory, latency or parameter limits make a smaller recurrent design attractive.
• You will verify behavior against the exact framework implementation used in production.

For any choice, establish a validation protocol first. Hold out sequences by time when forecasting, prevent information from future records leaking into features, and report the metric that matches the decision: token cross-entropy, classification F1, forecasting error, latency or another task-specific measure.

A practical RNN implementation checklist

1. Define the sequence unit. Decide whether one step is a token, audio frame, sensor reading, transaction or another event.
2. Set the context window. Choose the maximum number of steps used for training and document how longer sequences are split or truncated.
3. Handle padding and masks. If sequences have different lengths, ensure padded positions do not contribute to the loss or evaluation.
4. Choose the output shape. Use one output for sequence classification, one output per step for labeling, or autoregressive outputs for generation.
5. Initialize and monitor training. Track loss, validation metrics and gradient behavior; unstable or non-learning runs can indicate exploding or vanishing gradients.
6. Compare cells fairly. Keep data preparation, hidden width, number of layers, optimizer budget and stopping criteria comparable when evaluating vanilla RNN, LSTM and GRU.
7. Check deployment direction. A bidirectional model requires future context, while a streaming system generally needs a causal, left-to-right model.

Further reading

Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville is written for students and practitioners. Its Chapter 10, “Sequence Modeling: Recurrent and Recursive Nets,” provides broader mathematical and architectural context for recurrent models.

Quick Recap

SaleBestseller No. 1

Recurrent Neural Networks: From Simple to Gated Architectures (Engineering)

$39.99

Bestseller No. 2

Convolutional Neural Networks and Recurrent Neural Networks: Convolutional Neural Networks and Recurrent Neural Networks

$1.99

SaleBestseller No. 3

Neural Networks and Deep Learning: A Textbook

$61.95

Bestseller No. 4

Neural Network Design (2nd Edition)

$36.50

Bestseller No. 5

Neural Networks with R: Smart models using CNN, RNN, deep learning, and artificial intelligence principles

$45.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.