Back to Learning Area

Time & Sequences (RNNs)

The architecture that gave AI memory — before someone found a better way.

  • Architectures
  • 14 min
  • Expert

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: RNNs: neural networks with memory

What is an RNN?

An RNN is a neural network with memory. RNN stands for recurrent neural network. It processes data in order. Each step uses the result of the previous step.

The forgetting problem

RNNs have a problem with long texts. The problem is called vanishing gradient. This means: early information gets lost. The network forgets what was at the beginning.

LSTMs: memory with control

LSTMs solve the problem. LSTM means: memory cells with control. An LSTM has three switches. These switches are called gates. Gates decide: what stays? What is forgotten?

Every architecture you have encountered so far sees data as a snapshot — all at once, frozen in time. But language unfolds word by word, music note by note, and stock prices tick by tick. Rearrange the elements and the meaning changes.

Recurrent Neural Networks were the first architecture to give neural networks something resembling memory — a loop that carries information from one step to the next. This article explains how that loop works, why it inevitably fails on long sequences, and how LSTMs provided a decade-long solution before Transformers changed everything.

Why Order Matters — The Sequential Data Problem

Definition

Sequential Data & Recurrence

Analogy

Imagine someone rips out all the pages of a detective novel and hands them to you in random order. You have all the same words and clues — but the mystery is unsolvable. The revelation on page 200 only makes sense because you read pages 1 through 199 first. A feedforward network reads the shuffled stack. An RNN reads page by page, in order.

Example

A detective novel follows a plot structure planned by an author. Real-world sequential data (e.g., stock prices) has no planned narrative — but the temporal dependency is real.

Text, time series, and audio — three domains that all share the same property: the order of elements determines meaning.

Feedforward Network

Sees all inputs simultaneously. Given the words as a bag of words, it has no concept of order. Then it cannot distinguish 'The dog bites the man' from 'The man bites the dog.'

Recurrent Neural Network

Processes inputs step by step. The hidden state carries context from previous steps forward. Order is preserved.

'The dog bites the man' vs. 'The man bites the dog' — identical vocabulary, reversed meaning. A feedforward network treating each word independently cannot distinguish these sentences because it sees the same bag of words. An RNN processes left-to-right: by the time it reaches 'bites,' it already knows whether 'dog' or 'man' came first.

Caution

Misconception: CNNs Can Handle Sequences Too

Partial truth: 1D CNNs detect local sequential patterns effectively, but their receptive field is limited. For long-range dependencies (connecting word 1 to word 200), recurrence or attention mechanisms are necessary.

The Recurrence Loop — How RNNs Remember

Definition

Hidden State

Analogy

Imagine a factory conveyor belt where each station (time step) receives a product and adds its own modification based on the product's current state AND what the previous station passed along (the hidden state). The work instructions (weights) are identical at every station. Moving forward along the belt, we build the product step by step — that is the forward pass. But when the quality inspector (gradient) traces a defect backward through 100 stations, the feedback signal fades at each relay — like a game of telephone where the message becomes quieter with each pass.

Example

On a conveyor belt, the physical product is altered. In an RNN, the original input remains unchanged — the hidden state is a mathematical summary, not a physical copy.

The core idea of an RNN is simple: use the same weights at every time step and carry a hidden state from step to step. This enables processing sequences of arbitrary length.

  1. 1. x₁ → h₁First input produces first hidden state
  2. 2. x₂ + h₁ → h₂Second input merges with memory from step 1 (the + does not mean simple addition — the network combines both through learned weights)
  3. 3. x₃ + h₂ → h₃Third input merges with accumulated context
  4. 4. hₜ → OutputFinal hidden state yields the prediction
hₜ = f(Wₕ · hₜ₋₁ + Wₓ · xₜ + b)
Hidden State Update Equation

Parameter sharing: The same weight matrices W_h and W_x are reused at EVERY time step. This means: an RNN can process sequences of any length without the parameter count growing.

The Vanishing Gradient Problem

To train an RNN, the network is 'unrolled' through time and gradients are propagated backward through the entire chain — Backpropagation Through Time (BPTT). The chain rule is applied at every time step, which means multiplying by the same weight matrix. If the eigenvalues of this matrix are below 1 in magnitude, the gradient shrinks exponentially toward zero.

'I grew up in France. [...100 words...] I speak fluent ___.' The correct answer is 'French,' but the signal from 'France' must survive 100+ time steps. With a factor of 0.9 per step: 0.9¹⁰⁰ ≈ 0.0000266 — practically zero. The network cannot learn the connection.

Caution

Misconception: RNNs Remember Everything

No! The hidden state is a fixed-size vector. It compresses all past information into a finite representation. As new inputs arrive, older information is progressively overwritten. Even without vanishing gradients, the hidden state is a lossy summary, not a perfect recording.

The gradient for the first time step is: ∂L/∂h₁ = ∂L/∂h_T · ∏(∂h_t/∂h_{t-1}) for t=2..T. Each factor contains the weight matrix W_h. If the eigenvalues of W_h are less than 1 in magnitude, the product converges exponentially to zero. If greater than 1, it can explode — gradient clipping caps the gradient at a threshold. LSTMs bypass this through additive updates on the cell state instead of multiplicative chaining.

Interactive: RNN Step by Step

Click through the individual time steps of an unrolled RNN. Observe how the hidden state flows from cell to cell, absorbing new input information at each step. Pay special attention to the fact that all cells share the same weights.

x₀x₁x₂x₃h₋₁h₀h₁h₂h₃RNNRNNRNNRNN
Step 1 / 7Overview: Unrolled RNN

An RNN shown unrolled through time: 4 time steps, each with its own input x. The hidden state h flows from left to right, carrying context forward.

Controls
Steps

Gates to the Rescue — LSTM and GRU

The Long Short-Term Memory (LSTM) network, invented in 1997 by Hochreiter and Schmidhuber (the forget gate was added in 1999 by Gers et al.), replaces the simple hidden state update with a memory cell controlled by three learned gates.

LSTM Cell: Information Flow Through the Gates

Forget Gate

Sigmoid: Decides what to erase from long-term memory (0 = forget, 1 = keep)

Input Gate

Sigmoid: Decides what new information to write into memory

Output Gate

Sigmoid: Decides what part of memory to expose as the current hidden state

Cell State

Long-term memory: information flows nearly unchanged (Constant Error Carousel)

Think of the LSTM cell as a secure filing cabinet with three locks. The Forget Gate lock lets you shred old documents that are no longer relevant. The Input Gate lock lets you file new important documents. The Output Gate lock controls which documents you take out to work with right now. The crucial insight: you can leave documents in the cabinet untouched for as long as they remain relevant — they do not decay over time like a vanilla RNN's hidden state.

The Constant Error Carousel — the cell state — allows gradients to flow nearly unchanged through the cell. Because the forget gate can learn to keep its values near 1.0, the gradient is not exponentially attenuated.

GRU: The Simplified Alternative

The Gated Recurrent Unit (GRU), introduced in 2014 by Cho et al., simplifies the LSTM architecture to two gates (Update and Reset) and achieves comparable performance with fewer parameters.

1986

RNN Concept

Rumelhart et al. show how to unroll recurrent networks in time and train them with backpropagation

1991

Vanishing Gradient Identified

Hochreiter shows in his diploma thesis that gradients vanish exponentially in deep recurrent networks

1997

LSTM

Hochreiter & Schmidhuber invent Long Short-Term Memory — gates control information flow

2014

GRU

Cho et al. simplify LSTM to two gates — comparable performance with fewer parameters

2016

Google NMT

Google Translate switches to LSTM-based Neural Machine Translation — dramatic quality improvement

2017

Transformers

Vaswani et al. replace recurrence with Self-Attention — parallel processing of all positions

Real-World Example: Google Translate

Before 2016, Google Translate used phrase-based statistical models that translated sentence fragments independently — often producing grammatically incorrect output. In 2016, Google switched to Neural Machine Translation powered by deep LSTMs, producing dramatically more fluent translations. In 2017, a Google team introduced the Transformer; in 2020, Google Translate replaced its LSTM system with a model using a Transformer encoder (the decoder remained an RNN), adding another leap — foreshadowing the next article in your learning path. Since December 2025, Google Translate has also used Gemini, a large Transformer-based language model, for text translation, starting in the US and India between English and nearly 20 languages.

Caution

Misconception: LSTMs Have Infinite Memory

No! LSTMs forget too — but in a controlled, learned way rather than the uncontrolled exponential decay of vanilla RNNs. The Forget Gate actively decides what to discard. Over very long sequences, even LSTMs struggle with very distant dependencies.

Despite their success, both architectures retain the fundamental sequential processing constraint: each step must wait for the previous step to complete, making parallelization across time steps on GPUs impossible. This sequential bottleneck directly motivated the Transformer architecture — the next article in your learning path.

Transformers have quadratic cost with sequence length (O(n²) attention). State Space Models like Mamba (2023) achieve linear scaling through a learned recurrence. Streaming applications (real-time audio, sensor monitoring) where latency is critical benefit from RNN-like architectures. Hybrid architectures combine attention and recurrence: in the Qwen3.5 models (2026), for example, every three recurrent layers (Gated DeltaNet) are followed by one attention layer. The key message: the recurrence principle is not dead — it has evolved.

Note

Key Takeaways

  • Sequential data carries meaning in order — feedforward networks have no built-in concept of it (as a bag of words, the order is lost entirely), while RNNs process step-by-step with a hidden state that accumulates context.
  • The Vanishing Gradient Problem is not a bug but a mathematical inevitability: repeated multiplication through long chains causes gradients to shrink exponentially, erasing long-range memory.
  • LSTMs solve this with gates that learn WHAT to remember and forget — but their strictly sequential processing makes them slow, setting the stage for Transformers.

Check Your Understanding

Can you explain this in your own words? Tick what you've got.

0 of 3 done

Check

Quiz: Recurrent Neural Networks

Not completed

What is the primary purpose of the hidden state in a Recurrent Neural Network?

Select one answer

Answer Key: 1) B · 2) C · 3) B · 4) B · 5) B · 6) C