Time & Sequences (RNNs)
The architecture that gave AI memory — before someone found a better way.
Every architecture you have encountered so far sees data as a snapshot — all at once, frozen in time. But language unfolds word by word, music note by note, and stock prices tick by tick. Rearrange the elements and the meaning changes.
Recurrent Neural Networks were the first architecture to give neural networks something resembling memory — a loop that carries information from one step to the next. This article explains how that loop works, why it inevitably fails on long sequences, and how LSTMs provided a decade-long solution before Transformers changed everything.
Why Order Matters — The Sequential Data Problem
Definition
Sequential Data & Recurrence
Analogy
Example
Definition
Text, time series, and audio — three domains that all share the same property: the order of elements determines meaning.
Sees all inputs simultaneously. Given the words as a bag of words, it has no concept of order. Then it cannot distinguish 'The dog bites the man' from 'The man bites the dog.'
Processes inputs step by step. The hidden state carries context from previous steps forward. Order is preserved.
'The dog bites the man' vs. 'The man bites the dog' — identical vocabulary, reversed meaning. A feedforward network treating each word independently cannot distinguish these sentences because it sees the same bag of words. An RNN processes left-to-right: by the time it reaches 'bites,' it already knows whether 'dog' or 'man' came first.
Caution
Misconception: CNNs Can Handle Sequences Too
Partial truth: 1D CNNs detect local sequential patterns effectively, but their receptive field is limited. For long-range dependencies (connecting word 1 to word 200), recurrence or attention mechanisms are necessary.
The Recurrence Loop — How RNNs Remember
Definition
Hidden State
Analogy
Example
Definition
The core idea of an RNN is simple: use the same weights at every time step and carry a hidden state from step to step. This enables processing sequences of arbitrary length.
- 1. x₁ → h₁First input produces first hidden state
- 2. x₂ + h₁ → h₂Second input merges with memory from step 1 (the + does not mean simple addition — the network combines both through learned weights)
- 3. x₃ + h₂ → h₃Third input merges with accumulated context
- 4. hₜ → OutputFinal hidden state yields the prediction
Parameter sharing: The same weight matrices W_h and W_x are reused at EVERY time step. This means: an RNN can process sequences of any length without the parameter count growing.
The Vanishing Gradient Problem
To train an RNN, the network is 'unrolled' through time and gradients are propagated backward through the entire chain — Backpropagation Through Time (BPTT). The chain rule is applied at every time step, which means multiplying by the same weight matrix. If the eigenvalues of this matrix are below 1 in magnitude, the gradient shrinks exponentially toward zero.
'I grew up in France. [...100 words...] I speak fluent ___.' The correct answer is 'French,' but the signal from 'France' must survive 100+ time steps. With a factor of 0.9 per step: 0.9¹⁰⁰ ≈ 0.0000266 — practically zero. The network cannot learn the connection.
Caution
Misconception: RNNs Remember Everything
No! The hidden state is a fixed-size vector. It compresses all past information into a finite representation. As new inputs arrive, older information is progressively overwritten. Even without vanishing gradients, the hidden state is a lossy summary, not a perfect recording.
The gradient for the first time step is: ∂L/∂h₁ = ∂L/∂h_T · ∏(∂h_t/∂h_{t-1}) for t=2..T. Each factor contains the weight matrix W_h. If the eigenvalues of W_h are less than 1 in magnitude, the product converges exponentially to zero. If greater than 1, it can explode — gradient clipping caps the gradient at a threshold. LSTMs bypass this through additive updates on the cell state instead of multiplicative chaining.
Interactive: RNN Step by Step
Click through the individual time steps of an unrolled RNN. Observe how the hidden state flows from cell to cell, absorbing new input information at each step. Pay special attention to the fact that all cells share the same weights.
An RNN shown unrolled through time: 4 time steps, each with its own input x. The hidden state h flows from left to right, carrying context forward.
Gates to the Rescue — LSTM and GRU
The Long Short-Term Memory (LSTM) network, invented in 1997 by Hochreiter and Schmidhuber (the forget gate was added in 1999 by Gers et al.), replaces the simple hidden state update with a memory cell controlled by three learned gates.
LSTM Cell: Information Flow Through the Gates
Sigmoid: Decides what to erase from long-term memory (0 = forget, 1 = keep)
Sigmoid: Decides what new information to write into memory
Sigmoid: Decides what part of memory to expose as the current hidden state
Long-term memory: information flows nearly unchanged (Constant Error Carousel)
Think of the LSTM cell as a secure filing cabinet with three locks. The Forget Gate lock lets you shred old documents that are no longer relevant. The Input Gate lock lets you file new important documents. The Output Gate lock controls which documents you take out to work with right now. The crucial insight: you can leave documents in the cabinet untouched for as long as they remain relevant — they do not decay over time like a vanilla RNN's hidden state.
The Constant Error Carousel — the cell state — allows gradients to flow nearly unchanged through the cell. Because the forget gate can learn to keep its values near 1.0, the gradient is not exponentially attenuated.
GRU: The Simplified Alternative
The Gated Recurrent Unit (GRU), introduced in 2014 by Cho et al., simplifies the LSTM architecture to two gates (Update and Reset) and achieves comparable performance with fewer parameters.
RNN Concept
Rumelhart et al. show how to unroll recurrent networks in time and train them with backpropagation
Vanishing Gradient Identified
Hochreiter shows in his diploma thesis that gradients vanish exponentially in deep recurrent networks
LSTM
Hochreiter & Schmidhuber invent Long Short-Term Memory — gates control information flow
GRU
Cho et al. simplify LSTM to two gates — comparable performance with fewer parameters
Google NMT
Google Translate switches to LSTM-based Neural Machine Translation — dramatic quality improvement
Transformers
Vaswani et al. replace recurrence with Self-Attention — parallel processing of all positions
Real-World Example: Google Translate
Before 2016, Google Translate used phrase-based statistical models that translated sentence fragments independently — often producing grammatically incorrect output. In 2016, Google switched to Neural Machine Translation powered by deep LSTMs, producing dramatically more fluent translations. In 2017, a Google team introduced the Transformer; in 2020, Google Translate replaced its LSTM system with a model using a Transformer encoder (the decoder remained an RNN), adding another leap — foreshadowing the next article in your learning path. Since December 2025, Google Translate has also used Gemini, a large Transformer-based language model, for text translation, starting in the US and India between English and nearly 20 languages.
Attention Mechanism: The Key to Modern LLMs
September 2014: Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio published a paper that would lastingly change the NLP world. 'Neural Machine Translation by Jointly Learning to Align and Translate' solved a fundamental problem in sequence-to-sequence models. Previous encoder-decoder architectures compressed every input sentence into a single fixed-length vector — an information bottleneck for long sentences. Bahdanau attention was a significant step forward: instead of a fixed vector, the model used dynamic attention over different parts of the input sentence. Like the human eye jumping while reading, AI attention moves between relevant words. This 'additive attention' became the conceptual precursor to modern NLP systems. The later Transformer (2017) built on the attention idea, but replaced the additive variant with the more efficient Scaled Dot-Product Attention. Without Bahdanau's attention concept, no Transformers; without Transformers, no GPT family or BERT. This breakthrough happened three years before 'Attention Is All You Need.'
Caution
Misconception: LSTMs Have Infinite Memory
No! LSTMs forget too — but in a controlled, learned way rather than the uncontrolled exponential decay of vanilla RNNs. The Forget Gate actively decides what to discard. Over very long sequences, even LSTMs struggle with very distant dependencies.
Despite their success, both architectures retain the fundamental sequential processing constraint: each step must wait for the previous step to complete, making parallelization across time steps on GPUs impossible. This sequential bottleneck directly motivated the Transformer architecture — the next article in your learning path.
Transformers have quadratic cost with sequence length (O(n²) attention). State Space Models like Mamba (2023) achieve linear scaling through a learned recurrence. Streaming applications (real-time audio, sensor monitoring) where latency is critical benefit from RNN-like architectures. Hybrid architectures combine attention and recurrence: in the Qwen3.5 models (2026), for example, every three recurrent layers (Gated DeltaNet) are followed by one attention layer. The key message: the recurrence principle is not dead — it has evolved.
Note
Key Takeaways
- Sequential data carries meaning in order — feedforward networks have no built-in concept of it (as a bag of words, the order is lost entirely), while RNNs process step-by-step with a hidden state that accumulates context.
- The Vanishing Gradient Problem is not a bug but a mathematical inevitability: repeated multiplication through long chains causes gradients to shrink exponentially, erasing long-range memory.
- LSTMs solve this with gates that learn WHAT to remember and forget — but their strictly sequential processing makes them slow, setting the stage for Transformers.
Check Your Understanding
Can you explain this in your own words? Tick what you've got.
0 of 3 done
Check
Quiz: Recurrent Neural Networks
What is the primary purpose of the hidden state in a Recurrent Neural Network?
Select one answer
1. What is the primary purpose of the hidden state in a Recurrent Neural Network?
2. Why do standard (vanilla) RNNs struggle with long sequences?
3. An RNN processes a 200-word sentence. At each time step, the gradient is multiplied by a factor of 0.95. What is the approximate gradient magnitude reaching the first word (0.95²⁰⁰)?
4. In an LSTM cell, the Forget Gate outputs [0.1, 0.9, 0.0, 1.0] for a four-dimensional cell state. What happens to each dimension?
5. Google Translate switched from phrase-based models to LSTM-based NMT in 2016; in 2017, a Google team introduced the Transformer. What fundamental RNN limitation motivated this new architecture?
6. A colleague claims: 'Since Transformers have replaced RNNs in NLP, recurrent architectures are completely obsolete.' Why is this statement an oversimplification?