The architecture that gave AI memory — before someone found a better way.
Architectures 14 min Expert June 15, 2026
Every architecture you have encountered so far sees data as a snapshot — all at once, frozen in time. But language unfolds word by word, music note by note, and stock prices tick by tick. Rearrange the elements and the meaning changes.
Recurrent Neural Networks were the first architecture to give neural networks something resembling memory — a loop that carries information from one step to the next. This article explains how that loop works, why it inevitably fails on long sequences, and how LSTMs provided a decade-long solution before Transformers changed everything.
Why Order Matters — The Sequential Data Problem
Sequential Data & Recurrence
AnalogyDefinition
Imagine someone rips out all the pages of a detective novel and hands them to you in random order. You have all the same words and clues — but the mystery is unsolvable. The revelation on page 200 only makes sense because you read pages 1 through 199 first. A feedforward network reads the shuffled stack. An RNN reads page by page, in order.
Example
A detective novel follows a plot structure planned by an author. Real-world sequential data (e.g., stock prices) has no planned narrative — but the temporal dependency is real.
Analogy:
Imagine someone rips out all the pages of a detective novel and hands them to you in random order. You have all the same words and clues — but the mystery is unsolvable. The revelation on page 200 only makes sense because you read pages 1 through 199 first. A feedforward network reads the shuffled stack. An RNN reads page by page, in order.
Example
A detective novel follows a plot structure planned by an author. Real-world sequential data (e.g., stock prices) has no planned narrative — but the temporal dependency is real.
Definition:
Sequential data is any data where the order of elements carries meaning. Text, audio, stock prices, and sensor readings all share this property: rearranging the elements changes or destroys the information. Feedforward networks and CNNs process all inputs simultaneously — neither has an inherent concept of 'before' and 'after.'
Text, time series, and audio — three domains that all share the same property: the order of elements determines meaning.
Feedforward Network
Sees all inputs simultaneously. No concept of order. Cannot distinguish 'The dog bites the man' from 'The man bites the dog.'
Recurrent Neural Network
Processes inputs step by step. The hidden state carries context from previous steps forward. Order is preserved.
'The dog bites the man' vs. 'The man bites the dog' — identical vocabulary, reversed meaning. A feedforward network treating each word independently cannot distinguish these sentences because it sees the same bag of words. An RNN processes left-to-right: by the time it reaches 'bites,' it already knows whether 'dog' or 'man' came first.
Misconception: CNNs Can Handle Sequences Too
Partial truth: 1D CNNs detect local sequential patterns effectively, but their receptive field is limited. For long-range dependencies (connecting word 1 to word 200), recurrence or attention mechanisms are necessary.
The Recurrence Loop — How RNNs Remember
Hidden State
AnalogyDefinition
Imagine a factory conveyor belt where each station (time step) receives a product and adds its own modification based on the product's current state AND what the previous station passed along (the hidden state). The work instructions (weights) are identical at every station. Moving forward along the belt, we build the product step by step — that is the forward pass. But when the quality inspector (gradient) traces a defect backward through 100 stations, the feedback signal fades at each relay — like a game of telephone where the message becomes quieter with each pass.
Example
On a conveyor belt, the physical product is altered. In an RNN, the original input remains unchanged — the hidden state is a mathematical summary, not a physical copy.
Analogy:
Imagine a factory conveyor belt where each station (time step) receives a product and adds its own modification based on the product's current state AND what the previous station passed along (the hidden state). The work instructions (weights) are identical at every station. Moving forward along the belt, we build the product step by step — that is the forward pass. But when the quality inspector (gradient) traces a defect backward through 100 stations, the feedback signal fades at each relay — like a game of telephone where the message becomes quieter with each pass.
Example
On a conveyor belt, the physical product is altered. In an RNN, the original input remains unchanged — the hidden state is a mathematical summary, not a physical copy.
Definition:
An RNN processes a sequence one element at a time. At each time step t, it takes the current input x_t and the previous hidden state h_{t-1}, combines them through learned weight matrices, applies an activation function, and produces the new hidden state h_t. This hidden state is the network's 'memory' — a compressed representation of everything it has seen so far.
The core idea of an RNN is simple: use the same weights at every time step and carry a hidden state from step to step. This enables processing sequences of arbitrary length.
1
x₁ → h₁ First input produces first hidden state
2
x₂ + h₁ → h₂ Second input merges with memory from step 1 (the + does not mean simple addition — the network combines both through learned weights)
3
x₃ + h₂ → h₃ Third input merges with accumulated context
4
hₜ → Output Final hidden state yields the prediction
Hidden State Update Equation
hₜ = f(Wₕ · hₜ₋₁ + Wₓ · xₜ + b)
Parameter sharing: The same weight matrices W_h and W_x are reused at EVERY time step. This means: an RNN can process sequences of any length without the parameter count growing.
The Vanishing Gradient Problem
To train an RNN, the network is 'unrolled' through time and gradients are propagated backward through the entire chain — Backpropagation Through Time (BPTT). The chain rule is applied at every time step, which means multiplying by the same weight matrix. If this matrix has values consistently below 1, the gradient shrinks exponentially toward zero.
'I grew up in France. [...100 words...] I speak fluent ___.' The correct answer is 'French,' but the signal from 'France' must survive 100+ time steps. With a factor of 0.9 per step: 0.9¹⁰⁰ ≈ 0.0000265 — practically zero. The network cannot learn the connection.
Misconception: RNNs Remember Everything
No! The hidden state is a fixed-size vector. It compresses all past information into a finite representation. As new inputs arrive, older information is progressively overwritten. Even without vanishing gradients, the hidden state is a lossy summary, not a perfect recording.
Deep Dive: The Math Behind Vanishing Gradients
The gradient for the first time step is: ∂L/∂h₁ = ∂L/∂h_T · ∏(∂h_t/∂h_{t-1}) for t=2..T. Each factor contains the weight matrix W_h. If the eigenvalues of W_h are less than 1, the product converges exponentially to zero. If greater than 1, it explodes — gradient clipping caps the gradient at a threshold. LSTMs bypass this through additive updates on the cell state instead of multiplicative chaining.
Interactive: RNN Step by Step
Click through the individual time steps of an unrolled RNN. Observe how the hidden state flows from cell to cell, absorbing new input information at each step. Pay special attention to the fact that all cells share the same weights.
Step 1 / 7Overview: Unrolled RNN
An RNN shown unrolled through time: 4 time steps, each with its own input x. The hidden state h flows from left to right, carrying context forward.
Gates to the Rescue — LSTM and GRU
The Long Short-Term Memory (LSTM) network, invented in 1997 by Hochreiter and Schmidhuber, replaces the simple hidden state update with a memory cell controlled by three learned gates.
LSTM Cell: Information Flow Through the Gates
Forget Gate
Sigmoid: Decides what to erase from long-term memory (0 = forget, 1 = keep)
Input Gate
Sigmoid: Decides what new information to write into memory
Output Gate
Sigmoid: Decides what part of memory to expose as the current hidden state
Cell State
Long-term memory: information flows nearly unchanged (Constant Error Carousel)
Think of the LSTM cell as a secure filing cabinet with three locks. The Forget Gate lock lets you shred old documents that are no longer relevant. The Input Gate lock lets you file new important documents. The Output Gate lock controls which documents you take out to work with right now. The crucial insight: you can leave documents in the cabinet untouched for as long as they remain relevant — they do not decay over time like a vanilla RNN's hidden state.
The Constant Error Carousel — the cell state — allows gradients to flow nearly unchanged through the cell. Because the gates can learn to keep their values near 1.0, the gradient is not exponentially attenuated.
GRU: The Simplified Alternative
The Gated Recurrent Unit (GRU), introduced in 2014 by Cho et al., simplifies the LSTM architecture to two gates (Update and Reset) and achieves comparable performance with fewer parameters.
1986
RNN Concept
Rumelhart et al. describe the idea of recurrent connections in neural networks
1991
Vanishing Gradient Identified
Hochreiter shows in his diploma thesis that gradients vanish exponentially in deep recurrent networks
1997
LSTM
Hochreiter & Schmidhuber invent Long Short-Term Memory — gates control information flow
2014
GRU
Cho et al. simplify LSTM to two gates — comparable performance with fewer parameters
2016
Google NMT
Google Translate switches to LSTM-based Neural Machine Translation — dramatic quality improvement
2017
Transformers
Vaswani et al. replace recurrence with Self-Attention — parallel processing of all positions
Real-World Example: Google Translate
Before 2016, Google Translate used phrase-based statistical models that translated sentence fragments independently — often producing grammatically incorrect output. In 2016, Google switched to Neural Machine Translation powered by deep LSTMs, producing dramatically more fluent translations. In 2017, Transformers replaced LSTMs, adding another leap — foreshadowing the next article in your learning path.
2014 Papers
Attention Mechanism: The Key to Modern LLMs
September 2014: Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio published a paper that would lastingly change the NLP world. 'Neural Machine Translation by Jointly Learning to Align and Translate' solved a fundamental problem in sequence-to-sequence models. Previous encoder-decoder architectures compressed every input sentence into a single fixed-length vector — an information bottleneck for long sentences. Bahdanau attention was a significant step forward: instead of a fixed vector, the model used dynamic attention over different parts of the input sentence. Like the human eye jumping while reading, AI attention moves between relevant words. This 'additive attention' became the conceptual precursor to modern NLP systems. The later Transformer (2017) built on the attention idea, but replaced the additive variant with the more efficient Scaled Dot-Product Attention. Without Bahdanau's attention concept, no Transformers; without Transformers, no GPT family or BERT. This breakthrough happened three years before 'Attention Is All You Need.'
Misconception: LSTMs Have Infinite Memory
No! LSTMs forget too — but in a controlled, learned way rather than the uncontrolled exponential decay of vanilla RNNs. The Forget Gate actively decides what to discard. Over very long sequences, even LSTMs struggle with very distant dependencies.
Despite their success, both architectures retain the fundamental sequential processing constraint: each step must wait for the previous step to complete, making parallelization on GPUs impossible. This sequential bottleneck directly motivated the Transformer architecture — the next article in your learning path.
Deep Dive: The RNN Renaissance
Transformers have quadratic cost with sequence length (O(n²) attention). State Space Models like Mamba (2023) achieve linear scaling through a learned recurrence. Streaming applications (real-time audio, sensor monitoring) where latency is critical benefit from RNN-like architectures. Hybrid architectures combine attention and recurrence. The key message: the recurrence principle is not dead — it has evolved.
Key Takeaways
Sequential data carries meaning in order — feedforward networks destroy this by processing everything simultaneously, while RNNs process step-by-step with a hidden state that accumulates context.
The Vanishing Gradient Problem is not a bug but a mathematical inevitability: repeated multiplication through long chains causes gradients to shrink exponentially, erasing long-range memory.
LSTMs solve this with gates that learn WHAT to remember and forget — but their strictly sequential processing makes them slow, setting the stage for Transformers.
Check Your Understanding
Why can't feedforward networks distinguish 'The dog bites the man' from 'The man bites the dog'?
A gradient factor of 0.85 applied 150 times — what happens to the signal?
Which gate of an LSTM cell would you use to erase irrelevant information from the cell state?
Quiz: Recurrent Neural Networks
Question 1 / 6
Not completed
What is the primary purpose of the hidden state in a Recurrent Neural Network?
1. What is the primary purpose of the hidden state in a Recurrent Neural Network?
☐ A) To store the network's weight matrices for each time step
☐ B) To maintain a compressed representation of all previously processed inputs in the sequence
☐ C) To increase the number of parameters the network can learn
☐ D) To enable the network to process all inputs simultaneously
2. Why do standard (vanilla) RNNs struggle with long sequences?
☐ A) They run out of GPU memory for sequences longer than 50 elements
☐ B) Their hidden state vector is too small to store more than 20 words
☐ C) Gradients shrink exponentially during backpropagation through many time steps, preventing learning of long-range dependencies
☐ D) They can only process numerical data, not text or audio
3. An RNN processes a 200-word sentence. At each time step, the gradient is multiplied by a factor of 0.95. What is the approximate gradient magnitude reaching the first word (0.95²⁰⁰)?
☐ A) About 0.95 (nearly unchanged)
☐ B) About 0.00004 (nearly zero)
☐ C) About 0.5 (reduced by half)
☐ D) About 0.1 (reduced to 10%)
4. In an LSTM cell, the Forget Gate outputs [0.1, 0.9, 0.0, 1.0] for a four-dimensional cell state. What happens to each dimension?
☐ A) All dimensions are multiplied by 0.5 (averaged)
☐ B) Dimension 1 is mostly erased, dimension 2 mostly kept, dimension 3 completely erased, dimension 4 fully preserved
☐ C) Dimensions with values below 0.5 are set to zero, above 0.5 set to one
☐ D) The gate output is added to the cell state
5. Google Translate switched from phrase-based models to LSTM-based NMT in 2016, then to Transformers in 2017. What fundamental RNN limitation motivated the second switch?
☐ A) LSTMs could not handle languages with non-Latin alphabets
☐ B) LSTMs' strictly sequential processing prevented parallelization on GPUs, making training on massive datasets prohibitively slow
☐ C) LSTMs produced grammatically incorrect translations
☐ D) Transformers require fewer training examples than LSTMs
6. A colleague claims: 'Since Transformers have replaced RNNs in NLP, recurrent architectures are completely obsolete.' Why is this statement an oversimplification?
☐ A) Because RNNs are still faster than Transformers for all tasks
☐ B) Because Transformers cannot process sequential data at all
☐ C) Because RNN-like architectures (including modern State Space Models) offer linear-time scaling for very long sequences and real-time streaming, where Transformers' quadratic attention cost becomes prohibitive
☐ D) Because LSTMs have better accuracy than Transformers on every benchmark
Answer Key: 1) B · 2) C · 3) B · 4) B · 5) B · 6) C