Module 5 Lesson 1: Why Transformers Replaced Earlier Models

If you ask any AI researcher what the "Big Bang" of modern AI was, they will point to a single 2017 paper titled: "Attention Is All You Need." This paper introduced the Transformer.

But to understand why the Transformer is a genius invention, we first have to understand what it replaced: Recurrent Neural Networks (RNNs) and LSTMs.

1. The Sequential Problem: Life Before Transformers

Before 2017, AI processed text like a human reading a ticker tape—one word at a time, from left to right. This is called Sequential Processing.

The Weakness:

Limited Memory: By the time the AI reached word 50 of a sentence, it had often "forgotten" word 1. This made long-range context impossible.
Slow Training: Because it had to wait for word 1 to be finished before starting word 2, you couldn't take advantage of modern GPUs, which are designed to do thousands of things at once (Parallelization).

graph LR
    subgraph "The RNN Way (Slow & Sequential)"
        W1["Word 1"] --> H1["State 1"]
        H1 --> W2["Word 2"]
        W2 --> H2["State 2"]
        H2 --> W3["Word 3"]
        W3 --> H3["State 3 (Weak Memory of W1)"]
    end

2. The Transformer Breakthrough: Everything Everywhere All At Once

A Transformer doesn't read left to right. It reads the entire sentence at once.

Instead of passing a "hidden state" from word to word, the Transformer looks at every word in the sentence and asks: "Which other words are relevant to me?"

The Parallel Advantage

Because the Transformer looks at everything simultaneously, it can be trained on massive GPUs in parallel. This is why we can suddenly train models on the entire internet—a task that would have taken 1,000 years with older RSS/LSTM architectures.

3. Global Context

In an older model, the AI might see the word "it" and have no idea if "it" referred to a "dog" or a "bone" mentioned four sentences ago.

In a Transformer, the word "it" can "Attend" (look back) to any other word in the entire context window instantly. This allows the model to understand the deep structure of language, satire, and long-form logic.

graph TD
    subgraph "The Transformer Way (Fast & Parallel)"
        Sentence["'The dog didn't cross the road because IT was too tired.'"]
        IT["'IT'"] -- "Attends to..." --> DOG["'DOG' (High weight)"]
        IT -- "Attends to..." --> ROAD["'ROAD' (Low weight)"]
    end

4. Summary: The New King of Architecture

The Transformer became the standard because it solved the two biggest bottlenecks in AI:

Memory: It can remember relationships across thousands of tokens.
Scale: It can be trained massively in parallel across thousands of GPUs.

Lesson Exercise

Goal: Compare Sequential vs. Parallel processing.

Think about reading a book. If you have to read word-by-word, it takes a long time.
Now imagine if you could lay out every page of the book on a giant wall and see every "mention" of the main character at the same time.
Which method makes it easier to spot a hidden clue that appeared in Chapter 1 and Chapter 20?

Observation: The "Wall of Pages" is how a Transformer sees the world.

What’s Next?

In Lesson 2, we dive into the secret sauce that makes this possible: The Attention Mechanism. We'll use a simple analogy of "Keys, Queries, and Values" to explain how the model figures out what to focus on.