How Does ChatGPT Actually Think?
Understanding LLMs, Part 3 of 3: Transformers & Attention
This is Part 3 of a 3-Part Series. If you didn’t catch the first two articles, here they are:
How Does ChatGPT Actually Read? — Understanding LLMs: Tokens & Vectorization
How Does ChatGPT Actually Learn? — Understanding LLMs: Neural Networks
Nate had been wearing his sommelier lapel pin out on special occasions for about six months when something clicked.
Not in the studying sense that had clicked long before. I mean something different clicked. I watched him at a dinner party we were both invited to. A table of eight friends sat together with what looked like a genuinely chaotic set of preferences. One person was celebrating. Someone mentioned they didn’t love “heavy” reds. Someone else was asking their friend next to them for a recommendation in the same breath as “but I don’t really know wine.” Another was talking about the movie sideways while ragging on Merlot.
And the host looked awkwardly nervous about whether people were enjoying themselves.
Nate stood up and casually slipped away towards the host’s wine rack. He came back confidently with one bottle. He opened up and poured some glasses. And everyone, I mean everyone seemed to enjoy it. The mood completely shifted.
I asked him later how he’d done it — how he’d managed to hold all of the signals and clues at once and still come out the other side with a single, confident answer. He kind of shrugged. “You don’t answer each person separately,” he said. “You read the whole table.”
That’s not just good hospitality. That, almost exactly, is how a transformer works.
The Problem the Last Two Parts Didn’t Solve
In Part 1, we talked about how a large language model reads by breaking your text into tokens and converting each one into coordinates on a vast map of meaning. In Part 2, we covered how it learns through billions of small corrections, stacked into layers, until patterns that once seemed invisible become second nature.
But here’s what neither of those parts answered: how does the model understand context?
Specifically, what does “it” refer to in this sentence?
“The sommelier recommended a Burgundy because it paired beautifully with the fish.”
You knew immediately. The “it” is the Burgundy … not the sommelier and not the fish. You knew because you held the whole sentence in your head at once, and the context made the reference obvious. Early AI models couldn’t do this. They read the way someone might if they could only see one word at a time, left to right, without being able to glance back. By the time they reached “it,” they’d practically forgotten the word “Burgundy.”
Transformers were built to fix exactly this.
Reading the Whole Table at Once
A transformer doesn’t read left to right. It reads everything simultaneously. It “sees” every word in relation to every other word, all at once, in a single pass.
For each word in a sentence, the model is essentially asking: how much does every other word in this sentence matter for understanding this one? It assigns a weight to each relationship. A strong relationship gets a high weight, and a weak relationship a low weight. This process of weighing relevance across an entire sentence, or an entire conversation, is called the attention mechanism.
Think back to Nate’s interaction at our friend’s dinner party. He wasn’t processing each person in order – Person 1, then Person 2, then Person 3. He was holding the whole situation in his head, weighing each constraint against the others. The celebration matters. The “no heavy reds” matters. I guess even the comedy-movie-influenced opinion about Merlot matters. And crucially, those things matter in relation to each other, not independently.
The attention mechanism is the model doing the same thing, just with words. When it reads “Burgundy because it paired beautifully with the fish,” the model notices that “it” has a strong relationship to “Burgundy” — not because of a rule someone programmed in, but because, across billions of sentences, that’s how language actually behaves. Pronouns follow their antecedents. The map knows the pattern.
Why This Was the Actual Breakthrough
Before transformers, language models were sequential. They read text like a person scanning a page from left to right, word by word, holding a kind of running summary in memory. The problem is that by the time a computer generated model reached the end of a long, complex sentence, the beginning had effectively faded. Long-range relationships, such as a pronoun referring back to a noun from two clauses ago or a question being answered twenty words later, were genuinely hard for these systems to handle.
In 2017, a team of researchers at Google published a paper that changed that. Its title was almost comically direct: “Attention Is All You Need.”
The core insight was that attention (the ability to weigh the relevance of every word against every other word, simultaneously) was not just one useful technique among many. It was THE thing. You didn’t need to process text in sequence. You didn’t need to summarize and carry forward. You just needed to let every part of the input speak to every other part, all at once, and let the weights sort out what mattered.
That architecture became the backbone of every major AI language model that followed. GPT. BERT. Claude. All of them use transformers.
The Full Picture
Here’s what the three parts of this series add up to about LLMs:
Tokens gave the model a way to break language into manageable pieces.
Vectorization gave those pieces coordinates, like a location on a map of meaning, where similar concepts cluster together and dissimilar ones drift apart.
Neural networks gave the model a way to learn through billions of small corrections, across billions of examples, until the patterns became something resembling intuition or reason.
And transformers gave the model something none of the prior pieces could: the ability to hold context. This allowed a LLM to read a sentence, a paragraph, or a conversation the way Nate read that table. Not one thing at a time. All of it, together.
The result isn’t magic. That’s the thing worth sitting with. There’s no mystery ingredient, no moment where the machine becomes conscious. It’s pattern recognition. Ok, it’s extraordinarily sophisticated pattern recognition operating at a scale that genuinely strains comprehension, but pattern recognition nonetheless.
What is remarkable is what pattern recognition looks like at that scale. It looks like a system that can hold a nuanced multi-paragraph conversation and still remember what you said in your first message. It looks like a tool that can draft, explain, translate, and reason, not because it was programmed with answers, but because it was exposed to enough of the world’s written thought to navigate language the way a seasoned sommelier navigates a wine list.
Nate didn’t become a great sommelier because he memorized 500 flashcards. He became one because ten thousand hours of pattern recognition eventually became something that felt like taste.
The model got there too. It just did it faster, and it never got to drink the wine like Nate.
