Deep Dive into LLMs like ChatGPT

How ChatGPT-style models are built, from internet pretraining and tokenization through fine-tuning and reinforcement learning, and what that means for using them well.

A long, general-audience lecture that walks the whole pipeline behind a chatbot: collecting web text, pretraining a base model, post-training it into an assistant, then reinforcement learning. Along the way it builds mental models for why LLMs hallucinate, why they trip over counting and spelling, and how to use them without trusting them blindly. Published 5 February 2025.

These notes were compiled from several published write-ups of the talk, listed at the end, and cross-checked against the video’s chapter list.

Pretraining: data, tokens and the network (0:01)

It starts with a large, filtered crawl of the web. Hugging Face’s FineWeb (about 44 TB, roughly 15T tokens) is the running example, built from Common Crawl with URL filtering, text extraction, language filtering, deduplication and PII removal.
Tokenization (7:47). Text becomes bytes, then byte-pair encoding merges frequent pairs to shorten sequences. GPT-4’s vocabulary has 100,277 tokens; the tiktokenizer web app shows how text splits.
Network input and output (14:27). The model reads a window of tokens and outputs a probability for every token in the vocabulary. Training nudges the weights so the token that actually came next becomes more likely.
Internals (20:11). A Transformer is one big fixed mathematical function with billions of parameters. Nothing in it resembles a memory or a database.
Inference (26:01). Generation samples one token at a time from that distribution, so output is stochastic and varies between runs.

Base models: GPT-2 and Llama 3.1 (31:09)

GPT-2 (2019): 1.6B parameters, a 1,024-token context, about 100B training tokens. Karpathy’s llm.c reproduction shows how cheap that has become on rented 8×H100 nodes.
A base model is an “internet document simulator”. It autocompletes; it doesn’t answer. Its weights are a “lossy compression of the internet”: it can recite famous passages such as Wikipedia almost word for word but only vaguely recalls rarer facts.
Llama 3.1 405B base (42:52), demoed on Hyperbolic. Few-shot prompts, or a fake conversation written out as a document, can coax a base model into acting like an assistant.

Post-training: fine-tuning on conversations (59:23)

Post-training keeps training on a dataset of conversations. It takes hours of compute, not the months pretraining takes.
Turns are encoded with special tokens (user and assistant markers) that never appear in web text.
InstructGPT (OpenAI, 2022) used human labelers following long guidelines. Modern datasets such as UltraChat are mostly synthetic, generated with LLM help and edited by people.
The takeaway: talking to ChatGPT is talking to a statistical simulation of a labeler following the company’s instructions.

Hallucinations, tools and memory (1:20:32)

Hallucinations come from imitating confident answers even when the model doesn’t know.
Fix one: probe what the model knows by asking factual questions repeatedly (Meta did this for Llama 3). Where it’s consistently wrong, add training examples whose answer is “I don’t know”.
Fix two: tools. The model emits special tokens to run a web search, and the results are pasted into its context.
Knowledge in the weights is vague recollection; tokens in the context window are working memory. Pasting the source into the prompt beats relying on recall.
Sense of self (1:41:46). There is no persistent self. “Who are you?” is answered from hard-coded fine-tuning conversations or the system prompt.

Models need tokens to think (1:46:56)

Each token gets a fixed, finite amount of computation, so reasoning has to be spread across many tokens. Training answers that state the result before working it out teach bad habits.
For arithmetic and exact counting, it’s better to have the model write and run code than compute “in its head”.
Spelling and counting (2:01:11). The model sees tokens, not characters, so counting the r’s in “strawberry” or a row of dots is surprisingly hard.

Jagged intelligence (2:04:53)

Capabilities follow a “Swiss cheese” model: excellent at many hard tasks, with random holes where trivial ones fail.
The practical rule is to use LLMs as tools, check their work, and never trust them blindly.

Reinforcement learning (2:07:28)

A school analogy for the three stages: pretraining is reading the textbook, fine-tuning is studying worked examples, and RL is doing practice problems with only the answer key.
In RL the model samples many attempts at a problem with a checkable answer, and the attempts that reach it are reinforced. The model finds reasoning strategies that suit it, rather than copying a human’s.
DeepSeek-R1 (2:27:47). As RL training goes on, responses grow longer and the model learns to re-check its work and try other approaches. These thinking models show emergent chains of thought. OpenAI hides its reasoning models’ full chains of thought, partly to make distillation harder.
AlphaGo (2:42:07). RL lets a system go beyond human imitation, as move 37 did: a move no human would have played. The hope is for similarly novel strategies in LLM reasoning.

RLHF and its limits (2:48:26)

For tasks with no automatic check (jokes, poems, summaries), people rank model outputs, a reward model learns to imitate those rankings, and RL optimizes against it.
Ranking is easier for people than writing ideal answers, so RLHF does improve models.
The catch: the reward model is a gameable simulation of human judgment. Run RL long enough and the model finds adversarial nonsense that scores well, so RLHF is stopped after a limited number of steps. Karpathy calls it closer to a fine-tuning step than to “real” RL that can run indefinitely.

What’s next, and where to find models (3:09:39)

Coming: natively multimodal models (text, audio, images), agents that run long tasks under human supervision, models that disappear into other tools, and research on learning at test time.
Keeping up (3:15:15): the LMArena leaderboard (built on human comparisons) and the AI News newsletter.
Where to use them (3:18:34): proprietary models on their providers’ own sites; open-weight models such as DeepSeek and Llama through hosts like Together AI; base models on Hyperbolic; smaller and distilled models locally with LM Studio.

Sources

Watch on YouTube