From neural networks to large language models

Your phone keyboard suggests the next word as you type. A chatbot writes whole essays, answers questions and translates poems. Are these the same kind of machine? At the heart of both is one surprisingly modest task: given the text so far, guess what comes next. This last lesson shows how far that single question can go, and where its limits lie.

Try this

• Type a common word into the text field and read the list of likely next words and their probabilities. Do they make sense given what you know about the text?
• Move the temperature slider to the left and watch the top word take almost all of the probability. Then move it to the right and watch the probabilities even out.
• Try a word that is probably rare in the small text. With so few examples to count, how trustworthy are the probabilities the model shows?
• Follow a chain by hand: type the most likely next word, then the most likely word after that one, and so on. Does the sentence that comes out sound fluent? Does it go anywhere?

Now that you've tried it, let's see what was actually going on.

The idea in plain words

A language model assigns probabilities to what comes next in a text. After "I would like a cup of", the word "tea" is likely, "coffee" too, "gravel" hardly at all. Once you have such probabilities, you can generate text: pick a next word, append it, and ask again.

The simplest possible language model is a bigram model. It reads a body of text (a corpus) and simply counts which word follows which. This lesson looks at one built on a small text written for this course. For each word, the model knows the words that followed it in the corpus, with their probabilities.

Think of a person who has read only one book, many times, and whose only skill is remembering which word came after which. Ask them to continue a sentence and they will produce something that sounds vaguely like the book — locally fluent, globally aimless. That person is a bigram model.

Why it works

The bigram model estimates the probability of the next word b after the word a from counts:

P(next = b | current = a) = count(a b) / count(a)

If "cup" appears ten times in the corpus and is followed by "of" seven times, the model gives "of" a probability of 0.7 after "cup". That is all — counting and dividing. Learning, once again, means fitting a model to examples, and the score is how well it predicts the next word of real text.

To generate text the model has to choose among the candidates, and here temperature enters. Low temperature sharpens the distribution: the most likely word wins almost every time, and the text becomes repetitive and safe. High temperature flattens it: rarer words get a real chance, and the text becomes more varied and more erratic. Temperature does not change what the model knows, only how boldly it samples.

A large language model (LLM) does the same task with three big upgrades. First, it predicts tokens — pieces of words — so it can handle any word, even one it has never seen whole. Second, instead of looking only at the previous word, it is a neural network, usually a transformer (introduced in 2017), whose attention mechanism lets every position look back over thousands of previous tokens and decide which ones matter. Third, its probabilities are not counts but outputs of a network with billions of weights, trained with exactly the tools of this course: the log loss of lesson 5 on the next token, minimised by the gradient descent of lesson 4, with backpropagation from lesson 11. Trained on an enormous amount of text, the guessing task forces the network to pick up grammar, facts and patterns of reasoning, because they all help predict what comes next.

Common mistake

The biggest misunderstanding about language models is thinking they look answers up. They do not consult a database; they produce the continuation that their training made probable. Most of the time that is also the true one, but not always — when the plausible and the true come apart, the model states a falsehood fluently, which is called a hallucination. The second mistake is treating temperature as a quality dial. Low temperature is not "more correct", only more predictable; it suits factual answers, while higher temperature suits brainstorming.

That is the whole journey: from one threshold on one tumour measurement to a network predicting the next word of everything ever written. The recipe never changed — a family of models, a score, a search, and an honest test on data the model never saw. The Atlas links below are the place to go deeper into any step.

import re
from collections import Counter

src = open("site-src/course/labs/next-token.corpus.js", encoding="utf-8").read()
text = re.search(r"en: `(.*?)`", src, re.S).group(1)  # the lab's English sentences
words = re.findall(r"[a-z']+", text.lower())
after_the = Counter(b for a, b in zip(words, words[1:]) if a == "the")
counts = after_the.most_common()
print(f'"the" seen {sum(after_the.values())} times, {len(after_the)} different next words, top: {counts[:3]}')
for t in (1, 0.5, 0.1, 3):
    w = {b: c ** (1 / t) for b, c in counts}
    print(f"temperature {t}: P(morning | the) = {w['morning'] / sum(w.values()):.1%}")

"the" seen 42 times, 26 different next words, top: [('morning', 4), ('cup', 3), ('garden', 3)]
temperature 1: P(morning | the) = 9.5%
temperature 0.5: P(morning | the) = 18.6%
temperature 0.1: P(morning | the) = 85.0%
temperature 3: P(morning | the) = 5.3%