A short history of language models

Before 2000: counting words

An n-gram model uses a short history of words to predict what comes next. Smoothing and backoff address unseen or sparse sequences. {{figure:pe-history}}

A language model in the older sense is any system that assigns probabilities to sequences of words. The classic version is the n-gram model: count how often each word follows the previous one, two, or more words in a large body of text, and predict accordingly. [source]

N-gram models hit two walls. First, the number of possible word combinations explodes as you look further back (sometimes called the curse of dimensionality), so they can only consider a few words of context. Second, they have no sensible answer for a phrase they never counted (sparsity). The output could mimic the statistics of real text for a few words, but a reader would quickly see there was no thread connecting one sentence to the next. [source]

Mid-2000s: deeper neural networks

In 2006, Geoffrey Hinton and colleagues at the University of Toronto published a method for training networks with many layers one layer at a time, which helped make much deeper neural networks trainable. [source] Applied to language, neural networks represent words and phrases as points in a continuous numerical space rather than entries in a count table, so a phrase never seen in training can still land near similar phrases that were. [source]

These models learned the statistics of their training text well, but their output still tended to drift away from the input rather than respond to it.

2010s: recurrent networks and a running memory

Long short-term memory (LSTM) networks use gates to manage the information carried through a sequence. Long short-term memory networks (LSTMs), introduced by Hochreiter and Schmidhuber in the mid-1990s and published in 1997, found wide use in the 2010s. [source] An LSTM reads text one token at a time and updates an internal state as it goes, which gives it a kind of running memory. In 2015, Andrej Karpathy's widely read essay showed small character-level recurrent networks producing surprisingly convincing imitations of Shakespeare, Wikipedia markup, and source code. [source]

Two weaknesses remained. Long-range connections were hard: given "In France we travelled, ate pastries, drank wine … but never learned to speak ____", an LSTM could lose the thread back to "France". And reading strictly one token after another made training slow, because the work could not be split across many processors at once. [source]

2017: the transformer

Google researchers' paper "Attention Is All You Need" introduced the transformer. [source] Instead of reading in sequence, a transformer processes the whole input at once and uses a mechanism called attention to weigh how much each token should influence each other token. Two consequences made modern AI possible:

• Parallel training. Because the whole sequence is processed together, training can be spread across thousands of processors, so models could be made far larger.
• Long-range connections. Every output token can be influenced by every input token, so "France" and "French" can be linked across a long passage.

Research into variations continued. In 2022, for example, Amazon reported that its 20-billion-parameter AlexaTM model outperformed the far larger GPT-3 on several few-shot tasks, a reminder that size is not the only lever. [source]

The design also introduced a lasting constraint: a transformer works within a fixed-size context window, and the cost of attention grows quickly as that window gets longer. [source] What uploading a document does—and does not—establish explains what this means for the documents you upload.

2018–2020: pretraining at scale

In 2018, OpenAI showed that a transformer first trained on a large amount of unlabelled text, with no particular task in mind, could then be adjusted for specific tasks with comparatively little extra training and reach state-of-the-art results. This is the "generative pre-trained" in GPT. [source]

In 2020, the GPT-3 paper scaled the approach up roughly tenfold and found that many tasks no longer needed extra training at all: describe the task, perhaps give a few examples in the prompt, and the model performs it. [source] This is the origin of few-shot and zero-shot prompting, and of prompt engineering as a discipline.

2022: following instructions

A model trained only to continue text is not naturally a helpful assistant; asked a question, it may continue with more questions. In 2022 OpenAI published InstructGPT, which tuned a pretrained model to follow instructions using reinforcement learning from human feedback (RLHF): people ranked model outputs, a second model learned to predict those rankings, and the language model was trained to produce outputs that score well. [source] The underlying technique came from a 2017 paper by researchers at OpenAI and DeepMind. [source]

ChatGPT, released on 30 November 2022, applied the same approach to a conversational model. [source] One lasting effect matters for lawyers: a model tuned on human ratings learns what people prefer to read, which is not always what is accurate. The tendency to agree with the user, known as sycophancy, is one result.

2023 onward: many models, fast change

OpenAI's GPT-4 followed in March 2023. [source] Other developers released strong models too, including Meta's LLaMA family, whose weights researchers could download and run themselves. [source] Google (FLAN-T5, and earlier BERT), Meta (OPT), and open research groups (BLOOM, StableLM) also published models anyone could inspect. [source] That began the open-weights ecosystem that makes local AI (Why run AI on your own machine) possible.

Since then the main developments have been reasoning models that spend extra computation before answering (Ask for the working), models that call tools and act as agents (Give the model the facts, or give it tools), and context windows that have grown from a few thousand tokens to around a million in some current models (What uploading a document establishes). Models that once needed a cluster of servers were soon running, slowly, on laptops and even single-board computers. [source] The pace is fast enough that any list of "current" models dates within months, which is why this site links to dated primary sources rather than naming a best model.

Why the history matters in practice

• Fixed context windows come from the transformer design. They are why a long record must be selected, not just uploaded.
• Fluent but ungrounded text comes from training on what text looks like. It is why every material claim needs a source.
• Agreeable, confident tone partly comes from training on human preferences. It is why tone is not evidence.