Pretraining
Also: pre-training
An early, broad stage of model training. The model learns patterns from a large set of data before later training for some uses.
A model is a huge set of dials, each holding a number. Training turns the dials a little at a time, over millions of examples, until the model’s guesses are good. After training the dials are locked: using the model does not change them, and showing it your document does not teach it anything permanent.
The fuller definition
Pretraining is an initial broad training stage that develops reusable representations or capabilities before more specific adaptation.
Go deeper
- ExamplesInputs paired with the expected output.
- PredictionThe model produces an output.
- LossHow far off it was, as a number.
- UpdateParameters nudged to reduce the loss; repeat.
Training ends at the knowledge cutoff. Whatever the examples contained, including their gaps and biases, is what the model learned.
A concrete example
A language model learns to predict text continuations across a large corpus before instruction tuning.
The distinction that matters
Pretraining does not make every learned statement current, accurate, or traceable to a source.
Related terms
Training
Adjusting a model with examples and a learning rule. The aim is to improve how it handles tasks.
Training & generationFine-tuning
More training on a chosen set of examples. It changes a model to suit a task or style.
Training & generationDataset
A collection of examples used to train or test a model.
Where it is taught
How models learn from examplesSource and further reading
Hugging Face · LLM courseThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.