AI glossary / How it works

Tokenization

How it works

Chopping text into tokens before a model reads it.

More words in How it works

• TrainingThe learning stage, where a model is shown vast numbers of examples and adjusts itself until it gets good at the task.
• Training dataThe material a model learned from. The quality of what goes in shapes the quality of what comes out.
• Transfer learningReusing what a model learned on one task as a head start on another.
• TransformerThe model design, introduced in 2017, behind nearly all modern language AI. It's the T in GPT.
• Unsupervised learningTraining where the AI finds patterns in data on its own, without being given the right answers.