AI glossary / How it works

Token

How it works, one of the 17 words to learn first

The small chunks of text an AI reads and writes in, roughly three-quarters of a word each. Limits and prices are usually counted in tokens.

More words in How it works

• TokenizationChopping text into tokens before a model reads it.
• TrainingThe learning stage, where a model is shown vast numbers of examples and adjusts itself until it gets good at the task.
• Training dataThe material a model learned from. The quality of what goes in shapes the quality of what comes out.
• Transfer learningReusing what a model learned on one task as a head start on another.
• TransformerThe model design, introduced in 2017, behind nearly all modern language AI. It's the T in GPT.