Tokens and Tokenization

One model might read a short customer message as these 11 pieces:

My| refund| for| order| |482|13| is| still| pending|.

Each bar separates two pieces. Most words are one piece each, and a piece usually includes the space in front of it. The order number is split into three pieces: a lone space, "482", and "13".

These pieces are called tokens, the only units a language model reads. The model never sees letters or digits, only one number for each token. This explains what each request costs and how much text one request can hold