What is a Token?

In artificial intelligence, a token is the fundamental unit of text that a Large Language Model (LLM) processes.

Instead of reading full words or single characters like humans do, LLMs break text into fragments of characters called tokens.

The Rule of Thumb for Tokens

As a quick rule of thumb for English text:

  • 1 token ≈ 4 characters
  • 1 token ≈ 0.75 words
  • 100 tokens ≈ 75 words

For example, the word "unbelievable" might be split by a tokenizer into three tokens: ["un", "believ", "able"].

Why Tokens Matter to Students

  1. Context Limits: Every AI model has a maximum context window (e.g. 128,000 tokens). If your document exceeds this limit, the model will forget the earliest parts.
  2. Pricing & API Costs: API providers charge per million tokens processed.
  3. Response Length: AI outputs are measured in max output tokens (e.g. 4,096 tokens max output).

Test Your Tokens Locally

Use our free RankLLMs Token Counter to test token length and estimate prompt size directly in your browser.