Glossary

What is token (LLM)?

A token is the unit of text a language model processes — roughly a word-piece — and the thing models are billed by and limited to.

A tokenizer splits text into sub-word units before the model sees it; in English a token averages about four characters. Both context windows and API pricing are measured in tokens, so token efficiency directly affects cost and how much you can fit in a prompt.

Different models use different tokenizers, which changes how efficiently they handle code and non-English languages. Open models ship their own tokenizer so you can measure token counts exactly.

Trending AI & ML projects →

Trending token (LLM) projects

decolua/9router
Unlimited FREE AI coding. Connect Claude Code, Codex, Cursor, Cline, Copilot, Antigravity to FREE Claude/GPT/Gemini via 40+ providers. Auto-fallback, RTK -40% tokens, never hit limits.
★ 14.3K+165 · 24hmomentum 13JavaScript
jundot/omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
★ 15.2K+92 · 24hmomentum 12Python
iOfficeAI/AionUi
Free, local, open-source 24/7 Cowork app for OpenClaw, Hermes Agent, Claude Code, Codex, OpenCode, Gemini CLI and 20+ more CLI | Customize your assistants | Star if you like it!
★ 26.6K+34 · 24hmomentum 11TypeScript
NVlabs/Sana
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer
★ 7.6K+92 · 24hmomentum 11Python
Fincept-Corporation/FinceptTerminal
FinceptTerminal is a modern finance application offering advanced market analytics, investment research, and economic data tools, designed for interactive exploration and data-driven decision-making in a user-friendly environment.
★ 24K+56 · 24hmomentum 10Python
jlcodes99/cockpit-tools
🚀 通用 AI IDE 账号管理工具：支持 Antigravity / Codex / GitHub Copilot / Windsurf / Kiro / Cursor / Gemini-cli / CodeBuddy，多账号切换、配额监控、自动唤醒与多开实例管理。 🚀 Universal AI IDE account manager for Antigravity / Codex / GitHub Copilot / Windsurf / Kiro / Cursor / Gemini-cli / CodeBuddy, with multi-account switching, quota monitoring, wake-up automation, and multi-insta
★ 9.2K+31 · 24hmomentum 10Rust
openai/symphony
Symphony turns project work into isolated, autonomous implementation runs, allowing teams to manage work instead of supervising coding agents.
★ 24.6K+59 · 24hmomentum 10Elixir
OpenBMB/VoxCPM
VoxCPM2: Tokenizer-Free TTS for Multilingual Speech Generation, Creative Voice Design, and True-to-Life Cloning
★ 19.8K+26 · 24hmomentum 10Python

▌ token (LLM) — FAQ

What is token (LLM)?

A token is the unit of text a language model processes — roughly a word-piece — and the thing models are billed by and limited to. A tokenizer splits text into sub-word units before the model sees it; in English a token averages about four characters. Both context windows and API pricing are measured in tokens, so token efficiency directly affects cost and how much you can fit in a prompt.