# GigaToken GigaToken is a tokenization library for language models written in [[Rust]] (with Python bindings) by Marcel Rød. The claim that made it blow up on Hacker News: roughly 1000x faster than HuggingFace's tokenizers. Gigabytes per second instead of megabytes per second, consistent across x86 and ARM. SIMD-optimized pretokenization, caching of pretoken mappings (word distributions are long-tailed, so caching pays off big), minimized Python overhead, less inter-thread communication, and careful branch elimination. It supports nearly all common tokenizers (GPT-2, Llama, Qwen, DeepSeek, Gemma, ...) with both HuggingFace and Tiktoken compatibility modes, so adopting it can be a one-line change. MIT licensed. WordPiece and optimized SentencePiece are still missing. Tokenization is typically under 0.1% of total inference time, so this won't speed up your chatbot. It DOES matter for workloads that only tokenize: dataset preparation, counting tokens across corpora, filtering pipelines. ## References - GitHub: https://github.com/marcelroed/gigatoken - Hacker News discussion: https://news.ycombinator.com/item?id=49010167 ## Related - [[Rust]]