Gigatoken: A Stanford PhD student wrote the BPE tokenizer to 24.53 GB/s

On July 22, 2026, a Rust tokenizer was posted on the homepage of Hacker News, with 615 points and 119 comments. The headline number makes one suspect that the browser is rendering it wrong: 989 times faster than HuggingFace tokenizers. The comment section didn't really question whether the numbers were true or false, but what was discussed was "Is this number useful?" A typical objection comes from Amdahl's law: in standard GPU inference, tokenization accounts for less than 0.1% of wall time. If the car is accelerated 1000 times, the entire high-speed train will not be faster than one second. This logic is correct. But when you look away from