Post
20
πΉπΌ Releasing lianghsun/tw-tokenizer-v1 β a tokenizer trained from scratch for Traditional Chinese (Taiwan).
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex β it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured Ο = β0.59), so we validated two levels deeper:
**Segmentation** β boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
**Downstream** β trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win β with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info β Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak TΓ’i-lΓ΄ support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
π lianghsun/tw-tokenizer-v1
**46% better Chinese compression than Qwen3.8-27B with 81% of its vocab (201K vs 248K), and English essentially untouched (4.657 vs 4.674 chars/token).**
The gain isn't from the regex β it's the corpus. Qwen carries **27,364 Simplified-only multi-char tokens**, 11% of its vocab, dead weight for Traditional Chinese. Train on pure Traditional and that waste never appears.
Recent work is skeptical that compression predicts quality (Lotz et al. 2025 measured Ο = β0.59), so we validated two levels deeper:
**Segmentation** β boundary hit rate against jieba: **85.6%** vs Qwen's 77.8%. Single-character tokens: **17.6%** vs 41.7%.
ε°ζ₯η΄ ι€γηΉθ³ͺζηΆε
¬εε―©ζ₯εͺε
ours: ['ε°ζ₯η΄ ι€', 'γ', 'ηΉθ³ͺ', 'ζηΆ', 'ε
¬ε', 'ε―©ζ₯', 'εͺε']
Qwen: ['ε°ζ₯', 'η΄ ', 'ι€', ...] β γη΄ ι€γsplit mid-word**Downstream** β trained a 270M model from scratch with each tokenizer, compared bits-per-character (the only metric fair across tokenizers). At equal compute: **4.434 vs 4.591**, a 3.4% win β with 13% fewer parameters. Same token budget means our model saw 440M characters vs 308M: **43% more data for the same compute**.
Also: 6-char cap on pure-CJK tokens (long tokens obscure orthographic info β Haslett, CL 2025), NFC not NFKC, 1,024 reserved tokens.
Known limits (weak TΓ’i-lΓ΄ support, small-scale downstream validation, vocab sweep hadn't flattened) are in the card.
π lianghsun/tw-tokenizer-v1