README.md

metadata

title: Tokenizer Comparison
emoji: 📊
colorFrom: purple
colorTo: purple
sdk: gradio
sdk_version: 5.33.0
app_file: app.py
pinned: false
license: apache-2.0

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

• next up i want to add some sample texts that are interesting
• normalization of the tokenization