Tokenization, Robustness, LLMs
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Tokenization robustness leaderboard for language models
Evaluate models on multiple-choice questions
Compare tokenizers to split text into tokens
