Tokenizers documentation
Pre-tokenizers
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v1.0.0-rc.2).
Pre-tokenizers
TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See
REQUIRED_FOR_V1.mdfor the full list.
Python
Rust
Node
BertPreTokenizer
tokenizers.pre_tokenizers.BertPreTokenizer ByteLevel
tokenizers.pre_tokenizers.ByteLevel CharDelimiterSplit
tokenizers.pre_tokenizers.CharDelimiterSplit Digits
tokenizers.pre_tokenizers.Digits Metaspace
tokenizers.pre_tokenizers.Metaspace PreTokenizer
tokenizers.pre_tokenizers.PreTokenizer Punctuation
tokenizers.pre_tokenizers.Punctuation Sequence
tokenizers.pre_tokenizers.Sequence Split
tokenizers.pre_tokenizers.Split UnicodeScripts
tokenizers.pre_tokenizers.UnicodeScripts Whitespace
tokenizers.pre_tokenizers.Whitespace WhitespaceSplit
tokenizers.pre_tokenizers.WhitespaceSplit