# Using a BertWordPieceTokenizer trained from scratch from transformers

**URL:** <https://discuss.huggingface.co/t/using-a-bertwordpiecetokenizer-trained-from-scratch-from-transformers/4391>\
**Category:** 🤗Tokenizers\
**Created:** [March 14, 2021, 5:04pm UTC](https://discuss.huggingface.co/t/using-a-bertwordpiecetokenizer-trained-from-scratch-from-transformers/4391 "2021-03-14T17:04:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![yngtodd](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@yngtodd](https://discuss.huggingface.co/u/yngtodd)\
**Post date:** [March 14, 2021, 5:04pm UTC](https://discuss.huggingface.co/t/using-a-bertwordpiecetokenizer-trained-from-scratch-from-transformers/4391/1 "2021-03-14T17:04:35Z")

</div>

Hey everyone,

I’d like to load a `BertWordPieceTokenizer` I trained from scratch using the interface built in `transformers`, either with `BertTokenizer` or `BertTokenizerFast`. It looks like those two tokenizers in `transformers` expect different ways of loading in the saved data from `BertWordPieceTokenizer`, and I am wondering what is the best way to go about things.

### Example

I am training on a couple test files, saving the tokenizer, and the reloading it in `tokenizers.BertTokenizer` (there is a bit of ceremony here creating the test data, but this is everything you need to reproduce the behavior I am seeing):

```auto
from pathlib import Path

from tokenizers import BertWordPieceTokenizer
from transformers import BertTokenizer

def test_text():
    text = [
        "This is a test, just a test",
        "nothing more, nothing less"
    ]

    return text

def create_test_files():
    test_path = Path("tmp")
    test_path.mkdir()

    test_data = test_text()

    for idx, text in enumerate(test_data):
        file = test_path.joinpath(f"file{idx}.txt")
        with open(file, "w") as f:
            f.write(text)

    return test_path

def cleanup_test(path):
    path = Path(path)

    for child in path.iterdir():
        if child.is_file():
            child.unlink()
        else:
            rm_tree(child)

    path.rmdir()

def create_tokenizer_savepath():
    savepath = Path("./bert")
    savepath.mkdir()
    return str(savepath)

def main():
    # Saving two text files to train the tokenizer
    test_path = create_test_files()

    files = test_path.glob("**/*.txt")
    files = [str(f) for f in files]

    tokenizer = BertWordPieceTokenizer(
        clean_text=True,
        strip_accents=True,
        lowercase=True,
    )

    tokenizer.train(
        files,
        vocab_size=15,
        min_frequency=1,
        show_progress=True,
        special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
        limit_alphabet=1000,
        wordpieces_prefix="##",
    )

    savepath = create_tokenizer_savepath()
    tokenizer.save_model(savepath, "pubmed_bert")

    tokenizer = BertTokenizer.from_pretrained(
        f"{savepath}/pubmed_bert-vocab.txt",
        max_len=512
    )

    print(tokenizer)

    cleanup_test(test_path)
    cleanup_test(savepath)

if __name__ == " __main__":
    main()

```

## Loading the Trained Tokenizer

Specifying the path to the `pubmed_bert-vocab.txt` is deprecated:

```auto
Calling BertTokenizer.from_pretrained() with the path to a single file or url is deprecated
PreTrainedTokenizer(name_or_path='bert/pubmed_bert-vocab.txt', vocab_size=30, model_max_len=512, is_fast=False, padding_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'})

```

But, if I just specify the path to the directory containing `pubmed_bert-vocab.txt`:

```auto
Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1777, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load tokenizer for 'bert'. Make sure that:

- 'bert' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert' is the correct path to a directory containing relevant tokenizer files

```

The directory I am saving to only contains `pubmed_bert-vocab.txt`. If specifying the full path to that vocab is deprecated, what is the best way to load that tokenizer?

## Using BertTokenizerFast

If I swap out BertTokenizer for BertTokenizerFast, and pass in the path to the directory where I have saved my tokenizer trained from scratch, I get the same error:

```auto
Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1777, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load tokenizer for 'bert'. Make sure that:

- 'bert' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert' is the correct path to a directory containing relevant tokenizer files

```

And if I specify the path to file saved by my tokenizer (`pubmed_bert-vocab.txt`), I get a ValueError (vs the deprecation warning I was getting using BertTokenizer):

```auto
Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1696, in from_pretrained
    "Use a model identifier or the path to a directory instead.".format(cls. __name__ )
ValueError: Calling BertTokenizerFast.from_pretrained() with the path to a single file or url is not supported.Use a model identifier or the path to a directory instead.

```

## Current Approach

I am currently using `BertTokenizer`, specifying the full path the the `pubmed_bert-vocab.txt` and am ignoring the deprecation warning, but ideally I would like use `BertTokenizerFast` but don’t know how to load in my saved tokenizer. What is the best way to go forward on this?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [March 15, 2021, 8:21am UTC](https://discuss.huggingface.co/t/using-a-bertwordpiecetokenizer-trained-from-scratch-from-transformers/4391/2 "2021-03-15T08:21:09Z")

</div>

pinging @anthony

---

<div class="post-metadata">

**Author:** ![yngtodd](https://avatars.discourse-cdn.com/v4/letter/y/c0e974/32.png) [@yngtodd](https://discuss.huggingface.co/u/yngtodd)\
**Post date:** [March 26, 2021, 2:30pm UTC](https://discuss.huggingface.co/t/using-a-bertwordpiecetokenizer-trained-from-scratch-from-transformers/4391/3 "2021-03-26T14:30:59Z")

</div>

Should I open an issue on the tokenizers repository for this?
