# Building a Custom Tokenizer with HGF Dataset: Batch Iterator Best Practices

**URL:** <https://discuss.huggingface.co/t/building-a-custom-tokenizer-with-hgf-dataset-batch-iterator-best-practices/64927>\
**Category:** 🤗Datasets\
**Created:** [December 7, 2023, 11:23am UTC](https://discuss.huggingface.co/t/building-a-custom-tokenizer-with-hgf-dataset-batch-iterator-best-practices/64927 "2023-12-07T11:23:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rong-Tao](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rong-tao/32/26971_2.png) [@Rong-Tao](https://discuss.huggingface.co/u/Rong-Tao)\
**Post date:** [December 7, 2023, 11:23am UTC](https://discuss.huggingface.co/t/building-a-custom-tokenizer-with-hgf-dataset-batch-iterator-best-practices/64927/1 "2023-12-07T11:23:55Z")

</div>

Dear Hugging Face Community,

I am currently working on building my own tokenizer using a HGF dataset. The training of the tokenizer is initiated as follows:

```python
tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=25000)

```

My primary concern revolves around the creation of an efficient batch iterator. I am utilizing the OSCAR dataset, which is structured in this manner:

```php
Dataset({
    features: ['id', 'text', 'meta'],
    num_rows: 25113265
})

```

Based on the tutorials, a batch iterator can be constructed like this:

```python
def batch_iterator():
    for i in range(0, len(dataset), batch_size):
        yield dataset[i : i + batch_size]["text"]

```

However, I discovered that there’s a built-in iterator method available:

```python
batch_iter = ds.iter(batch_size=256)

```

This leads me to a couple of questions:

- How can I modify this built-in iterator to specifically select the ‘text’ subfield?
- Given the varying sizes of the text entries, should I apply any specific preprocessing or handling techniques to manage this variability?
- Also it seems that `ByteLevelBPETokenizer` have no attribute  
`train_new_from_iterator`. What can I do If I really want to use BPE? Do I really have to output the text into txt and then read it using, this sounds like a waste of disk space

```python
tokenizer.train(files, trainer)

```

Any insights or suggestions on these points would be greatly appreciated.

Thank you in advance for your assistance!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [December 7, 2023, 10:28pm UTC](https://discuss.huggingface.co/t/building-a-custom-tokenizer-with-hgf-dataset-batch-iterator-best-practices/64927/2 "2023-12-07T22:28:34Z")

</div>

Hi!

> How can I modify this built-in iterator to specifically select the ‘text’ subfield?

```python
def batch_iterator():
    for batch in ds.select_columns("text").iter(batch_size=256):
        yield batch["text"]

```

> Given the varying sizes of the text entries, should I apply any specific preprocessing or handling techniques to manage this variability?

No, this is not needed.

> Also it seems that `ByteLevelBPETokenizer` have no attribute  
> `train_new_from_iterator`. What can I do If I really want to use BPE? Do I really have to output the text into txt and then read it using, this sounds like a waste of disk space

You can use `train_from_iterator` to train it on an iterator. This method comes from the `tokenizers` library, and can be accessed via [`train_new_from_iterator`](https://github.com/huggingface/transformers/blob/0ea42ef0f9f71deba7775ead33afa0e493823d60/src/transformers/tokenization_utils_fast.py#L687) in `transformers`,

---

<div class="post-metadata">

**Author:** ![Rong-Tao](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rong-tao/32/26971_2.png) [@Rong-Tao](https://discuss.huggingface.co/u/Rong-Tao)\
**Post date:** [December 8, 2023, 2:07am UTC](https://discuss.huggingface.co/t/building-a-custom-tokenizer-with-hgf-dataset-batch-iterator-best-practices/64927/3 "2023-12-08T02:07:45Z")

</div>

Thanks! And the code goes as the following,

```python
from transformers import PreTrainedTokenizerFast
from tokenizers import ByteLevelBPETokenizer

tokenizer = PreTrainedTokenizerFast(
    tokenizer_object=ByteLevelBPETokenizer()
    )

```

Even though it is bit confusing to jump back and forth between transformer and tokenizer, I would expect all tokenizer stuff to be dealt within tokenizer classes, guess there are some architecture behind it. 🤔

---

<div class="post-metadata">

**Author:** ![Rong-Tao](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rong-tao/32/26971_2.png) [@Rong-Tao](https://discuss.huggingface.co/u/Rong-Tao)\
**Post date:** [December 8, 2023, 6:48am UTC](https://discuss.huggingface.co/t/building-a-custom-tokenizer-with-hgf-dataset-batch-iterator-best-practices/64927/4 "2023-12-08T06:48:50Z")

</div>

Hi, I am back again.

```python
from datasets import Dataset
from transformers import PreTrainedTokenizerFast
from tokenizers import ByteLevelBPETokenizer

#tokenizer = AutoTokenizer.from_pretrained("./gpt")
tokenizer = PreTrainedTokenizerFast(
    tokenizer_object=ByteLevelBPETokenizer()
    )

ds = Dataset.load_from_disk("../dataset/dutch.hf")
ds = ds.with_format("torch")

def batch_iterator():
    for batch in ds.select_columns("text").iter(batch_size=64):
        yield batch["text"]
        
tokenizer.train_new_from_iterator(batch_iterator(), 
                                  vocab_size=52_000)

tokenizer.save_pretrained("bpe-post")

```

I ran the code multiple times the kernel died

```php
[00:37:46] Pre-processing sequences █████████████████████████████████████ 0 / 0
[00:00:23] Tokenize words █████████████████████████████████████ 23818136 / 23818136
[00:02:43] Count pairs ██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
[00:10:26] Count pairs ███████████████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
[00:20:58] Count pairs ██████████████░ 23818100 / 23818136
Killed

```

Is it some hardware issue or some version issue?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [December 8, 2023, 3:20pm UTC](https://discuss.huggingface.co/t/building-a-custom-tokenizer-with-hgf-dataset-batch-iterator-best-practices/64927/5 "2023-12-08T15:20:12Z")

</div>

`PreTrainedTokenizerFast` is a class in `transformers` that wraps the `tokenizers` lib (they are implemented in Rust to make tokenization in `transformers` faster). Regarding the “Killed” issue, maybe these issues can help: [Issue search results · GitHub](https://github.com/search?q=repo%3Ahuggingface%2Ftokenizers+killed&type=issues). Otherwise, I suggest opening a new issue in the `tokenizers` repo.
