# Fine-Tune BART using "Fine-Tuning Custom Datasets" doc

**URL:** <https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628>\
**Category:** Beginners\
**Created:** [October 19, 2020, 2:34pm UTC](https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628 "2020-10-19T14:34:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Buckeyes2019](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Buckeyes2019](https://discuss.huggingface.co/u/Buckeyes2019)\
**Post date:** [October 19, 2020, 2:34pm UTC](https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628/1 "2020-10-19T14:34:47Z")

</div>

I am trying to fine-tune BART for a summarization task using the code on the “Fine Tuning with Custom Dataset” page ([https://huggingface.co/transformers/custom\_datasets.html](https://huggingface.co/transformers/custom_datasets.html)). The data is a subset of the CNN/Daily Mail data.

I am encountering two different errors. The first comes when I implement the exact code on the page: _"TypeError: new(): invalid data type ‘str’_

I assume this is caused by the fact that the labels are not encoded/tokenized, instead they are strings. I see this is the case in the sample code on the page, the labels are not tokenized.

If I tokenize the labels and run the code I receive a different error: _“Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers”_

I am not sure how to interpret that. I get the same errors whether I am using the Trainer class or using native PyTorch. Any suggestions? Thanks!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 19, 2020, 3:07pm UTC](https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628/2 "2020-10-19T15:07:23Z")

</div>

It’s a bit hard to help you without seeing the code you run.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [October 19, 2020, 3:15pm UTC](https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628/3 "2020-10-19T15:15:39Z")

</div>

Hi @Buckeyes2019,

Not a direct answer to your question, but you can use the scripts in `examples/seq2seq` [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) (`finetune.py` or `finetune_trainer.py`) for fine-tuning BART and other s2s models. It supports custom datasets as well. All you’ll need to do is get the data in the required format mentioned in the redme.

---

<div class="post-metadata">

**Author:** ![Buckeyes2019](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Buckeyes2019](https://discuss.huggingface.co/u/Buckeyes2019)\
**Post date:** [October 19, 2020, 4:54pm UTC](https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628/4 "2020-10-19T16:54:39Z")

</div>

Sorry, here is the code that produces the error: _“Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers”_

\<—\>  
from sklearn.model\_selection import train\_test\_split  
from transformers import BartTokenizer

tokenizer = BartTokenizer.from\_pretrained(‘facebook/bart-large-cnn’)  
train\_texts, val\_texts, train\_labels, val\_labels = train\_test\_split(df.articles, df.highlights, test\_size=.2)  
train\_texts = train\_texts.values.tolist()  
train\_labels = train\_labels.values.tolist()  
val\_texts = val\_texts.values.tolist()  
val\_labels = val\_labels.values.tolist()

train\_encodings = tokenizer(train\_texts, truncation=True, padding=True)  
train\_label\_encodings = tokenizer(train\_labels, truncation=True, padding=True)  
val\_encodings = tokenizer(val\_texts, truncation=True, padding=True)  
val\_label\_encodings = tokenizer(val\_labels, truncation=True, padding=True)

import torch

class PyTorchDatasetCreate(torch.utils.data.Dataset):  
def **init** (self, encodings, labels):  
self.encodings = encodings  
self.labels = labels

```
def __getitem__ (self, idx):
    item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
    item['labels'] = torch.tensor(self.labels[idx])
    return item

def __len__ (self):
    return len(self.labels)

```

train\_dataset = PyTorchDatasetCreate(train\_encodings, train\_label\_encodings)  
val\_dataset = PyTorchDatasetCreate(val\_encodings, val\_label\_encodings)

from transformers import BartForConditionalGeneration, Trainer, TrainingArguments

training\_args = TrainingArguments(  
output\_dir=’./results’, # output directory  
num\_train\_epochs=3, # total number of training epochs  
per\_device\_train\_batch\_size=1, # batch size per device during training  
per\_device\_eval\_batch\_size=1, # batch size for evaluation  
warmup\_steps=200, # number of warmup steps for learning rate scheduler  
weight\_decay=0.01, # strength of weight decay  
logging\_dir=’./logs’, # directory for storing logs  
logging\_steps=10)

model = BartForConditionalGeneration.from\_pretrained(“sshleifer/distilbart-cnn-12-6”)

trainer = Trainer(  
model=model, # the instantiated 🤗 Transformers model to be trained  
args=training\_args, # training arguments, defined above  
train\_dataset=train\_dataset, # training dataset  
eval\_dataset=val\_dataset) # evaluation dataset

trainer.train()

\<----\>  
The code for the other error (_"TypeError: new(): invalid data type ‘str’_) is identical except the labels (train and val) are not tokenized. Appreciate your help!

---

<div class="post-metadata">

**Author:** ![993](https://avatars.discourse-cdn.com/v4/letter/9/90ced4/32.png) [@993](https://discuss.huggingface.co/u/993)\
**Post date:** [October 26, 2020, 1:20pm UTC](https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628/5 "2020-10-26T13:20:26Z")

</div>

Is the correct format the following:  
6 files

- train.source
- train.target
- val.source
- val.target
- test.source
- test.target

Each with one text per line in the .source file and the corresponding summarization in the same line in the .target file?

Is there a better documentation what else need to be changed (e.g. data\_dir or output\_dir)?  
In the readme it only states: “you need to specify data\_dir, output\_dir and model\_name\_or\_path”.  
Or does anyone have a public respo where they finetuned one model?

---

<div class="post-metadata">

**Author:** ![Buckeyes2019](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Buckeyes2019](https://discuss.huggingface.co/u/Buckeyes2019)\
**Post date:** [October 26, 2020, 2:57pm UTC](https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628/6 "2020-10-26T14:57:55Z")

</div>

I was able to get this to work: [https://ohmeow.com/posts/2020/05/23/text-generation-with-blurr.html](https://ohmeow.com/posts/2020/05/23/text-generation-with-blurr.html)

---

<div class="post-metadata">

**Author:** ![993](https://avatars.discourse-cdn.com/v4/letter/9/90ced4/32.png) [@993](https://discuss.huggingface.co/u/993)\
**Post date:** [October 28, 2020, 10:33am UTC](https://discuss.huggingface.co/t/fine-tune-bart-using-fine-tuning-custom-datasets-doc/1628/7 "2020-10-28T10:33:57Z")

</div>

Thanks for the link! But how can you use a custom dataset on that apporach?  
They use a 🤗 dataset. In the official documentation they are using a single csv file for training: `pd.read_csv(path/'cnndm_sample.csv')`, but shouldn’t you use at least train and validation?

**Edit:**  
The answer is:  
`dblock = DataBlock(blocks=blocks, get_x=ColReader('article'), get_y=ColReader('highlights'), splitter=RandomSplitter())`  
They split the single csv file into two dataloader object.