# Streaming Dataset Roberta

**URL:** <https://discuss.huggingface.co/t/streaming-dataset-roberta/12479>\
**Category:** 🤗Datasets\
**Created:** [December 2, 2021, 6:26pm UTC](https://discuss.huggingface.co/t/streaming-dataset-roberta/12479 "2021-12-02T18:26:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![pere](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pere/32/40613_2.png) [@pere](https://discuss.huggingface.co/u/pere)\
**Post date:** [December 2, 2021, 6:26pm UTC](https://discuss.huggingface.co/t/streaming-dataset-roberta/12479/1 "2021-12-02T18:26:46Z")

</div>

Anyone know of RoBERTa pretraining script with support for Dataset streadmin?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [December 7, 2021, 12:07pm UTC](https://discuss.huggingface.co/t/streaming-dataset-roberta/12479/2 "2021-12-07T12:07:52Z")

</div>

Hi ! I don’t think the community has already shared a script for RoBERTa pretraining using dataset streaming yet. However if you’re interested in looking into this, here are a few pointers:

RoBERTa was trained with [BookCorpus](https://bookcorpus), [CC news](https://huggingface.co/datasets/cc_news) and [OpenWebText](https://openwebtext)

BookCorpus and OpenWebText have been replicated and open sourced as [BookCorpusOpen](https://huggingface.co/datasets/bookcorpusopen) and [OpenWebText2 (The Pile)](https://huggingface.co/datasets/the_pile_openwebtext2)

You can load and interleave the datasets with

```python
from datasets import load_dataset, interleave_datasets

def only_keep_text(example):
    return {"text": example["text"]}

bc = load_dataset("bookcorpusopen", split="train", streaming=True)
ccn = load_dataset("cc_news", split="train", streaming=True)
# this one currently has streaming issues - will fix soon
# owt = load_dataset("the_pile_openwebtext2", split="train", streaming=True)  

dataset = interleave_datasets([
    bc.map(only_keep_text),
    ccn.map(only_keep_text),
    # owt.map(only_keep_text)
])

```

Then you can check the documentation to see how to use it in a pytorch training loop: [Stream — datasets 1.16.1 documentation](https://huggingface.co/docs/datasets/stream.html#map)