# Pre-training datasets for base and roberta

**URL:** <https://discuss.huggingface.co/t/pre-training-datasets-for-base-and-roberta/17853>\
**Category:** 🤗Datasets\
**Created:** [May 12, 2022, 3:24am UTC](https://discuss.huggingface.co/t/pre-training-datasets-for-base-and-roberta/17853 "2022-05-12T03:24:43Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![LinJiangxian](https://avatars.discourse-cdn.com/v4/letter/l/8e8cbc/32.png) [@LinJiangxian](https://discuss.huggingface.co/u/LinJiangxian)\
**Post date:** [May 12, 2022, 3:24am UTC](https://discuss.huggingface.co/t/pre-training-datasets-for-base-and-roberta/17853/1 "2022-05-12T03:24:43Z")

</div>

I find that the pre-training datasets for bert-base-uncased, roberta-base, roberta large… is same. wikipedia 19.3k and bookcorpus 8.4k. This is not consistent with the datasets described in paper. Is this an error? If it is, you may revise it.