# Cardinality issue when training bert from scratch (tensorflow)

**URL:** <https://discuss.huggingface.co/t/cardinality-issue-when-training-bert-from-scratch-tensorflow/9546>\
**Category:** 🤗Transformers\
**Created:** [August 28, 2021, 2:13am UTC](https://discuss.huggingface.co/t/cardinality-issue-when-training-bert-from-scratch-tensorflow/9546 "2021-08-28T02:13:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [August 28, 2021, 2:13am UTC](https://discuss.huggingface.co/t/cardinality-issue-when-training-bert-from-scratch-tensorflow/9546/1 "2021-08-28T02:13:10Z")

</div>

Hello there!

I am trying to adapt the official [Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling_from_scratch.ipynb#scrollTo=n2ZRs1cL3l-R) for language generation to `tensorflow` and everything seems to work wonderfully by simply appending `TF` to most of the `huggingface` function calls (`TFAutoModel`, etc)

Unfortunately, this strategy fails at the `training` step:

```auto
from transformers import TFTrainer, TFTrainingArguments
import tensorflow as tf

training_args = TFTrainingArguments(
    "test-clm",
    evaluation_strategy = "epoch",
    learning_rate=2e-5)

trainer = TFTrainer(
    model=model,
    args = training_args,
    train_dataset=lm_datasets[0:1000],
    eval_dataset=lm_datasets[1000:])

trainer.train()

 self.num_train_examples = self.train_dataset.cardinality().numpy()
AttributeError: 'dict' object has no attribute 'cardinality'

```

I have absolutely no idea what this `cardinality` is. Do you know what the issue can be?  
Thanks!

---

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [August 29, 2021, 12:20am UTC](https://discuss.huggingface.co/t/cardinality-issue-when-training-bert-from-scratch-tensorflow/9546/2 "2021-08-29T00:20:40Z")

</div>

I saw a related issue here [TensorFlow Question-Answering example fails to run (cardinality error) · Issue #10246 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/10246) … huggingface masters, do you have an idea?

Thanks!

---

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [August 29, 2021, 8:56pm UTC](https://discuss.huggingface.co/t/cardinality-issue-when-training-bert-from-scratch-tensorflow/9546/3 "2021-08-29T20:56:28Z")

</div>

@BramVanroy sorry to pull you in, but I wonder if you have any clues about what is happening here? is this a tensorflow specific issue?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [August 30, 2021, 6:50am UTC](https://discuss.huggingface.co/t/cardinality-issue-when-training-bert-from-scratch-tensorflow/9546/4 "2021-08-30T06:50:23Z")

</div>

Seems like a potential bug. I’d recommend you to create a bug report on Github. But please provide the whole error trace and not just the last line, thanks.
