# KeyError: '\_data' when training on AWS **URL:** \ **Category:** Amazon SageMaker\ **Created:** [April 23, 2021, 5:07am UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711 "2021-04-23T05:07:16Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![cccx3](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cccx3/32/1546_2.png) [@cccx3](https://discuss.huggingface.co/u/cccx3)\ **Post date:** [April 23, 2021, 5:07am UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/1 "2021-04-23T05:07:16Z")
Hi all, I’ve been working through adapting [the getting started notebook](https://github.com/huggingface/notebooks/blob/master/sagemaker/01_getting_started_pytorch/sagemaker-notebook.ipynb) to my particular use case. I wrote out my data to s3, and kicked off .fit(), but am getting this error block: ``` 2021-04-23 04:58:40,552 sagemaker-training-toolkit ERROR ExecuteUserScriptError: Command "/opt/conda/bin/python3.6 train.py --epochs 1 --model_name distilbert-base-uncased --train_batch_size 32" Traceback (most recent call last): File "train.py", line 42, in train_dataset = load_from_disk(args.training_dir) File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 781, in load_from_disk return Dataset.load_from_disk(dataset_path, fs) File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 684, in load_from_disk state = {k: state[k] for k in dataset. __dict__.keys()} # in case we add new fields File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 684, in state = {k: state[k] for k in dataset. __dict__.keys()} # in case we add new fields KeyError: '_data' ``` What’s leaving me scratching my head is that when I reference the [arrow\_dataset.py](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py) file, I can’t find lines of this kind, making me think there’s some discrepancy in whatever AWS’ container is and that file. Regardless, does anyone have any advice/intuition on what may be going on here? I don’t know what the ‘\_data’ key would refer to in this case, and am looking for help. Thanks! ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [April 23, 2021, 5:47am UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/2 "2021-04-23T05:47:21Z")
Hey @cccx3, Thank you for creating this topic! There was an error in the `01_getting_started_pytorch` where it installed `datasets 1.6.0`, which has some changes, and in the DLC currently 1.5.0 is installed. This was already fixed in a [PR](https://github.com/huggingface/notebooks/blob/master/sagemaker/01_getting_started_pytorch/sagemaker-notebook.ipynb) yesterday. To solve this you need to install `datasets==1.5.0` in your notebook and pre-process the data again. ```python !pip install "sagemaker>=2.31.0" "transformers==4.4.2" "datasets[s3]==1.5.0" --upgrade ``` ---
**Author:** ![nreamaroon](https://avatars.discourse-cdn.com/v4/letter/n/eada6e/32.png) [@nreamaroon](https://discuss.huggingface.co/u/nreamaroon)\ **Post date:** [April 26, 2021, 3:32pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/3 "2021-04-26T15:32:15Z")
I had the same error and this fixed it for me, thanks! Do you know if/when `datasets` will be updated to 1.6.0 in the HuggingFace DLC? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [April 26, 2021, 3:46pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/4 "2021-04-26T15:46:38Z")
We are going to include it with the next `transformers` release (`4.6.0`). If you want to use it in advance you can add a `requirements.txt` in your `source_dir` with `datasets==1.6.1` in it. ---
**Author:** ![anikalburgi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anikalburgi/32/2530_2.png) [@anikalburgi](https://discuss.huggingface.co/u/anikalburgi)\ **Post date:** [August 19, 2021, 12:47pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/5 "2021-08-19T12:47:46Z")
I’m facing the same issue with following lib versions: datasets == 1.11.0 sagemaker == 2.48.1 UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2021-08-19-12-34-40-568: Failed. Reason: AlgorithmError: ExecuteUserScriptError: Command “/opt/conda/bin/python3.6 train.py --epochs 3 --model\_name bert-base-uncased --train\_batch\_size 16” Traceback (most recent call last): File “train.py”, line 41, in train\_dataset = load\_from\_disk(args.training\_dir) File “/opt/conda/lib/python3.6/site-packages/datasets/load.py”, line 781, in load\_from\_disk return Dataset.load\_from\_disk(dataset\_path, fs) File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 684, in load\_from\_disk state = {k: state[k] for k in dataset. **dict**.keys()} # in case we add new fields File “/opt/conda/lib/python3.6/site-packages/datasets/arrow\_dataset.py”, line 684, in state = {k: state[k] for k in dataset. **dict**.keys()} # in case we add new fields KeyError: ‘\_data’ ---
**Author:** ![anikalburgi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anikalburgi/32/2530_2.png) [@anikalburgi](https://discuss.huggingface.co/u/anikalburgi)\ **Post date:** [August 19, 2021, 12:51pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/7 "2021-08-19T12:51:07Z")
Is datasets version 1.5.0 strictly necessary? ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [August 19, 2021, 1:59pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/8 "2021-08-19T13:59:10Z")
Hey @anikalburgi, Since the creation of this topic we released new DLC container, take a look here: [Hugging Face on Amazon SageMaker](https://huggingface.co/docs/sagemaker/main#deep-learning-container-dlc-overview) You can use `transformers_version=4.6.1` in your Estimator to use the image with `datasets 1.6.2`. ---
**Author:** ![anikalburgi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anikalburgi/32/2530_2.png) [@anikalburgi](https://discuss.huggingface.co/u/anikalburgi)\ **Post date:** [August 19, 2021, 4:51pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/9 "2021-08-19T16:51:23Z")
Perfect, Thanks! This works! Although, I’m facing CUDA Out of Memory issue with batch size even as low as 8 on p3.2xlarge instance for bert-base-uncased. Usually, when trained from torch directly and outside sagemaker, the batch size of 14-16 safely works for me on both g4dn.xlarge and p3.2xlarge instances. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [August 19, 2021, 5:03pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/10 "2021-08-19T17:03:31Z")
Can you share your `HuggingFace` estimator and the script you use? ---
**Author:** ![anikalburgi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anikalburgi/32/2530_2.png) [@anikalburgi](https://discuss.huggingface.co/u/anikalburgi)\ **Post date:** [August 20, 2021, 1:12pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/11 "2021-08-20T13:12:55Z")
Nah, It was my bad, EOD rush - added \_'s instead of -'s in batch size arg It’s fixed and working smoothly! ---
**Author:** ![anikalburgi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anikalburgi/32/2530_2.png) [@anikalburgi](https://discuss.huggingface.co/u/anikalburgi)\ **Post date:** [August 20, 2021, 1:13pm UTC](https://discuss.huggingface.co/t/keyerror-data-when-training-on-aws/5711/12 "2021-08-20T13:13:37Z")
Thank you @philschmid