# KeyError: 'length' when loading dataset by load\_from\_disk **URL:** \ **Category:** 🤗Datasets\ **Created:** [March 25, 2023, 8:43am UTC](https://discuss.huggingface.co/t/keyerror-length-when-loading-dataset-by-load-from-disk/34627 "2023-03-25T08:43:01Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![yufengshen](https://avatars.discourse-cdn.com/v4/letter/y/a8b319/32.png) [@yufengshen](https://discuss.huggingface.co/u/yufengshen)\ **Post date:** [March 25, 2023, 8:43am UTC](https://discuss.huggingface.co/t/keyerror-length-when-loading-dataset-by-load-from-disk/34627/1 "2023-03-25T08:43:01Z")
1. I created and save the squad\_dataset using my mac machine by the using the code below. import os import torch from datasets import load\_dataset, load\_from\_disk dataset\_path = “./squad\_dataset” if not os.path.exists(dataset\_path): squad = load\_dataset(“squad”, split=“train[:5000]”) squad.save\_to\_disk(dataset\_path) then I load it to verify the saved squad\_dataset working good. squad = load\_from\_disk(dataset\_path) 2)then I code the squad\_dataset folder from my MAC to my another machine(with GPU installed) and got the exception "KeyError: ‘length’ ", so how to fix it ? ， import torch from datasets import load\_dataset from datasets import load\_from\_disk import sys squad = load\_from\_disk(“xxxxxx/squad\_dataset”) * * * KeyError Traceback (most recent call last) in 4 import sys 5 ----\> 6 squad = load\_from\_disk(“xxxxx/squad\_dataset”) 7 8 /conda/envs/notebook/lib/python3.6/site-packages/datasets/load.py in load\_from\_disk(dataset\_path, fs, keep\_in\_memory) 1804 raise FileNotFoundError(f"Directory {dataset\_path} not found") 1805 if fs.isfile(Path(dest\_dataset\_path, config.DATASET\_INFO\_FILENAME).as\_posix()): → 1806 return Dataset.load\_from\_disk(dataset\_path, fs, keep\_in\_memory=keep\_in\_memory) 1807 elif fs.isfile(Path(dest\_dataset\_path, config.DATASETDICT\_JSON\_FILENAME).as\_posix()): 1808 return DatasetDict.load\_from\_disk(dataset\_path, fs, keep\_in\_memory=keep\_in\_memory) /conda/envs/notebook/lib/python3.6/site-packages/datasets/arrow\_dataset.py in load\_from\_disk(dataset\_path, fs, keep\_in\_memory) 1211 state = json.load(state\_file) 1212 with open(Path(dataset\_path, config.DATASET\_INFO\_FILENAME).as\_posix(), encoding=“utf-8”) as dataset\_info\_file: → 1213 dataset\_info = DatasetInfo.from\_dict(json.load(dataset\_info\_file)) 1214 1215 dataset\_size = estimate\_dataset\_size( /conda/envs/notebook/lib/python3.6/site-packages/datasets/info.py in from\_dict(cls, dataset\_info\_dict) 270 def from\_dict(cls, dataset\_info\_dict: dict) → “DatasetInfo”: 271 field\_names = {f.name for f in dataclasses.fields(cls)} → 272 return cls(\*\*{k: v for k, v in dataset\_info\_dict.items() if k in field\_names}) 273 274 def update(self, other\_dataset\_info: “DatasetInfo”, ignore\_none=True): in **init** (self, description, citation, homepage, license, features, post\_processed, supervised\_keys, task\_templates, builder\_name, config\_name, version, splits, download\_checksums, download\_size, post\_processing\_size, dataset\_size, size\_in\_bytes) /conda/envs/notebook/lib/python3.6/site-packages/datasets/info.py in **post\_init** (self) 139 # Convert back to the correct classes when we reload from dict 140 if self.features is not None and not isinstance(self.features, Features): → 141 self.features = Features.from\_dict(self.features) 142 if self.post\_processed is not None and not isinstance(self.post\_processed, PostProcessedInfo): 143 self.post\_processed = PostProcessedInfo.from\_dict(self.post\_processed) /conda/envs/notebook/lib/python3.6/site-packages/datasets/features/features.py in from\_dict(cls, dic) 1595 {‘\_type’: Value(dtype=‘string’, id=None)} 1596 “”" → 1597 obj = generate\_from\_dict(dic) 1598 return cls(\*\*obj) 1599 /conda/envs/notebook/lib/python3.6/site-packages/datasets/features/features.py in generate\_from\_dict(obj) 1278 # Otherwise we have a dict or a dataclass 1279 if “\_type” not in obj or isinstance(obj[“\_type”], dict): → 1280 return {key: generate\_from\_dict(value) for key, value in obj.items()} 1281 class\_type = globals()[obj.pop(“\_type”)] 1282 /conda/envs/notebook/lib/python3.6/site-packages/datasets/features/features.py in (.0) 1278 # Otherwise we have a dict or a dataclass 1279 if “\_type” not in obj or isinstance(obj[“\_type”], dict): → 1280 return {key: generate\_from\_dict(value) for key, value in obj.items()} 1281 class\_type = globals()[obj.pop(“\_type”)] 1282 /conda/envs/notebook/lib/python3.6/site-packages/datasets/features/features.py in generate\_from\_dict(obj) 1282 1283 if class\_type == Sequence: → 1284 return Sequence(feature=generate\_from\_dict(obj[“feature”]), length=obj[“length”]) 1285 1286 field\_names = {f.name for f in fields(class\_type)} KeyError: ‘length’ ---
**Author:** ![qiuwenbo](https://avatars.discourse-cdn.com/v4/letter/q/a88e57/32.png) [@qiuwenbo](https://discuss.huggingface.co/u/qiuwenbo)\ **Post date:** [April 21, 2024, 8:03am UTC](https://discuss.huggingface.co/t/keyerror-length-when-loading-dataset-by-load-from-disk/34627/2 "2024-04-21T08:03:43Z")
After I installed datasets 2.12.0 (recent version), the error is disappered.