# Map result saved to a different folder than custom HF\_DATASETS\_CACHE

**URL:** <https://discuss.huggingface.co/t/map-result-saved-to-a-different-folder-than-custom-hf-datasets-cache/18978>\
**Category:** 🤗Datasets\
**Created:** [June 10, 2022, 10:05pm UTC](https://discuss.huggingface.co/t/map-result-saved-to-a-different-folder-than-custom-hf-datasets-cache/18978 "2022-06-10T22:05:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![luckynozomi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/luckynozomi/32/5042_2.png) [@luckynozomi](https://discuss.huggingface.co/u/luckynozomi)\
**Post date:** [June 10, 2022, 10:05pm UTC](https://discuss.huggingface.co/t/map-result-saved-to-a-different-folder-than-custom-hf-datasets-cache/18978/1 "2022-06-10T22:05:54Z")

</div>

I followed the documentation [here](https://huggingface.co/docs/datasets/v2.2.1/en/cache#cache-directory) to set my environment variable like this

```python
import os
os.environ["HF_DATASETS_CACHE"] = os.path.join(os.getcwd(), "cache")

```

but when I load a custom dataset

```python
from datasets import load_from_disk
my_dataset = load_from_disk("my_dataset")
train_dataset = my_dataset["train"]

```

the caching directory, from `train_dataset.cache_files`, seems to still be pointing to the directory of `my_dataset`, see code [here](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L2589)

(then `self._get_cache_file_path` calls `self.cache_files`)

Is this an intended behavior? Is there any way I can cache all intermediate results to HF\_DATASETS\_CACHE?

also `cache_dir` is not supported in `load_from_disk`

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 14, 2022, 1:23pm UTC](https://discuss.huggingface.co/t/map-result-saved-to-a-different-folder-than-custom-hf-datasets-cache/18978/2 "2022-06-14T13:23:47Z")

</div>

Hi! That’s expected. `load_from_disk` loads arrow files from a given directory, so `train_dataset.cache_files` should point to `my_dataset/train/dataset.arrow` (there is nothing to cache here).

> Is this an intended behavior? Is there any way I can cache all intermediate results to HF\_DATASETS\_CACHE?

Yes, if you save your dataset to `HF_DATASETS_CACHE`, but this is not advised as it can lead to name collisions (this directory stores the arrow files generated by `load_dataset` or `map` when called on the datasets from there).  
.