# Confusion in splitting dataset (from imagefolder) into train, test and validation

**URL:** <https://discuss.huggingface.co/t/confusion-in-splitting-dataset-from-imagefolder-into-train-test-and-validation/20187>\
**Category:** 🤗Datasets\
**Created:** [July 9, 2022, 4:09pm UTC](https://discuss.huggingface.co/t/confusion-in-splitting-dataset-from-imagefolder-into-train-test-and-validation/20187 "2022-07-09T16:09:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![joyjit1996](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joyjit1996/32/5324_2.png) [@joyjit1996](https://discuss.huggingface.co/u/joyjit1996)\
**Post date:** [July 9, 2022, 4:09pm UTC](https://discuss.huggingface.co/t/confusion-in-splitting-dataset-from-imagefolder-into-train-test-and-validation/20187/1 "2022-07-09T16:09:10Z")

</div>

Hi,

I am trying to load up images from dataset with the following structure for fine-tuning the vision transformer model. My dataset has following structure:

> DatasetFolder  
> — ClassA (x images)  
> ----ClassB (y images)  
> ----ClassC (z images)

I am quite confused on how to split the dataset into train, test and validation. I read various similar questions but couldn’t understand the process clearly. I have tried to do the following (train-test-validation split in a non-random manner, though actually I would like it to be randomly splitted):

```auto
from datasets import load_dataset

ds = load_dataset("imagefolder",data_dir="/Documents/DataSetFolder/",split="test")
# split up data into train + test

splits = ds.train_test_split(test_size=0.3)

train_ds = splits['train']

test_ds = splits['test']
# split up data into val + test

splits = ds.train_test_split(test_size=0.15)

test_ds = splits['test']

val_ds = splits['test']

```

Is this a correct process for randomly splitting the dataset into 70% training, 15% test and 15% validation? Also, how I can do a random train-test split and what is the significance of the **split** argument in **load\_dataset**? Would really appreciate some guidance on this as I am very confused even after reading the documentation at https://huggingface.co/docs/datasets/image\_process.

Thanks very much!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [July 13, 2022, 3:02pm UTC](https://discuss.huggingface.co/t/confusion-in-splitting-dataset-from-imagefolder-into-train-test-and-validation/20187/2 "2022-07-13T15:02:49Z")

</div>

Hi! Your code produces (potentially overlapping) splits of incorrect size. This is the fixed code:

```python
from datasets import load_dataset

ds = load_dataset("imagefolder", data_dir="/Documents/DataSetFolder/", split="test")

ds_split_train_test = ds.train_test_split(test_size=0.15)

train_ds, test_ds = ds_split_train_test["train"], ds_split_train_test["test"]

ds_split_train_val = train_ds.split_train_test(test_size=0.15/0.85)

train_ds, val_ds = ds_split_train_test["train"], ds_split_train_test["test"]

```

> what is the significance of the **split** argument in **load\_dataset**?

If specified, this argument returns a concrete dataset split/subset instead of returning a dictionary with all the subsets. You can think of it as being equal to `load_dataset("imagefolder", data_dir="/Documents/DataSetFolder/")[split]`. Note that this arg also supports the slicing syntax, but you shouldn’t use it here as this doesn’t shuffle the data.

---

<div class="post-metadata">

**Author:** ![akt42](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@akt42](https://discuss.huggingface.co/u/akt42)\
**Post date:** [August 12, 2022, 9:14am UTC](https://discuss.huggingface.co/t/confusion-in-splitting-dataset-from-imagefolder-into-train-test-and-validation/20187/3 "2022-08-12T09:14:30Z")

</div>

@mariosasko I have a slight variation of the question here. Maybe you can help me?

> [@Loading an imagenet-style image dataset with train/val directories](https://discuss.huggingface.co/t/loading-an-imagenet-style-image-dataset-with-train-val-directories/21554):
>
> This is my dataset structure: ROOT | -- train | ---- class\_1 | ---- class\_2 | -- val | ---- class\_1 | ---- class\_2 I want to load this in to a DataSet. I’m expecting the resulting DataSet to have a train and val split, but I only get a 'train' partition from the following: from datasets import load\_dataset dataset = load\_dataset( "imagefolder", data\_dir=ROOT ) \>\> dataset DatasetDict({ train: Dataset({ features: ['image', 'label'], num\_rows: 20 }) }) I can r…
