# Dataset Viewer not available on features of type datasets.Array2D(shape=(None, 768), dtype='float64')

**URL:** <https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828>\
**Category:** 🤗Datasets\
**Created:** [May 13, 2025, 5:22am UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828 "2025-05-13T05:22:30Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![stonezh](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@stonezh](https://discuss.huggingface.co/u/stonezh)\
**Post date:** [May 13, 2025, 5:22am UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828/1 "2025-05-13T05:22:30Z")

</div>

The data we upload to huggingface contains some features that have a flexible time axis. For example, for each sample it can be of shape (T, 768), T varies. According to the official doc, huggingface does support this type of 2D array by leaving the first dim as None, as I put here:

```python
features = datasets.Features(
    {
        "video_id": datasets.Value("string"),
        "label": datasets.Value("string"),
        "visual_feature": datasets.Array2D(shape=(None, 768), dtype='float64'), # (T, 768), T can vary
    },
)

```

But when I submit it to huggingface, I get this error on the dataset viewer panel:

```auto
Error code: StreamingRowsError
Exception: ValueError
Message: shape=(768,) and ndims=2 don't match
Traceback: Traceback (most recent call last):
                File "/src/services/worker/src/worker/utils.py", line 99, in get_rows_or_raise
                  return get_rows(
                File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
                  return func(*args, **kwargs)
                File "/src/services/worker/src/worker/utils.py", line 77, in get_rows
                  rows_plus_one = list(itertools.islice(ds, rows_max_number + 1))
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 2270, in __iter__
                  for key, example in ex_iterable:
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 1856, in __iter__
                  for key, pa_table in self._iter_arrow():
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 1879, in _iter_arrow
                  for key, pa_table in self.ex_iterable._iter_arrow():
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 476, in _iter_arrow
                  for key, pa_table in iterator:
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 323, in _iter_arrow
                  for key, pa_table in self.generate_tables_fn(**gen_kwags):
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 106, in _generate_tables
                  yield f"{file_idx}_{batch_idx}", self._cast_table(pa_table)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 73, in _cast_table
                  pa_table = table_cast(pa_table, self.info.features.arrow_schema)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1826, in arrow_schema
                  return pa.schema(self.type).with_metadata({"huggingface": json.dumps(hf_metadata)})
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1815, in type
                  return get_nested_type(self)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1253, in get_nested_type
                  {key: get_nested_type(schema[key]) for key in schema}
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1253, in <dictcomp>
                  {key: get_nested_type(schema[key]) for key in schema}
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1277, in get_nested_type
                  return schema()
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 551, in __call__
                  pa_type = globals()[self. __class__. __name__ + "ExtensionType"](self.shape, self.dtype)
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 665, in __init__
                  raise ValueError(f"shape={shape} and ndims={self.ndims} don't match")
              ValueError: shape=(768,) and ndims=2 don't match

```

Is it an expected behavior that Array2D with first dim being None cannot display and the whole dataset viewer panel stops functioning? Or I am doing something wrong with my dataset uploading script?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [May 13, 2025, 7:07am UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828/2 "2025-05-13T07:07:38Z")

</div>

The Dataset library anticipates cases where the dimensions of data change dynamically, but the DatasetViewer library does not yet anticipate this. This is probably a ToDo rather than a bug. @lhoestq

> <https://github.com/huggingface/dataset-viewer/blob/main/services/worker/src/worker/utils.py#L99>

> **[Dataset features](https://huggingface.co/docs/datasets/en/about_dataset_features)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> The array type also allows the first dimension of the array to be dynamic. This is useful for handling sequences with variable lengths such as sentences, without having to pad or truncate the input to a uniform shape.  
> features = Features({‘a’: Array3D(shape=(None, 5, 2), dtype=‘int32’)})

---

<div class="post-metadata">

**Author:** ![stonezh](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@stonezh](https://discuss.huggingface.co/u/stonezh)\
**Post date:** [May 13, 2025, 1:36pm UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828/3 "2025-05-13T13:36:30Z")

</div>

Hi thanks for confirming that! Also want to know will this array2d of flexible first dim prevent huggingface from automatically generating croissant meta? I’ve made several attempts on this, and notice that if the dataset contains one such array2d then after a short delay even the viewer still shows the error I can get croissant meta, but when the dataset contains multiple, say 3, each of dim (None, 768) (None, 1024) (None, 1024), then I get no croissant and no auto generated parquet branch

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [May 14, 2025, 2:26am UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828/4 "2025-05-14T02:26:08Z")

</div>

If the data can be converted to parquet format, I think metadata will probably be created eventually, but maybe there is an error in the converter…🤔

> [@Generating Croissant Metadata for Custom Image Dataset](https://discuss.huggingface.co/t/generating-croissant-metadata-for-custom-image-dataset/150255/6):
>
> The croissant metadata on HF are generated automatically for datasets in supported formats like Parquet or [ImageFolder (folder of images and a metadata file )](https://huggingface.co/docs/datasets/v3.5.0/en/image_dataset#imagefolder). If you convert your dataset to Parquet, or if you structure your dataset as a ImageFolder, the croissant metadata will be available. There is no way to automatically get a croissant metadata file for a dataset based on a python script.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [May 14, 2025, 1:03pm UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828/5 "2025-05-14T13:03:08Z")

</div>

hi, if the shape is (768,) then it’s a not a Array2D (but a Array1D but it doesn’t exist, you should use Sequence instead)

---

<div class="post-metadata">

**Author:** ![stonezh](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@stonezh](https://discuss.huggingface.co/u/stonezh)\
**Post date:** [May 14, 2025, 3:12pm UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828/6 "2025-05-14T15:12:11Z")

</div>

When I upload dataset containing 3 such Array2D ( (None, 768) (None, 1024) (None, 1024)) by loading them locally into memory, formatting to a `pandas.Dataframe`, calling `from_pandas()` function to convert to dataset, then using `push_to_hug` to submit it, I do see the parquet files in /data folder, but I do not see the parquet branch autogenerated even after say 8 hours. If I only keep 1 Array2D, then I do observed the parquet branch & croissant meta

---

<div class="post-metadata">

**Author:** ![stonezh](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@stonezh](https://discuss.huggingface.co/u/stonezh)\
**Post date:** [May 14, 2025, 3:16pm UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828/7 "2025-05-14T15:16:33Z")

</div>

No the actual shape is (T, 768) but T is not fixed. I used Sequence of Float on 1D array and it works fine, just this Array2D of shape (None, 768) causes trouble. My understanding is the `shape=(768,)` in the error log means huggingface processes Array2D of shape (None, 768) to a 1D array of shape 768 only, and `and ndims=2 don't match` means it actually gets the matrix of shape (T, 768), so the data is correctly uploaded, but the format is not correctly parsed by huggingface

---

<div class="post-metadata">

**Author:** ![stonezh](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@stonezh](https://discuss.huggingface.co/u/stonezh)\
**Post date:** [May 14, 2025, 3:30pm UTC](https://discuss.huggingface.co/t/dataset-viewer-not-available-on-features-of-type-datasets-array2d-shape-none-768-dtype-float64/154828/8 "2025-05-14T15:30:08Z")

</div>

@John6666 @lhoestq  
So my current fix is to pad all the (T, 768) to (max\_T, 768), then specify that max\_T in array2D. It roughly works, and huggingface can generate parquet branch & croissant meta, but the padding makes the files too big and croissant meta just skips those features and dataset viewer is also only showing the first 5GB, so idk if it is really a solution or just huggingface skip those Array2D therefore skip all the possible issues lol