# \[Bug?\] Array3D(dtype="uint8") persists as int64 after save\_to\_disk despite explicit casting

**URL:** <https://discuss.huggingface.co/t/bug-array3d-dtype-uint8-persists-as-int64-after-save-to-disk-despite-explicit-casting/173326>\
**Category:** 🤗Datasets\
**Created:** [February 11, 2026, 7:20am UTC](https://discuss.huggingface.co/t/bug-array3d-dtype-uint8-persists-as-int64-after-save-to-disk-despite-explicit-casting/173326 "2026-02-11T07:20:07Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 12, 2026, 3:19am UTC](https://discuss.huggingface.co/t/bug-array3d-dtype-uint8-persists-as-int64-after-save-to-disk-despite-explicit-casting/173326/4 "2026-02-12T03:19:07Z")

</div>

I tested the behavior of the `datasets` library on Colab for now.

* * *

### Summary for your case

- **Your data is not being physically stored as `int64`** (in the typical case). If inspecting the backing Arrow column shows the **leaf value type is `uint8`** , then the on-disk buffers holding pixel values are `uint8`.
- The **`int64` you observe comes from `with_format("numpy")`**, which applies a _separate, on-the-fly conversion layer_ at ` __getitem__ ` time. This conversion has known dtype-mismatch bugs/footguns for array-like columns (e.g., returning `int64` for integer arrays even when `Features` says `uint8`). ([Hugging Face](https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes "Tensor returned :"))
- The **~2.6× disk bloat is explainable even with `uint8` values** due to **nested Arrow ListArray offset buffers** (structural overhead), not because values widened to `int64`. ([GitHub](https://github.com/huggingface/datasets/issues/4964 "Column of arrays (2D+) are using unreasonably high memory"))

* * *

## 1) Why do you see `int64` after loading?

### `with_format("numpy")` is not “just reading Arrow”; it converts on access

Hugging Face Datasets explicitly documents that formatting (including `"numpy"`) is **applied on-the-fly** and that `format_kwargs` are passed to conversion functions like `np.array`. ([Hugging Face](https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes "Tensor returned :"))

So in:

```python
loaded_ds = loaded_ds.with_format("numpy")
sample = loaded_ds[0]["image"]

```

you are not directly retrieving the Arrow leaf buffer as a NumPy view; you’re going through a conversion path that (today) can change dtypes.

### This dtype mismatch is a known, repeatedly reported problem

There are multiple issues showing the same pattern:

- `uint8` features but NumPy outputs `int64` (very similar to your complaint). ([GitHub](https://github.com/huggingface/datasets/issues/5834 "Is uint8 supported? #5834 - huggingface/datasets"))
- dtype mismatch for `Array2D` with `with_format("numpy")` even when dtype is user-specified. ([GitHub](https://github.com/huggingface/datasets/issues/7254 "Issue #7254 · huggingface/datasets"))
- `with_format("numpy")` silently downcasts floats (shows that formatting can alter dtype). ([GitHub](https://github.com/huggingface/datasets/issues/5517 "`with\_format("numpy")` silently downcasts float64 to float32 ..."))

### Practical implication

Your observation **“after loading it’s int64”** is consistent with a _formatting-layer_ behavior/bug, not necessarily Arrow storage.

* * *

## 2) Why does disk usage blow up (~2.6×)?

### Background: `Array3D` is stored as nested lists with offsets

For multi-dimensional arrays, Datasets converts a NumPy array into an Arrow ListArray by:

1. flattening values
2. wrapping them repeatedly in list layers, creating `int32` **offset arrays** for each list nesting level

The conversion code is visible in `numpy_to_pyarrow_listarray`: it creates offsets using `np.arange(n_offsets + 1) * step_offsets` with `pa.int32()`, then wraps values with `pa.ListArray.from_arrays(offsets, values)` in a loop. ([Hugging Face](https://huggingface.co/docs/datasets/v1.13.3/_modules/datasets/features/features.html "datasets.features.features — datasets 1.13.3 documentation"))

### Why offsets cost so much for images

Arrow List arrays store an **offset buffer** (length = number of lists + 1) that describes where each list starts/ends. ([Apache Arrow](https://arrow.apache.org/rust/arrow_buffer/buffer/offset/struct.OffsetBuffer.html "OffsetBuffer in arrow\_buffer::buffer::offset - Rust"))

For an image `(H, W, C)` represented as nested lists, one major overhead is at the “pixel list” level:

- there are `H * W` lists per image (each list length = `C`)
- offsets are `int32` → 4 bytes each
- offset buffer size per image at that level is approximately `(H*W + 1) * 4`

For your CIFAR→256×256×3 example, per image:

- raw `uint8` payload = `256*256*3 = 196,608` bytes
- pixel offsets ≈ `(256*256 + 1) * 4 = 262,148` bytes
- plus additional (smaller) offsets for the next list level(s), plus Arrow file/metadata overhead

That already puts you around ~2× before counting higher-level offsets and file overhead; empirically, this aligns with reports that `Array2D/Array3D` can have “unreasonably high” memory/storage overhead. ([GitHub](https://github.com/huggingface/datasets/issues/4964 "Column of arrays (2D+) are using unreasonably high memory"))

**This is why a 26GB directory can be plausible even if the leaf values are `uint8`** , because you’re paying for structure (offsets), not value widening.

* * *

## 3) Why `ds.cast(features_numpy)` doesn’t “enforce uint8 on disk” (as you expect)

### Case A (most common): disk is already uint8; only NumPy formatting is wrong

If Arrow leaf values are `uint8`, then `cast()` _did_ its job for storage—your “int64” symptom is from the on-the-fly formatting layer. ([Hugging Face](https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes "Tensor returned :"))

### Case B: even if `cast()` ensures leaf dtype, it doesn’t change the physical representation choice

`Features` specifies the serialization format and schema. ([Hugging Face](https://huggingface.co/docs/datasets/en/about_dataset_features "Dataset features"))  
But `Array3D` still uses the nested ListArray storage strategy described above, so `cast()` cannot remove offset buffers or switch you to a fixed-size tensor representation. ([Hugging Face](https://huggingface.co/docs/datasets/v1.13.3/_modules/datasets/features/features.html "datasets.features.features — datasets 1.13.3 documentation"))

So `cast()` may enforce the **leaf value type** , but it doesn’t guarantee:

- “no dtype change during `with_format("numpy")`”
- “disk size equals raw `uint8` payload”

* * *

## 4) What you should do (verification + fixes)

### A) Verify whether the Arrow leaf is `uint8` (this is the decisive test)

Do this **before** `with_format("numpy")`:

```python
import pyarrow as pa
from datasets import load_from_disk

ds = load_from_disk("test_dataset")
tbl = ds.data
col = tbl.column("image").chunk(0)

# unwrap extension storage
arr = col.storage if hasattr(col, "storage") else col

# walk down nested lists to leaf
while pa.types.is_list(arr.type) or pa.types.is_large_list(arr.type) or pa.types.is_fixed_size_list(arr.type):
    arr = arr.values

print("Leaf type:", arr.type)

```

If it prints `uint8`, your data is stored as `uint8` and the “int64” comes from formatting.

### B) Fix the **output dtype** (workaround for formatting bug)

Because `with_format` passes kwargs to the conversion function, force the dtype:

```python
import numpy as np
loaded = load_from_disk("test_dataset").with_format(
    "numpy",
    columns=["image"],
    dtype=np.uint8,
    output_all_columns=True,
)

```

This is consistent with how `with_format` is documented (format kwargs passed to `np.array`). ([Hugging Face](https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes "Tensor returned :"))

### C) Fix **disk size** (change representation; `Array3D` is offset-heavy)

If disk footprint matters at 256×256×3 scale, consider alternatives:

1. **Flatten + fixed-length `Sequence(length=H*W*C)`**  
HF staff explicitly states that when `Sequence.length` is specified, it’s stored as a **fixed PyArrow list** and “does not store the offsets,” reducing memory. ([Hugging Face Forums](https://discuss.huggingface.co/t/intention-of-the-length-field-in-class-datasets-sequence/34401 "Intention of the `length` field in class datasets.Sequence?"))

2. **Store encoded images (`Image()` feature) or raw bytes (`Value("binary")`)**  
These avoid “offset per pixel” overhead and typically reduce size dramatically.

* * *

## Direct answers to your two questions

1. **Does `Array3D` default to `int64` during Arrow serialization?**  
Not as a general rule. What’s widely reported is that **NumPy formatting (`with_format("numpy")`) can return `int64` regardless of the feature dtype** for array-like columns. ([GitHub](https://github.com/huggingface/datasets/issues/7254 "Issue #7254 · huggingface/datasets"))

2. **Why does `cast()` fail to enforce `uint8` on physical disk?**  
Usually it doesn’t; the leaf can still be `uint8`. The mismatch you see is typically **formatting-time conversion**. And independently, `cast()` doesn’t change that `Array3D` uses **nested ListArrays with `int32` offsets** , which drives up disk usage. ([Hugging Face](https://huggingface.co/docs/datasets/v1.13.3/_modules/datasets/features/features.html "datasets.features.features — datasets 1.13.3 documentation"))

---

_[View the full topic](https://discuss.huggingface.co/t/bug-array3d-dtype-uint8-persists-as-int64-after-save-to-disk-despite-explicit-casting/173326)._