# All paid GPU pools fail to start: MOUNT\_ATTR\_IDMAP error at container init (Exit code 128) **URL:** \ **Category:** Spaces\ **Created:** [September 25, 2026, 8:03am UTC](https://discuss.huggingface.co/t/all-paid-gpu-pools-fail-to-start-mount-attr-idmap-error-at-container-init-exit-code-128/180730 "2026-09-25T08:03:51Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![miguralex](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/miguralex/32/63353_2.png) [@miguralex](https://discuss.huggingface.co/u/miguralex)\ **Post date:** [September 25, 2026, 8:03am UTC](https://discuss.huggingface.co/t/all-paid-gpu-pools-fail-to-start-mount-attr-idmap-error-at-container-init-exit-code-128/180730/1 "2026-09-25T08:03:51Z")
Hi HF team, Since this morning, paid GPU Spaces fail to start on every GPU pool I can try (L4, A10G) with the same OCI runtime error at container-init time. I’ve confirmed the identical failure live on several unrelated public Spaces across T4 / L4 / A10G. ZeroGPU Spaces seem unaffected. \*\*Error (verbatim from the runtime errorMessage):\*\* ``` Exit code: 128. Reason: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: failed to fulfil mount request: failed to set MOUNT\_ATTR\_IDMAP on /usr/bin/nvidia-cuda-mps-control: invalid argument (maybe the filesystem used doesn’t support idmap mounts on this kernel?) ``` \*\*Reproduced today (UTC) on public Spaces — all still RUNTIME\_ERROR:\*\* | Space | Hardware | Notes | | --- | --- | --- | | ggml-org/gguf-my-repo | a10g-large | same error | | AlicanKiraz0/Trendyol-TTS-Space | t4-small | same error | | HuggingFaceM4/faster-qwen3-tts-demo | a10g-small | same error (Docker SDK) | | vLAR/GPU-Test | l4x1 | failed with this exact error at 07:52:59 UTC today; owner seems to have switched it to cpu-basic since | \*\*My Spaces\*\* (private; were starting and running fine earlier today with unchanged code): - l4x1: failed identically - a10g-large: failed identically - happy to share full build/container logs on request \*\*Troubleshooting already done:\*\* 1. Minimal repro: a Gradio Space whose entire app.py is `import gradio as gr`, `import spaces` and a trivial `@spaces.GPU` function — fails at the same container-init step, before any application code executes. So this is not application code. 2. Build completes fully (all pip installs succeed). The failure is strictly at runc container creation (mount of `/usr/bin/nvidia-cuda-mps-control`). 3. Both Gradio SDK (sdk\_version 5.50.0 and 6.26.0) and Docker SDK Spaces are affected. 4. [status.huggingface.co](http://status.huggingface.co) shows Spaces “Operational” and no incident is listed. \*\*Questions:\*\* 1. Is there a known incident affecting idmap mounts / user namespaces on paid GPU nodes? 2. If not — is there anything we can change on our side? The failure happens before any app code runs, and the only real lever left (hardware flavor) reproduces the error on both pools I’ve tried. ZeroGPU works, but its quotas don’t fit GPU-heavy workloads. Thanks a lot! ---
**Author:** ![Mobe1](https://avatars.discourse-cdn.com/v4/letter/m/46a35a/32.png) [@Mobe1](https://discuss.huggingface.co/u/Mobe1)\ **Post date:** [September 25, 2026, 8:29am UTC](https://discuss.huggingface.co/t/all-paid-gpu-pools-fail-to-start-mount-attr-idmap-error-at-container-init-exit-code-128/180730/2 "2026-09-25T08:29:49Z")
Same error on Hugging Face Jobs, not only Spaces, and it started earlier than the first reports here. Since about 00:10 UTC on 2026-09-25, every GPU job on my account fails 4 s after it starts with exit code 128: failed to set MOUNT\_ATTR\_IDMAP on /usr/bin/nvidia-cuda-mps-control: invalid argument Flavours affected: t4-small, l4x1, a10g-small. cpu-basic still runs. The last GPU job that succeeded was at 20:34 UTC on 2026-09-24. It fails on stock public images running nothing but a GPU check: nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi 07:47 and ~08:10 UTC pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel torch.cuda check ~07:51 UTC python:3.12-slim nvidia-smi 07:45 UTC The status page still shows Jobs as operational. Job IDs are available to HF staff on request. ---
**Author:** ![stephen4](https://avatars.discourse-cdn.com/v4/letter/s/ecd19e/32.png) [@stephen4](https://discuss.huggingface.co/u/stephen4)\ **Post date:** [September 25, 2026, 10:29am UTC](https://discuss.huggingface.co/t/all-paid-gpu-pools-fail-to-start-mount-attr-idmap-error-at-container-init-exit-code-128/180730/3 "2026-09-25T10:29:43Z")
If it’s also happening on Jobs with stock CUDA images, that pretty much rules out anything in the Space code. Hopefully HF can trace the GPU node or runtime issue soon. ---
**Author:** ![meganariley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/meganariley/32/56966_2.png) [@meganariley](https://discuss.huggingface.co/u/meganariley)\ **Post date:** [September 25, 2026, 5:25pm UTC](https://discuss.huggingface.co/t/all-paid-gpu-pools-fail-to-start-mount-attr-idmap-error-at-container-init-exit-code-128/180730/4 "2026-09-25T17:25:37Z")
Hey all, thanks for reporting this! This should now be resolved. Sorry for the disruption here!