Inquiry About HF Spaces Usage — Multi-GPU Deployment Requirements

Spaces
1

Dear Hugging Face Team,

I hope this email finds you well. I am writing to inquire about deploying a multi-GPU environment on Hugging Face Spaces.

Specifically, my requirements are as follows:

  • I need a GPU environment with 8 GPUs, each with more than 40GB of VRAM per card.
  • Of these 8 GPUs, I plan to allocate 4 GPUs for deploying a generative model.
  • The remaining 4 GPUs will be used to deploy a vLLM model.

Could you please advise on the following:

  1. What is the recommended way to configure and deploy such a multi-GPU (8x GPU) Space on Hugging Face?
  2. Which GPU hardware tiers/instances on Spaces would meet the requirement of >40GB VRAM per card?
  3. Are there any specific configuration steps or best practices for splitting GPU resources across two different model deployments (4 GPUs for the generative model and 4 GPUs for vLLM) within the same Space, or would this require separate Spaces?
  4. Are there any pricing details or quota limitations I should be aware of for this kind of setup?

I would greatly appreciate your guidance on how to proceed with this deployment. Please let me know if you need any additional information about my use case.

Thank you very much for your time and support.

2

Hmm… for now, based on what can be confirmed from public information:

The short version is:

  1. The current Spaces hardware table does list two 8-GPU options that satisfy “more than 40 GB VRAM per card”: 8x L40S (48 GB per GPU) and 8x A100 (80 GB per GPU).
  2. A 4+4 split inside one Docker Space looks technically possible as a custom deployment, by running the two workloads as separate OS processes with disjoint GPU visibility, separate internal ports, and one public router. However, I could not find an HF document that presents this exact arrangement as a supported/recommended Spaces recipe.
  3. If the two model deployments are operationally independent, my default choice would be two separate 4-GPU deployments. Use one 8-GPU Space mainly when the two stages are tightly coupled, exchange substantial intermediate data, or you first want a single-node proof of concept.
  4. Current list pricing is public, but actual capacity, account access, reservations, region, quota, node topology, and support boundaries are things I would confirm directly with Hugging Face before committing to the design.

At the time of writing, the Spaces GPU hardware table shows:

Possible layout VRAM per GPU Published hourly price Operational boundary
1 × 8x L40S Space 48 GB $23.50/h One container/lifecycle
2 × 4x L40S Spaces 48 GB $16.60/h total Two independent containers/lifecycles
1 × 8x A100 Space 80 GB $20.00/h One container/lifecycle
2 × 4x A100 Spaces 80 GB $20.00/h total Two independent containers/lifecycles

Those are only the published list prices. The configurations are not otherwise identical: for example, the 8x L40S listing also includes substantially more CPU and system RAM than two 4x L40S listings combined. Separate Spaces also imply separate hosts and network communication between them, while a single 8-GPU Space may allow same-node communication. I would therefore not choose solely from the hourly-price column.

My default route

  • Two independent APIs, different release cycles, different traffic patterns, or a need to restart/scale either model separately: use two 4-GPU deployments. For a production API, also compare Inference Endpoints, since HF describes it as the managed production-serving product and manages container lifecycle, scaling, and health monitoring there.
  • Two stages of one request pipeline, especially if they exchange large tensors/latents/KV-like state frequently: benchmark one 8-GPU node with a 4+4 split before separating them over the network.
  • Still at PoC stage: one Docker Space can be a reasonable way to test whether the 4+4 layout works, but I would treat it as a measured baseline rather than assume it is the final architecture.

The most decision-changing information is not the GPU count itself, but the exact two model IDs/frameworks, whether this is inference or training, what crosses the boundary between the two workloads, dtype/quantization, context length or image/video dimensions, expected concurrency, latency/throughput targets, and why each side was estimated to require exactly four GPUs.

What a same-Space 4+4 deployment would actually look like

A Docker Space can have multiple internal ports, although the Space configuration exposes one application port externally. The Docker Spaces documentation explicitly says that a Space may have as many internal open ports as needed. In practice, the layout would be closer to this:

Public Space app_port
        |
        v
Thin router / reverse proxy
   |                     |
   v                     v
Service A              vLLM service
GPU 0-3                GPU 4-7
internal port A        internal port B

Conceptually, the child processes might be launched along these lines:

CUDA_VISIBLE_DEVICES=0,1,2,3 \
  python launch_generative_service.py --port 9000

CUDA_VISIBLE_DEVICES=4,5,6,7 \
  vllm serve MODEL_ID \
    --tensor-parallel-size 4 \
    --port 8000

This is illustrative rather than a complete production launcher. NVIDIA documents that CUDA_VISIBLE_DEVICES controls which devices a CUDA application can see and the order in which they are enumerated. Consequently, the second process will normally see its selected physical GPUs as its own logical cuda:0 through cuda:3. It is safer to record GPU UUIDs and the nvidia-smi process table instead of diagnosing placement from local ordinal numbers alone.

I would keep the two distributed engines in separate OS processes, rather than initialize two large distributed engines inside one Python process. A thin router can expose paths such as /generate/... and /v1/..., or perform whatever composition the pipeline requires. The vLLM maintainer answer in discussion #239 describes the general pattern of separate vLLM servers behind a custom load balancer for multiple instances/models.

The startup process also needs to be treated as part of the design:

  1. Start service A.
  2. Wait for a real readiness check, not merely an open process.
  3. Start service B.
  4. Wait for its readiness endpoint, such as vLLM’s health/model endpoint.
  5. Start or enable the public router only after both required backends are ready.
  6. Forward termination signals and fail clearly if a required child process exits.

There are public Spaces using a related, narrower pattern. For example, the Step-Audio-R1 startup script launches a TP=4 vLLM server and another application process in one Docker Space. The NuMarkdown startup script waits for the local vLLM service to pass a health check before starting the public Gradio application. These are useful process-layout examples, but neither proves that HF officially supports every two-engine 4+4 arrangement.

In addition to different HTTP ports, I would explicitly separate or inspect:

  • distributed rendezvous/master ports;
  • service-specific socket or IPC paths;
  • TorchInductor/Triton/vLLM compilation caches;
  • temporary directories;
  • log destinations;
  • health/readiness endpoints;
  • model download/cache paths where concurrent writers may be possible.

This is not because those conflicts must occur, but because there are reports showing that multiple same-node launches can fail for reasons unrelated to model fit. In vLLM issue #31199, sequential launches worked while concurrent launches failed during compilation, with a shared-cache race suspected. That issue was later closed as not planned/stale, so it is best used as a test-control idea, not as a prediction of the cause here. Likewise, a distributed process can fail when a rendezvous port is already occupied; issue #39015 is one current example of a 29500 address-in-use failure, although its exact stack and workload are not the same as this deployment.

Why one Space and two Spaces are different even after the GPUs are split

CUDA_VISIBLE_DEVICES separates device visibility. It does not turn one Space into two independently managed services.

Inside one Space, the two workloads still share or are coupled through:

  • the container start/stop/restart lifecycle;
  • the public ingress and routing layer;
  • the Space’s replica count;
  • sleep/pause and billing state;
  • CPU, host RAM, shared memory, disk and network namespace;
  • deployment version and rollback boundary;
  • top-level health behavior and logs;
  • failure handling by the parent process.

This matters particularly for horizontal scaling. The Spaces GPU documentation says that paid Spaces can use multiple replicas and that each replica is billed independently. A replica is a replica of the whole Space, so an 8-GPU Space containing both backends scales both together. It does not give you a way to add only vLLM capacity while keeping the other model at one copy.

Therefore, separate deployments are usually the cleaner default when:

  • either model can serve useful requests without the other;
  • their load peaks differ;
  • they have different update/restart schedules;
  • one side needs more replicas than the other;
  • a failure in one should not remove the other API;
  • the data passed between them is small enough for a network API.

A single 8-GPU Space becomes more attractive when:

  • nearly every request passes through both stages;
  • the stage boundary carries large or frequent intermediate state;
  • same-node shared memory or low-latency transfer is materially useful;
  • the two stages are versioned and released as one unit;
  • the whole system intentionally has one scaling and failure boundary.

There is no universal winner: it is mainly a component-boundary and data-flow decision, not a syntax question about GPU numbering.

Why I would verify the 4+4 sizing instead of treating it as fixed

Four GPUs per workload may be correct, but aggregate VRAM is not automatically pooled into one large virtual GPU. Each model must support an explicit sharding/parallelism strategy, and the useful strategy depends on the architecture and traffic target.

The current vLLM parallelism and scaling guide gives a useful sequence:

  • if the model fits on one GPU, distributed inference is probably unnecessary;
  • if it needs multiple GPUs on one node, tensor parallelism is a standard starting point;
  • after launch, inspect the reported GPU KV-cache capacity and estimated maximum concurrency;
  • if GPU count/model partitioning is uneven, pipeline parallelism may help;
  • for GPUs without NVLink, with L40S given as the example, vLLM specifically suggests considering pipeline parallelism to reduce communication overhead.

That means TP=4 should be tested rather than assumed, particularly for L40S. Depending on the exact models and SLO, useful alternatives could include:

  • 2 GPUs for one side and 6 for the other;
  • 3+5;
  • TP=2 with two replicas rather than one TP=4 replica;
  • pipeline parallelism for one workload;
  • expert parallelism for an MoE model;
  • quantization or a shorter max_model_len to free KV-cache capacity.

For the vLLM side, I would first verify that the exact architecture appears in the current supported-model documentation, then size it with the intended dtype/quantization, maximum sequence length, expected batch/concurrency, and acceptable latency. vLLM’s optimization guide also notes the trade-off: adding tensor/pipeline parallelism can provide memory, but can introduce synchronization or latency overhead.

A nearby but not identical case illustrates why placement and performance must be separated. In the currently open vLLM-Omni issue #4533, an 8x A100 system was split into an AR stage on GPUs 0-3 and a DiT stage on GPUs 4-7. The intended grouped path did run, but the reporter did not reproduce the expected end-to-end speedup at several concurrency levels. That does not imply the same result for your models; it only shows that a valid 4+4 placement is not by itself evidence that the end-to-end pipeline is faster.

A small validation matrix before committing to the architecture

I would use the 4+4 design as a controlled experiment and compare it with separate deployments. A compact test matrix would be:

Test What it isolates
Service A alone Its model fit, startup time, throughput and stability
vLLM alone Model/KV-cache fit and vLLM baseline
Start A, then B Shared-resource effects in the intended order
Start B, then A Startup-order dependency
Both resident, only A loaded Idle interference from B
Both resident, only B loaded Idle interference from A
Both under representative load CPU/RAM/shm/network and E2E interference
Terminate one child process Supervisor and failure-isolation behavior
Restart the whole Space Cold start, cache restoration and recovery time
Same node vs separate deployments Benefit/cost of the stage boundary

For each run, useful observations include:

  • nvidia-smi -L and the per-process GPU table;
  • nvidia-smi topo -m if available;
  • per-GPU memory and utilization;
  • CPU, host RAM and /dev/shm usage;
  • cold and warm startup time;
  • p50/p95 latency and throughput at the intended concurrency;
  • vLLM’s reported KV-cache capacity and estimated concurrency;
  • time and size of data transferred between the two stages;
  • recovery behavior after one backend exits.

If tensor-parallel communication is questionable, a small NCCL sanity test can distinguish a basic communication/topology problem from an application-level problem; NVIDIA’s nccl-tests repository is the usual reference. The result should still be interpreted as a communication baseline, not as an application benchmark.

A reasonable decision rule would be:

  • Choose two deployments if separate operation is stable and the network boundary is inexpensive relative to inference time.
  • Choose one 8-GPU deployment if same-node placement gives a measured end-to-end benefit large enough to justify coupled scaling and failure handling.
  • Revisit 4+4 if one stage is consistently idle, memory-bound rather than compute-bound, or unable to use four GPUs efficiently.
Pricing, storage and production-serving notes

The Spaces GPU page currently says billing is computed by the minute while a Space is Starting or Running. Paid upgraded Spaces run indefinitely by default unless a custom sleep time is configured, and each replica is billed independently.

For two large models, cold-start and storage behavior also matter. The Spaces disk documentation describes the normal Space disk as ephemeral: its content is lost when the Space restarts or stops. HF currently recommends attached Storage Buckets when data must persist beyond the Space lifecycle. Whether model caches should be persisted or simply re-downloaded depends on model size, update policy, startup target and storage cost, but the restart path should be measured rather than assumed.

For a production API, Inference Endpoints is worth comparing because HF manages starting, stopping, scaling, scale-to-zero and health/performance monitoring. It also supports vLLM and custom containers. However, I would not assume from the general product documentation that the exact 4-GPU/8-GPU type, region or topology you need is immediately available; that is another item for direct confirmation.

Items I would confirm directly with Hugging Face

Public documentation can answer the listed hardware specs and prices, but it cannot reliably answer current inventory or account-specific commercial/operational conditions. Before building around this, I would ask HF to confirm:

  1. Whether 8x A100 and/or 8x L40S is currently provisionable for your account and desired region.
  2. Whether access approval, enterprise terms, reservation, minimum duration or advance capacity planning is required.
  3. Whether all eight GPUs are on one host, and the exact GPU interconnect/topology for the offered instance.
  4. Whether HF considers two independent four-GPU distributed process groups inside one Docker Space a supported configuration, or merely something that may run as a custom container.
  5. Any practical limits relevant to two distributed engines, especially shared memory, IPC, startup timeout and health-check behavior.
  6. Whether a failure of either child process should terminate/restart the Space, and what health pattern HF recommends for a multi-service container.
  7. Whether separate 4-GPU Spaces or Inference Endpoints provide better capacity guarantees, observability, support or SLA for the intended production use.
  8. Whether the published price is the complete applicable compute price for the account/region, and whether there are quota or replica limits.

Those are the parts where an HF employee or support channel can add information that is not inferable from public code and documentation.

So, based only on the public information, I would not start from “one 8-GPU Space is required.” I would start from this default:

  • Independent services: two 4-GPU deployments.
  • Tightly coupled stages with a costly data boundary: benchmark one 8-GPU Docker Space with separate 4-GPU process groups and a thin router.
  • In either case: validate the exact model fit and parallelism first, because “four GPUs each” and “more than 40 GB per card” do not by themselves determine the efficient layout.

The current hardware table shows that the requested raw GPU capacity exists as a listed Spaces configuration; the remaining questions are mostly workload topology, operational boundaries, measured performance, and HF-specific availability/support conditions.