# Issues with Dataset Loading and Checkpoint Saving using FSDP with HuggingFace Trainer on SLURM Multi-Node Setup **URL:** \ **Category:** 🤗Accelerate\ **Created:** [April 7, 2025, 2:38pm UTC](https://discuss.huggingface.co/t/issues-with-dataset-loading-and-checkpoint-saving-using-fsdp-with-huggingface-trainer-on-slurm-multi-node-setup/149290 "2025-04-07T14:38:02Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Ryoo72](https://avatars.discourse-cdn.com/v4/letter/r/3da27b/32.png) [@Ryoo72](https://discuss.huggingface.co/u/Ryoo72)\ **Post date:** [April 7, 2025, 2:38pm UTC](https://discuss.huggingface.co/t/issues-with-dataset-loading-and-checkpoint-saving-using-fsdp-with-huggingface-trainer-on-slurm-multi-node-setup/149290/1 "2025-04-07T14:38:02Z")
Hello, I’m training a model using a SLURM multi-node setup with `accelerate`, HuggingFace `Trainer`, `FlashAttention2`, and `FSDP`. I’m working on a shared filesystem, which is mounted on all nodes so that every process can access the same data and output directories. I launch my training script without explicitly importing the `accelerate` Python package in code. Instead, I run it like this: ```auto accelerate launch --config_file my_config.yaml train_my_model.py ``` I have two questions: * * * **1. Dataset Loading** In my training script, I load the dataset like this: ```auto ds = load_dataset(ds_name, split='train') ``` Do I need to make sure this is executed only by the main process (e.g., using `main_process_first()` or a file lock) to prevent conflicts between processes in a distributed setting? Also, given this setup, is it safe to freely use `.map()` on the dataset inside the training script, or do I need to handle synchronization or process-specific behavior there as well? * * * **2. FSDP Checkpoint Saving** I’m using a shared filesystem, and during the checkpoint saving step, I get the following error. I suspect it’s related to concurrent writing in a multi-node setup with FSDP. ```bash 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2241, in train 1: [rank9]: return inner_training_loop( 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2612, in _inner_training_loop 1: [rank9]: self._maybe_log_save_evaluate( 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 3092, in _maybe_log_save_evaluate 1: [rank9]: self._save_checkpoint(model, trial) 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 3194, in _save_checkpoint 1: [rank9]: self._save_optimizer_and_scheduler(output_dir) 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 3311, in _save_optimizer_and_scheduler 1: [rank9]: save_fsdp_optimizer( 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/fsdp_utils.py", line 200, in save_fsdp_optimizer 1: [rank9]: dist_cp.save_state_dict( 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/checkpoint/state_dict_saver.py", line 41, in save_state_dict 1: [rank9]: return _save_state_dict( 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/checkpoint/state_dict_saver.py", line 271, in _save_state_dict 1: [rank9]: central_plan = distW.reduce_scatter("plan", local_step, global_step) 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/checkpoint/utils.py", line 167, in reduce_scatter 1: [rank9]: all_data = self.gather_object(local_data) 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/checkpoint/utils.py", line 106, in gather_object 1: [rank9]: dist.gather_object( 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 75, in wrapper 1: [rank9]: return func(*args, **kwargs) 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2529, in gather_object 1: [rank9]: gather( 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 75, in wrapper 1: [rank9]: return func(*args, **kwargs) 1: [rank9]: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 3105, in gather 1: [rank9]: work = default_pg.gather(output_tensors, input_tensors, opts) 1: [rank9]: RuntimeError: NCCL Error 2: unhandled system error (run with NCCL_DEBUG=INFO for details) ``` How should I configure `fsdp_config` properly in this case to avoid such errors during saving? Thank you! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 7, 2025, 2:54pm UTC](https://discuss.huggingface.co/t/issues-with-dataset-loading-and-checkpoint-saving-using-fsdp-with-huggingface-trainer-on-slurm-multi-node-setup/149290/2 "2025-04-07T14:54:18Z")
It seems to be an unsolved issue, but there may be a way to avoid it (such as downgrading the accelerate library). > > > \### System Info > > \`\`\`Shell > output of \`accelerate env\`: (note as shown below th…is prints the DEFAULT accelerate config and not the exact config being used for this job) > \- \`Accelerate\` version: 0.27.2 > \- Platform: Linux-5.15.0-1037-aws-x86\_64-with-glibc2.17 > \- Python version: 3.8.18 > \- Numpy version: 1.24.4 > \- PyTorch version (GPU?): 2.2.0+cu121 (False) > \- PyTorch XPU available: False > \- PyTorch NPU available: False > \- System RAM: 123.82 GB > \- \`Accelerate\` default config: > - compute\_environment: LOCAL\_MACHINE > - distributed\_type: FSDP > - mixed\_precision: bf16 > - use\_cpu: False > - debug: False > - num\_processes: 8 > - machine\_rank: 0 > - num\_machines: 2 > - main\_process\_ip: > - main\_process\_port: 1234 > - rdzv\_backend: static > - same\_network: True > - main\_training\_function: main > - fsdp\_config: {'fsdp\_auto\_wrap\_policy': 'TRANSFORMER\_BASED\_WRAP', 'fsdp\_backward\_prefetch': 'BACKWARD\_PRE', 'fsdp\_cpu\_ram\_efficient\_loading': True, 'fsdp\_forward\_prefetch': False, 'fsdp\_offload\_params': False, 'fsdp\_sharding\_strategy': 'FULL\_SHARD', 'fsdp\_state\_dict\_type': 'FULL\_STATE\_DICT', 'fsdp\_sync\_module\_states': True, 'fsdp\_transformer\_layer\_cls\_to\_wrap': 'LlamaDecoderLayer', 'fsdp\_use\_orig\_params': False} > - downcast\_bf16: no > - tpu\_use\_cluster: False > - tpu\_use\_sudo: False > - tpu\_env: \[\] > \`\`\` > > > \### Information > > \- \[\] The official example scripts > \- \[X\] My own modified scripts > > \### Tasks > > \- \[\] One of the scripts in the examples/ folder of Accelerate or an officially supported \`no\_trainer\` script in the \`examples\` folder of the \`transformers\` repo (such as \`run\_no\_trainer\_glue.py\`) > \- \[X\] My own task or dataset (give details below) > > \### Reproduction > > I'm attempting to train a model with multi-node training, using SLURM scheduler. I am launching the job in 2 nodes with 8 GPUs each. My training script runs fine in a single-node environment with FSDP, and it \*starts\* fine in the multi-node setting -- until there is actual communication required between the nodes. > > However, when the script gets to the parts that actually initialize multi-node training, it seems the processes are having issues communicating across nodes. I can see the logging output from all 16 processes, the data is loaded, etc. However, the script fails at \`accelerator.prepare()\`. > > Specifically I see the stack trace containing these lines (complete stack trace is below): > > \`\`\` > > torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1691, unhandled system error (run with NCCL\_DEBUG=INFO for details), NCCL version 2.19.3 > ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. > > \`\`\` > > Note that it is possible I have misconfigured the accelerate config, or the SLURM settings (tasks/node counts etc), but based on the example \[here\](https://gist.github.com/pacman100/1cb1f17b2f1b3139a63b764263e70b25) with corresponding FSDP config \[here\](https://github.com/pacman100/DHS-LLM-Workshop/blob/6093a2320543c2ac903a1fbb9b034ea714db43c9/personal\_copilot/training/configs/fsdp\_config.yaml#L4) this seems to be set up correctly to me. > > Any thoughts would be appreciated, I've tried lots of different configurations and tinkering with the environment to make sure the versions of pytorch/NCCL/accelerate are all compatible as well. > > > \## Contents of fsdp\_config\_base.yaml I am using: > > \`\`\` > compute\_environment: LOCAL\_MACHINE > distributed\_type: FSDP > downcast\_bf16: 'no' > fsdp\_config: > fsdp\_auto\_wrap\_policy: TRANSFORMER\_BASED\_WRAP > fsdp\_backward\_prefetch\_policy: BACKWARD\_PRE > fsdp\_forward\_prefetch: false > fsdp\_offload\_params: false > fsdp\_sharding\_strategy: 1 > fsdp\_state\_dict\_type: FULL\_STATE\_DICT > fsdp\_sync\_module\_states: true > fsdp\_transformer\_layer\_cls\_to\_wrap: LlamaDecoderLayer > fsdp\_use\_orig\_params: true > machine\_rank: 0 > main\_training\_function: main > num\_machines: 1 > num\_processes: 8 > mixed\_precision: bf16 > rdzv\_backend: c10d > same\_network: true > tpu\_env: \[\] > tpu\_use\_cluster: false > tpu\_use\_sudo: false > use\_cpu: false > \`\`\` > > \## Relevant chunks of the sbatch script I am launching the job with: > > \`\`\` > \#!/bin/bash > > \#SBATCH --nodes=2 > \#SBATCH --ntasks-per-node=1 > \#SBATCH --cpus-per-task=10 > \#SBATCH --partition=a40x > \#SBATCH --gpus-per-node=8 > \#SBATCH --exclusive > \#SBATCH --time=04-00:00:00 > \#SBATCH --account=nextgends > \#SBATCH --chdir=/admin/home-jpgard/rtfm > \#SBATCH --output=/admin/home-jpgard/rtfm/slurm-out/%j.out > \#SBATCH --err=/admin/home-jpgard/rtfm/slurm-out/%j.err > \#SBATCH --exclude=ip-10-0-201-106,ip-10-0-202-154 > > \################# code block adapted from https://gist.github.com/pacman100/1cb1f17b2f1b3139a63b764263e70b25 > set -x -e > > \# force crashing on nccl issues like hanging broadcast > export NCCL\_ASYNC\_ERROR\_HANDLING=1 > export NCCL\_DEBUG=INFO > > GPUS\_PER\_NODE=8 > NNODES=$SLURM\_NNODES > NUM\_PROCESSES=$(expr $NNODES \\\* $GPUS\_PER\_NODE) > > \# A function to parse slurm's node notation when it uses 'bracketed' values for SLURM\_JOB\_NODELIST > \# Function to parse and expand node list from SLURM\_JOB\_NODELIST > expand\_nodes() { > # The input is something like "ip-10-0-231-\[1,86\]" > local nodelist=$1 > > # Replace '\[' and '\]' with space and split the string > local base=$(echo $nodelist | sed -E 's/\\\[(\[0-9\]+),(\[0-9\]+)\\\]/ \\1 \\2 /') > > # Read into array > read -a parts \<\<\< "$base" > > # Check if we have three parts: prefix, start, end > if \[${#parts\[@\]} -eq 3 \]; then > local prefix=${parts\[0\]} > local start=${parts\[1\]} > local end=${parts\[2\]} > > # Generate sequence > for i in $(seq $start $end); do > echo "${prefix}${i}" > return # Return after first IP to mimic head node behavior > done > else > # If the format does not include a range, just echo the input > echo $nodelist > fi > } > > \# Extract the first node name from SLURM\_JOB\_NODELIST > \# This assumes the format "node-list: ip-10-0-209-157,ip-10-0-231-1" and extracts the first node name > echo "SLURM\_JOB\_NODELIST is $SLURM\_JOB\_NODELIST" > node\_name=$(echo $SLURM\_JOB\_NODELIST | sed 's/node-list: //' | cut -d, -f1) > > \# Now, resolve this node name to an IP address > \# Using getent ahosts (You can also use nslookup if getent does not work as expected) > MASTER\_ADDR=$(getent ahosts $node\_name | head -n 1 | awk '{print $1}') > > \# Check if we got an IP > if \[! -z "$MASTER\_ADDR" \]; then > echo "Head node IP: $MASTER\_ADDR" > else > echo "Failed to resolve head node IP address" > # Extract the first node name from SLURM\_JOB\_NODELIST and expand if needed > node\_name=$(expand\_nodes $SLURM\_JOB\_NODELIST) > # Now, resolve this node name to an IP address using getent ahosts > MASTER\_ADDR=$(getent ahosts $node\_name | head -n 1 | awk '{print $1}') > echo "Head node IP after parsing: $MASTER\_ADDR" > fi > > MASTER\_PORT=6999 > > \# OTHER LAUNCHERS CAN BE USED HERE > export LAUNCHER="/admin/home-jpgard/miniconda3/envs/rtfm/bin/accelerate launch \\ > --config\_file /admin/home-jpgard/rtfm/fsdp\_config\_base.yaml \\ > --num\_processes $NUM\_PROCESSES \\ > --main\_process\_ip $MASTER\_ADDR \\ > --num\_machines $NNODES \\ > --main\_process\_port $MASTER\_PORT \\ > --machine\_rank \\$SLURM\_PROCID \\ > " > > echo "SLURM\_JOB\_ID is ${SLURM\_JOB\_ID}" > > echo 'activating conda environment' > source /admin/home-jpgard/.bashrc > source /admin/home-jpgard/miniconda3/etc/profile.d/conda.sh > which conda > conda activate rtfm > which python > > export PROGRAM="\\ > scripts/train.py \\ > --more-args-here > --bf16 True \\ > --use\_amp \\ > " > > export CMD="$LAUNCHER $PROGRAM" > echo "about to run ${CMD}" > /opt/slurm/bin/srun --jobid $SLURM\_JOBID /usr/bin/bash -c "$CMD" > \`\`\` > > \## Full stack trace: > > \`\`\` > Traceback (most recent call last): > File "scripts/train.py", line 582, in \ > main( > File "scripts/train.py", line 206, in main > model = accelerator.prepare(model) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/accelerator.py", line 1228, in prepare > result = tuple( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/accelerator.py", line 1229, in \ > self.\_prepare\_one(obj, first\_pass=True, device\_placement=d) for obj, d in zip(args, device\_placement) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/accelerator.py", line 1105, in \_prepare\_one > return self.prepare\_model(obj, device\_placement=device\_placement) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/accelerator.py", line 1387, in prepare\_model > model = FSDP(model, \*\*kwargs) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/fully\_sharded\_data\_parallel.py", line 477, in \_\_init\_\_ > \_auto\_wrap( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/\_wrap\_utils.py", line 101, in \_auto\_wrap > \_recursive\_wrap(\*\*recursive\_wrap\_kwargs, \*\*root\_kwargs) # type: ignore\[arg-type\] > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 543, in \_recursive\_wrap > wrapped\_child, num\_wrapped\_params = \_recursive\_wrap( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 543, in \_recursive\_wrap > wrapped\_child, num\_wrapped\_params = \_recursive\_wrap( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 543, in \_recursive\_wrap > wrapped\_child, num\_wrapped\_params = \_recursive\_wrap( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 561, in \_recursive\_wrap > return \_wrap(module, wrapper\_cls, \*\*kwargs), nonwrapped\_numel > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 490, in \_wrap > return wrapper\_cls(module, \*\*kwargs) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/fully\_sharded\_data\_parallel.py", line 503, in \_\_init\_\_ > \_init\_param\_handle\_from\_module( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/\_init\_utils.py", line 587, in \_init\_param\_handle\_from\_module > \_sync\_module\_params\_and\_buffers( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/\_init\_utils.py", line 1068, in \_sync\_module\_params\_and\_buffers > \_sync\_params\_and\_buffers( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/utils.py", line 303, in \_sync\_params\_and\_buffers > dist.\_broadcast\_coalesced( > torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1691, unhandled system error (run with NCCL\_DEBUG=INFO for details), NCCL version 2.19.3 > ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. > Last error: > socketStartConnect: Connect to fe80::a849:bbff:fe73:19bd%veth8299cd8\<54785\> failed : Software caused connection abort > Traceback (most recent call last): > File "scripts/train.py", line 582, in \ > main( > File "scripts/train.py", line 206, in main > model = accelerator.prepare(model) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/accelerator.py", line 1228, in prepare > result = tuple( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/accelerator.py", line 1229, in \ > self.\_prepare\_one(obj, first\_pass=True, device\_placement=d) for obj, d in zip(args, device\_placement) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/accelerator.py", line 1105, in \_prepare\_one > return self.prepare\_model(obj, device\_placement=device\_placement) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/accelerator.py", line 1387, in prepare\_model > model = FSDP(model, \*\*kwargs) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/fully\_sharded\_data\_parallel.py", line 477, in \_\_init\_\_ > \_auto\_wrap( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/\_wrap\_utils.py", line 101, in \_auto\_wrap > \_recursive\_wrap(\*\*recursive\_wrap\_kwargs, \*\*root\_kwargs) # type: ignore\[arg-type\] > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 543, in \_recursive\_wrap > wrapped\_child, num\_wrapped\_params = \_recursive\_wrap( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 543, in \_recursive\_wrap > wrapped\_child, num\_wrapped\_params = \_recursive\_wrap( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 543, in \_recursive\_wrap > wrapped\_child, num\_wrapped\_params = \_recursive\_wrap( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 561, in \_recursive\_wrap > return \_wrap(module, wrapper\_cls, \*\*kwargs), nonwrapped\_numel > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/wrap.py", line 490, in \_wrap > return wrapper\_cls(module, \*\*kwargs) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/fully\_sharded\_data\_parallel.py", line 503, in \_\_init\_\_ > \_init\_param\_handle\_from\_module( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/\_init\_utils.py", line 587, in \_init\_param\_handle\_from\_module > \_sync\_module\_params\_and\_buffers( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/fsdp/\_init\_utils.py", line 1068, in \_sync\_module\_params\_and\_buffers > \_sync\_params\_and\_buffers( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/utils.py", line 303, in \_sync\_params\_and\_buffers > dist.\_broadcast\_coalesced( > torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1691, unhandled system error (run with NCCL\_DEBUG=INFO for details), NCCL version 2.19.3 > ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. > Last error: > > /admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/utils/launch.py:192: FutureWarning: \`fsdp\_backward\_prefetch\_policy\` is deprecated and will be removed in version 0.27.0 of 🤗 Accelerate. Use \`fsdp\_backward\_prefetch\` instead > warnings.warn( > \[2024-02-16 02:42:00,874\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 4025500 closing signal SIGTERM > \[2024-02-16 02:42:00,875\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 4025502 closing signal SIGTERM > \[2024-02-16 02:42:00,875\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 4025503 closing signal SIGTERM > \[2024-02-16 02:42:00,875\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 4025504 closing signal SIGTERM > \[2024-02-16 02:42:00,876\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 4025505 closing signal SIGTERM > \[2024-02-16 02:42:00,876\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 4025506 closing signal SIGTERM > \[2024-02-16 02:42:00,876\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 4025507 closing signal SIGTERM > /admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/utils/launch.py:192: FutureWarning: \`fsdp\_backward\_prefetch\_policy\` is deprecated and will be removed in version 0.27.0 of 🤗 Accelerate. Use \`fsdp\_backward\_prefetch\` instead > warnings.warn( > \[2024-02-16 02:42:00,885\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 1267868 closing signal SIGTERM > \[2024-02-16 02:42:00,885\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 1267870 closing signal SIGTERM > \[2024-02-16 02:42:00,886\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 1267872 closing signal SIGTERM > \[2024-02-16 02:42:00,886\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 1267873 closing signal SIGTERM > \[2024-02-16 02:42:00,886\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 1267874 closing signal SIGTERM > \[2024-02-16 02:42:00,886\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 1267875 closing signal SIGTERM > \[2024-02-16 02:42:00,886\] torch.distributed.elastic.multiprocessing.api: \[WARNING\] Sending process 1267876 closing signal SIGTERM > \[2024-02-16 02:42:03,298\] torch.distributed.elastic.multiprocessing.api: \[ERROR\] failed (exitcode: 1) local\_rank: 1 (pid: 1267869) of binary: /admin/home-jpgard/miniconda3/envs/rtfm/bin/python > Traceback (most recent call last): > File "/admin/home-jpgard/miniconda3/envs/rtfm/bin/accelerate", line 8, in \ > sys.exit(main()) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/commands/accelerate\_cli.py", line 47, in main > args.func(args) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/commands/launch.py", line 1010, in launch\_command > multi\_gpu\_launcher(args) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/accelerate/commands/launch.py", line 672, in multi\_gpu\_launcher > distrib\_run.run(args) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/run.py", line 803, in run > elastic\_launch( > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 135, in \_\_call\_\_ > return launch\_agent(self.\_config, self.\_entrypoint, list(args)) > File "/admin/home-jpgard/miniconda3/envs/rtfm/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 268, in launch\_agent > raise ChildFailedError( > torch.distributed.elastic.multiprocessing.errors.ChildFailedError: > ============================================================ > scripts/train.py FAILED > \------------------------------------------------------------ > Failures: > \ > \------------------------------------------------------------ > Root Cause (first observed failure): > \[0\]: > time : 2024-02-16\_02:42:00 > host : ip-10-0-209-157.us-west-2.compute.internal > rank : 9 (local\_rank: 1) > exitcode : 1 (pid: 1267869) > error\_file: \ > traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html > \`\`\` > > \### Expected behavior > > I expect training to work in the distributed setting just as it does in the single-node setting. > > > The error encountered is as follows: > > \`\`\` > File "torch/distributed/distribut…ed\_c10d.py", line 1451, in wrapper > return func(\*args, \*\*kwargs) > File "torch/distributed/distributed\_c10d.py", line 2053, in all\_gather\_object > all\_gather(object\_size\_list, local\_size, group=group) > File "torch/distributed/distributed\_c10d.py", line 1451, in wrapper > return func(\*args, \*\*kwargs) > File "torch/distributed/distributed\_c10d.py", line 2448, in all\_gather > work = default\_pg.allgather(\[tensor\_list\], \[tensor\]) > torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1296, internal error - please report this issue to the NCCL developers, NCCL version 2.18.1 > \`\`\`\` > > PyTorch API being used is: https://github.com/pytorch/pytorch/blob/main/torch/distributed/distributed\_c10d.py#L2322. > > NCCL\_DEBUG output is as follows: > > \`\`\` > NCCL version 2.18.1+cuda12.0 > node:2134736:2285781 \[0\] include/alloc.h:178 NCCL WARN Cuda failure 'out of memory' > node:2134736:2285781 \[0\] include/alloc.h:185 NCCL WARN Failed to CUDA calloc 6291456 bytes > node:2134736:2285781 \[0\] proxy.cc:1518 NCCL WARN \[Proxy Service 93\] Failed to execute operation Setup from rank 93, retcode 1 > node:2134736:2285759 \[0\] misc/socket.cc:49 NCCL WARN socketProgress: Connection closed by remote peer node.cm.cluster\<40031\> > node:2134736:2285759 \[0\] proxy.cc:1143 NCCL WARN Socket recv failed while polling for opId=0x152b48786ea0 > \`\`\` > > The NCCL log reports a CUDA OOM, but the number of bytes that were attempted to be allocated seems to be just 6MB which shouldn't cause an OOM. > > > \### System Info > > \`\`\`Shell > accelerate 1.0.1 > torch … 2.4.0 > python 3.10.14 > transformers 4.45.2 > > FullyShardedDataParallelPlugin(sharding\_strategy=\, backward\_prefetch=None, mixed\_precision\_policy=MixedPrecision(param\_dtype=torch.bfloat16, reduce\_dtype=torch.bfloat16, buffer\_dtype=torch.bfloat16, keep\_low\_precision\_grads=False, cast\_forward\_inputs=False, cast\_root\_forward\_inputs=True, \_module\_classes\_to\_ignore=(\,)), auto\_wrap\_policy=\, cpu\_offload=CPUOffload(offload\_params=False), ignored\_modules=None, state\_dict\_type=\, state\_dict\_config=ShardedStateDictConfig(offload\_to\_cpu=True, \_use\_dtensor=False), optim\_state\_dict\_config=ShardedOptimStateDictConfig(offload\_to\_cpu=True, \_use\_dtensor=False), limit\_all\_gathers=True, use\_orig\_params=True, param\_init\_fn=\.\ at 0x155230e00ee0\>, sync\_module\_states=True, forward\_prefetch=False, activation\_checkpointing=True, cpu\_ram\_efficient\_loading=True, transformer\_cls\_names\_to\_wrap=\['Qwen2DecoderLayer'\], min\_num\_params=None) > \`\`\` > > > \### Information > > \- \[\] The official example scripts > \- \[X\] My own modified scripts > > \### Tasks > > \- \[\] One of the scripts in the examples/ folder of Accelerate or an officially supported \`no\_trainer\` script in the \`examples\` folder of the \`transformers\` repo (such as \`run\_no\_trainer\_glue.py\`) > \- \[X\] My own task or dataset (give details below) > > \### Reproduction > > when I use FSDP to train a model on 16-32 node cluster. Sometime it hit this error "NCCL WARN Cuda failure 2 'out of memory'" when save the model checkpoint. resulting in an empty folder. > sometimes it works fine. Anyone had similar experience before? This really puzzles me as i already use offload\_to\_cpu = True for SHARDED\_STATE\_DICT > > \`\`\` > > Traceback (most recent call last): > \[rank150\]: File "/finetune\_orm\_bt\_listwise\_fsdp.py", line 353, in \ > \[rank150\]: trainer.train(resume\_from\_checkpoint=script\_args.resume\_from\_checkpoint) > \[rank150\]: File "lib/python3.10/site-packages/transformers/trainer.py", line 2052, in train > \[rank150\]: return inner\_training\_loop( > \[rank150\]: File "lib/python3.10/site-packages/transformers/trainer.py", line 2467, in \_inner\_training\_loop > \[rank150\]: self.\_maybe\_log\_save\_evaluate(tr\_loss, grad\_norm, model, trial, epoch, ignore\_keys\_for\_eval) > \[rank150\]: File "lib/python3.10/site-packages/transformers/trainer.py", line 2918, in \_maybe\_log\_save\_evaluate > \[rank150\]: self.\_save\_checkpoint(model, trial, metrics=metrics) > \[rank150\]: File "lib/python3.10/site-packages/transformers/trainer.py", line 3012, in \_save\_checkpoint > \[rank150\]: self.\_save\_optimizer\_and\_scheduler(output\_dir) > \[rank150\]: File "lib/python3.10/site-packages/transformers/trainer.py", line 3147, in \_save\_optimizer\_and\_scheduler > \[rank150\]: save\_fsdp\_model( > \[rank150\]: File "lib/python3.10/site-packages/accelerate/utils/fsdp\_utils.py", line 107, in save\_fsdp\_model > \[rank150\]: dist\_cp.save\_state\_dict( > \[rank150\]: File "lib/python3.10/site-packages/typing\_extensions.py", line 2853, in wrapper > \[rank150\]: return arg(\*args, \*\*kwargs) > \[rank150\]: File "lib/python3.10/site-packages/torch/distributed/checkpoint/state\_dict\_saver.py", line 47, in save\_state\_dict > \[rank150\]: return \_save\_state\_dict( > \[rank150\]: File "lib/python3.10/site-packages/torch/distributed/checkpoint/state\_dict\_saver.py", line 316, in \_save\_state\_dict > \[rank150\]: central\_plan: SavePlan = distW.reduce\_scatter("plan", local\_step, global\_step) > \[rank150\]: File "lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py", line 190, in reduce\_scatter > \[rank150\]: result = self.scatter\_object(all\_results) > \[rank150\]: File "lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py", line 135, in scatter\_object > \[rank150\]: dist.scatter\_object\_list( > \[rank150\]: File "lib/python3.10/site-packages/torch/distributed/c10d\_logger.py", line 79, in wrapper > \[rank150\]: return func(\*args, \*\*kwargs) > \[rank150\]: File "lib/python3.10/site-packages/torch/distributed/distributed\_c10d.py", line 3019, in scatter\_object\_list > \[rank150\]: scatter( > \[rank150\]: File "lib/python3.10/site-packages/torch/distributed/c10d\_logger.py", line 79, in wrapper > \[rank150\]: return func(\*args, \*\*kwargs) > \[rank150\]: File \`"lib/python3.10/site-packages/torch/distributed/distributed\_c10d.py",\` line 3480, in scatter > \[rank150\]: work = default\_pg.scatter(output\_tensors, input\_tensors, opts) > \[rank150\]: RuntimeError: NCCL Error 1: unhandled cuda error (run with NCCL\_DEBUG=INFO for details) > \`\`\` > \` > FullyShardedDataParallelPlugin(sharding\_strategy=\, backward\_prefetch=None, mixed\_precision\_policy=MixedPrecision(param\_dtype=torch.bfloat16, reduce\_dtype=torch.bfloat16, buffer\_dtype=torch.bfloat16, keep\_low\_precision\_grads=False, cast\_forward\_inputs=False, cast\_root\_forward\_inputs=True, \_module\_classes\_to\_ignore=(\,)), auto\_wrap\_policy=\, cpu\_offload=CPUOffload(offload\_params=False), ignored\_modules=None, state\_dict\_type=\, state\_dict\_config=ShardedStateDictConfig(offload\_to\_cpu=True, \_use\_dtensor=False), optim\_state\_dict\_config=ShardedOptimStateDictConfig(offload\_to\_cpu=True, \_use\_dtensor=False), limit\_all\_gathers=True, use\_orig\_params=True, param\_init\_fn=\.\ at 0x155230e00ee0\>, sync\_module\_states=True, forward\_prefetch=False, activation\_checkpointing=True, cpu\_ram\_efficient\_loading=True, transformer\_cls\_names\_to\_wrap=\['Qwen2DecoderLayer'\], min\_num\_params=None)\` > \### Expected behavior > > save the ckpt fine