The following have been reloaded with a version change: 1) GCCcore/.14.3.0 => GCCcore/14.3.0 Lmod is automatically replacing "GCC/14.3.0" with "nvidia-compilers/25.9-CUDA-13". Deactivating conda environment: /e/scratch/jureap59/feuer1/miniforge3/envs/otagent Activating RL environment: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl Python executable: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python path check: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python [ray] RAY_TMPDIR=/tmp/ray/ray_388832 [triton_cache] Triton cache: /tmp/triton_cache_feuer1_388832 [triton_cache] TorchInductor cache: /tmp/torchinductor_cache_feuer1_388832 [proxy] ✓ Found proxychains binary at /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 [proxy] Setting up SSH tunnel to jpbl-s01-01 [proxy] SSH key: /e/home/jusers/feuer1/jupiter/.ssh/authorized_keys/id_ed25519_jsc [proxy] Tunnel port: 7003 [proxy] Node IP: 10.128.33.33 (workers will connect here) [proxy] ✓ SSH tunnel started successfully [proxy] ✓ Generated proxychains config at /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_388832.conf [proxy] - Internal traffic (10.x.x.x, 172.x.x.x, 169.254.x.x) → DIRECT [proxy] - External traffic (internet) → PROXY via tunnel [proxy] ✓ Daytona timeout settings configured [proxy] Testing proxy connectivity... [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_388832.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [proxy] ✓ Proxy connectivity test passed (huggingface.co reachable via wrapped binary) [proxy] ⚠ Tunnel not accessible at 10.128.33.33:7003 (workers may fail) [proxy] ✓ Proxy setup complete (using wrapped binary for Ray workers) [container_runtime] Using cloud backend: daytona (no local container setup) === Universal RL Training Runner === Config: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/configs/a2-rl-stack_jest_v2_rl_config.json Working directory: /e/scratch/jureap59/feuer1/OpenThoughts-Agent Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Python version: Python 3.12.12 Proxy: DISABLED (direct internet or not configured) ======================================== === RLJobRunner: a2-rl-stack_jest_v2 === [wandb_utils] Fixing permissions on: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/wandb [wandb_utils] WandB directory ready: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/wandb HF_TOKEN=****pDbg HF_HUB_CACHE=/e/data1/datasets/playground/ot-baf/hf_hub SUPABASE_URL=https://rpzmyuapoqilpghynmza.s... (direct Supabase config) Environment configured: TENSOR_PARALLEL_SIZE=2 NUM_INFERENCE_ENGINES=32 POLICY_NUM_NODES=16 WANDB_DIR=/e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/wandb Starting Ray cluster with 16 nodes, 4 GPUs/node Cleaning up existing Ray instances... === Starting Ray Cluster === Nodes: 16 GPUs per node: 4 CPUs per node: 288 Head node: jpbo-047-01 (10.128.33.33) Ray port: 6379 ============================ Starting Ray head on jpbo-047-01 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_head_jpbo-047-01.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.33 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-01 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --head --node-ip-address=10.128.33.33 --port=6379 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray head on jpbo-047-01 Starting Ray worker on jpbo-047-02 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-02.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.34 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-02 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.34 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 1 on jpbo-047-02 Starting Ray worker on jpbo-047-03 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-03.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.35 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-03 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.35 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 2 on jpbo-047-03 Starting Ray worker on jpbo-047-04 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-04.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.36 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-04 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.36 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 3 on jpbo-047-04 Starting Ray worker on jpbo-047-05 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-05.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.37 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-05 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.37 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 4 on jpbo-047-05 Starting Ray worker on jpbo-047-06 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-06.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.38 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-06 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.38 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 5 on jpbo-047-06 Starting Ray worker on jpbo-047-07 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-07.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.39 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-07 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.39 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 6 on jpbo-047-07 Starting Ray worker on jpbo-047-08 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-08.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.40 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-08 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.40 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 7 on jpbo-047-08 Starting Ray worker on jpbo-047-09 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-09.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.41 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-09 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.41 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 8 on jpbo-047-09 Starting Ray worker on jpbo-047-10 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-10.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.42 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-10 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.42 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 9 on jpbo-047-10 Starting Ray worker on jpbo-047-11 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-11.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.43 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-11 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.43 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 10 on jpbo-047-11 Starting Ray worker on jpbo-047-12 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-12.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.44 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-12 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.44 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 11 on jpbo-047-12 Starting Ray worker on jpbo-047-13 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-13.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.45 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-13 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.45 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 12 on jpbo-047-13 Starting Ray worker on jpbo-047-14 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-14.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.46 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-14 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.46 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 13 on jpbo-047-14 Starting Ray worker on jpbo-047-15 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-15.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.47 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-15 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.47 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 14 on jpbo-047-15 Starting Ray worker on jpbo-047-16 (logging to /e/scratch/jureap59/feuer1/OpenThoughts-Agent/experiments/logs/ray_worker_jpbo-047-16.log)... Command: srun --export=ALL,VLLM_HOST_IP=10.128.33.48 --nodes=1 --ntasks=1 --gres=gpu:4 --gpu-bind=none --overlap --cpu-bind=none -w jpbo-047-16 bash -c /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f "$PROXYCHAINS_CONF_FILE" ray start --address=10.128.33.33:6379 --node-ip-address=10.128.33.48 --num-gpus=4 --num-cpus=288 --block --object-store-memory=42949672960 Started Ray worker 15 on jpbo-047-16 Waiting for cluster (64 GPUs, 16 nodes)... Connecting to Ray at 10.128.33.33:6379 (expecting 16 nodes, 64.0 GPUs) Ray connection established, polling for resources... [Ray wait] nodes=16/16 GPUs=64.0/64.0 resources={'GPU': 64.0, 'node:10.128.33.47': 1.0, 'object_store_memory': 687194767360.0, 'accelerator_type:GH200': 16.0, 'memory': 11834749616128.0, 'CPU': 4608.0, 'node:10.128.33.45': 1.0, 'node:10.128.33.33': 1.0, 'node:__internal_head__': 1.0, 'node:10.128.33.34': 1.0, 'node:10.128.33.37': 1.0, 'node:10.128.33.46': 1.0, 'node:10.128.33.38': 1.0, 'node:10.128.33.35': 1.0, 'node:10.128.33.41': 1.0, 'node:10.128.33.48': 1.0, 'node:10.128.33.36': 1.0, 'node:10.128.33.39': 1.0, 'node:10.128.33.42': 1.0, 'node:10.128.33.43': 1.0, 'node:10.128.33.40': 1.0, 'node:10.128.33.44': 1.0} ✓ Ray cluster ready === Ray Cluster Ready === Address: 10.128.33.33:6379 Total GPUs: 64 ========================= Ray cluster ready at 10.128.33.33:6379 Total GPUs available: 64 [RLJobRunner] Pinggy check: url=False, token=False, needs_tunnel=False (agent=terminus-2, env=daytona) [RLJobRunner] No Pinggy tunnel needed, using local vLLM Running SkyRL: Python: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python Entrypoint: examples.terminal_bench.entrypoints.main_tbench Args: 119 Hydra arguments Working dir: /e/scratch/jureap59/feuer1/OpenThoughts-Agent/SkyRL/skyrl-train Using proxychains binary: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 Executing command with srun: /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/bin/proxychains4 -f $PROXYCHAINS_CONF_FILE /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/bin/python -m examples.terminal_bench.entrypoints.main_tbench +terminal_bench_config=terminal_bench trainer.strategy=fsdp2 trainer.algorithm.advantage_estimator=rloo_n trainer.algorithm.use_kl_loss=false trainer.algorithm.kl_loss_coef=0.0 trainer.algorithm.eps_clip_low=0.2 trainer.algorithm.eps_clip_high=0.2 trainer.algorithm.loss_reduction=token_mean trainer.epochs=2 trainer.max_steps=60 trainer.update_epochs_per_batch=1 trainer.train_batch_size=64 trainer.policy_mini_batch_size=64 trainer.eval_batch_size=64 trainer.micro_forward_batch_size_per_gpu=1 trainer.micro_train_batch_size_per_gpu=1 trainer.max_prompt_length=999999 trainer.eval_interval=999999 trainer.eval_before_train=false trainer.ckpt_interval=1 trainer.resume_mode=latest trainer.hf_save_interval=5 ++trainer.hf_hub_repo_id=laion/a2-rl-stack_jest_v2 ++trainer.hf_hub_private=false ++trainer.hf_hub_revision=main ++trainer.enable_db_registration=true trainer.project_name=OpenThoughts-Agent trainer.log_level=INFO trainer.tracker_commit_each_step=true trainer.logger=console trainer.run_name=a2-rl-stack_jest_v2 trainer.ckpt_path=/e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints trainer.export_path=/e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/exports trainer.policy.optimizer_config.lr=9e-6 trainer.policy.optimizer_config.weight_decay=0.0 trainer.policy.optimizer_config.adam_betas=[0.9,0.999] trainer.policy.optimizer_config.max_grad_norm=10.0 trainer.policy.fsdp_config.cpu_offload=true trainer.policy.fsdp_config.reshard_after_forward=true trainer.policy.fsdp_config.fsdp_size=4 trainer.policy.model.path=/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137 trainer.ref.fsdp_config.cpu_offload=true trainer.ref.fsdp_config.reshard_after_forward=true trainer.ref.fsdp_config.fsdp_size=4 trainer.placement.colocate_all=false trainer.placement.policy_num_nodes=4 trainer.placement.ref_num_nodes=4 trainer.placement.policy_num_gpus_per_node=4 trainer.placement.ref_num_gpus_per_node=4 trainer.fully_async.max_staleness_steps=16 trainer.fully_async.num_parallel_generation_workers=512 generator.backend=vllm generator.timeout_multiplier=1.0 generator.model_dtype=bfloat16 generator.inference_engine_tensor_parallel_size=2 generator.num_inference_engines=24 generator.n_samples_per_prompt=8 generator.eval_n_samples_per_prompt=8 generator.gpu_memory_utilization=0.85 generator.max_num_seqs=24 generator.max_num_batched_tokens=16384 generator.enable_prefix_caching=true generator.enable_chunked_prefill=true generator.run_engines_locally=true generator.weight_sync_backend=nccl generator.async_engine=true generator.batched=false generator.enable_http_endpoint=true generator.enable_ray_prometheus_stats=false generator.vllm_stats_interval=1 generator.append_eos_token_after_stop_str_in_multi_turn=true generator.max_turns=999999 generator.sampling_params.max_generate_length=16384 generator.sampling_params.temperature=0.7 generator.sampling_params.top_p=0.95 generator.sampling_params.top_k=20 ++generator.engine_init_kwargs.max_model_len=32768 ++generator.engine_init_kwargs.custom_chat_template_chat_completion_path=chat_templates/qwen3_thinking_acc.jinja2 ++generator.engine_init_kwargs.served_model_name=9216db5781bf21249d130ec9da846c4624c16137 data.train_data=["/e/scratch/jureap59/feuer1/tasks/exp_rpt_stack-jest-v2"] data.val_data=["/e/scratch/jureap59/feuer1/tasks/OpenThoughts-TB-dev"] +terminal_bench_config.trials_dir=/e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/trace_jobs +terminal_bench_config.harbor.name=terminus-2 +terminal_bench_config.harbor.max_episodes=999999 +terminal_bench_config.harbor.enable_summarize=false +terminal_bench_config.harbor.store_all_messages=true +terminal_bench_config.harbor.trajectory_config.raw_content=true +terminal_bench_config.harbor.enable_episode_logging=false +terminal_bench_config.harbor.record_terminal_session=false +terminal_bench_config.harbor.enable_pane_logging=false +terminal_bench_config.harbor.strict_json_parser=true +terminal_bench_config.harbor.interleaved_thinking=true +terminal_bench_config.harbor.extra_body.chat_template_kwargs.enable_thinking=true +terminal_bench_config.harbor.override_timeout_sec=1800 +terminal_bench_config.harbor.override_cpus=1 +terminal_bench_config.harbor.override_memory_mb=2048 +terminal_bench_config.harbor.override_storage_mb=2048 +terminal_bench_config.harbor.auto_snapshot=true +terminal_bench_config.harbor.verifier_override_timeout_sec=120 +terminal_bench_config.harbor.max_retries=3 +terminal_bench_config.harbor.min_wait_sec=60.0 +terminal_bench_config.harbor.max_wait_sec=600.0 +terminal_bench_config.harbor.wait_multiplier=2.0 +terminal_bench_config.harbor.exclude_exceptions=["VerifierTimeoutError","VerifierRuntimeError","RewardFileNotFoundError","RewardFileEmptyError","VerifierOutputParseError"] +terminal_bench_config.harbor.n_concurrent_trials=256 +terminal_bench_config.harbor.log_level=INFO +terminal_bench_config.harbor.enable_reward_shaping=false +terminal_bench_config.harbor.enable_error_classification=true +terminal_bench_config.harbor.mask_exceptions=["DaytonaError","EnvironmentStartTimeoutError","NetworkError","ConnectionError","RewardFileNotFoundError","RewardFileEmptyError","AgentEnvironmentTimeoutError"] +terminal_bench_config.harbor.default_error_treatment=zero +terminal_bench_config.harbor.passthrough_exceptions=["AgentTimeoutError","ContextLengthExceededError"] +terminal_bench_config.model_info.max_input_tokens=32768 +terminal_bench_config.model_info.max_output_tokens=4096 +terminal_bench_config.archiving.enabled=false +terminal_bench_config.trace_upload.enabled=true +terminal_bench_config.trace_upload.repo_org=DCAgent +terminal_bench_config.trace_upload.episodes=last +terminal_bench_config.trace_upload.dataset_type=SFT +terminal_bench_config.trace_upload.cleanup=true [proxychains] config file found: /e/home/jusers/feuer1/jupiter/.proxychains/proxychains_388832.conf [proxychains] preloading /e/scratch/jureap59/feuer1/proxychains-ng-aarch64/lib/libproxychains4.so [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e 2026-04-24 06:37:44.831 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:590 - Exporting wandb api key to ray runtime env 2026-04-24 06:37:44.832 | INFO | skyrl_train.utils.utils:prepare_runtime_environment:609 - Exporting RAY_ADDRESS to ray runtime env 2026-04-24 06:37:44,832 INFO worker.py:1680 -- Using address 10.128.33.33:6379 set in the environment variable RAY_ADDRESS 2026-04-24 06:37:44,861 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.33.33:6379... 2026-04-24 06:37:44,870 INFO worker.py:2007 -- Connected to Ray cluster. /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( (raylet, ip=10.128.33.46) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (pid=2093226) [2026-04-24 06:37:45,192 E 2093226 2094275] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 2026-04-24 06:37:46.913 | INFO | skyrl_train.utils.ppo_utils:sync_registries:546 - Synced registries to ray actor (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 17x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (pid=2093242) [2026-04-24 06:37:50,741 E 2093242 2094821] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 13x across cluster] (skyrl_entrypoint pid=2109818) 2026-04-24 06:37:55.806 | INFO  | skyrl_train.entrypoints.main_base:_configure_log_level:199 - SkyRL log level set to: INFO (pid=2093427) [2026-04-24 06:37:55,946 E 2093427 2103528] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 121x across cluster] (skyrl_entrypoint pid=2109818) 2026-04-24 06:37:56.047 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/exp_rpt_stack-jest-v2 (skyrl_entrypoint pid=2109818) 2026-04-24 06:37:56.363 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 500 valid task directories out of 500 total directories (skyrl_entrypoint pid=2109818) 2026-04-24 06:37:56.363 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 500 task paths (skyrl_entrypoint pid=2109818) 2026-04-24 06:37:56.364 | INFO  | examples.terminal_bench.dataset:_load_data_files:40 - Loading data from: /e/scratch/jureap59/feuer1/tasks/OpenThoughts-TB-dev (skyrl_entrypoint pid=2109818) 2026-04-24 06:37:56.407 | INFO  | examples.terminal_bench.dataset:_load_data_files:50 - Found 70 valid task directories out of 70 total directories (skyrl_entrypoint pid=2109818) 2026-04-24 06:37:56.407 | INFO  | examples.terminal_bench.dataset:__init__:27 - TerminalBenchTaskDataset initialized with 70 task paths (skyrl_entrypoint pid=2109818) 2026-04-24 06:37:56.418 | INFO  | skyrl_train.entrypoints.main_base:_setup_trainer:352 - data: (skyrl_entrypoint pid=2109818) train_data: (skyrl_entrypoint pid=2109818) - /e/scratch/jureap59/feuer1/tasks/exp_rpt_stack-jest-v2 (skyrl_entrypoint pid=2109818) val_data: (skyrl_entrypoint pid=2109818) - /e/scratch/jureap59/feuer1/tasks/OpenThoughts-TB-dev (skyrl_entrypoint pid=2109818) trainer: (skyrl_entrypoint pid=2109818) placement: (skyrl_entrypoint pid=2109818) colocate_all: false (skyrl_entrypoint pid=2109818) colocate_policy_ref: true (skyrl_entrypoint pid=2109818) policy_num_nodes: 4 (skyrl_entrypoint pid=2109818) policy_num_gpus_per_node: 4 (skyrl_entrypoint pid=2109818) critic_num_nodes: 1 (skyrl_entrypoint pid=2109818) critic_num_gpus_per_node: 4 (skyrl_entrypoint pid=2109818) ref_num_nodes: 4 (skyrl_entrypoint pid=2109818) ref_num_gpus_per_node: 4 (skyrl_entrypoint pid=2109818) sequence_parallel_backend: ulysses (skyrl_entrypoint pid=2109818) strategy: fsdp2 (skyrl_entrypoint pid=2109818) policy: (skyrl_entrypoint pid=2109818) model: (skyrl_entrypoint pid=2109818) path: /e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137 (skyrl_entrypoint pid=2109818) lora: (skyrl_entrypoint pid=2109818) rank: 0 (skyrl_entrypoint pid=2109818) alpha: 16 (skyrl_entrypoint pid=2109818) dropout: 0 (skyrl_entrypoint pid=2109818) lora_sync_path: /tmp/skyrl_lora_sync (skyrl_entrypoint pid=2109818) target_modules: all-linear (skyrl_entrypoint pid=2109818) exclude_modules: null (skyrl_entrypoint pid=2109818) deepspeed_config: ${deepspeed_config.train} (skyrl_entrypoint pid=2109818) optimizer_config: (skyrl_entrypoint pid=2109818) optimizer: AdamW (skyrl_entrypoint pid=2109818) lr: 9.0e-06 (skyrl_entrypoint pid=2109818) adam_betas: (skyrl_entrypoint pid=2109818) - 0.9 (skyrl_entrypoint pid=2109818) - 0.999 (skyrl_entrypoint pid=2109818) weight_decay: 0.0 (skyrl_entrypoint pid=2109818) max_grad_norm: 10.0 (skyrl_entrypoint pid=2109818) offload_after_step: true (skyrl_entrypoint pid=2109818) num_warmup_steps: 0 (skyrl_entrypoint pid=2109818) scheduler: constant_with_warmup (skyrl_entrypoint pid=2109818) optimizer_kwargs: {} (skyrl_entrypoint pid=2109818) fsdp_config: (skyrl_entrypoint pid=2109818) cpu_offload: true (skyrl_entrypoint pid=2109818) reshard_after_forward: true (skyrl_entrypoint pid=2109818) fsdp_size: 4 (skyrl_entrypoint pid=2109818) sequence_parallel_size: 1 (skyrl_entrypoint pid=2109818) use_torch_compile: false (skyrl_entrypoint pid=2109818) record_memory: false (skyrl_entrypoint pid=2109818) megatron_config: (skyrl_entrypoint pid=2109818) tensor_model_parallel_size: 1 (skyrl_entrypoint pid=2109818) pipeline_model_parallel_size: 1 (skyrl_entrypoint pid=2109818) context_parallel_size: 1 (skyrl_entrypoint pid=2109818) expert_model_parallel_size: 1 (skyrl_entrypoint pid=2109818) expert_tensor_parallel_size: null (skyrl_entrypoint pid=2109818) ddp_config: (skyrl_entrypoint pid=2109818) grad_reduce_in_fp32: true (skyrl_entrypoint pid=2109818) overlap_grad_reduce: false (skyrl_entrypoint pid=2109818) overlap_param_gather: false (skyrl_entrypoint pid=2109818) average_in_collective: true (skyrl_entrypoint pid=2109818) model_config_kwargs: {} (skyrl_entrypoint pid=2109818) torch_profiler_config: (skyrl_entrypoint pid=2109818) enable: false (skyrl_entrypoint pid=2109818) ranks: [] (skyrl_entrypoint pid=2109818) save_path: null (skyrl_entrypoint pid=2109818) optimizer_config_kwargs: (skyrl_entrypoint pid=2109818) overlap_cpu_optimizer_d2h_h2d: false (skyrl_entrypoint pid=2109818) use_precision_aware_optimizer: false (skyrl_entrypoint pid=2109818) optimizer_cpu_offload: false (skyrl_entrypoint pid=2109818) optimizer_offload_fraction: 0.0 (skyrl_entrypoint pid=2109818) transformer_config_kwargs: (skyrl_entrypoint pid=2109818) recompute_granularity: full (skyrl_entrypoint pid=2109818) recompute_modules: (skyrl_entrypoint pid=2109818) - core_attn (skyrl_entrypoint pid=2109818) recompute_method: uniform (skyrl_entrypoint pid=2109818) recompute_num_layers: 1 (skyrl_entrypoint pid=2109818) empty_cuda_cache: true (skyrl_entrypoint pid=2109818) ref: (skyrl_entrypoint pid=2109818) model: (skyrl_entrypoint pid=2109818) path: ${trainer.policy.model.path} (skyrl_entrypoint pid=2109818) sequence_parallel_size: 1 (skyrl_entrypoint pid=2109818) deepspeed_config: ${deepspeed_config.eval} (skyrl_entrypoint pid=2109818) fsdp_config: (skyrl_entrypoint pid=2109818) cpu_offload: true (skyrl_entrypoint pid=2109818) reshard_after_forward: true (skyrl_entrypoint pid=2109818) fsdp_size: 4 (skyrl_entrypoint pid=2109818) megatron_config: (skyrl_entrypoint pid=2109818) tensor_model_parallel_size: 1 (skyrl_entrypoint pid=2109818) pipeline_model_parallel_size: 1 (skyrl_entrypoint pid=2109818) context_parallel_size: 1 (skyrl_entrypoint pid=2109818) expert_model_parallel_size: 1 (skyrl_entrypoint pid=2109818) expert_tensor_parallel_size: 1 (skyrl_entrypoint pid=2109818) model_config_kwargs: {} (skyrl_entrypoint pid=2109818) transformer_config_kwargs: {} (skyrl_entrypoint pid=2109818) critic: (skyrl_entrypoint pid=2109818) model: (skyrl_entrypoint pid=2109818) path: null (skyrl_entrypoint pid=2109818) lora: (skyrl_entrypoint pid=2109818) rank: 0 (skyrl_entrypoint pid=2109818) alpha: 16 (skyrl_entrypoint pid=2109818) dropout: 0 (skyrl_entrypoint pid=2109818) target_modules: all-linear (skyrl_entrypoint pid=2109818) exclude_modules: null (skyrl_entrypoint pid=2109818) deepspeed_config: ${deepspeed_config.train} (skyrl_entrypoint pid=2109818) optimizer_config: (skyrl_entrypoint pid=2109818) optimizer: AdamW (skyrl_entrypoint pid=2109818) lr: 5.0e-06 (skyrl_entrypoint pid=2109818) adam_betas: (skyrl_entrypoint pid=2109818) - 0.9 (skyrl_entrypoint pid=2109818) - 0.999 (skyrl_entrypoint pid=2109818) weight_decay: 0.01 (skyrl_entrypoint pid=2109818) max_grad_norm: 1.0 (skyrl_entrypoint pid=2109818) offload_after_step: true (skyrl_entrypoint pid=2109818) num_warmup_steps: 0 (skyrl_entrypoint pid=2109818) scheduler: constant_with_warmup (skyrl_entrypoint pid=2109818) optimizer_kwargs: {} (skyrl_entrypoint pid=2109818) fsdp_config: (skyrl_entrypoint pid=2109818) cpu_offload: false (skyrl_entrypoint pid=2109818) reshard_after_forward: true (skyrl_entrypoint pid=2109818) fsdp_size: -1 (skyrl_entrypoint pid=2109818) sequence_parallel_size: 1 (skyrl_entrypoint pid=2109818) algorithm: (skyrl_entrypoint pid=2109818) advantage_estimator: rloo_n (skyrl_entrypoint pid=2109818) kl_ctrl: (skyrl_entrypoint pid=2109818) type: fixed (skyrl_entrypoint pid=2109818) kl_target: 0.1 (skyrl_entrypoint pid=2109818) horizon: 10000 (skyrl_entrypoint pid=2109818) kl_estimator_type: k3 (skyrl_entrypoint pid=2109818) use_kl_estimator_k3: false (skyrl_entrypoint pid=2109818) use_abs_kl: false (skyrl_entrypoint pid=2109818) use_kl_in_reward: false (skyrl_entrypoint pid=2109818) use_kl_loss: false (skyrl_entrypoint pid=2109818) kl_loss_coef: 0.0 (skyrl_entrypoint pid=2109818) use_entropy_loss: false (skyrl_entrypoint pid=2109818) entropy_loss_coef: 0.01 (skyrl_entrypoint pid=2109818) advantage_batch_normalize: false (skyrl_entrypoint pid=2109818) value_head_prefix: value_head (skyrl_entrypoint pid=2109818) policy_loss_type: regular (skyrl_entrypoint pid=2109818) loss_reduction: token_mean (skyrl_entrypoint pid=2109818) grpo_norm_by_std: true (skyrl_entrypoint pid=2109818) rloo_n_min_group_size: 4 (skyrl_entrypoint pid=2109818) rloo_n_filter_zero_reward_groups: true (skyrl_entrypoint pid=2109818) lambd: 1.0 (skyrl_entrypoint pid=2109818) gamma: 1.0 (skyrl_entrypoint pid=2109818) eps_clip_low: 0.2 (skyrl_entrypoint pid=2109818) eps_clip_high: 0.2 (skyrl_entrypoint pid=2109818) clip_ratio_c: 3.0 (skyrl_entrypoint pid=2109818) tis_imp_ratio_cap: -1.0 (skyrl_entrypoint pid=2109818) use_tis: false (skyrl_entrypoint pid=2109818) sapo: (skyrl_entrypoint pid=2109818) tau_pos: 1.0 (skyrl_entrypoint pid=2109818) tau_neg: 1.05 (skyrl_entrypoint pid=2109818) value_clip: 0.2 (skyrl_entrypoint pid=2109818) dynamic_sampling: (skyrl_entrypoint pid=2109818) type: null (skyrl_entrypoint pid=2109818) max_sample_batches: 30 (skyrl_entrypoint pid=2109818) min_replace_ratio: 0.3 (skyrl_entrypoint pid=2109818) clip_cov: (skyrl_entrypoint pid=2109818) clip_ratio: 0.0002 (skyrl_entrypoint pid=2109818) clip_cov_lb: 1.0 (skyrl_entrypoint pid=2109818) clip_cov_ub: 5.0 (skyrl_entrypoint pid=2109818) kl_cov: (skyrl_entrypoint pid=2109818) kl_cov_frac: 0.2 (skyrl_entrypoint pid=2109818) ppo_kl_coef: 1.0 (skyrl_entrypoint pid=2109818) cispo: (skyrl_entrypoint pid=2109818) cispo_eps_clip_low: 0 (skyrl_entrypoint pid=2109818) cispo_eps_clip_high: 5 (skyrl_entrypoint pid=2109818) max_seq_len: 1016383 (skyrl_entrypoint pid=2109818) fully_async: (skyrl_entrypoint pid=2109818) max_staleness_steps: 16 (skyrl_entrypoint pid=2109818) num_parallel_generation_workers: 512 (skyrl_entrypoint pid=2109818) gradient_checkpointing: true (skyrl_entrypoint pid=2109818) gradient_checkpointing_use_reentrant: false (skyrl_entrypoint pid=2109818) seed: 42 (skyrl_entrypoint pid=2109818) resume_mode: latest (skyrl_entrypoint pid=2109818) resume_path: null (skyrl_entrypoint pid=2109818) ckpt_path: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints (skyrl_entrypoint pid=2109818) max_ckpts_to_keep: -1 (skyrl_entrypoint pid=2109818) ckpt_interval: 1 (skyrl_entrypoint pid=2109818) hf_save_interval: 5 (skyrl_entrypoint pid=2109818) export_path: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/exports (skyrl_entrypoint pid=2109818) bf16: true (skyrl_entrypoint pid=2109818) epochs: 2 (skyrl_entrypoint pid=2109818) max_steps: 60 (skyrl_entrypoint pid=2109818) update_epochs_per_batch: 1 (skyrl_entrypoint pid=2109818) train_batch_size: 64 (skyrl_entrypoint pid=2109818) policy_mini_batch_size: 64 (skyrl_entrypoint pid=2109818) critic_mini_batch_size: 256 (skyrl_entrypoint pid=2109818) micro_train_batch_size_per_gpu: 1 (skyrl_entrypoint pid=2109818) micro_forward_batch_size_per_gpu: 1 (skyrl_entrypoint pid=2109818) update_ref_every_epoch: false (skyrl_entrypoint pid=2109818) use_sample_packing: true (skyrl_entrypoint pid=2109818) eval_batch_size: 64 (skyrl_entrypoint pid=2109818) eval_before_train: false (skyrl_entrypoint pid=2109818) eval_interval: 999999 (skyrl_entrypoint pid=2109818) max_prompt_length: 999999 (skyrl_entrypoint pid=2109818) flash_attn: true (skyrl_entrypoint pid=2109818) disable_fast_tokenizer: false (skyrl_entrypoint pid=2109818) target_modules: null (skyrl_entrypoint pid=2109818) exclude_modules: null (skyrl_entrypoint pid=2109818) project_name: OpenThoughts-Agent (skyrl_entrypoint pid=2109818) run_name: a2-rl-stack_jest_v2 (skyrl_entrypoint pid=2109818) logger: console (skyrl_entrypoint pid=2109818) tracker_commit_each_step: true (skyrl_entrypoint pid=2109818) dump_data_batch: false (skyrl_entrypoint pid=2109818) dump_eval_results: true (skyrl_entrypoint pid=2109818) log_level: INFO (skyrl_entrypoint pid=2109818) rope_scaling: null (skyrl_entrypoint pid=2109818) rope_theta: null (skyrl_entrypoint pid=2109818) step_wise_training: false (skyrl_entrypoint pid=2109818) hf_hub_repo_id: laion/a2-rl-stack_jest_v2 (skyrl_entrypoint pid=2109818) hf_hub_private: false (skyrl_entrypoint pid=2109818) hf_hub_revision: main (skyrl_entrypoint pid=2109818) enable_db_registration: true (skyrl_entrypoint pid=2109818) generator: (skyrl_entrypoint pid=2109818) model_name: ${trainer.policy.model.path} (skyrl_entrypoint pid=2109818) model_dtype: bfloat16 (skyrl_entrypoint pid=2109818) timeout_multiplier: 1.0 (skyrl_entrypoint pid=2109818) run_engines_locally: true (skyrl_entrypoint pid=2109818) num_inference_engines: 24 (skyrl_entrypoint pid=2109818) backend: vllm (skyrl_entrypoint pid=2109818) weight_sync_backend: nccl (skyrl_entrypoint pid=2109818) fuse_weights: false (skyrl_entrypoint pid=2109818) weight_transfer_threshold_cuda_ipc_GB: 1.0 (skyrl_entrypoint pid=2109818) inference_engine_tensor_parallel_size: 2 (skyrl_entrypoint pid=2109818) inference_engine_pipeline_parallel_size: 1 (skyrl_entrypoint pid=2109818) inference_engine_expert_parallel_size: 1 (skyrl_entrypoint pid=2109818) inference_engine_data_parallel_size: 1 (skyrl_entrypoint pid=2109818) n_samples_per_prompt: 8 (skyrl_entrypoint pid=2109818) async_engine: true (skyrl_entrypoint pid=2109818) batched: false (skyrl_entrypoint pid=2109818) max_input_length: ${trainer.max_prompt_length} (skyrl_entrypoint pid=2109818) vllm_v1_disable_multiproc: true (skyrl_entrypoint pid=2109818) enable_prefix_caching: true (skyrl_entrypoint pid=2109818) enable_chunked_prefill: true (skyrl_entrypoint pid=2109818) max_num_batched_tokens: 16384 (skyrl_entrypoint pid=2109818) enforce_eager: true (skyrl_entrypoint pid=2109818) fully_sharded_loras: false (skyrl_entrypoint pid=2109818) enable_ray_prometheus_stats: false (skyrl_entrypoint pid=2109818) vllm_stats_interval: 1 (skyrl_entrypoint pid=2109818) gpu_memory_utilization: 0.85 (skyrl_entrypoint pid=2109818) max_num_seqs: 24 (skyrl_entrypoint pid=2109818) remote_inference_engine_urls: (skyrl_entrypoint pid=2109818) - 127.0.0.1:8001 (skyrl_entrypoint pid=2109818) enable_http_endpoint: true (skyrl_entrypoint pid=2109818) http_endpoint_host: 127.0.0.1 (skyrl_entrypoint pid=2109818) http_endpoint_port: 8000 (skyrl_entrypoint pid=2109818) max_turns: 999999 (skyrl_entrypoint pid=2109818) chat_template: (skyrl_entrypoint pid=2109818) source: name (skyrl_entrypoint pid=2109818) name_or_path: null (skyrl_entrypoint pid=2109818) chat_template_kwargs: {} (skyrl_entrypoint pid=2109818) engine_init_kwargs: (skyrl_entrypoint pid=2109818) max_model_len: 32768 (skyrl_entrypoint pid=2109818) custom_chat_template_chat_completion_path: chat_templates/qwen3_thinking_acc.jinja2 (skyrl_entrypoint pid=2109818) served_model_name: 9216db5781bf21249d130ec9da846c4624c16137 (skyrl_entrypoint pid=2109818) override_existing_update_group: disable (skyrl_entrypoint pid=2109818) sampling_params: (skyrl_entrypoint pid=2109818) max_generate_length: 16384 (skyrl_entrypoint pid=2109818) repetition_penalty: 1.0 (skyrl_entrypoint pid=2109818) temperature: 0.7 (skyrl_entrypoint pid=2109818) top_p: 0.95 (skyrl_entrypoint pid=2109818) min_p: 0.0 (skyrl_entrypoint pid=2109818) top_k: 20 (skyrl_entrypoint pid=2109818) logprobs: null (skyrl_entrypoint pid=2109818) stop: null (skyrl_entrypoint pid=2109818) use_conversation_multi_turn: true (skyrl_entrypoint pid=2109818) append_eos_token_after_stop_str_in_multi_turn: true (skyrl_entrypoint pid=2109818) eval_sampling_params: (skyrl_entrypoint pid=2109818) max_generate_length: ${generator.sampling_params.max_generate_length} (skyrl_entrypoint pid=2109818) repetition_penalty: 1.0 (skyrl_entrypoint pid=2109818) temperature: 0.0 (skyrl_entrypoint pid=2109818) top_p: 1.0 (skyrl_entrypoint pid=2109818) min_p: 0.0 (skyrl_entrypoint pid=2109818) top_k: -1 (skyrl_entrypoint pid=2109818) logprobs: null (skyrl_entrypoint pid=2109818) stop: null (skyrl_entrypoint pid=2109818) eval_n_samples_per_prompt: 8 (skyrl_entrypoint pid=2109818) zero_reward_on_non_stop: false (skyrl_entrypoint pid=2109818) apply_overlong_filtering: false (skyrl_entrypoint pid=2109818) rope_scaling: ${trainer.rope_scaling} (skyrl_entrypoint pid=2109818) rope_theta: ${trainer.rope_theta} (skyrl_entrypoint pid=2109818) teacher: (skyrl_entrypoint pid=2109818) model_path: null (skyrl_entrypoint pid=2109818) top_k_logprobs: 256 (skyrl_entrypoint pid=2109818) num_inference_engines: 1 (skyrl_entrypoint pid=2109818) inference_engine_tensor_parallel_size: 1 (skyrl_entrypoint pid=2109818) inference_engine_pipeline_parallel_size: 1 (skyrl_entrypoint pid=2109818) gpu_memory_utilization: 0.9 (skyrl_entrypoint pid=2109818) enforce_eager: false (skyrl_entrypoint pid=2109818) backend: vllm (skyrl_entrypoint pid=2109818) engine_init_kwargs: {} (skyrl_entrypoint pid=2109818) environment: (skyrl_entrypoint pid=2109818) env_class: gsm8k (skyrl_entrypoint pid=2109818) skyrl_gym: (skyrl_entrypoint pid=2109818) max_env_workers: 32 (skyrl_entrypoint pid=2109818) text2sql: (skyrl_entrypoint pid=2109818) db_path: /home/ray/default/sql_data (skyrl_entrypoint pid=2109818) llm_as_a_judge: (skyrl_entrypoint pid=2109818) model: gpt-4o-mini (skyrl_entrypoint pid=2109818) base_url: null (skyrl_entrypoint pid=2109818) search: (skyrl_entrypoint pid=2109818) log_requests: false (skyrl_entrypoint pid=2109818) search_url: http://127.0.0.1:8000/retrieve (skyrl_entrypoint pid=2109818) topk: 3 (skyrl_entrypoint pid=2109818) timeout: 30 (skyrl_entrypoint pid=2109818) deepspeed_config: (skyrl_entrypoint pid=2109818) train: (skyrl_entrypoint pid=2109818) zero_optimization: (skyrl_entrypoint pid=2109818) stage: 3 (skyrl_entrypoint pid=2109818) offload_param: (skyrl_entrypoint pid=2109818) device: none (skyrl_entrypoint pid=2109818) offload_optimizer: (skyrl_entrypoint pid=2109818) device: none (skyrl_entrypoint pid=2109818) pin_memory: true (skyrl_entrypoint pid=2109818) sub_group_size: auto (skyrl_entrypoint pid=2109818) reduce_bucket_size: auto (skyrl_entrypoint pid=2109818) stage3_param_persistence_threshold: auto (skyrl_entrypoint pid=2109818) stage3_prefetch_bucket_size: auto (skyrl_entrypoint pid=2109818) stage3_max_live_parameters: auto (skyrl_entrypoint pid=2109818) stage3_max_reuse_distance: auto (skyrl_entrypoint pid=2109818) round_robin_gradients: true (skyrl_entrypoint pid=2109818) zero_hpz_partition_size: 1 (skyrl_entrypoint pid=2109818) zero_quantized_weights: false (skyrl_entrypoint pid=2109818) zero_quantized_gradients: false (skyrl_entrypoint pid=2109818) torch_autocast: (skyrl_entrypoint pid=2109818) enabled: true (skyrl_entrypoint pid=2109818) dtype: bfloat16 (skyrl_entrypoint pid=2109818) disable_trace_cache: false (skyrl_entrypoint pid=2109818) data_types: (skyrl_entrypoint pid=2109818) grad_accum_dtype: fp32 (skyrl_entrypoint pid=2109818) gradient_clipping: 1.0 (skyrl_entrypoint pid=2109818) wall_clock_breakdown: false (skyrl_entrypoint pid=2109818) prescale_gradient: false (skyrl_entrypoint pid=2109818) eval: (skyrl_entrypoint pid=2109818) zero_optimization: (skyrl_entrypoint pid=2109818) stage: 3 (skyrl_entrypoint pid=2109818) stage3_param_persistence_threshold: auto (skyrl_entrypoint pid=2109818) offload_param: (skyrl_entrypoint pid=2109818) device: cpu (skyrl_entrypoint pid=2109818) pin_memory: true (skyrl_entrypoint pid=2109818) torch_autocast: (skyrl_entrypoint pid=2109818) enabled: true (skyrl_entrypoint pid=2109818) dtype: bfloat16 (skyrl_entrypoint pid=2109818) gradient_clipping: 1.0 (skyrl_entrypoint pid=2109818) prescale_gradient: false (skyrl_entrypoint pid=2109818) wall_clock_breakdown: false (skyrl_entrypoint pid=2109818) terminal_bench_config: (skyrl_entrypoint pid=2109818) trials_dir: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/trace_jobs (skyrl_entrypoint pid=2109818) harbor: (skyrl_entrypoint pid=2109818) name: terminus-2 (skyrl_entrypoint pid=2109818) max_episodes: 999999 (skyrl_entrypoint pid=2109818) enable_summarize: false (skyrl_entrypoint pid=2109818) store_all_messages: true (skyrl_entrypoint pid=2109818) trajectory_config: (skyrl_entrypoint pid=2109818) raw_content: true (skyrl_entrypoint pid=2109818) enable_episode_logging: false (skyrl_entrypoint pid=2109818) record_terminal_session: false (skyrl_entrypoint pid=2109818) enable_pane_logging: false (skyrl_entrypoint pid=2109818) strict_json_parser: true (skyrl_entrypoint pid=2109818) interleaved_thinking: true (skyrl_entrypoint pid=2109818) extra_body: (skyrl_entrypoint pid=2109818) chat_template_kwargs: (skyrl_entrypoint pid=2109818) enable_thinking: true (skyrl_entrypoint pid=2109818) override_timeout_sec: 1800 (skyrl_entrypoint pid=2109818) override_cpus: 1 (skyrl_entrypoint pid=2109818) override_memory_mb: 2048 (skyrl_entrypoint pid=2109818) override_storage_mb: 2048 (skyrl_entrypoint pid=2109818) auto_snapshot: true (skyrl_entrypoint pid=2109818) verifier_override_timeout_sec: 120 (skyrl_entrypoint pid=2109818) max_retries: 3 (skyrl_entrypoint pid=2109818) min_wait_sec: 60.0 (skyrl_entrypoint pid=2109818) max_wait_sec: 600.0 (skyrl_entrypoint pid=2109818) wait_multiplier: 2.0 (skyrl_entrypoint pid=2109818) exclude_exceptions: (skyrl_entrypoint pid=2109818) - VerifierTimeoutError (skyrl_entrypoint pid=2109818) - VerifierRuntimeError (skyrl_entrypoint pid=2109818) - RewardFileNotFoundError (skyrl_entrypoint pid=2109818) - RewardFileEmptyError (skyrl_entrypoint pid=2109818) - VerifierOutputParseError (skyrl_entrypoint pid=2109818) n_concurrent_trials: 256 (skyrl_entrypoint pid=2109818) log_level: INFO (skyrl_entrypoint pid=2109818) enable_reward_shaping: false (skyrl_entrypoint pid=2109818) enable_error_classification: true (skyrl_entrypoint pid=2109818) mask_exceptions: (skyrl_entrypoint pid=2109818) - DaytonaError (skyrl_entrypoint pid=2109818) - EnvironmentStartTimeoutError (skyrl_entrypoint pid=2109818) - NetworkError (skyrl_entrypoint pid=2109818) - ConnectionError (skyrl_entrypoint pid=2109818) - RewardFileNotFoundError (skyrl_entrypoint pid=2109818) - RewardFileEmptyError (skyrl_entrypoint pid=2109818) - AgentEnvironmentTimeoutError (skyrl_entrypoint pid=2109818) default_error_treatment: zero (skyrl_entrypoint pid=2109818) passthrough_exceptions: (skyrl_entrypoint pid=2109818) - AgentTimeoutError (skyrl_entrypoint pid=2109818) - ContextLengthExceededError (skyrl_entrypoint pid=2109818) model_info: (skyrl_entrypoint pid=2109818) max_input_tokens: 32768 (skyrl_entrypoint pid=2109818) max_output_tokens: 4096 (skyrl_entrypoint pid=2109818) archiving: (skyrl_entrypoint pid=2109818) enabled: false (skyrl_entrypoint pid=2109818) trace_upload: (skyrl_entrypoint pid=2109818) enabled: true (skyrl_entrypoint pid=2109818) repo_org: DCAgent (skyrl_entrypoint pid=2109818) episodes: last (skyrl_entrypoint pid=2109818) dataset_type: SFT (skyrl_entrypoint pid=2109818) cleanup: true (skyrl_entrypoint pid=2109818)  (skyrl_entrypoint pid=2109818) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (skyrl_entrypoint pid=2109818) No module named 'vllm._version' (skyrl_entrypoint pid=2109818) from .version import __version__, __version_tuple__ # isort:skip (skyrl_entrypoint pid=2109818) W0424 06:38:01.093000 2109818 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' (pid=2093357) [2026-04-24 06:37:56,337 E 2093357 2108164] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 147x across cluster] (raylet) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e (pid=1698116, ip=10.128.33.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (pid=1698116, ip=10.128.33.39) No module named 'vllm._version' (pid=1698116, ip=10.128.33.39) from .version import __version__, __version_tuple__ # isort:skip (raylet, ip=10.128.33.42) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 41x across cluster] (pid=831169, ip=10.128.33.43) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 3x across cluster] (pid=831169, ip=10.128.33.43) No module named 'vllm._version' [repeated 3x across cluster] (pid=831169, ip=10.128.33.43) from .version import __version__, __version_tuple__ # isort:skip [repeated 3x across cluster] (raylet, ip=10.128.33.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 62x across cluster] [2026-04-24 06:38:15,092 E 2109408 2109790] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (RegistryActor pid=1955327, ip=10.128.33.46) [2026-04-24 06:38:15,813 E 1955327 1955367] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (pid=1698116, ip=10.128.33.39) W0424 06:38:16.000000 1698116 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' (pid=2361797, ip=10.128.33.37) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 9x across cluster] (pid=2361797, ip=10.128.33.37) No module named 'vllm._version' [repeated 9x across cluster] (pid=2361797, ip=10.128.33.37) from .version import __version__, __version_tuple__ # isort:skip [repeated 9x across cluster] (raylet, ip=10.128.33.35) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 46x across cluster] (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) 2026-04-24 06:38:17.724 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:110 - creating LLM with bundle_indices=[0, 1] (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) 2026-04-24 06:38:17.724 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=ray, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=, VLLM_ENABLE_V1_MULTIPROCESSING= (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) 2026-04-24 06:38:17.724 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING= (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) 2026-04-24 06:38:17.724 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) 2026-04-24 06:38:17.724 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) 2026-04-24 06:38:17.743 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 2.00s to avoid port collisions (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. (skyrl_entrypoint pid=2109818) [2026-04-24 06:38:17,652 E 2109818 2109859] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 2x across cluster] (pid=831169, ip=10.128.33.43) W0424 06:38:19.971000 831169 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) No module named 'vllm._version' [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) from .version import __version__, __version_tuple__ # isort:skip [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 72x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) 2026-04-24 06:38:22.824 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:110 - creating LLM with bundle_indices=[8, 9] [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) 2026-04-24 06:38:22.824 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=ray, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) 2026-04-24 06:38:22.824 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) 2026-04-24 06:38:22.824 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) 2026-04-24 06:38:22.824 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) 2026-04-24 06:38:22.844 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 1.72s to avoid port collisions [repeated 6x across cluster] (skyrl_entrypoint pid=2109818) [2026-04-24 06:38:23] INFO inference_engine_client_http_endpoint.py:350: Starting server on 127.0.0.1:8000 (skyrl_entrypoint pid=2109818) [2026-04-24 06:38:23] INFO inference_engine_client_http_endpoint.py:242: Starting inference HTTP endpoint... (skyrl_entrypoint pid=2109818) [2026-04-24 06:38:24] INFO inference_engine_client_http_endpoint.py:229: Server ready after 2 attempts (2 seconds) (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.237 | INFO  | skyrl_train.inference_engines.inference_engine_client:_spin_up_http_endpoint:960 - InferenceEngineClient HTTP endpoint started on 127.0.0.1:8000 (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.237 | INFO  | skyrl_train.inference_engines.inference_engine_client:__init__:61 - InferenceEngineClient initialized with 24 engines. (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.239 | INFO  | examples.terminal_bench.terminal_bench_generator:_configure_harbor_logging:172 - Harbor logging level set to INFO (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.240 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:106 - TerminalBenchGenerator initialized with HarborConfigBuilder. Exposed fields: ['name', 'max_episodes', 'enable_summarize', 'store_all_messages', 'trajectory_config', 'enable_episode_logging', 'record_terminal_session', 'enable_pane_logging', 'strict_json_parser', 'interleaved_thinking', 'extra_body', 'override_timeout_sec', 'override_cpus', 'override_memory_mb', 'override_storage_mb', 'auto_snapshot', 'verifier_override_timeout_sec', 'max_retries', 'min_wait_sec', 'max_wait_sec', 'wait_multiplier', 'exclude_exceptions', 'n_concurrent_trials', 'log_level', 'enable_reward_shaping', 'enable_error_classification', 'mask_exceptions', 'default_error_treatment', 'passthrough_exceptions']. Retry config: max_retries=3, backoff=60.0-600.0s. Concurrent trials: 256. Reward shaping: enabled=False, shaper=pass_ratio. Error classification: enabled=True (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.241 | INFO  | examples.terminal_bench.terminal_bench_generator:__init__:122 - TerminalBenchGenerator initialized with custom chat template read from: chat_templates/qwen3_thinking_acc.jinja2 (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.242 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:656 - Total steps: 14 (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.242 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:351 - Length of train_dataloader: 500 (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.242 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:352 - Number of steps per epoch: 7 (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.242 | INFO  | skyrl_train.fully_async_trainer:_build_train_dataloader_and_compute_training_steps:353 - Total training steps: 14 (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:24.242 | INFO  | skyrl_train.utils.trainer_utils:build_dataloader:658 - Validation set size: 2 (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 12x across cluster] (pid=1955400, ip=10.128.33.46) W0424 06:38:26.204000 1955400 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' [repeated 8x across cluster] (pid=4151080, ip=10.128.33.41) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) No module named 'vllm._version' [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) from .version import __version__, __version_tuple__ # isort:skip [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1690483, ip=10.128.33.36) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 27x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) 2026-04-24 06:38:27.895 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:110 - creating LLM with bundle_indices=[38, 39] [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) 2026-04-24 06:38:27.895 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=ray, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) 2026-04-24 06:38:27.895 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) 2026-04-24 06:38:27.895 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) 2026-04-24 06:38:27.896 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) 2026-04-24 06:38:27.910 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 2.10s to avoid port collisions [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) 2026-04-24 06:38:28,069 INFO worker.py:1680 -- Using address 10.128.33.33:6379 set in the environment variable RAY_ADDRESS (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) 2026-04-24 06:38:28,097 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.33.33:6379... (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) 2026-04-24 06:38:28,107 INFO worker.py:2007 -- Connected to Ray cluster. (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 18x across cluster] (pid=1994236, ip=10.128.33.45) W0424 06:38:30.776000 1994236 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) warnings.warn( (bundle_reservation_check_func pid=2109901) [2026-04-24 06:38:32,809 E 2109901 2109941] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) No module named 'vllm._version' [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) from .version import __version__, __version_tuple__ # isort:skip [repeated 6x across cluster] (raylet, ip=10.128.33.43) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 1074x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) 2026-04-24 06:38:32.121 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:110 - creating LLM with bundle_indices=[42, 43] [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) 2026-04-24 06:38:32.121 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=ray, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) 2026-04-24 06:38:32.121 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) 2026-04-24 06:38:32.121 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) 2026-04-24 06:38:32.121 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) 2026-04-24 06:38:32.143 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 1.92s to avoid port collisions [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) 2026-04-24 06:38:32,111 INFO worker.py:1680 -- Using address 10.128.33.33:6379 set in the environment variable RAY_ADDRESS [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) 2026-04-24 06:38:32,143 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.33.33:6379... [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) 2026-04-24 06:38:32,152 INFO worker.py:2007 -- Connected to Ray cluster. [repeated 4x across cluster] (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) 2026-04-24 06:38:33 INFO [ipv4-debug] hostname=jpbo-047-09.jupiter.internal (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) 2026-04-24 06:38:33 INFO [ipv4-debug] _global_node.node_ip_address=10.128.33.41 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) 2026-04-24 06:38:33 INFO [ipv4-debug] get_node_ip_address()=10.128.33.41 (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:33.368 | INFO  | skyrl_train.workers.worker:_initiate_actors:496 - Initializing process group for RayActorGroup (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [W424 06:38:33.489203296 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-047-09-interconnect-1.jupiter.internal]:45717 (errno: 97 - Address family not supported by protocol). (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 12x across cluster] (pid=4151157, ip=10.128.33.41) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. (pid=1671783, ip=10.128.33.47) W0424 06:38:33.066000 1671783 envs/rl/lib/python3.12/site-packages/torch/utils/cpp_extension.py:117] No CUDA runtime is found, using CUDA_HOME='/e/software/default/stages/2026/software/CUDA/13' [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (raylet, ip=10.128.33.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 980x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (AsyncVLLMInferenceEngine pid=1989927, ip=10.128.33.40) (EngineCore_DP0 pid=1990134) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 (AsyncVLLMInferenceEngine pid=1989927, ip=10.128.33.40) (EngineCore_DP0 pid=1990134) warnings.warn( (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) [2026-04-24 06:38:36,694 E 2215993 2216096] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) No module named 'vllm._version' [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) from .version import __version__, __version_tuple__ # isort:skip [repeated 7x across cluster] (raylet, ip=10.128.33.44) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 965x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) 2026-04-24 06:38:34.328 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:110 - creating LLM with bundle_indices=[46, 47] [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) 2026-04-24 06:38:34.329 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:setup_envvars_for_vllm:123 - setup_envvars_for_vllm: distributed_executor_backend=ray, SKYRL_ENABLE_NUMA_AFFINITY=1, CUDA_VISIBLE_DEVICES=, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) 2026-04-24 06:38:34.329 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:481 - BaseVLLMInferenceEngine: vllm_v1_disable_multiproc=True, vllm.__version__=dev, VLLM_ENABLE_V1_MULTIPROCESSING= [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) 2026-04-24 06:38:34.329 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:__init__:489 - BaseVLLMInferenceEngine: set VLLM_ENABLE_V1_MULTIPROCESSING=0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) 2026-04-24 06:38:34.329 | WARNING | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1047 - OpenAI API sampling params overridden: temperature=0.7, top_p=0.95, top_k=20 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) 2026-04-24 06:38:34.350 | INFO | skyrl_train.inference_engines.vllm.vllm_engine:_create_engine:1078 - Engine startup stagger: sleeping 2.79s to avoid port collisions [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) 2026-04-24 06:38:37,716 INFO worker.py:1680 -- Using address 10.128.33.33:6379 set in the environment variable RAY_ADDRESS [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) 2026-04-24 06:38:37,744 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.33.33:6379... [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) 2026-04-24 06:38:37,804 INFO worker.py:2007 -- Connected to Ray cluster. [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1958270, ip=10.128.33.44) (EngineCore_DP0 pid=1958353) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 (AsyncVLLMInferenceEngine pid=1958270, ip=10.128.33.44) (EngineCore_DP0 pid=1958353) warnings.warn( (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 4x across cluster] (pid=2370204, ip=10.128.33.48) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=4151157, ip=10.128.33.41) [W424 06:38:40.901343659 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-047-09.jupiter.internal]:45717 (errno: 97 - Address family not supported by protocol). (AsyncVLLMInferenceEngine pid=1989927, ip=10.128.33.40) (EngineCore_DP0 pid=1990134) (raylet, ip=10.128.33.34) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 981x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (AsyncVLLMInferenceEngine pid=1989927, ip=10.128.33.40) [2026-04-24 06:38:41,887 E 1989927 1990030] core_worker_process.cc:842: Failed to establish connection to the metrics exporter agent. Metrics will not be exported. Exporter agent status: RpcError: Running out of retries to initialize the metrics agent. rpc_code: 14 [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (raylet, ip=10.128.33.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 1086x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/vllm/__init__.py:7: RuntimeWarning: Failed to read commit hash: (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) No module named 'vllm._version' (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) from .version import __version__, __version_tuple__ # isort:skip (raylet, ip=10.128.33.45) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 1016x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) 2026-04-24 06:38:43,043 INFO worker.py:1680 -- Using address 10.128.33.33:6379 set in the environment variable RAY_ADDRESS [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) 2026-04-24 06:38:43,075 INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.128.33.33:6379... [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) 2026-04-24 06:38:43,084 INFO worker.py:2007 -- Connected to Ray cluster. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1958270, ip=10.128.33.44) (EngineCore_DP0 pid=1958353) (raylet, ip=10.128.33.47) [proxychains] DLL init: proxychains-ng 4.17-git-9-g78ead0e [repeated 1013x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) warnings.warn( (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) /e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) warnings.warn( (FSDPPolicyWorkerBase pid=2370205, ip=10.128.33.48) [W424 06:38:45.073942351 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [jpbo-047-09.jupiter.internal]:45717 (errno: 97 - Address family not supported by protocol). [repeated 14x across cluster] (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) `torch_dtype` is deprecated! Use `dtype` instead! (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:46.477 | INFO  | skyrl_train.workers.worker:_initiate_actors:498 - Initialized process group for RayActorGroup (skyrl_entrypoint pid=2109818) 2026-04-24 06:38:46.494 | INFO  | skyrl_train.workers.worker:_initiate_actors:500 - Mesh Ranks: [MeshRank(dp=0, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=1, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=2, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=3, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=4, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=5, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=6, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=7, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=8, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=9, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=10, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=11, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=12, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=13, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=14, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1), MeshRank(dp=15, sp=0, tp=0, pp=0, world_size=16, dp_size=16, pp_size=1)] (FSDPPolicyWorkerBase pid=2370205, ip=10.128.33.48) Loading checkpoint shards: 0%| | 0/17 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) WARNING 04-24 06:38:30 [arg_utils.py:1256] The global random seed is set to 61. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) INFO 04-24 06:38:30 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) INFO 04-24 06:38:30 [model.py:1549] Using max model len 32768 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) INFO 04-24 06:38:30 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) INFO 04-24 06:38:30 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=16384. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) WARNING 04-24 06:38:30 [vllm.py:672] Async scheduling will be disabled because it is not supported with the `ray` distributed executor backend (only `mp`, `uni`, and `external_launcher` are supported). [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) INFO 04-24 06:38:30 [vllm.py:690] Asynchronous scheduling is disabled. [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) WARNING 04-24 06:38:30 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) INFO 04-24 06:38:30 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) WARNING 04-24 06:38:30 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) WARNING 04-24 06:38:30 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 9x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) INFO 04-24 06:38:32 [ray_utils.py:469] Using the existing placement group (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) INFO 04-24 06:38:32 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=46, served_model_name=9216db5781bf21249d130ec9da846c4624c16137, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 4x across cluster] (pid=4151080, ip=10.128.33.41) ⚙️ Running in WANDB offline mode (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) WARNING 04-24 06:38:34 [arg_utils.py:1256] The global random seed is set to 62. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) INFO 04-24 06:38:34 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) INFO 04-24 06:38:34 [model.py:1549] Using max model len 32768 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) INFO 04-24 06:38:34 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) INFO 04-24 06:38:34 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=16384. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) WARNING 04-24 06:38:34 [vllm.py:672] Async scheduling will be disabled because it is not supported with the `ray` distributed executor backend (only `mp`, `uni`, and `external_launcher` are supported). [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) INFO 04-24 06:38:34 [vllm.py:690] Asynchronous scheduling is disabled. [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) WARNING 04-24 06:38:34 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) INFO 04-24 06:38:34 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) WARNING 04-24 06:38:35 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) WARNING 04-24 06:38:35 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1989927, ip=10.128.33.40) (EngineCore_DP0 pid=1990134) INFO 04-24 06:38:35 [ray_utils.py:469] Using the existing placement group (AsyncVLLMInferenceEngine pid=1958270, ip=10.128.33.44) (EngineCore_DP0 pid=1958353) INFO 04-24 06:38:38 [ray_utils.py:469] Using the existing placement group (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) INFO 04-24 06:38:37 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=60, served_model_name=9216db5781bf21249d130ec9da846c4624c16137, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 7x across cluster] (pid=4151157, ip=10.128.33.41) ⚙️ Running in WANDB offline mode (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) WARNING 04-24 06:38:37 [arg_utils.py:1256] The global random seed is set to 65. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) INFO 04-24 06:38:37 [model.py:529] Resolved architecture: Qwen3ForCausalLM [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) INFO 04-24 06:38:37 [model.py:1549] Using max model len 32768 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) INFO 04-24 06:38:37 [arg_utils.py:1469] Using ray runtime env (env vars redacted): {'env_vars': {'NCCL_CUMEM_ENABLE': '***', 'RAY_ADDRESS': '***', 'VLLM_ALLOW_INSECURE_SERIALIZATION': '***', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': '***', 'VLLM_DISABLE_COMPILE_CACHE': '***', 'WANDB_API_KEY': '***'}} [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) INFO 04-24 06:38:37 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=16384. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) WARNING 04-24 06:38:37 [vllm.py:672] Async scheduling will be disabled because it is not supported with the `ray` distributed executor backend (only `mp`, `uni`, and `external_launcher` are supported). [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) INFO 04-24 06:38:37 [vllm.py:690] Asynchronous scheduling is disabled. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) WARNING 04-24 06:38:37 [vllm.py:728] Enforce eager set, overriding optimization level to -O0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) INFO 04-24 06:38:37 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) WARNING 04-24 06:38:37 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) WARNING 04-24 06:38:37 [system_utils.py:140] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) INFO 04-24 06:38:43 [ray_utils.py:469] Using the existing placement group [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) INFO 04-24 06:38:43 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=64, served_model_name=9216db5781bf21249d130ec9da846c4624c16137, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 5x across cluster] (pid=2370205, ip=10.128.33.48) ⚙️ Running in WANDB offline mode [repeated 14x across cluster] (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:38:52 [ray_utils.py:469] Using the existing placement group (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) INFO 04-24 06:38:53 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=48, served_model_name=9216db5781bf21249d130ec9da846c4624c16137, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) INFO 04-24 06:38:53 [ray_utils.py:469] Using the existing placement group (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) (EngineCore_DP0 pid=1698330) INFO 04-24 06:38:54 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=49, served_model_name=9216db5781bf21249d130ec9da846c4624c16137, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) INFO 04-24 06:38:59 [ray_utils.py:469] Using the existing placement group [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:39:00 [ray_env.py:66] RAY_NON_CARRY_OVER_ENV_VARS from config: set() (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:39:00 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ENABLE_V1_MULTIPROCESSING', 'HF_TOKEN', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'CUDA_HOME', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_CACHE_ROOT', 'VLLM_RAY_BUNDLE_INDICES', 'LD_LIBRARY_PATH', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_RAY_PER_WORKER_GPUS'] (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:39:00 [ray_env.py:74] If certain env vars should NOT be copied, add them to /e/home/jusers/feuer1/jupiter/.config/vllm/ray_non_carry_over_env_vars.json file (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) INFO 04-24 06:39:02 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_RAY_BUNDLE_INDICES', 'HF_TOKEN', 'VLLM_CACHE_ROOT', 'CUDA_HOME', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_DISABLE_COMPILE_CACHE', 'LD_LIBRARY_PATH', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_ENABLE_V1_MULTIPROCESSING'] (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) INFO 04-24 06:39:03 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_DISABLE_COMPILE_CACHE', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'HF_TOKEN', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_RAY_BUNDLE_INDICES', 'LD_LIBRARY_PATH', 'VLLM_CACHE_ROOT', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'CUDA_HOME'] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) (EngineCore_DP0 pid=1698330) INFO 04-24 06:39:03 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_RAY_BUNDLE_INDICES', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_RAY_PER_WORKER_GPUS', 'HF_TOKEN', 'VLLM_CACHE_ROOT', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_WORKER_MULTIPROC_METHOD', 'LD_LIBRARY_PATH', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'CUDA_HOME', 'VLLM_ALLOW_INSECURE_SERIALIZATION'] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) INFO 04-24 06:39:03 [ray_env.py:69] Copying the following environment variables to workers: ['LD_LIBRARY_PATH', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_CACHE_ROOT', 'CUDA_HOME', 'VLLM_USE_FLASHINFER_SAMPLER', 'HF_TOKEN', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_ALLREDUCE_USE_SYMM_MEM'] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) INFO 04-24 06:39:03 [ray_env.py:69] Copying the following environment variables to workers: ['HF_TOKEN', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'LD_LIBRARY_PATH', 'VLLM_CACHE_ROOT', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_RAY_PER_WORKER_GPUS', 'CUDA_HOME', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING'] (AsyncVLLMInferenceEngine pid=1690483, ip=10.128.33.36) (EngineCore_DP0 pid=1690574) INFO 04-24 06:39:03 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_CACHE_ROOT', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_RAY_PER_WORKER_GPUS', 'HF_TOKEN', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'LD_LIBRARY_PATH', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'CUDA_HOME'] (AsyncVLLMInferenceEngine pid=831041, ip=10.128.33.43) (EngineCore_DP0 pid=831322) INFO 04-24 06:39:04 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_RAY_BUNDLE_INDICES', 'CUDA_HOME', 'HF_TOKEN', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'LD_LIBRARY_PATH', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_CACHE_ROOT'] (AsyncVLLMInferenceEngine pid=1989927, ip=10.128.33.40) (EngineCore_DP0 pid=1990134) INFO 04-24 06:39:07 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'CUDA_HOME', 'VLLM_USE_FLASHINFER_SAMPLER', 'HF_TOKEN', 'VLLM_RAY_BUNDLE_INDICES', 'LD_LIBRARY_PATH', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_CACHE_ROOT', 'VLLM_ALLOW_INSECURE_SERIALIZATION'] (AsyncVLLMInferenceEngine pid=1706685, ip=10.128.33.35) (EngineCore_DP0 pid=1706902) INFO 04-24 06:39:04 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=56, served_model_name=9216db5781bf21249d130ec9da846c4624c16137, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1706685, ip=10.128.33.35) (EngineCore_DP0 pid=1706902) INFO 04-24 06:39:04 [ray_utils.py:469] Using the existing placement group [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=1989927, ip=10.128.33.40) (EngineCore_DP0 pid=1990134) INFO 04-24 06:39:07 [ray_env.py:66] RAY_NON_CARRY_OVER_ENV_VARS from config: set() [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1989927, ip=10.128.33.40) (EngineCore_DP0 pid=1990134) INFO 04-24 06:39:07 [ray_env.py:74] If certain env vars should NOT be copied, add them to /e/home/jusers/feuer1/jupiter/.config/vllm/ray_non_carry_over_env_vars.json file [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) INFO 04-24 06:39:07 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_RAY_PER_WORKER_GPUS', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_CACHE_ROOT', 'LD_LIBRARY_PATH', 'HF_TOKEN', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_WORKER_MULTIPROC_METHOD', 'CUDA_HOME', 'VLLM_RAY_BUNDLE_INDICES'] (AsyncVLLMInferenceEngine pid=1958270, ip=10.128.33.44) (EngineCore_DP0 pid=1958353) INFO 04-24 06:39:09 [ray_env.py:69] Copying the following environment variables to workers: ['LD_LIBRARY_PATH', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_CACHE_ROOT', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_RAY_PER_WORKER_GPUS', 'HF_TOKEN', 'VLLM_USE_FLASHINFER_SAMPLER', 'CUDA_HOME', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING'] (AsyncVLLMInferenceEngine pid=1958143, ip=10.128.33.44) (EngineCore_DP0 pid=1958349) INFO 04-24 06:39:09 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ALLOW_INSECURE_SERIALIZATION', 'HF_TOKEN', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_RAY_BUNDLE_INDICES', 'LD_LIBRARY_PATH', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'CUDA_HOME', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_CACHE_ROOT'] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) INFO 04-24 06:39:09 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'CUDA_HOME', 'LD_LIBRARY_PATH', 'VLLM_USE_FLASHINFER_SAMPLER', 'HF_TOKEN', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_CACHE_ROOT'] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) INFO 04-24 06:39:10 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'CUDA_HOME', 'LD_LIBRARY_PATH', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_CACHE_ROOT', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'HF_TOKEN', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_DISABLE_COMPILE_CACHE'] (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) INFO 04-24 06:39:10 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_RAY_BUNDLE_INDICES', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_CACHE_ROOT', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'HF_TOKEN', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'LD_LIBRARY_PATH', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'CUDA_HOME', 'VLLM_WORKER_MULTIPROC_METHOD'] (AsyncVLLMInferenceEngine pid=1955400, ip=10.128.33.46) (EngineCore_DP0 pid=1955609) INFO 04-24 06:39:10 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'LD_LIBRARY_PATH', 'VLLM_USE_FLASHINFER_SAMPLER', 'CUDA_HOME', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_RAY_PER_WORKER_GPUS', 'HF_TOKEN', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_CACHE_ROOT', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_ENABLE_V1_MULTIPROCESSING'] (AsyncVLLMInferenceEngine pid=3867661, ip=10.128.33.34) (EngineCore_DP0 pid=3867869) INFO 04-24 06:39:11 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_RAY_BUNDLE_INDICES', 'LD_LIBRARY_PATH', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_CACHE_ROOT', 'CUDA_HOME', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'HF_TOKEN', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_INSECURE_SERIALIZATION'] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) INFO 04-24 06:39:11 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_CACHE_ROOT', 'HF_TOKEN', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'LD_LIBRARY_PATH', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'CUDA_HOME'] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) INFO 04-24 06:39:12 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_CACHE_ROOT', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'HF_TOKEN', 'VLLM_WORKER_MULTIPROC_METHOD', 'LD_LIBRARY_PATH', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'CUDA_HOME', 'VLLM_RAY_PER_WORKER_GPUS'] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) INFO 04-24 06:39:08 [core.py:97] Initializing a V1 LLM engine (vdev) with config: model='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', speculative_config=None, tokenizer='/e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=65, served_model_name=9216db5781bf21249d130ec9da846c4624c16137, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []} (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) INFO 04-24 06:39:08 [ray_utils.py:469] Using the existing placement group [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) INFO 04-24 06:39:12 [ray_env.py:66] RAY_NON_CARRY_OVER_ENV_VARS from config: set() [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) INFO 04-24 06:39:12 [ray_env.py:74] If certain env vars should NOT be copied, add them to /e/home/jusers/feuer1/jupiter/.config/vllm/ray_non_carry_over_env_vars.json file [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1706685, ip=10.128.33.35) (EngineCore_DP0 pid=1706902) INFO 04-24 06:39:12 [ray_env.py:69] Copying the following environment variables to workers: ['CUDA_HOME', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'LD_LIBRARY_PATH', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'HF_TOKEN', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_CACHE_ROOT', 'VLLM_WORKER_MULTIPROC_METHOD'] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) INFO 04-24 06:39:14 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_RAY_PER_WORKER_GPUS', 'VLLM_WORKER_MULTIPROC_METHOD', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'LD_LIBRARY_PATH', 'VLLM_USE_FLASHINFER_SAMPLER', 'CUDA_HOME', 'VLLM_DISABLE_COMPILE_CACHE', 'HF_TOKEN', 'VLLM_CACHE_ROOT', 'VLLM_ENABLE_V1_MULTIPROCESSING'] (AsyncVLLMInferenceEngine pid=1994236, ip=10.128.33.45) (EngineCore_DP0 pid=1994315) INFO 04-24 06:39:15 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_DISABLE_COMPILE_CACHE', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_CACHE_ROOT', 'LD_LIBRARY_PATH', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'VLLM_WORKER_MULTIPROC_METHOD', 'HF_TOKEN', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'CUDA_HOME', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'VLLM_RAY_PER_WORKER_GPUS'] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) INFO 04-24 06:39:16 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_DISABLE_COMPILE_CACHE', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_CACHE_ROOT', 'VLLM_ALLOW_INSECURE_SERIALIZATION', 'CUDA_HOME', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'VLLM_WORKER_MULTIPROC_METHOD', 'LD_LIBRARY_PATH', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'HF_TOKEN'] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) INFO 04-24 06:39:16 [ray_env.py:69] Copying the following environment variables to workers: ['VLLM_ALLOW_INSECURE_SERIALIZATION', 'HF_TOKEN', 'VLLM_ALLREDUCE_USE_SYMM_MEM', 'LD_LIBRARY_PATH', 'VLLM_RAY_PER_WORKER_GPUS', 'VLLM_DISABLE_COMPILE_CACHE', 'VLLM_CACHE_ROOT', 'VLLM_USE_FLASHINFER_SAMPLER', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING', 'VLLM_RAY_BUNDLE_INDICES', 'VLLM_ENABLE_V1_MULTIPROCESSING', 'CUDA_HOME', 'VLLM_WORKER_MULTIPROC_METHOD'] (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151152 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151152 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151152 [0] NCCL INFO Bootstrap: Using ib0:10.128.33.41<0> (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151152 [0] NCCL INFO cudaDriverVersion 13010 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151152 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151152 [0] NCCL INFO Comm config Blocking set to 1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.33.41<0> (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Initialized NET plugin IB (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Assigned NET plugin IB to comm (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Using network IB (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO DMA-BUF is available on GPU device 0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO ncclCommInitRankConfig comm 0x400eb25f55e0 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 901000 commId 0x993b2bbee9ef7c7e - Init START (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Bootstrap timings total 6.524683 (create 0.000027, send 0.000092, recv 0.000167, ring 6.522021, delay 0.000000) (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO comm 0x400eb25f55e0 rank 0 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 00/08 : 0 1 2 3 7 6 5 4 8 9 10 11 15 14 13 12 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 01/08 : 0 4 5 6 7 11 10 9 8 12 13 14 15 3 2 1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 02/08 : 0 3 1 5 4 7 6 10 8 11 9 13 12 15 14 2 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 03/08 : 0 3 2 6 4 7 5 9 8 11 10 14 12 15 13 1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 04/08 : 0 1 2 3 7 6 5 4 8 9 10 11 15 14 13 12 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 05/08 : 0 4 5 6 7 11 10 9 8 12 13 14 15 3 2 1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 06/08 : 0 3 1 5 4 7 6 10 8 11 9 13 12 15 14 2 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 07/08 : 0 3 2 6 4 7 5 9 8 11 10 14 12 15 13 1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] -1/-1/-1->0->3 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->3 [4] 1/-1/-1->0->4 [5] -1/-1/-1->0->3 [6] 1/-1/-1->0->2 [7] 2/-1/-1->0->3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO P2P Chunksize set to 131072 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151480 [0] NCCL INFO [Proxy Service] Device 0 CPU core 1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151482 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 2 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 02/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 03/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 06/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 07/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151486 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 00/0 : 12[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 04/0 : 12[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 01/0 : 0[0] -> 4[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 05/0 : 0[0] -> 4[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151480 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 60. (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 02/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 06/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 07/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 01/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 05/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 04/0 : 0[0] -> 4[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Channel 04/0 : 4[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Connected all trees (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 2 p2p channels per peer (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO CC Off, workFifoBytes 1048576 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO ncclCommInitRankConfig comm 0x400eb25f55e0 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 901000 commId 0x993b2bbee9ef7c7e - Init COMPLETE (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151446 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 16 total 8.70 (kernels 0.23, alloc 1.41, bootstrap 6.52, allgathers 0.02, topo 0.06, graphs 0.01, connections 0.45, rest 0.01) (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151152 [0] NCCL INFO Comm config Blocking set to 1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Assigned NET plugin IB to comm (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Using network IB (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO DMA-BUF is available on GPU device 0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO ncclCommInitRankConfig comm 0x400eb12764b0 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 901000 commId 0xfc2340fcc323d1bc - Init START (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Bootstrap timings total 0.014487 (create 0.000035, send 0.000075, recv 0.013798, ring 0.000376, delay 0.000000) (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO comm 0x400eb12764b0 rank 0 nRanks 4 nNodes 4 localRanks 1 localRank 0 MNNVL 0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 00/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 01/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 02/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 03/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 04/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 05/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 06/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 07/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 08/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 09/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 10/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 11/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 12/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 13/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 14/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 15/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 16/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 17/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 18/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 19/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 20/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 21/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 22/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 23/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 24/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 25/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 26/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 27/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 28/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 29/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 30/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 31/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 32/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 33/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 34/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 35/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 36/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 37/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 38/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 39/40 : 0 1 2 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 2/-1/-1->0->-1 [3] 2/-1/-1->0->-1 [4] 2/-1/-1->0->-1 [5] 2/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 2/-1/-1->0->-1 [9] 2/-1/-1->0->-1 [10] 2/-1/-1->0->-1 [11] 2/-1/-1->0->-1 [12] 2/-1/-1->0->-1 [13] 2/-1/-1->0->-1 [14] 2/-1/-1->0->-1 [15] 2/-1/-1->0->-1 [16] 2/-1/-1->0->-1 [17] 2/-1/-1->0->-1 [18] 2/-1/-1->0->-1 [19] 2/-1/-1->0->-1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 [24] -1/-1/-1->0->1 [25] -1/-1/-1->0->1 [26] -1/-1/-1->0->1 [27] -1/-1/-1->0->1 [28] -1/-1/-1->0->1 [29] -1/-1/-1->0->1 [30] -1/-1/-1->0->1 [31] -1/-1/-1->0->1 [32] -1/-1/-1->0->1 [33] -1/-1/-1->0->1 [34] -1/-1/-1->0->1 [35] -1/-1/-1->0->1 [36] -1/-1/-1->0->1 [37] -1/-1/-1->0->1 [38] -1/-1/-1->0->1 [39] -1/-1/-1->0->1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO P2P Chunksize set to 131072 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151502 [0] NCCL INFO [Proxy Service] Device 0 CPU core 2 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151503 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 4 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151507 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 5 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 00/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 01/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 02/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 03/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 04/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 05/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 06/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 07/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 08/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 09/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 10/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 11/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 12/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 13/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 14/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 15/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 16/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 17/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 18/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 19/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 20/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 21/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 22/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 23/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 24/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 25/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 26/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 27/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 28/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 29/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 30/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 31/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 32/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 33/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 34/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 35/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 36/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 37/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 38/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 39/0 : 3[0] -> 0[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151491 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:415149 (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) INFO 04-24 06:39:16 [ray_env.py:66] RAY_NON_CARRY_OVER_ENV_VARS from config: set() [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) INFO 04-24 06:39:16 [ray_env.py:74] If certain env vars should NOT be copied, add them to /e/home/jusers/feuer1/jupiter/.config/vllm/ray_non_carry_over_env_vars.json file [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=4151157, ip=10.128.33.41) j (FSDPPolicyWorkerBase pid=4151158, ip=10.128.33.41) jpbo-047-09:4151158:4 (FSDPPolicyWorkerBase pid=2109996) jpbo-047-01:2109996:2110423 [0] (FSDPPolicyWorkerBase pid=4151159, ip=10.128.33.41) jpbo-047-09:4151159:4151493 [0] NCCL INFO Chan (FSDPPolicyWorkerBase pid=2109997) jpbo-047-01:2109997:2110424 [0] NCCL INFO Channel 06/0 : 2[3] -> 3[3] [receive] via NE (FSDPPolicyWorkerBase pid=2109998) jpbo-047-01:2109998:2110422 [0] NCCL INFO Channel 08/0 : 2[1] -> 3[1] [recei (FSDPPolicyWorkerBase pid=2370207, ip=10.128.33.48) jpbo-047-16 (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232400) WARNING 04-24 06:39:00 [system_utils.py:37] Overwriting environment variable CUDA_VISIBLE_DEVICES from '0,1,2,3' to '1,2' (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232400) WARNING 04-24 06:39:00 [system_utils.py:37] Overwriting environment variable LD_LIBRARY_PATH from '/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' to '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:04 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) WARNING 04-24 06:39:04 [worker_base.py:307] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232400) INFO 04-24 06:39:05 [parallel_state.py:1234] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:46057 backend=nccl (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:06 [pynccl.py:111] vLLM is using nccl==2.27.7 (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) WARNING 04-24 06:39:00 [system_utils.py:37] Overwriting environment variable CUDA_VISIBLE_DEVICES from '0,1,2,3' to '1,2' (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) WARNING 04-24 06:39:00 [system_utils.py:37] Overwriting environment variable LD_LIBRARY_PATH from '/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' to '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232400) INFO 04-24 06:39:07 [parallel_state.py:1445] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:08 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137... (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:08 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:29 [default_loader.py:293] Loading weights took 20.51 seconds (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232400) INFO 04-24 06:39:04 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232400) WARNING 04-24 06:39:04 [worker_base.py:307] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:05 [parallel_state.py:1234] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:46057 backend=nccl (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:07 [parallel_state.py:1445] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:30 [gpu_model_runner.py:4222] Model loading took 30.59 GiB memory and 21.093942 seconds (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232399) INFO 04-24 06:39:35 [gpu_worker.py:373] Available KV cache memory: 47.96 GiB (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:39:35 [kv_cache_utils.py:1307] GPU KV cache size: 392,864 tokens (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:39:35 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 11.99x (FSDPPolicyWorkerBase pid=2370205, ip=10.128.33.48) jpbo-047-16:2370205:23 (FSDPPolicyWorkerBase pid=2417105, ip=10.128.33.38) jpbo-047-06:2417105:2417504 [0] NCCL INFO Channel 05/0 : 0[1] -> 1[1] [receive] via NET/I (FSDPPolicyWorkerBase pid=2370204, ip=10.128.33.48) jpbo-047-16:2370204:2370605 [0] (FSDPPolicyWorkerBase pid=2109997) T/IB/3 (FSDPPolicyWorkerBase pid=2109997) jpbo-047-01:2109997:2110424 [0] NCCL (FSDPPolicyWorkerBase pid=2109998) ve] via NET/IB/1 (FSDPPolicyWorkerBase pid=2109998) jpbo-047-01:2109998:2110422 (FSDPPolicyWorkerBase pid=2370205, ip=10.128.33.48) jpbo-047-16:2370205:2370606 [0] NCCL INFO Channel 10/0 : 0[1] -> 2 (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) nel 04/0 : 0[0] -> 1[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417506 [0] NCCL INFO Channel 32/0 : 3[0] -> 1[0] [receive] via NET/IB (FSDPPolicyWorkerBase pid=2370207, ip=10.128.33.48) jpbo-047-16:2370207:2370607 [0] NCCL INFO Channel 14/0 (FSDPPolicyWorkerBase pid=2417105, ip=10.128.33.38) B/1 (FSDPPolicyWorkerBase pid=2417105, ip=10.128.33.38) jpbo-047-06:2417105:2417504 [0] NCCL INF (FSDPPolicyWorkerBase pid=2370206, ip=10.128.33.48) 2] -> 2[2] [receive] via NET/IB/2 (FSDPPolicyWorkerBase pid=2370206, ip=10.128.33.48) jpbo-047-1 (FSDPPolicyWorkerBase pid=2417107, ip=10.128.33.38) jpbo-047-06:2417107:2417505 [0] NCCL INFO Ch (FSDPPolicyWorkerBase pid=2109999) T/IB/2 (FSDPPolicyWorkerBase pid=2370204, ip=10.128.33.48) jpbo-047-16:2370204:2370605 [0] NCCL INFO Channel 13/0 : 0[3] -> 2[3] [rece (FSDPPolicyWorkerBase pid=2370205, ip=10.128.33.48) [1] [receive] via NET/IB/1 (FSDPPolicyWorkerBase pid=2370205, ip=10.128.33.48) jpbo-0 (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) /0 (FSDPPolicyWorkerBase pid=2417105, ip=10.128.33.38) O Channel 34/0 : 3[1] -> 1[1] [receive] via NET/IB/1 (FSDPPolicyWorkerBase pid=2417105, ip=10.128.33.38) jpbo-047-06:2417105:24 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080: (FSDPPolicyWorkerBase pid=4151159, ip=10.128.33.41) nel 25/0 : 0[1] -> 1[1] [send] via NET/IB/1 (FSDPPolicyWorkerBase pid=4151159, ip=10.128.33.41) jpbo-047-09:4 (FSDPPolicyWorkerBase pid=2109998) jpbo-047-01:210999 (FSDPPolicyWorkerBase pid=2370207, ip=10.128.33.48) : 0[0] -> 2[0] [receive] via NET/IB/0 (FSDPPolicyWorkerBase pid=2370207, ip=10.128.33.48) jpbo-047-16:2370207:2370607 [0] NCCL INFO Channel 23/0 : 3[0] -> 2[0] [receive] via NET/ (FSDPPolicyWorkerBase pid=2370206, ip=10.128.33.48) jpbo-047-16:2370206:2370604 [0] NCCL INFO C (FSDPPolicyWorkerBase pid=2370204, ip=10.128.33.48) ive] via NET/IB/3 (FSDPPolicyWorkerBase pid=2370204, ip=10.128.33.48) jpbo-047-16:237 (FSDPPolicyWorkerBase pid=2417107, ip=10.128.33.38) annel 32/0 : 3[3] -> 1[3] [receive] via NET/IB/3 (FSDPPolicyWorkerBase pid=2417107, ip=10.128.33.38) jpbo-047-06:2417107: (FSDPPolicyWorkerBase pid=2417108, ip=10.128.33.38) jpbo-047-06:2417108:241750 (FSDPPolicyWorkerBase pid=2109997) INFO Channel 35/0 : 1[3] -> 3[3] [receive] via NET/IB/3 (FSDPPolicyWorkerBase pid=2109997) jpbo-047-01:2109997:21 (FSDPPolicyWorkerBase pid=2109999) jpbo-047-01:2109999:21 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo (FSDPPolicyWorkerBase pid=4151159, ip=10.128.33.41) jpbo-047-09:4151159:4151493 [0] NCCL INFO Connected binomial trees (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) (RayWorkerWrapper pid=2232400) INFO 04-24 06:39:35 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:39:36 [core.py:278] init engine (profile, create kv cache, warmup model) took 6.29 seconds (FSDPPolicyWorkerBase pid=4151157, ip=10.128.33.41) jpbo-047-09:4151157:4151495 [0] NCCL INFO Init timings - ncclCommI (FSDPPolicyWorkerBase pid=4151158, ip=10.128.33.41) jpbo-047-09:4151158:4151489 [0] NCCL INFO ncclCommInitRankConfig comm 0x400e414f5f0 (FSDPPolicyWorkerBase pid=2109996) : 3[0] -> 2[0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=2109996) jpbo-047-01:2109996:2110438 [0] NCCL IN (FSDPPolicyWorkerBase pid=2370207, ip=10.128.33.48) IB/0 (FSDPPolicyWorkerBase pid=2370207, ip=10.128.33.48) jpbo-047-16:2370207:2370621 [0] NCCL INFO Cha (FSDPPolicyWorkerBase pid=2370206, ip=10.128.33.48) hannel 23/0 : 3[2] -> 2[2] [receive] via NET/IB/2 (FSDPPolicyWorkerBase pid=2370206, ip=10.128.33.48) jpbo-047-16:2370206:2370624 [0] NCCL INFO Channel 20/0 : 2[2] -> 3[3] v (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) [0] [send] via NET/IB/0 (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:241752 (FSDPPolicyWorkerBase pid=2370204, ip=10.128.33.48) jpbo-047-16:2370204:2370623 [0] NCC (FSDPPolicyWorkerBase pid=2417105, ip=10.128.33.38) jpbo-047-06:2417105:24 (FSDPPolicyWorkerBase pid=2417107, ip=10.128.33.38) jpbo-047-06:2417107:2417522 [0] NCCL INFO Channel 15/0 : 3[3] -> 0[0] via P (FSDPPolicyWorkerBase pid=2417108, ip=10.128.33.38) jpbo-047-06:2417108:241 (FSDPPolicyWorkerBase pid=2109997) jpbo-047-01:2109997 (FSDPPolicyWorkerBase pid=2109998) jpbo-047-01:2109998:2110439 [0] NCCL INFO (FSDPPolicyWorkerBase pid=2109999) jpbo-047-01:2109999:2110441 (FSDPPolicyWorkerBase pid=2370207, ip=10.128.33.48) nnel 21/24 : 0 2 1 3 (FSDPPolicyWorkerBase pid=2109996) FO Channel 20/24 : 0 2 3 1 (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) WARNING 04-24 06:39:38 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:39:38 [vllm.py:690] Asynchronous scheduling is disabled. (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) WARNING 04-24 06:39:38 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) (EngineCore_DP0 pid=2216202) INFO 04-24 06:39:38 [vllm.py:846] Cudagraph is disabled under eager mode (AsyncVLLMInferenceEngine pid=2215993, ip=10.128.33.42) WARNING 04-24 06:39:38 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=2232721, ip=10.128.33.42) jpbo-047-10:2232721:2232721 [0] NCCL INFO ncclCommInitRank comm 0xaaaaf132b390 rank 1 nranks 2 cudaDev 0 nvmlDev 0 busId 901000 commId 0xd5e8ac5e065ebbfe - Init START (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=2232721, ip=10.128.33.42) jpbo-047-1 (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) jp (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) WARNING 04-24 06:39:10 [custom_all_reduce.py:92] Custom allreduce is disabled because this process group spans across nodes. (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) jpbo-047-07:1714615:1714615 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) jpbo-047-07:1714615:1714615 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) jpbo-047-07:1714615:1714615 [0] NCCL INFO ncclCommInitRank comm 0xaaab0b49e600 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 901000 commId 0xd5e8ac5e065ebbfe - Init START (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) jpbo-047-07:1714615:1714615 [0] NCCL INFO Channel 26/0 : 1[0] -> 0[0] [receive] via NET/IB/0 [repeated 27x across cluster] (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=2232721, ip=10.128.33.42) WARNING 04-24 06:39:10 [custom_all_reduce.py:92] Custom allreduce is disabled because this process group spans across nodes. (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=2232721, ip=10.128.33.42) jpbo-047-10:2232721:2233406 [0] NCCL INFO Channel 27/0 : 1[0] -> 0[0] [send] via NET/ (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=2232721, ip=10.128.33.42) jpbo-047-10:2232721:2233406 [0] NCCL INFO Channel 39/0 : 0[0] -> 1[0] [receive] via NET/IB/0 [repeated 40x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) jpbo-04 (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NC (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) jpbo-047-07:1702991:1715538 [0] NCCL INFO Channel 27/0 : 1[1] -> 0[3] [send] via N (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) jpbo-047-07:1702991:1702991 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET [repeated 34x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) jpbo-047-07:1702991:1702991 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 [repeated 34x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO Bootstrap: Using ib0:10.128.33.42<0> [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO cudaDriverVersion 13010 [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO Comm config Blocking set to 1 [repeated 45x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.  [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.33.42<0> [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO Initialized NET plugin IB [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Assigned NET plugin IB to comm [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Using network IB [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO DMA-BUF is available on GPU device 0 [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaaaf7609700 rank 0 nranks 2 cudaDev 0 nvmlDev 3 busId 3901000 commId 0x7a3727cb8540b9c2 - Init START [repeated 45x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Bootstrap timings total 0.002699 (create 0.000028, send 0.000090, recv 0.002113, ring 0.000051, delay 0.000000) [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Setting affinity for GPU 3 to 216-287 [repeated 49x across cluster] (FSDPPolicyWorkerBase pid=2109999) jpbo-047-01:2109999:2110441 [0] NCCL INFO NVLS multicast support is not available on dev 0 (NVLS_NCHANNELS 0) [repeated 28x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO comm 0xaaaaf7609700 rank 0 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0 [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Channel 39/40 : 0 1 [repeated 376x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 [24] -1/-1/-1->0->1 [25] -1/-1/-1->0->1 [26] -1/-1/-1->0->1 [27] -1/-1/-1->0->1 [28] -1/-1/-1->0->1 [29] -1/-1/-1->0->1 [30] -1/-1/-1->0->1 [31] -1/-1/-1->0->1 [32] -1/-1/-1->0->1 [33] -1/-1/-1->0->1 [34] -1/-1/-1->0->1 [35] -1/-1/-1->0->1 [36] -1/-1/-1->0->1 [37] -1/-1/-1->0->1 [38] -1/-1/-1->0->1 [39] -1/-1/-1->0->1 [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO P2P Chunksize set to 131072 [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.  [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233409 [0] NCCL INFO [Proxy Service] Device 0 CPU core 245 [repeated 49x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233410 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 247 [repeated 49x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417539 [0] NCCL INFO Channel 19/1 : 0[0] -> 2[2] via P2P/IPC [repeated 545x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233411 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 278 [repeated 39x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) jpbo-047-07:1702991:1702991 [0] NCCL INFO Channel 39/0 : 0[3] -> 1[1] [receive] via NET/IB/1 [repeated 2012x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) jpbo-047-07:1702991:1715538 [0] NCCL INFO Channel 26/0 : 1[1] -> 0[3] [send] via NET/IB/1 [repeated 2057x across cluster] (FSDPPolicyWorkerBase pid=2417108, ip=10.128.33.38) jpbo-047-06:2417108:2417491 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 60. [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417520 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 [repeated 35x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417520 [0] NCCL INFO Connected all trees [repeated 34x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417520 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 [repeated 34x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417520 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 16 p2p channels per peer [repeated 34x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417520 [0] NCCL INFO CC Off, workFifoBytes 1048576 [repeated 6x across cluster] (FSDPPolicyWorkerBase pid=2417108, ip=10.128.33.38) jpbo-047-06:2417108:2417463 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. [repeated 15x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417520 [0] NCCL INFO ncclCommInitRankConfig comm 0x400e427d4340 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 901000 commId 0xd132ae93804c20f3 - Init COMPLETE [repeated 32x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417520 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 4 total 0.34 (kernels 0.00, alloc 0.00, bootstrap 0.17, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.14, rest 0.00) [repeated 31x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417506 [0] NCCL INFO Chan (FSDPPolicyWorkerBase pid=2109999) jpbo-047-01:2109999:2110421 [0] NCCL INFO Channel 06/0 : 2[2] -> 3[2] [receive] via NE (FSDPPolicyWorkerBase pid=2109996) jpbo-047-01:2109996:2110423 [0] NCCL INFO Channel 33/0 : 1[0] -> 3[0] [recei (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) WARNING 04-24 06:39:03 [system_utils.py:37] Overwriting environment variable CUDA_VISIBLE_DEVICES from '0,1,2,3' to '2,3' [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) WARNING 04-24 06:39:03 [system_utils.py:37] Overwriting environment variable LD_LIBRARY_PATH from '/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' to '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) INFO 04-24 06:39:08 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) WARNING 04-24 06:39:08 [worker_base.py:307] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) INFO 04-24 06:39:08 [parallel_state.py:1234] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:36759 backend=nccl [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) INFO 04-24 06:39:09 [pynccl.py:111] vLLM is using nccl==2.27.7 [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) INFO 04-24 06:39:10 [parallel_state.py:1445] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) INFO 04-24 06:39:10 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137... [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) INFO 04-24 06:39:11 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) INFO 04-24 06:39:33 [default_loader.py:293] Loading weights took 21.74 seconds [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831914) INFO 04-24 06:39:34 [gpu_model_runner.py:4222] Model loading took 30.59 GiB memory and 22.816749 seconds [repeated 7x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) INFO 04-24 06:39:39 [gpu_worker.py:373] Available KV cache memory: 47.99 GiB [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) INFO 04-24 06:39:40 [kv_cache_utils.py:1307] GPU KV cache size: 392,864 tokens [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) INFO 04-24 06:39:40 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 11.99x [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL  [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=2109996) ve] via NET/IB/0 (FSDPPolicyWorkerBase pid=2109996) jpbo-047-01:2109996:2110456 [0] NCCL INFO Channel 23/1 : 0[0] -> 2 [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=4151157, ip=10.128.33.41) jpbo-047-09:4151157:4151495 [0] NCCL INFO Channel [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=2417108, ip=10.128.33.38) jpbo-047-06:2417108:2417503 [0] NCCL INFO Ch (FSDPPolicyWorkerBase pid=4151158, ip=10.128.33.41) jpbo-047-09:4151158:4151489 [0] NCCL INFO Channel 14/0 : 1[2] -> 0[2] [rece (FSDPPolicyWorkerBase pid=4151157, ip=10.128.33.41) 16/0 : 1[3] -> 0[3] [receive] via NET/IB/3 (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO Channel 26/0 : 1[1] -> 0[3] [receive] via NET/ (FSDPPolicyWorkerBase pid=4151158, ip=10.128.33.41) ive] via NET/IB/2 (FSDPPolicyWorkerBase pid=2417108, ip=10.128.33.38) annel 34/0 : 3[2] -> 1[2] [receive] via NET/IB/2 (FSDPPolicyWorkerBase pid=2109999) INFO Channel 35/0 : 1[2] -> 3[2] [receive] via NET/IB/2 (FSDPPolicyWorkerBase pid=2370205, ip=10.128.33.48) jpbo (FSDPPolicyWorkerBase pid=2109999) jpbo-047-01:2109999:2110421 [0] NCCL INFO Connected binomial trees [repeated 15x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) (RayWorkerWrapper pid=831854) INFO 04-24 06:39:40 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=831169, ip=10.128.33.43) (EngineCore_DP0 pid=831330) INFO 04-24 06:39:41 [core.py:278] init engine (profile, create kv cache, warmup model) took 6.49 seconds [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=2417106, ip=10.128.33.38) jpbo-047-06:2417106:2417539 [0] NCCL INFO Channel 21/1 : 0[0] -> 2[2] v (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) jpbo-047-07:1714615:1715537 [0] NCCL INFO (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) (EngineCore_DP0 pid=1698330) WARNING 04-24 06:39:42 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 6x across cluster] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) (EngineCore_DP0 pid=1698330) INFO 04-24 06:39:42 [vllm.py:690] Asynchronous scheduling is disabled. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) (EngineCore_DP0 pid=1698330) WARNING 04-24 06:39:42 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) (EngineCore_DP0 pid=1698330) INFO 04-24 06:39:42 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=1698245, ip=10.128.33.39) WARNING 04-24 06:39:42 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 5x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO ncclCommInitRank comm 0xaaaadca8ea30 rank 0 nranks 2 cudaDev 0 nvmlDev 3 busId 3901000 commId 0x94afb523beebf273 - Init START [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) WARNING 04-24 06:39:08 [custom_all_reduce.py:92] Custom allreduce is disabled because this process group spans across nodes. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2232505 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Channel 38/0 : 1[1] -> 0[3] [receive] via NET/IB/3 [repeated 39x across cluster] [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706851) WARNING 04-24 06:39:03 [system_utils.py:37] Overwriting environment variable CUDA_VISIBLE_DEVICES from '0,1,2,3' to '0,1' [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706851) WARNING 04-24 06:39:03 [system_utils.py:37] Overwriting environment variable LD_LIBRARY_PATH from '/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' to '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706852) INFO 04-24 06:39:09 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706852) WARNING 04-24 06:39:09 [worker_base.py:307] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706851) INFO 04-24 06:39:10 [parallel_state.py:1234] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:39121 backend=nccl [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706852) INFO 04-24 06:39:10 [pynccl.py:111] vLLM is using nccl==2.27.7 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706851) INFO 04-24 06:39:12 [parallel_state.py:1445] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706852) INFO 04-24 06:39:13 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137... [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706852) INFO 04-24 06:39:14 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706852) INFO 04-24 06:39:38 [default_loader.py:293] Loading weights took 24.36 seconds [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706852) INFO 04-24 06:39:39 [gpu_model_runner.py:4222] Model loading took 30.59 GiB memory and 25.477003 seconds [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) (RayWorkerWrapper pid=1706852) INFO 04-24 06:39:44 [gpu_worker.py:373] Available KV cache memory: 47.96 GiB [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) INFO 04-24 06:39:44 [kv_cache_utils.py:1307] GPU KV cache size: 392,864 tokens [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690356, ip=10.128.33.36) (EngineCore_DP0 pid=1690566) INFO 04-24 06:39:44 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 11.99x [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Bootstrap: Using ib0:10.128.33.37<0> (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO cudaDriverVersion 13010 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.33.37<0> (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Initialized NET plugin IB (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Assigned NET plugin IB to comm (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Using network IB (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO DMA-BUF is available on GPU device 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Bootstrap timings total 0.002899 (create 0.000050, send 0.000101, recv 0.000218, ring 0.000397, delay 0.000001) (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO comm 0xaaab12a1dd60 rank 0 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 00/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 01/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 02/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 03/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 04/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 05/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 06/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 07/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 08/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 09/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 10/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 11/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 12/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 13/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 14/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 15/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 16/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 17/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 18/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 19/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 20/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 21/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 22/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 23/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 24/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 25/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 26/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 27/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 28/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 29/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 30/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 31/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 32/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 33/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 34/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 35/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 36/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 37/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 38/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 39/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 [24] -1/-1/-1->0->1 [25] -1/-1/-1->0->1 [26] -1/-1/-1->0->1 [27] -1/-1/-1->0->1 [28] -1/-1/-1->0->1 [29] -1/-1/-1->0->1 [30] -1/-1/-1->0->1 [31] -1/-1/-1->0->1 [32] -1/-1/-1->0->1 [33] -1/-1/-1->0->1 [34] -1/-1/-1->0->1 [35] -1/-1/-1->0->1 [36] -1/-1/-1->0->1 [37] -1/-1/-1->0->1 [38] -1/-1/-1->0->1 [39] -1/-1/-1->0->1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO P2P Chunksize set to 131072 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379053 [0] NCCL INFO [Proxy Service] Device 0 CPU core 51 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379054 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 28 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379055 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 69 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 00/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 01/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 02/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 03/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 04/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 05/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 06/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 07/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 08/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 09/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 10/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 11/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 12/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 13/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 14/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 15/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 16/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 17/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 18/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 19/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 20/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 21/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 22/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 23/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 24/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 25/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Channel 26/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jp (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 00/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 01/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 02/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 03/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 04/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 05/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 06/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 07/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 08/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 09/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 10/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 11/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 12/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 13/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Channel 14/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Bootstrap: Using ib0:10.128.33.46<0> (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO cudaDriverVersion 13010 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.33.46<0> (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Initialized NET plugin IB (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Assigned NET plugin IB to comm (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Using network IB (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO DMA-BUF is available on GPU device 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Bootstrap timings total 0.236228 (create 0.000037, send 0.000137, recv 0.233538, ring 0.000056, delay 0.000000) (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO comm 0xaaaaec6bd650 rank 1 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] -1/-1/-1->1->0 [7] -1/-1/-1->1->0 [8] -1/-1/-1->1->0 [9] -1/-1/-1->1->0 [10] -1/-1/-1->1->0 [11] -1/-1/-1->1->0 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] -1/-1/-1->1->0 [19] -1/-1/-1->1->0 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 [24] 0/-1/-1->1->-1 [25] 0/-1/-1->1->-1 [26] 0/-1/-1->1->-1 [27] 0/-1/-1->1->-1 [28] 0/-1/-1->1->-1 [29] 0/-1/-1->1->-1 [30] 0/-1/-1->1->-1 [31] 0/-1/-1->1->-1 [32] 0/-1/-1->1->-1 [33] 0/-1/-1->1->-1 [34] 0/-1/-1->1->-1 [35] 0/-1/-1->1->-1 [36] 0/-1/-1->1->-1 [37] 0/-1/-1->1->-1 [38] 0/-1/-1->1->-1 [39] 0/-1/-1->1->-1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO P2P Chunksize set to 131072 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972670 [0] NCCL INFO [Proxy Service] Device 0 CPU core 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972671 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 2 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972672 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 3 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2378244 [0] NCCL INFO Comm config Blocking set to 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Assigned NET plugin IB to comm (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Using network IB (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO DMA-BUF is available on GPU device 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaab2d76fa20 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 901000 commId 0xf87a25994ccf61d3 - Init START (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Bootstrap timings total 0.001121 (create 0.000028, send 0.000097, recv 0.000597, ring 0.000056, delay 0.000000) (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO comm 0xaaab2d76fa20 rank 0 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 00/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 01/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 02/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 03/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 04/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 05/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 06/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 07/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 08/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 09/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 10/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 11/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 12/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 13/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 14/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 15/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 16/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 17/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 18/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 19/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 20/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 21/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 22/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 23/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 24/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 25/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 26/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 27/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 28/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 29/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 30/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 31/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 32/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 33/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 34/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 35/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 36/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 37/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 38/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 39/40 : 0 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1 [24] -1/-1/-1->0->1 [25] -1/-1/-1->0->1 [26] -1/-1/-1->0->1 [27] -1/-1/-1->0->1 [28] -1/-1/-1->0->1 [29] -1/-1/-1->0->1 [30] -1/-1/-1->0->1 [31] -1/-1/-1->0->1 [32] -1/-1/-1->0->1 [33] -1/-1/-1->0->1 [34] -1/-1/-1->0->1 [35] -1/-1/-1->0->1 [36] -1/-1/-1->0->1 [37] -1/-1/-1->0->1 [38] -1/-1/-1->0->1 [39] -1/-1/-1->0->1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO P2P Chunksize set to 131072 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379225 [0] NCCL INFO [Proxy Service] Device 0 CPU core 52 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379226 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 27 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379229 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 37 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 00/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 01/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 02/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 03/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 04/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 05/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 06/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 07/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 08/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 09/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 10/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 11/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 12/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 13/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 14/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 15/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 16/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 17/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 18/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 19/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 20/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 21/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 22/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 23/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 24/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 25/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 26/0 : 1[0] -> 0[0] [send] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 27/0 : 1[0] -> 0[0] [send] via NET/IB/ (AsyncVLLMInferenceEngine pid=1690483, ip=10.128.33.36) (EngineCore_DP0 pid=1690574) (RayWorkerWrapper pid=1695347) INFO 04-24 06:39:44 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690483, ip=10.128.33.36) (EngineCore_DP0 pid=1690574) INFO 04-24 06:39:45 [core.py:278] init engine (profile, create kv cache, warmup model) took 6.55 seconds [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-0 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO Channel 26/0 : 1[1] -> 0[3] [receive] via NET/ (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NC (AsyncVLLMInferenceEngine pid=1690483, ip=10.128.33.36) (EngineCore_DP0 pid=1690574) WARNING 04-24 06:39:47 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690483, ip=10.128.33.36) (EngineCore_DP0 pid=1690574) INFO 04-24 06:39:47 [vllm.py:690] Asynchronous scheduling is disabled. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690483, ip=10.128.33.36) (EngineCore_DP0 pid=1690574) WARNING 04-24 06:39:47 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1690483, ip=10.128.33.36) (EngineCore_DP0 pid=1690574) INFO 04-24 06:39:47 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) WARNING 04-24 06:39:49 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO ncclCommInitRank comm 0xaaab399cebc0 rank 0 nranks 2 cudaDev 0 nvmlDev 3 busId 3901000 commId 0x3bfe8e79642bfe68 - Init START [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) WARNING 04-24 06:39:16 [custom_all_reduce.py:92] Custom allreduce is disabled because this process group spans across nodes. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 39/0 : 0[3] -> 1[1] [receive] via NET/IB/1 [repeated 40x across cluster] [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868654) WARNING 04-24 06:39:11 [system_utils.py:37] Overwriting environment variable CUDA_VISIBLE_DEVICES from '0,1,2,3' to '2,3' [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868654) WARNING 04-24 06:39:11 [system_utils.py:37] Overwriting environment variable LD_LIBRARY_PATH from '/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' to '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868654) INFO 04-24 06:39:16 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868654) WARNING 04-24 06:39:16 [worker_base.py:307] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868654) INFO 04-24 06:39:17 [parallel_state.py:1234] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:39993 backend=nccl [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868681) INFO 04-24 06:39:17 [pynccl.py:111] vLLM is using nccl==2.27.7 [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868654) INFO 04-24 06:39:19 [parallel_state.py:1445] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A [repeated 20x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868681) INFO 04-24 06:39:20 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137... [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868681) INFO 04-24 06:39:20 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868681) INFO 04-24 06:39:43 [default_loader.py:293] Loading weights took 21.78 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) (RayWorkerWrapper pid=3868681) INFO 04-24 06:39:43 [gpu_model_runner.py:4222] Model loading took 30.59 GiB memory and 22.829386 seconds [repeated 12x across cluster] (AsyncVLLMInferenceEngine pid=3867661, ip=10.128.33.34) (EngineCore_DP0 pid=3867869) (RayWorkerWrapper pid=3884173) INFO 04-24 06:39:48 [gpu_worker.py:373] Available KV cache memory: 47.96 GiB [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) INFO 04-24 06:39:48 [kv_cache_utils.py:1307] GPU KV cache size: 392,864 tokens [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=3867788, ip=10.128.33.34) (EngineCore_DP0 pid=3867877) INFO 04-24 06:39:48 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 11.99x [repeated 10x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2362727 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2362727 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO Bootstrap: Using ib0:10.128.33.46<0> [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO cudaDriverVersion 13010 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO NCCL version 2.27.7+cuda13.0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so.  [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [RO]; OOB ib0:10.128.33.46<0> [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO Initialized NET plugin IB [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Assigned NET plugin IB to comm [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Using network IB [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO DMA-BUF is available on GPU device 0 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Bootstrap timings total 0.000624 (create 0.000027, send 0.000099, recv 0.000258, ring 0.000072, delay 0.000000) [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to LOC [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Setting affinity for GPU 1 to 72-143 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO comm 0xaaab24408ae0 rank 1 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 39/40 : 0 1 [repeated 80x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] -1/-1/-1->1->0 [7] -1/-1/-1->1->0 [8] -1/-1/-1->1->0 [9] -1/-1/-1->1->0 [10] -1/-1/-1->1->0 [11] -1/-1/-1->1->0 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] -1/-1/-1->1->0 [19] -1/-1/-1->1->0 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 [24] 0/-1/-1->1->-1 [25] 0/-1/-1->1->-1 [26] 0/-1/-1->1->-1 [27] 0/-1/-1->1->-1 [28] 0/-1/-1->1->-1 [29] 0/-1/-1->1->-1 [30] 0/-1/-1->1->-1 [31] 0/-1/-1->1->-1 [32] 0/-1/-1->1->-1 [33] 0/-1/-1->1->-1 [34] 0/-1/-1->1->-1 [35] 0/-1/-1->1->-1 [36] 0/-1/-1->1->-1 [37] 0/-1/-1->1->-1 [38] 0/-1/-1->1->-1 [39] 0/-1/-1->1->-1 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO P2P Chunksize set to 131072 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.  [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379227 [0] NCCL INFO [Proxy Service] Device 0 CPU core 74 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379228 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 108 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379230 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 135 [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2362727 [0] NCCL INFO Channel 39/0 : 0[3] -> 1[1] [receive] via NET/IB/1 [repeated 40x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 27/0 : 1[1] -> 0[3] [send] via [repeated 43x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO NCCL_SOCKET_FAMILY set by environment to AF_INET [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1956420 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2362727 [0] NCCL INFO Comm config Blocking set to 1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaab24408ae0 rank 1 nranks 2 cudaDev 0 nvmlDev 1 busId 1901000 commId 0xc0090a9fe9957abd - Init START (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) (RayWorkerWrapper pid=1994958) INFO 04-24 06:39:52 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) INFO 04-24 06:39:53 [core.py:278] init engine (profile, create kv cache, warmup model) took 6.19 seconds [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) WARNING 04-24 06:39:54 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) INFO 04-24 06:39:54 [vllm.py:690] Asynchronous scheduling is disabled. [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) WARNING 04-24 06:39:54 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) (EngineCore_DP0 pid=1994319) INFO 04-24 06:39:54 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 11x across cluster] (AsyncVLLMInferenceEngine pid=1994108, ip=10.128.33.45) WARNING 04-24 06:39:54 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 5x across cluster] (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Loading model from /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints/global_step_18/policy/model_world_size_16_rank_0.pt (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Loading extra_state from /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints/global_step_18/policy/extra_state_world_size_16_rank_0.pt (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Loading optim from /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints/global_step_18/policy/optim_world_size_16_rank_0.pt (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723126) WARNING 04-24 06:39:12 [system_utils.py:37] Overwriting environment variable CUDA_VISIBLE_DEVICES from '0,1,2,3' to '2,3' [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723126) WARNING 04-24 06:39:12 [system_utils.py:37] Overwriting environment variable LD_LIBRARY_PATH from '/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' to '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723126) INFO 04-24 06:39:17 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723126) WARNING 04-24 06:39:17 [worker_base.py:307] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723126) INFO 04-24 06:39:18 [parallel_state.py:1234] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:56411 backend=nccl [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723127) INFO 04-24 06:39:19 [pynccl.py:111] vLLM is using nccl==2.27.7 [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723126) INFO 04-24 06:39:20 [parallel_state.py:1445] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A [repeated 8x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723127) INFO 04-24 06:39:21 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137... [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723127) INFO 04-24 06:39:22 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723127) INFO 04-24 06:39:49 [default_loader.py:293] Loading weights took 26.10 seconds [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) (RayWorkerWrapper pid=1723127) INFO 04-24 06:39:49 [gpu_model_runner.py:4222] Model loading took 30.59 GiB memory and 27.341083 seconds [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1706685, ip=10.128.33.35) (EngineCore_DP0 pid=1706902) (RayWorkerWrapper pid=1723202) INFO 04-24 06:39:54 [gpu_worker.py:373] Available KV cache memory: 47.96 GiB [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) INFO 04-24 06:39:55 [kv_cache_utils.py:1307] GPU KV cache size: 392,864 tokens [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1706814, ip=10.128.33.35) (EngineCore_DP0 pid=1706894) INFO 04-24 06:39:55 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 11.99x [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688118) INFO 04-24 06:39:57 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) INFO 04-24 06:39:58 [core.py:278] init engine (profile, create kv cache, warmup model) took 6.75 seconds [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) WARNING 04-24 06:40:00 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) INFO 04-24 06:40:00 [vllm.py:690] Asynchronous scheduling is disabled. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) WARNING 04-24 06:40:00 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) INFO 04-24 06:40:00 [vllm.py:846] Cudagraph is disabled under eager mode [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1706685, ip=10.128.33.35) WARNING 04-24 06:39:57 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster] (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Successfully loaded model state dict (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Successfully loaded optimizer state (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Successfully loaded scheduler state (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688123) WARNING 04-24 06:39:16 [system_utils.py:37] Overwriting environment variable CUDA_VISIBLE_DEVICES from '0,1,2,3' to '0,1' [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688123) WARNING 04-24 06:39:16 [system_utils.py:37] Overwriting environment variable LD_LIBRARY_PATH from '/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' to '/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/e/scratch/jureap59/feuer1/OpenThoughts-Agent/envs/rl/lib/python3.12/site-packages/cv2/../../lib64:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real/:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nvshmem/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/comm_libs/13.0/nccl/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/math_libs/13.0/targets/sbsa-linux/lib:/e/software/default/stages/2026/software/nvidia-compilers/25.9-CUDA-13/Linux_aarch64/25.9/compilers/lib:/e/software/default/stages/2026/software/numactl/2.0.19-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/GCCcore/14.3.0/lib64:/e/software/default/stages/2026/software/CUDA/13/nvvm/lib64:/e/software/default/stages/2026/software/CUDA/13/extras/CUPTI/lib64:/e/software/default/stages/2026/software/CUDA/13/targets/sbsa-linux/lib:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/.singularity.d/libs:/usr/local/cuda/compat/lib.real:/usr/local/cuda-13/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda-13/compat/lib.real:/e/software/default/stages/2026/software/binutils/2.44-GCCcore-14.3.0/lib:/e/software/default/stages/2026/software/zlib/1.3.1-GCCcore-14.3.0/lib' [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688118) INFO 04-24 06:39:21 [worker_base.py:289] Injected into for extended collective_rpc calls ['_apply_fp8_weight_loader_patches', '_is_fp8_model', '_quantize_weights_for_fp8', '_restore_param_subclasses', '_undo_param_subclasses', 'begin_weight_update', 'destroy_weights_update_group', 'end_weight_update', 'init_weight_update_communicator', 'load_weights', 'set_numa_affinity', 'test_rpc'] [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688118) WARNING 04-24 06:39:21 [worker_base.py:307] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688123) INFO 04-24 06:39:22 [parallel_state.py:1234] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:46189 backend=nccl [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688118) INFO 04-24 06:39:22 [pynccl.py:111] vLLM is using nccl==2.27.7 [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688123) INFO 04-24 06:39:24 [parallel_state.py:1445] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank N/A [repeated 4x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688118) INFO 04-24 06:39:25 [gpu_model_runner.py:4125] Starting to load model /e/data1/datasets/playground/ot-baf/hf_hub/models--Qwen--Qwen3-32B/snapshots/9216db5781bf21249d130ec9da846c4624c16137... [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688118) INFO 04-24 06:39:26 [cuda.py:367] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688118) INFO 04-24 06:39:51 [default_loader.py:293] Loading weights took 24.70 seconds [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) (RayWorkerWrapper pid=1688118) INFO 04-24 06:39:51 [gpu_model_runner.py:4222] Model loading took 30.59 GiB memory and 25.752638 seconds [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) (RayWorkerWrapper pid=1688122) INFO 04-24 06:39:57 [gpu_worker.py:373] Available KV cache memory: 47.96 GiB [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) INFO 04-24 06:39:57 [kv_cache_utils.py:1307] GPU KV cache size: 392,864 tokens [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671655, ip=10.128.33.47) (EngineCore_DP0 pid=1671863) INFO 04-24 06:39:57 [kv_cache_utils.py:1312] Maximum concurrency for 32,768 tokens per request: 11.99x [repeated 2x across cluster] (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) (RayWorkerWrapper pid=1688122) INFO 04-24 06:39:57 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) INFO 04-24 06:39:58 [core.py:278] init engine (profile, create kv cache, warmup model) took 6.82 seconds (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) WARNING 04-24 06:40:00 [serial_utils.py:57] Allowing insecure serialization using pickle due to VLLM_ALLOW_INSECURE_SERIALIZATION=1 (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) INFO 04-24 06:40:00 [vllm.py:690] Asynchronous scheduling is disabled. (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) WARNING 04-24 06:40:00 [vllm.py:735] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) (EngineCore_DP0 pid=1671871) INFO 04-24 06:40:00 [vllm.py:846] Cudagraph is disabled under eager mode (AsyncVLLMInferenceEngine pid=1671783, ip=10.128.33.47) WARNING 04-24 06:40:00 [model.py:1350] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 2x across cluster] (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Checkpoint loaded successfully from /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints/global_step_18/policy (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) INFO Channel 18/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO Channel 19/0 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO Channel 08/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO Channel 09/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO Channel 20/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO Channel 21/0 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO Connected all trees (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151521 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 16 p2p channels per peer (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO CC Off, workFifoBytes 1048576 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO ncclCommInitRankConfig comm 0x400eb3980500 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 901000 commId 0x9ec23970e326a807 - Init COMPLETE (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151509 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 4 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.16, allgathers 0.00, topo 0.02, graphs 0.01, connections 0.14, rest 0.00) (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 01/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 03/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 05/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 07/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 09/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 11/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 13/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 15/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 17/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 19/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 21/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 23/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 25/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 27/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 29/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 31/1 : 0[0] -> 1[1] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 01/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 03/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 05/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 07/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 09/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 11/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 13/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 15/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 17/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 19/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 21/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 23/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 25/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 27/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 29/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 31/1 : 0[0] -> 2[2] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 00/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 02/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 04/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 06/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 08/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 10/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 12/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 14/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 16/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 18/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 20/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 22/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 24/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 26/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 28/1 : 0[0] -> 3[3] via P2P/IPC (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) jpbo-047-09:4151080:4151528 [0] NCCL INFO Channel 30/1 : 0[0] -> 3[3] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) Channel 39/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Connected binomial trees (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1961339 [0] NCCL INFO Comm config Blocking set to 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Assigned NET plugin IB to comm (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Using network IB (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO DMA-BUF is available on GPU device 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaab073f4ce0 rank 1 nranks 2 cudaDev 0 nvmlDev 0 busId 901000 commId 0xf87a25994ccf61d3 - Init START (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Bootstrap timings total 0.000787 (create 0.000061, send 0.000182, recv 0.000179, ring 0.000109, delay 0.000000) (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Setting affinity for GPU 0 to 0-71 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO comm 0xaaab073f4ce0 rank 1 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] -1/-1/-1->1->0 [7] -1/-1/-1->1->0 [8] -1/-1/-1->1->0 [9] -1/-1/-1->1->0 [10] -1/-1/-1->1->0 [11] -1/-1/-1->1->0 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] -1/-1/-1->1->0 [19] -1/-1/-1->1->0 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 [24] 0/-1/-1->1->-1 [25] 0/-1/-1->1->-1 [26] 0/-1/-1->1->-1 [27] 0/-1/-1->1->-1 [28] 0/-1/-1->1->-1 [29] 0/-1/-1->1->-1 [30] 0/-1/-1->1->-1 [31] 0/-1/-1->1->-1 [32] 0/-1/-1->1->-1 [33] 0/-1/-1->1->-1 [34] 0/-1/-1->1->-1 [35] 0/-1/-1->1->-1 [36] 0/-1/-1->1->-1 [37] 0/-1/-1->1->-1 [38] 0/-1/-1->1->-1 [39] 0/-1/-1->1->-1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO P2P Chunksize set to 131072 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972827 [0] NCCL INFO [Proxy Service] Device 0 CPU core 1 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972828 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 2 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972824 [0] NCCL INFO Channel 39/0 : 0[0] -> 1[0] [receive] via NET/IB/0 [repeated 40x across cluster] (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Channel 39/0 : 0[0] -> 1[0] [send] via NET/IB/0 [repeated 40x across cluster] (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) INFO 04-24 06:39:47 [gpu_worker.py:373] Available KV cache memory: 47.99 GiB (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) INFO 04-24 06:39:47 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) 0 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) INFO 04-24 06:40:33 [block_pool.py:452] Successfully reset prefix cache (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Connected all trees (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=1961339, ip=10.128.33.46) jpbo-047-14:1961339:1972831 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 4 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO 40 coll channels, 40 collnet channels, 0 nvls channels, 64 p2p channels, 2 p2p channels per peer (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO CC Off, workFifoBytes 1048576 (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO ncclCommInitRankConfig comm 0xaaab2d76fa20 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 901000 commId 0xf87a25994ccf61d3 - Init COMPLETE (AsyncVLLMInferenceEngine pid=2361797, ip=10.128.33.37) (EngineCore_DP0 pid=2362007) (RayWorkerWrapper pid=2378244) jpbo-047-05:2378244:2379223 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.33 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.04, graphs 0.00, connections 0.29, rest 0.00) (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Channel 00/0 : 1[3] -> 0[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Channel 01/0 : 1[3] -> 0[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Channel 02/0 : 1[3] -> 0[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Channel 03/0 : 1[3] -> 0[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Channel 04/0 : 1[3] -> 0[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Channel 05/0 : 1[3] -> 0[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Channel 06/0 : 1[3] -> 0[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Channel 07/0 : 1[3] -> 0[2] via P2P/IPC (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Connected all trees (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379220 [1] NCCL INFO [Proxy Progress] Device 1 CPU core 220 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 8 p2p channels per peer (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO ncclCommInitRankConfig comm 0xaaaaf496ae50 rank 1 nranks 2 cudaDev 1 nvmlDev 3 busId 3901000 commId 0x56b155e92217a124 - Init COMPLETE (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378344) jpbo-047-05:2378344:2379215 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.03 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.02, rest 0.00) (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Channel 00/08 : 0 1 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Channel 01/08 : 0 1 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Channel 02/08 : 0 1 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Channel 03/08 : 0 1 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Channel 04/08 : 0 1 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Channel 05/08 : 0 1 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Channel 06/08 : 0 1 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Channel 07/08 : 0 1 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 (AsyncVLLMInferenceEngine pid=2361929, ip=10.128.33.37) (EngineCore_DP0 pid=2362015) (RayWorkerWrapper pid=2378339) jpbo-047-05:2378339:2379214 [0] NCCL INFO CC Off, workFifoBytes 1048576 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 28/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 29/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 30/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 31/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 32/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 33/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 34/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 35/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 36/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 37/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 38/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=2362727, ip=10.128.33.37) jpbo-047-05:2362727:2379224 [0] NCCL INFO Channel 39/0 : 1[1] -> 0[3] [send] via NET/IB/1 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) CL INFO Channel 39/0 : 1[1] -> 0[3] [receive] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 00/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 01/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 02/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 03/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 04/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 05/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 06/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 07/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 08/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 09/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 10/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 11/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 12/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 13/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 14/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 15/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 16/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 17/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 18/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 19/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 20/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 21/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 22/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 23/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 24/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 25/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 26/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 27/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 28/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 29/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 30/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 31/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 32/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 33/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 34/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 35/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 36/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 37/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 38/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1955530, ip=10.128.33.46) (EngineCore_DP0 pid=1955613) (RayWorkerWrapper pid=1956420) jpbo-047-14:1956420:1972826 [0] NCCL INFO Channel 39/0 : 0[3] -> 1[1] [send] via NET/IB/3 (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=2232721, ip=10.128.33.42) IB/0 (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) ET/IB/1 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Saving model to /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints/global_step_19/policy/model_world_size_16_rank_0.pt (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) Channel 39/0 : 1[0] -> 0[0] [receive] via NET/IB/0 (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Connected binomial trees [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2006411 [1] NCCL INFO Comm config Blocking set to 1 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Assigned NET plugin IB to comm [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Using network IB [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO DMA-BUF is available on GPU device 1 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO ncclCommInitRankConfig comm 0xaaab4dbdfb80 rank 1 nranks 2 cudaDev 1 nvmlDev 3 busId 3901000 commId 0x1cea5b0b33ddab3 - Init START [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Bootstrap timings total 0.000316 (create 0.000029, send 0.000076, recv 0.000103, ring 0.000010, delay 0.000000) [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Setting affinity for GPU 3 to 216-287 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO comm 0xaaab4dbdfb80 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] 0/-1/-1->1->-1 [3] 0/-1/-1->1->-1 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO P2P Chunksize set to 524288 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007344 [1] NCCL INFO [Proxy Service] Device 1 CPU core 255 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007345 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 258 [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) jpbo-047-07:1702991:1715538 [0] NCCL INFO Channel 39/0 : 0[3] -> 1[1] [receive] via NET/IB/1 [repeated 40x across cluster] [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) jpbo-047-10:2232505:2233405 [0] NCCL INFO Channel 39/0 : 0[3] -> 1[1] [send] via NET/IB/3 [repeated 40x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=1702991, ip=10.128.33.39) INFO 04-24 06:39:40 [gpu_worker.py:373] Available KV cache memory: 47.99 GiB [repeated 3x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006410) INFO 04-24 06:39:46 [kernel_warmup.py:44] Skipping FlashInfer autotune because it is disabled. [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) INFO 04-24 06:40:33 [block_pool.py:452] Successfully reset prefix cache [repeated 23x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Channel 07/0 : 1[3] -> 0[2] via P2P/IPC [repeated 152x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Connected all trees [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007351 [1] NCCL INFO [Proxy Progress] Device 1 CPU core 267 [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO 8 coll channels, 8 collnet channels, 0 nvls channels, 8 p2p channels, 8 p2p channels per peer [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO ncclCommInitRankConfig comm 0xaaab4dbdfb80 rank 1 nranks 2 cudaDev 1 nvmlDev 3 busId 3901000 commId 0x1cea5b0b33ddab3 - Init COMPLETE [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006411) jpbo-047-08:2006411:2007339 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.04 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.03, rest 0.00) [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006410) jpbo-047-08:2006410:2007338 [0] NCCL INFO Channel 07/08 : 0 1 [repeated 152x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006410) jpbo-047-08:2006410:2007338 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 [repeated 19x across cluster] (AsyncVLLMInferenceEngine pid=1990057, ip=10.128.33.40) (EngineCore_DP0 pid=1990138) (RayWorkerWrapper pid=2006410) jpbo-047-08:2006410:2007338 [0] NCCL INFO CC Off, workFifoBytes 1048576 [repeated 22x across cluster] (AsyncVLLMInferenceEngine pid=1698116, ip=10.128.33.39) (EngineCore_DP0 pid=1698322) (RayWorkerWrapper pid=1714615) jpbo-047-07:1714615:1715537 [0] NCCL INFO Channel 39/0 : 0[0] -> 1[0] [send] via NET/IB/0 [repeated 52x across cluster] (AsyncVLLMInferenceEngine pid=2216121, ip=10.128.33.42) (EngineCore_DP0 pid=2216210) (RayWorkerWrapper pid=2232505) CL INFO Channel 39/0 : 1[1] -> 0[3] [receive] via NET/IB/3 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Saving optim to /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints/global_step_19/policy/optim_world_size_16_rank_0.pt (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Saving extra_state to /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints/global_step_19/policy/extra_state_world_size_16_rank_0.pt (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Checkpoint saved to /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/checkpoints/global_step_19/policy (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Created output directory: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/exports/global_step_19/policy (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Detected FSDP version: 2 (FSDPPolicyWorkerBase pid=4151080, ip=10.128.33.41) [rank-0]: Successfully saved model to /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2/exports/global_step_19/policy Stopping Ray cluster... Ray cluster stopped [RLJobRunner] Launching trace upload (training exit code: 0): repo_id: DCAgent/a2-rl-stack_jest_v2 job_dir: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/a2-rl-stack_jest_v2 episodes: last log: /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/logs/a2-rl-stack_jest_v2_trace_upload.log [RLJobRunner] Waiting for trace upload to complete... [RLJobRunner] Trace upload failed with exit code 1. Preserving Ray logs to /e/data1/datasets/playground/ot-baf/a2-rl-stack_jest_v2/ray_logs/ Collecting Ray logs from worker jpbo-047-02... Collecting Ray logs from worker jpbo-047-03... Collecting Ray logs from worker jpbo-047-04... Collecting Ray logs from worker jpbo-047-05... Collecting Ray logs from worker jpbo-047-06... Collecting Ray logs from worker jpbo-047-07... Collecting Ray logs from worker jpbo-047-08... Collecting Ray logs from worker jpbo-047-09... Collecting Ray logs from worker jpbo-047-10... Collecting Ray logs from worker jpbo-047-11... Collecting Ray logs from worker jpbo-047-12... Collecting Ray logs from worker jpbo-047-13... Collecting Ray logs from worker jpbo-047-14... Collecting Ray logs from worker jpbo-047-15... Collecting Ray logs from worker jpbo-047-16... Ray log preservation complete