# I have a question about multi-GPU inference

**URL:** <https://discuss.huggingface.co/t/i-have-a-question-about-multi-gpu-inference/33384>\
**Category:** DeepSpeed\
**Created:** [March 9, 2023, 1:51am UTC](https://discuss.huggingface.co/t/i-have-a-question-about-multi-gpu-inference/33384 "2023-03-09T01:51:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![gaoxt1983](https://avatars.discourse-cdn.com/v4/letter/g/b4bc9f/32.png) [@gaoxt1983](https://discuss.huggingface.co/u/gaoxt1983)\
**Post date:** [March 9, 2023, 1:51am UTC](https://discuss.huggingface.co/t/i-have-a-question-about-multi-gpu-inference/33384/1 "2023-03-09T01:51:24Z")

</div>

In the inference tutorial: [Getting Started with DeepSpeed for Inferencing Transformer based Models - DeepSpeed](https://www.deepspeed.ai/tutorials/inference-tutorial/) , for this example:

```auto
# Filename: gpt-neo-2.7b-generation.py
import os
import deepspeed
import torch
from transformers import pipeline

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))
generator = pipeline('text-generation', model='EleutherAI/gpt-neo-2.7B',
                     device=local_rank)

generator.model = deepspeed.init_inference(generator.model,
                                           mp_size=world_size,
                                           dtype=torch.float,
                                           replace_with_kernel_inject=True)

string = generator("DeepSpeed is", do_sample=True, min_length=50)
if not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0:
    print(string)

```

I want to know:

1. if I’m using deepspeed --num\_gpus 2 gpt-neo-2.7b-generation.py, the “generator” statement runs once or twice?
2. should I do something necessary for different rank of machine?