# Sagemaker Huggingface zero shot classification with candidate\_labels in predict method

**URL:** <https://discuss.huggingface.co/t/sagemaker-huggingface-zero-shot-classification-with-candidate-labels-in-predict-method/19194>\
**Category:** Amazon SageMaker\
**Created:** [June 15, 2022, 1:34pm UTC](https://discuss.huggingface.co/t/sagemaker-huggingface-zero-shot-classification-with-candidate-labels-in-predict-method/19194 "2022-06-15T13:34:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dineshmane](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@dineshmane](https://discuss.huggingface.co/u/dineshmane)\
**Post date:** [June 15, 2022, 1:34pm UTC](https://discuss.huggingface.co/t/sagemaker-huggingface-zero-shot-classification-with-candidate-labels-in-predict-method/19194/1 "2022-06-15T13:34:40Z")

</div>

Hi,  
I am using HF for zero shot classification for doc classification. I successfully implemented code with native HF library. After working code I deploy the model using SM, but I am not able to find anywhere how to pass the custom labels to the predict method?

Using HF library:

```auto
classifier = pipeline("zero-shot-classification")
a = classifier(
    "Sample Vaccination Record Card Mary Major M Last Name First Name MI 1/6/58 012345abcd67 blah blah blah….",
    candidate_labels=["Medical Card", "ID card", "Passport" ,"education", "politics", "business"],
)

```

I got the prediction score for each candidate label. So all good until here 🙂

Now I deployed the model using our SM:

```auto
hub_v1 = {
	'HF_MODEL_ID':'cross-encoder/nli-distilroberta-base',
	'HF_TASK':'text-classification'
}
huggingface_model_v1 = HuggingFaceModel(
	transformers_version='4.17.0',
	pytorch_version='1.10.2',
	py_version='py38',
	env=hub_v1,
	role=role, 
)
serverless_config_v1 = ServerlessInferenceConfig(
    memory_size_in_mb=4096, max_concurrency=10,
)
predictor_v1 = huggingface_model_v1.deploy(
    serverless_inference_config=serverless_config_v1
)

```

I called the endpoint like this

```auto
predictor_v1.predict({
	'inputs': "Sample Vaccination Record Card Mary Major M Last Name First Name MI 1/6/58 012345abcd67 blah blah blah…",
    'candidate_labels' : ['WILL', 'ID', 'P60', 'OTHER']
})

```

i get output like this:

`[{'label': 'ENTAILMENT', 'score': 0.5859580636024475}]`

I need the prediction score for the input string per each of the four categories.  
Any idea? Am i missing something here? Appreciate your help

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [June 16, 2022, 7:04am UTC](https://discuss.huggingface.co/t/sagemaker-huggingface-zero-shot-classification-with-candidate-labels-in-predict-method/19194/2 "2022-06-16T07:04:11Z")

</div>

@dineshmane you deploy your model on sagemaker with the task `text-classification` and in the pipeline you use `zero-shot-classification`. If you change your sagemaker task it should work.

Also you payload was a bit off. An example of a `zero-shot-classifcation` payload can be found in the documentation and I also added a json below. [Reference](https://huggingface.co/docs/sagemaker/reference#inference-toolkit-api)

```json
{
  "inputs": "Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!",
  "parameters": {
    "candidate_labels": ["refund", "legal", "faq"]
  }
}

```

---

<div class="post-metadata">

**Author:** ![dineshmane](https://avatars.discourse-cdn.com/v4/letter/d/e495f1/32.png) [@dineshmane](https://discuss.huggingface.co/u/dineshmane)\
**Post date:** [June 16, 2022, 2:17pm UTC](https://discuss.huggingface.co/t/sagemaker-huggingface-zero-shot-classification-with-candidate-labels-in-predict-method/19194/3 "2022-06-16T14:17:24Z")

</div>

You the best @philschmid ! Thank you 🙂  
It is working now.  
working code base

```auto

from sagemaker.huggingface.model import HuggingFaceModel
from sagemaker.serverless import ServerlessInferenceConfig

# Hub Model configuration. <https://huggingface.co/models>
hub_v1 = {
	'HF_MODEL_ID':'cross-encoder/nli-distilroberta-base',
	'HF_TASK':'zero-shot-classification'
}

# create Hugging Face Model Class
huggingface_model_v1 = HuggingFaceModel(
	transformers_version='4.17.0',
	pytorch_version='1.10.2',
	py_version='py38',
	env=hub_v1,
	role=role, 
)

# Specify MemorySizeInMB and MaxConcurrency in the serverless config object
serverless_config_v1 = ServerlessInferenceConfig(
    memory_size_in_mb=4096, max_concurrency=10,
)

# deploy the endpoint endpoint
predictor_v1 = huggingface_model_v1.deploy(
    serverless_inference_config=serverless_config_v1
)

predictor_v1.predict({
	'inputs': " 
Sample Vaccination Record Card Mary Major M Last Name First Name MI 1/6/58 012345abcd67 blah blah ",
    "parameters":{'candidate_labels' : label_list}
})

```

output was like below:

```auto

{'sequence': ' Sample Vaccination Record Card Mary Major M Last Name First Name MI 1/6/58 012345abcd67 blah blah ',
 'labels': ['RENT',
  'Medical_Document',
  'p60 form',
  'uk_driving_licence',
  'WILL',
  'passport'],
 'scores': [0.27988865971565247,
  0.17847613990306854,
  0.16656287014484406,
  0.15448175370693207,
  0.125006765127182,
  0.09558377414941788]}

```