Instructions to use 01-ai/Yi-1.5-6B-Chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use 01-ai/Yi-1.5-6B-Chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="01-ai/Yi-1.5-6B-Chat")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("01-ai/Yi-1.5-6B-Chat")
model = AutoModelForCausalLM.from_pretrained("01-ai/Yi-1.5-6B-Chat", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use 01-ai/Yi-1.5-6B-Chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "01-ai/Yi-1.5-6B-Chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "01-ai/Yi-1.5-6B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/01-ai/Yi-1.5-6B-Chat

SGLang

How to use 01-ai/Yi-1.5-6B-Chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "01-ai/Yi-1.5-6B-Chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "01-ai/Yi-1.5-6B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "01-ai/Yi-1.5-6B-Chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "01-ai/Yi-1.5-6B-Chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use 01-ai/Yi-1.5-6B-Chat with Docker Model Runner:
```
docker model run hf.co/01-ai/Yi-1.5-6B-Chat
```

Alternate quantizations

by ZeroWw - opened Jun 25, 2024

Discussion

ZeroWw

Jun 25, 2024

•

edited Jul 23, 2024

These are my own quantizations (updated almost daily).

The difference with normal quantizations is that I quantize the output and embed tensors to f16.
and the other tensors to 15_k,q6_k or q8_0.
This creates models that are little or not degraded at all and have a smaller size. They run at about 3-6 t/sec on CPU only using llama.cpp
And obviously faster on computers with potent GPUs

https://huggingface.co/ZeroWw/Yi-1.5-6B-Chat-GGUF

lorinma

01-ai org Jul 1, 2024

Thank you ZeroWw, That's a fresh perspective for me! Did you write your own tool for this purpose?
Also, by little or no degrade, did you run a quantitative eval or just subject evaluation?

ZeroWw

Jul 1, 2024

Thank you ZeroWw, That's a fresh perspective for me! Did you write your own tool for this purpose?
Also, by little or no degrade, did you run a quantitative eval or just subject evaluation?

Subject evaluation, I don't have the resource to do much (but you are welcome to offer them to me)

I use the normal quantizing tool I just set the output and embed tensors to f16 and quantized everything else.
That's because the output and embed tensors are the ones responsible for the main "understanding and expressing".
Most quantizations quantize everything in the same way and that's a mistake in my opinion.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment