Instructions to use TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ")
model = AutoModelForMultimodalLM.from_pretrained("TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ

SGLang

How to use TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ with Docker Model Runner:
```
docker model run hf.co/TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ
```

Swift-Qwen3.8-27B-W4A16-AWQ

W4A16 (4-bit weights, 16-bit activations) AWQ compressed-tensors quantization of ukisai/Swift-Qwen3.8-27b.

Quantization method

Scheme: W4A16_ASYM — 4-bit asymmetric per-group quantization (group size 128) of all Linear weights, stored in the compressed-tensors pack-quantized format (weight_packed / weight_scale / weight_zero_point / weight_shape), which LMDeploy turbomind auto-detects and loads natively (including the MTP head and vision tower, which stay BF16).
Tooling: llmcompressor one-shot offline quantization with CPU offloading (compressed_tensors.offload.load_offloaded_model), so the full-precision source fits on a 2×16 GB VRAM setup.
AWQ activation smoothing: AWQModifier with the layer-scoped hybrid-attention mappings from build_hybrid_attention_mappings — full-attention input_layernorm → self_attn.q/k/v, post_attention_layernorm → mlp.gate/up, and mlp.up_proj → mlp.down_proj, with duo_scaling="both" and CPU offload, followed by W4A16 quantization. This layer-scoped recipe is required for hybrid-attention (Qwen3.5-family) architectures — grouped-regex smoothing or mismatched mappings corrupt decoding.
Unquantized (kept BF16): embeddings, lm_head, norms, linear_attn.in_proj_a/b, the vision tower, and the MTP head.
Run command:

CUDA_VISIBLE_DEVICES=1,2 python3 quantize-awq-hybrid.py \
  --model_path ./Swift-Qwen3.8-27b \
  --quant_path ./Swift-Qwen3.8-27b-W4A16-AWQ \
  --offload_dir ./Swift-Qwen3.8-27b-W4A16-AWQ

(Default calibration: UltraChat 200k train_sft.)

Recommended parameters

From the base model: temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1.0.

Usage

Tested with LMDeploy turbomind:

from lmdeploy import pipeline, TurbomindEngineConfig

pipe = pipeline(
    "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ",
    backend_config=TurbomindEngineConfig(
        tp=2,
        model_format="compressed-tensors",
        language_model_only=True,
    ),
)
print(pipe("Hello, who are you?").text)

License and access

Swift weights are distributed through gated access under the Swift Open License v1.0; this quantization inherits that license from the base model. Personal, research, educational, evaluation, and commercial use are free for individuals and organizations with annual recurring revenue, including affiliates, of up to US$1,000,000. Above that threshold, commercial use requires a separate Swift Enterprise License. Contact UkisAI for terms.

Files

quantize-awq-hybrid.py — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered -of-N shards). --offload_dir selects where per-rank CPU offload temp folders live (defaults to the current working directory).
model-nonquant.safetensors — unquantized tensors (mtp.* and model.visual.*) preserved BF16 so the full model architecture is loadable.

Downloads last month: 3,245

Safetensors

Model size

28B params

Tensor type

BF16

I32

Model tree for TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ

Base model

Qwen/Qwen3.8-27B

Finetuned

ukisai/Swift-Qwen3.8-27b

Quantized

(48)

this model

Quantizations

1 model