WebWorld-8B-Onnx / INTEL_NPU_GUIDE.md

Prince-1

Add files using upload-large-folder tool

5abb996 verified 4 months ago

preview code

Raw

History Blame Contribute Delete

10.6 kB

Intel NPU & GPU Acceleration Guide for Qwen ONNX Model

Complete guide for using Intel Neural Processing Unit (NPU), Integrated GPU, and Arc GPU with onnxruntime-genai.

Hardware Support Overview

Intel Execution Providers in ONNX Runtime

Hardware	Provider	Support	Status
Intel NPU	OpenVINO	Yes	✓ Recommended
Intel iGPU (Arc/Xe)	OpenVINO/DML	Yes	✓ Supported
Intel Arc GPU	OpenVINO/DML	Yes	✓ Supported
Intel CPU	OpenVINO/CPU	Yes	✓ Always available

Modern Intel CPUs with NPU

Generation	Processor	NPU	iGPU	Notes
Meteor Lake (2023)	Core Ultra	✓	Xe-iGPU	Best for NPU
Arrow Lake (2024)	Core Ultra 200	✓	Xe-iGPU	Latest
Raptor Lake	Core i7/i9 13th	✗	Iris Xe	GPU only
12th Gen	Core i7/i9 12th	✗	Iris Xe	GPU only

Installation

Step 1: Install OpenVINO Runtime

Option A: Quick Install (Recommended)

pip install onnxruntime-openvino

Option B: Full Installation

# Install ONNX Runtime with OpenVINO support
pip install onnxruntime>=1.17.0

# Install OpenVINO toolkit
pip install openvino>=2024.0

# Install Intel GPU support (Windows)
pip install intel-level-zero-loader

Step 2: Verify Installation

import onnxruntime as ort

providers = ort.get_available_providers()
print("Available Providers:")
for p in providers:
    print(f"  - {p}")

# Should show:
# - OpenVINOExecutionProvider  (Intel optimized)
# - CUDAExecutionProvider      (if NVIDIA GPU available)
# - CPUExecutionProvider       (fallback)

Step 3: Verify Hardware Detection

# Windows: Check for Intel GPU
wmic path win32_videocontroller get name

# Linux: 
# lspci | grep -i intel
# lspci | grep -i vga

Step 4: Install GPU Drivers (Windows)

For Intel Arc GPU / Integrated GPU:

# Download and install Intel Arc GPU drivers from:
# https://www.intel.com/content/www/us/en/support/articles/000090010/graphics.html

# Or use Intel Driver & Support Assistant:
# https://www.intel.com/content/www/us/en/support/detect.html

Usage Examples

1. Auto-Detection (Recommended)

from intel_acceleration import IntelAcceleratedInference

# Automatically uses best available Intel hardware
model = IntelAcceleratedInference(provider="AUTO")

response = model.generate(
    "What is the future of AI?",
    max_length=200
)
print(response)

2. Specify NPU

from intel_acceleration import IntelAcceleratedInference

# Force NPU (if available)
model = IntelAcceleratedInference(provider="NPU")

response = model.generate("Hello!", max_length=100)
print(response)

3. Specify iGPU

from intel_acceleration import IntelAcceleratedInference

# Use Intel iGPU
model = IntelAcceleratedInference(provider="GPU")

response = model.generate("Explain machine learning", max_length=200)
print(response)

4. Check Available Hardware

from intel_acceleration import check_intel_hardware

check_intel_hardware()

Output: ```

Intel Hardware Detection

ONNX Runtime Execution Providers:

OpenVINOExecutionProvider
TensorrtExecutionProvider
CudaExecutionProvider
CPUExecutionProvider

✓ Intel-related providers found: - OpenVINOExecutionProvider


### 5. Benchmark Different Providers

```python
from intel_acceleration import compare_providers

# Compare NPU vs GPU vs CPU
compare_providers()

6. Detailed Benchmarking

from intel_acceleration import IntelAcceleratedInference

# Benchmark on specific hardware
model = IntelAcceleratedInference(provider="NPU")
benchmark = model.benchmark_hardware()

print(f"Speed: {benchmark['speed']:.2f} tokens/sec")
print(f"Time: {benchmark['time']:.2f}s")
print(f"Tokens: {benchmark['tokens']}")

Configuration Options

OpenVINO Provider Options

Edit genai_config.json for fine-tuned control:

{
    "model": {
        "decoder": {
            "session_options": {
                "provider": "OpenVINOExecutionProvider",
                "provider_options": {
                    "device_type": "AUTO",
                    "enable_caching": true,
                    "cache_dir": "./ov_cache",
                    "num_streams": "4",
                    "ov_device_type": "NPU.0"
                }
            }
        }
    }
}

Device Type Options

# NPU (Neural Processing Unit) - Best performance for inference
device_type = "NPU"

# GPU - Intel Arc / integrated GPU
device_type = "GPU_FP32"    # Full precision
device_type = "GPU_FP16"    # Half precision (faster)

# AUTO - Let OpenVINO choose
device_type = "AUTO"

# CPU - Fallback
device_type = "CPU"

Performance Tuning Options

provider_options = {
    "device_type": "NPU",
    "enable_caching": True,           # Cache compiled models
    "cache_dir": "./ov_cache",        # Cache location
    "num_streams": "4",               # Parallel inference
    "enable_profiling": False,        # Profile performance
}

Performance Comparison

Typical Performance (Qwen3 Model)

On Intel Meteor Lake (Core Ultra with NPU)

Hardware	Speed	Tokens/sec	Latency	Power
NPU	⚡⚡⚡	50-80	12-20ms	Low
iGPU	⚡⚡	30-50	20-33ms	Medium
CPU	⚡	5-15	66-200ms	Medium

Note: Actual performance depends on:

Specific CPU model
Model size
Context length
Quantization level

Optimization Strategies

Use NPU for best balance of speed and power
Enable model caching to avoid recompilation
Use lower precision (FP16) for faster inference
Batch requests when possible
Profile your specific hardware with benchmark_hardware()

Quantization for Intel Hardware

Lower Precision = Faster Inference

# FP32 - Best quality, slower
provider_options = {"precision": "FP32"}

# FP16 - Good quality, faster (recommended)
provider_options = {"precision": "FP16"}

# INT8 - Smallest size, fastest (requires quantization)
provider_options = {"precision": "INT8"}

To use quantized models:

# Convert to quantized ONNX
python -m onnxruntime.quantization.quantize --model_input model.onnx \
    --model_output model_int8.onnx --op_types_to_quantize MatMul,Add \
    --weight_type QInt8

Troubleshooting

Issue: OpenVINO not found

Error: OpenVINOExecutionProvider not available

Solution:

pip install --upgrade onnxruntime-openvino
pip install --upgrade openvino

Issue: NPU not detected

Warning: NPU not available, falling back to CPU

Solutions:

Check if your CPU has NPU (Meteor Lake or newer)
Update Intel drivers
Restart computer after driver installation
Check BIOS settings (ensure NPU is enabled)

# Debug: Check what's available
import onnxruntime as ort
print(ort.get_available_providers())

Issue: GPU not detected

Warning: GPU not detected, using CPU

Solutions (Windows):

# Update Intel GPU drivers
# https://www.intel.com/content/www/us/en/support/detect.html

# Or for Arc GPU:
# https://www.intel.com/content/www/us/en/download/785597/

# Verify in Device Manager:
# Look for "Intel Arc" or "Intel Iris Xe Graphics"

Issue: Memory errors

RuntimeError: out of memory

Solutions:

# Reduce batch size
# Reduce model quantization precision
# Close other applications

# Use memory efficient mode
import gc
import torch

gc.collect()
if torch.cuda.is_available():
    torch.cuda.empty_cache()

Issue: Slow inference

Optimization steps:

# 1. Enable caching
provider_options = {
    "device_type": "NPU",
    "enable_caching": True
}

# 2. Use FP16 precision
provider_options = {"precision": "FP16"}

# 3. Reduce max_length
model.generate(prompt, max_length=100)  # Smaller output

# 4. Use temperature=0 for faster greedy decoding
model.generate(prompt, temperature=0.0)

# 5. Benchmark to identify bottleneck
model.benchmark_hardware()

Advanced: Manual Configuration

Modify genai_config.json

{
    "model": {
        "bos_token_id": 151643,
        "context_length": 40960,
        "decoder": {
            "session_options": {
                "log_id": "onnxruntime-genai",
                "provider": "OpenVINOExecutionProvider",
                "provider_options": {
                    "device_type": "AUTO",
                    "enable_caching": true,
                    "cache_dir": "./ov_cache",
                    "num_streams": "4",
                    "profiling_verbosity": 0
                }
            },
            "filename": "model.onnx",
            "head_size": 128,
            "hidden_size": 4096,
            "num_attention_heads": 32,
            "num_hidden_layers": 36,
            "num_key_value_heads": 8
        }
    },
    "search": {
        "max_length": 512,
        "temperature": 0.6,
        "top_p": 0.95,
        "top_k": 20
    }
}

Performance Monitoring

Enable Profiling

from intel_acceleration import IntelAcceleratedInference

model = IntelAcceleratedInference(provider="NPU")

# Benchmark with profiling
result = model.benchmark_hardware(prompt="Your test prompt")

print(f"Provider: {result['provider']}")
print(f"Speed: {result['speed']:.2f} tokens/sec")
print(f"Time: {result['time']:.2f}s")

Compare Providers

python intel_acceleration.py
# This will:
# 1. Detect available hardware
# 2. Test each provider
# 3. Show performance comparison

Resources

Quick Reference

# Quickest setup
from intel_acceleration import IntelAcceleratedInference

model = IntelAcceleratedInference(provider="AUTO")  # Best available
response = model.generate("Your prompt", max_length=100)
print(response)

That's it! The Intel hardware acceleration will automatically be used if available.