# Intel NPU & GPU Acceleration Guide for Qwen ONNX Model

Complete guide for using Intel Neural Processing Unit (NPU), Integrated GPU, and Arc GPU with onnxruntime-genai.

## Hardware Support Overview

### Intel Execution Providers in ONNX Runtime

| Hardware | Provider | Support | Status |
|----------|----------|---------|--------|
| **Intel NPU** | OpenVINO | Yes | ✓ Recommended |
| **Intel iGPU** (Arc/Xe) | OpenVINO/DML | Yes | ✓ Supported |
| **Intel Arc GPU** | OpenVINO/DML | Yes | ✓ Supported |
| **Intel CPU** | OpenVINO/CPU | Yes | ✓ Always available |

### Modern Intel CPUs with NPU

| Generation | Processor | NPU | iGPU | Notes |
|------------|-----------|-----|------|-------|
| **Meteor Lake** (2023) | Core Ultra | ✓ | Xe-iGPU | **Best for NPU** |
| **Arrow Lake** (2024) | Core Ultra 200 | ✓ | Xe-iGPU | Latest |
| **Raptor Lake** | Core i7/i9 13th | ✗ | Iris Xe | GPU only |
| **12th Gen** | Core i7/i9 12th | ✗ | Iris Xe | GPU only |

---

## Installation

### Step 1: Install OpenVINO Runtime

**Option A: Quick Install (Recommended)**
```bash
pip install onnxruntime-openvino
```

**Option B: Full Installation**
```bash
# Install ONNX Runtime with OpenVINO support
pip install onnxruntime>=1.17.0

# Install OpenVINO toolkit
pip install openvino>=2024.0

# Install Intel GPU support (Windows)
pip install intel-level-zero-loader
```

### Step 2: Verify Installation

```python
import onnxruntime as ort

providers = ort.get_available_providers()
print("Available Providers:")
for p in providers:
    print(f"  - {p}")

# Should show:
# - OpenVINOExecutionProvider  (Intel optimized)
# - CUDAExecutionProvider      (if NVIDIA GPU available)
# - CPUExecutionProvider       (fallback)
```

### Step 3: Verify Hardware Detection

```bash
# Windows: Check for Intel GPU
wmic path win32_videocontroller get name

# Linux: 
# lspci | grep -i intel
# lspci | grep -i vga
```

### Step 4: Install GPU Drivers (Windows)

For Intel Arc GPU / Integrated GPU:

```bash
# Download and install Intel Arc GPU drivers from:
# https://www.intel.com/content/www/us/en/support/articles/000090010/graphics.html

# Or use Intel Driver & Support Assistant:
# https://www.intel.com/content/www/us/en/support/detect.html
```

---

## Usage Examples

### 1. Auto-Detection (Recommended)

```python
from intel_acceleration import IntelAcceleratedInference

# Automatically uses best available Intel hardware
model = IntelAcceleratedInference(provider="AUTO")

response = model.generate(
    "What is the future of AI?",
    max_length=200
)
print(response)
```

### 2. Specify NPU

```python
from intel_acceleration import IntelAcceleratedInference

# Force NPU (if available)
model = IntelAcceleratedInference(provider="NPU")

response = model.generate("Hello!", max_length=100)
print(response)
```

### 3. Specify iGPU

```python
from intel_acceleration import IntelAcceleratedInference

# Use Intel iGPU
model = IntelAcceleratedInference(provider="GPU")

response = model.generate("Explain machine learning", max_length=200)
print(response)
```

### 4. Check Available Hardware

```python
from intel_acceleration import check_intel_hardware

check_intel_hardware()
```

Output:
```
============================================================
Intel Hardware Detection
============================================================

ONNX Runtime Execution Providers:
  - OpenVINOExecutionProvider
  - TensorrtExecutionProvider
  - CudaExecutionProvider
  - CPUExecutionProvider

✓ Intel-related providers found:
    - OpenVINOExecutionProvider
```

### 5. Benchmark Different Providers

```python
from intel_acceleration import compare_providers

# Compare NPU vs GPU vs CPU
compare_providers()
```

### 6. Detailed Benchmarking

```python
from intel_acceleration import IntelAcceleratedInference

# Benchmark on specific hardware
model = IntelAcceleratedInference(provider="NPU")
benchmark = model.benchmark_hardware()

print(f"Speed: {benchmark['speed']:.2f} tokens/sec")
print(f"Time: {benchmark['time']:.2f}s")
print(f"Tokens: {benchmark['tokens']}")
```

---

## Configuration Options

### OpenVINO Provider Options

Edit `genai_config.json` for fine-tuned control:

```json
{
    "model": {
        "decoder": {
            "session_options": {
                "provider": "OpenVINOExecutionProvider",
                "provider_options": {
                    "device_type": "AUTO",
                    "enable_caching": true,
                    "cache_dir": "./ov_cache",
                    "num_streams": "4",
                    "ov_device_type": "NPU.0"
                }
            }
        }
    }
}
```

### Device Type Options

```python
# NPU (Neural Processing Unit) - Best performance for inference
device_type = "NPU"

# GPU - Intel Arc / integrated GPU
device_type = "GPU_FP32"    # Full precision
device_type = "GPU_FP16"    # Half precision (faster)

# AUTO - Let OpenVINO choose
device_type = "AUTO"

# CPU - Fallback
device_type = "CPU"
```

### Performance Tuning Options

```python
provider_options = {
    "device_type": "NPU",
    "enable_caching": True,           # Cache compiled models
    "cache_dir": "./ov_cache",        # Cache location
    "num_streams": "4",               # Parallel inference
    "enable_profiling": False,        # Profile performance
}
```

---

## Performance Comparison

### Typical Performance (Qwen3 Model)

#### On Intel Meteor Lake (Core Ultra with NPU)

| Hardware | Speed | Tokens/sec | Latency | Power |
|----------|-------|-----------|---------|-------|
| **NPU** | ⚡⚡⚡ | 50-80 | 12-20ms | Low |
| **iGPU** | ⚡⚡ | 30-50 | 20-33ms | Medium |
| **CPU** | ⚡ | 5-15 | 66-200ms | Medium |

**Note**: Actual performance depends on:
- Specific CPU model
- Model size
- Context length
- Quantization level

### Optimization Strategies

1. **Use NPU for best balance** of speed and power
2. **Enable model caching** to avoid recompilation
3. **Use lower precision** (FP16) for faster inference
4. **Batch requests** when possible
5. **Profile** your specific hardware with `benchmark_hardware()`

---

## Quantization for Intel Hardware

### Lower Precision = Faster Inference

```python
# FP32 - Best quality, slower
provider_options = {"precision": "FP32"}

# FP16 - Good quality, faster (recommended)
provider_options = {"precision": "FP16"}

# INT8 - Smallest size, fastest (requires quantization)
provider_options = {"precision": "INT8"}
```

To use quantized models:
```bash
# Convert to quantized ONNX
python -m onnxruntime.quantization.quantize --model_input model.onnx \
    --model_output model_int8.onnx --op_types_to_quantize MatMul,Add \
    --weight_type QInt8
```

---

## Troubleshooting

### Issue: OpenVINO not found

```
Error: OpenVINOExecutionProvider not available
```

**Solution:**
```bash
pip install --upgrade onnxruntime-openvino
pip install --upgrade openvino
```

### Issue: NPU not detected

```
Warning: NPU not available, falling back to CPU
```

**Solutions:**
1. Check if your CPU has NPU (Meteor Lake or newer)
2. Update Intel drivers
3. Restart computer after driver installation
4. Check BIOS settings (ensure NPU is enabled)

```python
# Debug: Check what's available
import onnxruntime as ort
print(ort.get_available_providers())
```

### Issue: GPU not detected

```
Warning: GPU not detected, using CPU
```

**Solutions (Windows):**
```bash
# Update Intel GPU drivers
# https://www.intel.com/content/www/us/en/support/detect.html

# Or for Arc GPU:
# https://www.intel.com/content/www/us/en/download/785597/

# Verify in Device Manager:
# Look for "Intel Arc" or "Intel Iris Xe Graphics"
```

### Issue: Memory errors

```
RuntimeError: out of memory
```

**Solutions:**
```python
# Reduce batch size
# Reduce model quantization precision
# Close other applications

# Use memory efficient mode
import gc
import torch

gc.collect()
if torch.cuda.is_available():
    torch.cuda.empty_cache()
```

### Issue: Slow inference

**Optimization steps:**
```python
# 1. Enable caching
provider_options = {
    "device_type": "NPU",
    "enable_caching": True
}

# 2. Use FP16 precision
provider_options = {"precision": "FP16"}

# 3. Reduce max_length
model.generate(prompt, max_length=100)  # Smaller output

# 4. Use temperature=0 for faster greedy decoding
model.generate(prompt, temperature=0.0)

# 5. Benchmark to identify bottleneck
model.benchmark_hardware()
```

---

## Advanced: Manual Configuration

### Modify genai_config.json

```json
{
    "model": {
        "bos_token_id": 151643,
        "context_length": 40960,
        "decoder": {
            "session_options": {
                "log_id": "onnxruntime-genai",
                "provider": "OpenVINOExecutionProvider",
                "provider_options": {
                    "device_type": "AUTO",
                    "enable_caching": true,
                    "cache_dir": "./ov_cache",
                    "num_streams": "4",
                    "profiling_verbosity": 0
                }
            },
            "filename": "model.onnx",
            "head_size": 128,
            "hidden_size": 4096,
            "num_attention_heads": 32,
            "num_hidden_layers": 36,
            "num_key_value_heads": 8
        }
    },
    "search": {
        "max_length": 512,
        "temperature": 0.6,
        "top_p": 0.95,
        "top_k": 20
    }
}
```

---

## Performance Monitoring

### Enable Profiling

```python
from intel_acceleration import IntelAcceleratedInference

model = IntelAcceleratedInference(provider="NPU")

# Benchmark with profiling
result = model.benchmark_hardware(prompt="Your test prompt")

print(f"Provider: {result['provider']}")
print(f"Speed: {result['speed']:.2f} tokens/sec")
print(f"Time: {result['time']:.2f}s")
```

### Compare Providers

```bash
python intel_acceleration.py
# This will:
# 1. Detect available hardware
# 2. Test each provider
# 3. Show performance comparison
```

---

## Resources

- [OpenVINO Documentation](https://docs.openvino.ai/)
- [ONNX Runtime Intel Support](https://onnxruntime.ai/docs/)
- [Intel NPU Specification](https://www.intel.com/content/www/us/en/products/details/processors/core-ultra.html)
- [Intel Arc GPU](https://www.intel.com/content/www/us/en/products/details/discrete-gpus/arc.html)

---

## Quick Reference

```python
# Quickest setup
from intel_acceleration import IntelAcceleratedInference

model = IntelAcceleratedInference(provider="AUTO")  # Best available
response = model.generate("Your prompt", max_length=100)
print(response)
```

**That's it!** The Intel hardware acceleration will automatically be used if available.