Image-Text-to-Text
GGUF
English
Chinese
llama.cpp
quantized
qwen3.8
qwen3.5
ridge
gated-deltanet
imatrix
reasoning
multimodal
vision
mtp
long-context
conversational
Instructions to use empero-ai/Qwen3.8-27B-Ridge-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use empero-ai/Qwen3.8-27B-Ridge-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16 # Run inference directly in the terminal: llama cli -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16 # Run inference directly in the terminal: llama cli -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16 # Run inference directly in the terminal: ./llama-cli -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16 # Run inference directly in the terminal: ./build/bin/llama-cli -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Use Docker
docker model run hf.co/empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
- LM Studio
- Jan
- vLLM
How to use empero-ai/Qwen3.8-27B-Ridge-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "empero-ai/Qwen3.8-27B-Ridge-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "empero-ai/Qwen3.8-27B-Ridge-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
- Ollama
How to use empero-ai/Qwen3.8-27B-Ridge-GGUF with Ollama:
ollama run hf.co/empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
- Unsloth Desktop
- Pi
How to use empero-ai/Qwen3.8-27B-Ridge-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use empero-ai/Qwen3.8-27B-Ridge-GGUF with Docker Model Runner:
docker model run hf.co/empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
- Lemonade
How to use empero-ai/Qwen3.8-27B-Ridge-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Run and chat with the model
lemonade run user.Qwen3.8-27B-Ridge-GGUF-BF16
List all available models
lemonade list
- Hermes Agent
How to use empero-ai/Qwen3.8-27B-Ridge-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use empero-ai/Qwen3.8-27B-Ridge-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "empero-ai/Qwen3.8-27B-Ridge-GGUF:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Update README.md
Browse files
README.md
CHANGED
|
@@ -32,23 +32,20 @@ A Gated-DeltaNet-aware mixed GGUF of official
|
|
| 32 |
for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio,
|
| 33 |
jan, KoboldCpp, and other stock GGUF runtimes.
|
| 34 |
|
| 35 |
-
This is a quantization of the
|
| 36 |
-
not a new codec. Ridge is a probed mix of types llama.cpp already ships,
|
| 37 |
written for this architecture: 64 layers =
|
| 38 |
16 × `(3 × GatedDeltaNet → FFN + 1 × GatedAttn → FFN)`. Generic `IQ2_XS`
|
| 39 |
-
and
|
| 40 |
-
the GDN mixers as first-class.
|
| 41 |
|
| 42 |
Nothing was stripped to make the file fit. The native MTP draft head
|
| 43 |
(`blk.64` / `nextn`) stays in the GGUF. Vision is a separate BF16
|
| 44 |
`mmproj`.
|
| 45 |
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
capability writeup lives on the
|
| 51 |
-
**[base model card](https://huggingface.co/Qwen/Qwen3.8-27B)**.
|
| 52 |
|
| 53 |
---
|
| 54 |
|
|
@@ -68,7 +65,7 @@ If you only want text, download the Ridge GGUF. Add the `mmproj` for image input
|
|
| 68 |
|
| 69 |
These are practical **weight-size-based estimates**, not a VRAM benchmark.
|
| 70 |
They assume a modest context and leave room for runtime and the KV cache.
|
| 71 |
-
Image input adds the 0.87 GiB `mmproj`. The native 262k window
|
| 72 |
1M YaRN extension — make KV the dominant cost and may need offload
|
| 73 |
regardless of weight quant.
|
| 74 |
|
|
@@ -92,18 +89,6 @@ layer. GDN state is disproportionately sensitive to low-bit quantization,
|
|
| 92 |
so Ridge holds that path high and spends the saved bits by dropping
|
| 93 |
mid-stack FFN.
|
| 94 |
|
| 95 |
-
| Role | Tensors | Type |
|
| 96 |
-
|---|---|---|
|
| 97 |
-
| Embeddings, `lm_head` | `token_embd`, `output.weight` | Q6_K |
|
| 98 |
-
| GDN state | `ssm_alpha`, `ssm_beta`, `ssm_conv1d`, `ssm_a`, `ssm_dt`, `ssm_norm` | Q8_0 |
|
| 99 |
-
| Norms | `attn_norm`, `post_attention_norm`, `output_norm`, `attn_q_norm`, `attn_k_norm` | Q8_0 |
|
| 100 |
-
| GDN mixers (48 linear-attn layers) | `attn_qkv`, `attn_gate`, `ssm_out` | Q4_K |
|
| 101 |
-
| Full attention (layers 3, 7, …, 63) | `attn_q`, `attn_k`, `attn_v`, `attn_output` | Q5_K |
|
| 102 |
-
| FFN first / last 4 layers (`blk.0–3`, `blk.60–63`) | `ffn_{gate,up,down}` | IQ3_S |
|
| 103 |
-
| FFN mid (bulk of the bits) | `ffn_{gate,up,down}` | IQ2_M |
|
| 104 |
-
| MTP draft (`blk.64`; no imatrix) | `nextn.*`, `blk.64.ffn_*` | Q6_K |
|
| 105 |
-
| Vision | separate file | BF16 |
|
| 106 |
-
|
| 107 |
**The Gated-DeltaNet state path is Q8_0.** Mixers are Q4_K, not IQ2.
|
| 108 |
That is the difference between this file and a flat 2-bit dump of the
|
| 109 |
same model.
|
|
@@ -125,11 +110,6 @@ Same box, same calibration file, `llama-perplexity`, 80 chunks,
|
|
| 125 |
| BF16 GGUF (this convert) | 50.89 GiB | 16.00 | **7.15 ± 0.12** | — |
|
| 126 |
| **Ridge-3.7bpw** | **11.73 GiB** | **3.69** | **7.82 ± 0.14** | **+9.3 %** |
|
| 127 |
|
| 128 |
-
Not lossless. +9 % PPL at 23 % of BF16 size is the first-bake number.
|
| 129 |
-
A same-suite bake-off against Unsloth `UD-Q3_K_XL` and bartowski
|
| 130 |
-
`IQ3_XXS` has not been run on this box yet — do not read the size table
|
| 131 |
-
below as a quality ranking.
|
| 132 |
-
|
| 133 |
---
|
| 134 |
|
| 135 |
## Comparison
|
|
@@ -143,24 +123,10 @@ where we measured the file ourselves.
|
|
| 143 |
| `UD-IQ2_XXS` | [unsloth](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) | 8.39 GiB | ~2.1 bpw | *not measured here* (Unsloth quotes 82.5 % top-1 vs BF16) |
|
| 144 |
| `UD-IQ2_M` | unsloth | 9.61 GiB | ~2.4 bpw | *not measured* |
|
| 145 |
| `IQ2_XXS` | [bartowski](https://huggingface.co/bartowski/Qwen3.8-27B-GGUF) | 8.75 GiB | ~2.2 bpw | *not measured* |
|
| 146 |
-
| `IQ2_M` | bartowski | 10.13 GiB | ~2.5 bpw | *not measured* |
|
| 147 |
-
| `UD-IQ3_XXS` | unsloth | 11.10 GiB | ~2.8 bpw | *not measured* |
|
| 148 |
-
| `Q2_K` | bartowski | 11.03 GiB | ~2.7 bpw | *not measured* |
|
| 149 |
| `Q3_K_S` | unsloth | 11.71 GiB | ~3.1 bpw | *not measured* |
|
| 150 |
| **Ridge-3.7bpw** | **empero-ai** | **11.73 GiB** | **3.69 bpw** | **7.82 (+9 %)** |
|
| 151 |
| `IQ3_XXS` | bartowski | 11.76 GiB | ~2.9 bpw | *not measured* |
|
| 152 |
| `UD-Q3_K_XL` | unsloth | 12.52 GiB | ~3.4 bpw | *not measured* |
|
| 153 |
-
| `Q3_K_M` | unsloth | 12.87 GiB | ~3.5 bpw | *not measured* |
|
| 154 |
-
| `Q4_K_M` | unsloth | 15.93 GiB | ~4.3 bpw | *not measured* |
|
| 155 |
-
|
| 156 |
-
How to read this:
|
| 157 |
-
|
| 158 |
-
- Ridge-3.7bpw sits in the **~12 GiB** band, next to Unsloth `Q3_K_S` and
|
| 159 |
-
bartowski `IQ3_XXS` — not next to UD-IQ2_XXS.
|
| 160 |
-
- The extra bits versus a flat IQ2 go into **GDN state (Q8_0)**, **GDN
|
| 161 |
-
mixers (Q4_K)**, and **full attention (Q5_K)**. Mid-stack FFN stays IQ2_M.
|
| 162 |
-
- We are not claiming this file beats Unsloth 2-bit at the same size. We
|
| 163 |
-
spent the bits on GDN on purpose.
|
| 164 |
|
| 165 |
---
|
| 166 |
|
|
@@ -288,12 +254,7 @@ YaRN. Set `-c` to what you actually need — the KV cache, not the
|
|
| 288 |
|
| 289 |
## Limitations
|
| 290 |
|
| 291 |
-
- **Not lossless.** +9 % wiki-style PPL vs our BF16 convert
|
| 292 |
-
gap. Do not describe this as an almost-lossless 2-bit.
|
| 293 |
-
- **Not a 2-bit file.** The name is Ridge-**3.7bpw** because that is
|
| 294 |
-
what llama-quantize reported (3.69).
|
| 295 |
-
- **Same-suite bake-off pending.** Unsloth / bartowski PPL on this
|
| 296 |
-
calibration set has not been measured here.
|
| 297 |
- **Context costs memory.** Weight size is only part of the hardware
|
| 298 |
budget.
|
| 299 |
- **MTP is runtime-dependent.** The head is in the file; the speedup
|
|
|
|
| 32 |
for [llama.cpp](https://github.com/ggml-org/llama.cpp), Ollama, LM Studio,
|
| 33 |
jan, KoboldCpp, and other stock GGUF runtimes.
|
| 34 |
|
| 35 |
+
This is a quantization of the Qwen3.8-27B checkpoint. Ridge is a probed mix of types
|
|
|
|
| 36 |
written for this architecture: 64 layers =
|
| 37 |
16 × `(3 × GatedDeltaNet → FFN + 1 × GatedAttn → FFN)`. Generic `IQ2_XS`
|
| 38 |
+
and UD-IQ2 do not treat GDN state (`ssm_alpha` / `ssm_beta`) or
|
| 39 |
+
the GDN mixers as first-class. We fixed that.
|
| 40 |
|
| 41 |
Nothing was stripped to make the file fit. The native MTP draft head
|
| 42 |
(`blk.64` / `nextn`) stays in the GGUF. Vision is a separate BF16
|
| 43 |
`mmproj`.
|
| 44 |
|
| 45 |
+
> [!Note]
|
| 46 |
+
> This card is about choosing the file and running it. The official
|
| 47 |
+
> capability writeup lives on the
|
| 48 |
+
> **[base model card](https://huggingface.co/Qwen/Qwen3.8-27B)**.
|
|
|
|
|
|
|
| 49 |
|
| 50 |
---
|
| 51 |
|
|
|
|
| 65 |
|
| 66 |
These are practical **weight-size-based estimates**, not a VRAM benchmark.
|
| 67 |
They assume a modest context and leave room for runtime and the KV cache.
|
| 68 |
+
Image input adds the 0.87 GiB `mmproj`. The native 262k window and the
|
| 69 |
1M YaRN extension — make KV the dominant cost and may need offload
|
| 70 |
regardless of weight quant.
|
| 71 |
|
|
|
|
| 89 |
so Ridge holds that path high and spends the saved bits by dropping
|
| 90 |
mid-stack FFN.
|
| 91 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
**The Gated-DeltaNet state path is Q8_0.** Mixers are Q4_K, not IQ2.
|
| 93 |
That is the difference between this file and a flat 2-bit dump of the
|
| 94 |
same model.
|
|
|
|
| 110 |
| BF16 GGUF (this convert) | 50.89 GiB | 16.00 | **7.15 ± 0.12** | — |
|
| 111 |
| **Ridge-3.7bpw** | **11.73 GiB** | **3.69** | **7.82 ± 0.14** | **+9.3 %** |
|
| 112 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
---
|
| 114 |
|
| 115 |
## Comparison
|
|
|
|
| 123 |
| `UD-IQ2_XXS` | [unsloth](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) | 8.39 GiB | ~2.1 bpw | *not measured here* (Unsloth quotes 82.5 % top-1 vs BF16) |
|
| 124 |
| `UD-IQ2_M` | unsloth | 9.61 GiB | ~2.4 bpw | *not measured* |
|
| 125 |
| `IQ2_XXS` | [bartowski](https://huggingface.co/bartowski/Qwen3.8-27B-GGUF) | 8.75 GiB | ~2.2 bpw | *not measured* |
|
|
|
|
|
|
|
|
|
|
| 126 |
| `Q3_K_S` | unsloth | 11.71 GiB | ~3.1 bpw | *not measured* |
|
| 127 |
| **Ridge-3.7bpw** | **empero-ai** | **11.73 GiB** | **3.69 bpw** | **7.82 (+9 %)** |
|
| 128 |
| `IQ3_XXS` | bartowski | 11.76 GiB | ~2.9 bpw | *not measured* |
|
| 129 |
| `UD-Q3_K_XL` | unsloth | 12.52 GiB | ~3.4 bpw | *not measured* |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
---
|
| 132 |
|
|
|
|
| 254 |
|
| 255 |
## Limitations
|
| 256 |
|
| 257 |
+
- **Not lossless.** +9 % wiki-style PPL vs our BF16 convert
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 258 |
- **Context costs memory.** Weight size is only part of the hardware
|
| 259 |
budget.
|
| 260 |
- **MTP is runtime-dependent.** The head is in the file; the speedup
|