Instructions to use Archsloth/Qwen3-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Archsloth/Qwen3-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Archsloth/Qwen3-4B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Archsloth/Qwen3-4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Archsloth/Qwen3-4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Archsloth/Qwen3-4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Archsloth/Qwen3-4B-GGUF:Q4_K_M

Ollama
How to use Archsloth/Qwen3-4B-GGUF with Ollama:
```
ollama run hf.co/Archsloth/Qwen3-4B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use Archsloth/Qwen3-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Archsloth/Qwen3-4B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Archsloth/Qwen3-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/Archsloth/Qwen3-4B-GGUF:Q4_K_M
```

Lemonade

How to use Archsloth/Qwen3-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Archsloth/Qwen3-4B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3-4B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Archsloth/Qwen3-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Archsloth/Qwen3-4B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Archsloth/Qwen3-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Archsloth/Qwen3-4B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Archsloth/Qwen3-4B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

openfree commited on 23 days ago

Commit

3240ace

verified ·

1 Parent(s): d070498

card: direct unsloth comparison + per-language and frontier charts

Browse files

Files changed (3) hide show

README.md +46 -15
chart_axes.svg +94 -0
chart_frontier.svg +56 -0

README.md CHANGED Viewed

@@ -42,6 +42,17 @@ tags:
   <b>Most quantized weights ship with an adjective. Ours ship with a table.</b>
 </p>
 ---
 > ### 📚 Collection
@@ -80,23 +91,28 @@ There is no Q5 file here. [Why not.](#what-we-did-not-win)
 ## `[measured]` Q4_K_M — ten axes, 2,497,280,800 bytes on both sides
 KL divergence from the bf16 original. **Lower is better.** `llama-perplexity --kl-divergence`,
 ctx 512, 60 chunks for Korean and English, 40 for the rest. Every ARCHsloth number below comes
 from **one single build**.
-| | script | **ARCHsloth Q4** | `Qwen3-4B-Q4_K_M.gguf` | `Qwen3-4B-UD-Q4_K_XL.gguf` | stock `llama-quantize` |
-|---|---|---|---|---|---|
-| | file size | **2.4973 GB** | 2.4973 GB | 2.5463 GB | 2.4973 GB |
-| Korean | Hangul | **0.024297** | 0.053290 | 0.046637 | 0.081506 |
-| English | Latin | **0.032344** | 0.048443 | 0.043660 | 0.066203 |
-| Spanish | Latin | **0.033679** | 0.048234 | 0.043093 | 0.069169 |
-| Japanese | Kana·Han | **0.034148** | 0.051534 | 0.045666 | 0.075946 |
-| Chinese | Han | **0.033088** | 0.048051 | 0.042510 | 0.073735 |
-| Russian | Cyrillic | **0.032852** | 0.043831 | 0.038792 | 0.065831 |
-| Arabic | Arabic | **0.040871** | 0.053212 | 0.047022 | 0.067085 |
-| Hindi | Devanagari | **0.031617** | 0.039979 | 0.035678 | 0.053405 |
-| Thai | Thai | **0.024561** | 0.031301 | 0.027782 | 0.045149 |
-| **Source code** | — | **0.010191** | 0.021659 | 0.018216 | 0.039165 |
 Ten axes, seven writing systems, ahead on every one — against the same-size file and against
 the larger one. Lowest separation is 3.8 σ (Hindi); highest is 20.4 σ (Korean).
@@ -105,14 +121,29 @@ the larger one. Lowest separation is 3.8 σ (Hindi); highest is 20.4 σ (Korean)
 not.** They are the control, so that a build which buys one language with another has somewhere
 to show it.
 ### `[measured]` Q6_K and Q8_0
 | | size | Korean | English |
 |---|---|---|---|
 | **ARCHsloth Q6_K** | 3.3063 GB | **0.004077** | **0.004022** |
-| `Qwen3-4B-Q6_K.gguf` | 3.3063 GB | 0.006969 | 0.005765 |
 | **ARCHsloth Q8_0** | 4.2804 GB | **0.001624** | **0.001371** |
-| `Qwen3-4B-Q8_0.gguf` | 4.2804 GB | 0.001827 | 0.001502 |
 Same-top-p and RMS Δp move the same direction on every row. Full statistics: [`EVAL.md`](EVAL.md).
 Raw per-run logs: [`eval/logs/`](eval/logs).

   <b>Most quantized weights ship with an adjective. Ours ship with a table.</b>
 </p>
+<p align="center">
+  <b>Head to head with <a href="https://huggingface.co/unsloth/Qwen3-4B-GGUF"><code>unsloth/Qwen3-4B-GGUF</code></a> at the identical
+  2,497,280,800 bytes &mdash; KL divergence from bf16, lower is better:</b><br>
+  Korean <b>&minus;54.4&#8201;%</b> &#183; Code <b>&minus;52.9&#8201;%</b> &#183;
+  Japanese <b>&minus;33.7&#8201;%</b> &#183; English <b>&minus;33.2&#8201;%</b> &#183;
+  Chinese <b>&minus;31.1&#8201;%</b> &#183; Spanish <b>&minus;30.2&#8201;%</b> &#183;
+  Russian <b>&minus;25.0&#8201;%</b> &#183; Arabic <b>&minus;23.2&#8201;%</b> &#183;
+  Thai <b>&minus;21.5&#8201;%</b> &#183; Hindi <b>&minus;20.9&#8201;%</b><br>
+  <sub>Ten axes measured, ten ahead. Seven of them are not in our calibration set.</sub>
+</p>
 ---
 > ### 📚 Collection
 ## `[measured]` Q4_K_M — ten axes, 2,497,280,800 bytes on both sides
+<p align="center">
+  <img src="chart_axes.svg" width="1000"
+       alt="KL divergence by language — ARCHsloth Q4 against unsloth Q4_K_M and UD-Q4_K_XL, ten axes">
+</p>
 KL divergence from the bf16 original. **Lower is better.** `llama-perplexity --kl-divergence`,
 ctx 512, 60 chunks for Korean and English, 40 for the rest. Every ARCHsloth number below comes
 from **one single build**.
+| | script | **ARCHsloth Q4** | unsloth<br>`Q4_K_M` | unsloth<br>`UD-Q4_K_XL` | stock<br>`llama-quantize` | **vs unsloth**<br>same size |
+|---|---|---|---|---|---|---|
+| | file size | **2.4973 GB** | 2.4973 GB | 2.5463 GB | 2.4973 GB | — |
+| Korean | Hangul | **0.024297** | 0.053290 | 0.046637 | 0.081506 | **−54.4 %** |
+| English | Latin | **0.032344** | 0.048443 | 0.043660 | 0.066203 | **−33.2 %** |
+| Spanish | Latin | **0.033679** | 0.048234 | 0.043093 | 0.069169 | **−30.2 %** |
+| Japanese | Kana·Han | **0.034148** | 0.051534 | 0.045666 | 0.075946 | **−33.7 %** |
+| Chinese | Han | **0.033088** | 0.048051 | 0.042510 | 0.073735 | **−31.1 %** |
+| Russian | Cyrillic | **0.032852** | 0.043831 | 0.038792 | 0.065831 | **−25.0 %** |
+| Arabic | Arabic | **0.040871** | 0.053212 | 0.047022 | 0.067085 | **−23.2 %** |
+| Hindi | Devanagari | **0.031617** | 0.039979 | 0.035678 | 0.053405 | **−20.9 %** |
+| Thai | Thai | **0.024561** | 0.031301 | 0.027782 | 0.045149 | **−21.5 %** |
+| **Source code** | — | **0.010191** | 0.021659 | 0.018216 | 0.039165 | **−52.9 %** |
 Ten axes, seven writing systems, ahead on every one — against the same-size file and against
 the larger one. Lowest separation is 3.8 σ (Hindi); highest is 20.4 σ (Korean).
 not.** They are the control, so that a build which buys one language with another has somewhere
 to show it.
+### `[measured]` The whole curve, not one point
+<p align="center">
+  <img src="chart_frontier.svg" width="1000"
+       alt="File size against KL divergence — the ARCHsloth curve sits under the unsloth curve at every size">
+</p>
+Nine of those points are real files: six downloaded from
+[`unsloth/Qwen3-4B-GGUF`](https://huggingface.co/unsloth/Qwen3-4B-GGUF), three built here. Nothing is interpolated.
+Their curve includes the `UD-*-XL` files, which are **larger** than the same-named standard
+file — and our curve is still below theirs. Read it vertically for "same size, closer to the
+original"; read it horizontally for "same quality, smaller file".
 ### `[measured]` Q6_K and Q8_0
 | | size | Korean | English |
 |---|---|---|---|
 | **ARCHsloth Q6_K** | 3.3063 GB | **0.004077** | **0.004022** |
+| unsloth `Q6_K` | 3.3063 GB | 0.006969 | 0.005765 |
+| | | **−41.5 %** | **−30.2 %** |
 | **ARCHsloth Q8_0** | 4.2804 GB | **0.001624** | **0.001371** |
+| unsloth `Q8_0` | 4.2804 GB | 0.001827 | 0.001502 |
+| | | **−11.1 %** | **−8.7 %** |
 Same-top-p and RMS Δp move the same direction on every row. Full statistics: [`EVAL.md`](EVAL.md).
 Raw per-run logs: [`eval/logs/`](eval/logs).

chart_axes.svg ADDED Viewed

chart_frontier.svg ADDED Viewed