Publish PrismQuant paper, concise model card, and loader manifest
Browse files- README.md +16 -30
- config.json +13 -0
README.md
CHANGED
|
@@ -9,18 +9,20 @@ tags:
|
|
| 9 |
- gptq
|
| 10 |
- w4a4kv4
|
| 11 |
- custom-code
|
| 12 |
-
-
|
| 13 |
---
|
| 14 |
|
| 15 |
-
# Llama
|
| 16 |
|
| 17 |
-
|
| 18 |
|
| 19 |
-
**
|
| 20 |
|
| 21 |
-
|
| 22 |
|
| 23 |
-
|
|
|
|
|
|
|
| 24 |
|
| 25 |
## Quick start
|
| 26 |
|
|
@@ -28,41 +30,25 @@ Released checkpoints for [PrismQuant](https://github.com/ForeverBlue816/PrismQua
|
|
| 28 |
git clone --depth 1 https://github.com/ForeverBlue816/PrismQuant.git
|
| 29 |
cd PrismQuant
|
| 30 |
pip install -e .
|
| 31 |
-
prismquant models
|
| 32 |
```
|
| 33 |
|
| 34 |
```python
|
| 35 |
from prismquant import load_model
|
| 36 |
|
| 37 |
-
loaded = load_model("llama31_70b")
|
| 38 |
print(loaded.generate("The key idea behind quantization is", max_new_tokens=64))
|
| 39 |
```
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
-
|
| 44 |
|
| 45 |
-
|
| 46 |
|
| 47 |
-
|
| 48 |
|
| 49 |
-
|
| 50 |
|
| 51 |
-
|
| 52 |
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
- `checkpoints/<model>/<checkpoint>/layer_XX.pt`: the folded, GPTQ-quantized decoder linear tensors.
|
| 56 |
-
- `DONE.json` and `gptq_audit.csv`: quantization settings and provenance.
|
| 57 |
-
- `rotations/<model>/...`: the calibrated factors used by the runtime.
|
| 58 |
-
|
| 59 |
-
The `nar` strings in filenames are the original internal name of PrismQuant and are retained for compatibility. Existing weight files are unchanged by this public-release update. The two smaller Llama repositories additionally include the previously omitted `k=8` down-projection factors.
|
| 60 |
-
|
| 61 |
-
## Available checkpoints
|
| 62 |
-
|
| 63 |
-
Download sizes below cover one variant and its required factors, excluding the separately downloaded base model.
|
| 64 |
-
|
| 65 |
-
| Model key | Checkpoint | Weight protocol | Download (GB) |
|
| 66 |
-
| --- | --- | --- | --- |
|
| 67 |
-
| `llama31_70b` | `gptq_nar_k8_seed0` | default | 273.92 |
|
| 68 |
-
| `llama31_70b` | `gptq_nar_kmax_seed0` | default | 275.90 |
|
|
|
|
| 9 |
- gptq
|
| 10 |
- w4a4kv4
|
| 11 |
- custom-code
|
| 12 |
+
- arxiv:2609.32429
|
| 13 |
---
|
| 14 |
|
| 15 |
+
# PrismQuant-Llama-3.1-70B
|
| 16 |
|
| 17 |
+
[Paper](https://arxiv.org/abs/2609.32429) · [Code](https://github.com/ForeverBlue816/PrismQuant) · [Loading guide](https://github.com/ForeverBlue816/PrismQuant/blob/main/docs/models.md)
|
| 18 |
|
| 19 |
+
Official checkpoints for **PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers**. PrismQuant aligns dominant activation directions with the constant directions of asymmetric quantization groups.
|
| 20 |
|
| 21 |
+
## Models
|
| 22 |
|
| 23 |
+
| Model key | Base model | Quantization |
|
| 24 |
+
| --- | --- | --- |
|
| 25 |
+
| `llama31_70b` | [unsloth/Meta-Llama-3.1-70B](https://huggingface.co/unsloth/Meta-Llama-3.1-70B) | W4A4KV4 |
|
| 26 |
|
| 27 |
## Quick start
|
| 28 |
|
|
|
|
| 30 |
git clone --depth 1 https://github.com/ForeverBlue816/PrismQuant.git
|
| 31 |
cd PrismQuant
|
| 32 |
pip install -e .
|
|
|
|
| 33 |
```
|
| 34 |
|
| 35 |
```python
|
| 36 |
from prismquant import load_model
|
| 37 |
|
| 38 |
+
loaded = load_model("llama31_70b", device_map="balanced")
|
| 39 |
print(loaded.generate("The key idea behind quantization is", max_new_tokens=64))
|
| 40 |
```
|
| 41 |
|
| 42 |
+
The loader selects the default PrismQuant checkpoint, downloads its required files at a pinned revision, and obtains the matching base model and tokenizer. These are base models for text completion.
|
| 43 |
|
| 44 |
+
## Checkpoint format
|
| 45 |
|
| 46 |
+
The GPTQ INT4 values are stored as **dequantized floating-point tensors**. The reference runtime applies activation and KV quantization while retaining floating-point storage. Use the PrismQuant loader; this repository is not a standalone Transformers checkpoint or a packed INT4 serving model.
|
| 47 |
|
| 48 |
+
This fp32 reference requires roughly **280 GB for model parameters**, plus runtime overhead. Use sufficient aggregate GPU memory and, if needed, `max_memory={...}` with balanced placement.
|
| 49 |
|
| 50 |
+
`config.json` declares the PrismQuant artifact format and default models. `checkpoints/` contains decoder weights; `rotations/` contains the factors required by the loader.
|
| 51 |
|
| 52 |
+
## License
|
| 53 |
|
| 54 |
+
**Built with Llama.** The upstream Llama Community License applies. See [LICENSE](LICENSE), [NOTICE](NOTICE) and [Acceptable Use Policy](USE_POLICY.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
config.json
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"format": "prismquant-reference-v1",
|
| 3 |
+
"paper": "https://arxiv.org/abs/2609.32429",
|
| 4 |
+
"models": {
|
| 5 |
+
"llama31_70b": {
|
| 6 |
+
"base_model": "unsloth/Meta-Llama-3.1-70B",
|
| 7 |
+
"default_checkpoint": "gptq_nar_kmax_seed0",
|
| 8 |
+
"architecture": "llama",
|
| 9 |
+
"quantization": "W4A4KV4",
|
| 10 |
+
"storage_dtype": "float32"
|
| 11 |
+
}
|
| 12 |
+
}
|
| 13 |
+
}
|