K2-Horizon-375B-A23B-MXFP4

Model Overview

• Model Architecture: K2HorizonForCausalLM
  • Input: Text
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI355 / MI350 (gfx950)
• ROCm: 7.2.3
• PyTorch: 2.12.0
• Transformers: 5.17.0
• Operating System(s): Linux
• Inference Engine: vLLM
• Model Optimizer: AMD-Quark (v0.12.0)
  • Weight quantization: routed MOE-only, OCP MXFP4, Static
  • Activation quantization: routed MOE-only, OCP MXFP4, Dynamic

Model Quantization

Quantized from IFM/K2-Horizon-375B-A23B with AMD Quark. The routed MoE experts are quantized to OCP MXFP4, using static weight quantization and dynamic activation quantization. All remaining modules are kept in their original precision.

Quantization script

from quark.torch import LLMTemplate, ModelQuantizer

MODEL_DIR = "IFM/K2-Horizon-375B-A23B"
OUTPUT_DIR = "amd/K2-Horizon-375B-A23B-MXFP4"

template = LLMTemplate(
    model_type="k2moe375B",
    kv_layers_name=["*self_attn.k_proj", "*self_attn.v_proj"],
    q_layer_name="*self_attn.q_proj",
    exclude_layers_name=[
        "lm_head",
        "model.embed_tokens",
        "model.norm",
        "*layernorm",
        "*self_attn*",
        "*mlp.gate",
        "*mlp.shared_experts*",
        "model.layers.0.mlp*",
        "model.layers.1.mlp*",
        "model.layers.2.mlp*",
    ],
)
LLMTemplate.register_template(template)

quant_config = template.get_config(scheme="mxfp4")

quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=MODEL_DIR,
    save_path=OUTPUT_DIR,
)

Deployment

Use with vLLM

This model can be deployed efficiently using the vLLM backend based on the Docker image alexsu52/vllm-openai-rocm:v0.30.0-k2horizon.

Evaluation

The model was evaluated on GSM8K (5-shot) and GPQA Diamond (0-shot) using the vLLM framework.

Accuracy

Benchmark	IFM/K2-Horizon-375B-A23B	amd/K2-Horizon-375B-A23B-MXFP4 (this model)	Recovery
GSM8K (5-shot, strict-matcht)	92.42%	91.89%	99.43%
GPQA Diamond (0-shot, thinking)	87.3%	86.79%	99.42%

GSM8K is measured with lm-eval and GPQA Diamond with sgl-eval. For each benchmark the base and quantized models are run with the same framework, settings, and host.

Reproduction

The evaluation runs against a vLLM server, using the Docker image alexsu52/vllm-openai-rocm:v0.30.0-k2horizon.

Launching server

export VLLM_ROCM_USE_AITER=1

vllm serve amd/K2-Horizon-375B-A23B-MXFP4 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --dtype auto \
  --gpu-memory-utilization 0.9 \
  --generation-config auto \
  --reasoning-parser k2_horizon \
  --chat-template-content-format string \
  --enable-auto-tool-choice \
  --tool-call-parser k2_horizon \
  --attention-backend ROCM_AITER_UNIFIED_ATTN

The context length must accommodate GPQA Diamond's 65536-token generation budget in thinking mode with reasoning effort high; GSM8K alone runs comfortably at --max-model-len 8192.

Evaluating GSM8K in a new terminal

lm_eval --model local-completions \
    --model_args model=amd/K2-Horizon-375B-A23B-MXFP4,base_url=http://localhost:8000/v1/completions,num_concurrent=32,max_retries=5,tokenized_requests=False \
    --tasks gsm8k --num_fewshot 5 --seed 0,1234,1234,1234 --gen_kwargs "temperature=0,top_p=1"

Evaluating GPQA Diamond

pip install sgl-eval
sgl-eval run gpqa \
    --model amd/K2-Horizon-375B-A23B-MXFP4 \
    --api-key EMPTY \
    --base-url http://localhost:8000/v1 \
    --num-threads 8 \
    --n-repeats 1 \
    --max-tokens 65536 \
    --temperature 1.0 \
    --top-p 0.95 \
    --thinking \
    --reasoning-effort high