Sumitc13

/

sarvam-30b-GGUF

like 7

The PR for gguf support on llama cpp is still open. How to use this?

The PR for gguf support on llama cpp is still open. How to use this?

The GGUF files are ready to download from this repo. However, since the sarvam_moe architecture isn't merged into official llama.cpp yet, you will need to build from my fork to run them.

git clone https://github.com/sumitchatterjee13/llama.cpp
cd llama.cpp
git checkout add-sarvam-moe
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
Then download any GGUF from this repo and run:

./build/bin/llama-cli -m sarvam-30B-Q6_K.gguf -p "Your prompt here" -n 512 -ngl 99

Replace -DGGML_CUDA=ON with -DGGML_VULKAN=ON or -DGGML_METAL=ON depending on your GPU. Once the PR is merged, any standard llama.cpp build will work.

Support this. We need https://huggingface.co/aoxo/sarvam-30b-uncensored in gguf, still no working version of uncensored for llama.cpp
But your censored version works with latest llama.cpp, already some progress.

Support this. We need https://huggingface.co/aoxo/sarvam-30b-uncensored in gguf, still no working version of uncensored for llama.cpp
But your censored version works with latest llama.cpp, already some progress.

Hey, since the llama.cpp PR is merged now, anyone can quantize this with a fresh checkout. Should take about an hour on a decent GPU.

Quick recipe:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j

huggingface-cli download aoxo/sarvam-30b-uncensored --local-dir ./model
python convert_hf_to_gguf.py ./model --outtype bf16
./build/bin/llama-quantize ./model/*-BF16.gguf sarvam-uncensored-Q6_K.gguf Q6_K

Swap -DGGML_CUDA=ON for -DGGML_METAL=ON on Mac or -DGGML_VULKAN=ON for other GPUs.

I am not planning to host the uncensored one myself, but the pipeline is the same as the base model so it should just work. Let me know if you hit any snags.

Thanks for the recipe, I will try.

If your system is underpowered to quantize into a gguf, you simply put in a gguf request to mradermacher's Hf page. they will do it for you within a reasonable time.

https://huggingface.co/mradermacher/sarvam-30b-uncensored-GGUF
Mradermacher made uncensored GGUF today and it works in llama latest release!

For some reason thinking doesn't work there for me, it jumps straight to text generation without any thinking tags. But thinking works just fine in this censored GGUF. Any idea how to restore thinking in uncensored?

Tried this as llama.cpp param, it did not enable thinking --chat-template-kwargs "{"enable_thinking": true}"

I checked the chat template in the source repo — it's identical to the base Sarvam template, including the enable_thinking / <|nothink|> machinery. So in theory it should work.

Two things to check:

First, does mradermacher's GGUF actually have the chat template embedded? Run this and look for the template in the metadata:

llama-cli -m <their.gguf> --no-warmup 2>&1 | grep "chat_template"

If nothing shows up or it's a generic default, that's your problem — kwargs can't enable thinking if there's no template to read them.

Second, see if the model itself still knows how to think. Force the prefix manually:

llama-cli -m <their.gguf> -p "<|start_of_turn|><|user|>\nHello<|end_of_turn|>\n<|start_of_turn|><|assistant|>\n"

If it generates reasoning after , the model's fine and it's just a template/flag issue. If it ignores the prefix and jumps straight to answering, abliteration probably damaged the thinking behavior — refusal-direction surgery sometimes affects nearby behaviors and there's nothing you can do at inference time.

Also your earlier kwargs command had broken quoting. Use single quotes around the JSON:

--chat-template-kwargs '{"enable_thinking": true}'

The model has some chat template, it says this at startup:

init: chat template, example_format: '[@BOS@]
<|start_of_turn|><|system|>
You are a helpful assistant<|end_of_turn|>
<|start_of_turn|><|user|>
Hello<|end_of_turn|>
<|start_of_turn|><|assistant|>
<think></think>
Hi there<|end_of_turn|>
<|start_of_turn|><|user|>
How are you?<|end_of_turn|>
<|start_of_turn|><|assistant|>
'
srv          init: init: chat template, thinking = 1

That llama-cli command with -p "<|start_of_turn|><|user|>\nHello<|end_of_turn|>\n<|start_of_turn|><|assistant|>\n"
does not produce thinking - just straight text generation

I also tried passing modified -p strings based on that template above, but it did not work. Including instructions about thinking in system prompt did not help either.

But i've found something else, if I pass a custom template via --chat-template-file sarvamtemplate.txt --chat-template-kwargs "{\"enable_thinking\": true}"
it starts working - the whole thinking process. But not on every try, about 2/3 of tries it starts thinking, otherwise just generating. But I can click Retry in most cases, so it's not a big problem. The only problem now is that it does not include starting <think> tag for some reason like your gguf - so I have to remove thinking text from the answer after every turn or use frontend that can still detect it without starting thinking tag (kobold frontend does support that).

And here's the custom template that worked for enabling thinking:

{% for message in messages %}
{% if message['role'] == 'user' %}
{{ '<|start_of_turn|>user\n' + message['content'] + '<|end_of_turn|>\n' }}
{% elif message['role'] == 'assistant' %}
{{ '<|start_of_turn|>assistant\n' + message['content'] + '<|end_of_turn|>\n' }}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}
{{ '<|start_of_turn|>assistant\n<think>\n' }}
{% endif %}

Nice debugging. That confirms it — the template is fine (thinking = 1 proves llama.cpp detected it), so the issue is the model itself. The base Sarvam-30B emits ... reliably after the assistant turn marker; the abliterated version has lost some of that reliability. That 2/3 success rate is the abliteration damage I was guessing about — refusal-direction surgery affecting nearby behaviors.

Your workaround (prefilling \n in the generation prompt) is exactly the right call. It's basically forcing the model into "thinking mode" by giving it no choice but to continue from inside a block.

The missing opening tag in the output is expected — llama.cpp's reasoning parser strips it on output because it assumes the model generated it. Since you prefilled it manually, it never appears in the output stream. Two ways to deal with that:

1. Use a frontend that detects reasoning without the opening tag (you mentioned kobold already does this — that's the easiest path)

2. Pre-pend \n to the response manually before displaying

For the unreliable 1/3 — that's a model-level issue and nothing template tricks will fully fix. Sampling parameter tweaks might help slightly (lower temperature, higher repetition penalty might push it toward the trained pattern), but the real fix would be a different abliteration approach by the model author.

I figured it out with the help of Claude.
--chat-template-kwargs was not needed, it was only making everything worse
what was needed is detection of thinking from template - it looks for reasoning_content within template
when llama detects that, it writes
srv init: init: chat template, thinking = 1
in the console
and it also needed correct reasoning-format inside llama to wrap the thinking block

so the full fix:
command line parameters for llama.cpp (only these 2):
--chat-template-file sarvamClaudeTemplate.txt --reasoning-format deepseek-legacy

and then the new template:

{{- '[@BOS@]\n' -}}
{%- for m in messages -%}
    {%- if m.role == 'system' -%}
        {{- '<|start_of_turn|><|system|>\n' + m.content + '<|end_of_turn|>\n' -}}
    {%- elif m.role == 'user' -%}
        {{- '<|start_of_turn|><|user|>\n' + m.content + '<|end_of_turn|>\n' -}}
    {%- elif m.role == 'assistant' -%}
        {{- '<|start_of_turn|><|assistant|>\n' -}}
        {%- if m.reasoning_content -%}
            {{- '<think>\n' + m.reasoning_content + '\n</think>\n' -}}
        {%- endif -%}
        {{- m.content + '<|end_of_turn|>\n' -}}
    {%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
    {{- '<|start_of_turn|><|assistant|>\n<think>\nLet me think through this carefully.\n' -}}
{%- endif -%}

now it always reasons and outputs reasoning text correctly within reasoning block in frontend
generated text is also printed correctly, outside the thinking block