How do I combine the file parts?
#1
by Frank2478 - opened
Hey, I tried to combine "Behemoth-123B-v2f-Q4_K_M-00001-of-00002.gguf" and "Behemoth-123B-v2f-Q4_K_M-00002-of-00002.gguf" by using the following commands:
cat Behemoth-123B-v2f-Q4_K_M-00001-of-00002.gguf Behemoth-123B-v2f-Q4_K_M-00002-of-00002.gguf > Behemoth-123B-v2f-Q4_K_M.gguf
But it returns the following error when I try loading Behemoth-123B-v2f-Q4_K_M.gguf
Traceback (most recent call last):
File "/workspace/text-generation-webui/modules/ui_model_menu.py", line 232, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/modules/models.py", line 93, in load_model
output = load_func_map[loader](model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/modules/models.py", line 278, in llamacpp_loader
model, tokenizer = LlamaCppModel.from_pretrained(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/modules/llamacpp_model.py", line 85, in from_pretrained
result.model = Llama(**params)
^^^^^^^^^^^^^^^
File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 372, in init
_LlamaModel(
File "/workspace/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp_cuda/_internals.py", line 55, in init
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: models/Behemoth-123B-v2f-Q4_K_M.gguf
I assumed that I needed to follow the instructions here to join the files together. I am using oobabooga/text-generation-webui on runpod.
Point koboldcpp at the first part with part 2 in the same dir.
On Runpod there is a behemoth specific template which will get you completely setup automatically.
Thanks, but is there a solution for text-generation-webui?