Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf chatpdflocal/llama3.1-8b-gguf:# Run inference directly in the terminal:
llama cli -hf chatpdflocal/llama3.1-8b-gguf:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf chatpdflocal/llama3.1-8b-gguf:# Run inference directly in the terminal:
./llama-cli -hf chatpdflocal/llama3.1-8b-gguf:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf chatpdflocal/llama3.1-8b-gguf:# Run inference directly in the terminal:
./build/bin/llama-cli -hf chatpdflocal/llama3.1-8b-gguf:Use Docker
docker model run hf.co/chatpdflocal/llama3.1-8b-gguf:Llama3.1 8b Instruct GGUF format models which can be runned on the PCs of MacOS, Windows or Linux, Cell phones and smaller devices.
This repo focuses on the available and excellent tiny LLMs which can be easily runned for chatting PDFs on MacOS, balancing the LLM's effect and inference speed.
Useful local intelligent documents assistant AI tools:
MyDocs is a desktop App which supports Windows and MacOS, and especially for professionals who demand absolute privacy. 100% local processing ensuring total documents sovereignty and zero-cloud dependency.
If you are using Zotero for managing and reading your personal PDFs, PapersGPT is a free plugin which can assist you to chat PDFs effectively by your local llama3.1.
You can download the beautiful ChatPDFLocal MacOS app from here, load one or batch PDF files at will, and quickly experience the effect of the model through chat reading.
The default model used by local LLM is ggml-model-Q3_K_M.gguf, you can also load any customerized open source model that suits your Mac configuration size by inputing huggingface repo.
Enjoy, thank you!
- Downloads last month
- 292
2-bit
3-bit
4-bit
8-bit
16-bit
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf chatpdflocal/llama3.1-8b-gguf:# Run inference directly in the terminal: llama cli -hf chatpdflocal/llama3.1-8b-gguf: