Views
No views yet
llama.cpp.llama-cpp-python and other compatible loaders. It contains the merged weights for local, low-resource deployment.llama-cpp-python1from llama_cpp import Llama
2
3# Load the model
4llm = Llama(
5 model_path="merged_model.gguf",
6 n_ctx=2048, # Context window
7 n_gpu_layers=0 # Increase this to offload layers to GPU
8)
9
10# Generate completion
11output = llm(
12 prompt="### Human: Hello!\n### Assistant:",
13 max_tokens=256,
14 stop=["### Human:"],
15 temperature=0.7
16)
17print(output["choices"][0]["text"])