You can deploy this model and start earning money today!

np-n/
meditron-7b_Q4_K_M.gguf

Name: meditron-7b_Q4_K_M.gguf
Availability: InStock

Views

No views yet

This model the 4-bit quantized version of the Meditron-7b.Please follow the following instruction to run the model on your device:

There are multiple ways to infer the model. Firstly, let's install llama.cpp and use it for the inference

Install

git clone https://github.com/ggerganov/llama.cpp
!mkdir llama.cpp/build && cd llama.cpp/build && cmake .. && cmake --build . --config Release

Inference

./llama.cpp/build/bin/llama-cli -m ./meditron-7b_Q4_K_M.gguf -cnv -p "You are a helpful assistant"

Here, you can interact with model from your terminal.

Alternatively, we can use python binding of the llama.cpp to run the model on both CPU and GPU.

Install

pip install --no-cache-dir llama-cpp-python==0.2.85 --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu122

Inference on CPU

from llama_cpp import Llama

model_path = "./meditron-7b_Q4_K_M.gguf"
llm = Llama(model_path=model_path, n_threads=8, verbose=False)

prompt = "What should I do when my eyes are dry?"
output = llm(
        prompt=f"<|user|>\n{prompt}<|end|>\n<|assistant|>",
        max_tokens=4096,
        stop=["<|end|>"],
        echo=False,  # Whether to echo the prompt
)
print(output)

Inference on GPU

from llama_cpp import Llama

model_path = "./meditron-7b_Q4_K_M.gguf"
llm = Llama(model_path=model_path, n_threads=8, n_gpu_layers=-1, verbose=False)

prompt = "What should I do when my eyes are dry?"
output = llm(
        prompt=f"<|user|>\n{prompt}<|end|>\n<|assistant|>",
        max_tokens=4096,
        stop=["<|end|>"],
        echo=False,  # Whether to echo the prompt
)
print(output)

np-n/meditron-7b_Q4_K_M.gguf

np-n/
meditron-7b_Q4_K_M.gguf