# pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
repo_id="Knixee/Atlas-nms-v2",
filename="Atlas-2.0-Q4_K_M.gguf", # Or Q4_K_S, Q8_0, BF16 
)

output = llm("User: Hello!\nAssistant:", max_tokens=512)
print(output["choices"][0]["text"])

AWQ Version (via vLLM)

To launch 4bit AWQ version you need to download this folder

bash

# pip install vllm

vllm serve Knixee/Atlas-nms-v2 \
  --subfolder Atlas-2.0-AWQ-4bit \
  --served-model-name Atlas-nms-v2 \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.88 \
  --max-num-seqs 6 \
  --port 8148

API Request Example:

bash

curl http://localhost:8148/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Atlas-nms-v2", "messages": [{"role": "user", "content": "Hello!"}]}'

Knixee/Atlas-nms-v2

Atlas

Serving GGUF & AWQ Models with vLLM

GGUF Version (via llama.cpp)

AWQ Version (via vLLM)

Contacts

Knixee/
Atlas-nms-v2