Because Llama 3.2 features a massive native 128k context window, running it in low-RAM environments (like a standard 12GB Google Colab CPU instance) without limits will result in an instant Out-Of-Memory (OOM) /killed crash.

You must cap the context window (-c) and disable standard memory mapping (--no-mmap) to keep the KV cache safe within memory limits.

💻 How to Run Directly via llama.cpp CLI

You can stream and execute this model directly from the Hugging Face Hub using the following command:

bash

# Clone and build llama.cpp first, then run:
./build/bin/llama-cli \
  -hf KrishnaAgarwal-123/Llama-3.2-3B-Instruct-GGUF:Llama-3.2-3B-Instruct-Q4_K_M.gguf \
  --no-mmap \
  -c 2048 \
  -p "Introduce yourself in three sentences."

Explanations of Flags Used:

-hf: Pulls and caches the specified GGUF file directly from this Hugging Face repository.
--no-mmap: Prevents violent virtual memory allocation spikes.
-c 2048: Forcefully limits the context cache size to roughly 220 MiB (instead of the 11.1 GiB default).

KrishnaAgarwal-123/Llama-3.2-3B-Instruct-Q4_K_M-GGUF

Llama-3.2-3B-Instruct-Q4_K_M.gguf

🚀 Memory Optimization Notice (Google Colab / Low-RAM Devices)

💻 How to Run Directly via llama.cpp CLI

Explanations of Flags Used:

KrishnaAgarwal-123/
Llama-3.2-3B-Instruct-Q4_K_M-GGUF