Local GGUF Chat (Q2_K_L) — Run on CPU (16GB RAM)

1. Download a single GGUF quantized weight (*Q2_K_L.gguf) from Hugging Face by pasting your token into a file.
2. Run a small local Flask chat UI that talks to the model using llama-cpp-python.

Files

• download_model.py — edit & paste your HF token, then run to download only the Q2_K_L gguf file.
• app.py — Flask server + model loader + chat endpoints.
• templates/index.html — Chat UI (ChatGPT-like).
• requirements.txt — Python dependencies.

Requirements

• Python 3.10.9 (recommend)
• ~16 GB RAM (CPU-only); speed depends on quantization & CPU cores.

Quick start

1. Create & activate a virtual environment:
   bash

   1python -m venv oss_env 2# Windows 3oss_env\Scripts\activate 4# Linux / macOS 5source oss_env/bin/activate 6 7
2. Install Python dependencies: pip install -r requirements.txt
3. Edit download_model.py: Paste your Hugging Face token into HUGGINGFACE_TOKEN. If your model repo is different, update REPO_ID.
4. Download the Q2_K_L GGUF: python download_model.py The script will print the full path to the downloaded .gguf file.
5. (Optional) Edit app.py: If you want to explicitly set the exact .gguf path, set MODEL_PATH at top of app.py. Otherwise app.py will auto-detect the first .gguf under models/.
6. Run the Flask app: python app.py

Open http://localhost:5000

1. If need you can run the inference.py code for the single stage demo without chat loop

Trouble shooting

• May be you can face issues while installing the libraries in your laptop to solve them follow the below steps
• Go through this link and install the Visual Studio Build Tools in your laptop https://visualstudio.microsoft.com/visual-cpp-build-tools/?utm_source=chatgpt.com
• After completion of download select Desktop development with C++ workload the Visual Studio Build Tools
• After that check the boxes of MSVC v142/143, C++ CMake tools for Windows, Windows SDK
• Wait for 40 minutes for to installing all the packages
• Then once again run the pip install -r requirements.txt command then all the libraries will download without any errors or issues.