Quantized versions of SmolLM3-3B running on consumer hardware under 6 GB VRAM, with custom matrix kernels written in Nim for hot-path matrix-multiply operations.

Full write-up: Run LLMs on a 6 GB Laptop GPU & CPU — Medium

Memory footprint is reduced ~75% from the FP16 baseline.

Hardware used: NVIDIA GPU with 6 GB VRAM, Windows, CUDA 12.4.

Requires llama.cpp 0.3.14+ built with CUDA support for GPU runs.

The repo ships two companion DLLs — libsmolkernels_q4.dll and libsmolkernels_q8.dll — written in Nim, compiled with -O3 and ORC memory management. They implement a custom dequantization + matrix-multiply path (mulQ4Mat / mulQ8Mat) that is dynamically loaded by a patched main.cpp at runtime based on the detected model type. llama-cli.exe detects "q4_k_m" in filename → loads libsmolkernels_q4.dll → mulQ4Mat() llama-cli.exe detects "q8_0" in filename → loads libsmolkernels_q8.dll → mulQ8Mat() Full build instructions, kernel source code, and the main.cpp patch are in the Medium article linked above.

This work is the research foundation for QBench CLI — a C++ command-line tool that automates LLM quantization workflows, model selection, and hardware compatibility checks using the same quantization and kernel techniques developed here.