Views
No views yet
google/gemma-4-8b-it in a deterministic, Rust-backed
orchestrator designed for the Edge: PagedAttention with Copy-on-Write
prefix sharing, a Zero-Trust MCP gatekeeper, lock-free hardware
telemetry, and a continuous-batching scheduler that survives
oversubscribed workloads without OOM. The repository ships
pre-compiled wheels for Linux (x86_64 / aarch64), macOS (x86_64 /
aarch64), and Windows (x86_64); end users never touch a Rust toolchain.| Quantization | Use Case | VRAM (PagedAttention Est.) | Target Hardware |
|---|---|---|---|
Q4_K_M | Balanced laptop / on-device assistant; best size-quality trade-off for interactive agents. | ~5.5 GB at 32k ctx, ~7 GB at 128k ctx with CoW prefix sharing. | Apple M-series (8–16 GB unified memory), NVIDIA RTX 4060 / 4070 mobile, ROCm 7900M. |
Q8_0 | Server-side accuracy; near-FP16 fidelity for evaluation, distillation, or compliance-grade inference. | ~9 GB at 32k ctx, ~11 GB at 128k ctx. | NVIDIA RTX 4090 / 5090, A100 40 GB, H100 PCIe slice. |
AWQ | Pure GPU throughput; activation-aware 4-bit weights for high-QPS deployments behind the continuous-batching scheduler. | ~6 GB at 32k ctx with batched KV-cache reuse. | NVIDIA L4 / L40S, RTX 5080, Jetson AGX Orin 64 GB. |