bounded-moe
Bounded-memory inference experiments for running large Mixture-of-Experts models on consumer hardware.
Can we run a large MoE model without letting its expert weights consume most of system RAM?
bounded-moe is an experimental project exploring SSD-backed expert storage and bounded-memory caching for llama.cpp / GGML.
The goal is to make large MoE models more practical on memory-constrained consumer PCs by keeping only a controlled working set of experts resident in memory.
Architecture
1 MoE Router
2 │
3 ▼
4 Requested Experts
5 │
6 ▼
7 ┌───────────────────┐
8 │ Bounded Expert │
9 │ Cache │
10 └─────────┬─────────┘
11 hit │ miss
12 │
13 ▼
14 External Expert Storage
15 │
16 SSD
17 │
18 ▼
19 RAM / VRAM
20 │
21 ▼
22 Compute
In simplified form:
SSD → Bounded Expert Cache → RAM/VRAM → Inference
Instead of allowing all expert weights to remain resident in system memory, the runtime resolves and caches the experts required by MoE routing.
Test Hardware
Current development and testing is performed on consumer hardware:
- GPU: NVIDIA GeForce RTX 4060 8 GB
- System RAM: 32 GB
- OS: Windows 11
- Runtime: llama.cpp / GGML
- Model class: ~35B Mixture-of-Experts, ~3B active parameters
The project specifically explores scenarios where model weights, applications, and the operating system compete for limited RAM.
Implemented Research
The project currently includes experiments around:
- MoE routing tracing
- External expert storage
- Bounded expert caching
- Cache hit/miss accounting
- Expert pin/unpin lifecycle
- Safe cache eviction
- Direct expert reads
- Resolver indirection
- Cache-size experiments
- Working-set / RAM measurements
- SSD-backed inference
- Performance and correctness validation
Current Status
The architecture is experimental.
The bounded-storage path works and demonstrates that expert memory can be managed independently from normal full-model residency.
The main research challenge is now performance.
Normal memory-mapped inference is significantly faster than the current experimental SSD-backed expert path. Current work therefore focuses on reducing expert-resolution and storage latency while preserving the bounded-memory property.
This repository should currently be considered an engineering/research prototype, not a production inference runtime.
Why?
Large MoE models are interesting for consumer hardware because only a subset of their parameters is active for each token.
However, inactive experts can still consume significant system memory.
This project asks a slightly different question:
What if model capacity could be much larger than the amount of RAM we're willing to dedicate to inference?
Rather than treating available RAM as the hard limit, bounded-moe explores using a hierarchy of:
SSD → bounded RAM cache → GPU → compute
while exploiting MoE routing locality to keep the frequently requested experts close to compute.
Future Research
Planned experiments include:
- asynchronous expert prefetching
- predictive prefetch based on routing behavior
- overlapping SSD I/O with GPU computation
- smarter eviction policies
- cache locality analysis
- larger-context testing
- Windows I/O optimization
- reducing resolver overhead
- dense-model layer/block streaming experiments
The last item is particularly interesting: some of the infrastructure may eventually be generalized beyond MoE models into a bounded-memory inference runtime for dense models.
Source Code
The implementation, experiments, benchmark notes, and development history are available on GitHub:
Feedback
Feedback is very welcome, particularly from people working with:
llama.cpp · GGML · MoE routing · model offloading · caching · mmap · SSD I/O · CUDA · inference optimization
If you've experimented with similar SSD-backed or bounded-memory inference architectures, I'd especially like to hear about approaches for hiding cache-miss and storage latency.
Status: Experimental / Research Prototype
