Views
No views yet
<|memory_start|> and
<|memory_end|>.Running these checkpoints requires the LCLM codebase: https://github.com/LeonLixyz/LCLM. Standardtransformers.AutoModel/vllm.LLMwill not load this format on its own.
1from latent_context import LCLM
2
3model = LCLM.from_pretrained("latent-context/0.6b-4b-LCLM-16x")
4
5prompt = (
6 "<|memory_start|>"
7 "<long document, code, or text to compress>"
8 "<|memory_end|> "
9 "Summarize the document above."
10)
11# model.generate(...) — see latent_context/inference/hf.py.pt file on disk. Running both in
one process OOMs — vLLM grabs all GPU memory at init, leaving none
for the HF encoder.1# Step 1: HF encoder over a jsonl of prompts → embeds.pt
2python -m inference.vllm_inference.encode --checkpoint latent-context/0.6b-4b-LCLM-16x --prompts-jsonl prompts.jsonl --out embeds.pt
3
4# Step 2: vLLM decoder reads embeds.pt → completions.jsonl
5python -m inference.vllm_inference.decode --checkpoint latent-context/0.6b-4b-LCLM-16x --embeds-pt embeds.pt --out completions.jsonlinference/examples/README.md in the codebase for the
prompts.jsonl schema and an end-to-end RULER NIAH eval driver.| field | value |
|---|---|
| encoder | Qwen/Qwen3-Embedding-0.6B |
| decoder | Qwen/Qwen3-4B-Instruct-2507 |
| compression_ratio | 16 |
| encoder_window_size | 1024 |
| pooling | mean |
| encoder_mask_type | causal |
| boundary_overlap | 0 |
| adapter_type | mlp |