Views
No views yet
samuelfaj/distill-E4B-it-4-bit,
for fast, accurate local inference on Apple Silicon. Distilled for CLI / command-output
compression → single-line {"response": <str>, "confidence": <int 0-100>}.✅ Recommended build. Upgraded from plain 4-bit to a Q4_K_M-style mixed recipe (lm_head + sensitive v_proj/down_proj at 6-bit, rest 4-bit). Pair it with the constrained decoding snippet below for near-perfect format compliance.
| Field | Value |
|---|---|
| Base | google/gemma-4-E4B-it |
| Adapter | QLoRA r=64, α=128 (published root, sha256 a229d12c…) |
| Quantization | mixed mixed_4_6 (4-bit base, 6-bit lm_head + sensitive layers) — 4.65 bits/weight |
| Group size | 64 |
| Size on disk | ~4.0 GB |
| Peak memory | ~4.4 GB |
eval_gold_v1 — 130 held-out CLI-compression cases. Decode: greedy.| Build | P1 accuracy | JSON compliance | json_extraction |
|---|---|---|---|
| plain 4-bit (old) | 64.6% | 89.2% | 37.5% |
| mixed 4-bit | 67.7% | 92.3% | 37.5% |
| mixed 4-bit + constrained decode (this) | 69.2% | 99.2% | 56.2% |
| fp16 reference (ceiling) | 72.3% | 98.5% | 75.0% |
confidence as a soft signal.{"response","confidence"} JSON contract. Prefilling the
opening of that object forces the schema and suppresses any reasoning preamble — this is
what drives JSON compliance from 92% → 99%.pip install mlx-lm1import json
2from mlx_lm import load, generate
3
4model, tok = load("samuelfaj/distill-E4B-it-4-bit-MLX")
5
6def ask(prompt: str) -> dict:
7 text = tok.apply_chat_template(
8 [{"role": "user", "content": prompt}],
9 add_generation_prompt=True, tokenize=False,
10 )
11 prefix = '{"response": "' # constrained decode: pin the contract
12 raw = prefix + generate(model, tok, prompt=text + prefix, max_tokens=800, verbose=False)
13 return json.loads(raw)
14
15print(ask('Did the tests pass? Return PASS or FAIL, then failing tests.\n\n'
16 'PASS src/auth.test.ts\nFAIL src/queue.test.ts\n1 passed, 1 failed.'))
17# {'response': 'FAIL src/queue.test.ts', 'confidence': 98}1python -m mlx_lm generate --model samuelfaj/distill-E4B-it-4-bit-MLX \
2 --prompt "Is this safe? SAFE/REVIEW/UNSAFE.\n\nDROP TABLE users;" --max-tokens 256