A fine-tune of NVIDIA's Nemotron-3-Nano-30B-A3B (31.6B-parameter hybrid Mamba-2 + MoE + Attention, thinking-capable) that does one thing: turn rough, underspecified prompts into detailed, structured prompts ready to hand to any AI model.

This is the exact model serving koleslaw.ai in production — try it there first if you want to see what it does before downloading 17 GiB. The IQ4_NL quant runs on a single 24GB GPU (RTX 3090 / 4090, L4) with room to spare.

Read the full write-up: My "4-bit" quant was 6.2 bits per weight — how the IQ4_NL below was made, why the usual K-quants balloon on this architecture, and the eval that gated it into production. First post in a series on serving this model; the rest lands weekly on the Koleslaw blog.

What it does

Give it a lazy prompt; it returns a structured one with labeled sections — ROLE / OBJECTIVE / CONTEXT / DELIVERABLES / CONSTRAINTS / OUTPUT FORMAT — with bracketed <fill-in> placeholders where it would otherwise have to invent facts, and an instruction to the answering model to state assumptions rather than guess.

Real example — input: "write a script to back up my postgres database" (unedited output from the IQ4_NL quant in production)

ROLE
Act as a senior DevOps engineer with deep expertise in PostgreSQL administration, Linux bash scripting, and database automation. You are expected to provide polished, production-ready code and clear explanations.

OBJECTIVE
Generate a complete, reusable backup script for a PostgreSQL database that automates the backup process with logging, error handling, compression, and retention management.

SETUP PARAMETERS — use the following placeholder variables (marked clearly) instead of hardcoding real values. Instruct the model to make reasonable default assumptions where no value is provided:
- `DB_HOST`: <host_ip_or_name>
- `DB_PORT`: 5432
- `DB_NAME`: your_database_name
- `BACKUP_USER`: your_db_username
- `BACKUP_PASS`: your_db_password (handle securely — note recommended approach for production)
- `BACKUP_PATH`: /path/to/backup/directory/<date>/database_name
- `RETENTION_DAYS`: 7

**TASK REQUIREMENTS**
Produce the following in a single cohesive response:

1. **SQL Backup Command** using `pg_dump` with all relevant flags (compression, custom format if applicable), properly quoting/protecting credentials.

2. **Full Bash Script** that:
   - Checks the database connection before starting
   - Creates a timestamped backup directory and filenames
   - Executes the backup command safely
   - Compresses the output using gzip (or notes when compression would add value)
   - Logs every step to a logfile with timestamps, errors, and success indicators
   - Handles common failure cases gracefully (missing pg_dump, connection refused, permission denied, disk full)

3. **Retention Rotation Logic** — automatically delete backup files older than `RETENTION_DAYS`

4. **Restore Script Snippet** — a minimal companion restore guide showing how to use the generated `.sql` file with `psql`

5. **Usage Instructions** — include a sample crontab entry to run the script daily and a note on how to test it manually

6. **Security Notes Section** — flag any security concerns (e.g., plaintext password handling) and recommend production-grade alternatives such as storing credentials in `.pgpass`, using role-based access, or integrating with a secrets manager.

7. **Assumptions List** — end by listing every assumption you made about configuration choices if the user provided placeholder values instead of real details

FORMAT REQUIREMENTS
- All code blocks must be properly syntax-highlighted
- Use descriptive variable names and inline comments in all scripts
- Do NOT hardcode any database credentials in plain text within the script — explicitly note where substitutions or secure alternatives should be used
- Assume Linux/CentOS/RHEL environment with Bash as the default shell
- PostgreSQL version assumed: 12+

Files

File	Size	Use it when
`koleslaw-nemotron-IQ4_NL.gguf`	17.0 GiB	Recommended. Fits a 24GB GPU with 8K context. What koleslaw.ai serves
`koleslaw-nemotron-Q4_1.gguf`	18.6 GiB	Alternative 32-block quant if IQ4_NL is slow on your backend
`nemotron-finetuned-Q8_0.gguf`	~34 GB	Near-lossless; needs ~40GB+ (M-series Mac with 64GB+, or 48GB GPU)
`nemotron-finetuned-f16.gguf`	~63 GB	Requantize your own variants from this
`model-*.safetensors`	~63 GB	Merged bf16 weights (LoRA already fused) for transformers/vLLM

⚠️ Quantize this model carefully

Nemotron-3-Nano's hidden dimension is 2688, which is not divisible by 256. Standard K-quants (Q4_K_M, Q5_K_M, …) use 256-element superblocks, so llama.cpp silently falls back to q5_0 for the affected tensors — a "Q4_K_M" of this model balloons to ~24.5GB and no longer fits a 24GB card.

Use 32-element block types only: IQ4_NL, Q4_1, Q4_0, Q5_0, Q5_1, Q8_0. Requantizing from the Q8_0 with --allow-requantize works and is how the IQ4_NL here was made (quality-gated against the Q8 — see Evaluation). The GGUFs in this repo were produced with llama.cpp at commit c96f608 (known-good nemotron_h_moe support); newer builds should work, but if a conversion or quantization pass crashes on this architecture, try that pin first. Which tensors fall back, why, and the log to prove it: the write-up.

Also: CPU inference via llama-cli hangs on the nemotron_h_moe architecture — don't use it to smoke-test your quant. ollama create completing its hash pass is the practical integrity check; then test on GPU.

Usage (Ollama)

Requires Ollama ≥ v0.32 — the RENDERER/PARSER directives below are newer Modelfile syntax and older builds fail with a parse error (tested on v0.32.0). Create a Modelfile:

FROM ./koleslaw-nemotron-IQ4_NL.gguf
RENDERER nemotron-3-nano
PARSER nemotron-3-nano

# temperature backed off from the NVIDIA-recommended 1.0: structured output
# was too loose at 1.0 (duplicated headings, stray lists). top_p per NVIDIA.
PARAMETER temperature 0.85
PARAMETER top_p 1.0

# Reasoning tokens share the context window; Ollama's default is tight.
PARAMETER num_ctx 8192

SYSTEM """You are a prompt enhancement assistant. You transform raw user prompts into detailed, specific, well-structured prompts ready to hand to an AI model. When the task is complex, reason deeply about what makes the prompt weak and which improvements would be most impactful before producing the enhanced version.

Structure every enhanced prompt with labeled sections, adapting to the task:

ROLE — the expert persona the answering model should adopt, with concrete, domain-specific expertise.
OBJECTIVE — precisely what the user wants, including explicit non-goals (what must NOT be done).
CONTEXT — the user's situation and inputs. Where a detail is unknown, add a bracketed fill-in placeholder listing likely options, e.g. <PostgreSQL | MySQL | other>, instead of inventing facts.
DELIVERABLES — a numbered list of concrete outputs, each specific enough to verify: comparison tables with named criteria, code with language and library versions, recommendations with rationale.
CONSTRAINTS — hard requirements, preferences, and risks the answer must respect, called out explicitly.
OUTPUT FORMAT — the exact shape the response should take.

Also: preserve the user's original intent exactly; carry the user's stated facts into the prompt and mark time-sensitive ones for verification; end the enhanced prompt by instructing the model to ask about any missing bracketed inputs and to state its assumptions rather than guessing. Return only the enhanced prompt with no preamble or explanation."""

bash

ollama create koleslaw-nemotron -f Modelfile
ollama run koleslaw-nemotron "help me plan a team offsite"

The model is thinking-capable; reasoning tokens count against num_ctx.

Training

Supervised bf16 LoRA with Unsloth on a single A100 80GB (RunPod). The constraints are the interesting part — this architecture fights you:

QLoRA is impossible: bitsandbytes can't 4-bit-quantize MoE expert weights (stored as nn.Parameter, not nn.Linear). bf16 LoRA loads all 30B params in 16-bit — ~60GB before activations. A100 80GB minimum.
Gradient checkpointing is unsupported: NemotronHForCausalLM (hybrid Mamba-2 + MoE + Attention) doesn't implement the hooks. It must be disabled in both Unsloth and SFTConfig, or the Trainer crashes.
Consequences: max_seq_length=1024, per_device_train_batch_size=1, ~80.5GB of 81.9GB VRAM in steady state. One epoch over the dataset ≈ 2,567 steps.

Data: 10,265 train / 1,282 validation examples of (rough prompt → enhanced prompt) pairs across three enhancement modes (expand, refine, structure), synthetically generated with Claude (claude-sonnet-4) and curated. No user data — the dataset predates the hosted service.

Evaluation

LLM-as-judge (claude-sonnet-4-6) over 50 validation prompts, scored 1–5 on three dimensions (fine-tune, pre-quantization):

Dimension	Mean	≥4
Clarity	4.90	100%
Completeness	4.68	100%
Faithfulness to intent	4.90	98%
Overall	4.83	98%

Sampling-temperature sweep (0.3 / 0.85 / 1.0) scored within noise of each other (4.86 / 4.83 / 4.81 overall); 0.85 shipped because 1.0 showed structure drift (duplicated headings, stray lists) the judge underweights.

Quantization gate (IQ4_NL vs Q8_0, before the quant was allowed into production). A 70-prompt A/B generation run on the target L4 hardware, of which 30 items were scored by a claude-opus-4-8 judge, including a blind pairwise comparison:

Metric (judge: claude-opus-4-8, n=30)	Q8_0 baseline	IQ4_NL (this release)
Composite	4.69	4.70
Clarity	4.67	4.60
Completeness	4.80	4.80
Faithfulness	4.60	4.70
Blind pairwise preference	14	16 (0 ties)

Latency from the 70-prompt A/B on a single L4 24GB: mean 14.5s / p95 19.8s.

Intended use & limitations

Narrow model: it enhances prompts. It is not a general assistant, and output quality outside English is untested.
Training examples were ≤1024 tokens; very long inputs are out of distribution (the production service runs num_ctx 8192 to give reasoning headroom, not for long documents).
It occasionally deviates from the canonical section names when the task warrants (see the example above — SETUP PARAMETERS instead of CONTEXT). By design it prefers bracketed placeholders over invented facts; downstream models should be told to ask about unfilled brackets.
Outputs are prompts, not answers: treat time-sensitive claims carried into the prompt as needing verification.

License & provenance

Base model: NVIDIA Nemotron-3-Nano-30B-A3B, under the NVIDIA Nemotron Open Model License (copy in LICENSE; attribution in NOTICE). Commercial use and derivatives permitted; this repo complies with its notice requirements.
Training data was synthetically generated with Claude, as disclosed above.
The serving stack (API, clients, portal) is open source under MIT: github.com/thunkaboutit. The full write-up of how this model is served in production for ~$245/month is rolling out on the Koleslaw blog, starting with the quantization story.

Questions / commercial interest: cp@thunkabout.it (.it is correct — the company is ThunkAboutIt)

thunkaboutit/Koleslaw-Nemotron-30B-A3B-PromptEnhance

Koleslaw Nemotron 30B — Prompt Enhancement