A paragraph-level prose rewriter: it takes prose written by a large model and re-renders it to be more human, preserving the semantics it was given.

Qwen/Qwen3-4B-Base with a rank-32 LoRA merged in at strength 1.15.

This is the larger of two checkpoints trained on the same pool and pipeline. A smaller version here: 1.7B

Variants

Path	Format	Use with
`/`	safetensors bf16, `qwen3` arch	transformers
`GGUF/prose-rewriter-4b-v1.2-Q8_0.gguf`	GGUF Q8_0, 4.69 GB	llama.cpp / llama-cpp-python
`GGUF/prose-rewriter-4b-v1.2-Q4_K_M.gguf`	GGUF Q4_K_M, 2.72 GB	llama.cpp / llama-cpp-python

VRAM

Measured on an RTX 3090 with llama-server -ngl 99 at stock settings (flash attention auto, f16 KV), read as the server process's resident VRAM after load and before the first request.

-c is the TOTAL context and llama.cpp divides it by --parallel. A -c 4096 --parallel 8 server gives each slot 512 tokens and silently truncates long paragraphs. Ask for slots x 1280: 512 source tokens is the documented input ceiling, the generation budget never exceeds 512, and the prompt's three blocks are a dozen more.

slots	`-c`	Q8_0	Q4_K_M
1	1280	4,624 MiB	2,922 MiB
2	2560	4,804 MiB	3,102 MiB
4	5120	5,164 MiB	3,462 MiB
8	10240	5,884 MiB	4,182 MiB

It is linear, so extrapolate freely: the weights are a fixed 4,444 MiB (Q8_0) or 2,742 MiB (Q4_K_M), and every 1280-token slot adds 180 MiB of KV cache on either quant. Only the total matters -- -c 2048 --parallel 1 and -c 2048 --parallel 2 allocate the same bytes.

Four slots is a sensible default. Eight reserves better than a gigabyte of KV before the first request arrives, which on a small card is the difference between fitting and not. The 1.7B runs the same eight slots in 3,202 MiB.

Prompt format

<|im_start|>source
{paragraph}<|im_end|>
<|im_start|>edit
match<|im_end|>
<|im_start|>rewrite

The chat template in this repo builds exactly that string, byte for byte, from two roles:

python

messages = [
    {"role": "source", "content": paragraph},
    {"role": "edit",   "content": "match"},
]
tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

It is not a chat model. The template rejects user / assistant / system, and an edit value outside the three modes, rather than quietly building a prompt the weights have never seen.

One paragraph per call. Generation stops on <|im_end|>, which is set as an eos_token_id in generation_config.json. Temperature is the only knob.

The `edit` block is mandatory

edit names which of three length transforms is being asked for. The values are the corruptor's, so they read backwards. They name what was done to build the input, not what the model should do to it:

`edit`	what it says about the input	what the model does
`match`	the source is the human's length	rewrite in place
`inflate`	the source was padded relative to the human original	cut
`compress`	the source was flattened and shortened	open it back out

match is the setting for "rewrite it, do not trim it". It's strongly recommended you use this mode.

Sending no block is the worst thing you can do to this checkpoint. It was trained with the block, so omitting it collapses the model onto its deletion-heaviest mode.

Serving recipe

Sampled at temperature=0.9, top_p=0.9.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "chartreuse-verte/prose-rewriter-4b-v1.2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()

def rewrite(paragraph, mode="match"):
    text = tok.apply_chat_template(
        [{"role": "source", "content": paragraph},
         {"role": "edit",   "content": mode}],
        tokenize=False, add_generation_prompt=True,
    )
    ids = tok(text, return_tensors="pt", add_special_tokens=False).input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=512, do_sample=True, temperature=0.9, top_p=0.9)
    return tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip()

Same thing under llama.cpp. The roles are source and edit, which no chat API models, so build the string yourself; <|im_end|> stops it:

bash

llama-cli -m GGUF/prose-rewriter-4b-v1.2-Q8_0.gguf -st -n 512 --temp 0.9 --top-p 0.9 \
  -p '<|im_start|>source
{paragraph}<|im_end|>
<|im_start|>edit
match<|im_end|>
<|im_start|>rewrite
'

Input length

input words	generations	length ratio	invention	3-gram self-repetition
< 15	288	1.41	68%	0.000
15-24	219	1.00	31%	0.001
25-39	108	0.96	32%	0.000
40-59	60	0.95	27%	0.001
60-79	180	0.91	24%	0.002
80-119	170	0.90	24%	0.002

length ratio is median output words over input words. invention is the share of generations carrying a sentence the input does not entail (DeBERTa-MNLI, min entailment < 0.5). 3-gram self-repetition is the fraction of a generation's 3-grams that repeat inside it -- the degeneracy signal.

Nothing degenerates at any length. Self-repetition sits at noise across the whole range, and not one of these generations failed to stop or came back empty. What actually happens below ~15 words is that the model pads and invents: it stretches the line toward its learned length and roughly 70% of outputs add material the input never supported, against ~25% at paragraph length. Between 15 and 120 words nothing moves much.

So serve it on anything from a full sentence up. Below 80 bytes, pass the text through unchanged.

Evaluation

	1.7B	4B
val loss	0.8214	0.7539
train loss	0.9735	0.9104
invention rate	0.177	0.146
copy rate	0.062	0.062
copy excess vs target	+0.001	−0.009
distinct-4	0.982	0.982
entropy	0.530	0.506
length ratio	1.025	1.004

The 4B invents less and copies no more, at flat diversity, which is the direction scale was expected to move.

A known quirk is that human-written texts use less rare vocabulary than LLM texts. This is a trade-off that must be accepted.

Training

Corrupt forward, train backward. The human paragraph is the target; an on-policy LLM manufactures the input by slop-ifying it.

The target side is human prose: roughly 60/40 r/WritingPrompts (Mollymo/Human-to-AI-writing) and AO3 (midwestern-simulation-active/ao3_random_subset), with a sliver of fanfiction.net (atom-in-the-universe/fanfics-10k-10k).

The input side was generated by eight corruptor endpoints, weighted so no single model's tics dominate:

pool axis	composition
rows	60,323 over 48,395 distinct targets
corruptor	ds-flash-nano 21%, artemis 21%, ds-flash 18%, gemma-31b 16%, qwen-flash 12%, then muse-spark, ds-pro
corruption band	medium 38%, heavy 34%, light 25%
`len_mode`	match 60%, inflate 23%, compress 10%, unmarked 7%
kind	prose 93%, dialogue 7%, structural no-ops 0.7%

Pairs pass invariant gates before they reach the GPU: POV, tense, who is in the scene, grammatical correctness on the target side, and NLI entailment both ways. About 44k of 317k raw corruptions survive; the training pool is the seeded subset of those. This is the same pool the 1.7B was trained on, in the same order.

Loss on the target paragraph only. Everything before rewrite is masked.


LoRA	r=32, alpha=32, dropout 0.05
target modules	q, k, v, o, gate, up, down, and `lm_head`
trainable	71,004,160 params (1.73%)
schedule	1 epoch, lr 1e-4 cosine, batch 4 × accum 8, seq 2048
steps	1,659 on one RTX 3090, 2h14
loss	train 0.9104, val 0.7539 (601 val rows, document-disjoint)

The merge, and one thing not to do

Merged at strength 1.15. Rank 32 with alpha 32 is a LoRA scaling of 1.0, so the effective scaling is 1.15: W + (B @ A) * 1.15. Merged in float32, stored bfloat16.

lm_head is adapted, and Qwen3-4B-Base ties lm_head.weight to embed_tokens.weight. This checkpoint is untied: the merged output head is stored separately and the input embeddings are bit-identical to the base model's, which is what training assumed. config.json says tie_word_embeddings: false and it means it. Do not re-tie it, and if you convert to another format, check that the head survived.

Limitations

Not an instruct model. It has one job and one prompt. There is nothing to ask it.
Works on fictional prose only. May not work on technical documentation.
One paragraph per call. Longer input degrades; split it.
Will not pass AI detectors. Pangram and such will still know because this model preserves word choices and certain sentence structures.
English only, narrative register (third and first person fiction, dialogue with quoted speech).
Short input pads and invents. The floor is about 15 words, and below it the failure is fabrication rather than gibberish. See Input length.

License

The weights in this repository are released under the GNU Affero General Public License, version 3. The full text is in LICENSE.

This is a derivative of Qwen/Qwen3-4B-Base, which is licensed under Apache License 2.0. That license is preserved and its terms continue to apply to the base weights this model was built from; the AGPL covers the combined work as distributed here. Apache-2.0 is one-way compatible with AGPLv3, which is what makes this combination possible.

If you run a modified version of this model as a network service, AGPL section 13 requires you to offer the corresponding source of your modifications to its users.

chartreuse-verte/prose-rewriter-4b-v1.2

prose-rewriter-4b-v1.2