Korean styled handwriting line-image generation, fine-tuned from blowing-up-groundhogs/eruku (autoregressive styled handwriting generation, arXiv 2510.23240). Give it a style reference line and a Korean/English string; it draws that string in that style.

Training code and experiment log: https://github.com/HERIUN/Eruku_korean_finetuning

This revision requires the paired Korean VAE. config.vae_name_or_path points at HERIUN/emuru_vae_korean. The VAE was fine-tuned with encoder+decoder unfrozen, so its latent space moved and the T5 decoder here was re-adapted on top of it. Swapping in the original emuru_vae produces garbage.

Quick start

python

from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("HERIUN/eruku_korean", trust_remote_code=True).eval().cuda()

img = model.generate_handwriting(
    style_image=Image.open("style_line.png"),   # one line of handwriting
    style_text="조용히 책을 읽었다.",              # transcription of the style line (optional, but helps a lot)
    gen_text="한국어 손글씨 생성",                 # what to write
    cfg_scale=1.5,
)
img.save("out.png")

style_text — passing the style line's transcription markedly improves fidelity.
cfg_scale — 1.0 is the most faithful on the eval battery; 1.5-2.0 pushes style strength.
bbox_crop=True (default) crops the style image to its ink bounding box. Real scans carry page margins that blow up the VAE latent std and make generation run away or come back blank; this closes that gap. Pass bbox_crop=False for crops that already fill the frame.
Preprocessing is handled for you and matches training: the style image is converted to RGB, resized to height 64, and normalised to [-1, 1] before it reaches the VAE. Width is not padded to a multiple of 8 — training never padded either, so do not add it.

Results

CER measured with a Korean HTR reader (floor on font renders ~0.00; easyocr saturates at 0.236 even on Korean ground truth and cannot be used). n=100 sentences split into length buckets, cfg=1.0, coherent sentences, seen fonts.

bucket	decoder-only VAE	full VAE, before re-adaptation	this model
Korean, all	0.203	0.211	0.127
Korean, short (1-3 words)	0.497	0.540	0.286
Korean, mid (4-8 words)	—	—	0.049
Korean, long (9-15 words)	—	—	0.059
English, all	0.084	0.056	0.045

Two things had to happen together: the Korean VAE raised the reconstruction ceiling (roundtrip MSE -83%), and a short-text-weighted re-adaptation of T5 closed the gap to that ceiling — short Korean strings were the actual bottleneck (CER 0.54 -> 0.286).

Before / after

Same target text through the original English pretrained Eruku (with the original VAE) and through this release. Columns: style reference | target rendered in that font | before | after.

Korean, with Korean-font style references. The baseline has seen neither Hangul glyphs nor Korean-font styles, so it returns nothing usable:

English, with Latin style references that are in distribution for both models — English is kept, not traded away for Korean, and strokes come out crisper (the Korean VAE is a better decoder for Latin too):

How it was trained

English pretrained -> Korean (--english-frac 0.15 to hold English) -> VAE swap (--reset-vae) + re-adaptation with short-text emphasis, 30000 optimizer steps total. Data is synthesized online: fonts act as writers (83 Korean handwriting/display + 177 Latin), text is sampled from a Korean corpus, rendered and augmented per sample, so every sample is unique. Backbone: T5-large + ByT5 byte tokenizer (byte-level input handles Hangul without a Korean vocab), frozen VAE.

Limitations

Short strings are still the weak spot (CER 0.286 vs 0.059 for long lines) — fewer latent columns means less context for the autoregressive decoder.
Unseen extreme display/brush fonts degrade; the training set has 99 fonts, upstream had ~100k.
The VAE snaps ink density to crisp black, so it does not preserve faint/pencil strokes (prototype collapse; see docs/VAE_ROBUSTNESS.md in the repo).
Long English lines are weaker than the original English model — Korean is the target here.

License: Apache-2.0, inherited from the base model.