The original VAE was trained on Latin script and is the hard ceiling for Korean
fidelity: Eruku conditions and predicts only VAE latents (8 dims per latent column), so
anything decode(encode(x)) cannot represent is invisible to the model. Hangul stacks
initial/medial/final jamo in 2D inside one syllable cell, which costs ~12x more
reconstruction error than Latin. This checkpoint removes most of that gap.
English improves too, because the recipe is script-neutral (pure L1, no OCR loss).
Before / after
Same input, roundtrip through each VAE (source / original / this model), with per-line MSE and SSIM.
Korean — the original smears dense syllables, this model keeps the strokes:
Korean reconstruction
English — the recipe is script-neutral, so Latin improves as well:
English reconstruction
What it changes in generation
The VAE only sets the ceiling — the T5 decoder must be re-adapted to reach it (its latent space moved). Original English pretrained Eruku with the original VAE vs. the paired release HERIUN/eruku_korean, which uses this VAE. Columns: style reference | target rendered in that font | before | after.
Korean, with Korean-font style references — note the baseline has seen neither Hangul glyphs nor Korean-font styles:
Korean generation
English, with Latin style references that are in distribution for both models, so only the model differs — English survives, and strokes come out crisper:
Pure L1 reconstruction (+ negligible KL), encoder and decoder both unfrozen, warm-started
through a decoder-only stage and then a 15k-step full stage; this checkpoint is 28k further
steps on top of that. Training data: synthetic Korean/English line renders
from 83 Korean + 177 Latin fonts (fonts act as writers), held-out font split for
validation. See train_vae_korean.py.
Adding an OCR/HTR readability loss made things worse for this task (Korean MSE
0.0089 vs 0.0031 with it off) — text is already high contrast, so L1 keeps strokes crisp
on its own, while a Korean HTR loss biases the shared decoder and degrades English.
Usage
Minimum — what the VAE strictly requires
Three things, and nothing else:
requirement
if broken
3 input channels (repeat your grayscale line 3x)
hard error: expected input[1, 1, 64, 872] to have 3 channels
values near unit scale ([0, 1] or [-1, 1], both work)
raw 0-255 input breaks the latents (cosine 0.55 to the correct ones, latent std 1.04 -> 0.53)
That is all it takes to run:
python
1from diffusers import AutoencoderKL
2from PIL import Image
3import numpy as np, torch
45vae = AutoencoderKL.from_pretrained("HERIUN/emuru_vae_korean").eval().cuda()67g = Image.open("line.png").convert("L")# dark ink, light background8x = torch.from_numpy(np.array(g, np.float32)/255.0)[None,None].repeat(1,3,1,1)910with torch.no_grad():11 z = vae.encode(x.cuda()).latent_dist.mode()# [1, 1, H/8, W/8]12 rec = vae.decode(z).sample[:,:1]# decoder returns 1 channel1314print(tuple(x.shape),"->",tuple(z.shape),"->",tuple(rec.shape))15# a 99px-tall scan: (1, 3, 99, 1347) -> (1, 1, 12, 168) -> (1, 1, 96, 1344)16# note the height is floored to a multiple of 8, and 12 latent rows != Eruku's 8
No resizing, no normalisation, no padding. It works — but the reconstruction is only good if
your line already happens to be about 64px tall, which is what the next section fixes.
Recommended — resize to height 64 first
This is the one preprocessing step with a large measured effect, because the VAE was trained on
64px lines. Everything else you might be tempted to add is optional (see the table below).
python
1import torch.nn.functional as F
23defto_vae_input(image, height=64):4# PIL text-line image -> [1, 3, height, W] in [-1, 1], width padded to a multiple of 85 g = image.convert("L")6 w =max(1,round(height * g.width / g.height))7 x = torch.from_numpy(np.array(g, dtype=np.float32)/255.0)[None,None]8 x = F.interpolate(x, size=(height, w), mode="bicubic", align_corners=False).clamp(0,1)9 x = F.pad(x *2-1,(0,-w %8), value=1.0)# [-1,1], white pad10return x.repeat(1,3,1,1)# encoder wants 3 channels1112defto_pil(t):13# VAE output [1, 1, H, W] in [-1, 1] -> grayscale PIL image14return Image.fromarray((((t[0,0].clamp(-1,1)+1)/2)*255).byte().cpu().numpy())1516x = to_vae_input(Image.open("line.png"))17with torch.no_grad():18 z = vae.encode(x.cuda()).latent_dist.mode()# [1, 1, 8, W/8] (.sample() adds noise)19 rec = vae.decode(z).sample[:,:1]2021to_pil(rec).save("line_recon.png")22print(tuple(x.shape),"->",tuple(z.shape),"->",tuple(rec.shape))23# (1, 3, 64, 872) -> (1, 1, 8, 109) -> (1, 1, 64, 872)
One latent column covers 8 pixels and carries 8 dims — that column sequence is exactly what
Eruku conditions on and predicts. If you are feeding Eruku, height 64 is not optional: the
latent row count follows the input height and its interface expects 8.
What the recommended version actually buys
12 clean font lines + 20 real scanned lines, each reconstruction resampled to a common 64px
canvas and compared against the same reference (so different input resolutions stay comparable):
preprocessing
font lines MSE / SSIM
scans MSE / SSIM
minimum (native size, [0, 1])
0.1417 / 0.481
0.0170 / 0.806
+ resize to height 64
0.0047 / 0.967
0.0042 / 0.937
+ [-1, 1] and width padding (recommended)
0.0044 / 0.970
0.0059 / 0.943
So the resize is worth ~30x in MSE on clean lines (0.1417 -> 0.0047); the normalisation and
padding on top of it are within noise — including for Eruku generation, which we checked
separately (CER 0.045 with [0, 1] vs 0.046 with [-1, 1] over 24 lines). They are kept because
[-1, 1] is the convention training used and the padding returns a reconstruction at exactly the
input width, not because they buy fidelity.
HTR-reader CER does not separate these variants (0.026 / 0.041 / 0.044 on font lines — all
readable, differences are noise; on our scans the reader's own floor is 0.401), so the pixel
metrics above are the ones to go by.
Height 64 is what the VAE was trained on and what Eruku requires — but for standalone
reconstruction it is not the optimum. Each reconstruction was resampled back to the source's
native resolution and compared there, so taller inputs get no free advantage:
input height
latent rows
font lines, native 176px
real scans, native 99px
48
6
0.0941 / 0.703
0.0049 / 0.947
64 (training resolution)
8
0.0377 / 0.812
0.0035 / 0.968
80
10
0.0165 / 0.875
0.0032 / 0.975
96
12
0.0105 / 0.906
0.0033 / 0.976
128
16
0.0051 / 0.936
0.0036 / 0.974
192
24
0.0021 / 0.956
0.0042 / 0.960
(MSE / SSIM, 12 font lines and 10 scanned lines.)
The pattern is about pixels per glyph, not the number 64: a taller canvas gives the encoder
more pixels for the same characters, so fidelity keeps climbing — as long as the source really
has that resolution. The font lines are rendered at 176px native and improve all the way to 192
(MSE 18x lower than at 64). The scans are only 99px native, so they peak around 80-96 and then
get slowly worse: upscaling past the source adds no information and moves further from the
training scale. Cost grows with the latent area, roughly (height / 64)^2 for the same line.
So: feeding Eruku -> height 64, no choice. Reconstruction only -> use the source's native
height (capped around 128-192) and you get a much better roundtrip.
What Eruku itself does with this VAE
For reference, the paired model's own preprocessing (generate_handwriting) and the training
pipeline agree on every load-bearing point, so the recommended settings above are exactly the
distribution this VAE saw:
step
training pipeline
generate_handwriting
height
64 (bilinear, antialias)
64 (Lanczos)
value range
Normalize(0.5, 0.5) -> [-1, 1]
same, applied inside the model
channels
3
3 (convert("RGB"))
polarity
dark ink on light paper
unchanged from your input
width padding to a multiple of 8
none
none
Note the last row: do not pad the width when conditioning Eruku. Padding would append a
white latent column that training never produced. The padding in the recommended snippet above
exists only so a reconstruction comes back at exactly the input width.
Everything else we swept — no effect or harmful
knob
verdict
resize filter
almost irrelevant among smooth filters: bilinear 0.0044, bicubic 0.0044, Lanczos 0.0045, area 0.0051 (font lines). Only nearest-neighbour is clearly bad (0.0063)
contrast cleanup
every attempt made it worse: Otsu binarisation 0.0072, autocontrast 0.0042, min-max stretch 0.0036, untouched 0.0034. The decoder already snaps ink to crisp black — pre-binarising just discards the grey levels it uses
[-1, 1] vs [0, 1]
interchangeable. Latents differ only with cosine 0.9997, reconstruction is the same (MSE 0.0048 vs 0.0058, SSIM 0.946 vs 0.951, n=32), and Eruku generation is unaffected too (CER 0.045 vs 0.046 over 24 lines; 21 of 24 scored identically). A GroupNorm right after the first conv absorbs a global shift/scale. Training used [-1, 1], so that stays the default here
width multiple of 8
no fidelity effect; without it the encoder floors the width and you get back up to 7px less (871 -> 864)
Real scans reconstruct a little worse than clean font renders and the decoder does not preserve
faint or pencil strokes — it snaps ink to crisp black.
If you are feeding Eruku, also crop scan margins to the ink bounding box: empty margins
inflate the latent std and make generation run away. The paired model does this for you
(bbox_crop=True). Cropping is not comparable in these tables — it raises ink coverage from
1.5% to 6.0% of pixels, so per-pixel MSE rises by construction.
To generate handwriting rather than reconstruct it, use the paired model
HERIUN/eruku_korean — it loads this VAE automatically and
does the preprocessing for you.
Warning — latent space moved. This was trained with --train-part full
(encoder + decoder), so its latents are not interchangeable with the original
emuru_vae. Pairing it with the original blowing-up-groundhogs/eruku weights will
produce garbage. Use it with HERIUN/eruku_korean,
which was re-adapted on top of these latents.
Limitations
Held-out extreme display/brush fonts remain considerably worse than regular fonts
(0.0128 vs 0.0031); font diversity is the bottleneck (99 fonts here vs
~100k upstream), and augmentation did not help.
