You can deploy this model and start earning money today!

build-small-hackathon/
lolaby-llama-3b

Name: lolaby-llama-3b
Availability: InStock

Views

No views yet

Lolaby — Llama 3.2 3B, fine-tuned to write singable lullabies

A small (3B) instruction-tuned model that writes original, personalised lullabies with chord markers and a tempo/meter header, designed for use with a downstream singing/synthesis pipeline.

This is the lyric-generation model used by Lolaby, a small, AI-powered app that turns a child's drawing into a personalised lullaby. Built for the Build Small Hackathon 2026 (Backyard AI track).

Write a lullaby for: Mia, age 3 Loves: a stuffed fox named after a color Fears: the dark when the light goes out

[C] Mia, your fox keeps [Am] watch tonight, [F] curled up close where it's [G] warm and bright, [C] the dark outside is [Am] soft and small, [F] and your fox can hear me [G] sing through it all...

Training data — built from scratch with anti-boilerplate gates

1,500 lullabies, distilled from Claude Haiku 4.5 with several hard gates that reject boilerplate at generation time:

Gate	What it does
Concrete love steer	Each prompt samples a specific sub-topic from a pool (e.g. "a hot-air balloon drifting up", "the corner bakery at dawn") instead of vague categories — the teacher can't default to its favourite cliché.
N-gram dedup	Every generated lyric line is normalised and checked against all previously-accepted lines via 4-gram Jaccard overlap. Examples with > 0.4 overlap on too many lines are rejected.
Theme cap	No single love-category (animal, nature, comfort object, etc.) may exceed 16% of the dataset. Forces broad coverage.
Opener dedup	The first 4 words of each lullaby (with the name normalised) are tracked. Over-used opening shapes are rejected after 3 uses — kills " watches..." / "...eyes are growing soft".
Format gate	Every example must parse — tempo header, valid chord markers from the declared progression, sensible line count — or it's regenerated.
Safety re-check	Even though the teacher is told to stay wholesome, the synthesised LOVE, FEAR, and lyric body are each re-screened with a safety filter before acceptance.

Result: 99.4% line uniqueness (11,925 unique lines out of 12,001 total lyric lines across 1,500 examples).

Diversity dimensions sampled per example

49 names across multiple naming traditions (Mia, Aiko, Mateo, Tariq, Saoirse, ...).
12 ages (1-7, weighted toward 2-5).
10 love categories × ~10-17 specific sub-topics each (≈ 120 concrete loves).
15 fear topics (and ~25% of examples have no fear, the natural "just comfort" case).
12 moods (cosy and content / restless but settling / clingy and needing reassurance / ...).
12 keys, 3 meters (6/8, 3/4, 4/4), 2-4 diatonic progressions per key.

The training-prompt format mirrors what the downstream app sends at inference, so there's no distribution shift between training and serving:

Write a lullaby for: <name>, age <n>
Loves: <concrete love>
Fears: <concrete fear>      # omitted if no fear
Mood: <mood>
Key: <key>
Meter: <meter>

Training setup

Base model: unsloth/Llama-3.2-3B-Instruct (4-bit loaded via Unsloth)
Method: LoRA via PEFT
LoRA config: r=16, alpha=32, dropout 0, applied to all 7 attention + MLP projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
Sequence length: 1024
Optimiser: AdamW 8-bit, learning rate 2e-4, cosine schedule, 20 warmup steps, weight decay 0.01
Batch: per-device 4, gradient accumulation 4 (effective batch 16)
Epochs: 3, with load_best_model_at_end on eval loss
Precision: bf16 where supported, fp16 otherwise
Eval/save strategy: per-epoch; the best checkpoint by eval_loss was kept (epoch 2 in our run)
Seed: 42

Trained on a single Colab T4 in ~45 minutes via the included train_lullaby.ipynb notebook.

Files in this repo

The repo ships in three forms so it's usable across runtimes:

LoRA adapter (adapter_model.safetensors) — smallest, requires the base model at inference.
Merged FP16 weights — standalone HF transformers checkpoint.
GGUF Q4_K_M — for llama.cpp and downstream tooling. The Lolaby app uses these on CPU.

How to use

With `llama-cpp-python` (CPU, what Lolaby itself uses)

python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="build-small-hackathon/lolaby-llama-3b",
    filename="*Q4_K_M.gguf",
    n_ctx=2048,
    verbose=False,
)

prompt = (
    "Write a lullaby for: Mia, age 3\n"
    "Loves: a stuffed fox named after a color\n"
    "Fears: the dark when the light goes out\n"
    "Mood: sleepy and comforted\n"
    "Key: C major\n"
    "Meter: 6/8"
)

out = llm.create_chat_completion(
    messages=[
        {"role": "system",
         "content": "You write personalized lullabies for small children, "
                    "with chord markers and a tempo/meter header so a guitar "
                    "accompaniment can be rendered. Output only the lullaby — "
                    "no preamble."},
        {"role": "user", "content": prompt},
    ],
    max_tokens=512,
    temperature=0.85,
)
print(out["choices"][0]["message"]["content"])

With `transformers` (GPU)

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "build-small-hackathon/lolaby-llama-3b"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system",
     "content": "You write personalized lullabies for small children, "
                "with chord markers and a tempo/meter header so a guitar "
                "accompaniment can be rendered. Output only the lullaby — "
                "no preamble."},
    {"role": "user", "content":
        "Write a lullaby for: Mia, age 3\n"
        "Loves: a stuffed fox named after a color\n"
        "Mood: sleepy and comforted\n"
        "Key: C major\n"
        "Meter: 6/8"},
]
inputs = tok.apply_chat_template(messages, return_tensors="pt",
                                  add_generation_prompt=True).to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=True,
                     temperature=0.85, top_p=0.95)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Output format

Every generation begins with a two-line header, then a blank line, then the lullaby itself. Lyric lines are prefixed with chord markers in [brackets] inline, drawn from the requested progression. Two 4-line verses by default.

Tempo: 54 bpm, 6/8
Progression: C - Am - F - G

[C] First line of [Am] first verse,
[F] continues here [G] in rhyme,
[C] third line of [Am] the verse,
[F] gentle close in [G] time...

[C] Second verse [Am] line one,
[F] line two [G] continues,
[C] third line of [Am] verse,
[F] final line [G] ending...

This format is consumed downstream by the Lolaby app's synth and TTS pipeline — the chord markers drive instrument rendering, the tempo header drives playback timing.

Safety

Every training example was safety-screened during generation: the synthesised LOVE, FEAR, and lyric body were each passed through a content filter before acceptance, with anything that slipped through the teacher's own guidance rejected and regenerated. This is a deliberate choice for a model intended for children — the training distribution itself is wholesome by construction, not retrofitted at inference time.

The teacher model was also explicitly instructed to keep all invented content age-appropriate (no death, violence, weapons, horror, substances, or anything frightening beyond a gentle, easily-soothed childhood worry).

Evaluation

Trained on 1,425 examples / held out 75 (5%) for eval. The best checkpoint by held-out loss was kept (epoch 2 of 3).

Beyond eval loss, the model was assessed on a deliberately-varied 10-spec sanity battery stressing diversity (different cultures of name, different love categories, edge-case loves the previous model failed on, fears that need actual soothing). Each generation was scored for:

Coherence — lines parse as English, no garbled tokens.
Faithfulness — the love and fear from the prompt appear in the lyric in recognisable form.
Variety — no two generations from the battery share opening or refrain shapes.
Format compliance — tempo header present, chord markers from the declared progression only.

The model passes all four on the held-out battery. The previous boilerplate-trained model passed only format compliance.

Handling edge cases

When given a love or fear the model has never seen at training time, it generalises gracefully — finding the nearest comforting concept and weaving it in. A specific fictional character or unusual object will typically be rendered through a familiar lens (e.g. "your little friend", "a quiet companion") so the lullaby stays singable and warm rather than breaking or refusing. This is the right behaviour for a bedtime song: soft landing, not literal lookup.

Limitations

English only. Training data is English. The model will attempt other languages but quality drops fast and chord markers may be inserted at awkward positions.
Dataset provenance. Training data was generated by Claude Haiku 4.5 under Anthropic's API. The dataset is not distributed with this repo. Anthropic's usage policy restricts redistribution of Claude-generated outputs, so the dataset is kept private to stay compliant.
Not a general LLM. This model is narrow-purpose. Asked off-topic questions, it will sometimes attempt to answer in lullaby form.

Intended use and out-of-scope use

Intended:

Generating personalised bedtime lullabies for use in apps, devices, or printed/digital lyric sheets.
A teaching example of dataset-quality-first fine-tuning.

Not intended:

General-purpose conversation or instruction following (the base Llama-3.2-3B-Instruct is better suited for that).
Content involving minors that goes beyond gentle bedtime themes.
Anything safety-critical.

Citation

If you use this model or the dataset-building approach, a link back to the Lolaby Space or this model is enough. If you want a Bib entry:

bibtex

@software{lolaby_2026,
  author  = {André Oliveira and Vasco Oliveira},
  title   = {Lolaby: a small-AI lullaby generator},
  year    = {2026},
  url     = {https://huggingface.co/build-small-hackathon/lolaby-llama-3b},
  note    = {Built for the Hugging Face Build Small Hackathon 2026.}
}

License

This model inherits the Llama 3.2 Community License from its base. You agree to that license by using these weights — see https://www.llama.com/llama3_2/license/.

Acknowledgements

Meta for releasing Llama 3.2 3B Instruct as the base.
Unsloth for the 4-bit + LoRA training stack that made this trainable on a free Colab T4.
Anthropic for the Claude Haiku 4.5 API used as the teacher model during dataset distillation.
Hugging Face & Gradio for hosting the Build Small Hackathon 2026 and shaping a thoughtful "small AI" prompt.

build-small-hackathon/lolaby-llama-3b