Lolaby — AI-powered lullabies
My partner's sister teaches kindergarten. Every day at nap time she has fifteen 4-year-olds on tiny mats, and ever since the class started learning about music and instruments, it begins the same way: "sing a song for me." She'd love to give each child their own song, one built around whatever they love that week: the stuffed fox, the new puppy, the rainbow. But she's one person, with no musical training, and thirty minutes to get fifteen children to sleep.
Lolaby is that tool. The child shows Lola what they love, doodling on screen, or handing over a paper drawing to photograph. The teacher types the child's name. A small, on-device AI looks at the drawing, writes a lullaby about it, and sings it back.
Everything runs locally. No cloud LLM, no per-song API cost, no child's data ever leaving the device.
Meet Lola, your personal bedtime singer.
Watch the demo ↓
Try it here: Lolaby on Hugging Face.
What Lolaby does
You give it:
- the child's name.
- their age.
- something they love (drawing or typing).
- something they're scared of.
- and a mood.
It writes them a lullaby and sings it back over original instrumental music.
A sample input looks like this:
Mia, age 3 · Loves: her stuffed elephant Pip · Scared of: the dark · Mood: sleepy and comforted
And the output isn't just generic "sleep tight" boilerplate. The song mentions Pip directly. The dark becomes something gentle instead of threatening. The lyrics are structured around reassurance instead of spectacle.
The first generated lullaby that actually felt real to us was one where the model turned the stuffed elephant into an active character in the song:
"Pip keeps watch beside your bed tonight…"
That was the first moment it stopped feeling like a prompt-completion system and started feeling like a bedtime interaction.
When we showed the prototype to the kindergarten teacher who inspired it, she laughed halfway through the song — because hearing a child's toy named back to them immediately sounded more emotionally specific than any canned lullaby app she'd seen before. That reaction became the product target.
How it works
The pipeline starts with a drawing and ends with a song.
Drawing. The child doodles on screen, or the teacher photographs a paper drawing, read by MiniCPM-V 4.6 1.3B, running locally. If there's no drawing, Lolaby falls back to the What do they love? field automatically. But a drawing makes the song personal in a way a text field can't — the child sees their own fox, their own balloon, turned into a song.
Lyrics. A fine-tuned Llama 3.2 3B reads the drawing description (if available) and the child's details, then generates lullaby lyrics in a structured format with tempo, chord progression, and inline chord markers. The lyrics stream in as they're written, so the teacher can start reading aloud before the last verse is finished.
Music. Every instrument is synthesized from scratch using custom DSP — no prerecorded samples and no MIDI libraries. Guitar, piano, xylophone, harp, and ocarina are all generated in real time. The target wasn't realism but non-stimulating comfort: soft attacks, slow decays, harmonic profiles that stay warm on laptop speakers.
Voice. Kokoro TTS runs locally and reads the lyrics back in a calm, understated voice — closer to being quietly read to than performed at. Bedtime audio should settle energy, not add more of it.
The whole stack runs on CPU. No GPU, no cloud, no latency surprise mid-nap.
The drawing
The drawing is the emotional heart of Lolaby, so getting it right mattered.
We use MiniCPM-V 4.6, a 1.3B vision-language model from OpenBMB, to convert the drawing into a short text description that feeds directly into the lyric prompt. At 1.3B parameters it's small enough to run on CPU alongside everything else, without blowing the "build small" constraint or adding a cloud dependency just for the vision step.
The prompt we send it is deliberately narrow: describe what you see in simple terms, as if explaining a child's drawing to someone who can't see it. Children's drawings are rough. A stick figure becomes "a little person." A wobbly square with a triangle on top becomes "a house." That level of interpretation is fine, it's exactly what a parent does when a child hands them a drawing. The model doesn't need to identify a specific character, it needs to find the warmth in the shape.
If the vision model can't load for any reason, the app falls back to a lightweight NumPy stroke-and-colour analyser. It's less expressive, but it keeps songs generating rather than breaking.
The lyrics
The base Llama 3.2 3B was good at writing songs but unreliable at writing them in a machine-readable format with consistent structure and emotionally appropriate content. So we fine-tuned it on a synthetic dataset of 1,500 lullabies distilled from Claude Haiku 4.5.
The dataset was built with hard anti-boilerplate gates. Each prompt sampled a specific sub-topic rather than a vague category — "a stuffed fox named after a colour" rather than "comfort object" — so the teacher model couldn't default to its favourite clichés. N-gram deduplication rejected lines with too much overlap against previously accepted ones. Opener deduplication killed repeated opening shapes after three uses. A theme cap stopped any single category from exceeding 16% of the dataset. Every example had to parse correctly, and the lyric body was safety-screened before acceptance.
The result was 99.4% line uniqueness across 12,001 total lyric lines. Training used LoRA rank 16 on a Colab T4, three epochs, about 45 minutes. The best checkpoint passes four quality checks: coherence, faithfulness to the prompt, variety, and format compliance. The previous boilerplate-trained model passed only the last one.
What the fine-tune actually taught us: dataset shaping mattered more than model size. The model didn't become smarter. It became stylistically disciplined.
The model card, LoRA adapter, merged FP16 weights, and GGUF are at build-small-hackathon/lolaby-llama-3b.
The music
The hard work split evenly between the AI and the audio. Generating lyrics was one problem. Making synthesized instruments sound emotionally believable without samples, in real time, was another.
So we started analyzing real instrument recordings spectrally. The workflow for every synth became:
- Find a clean reference recording.
- Isolate a sustained note.
- Measure its FFT (Fast Fourier Transform).
- Extract harmonic balance and decay.
- Rebuild those properties in code.
Every instrument had a challenge to overcome. The xylophone was almost pure sine. The keyboard sounded like an organ until we fixed the harmonic suppression. The guitar's third harmonic was stronger than the fundamental. Those tiny spectral details were the difference between synthetic and warm. Most of the final comfort in the app comes from those small spectral decisions, not just the language model.
The voice
The final layer is the one the child actually hears.
We use Kokoro 82M for text-to-speech, a lightweight model small enough that it adds almost nothing to the pipeline's latency on CPU. No cloud TTS, no API call, no audio leaving the device. The lyrics go in, a WAV comes out, entirely on-device.
The voice choice was deliberate. Kokoro's output is calm and slightly understated — closer to being quietly read to than performed at. That matters more than it might seem. A lullaby is not a performance. Expressiveness, dynamic range, dramatic pauses — all of it works against the goal. Bedtime audio should reduce stimulation, not add more of it. A voice that sounds like it's winding down is more useful than one that sounds impressive.
What we learned
Streaming changes the interaction. Lyrics arriving token by token mean a teacher can start singing before generation finishes. That's the difference between a tool that fits into nap time and one that interrupts it.
Local-first shapes everything. The offline constraint forced the architecture to be coherent. A model that responds in 30 seconds on a laptop is useful in a classroom. A cloud model that needs reliable WiFi isn't.
Specificity beats scale. The most emotionally effective moments were always the small ones — the child's toy becoming an active character, their fear being named and gently resolved. That came from dataset curation, not model size.
