📌 Part of Blueprint: AI Agent & Local GPU Automation → L3: Local GPU Inference Stack (RTX 3060)

Generating Subtitles for Songs That Ship Without Timestamps

· 29 min read · Tutorials subtitleslrcsrtwhisperforced-alignmentai-musicpython

Every AI music tool I have tried returns the same shape of output: an audio file. Not one of them tells you when each line is sung. That is fine if you only want to listen, and useless the moment you want lyrics on screen — a karaoke line, a video caption, a scrolling block under a player.

Timing is the whole problem. The audio is the easy part.

This is the pipeline I ended up with, running on 18 English songs generated locally, plus the two places it broke in the browser. Everything below is reproducible: the only thing it needs from you is the lyrics you already wrote.

What you actually need

Two subtitle formats, one shared data structure:

FormatUsed byShape
LRCmusic players, karaoke blocks, <audio> overlays[mm:ss.xx]lyric line
SRTVLC, YouTube, Premiere, ffmpeg burn-inindex / hh:mm:ss,mmm --> hh:mm:ss,mmm / text

Both are just line → start time (+ end time for SRT). So the real task is: given a list of lyric lines and an audio file, find where each line starts.

Why “just transcribe it” fails

The obvious approach: run speech recognition, take its segments, write them out. It fails for two reasons.

1. Recognition of singing is much worse than recognition of speech. The model is fighting instruments, reverb, and vowel stretching. On my batch the word error rate on sung Mandarin hovered around 25–35%; dense electronic mixes were worse.

2. It drops whole sections. On a 140 BPM remix, one song’s entire first verse simply never appeared in the transcript — the vocal is buried and the model moved on. Two other songs lost their outros.

A transcript with holes cannot become subtitles with no holes. But here is the reframe that fixes everything:

You are not transcribing. You are aligning. You already know the exact words — they are your lyrics. You only need the audio for anchors.

The inversion that makes it work

Feed the recogniser to get word-level timestamps, then align its output against your known lyrics with a longest-common-subsequence diff. You do not care whether the recogniser got the words right — you care that the words it did get are in the right order, because each match is a timestamp anchor.

import difflib

ref = [u for line in lines for u in units(line)]      # your lyrics
hyp = [w[0] for w in asr_words]                        # what it heard
sm = difflib.SequenceMatcher(a=ref, b=hyp, autojunk=False)

for tag, i1, i2, j1, j2 in sm.get_opcodes():
    if tag != "equal":
        continue            # mismatches are just gaps, not errors
    for k in range(i2 - i1):
        matched[ref_index[i1 + k]] = asr_words[j1 + k][1:]   # (start, end)

Each lyric line then takes the timestamp of its first matched unit. This is robust in a way transcription is not: 60–70% word accuracy is plenty, because every correctly-placed word pins the line it belongs to.

Measured coverage on this batch — the share of lines that got at least one real anchor:

LanguageUnitMean anchor coverageSongs fully anchored
Englishword0.969/18
Mandarincharacter0.7011/11

English aligns better for a boring reason: words are delimited by spaces, so each ASR token maps cleanly onto one lyric token. Mandarin needs each character spread across the token’s time span:

span = w.end - w.start
for i, ch in enumerate(hanzi_in_word):
    chars.append((ch, w.start + span * i / n, w.start + span * (i + 1) / n))

The governor: three rules that stop nonsense timings

Anchors get you most of the way. Left alone, the gaps between them produce timings that are physically impossible, and two of those bit me for real:

Failure A — a line at 0.0 s. One song’s first lyric landed at exactly zero, before the intro had even finished, because the anchor search matched a stray word in the opening shout.

Failure B — four lines in 0.6 s. A song whose outro the recogniser missed entirely had its remaining lines compressed to 0.18 s apart. Twelve characters cannot be sung in 0.18 s.

Failure C — six lines pinned to the floor. One track’s entire first verse landed with no usable anchor, so rule 3 below swept six consecutive lines onto the minimum gap: exactly 1.20 s each, which is roughly ten words a second. The output still looked plausible — monotonic, in range, inside the audio — and it only fell apart when I listened to that track and the whole verse went past in six seconds. Re-running alignment on that one file put its first line at 10.8 s instead of 28.2 s: the anchors had not been missing, they had been matched to the wrong part of the song.

So after alignment, run three rules:

  1. Estimate the song’s own rate from neighbouring anchored pairs (units ÷ seconds, take the median). Every line’s expected duration is units / rate, floored at ~1.8 s. Rates across one batch ranged from 0.96 to 2.33 words/s in English, and 2.3 to 4.1 characters/s in Mandarin — you cannot use a global constant.
  2. Refill unanchored runs proportionally, not with a fixed spacing. If a run sits between two anchors, distribute it by expected duration; if there is not enough room, compress proportionally.
  3. Reject impossible anchors and enforce a minimum gap. An anchor whose start is earlier than 5 s + everything before it could possibly take is a false match — demote it and re-interpolate. Then sweep forward so no two lines start less than 1.2 s apart, and pull back from the end if the tail overruns the audio.

That last sweep is what turns “looks plausible” into “cannot flicker”.

Emitting both formats

Same timing array, two writers:

def lrc(ts, lines):                      # [mm:ss.xx]
    return "\n".join(f"[{m:02d}:{s:05.2f}]{l}" for (m, s), l in zip(ts, lines))

def srt(spans, lines):                   # hh:mm:ss,mmm
    out = []
    for i, (line, (a, b)) in enumerate(zip(lines, spans), 1):
        out.append(f"{i}\n{stamp(a)} --> {stamp(b)}\n{line}\n")
    return "\n".join(out)

LRC needs only the start; SRT needs start and end, so use the next line’s start as the current line’s end (this is why the governor’s minimum gap matters — without it, SRT blocks get zero duration and some players skip them entirely).

Verification: the part I got wrong twice

The subtitles were correct long before the display was. Two bugs, both in the 20-line player script, both invisible to a code read:

Bug 1 — the box started at the bottom. To find the current line I wrote lines.findIndex(l => l.t > now) - 1. Before the first line, findIndex returns 0, so i became -1, and the guard if (i < 0) i = lines.length - 1 turned “nothing yet” into “the last line”. Playback opened scrolled to the outro.

Bug 2 — it then scrolled to the bottom on the first chord. Fixing bug 1, I centred the active line with p.offsetTop. offsetTop is measured from the nearest positioned ancestor — and the lyric box had no position: relative, so it returned the line’s distance from the top of the page, thousands of pixels. scrollTop clamped to maxScroll: straight to the bottom.

The correct form does not care about positioning contexts:

const br = box.getBoundingClientRect(), pr = p.getBoundingClientRect();
box.scrollTop += (pr.top - br.top) - (box.clientHeight - pr.height) / 2;

And the lesson about testing matters more than the fix. I verified the index logic in Node — it passed. I looked at a screenshot — I misread which line was highlighted. Neither could see a layout bug. What worked was a numeric probe in headless Chrome: stub currentTime, dispatch timeupdate, then read the real DOM back out.

chrome --headless=new --dump-dom --virtual-time-budget=8000 \
       --window-size=760,520 "file:///probe.html"
Object.defineProperty(a, 'currentTime', { get: () => 100, configurable: true });
a.dispatchEvent(new Event('timeupdate'));
// then report: active line index, its offset inside the box, scrollTop, maxScroll

The pass condition is a number, not an opinion: the active line’s offset inside the box should be about half the box height (I got 124 px in a 274 px box), scrollTop = 0 with nothing highlighted at t = 0, and scrollTop = maxScroll only for the final lines.

Results

Eighteen songs, none of which shipped a timestamp. Every one now has line-level LRC and SRT, generated from lyrics plus audio alone. The numbers below are the English page set; the Mandarin row in the coverage table above comes from a separate 11-track validation batch I ran through the same alignment code, not from these players.

Songs18
Subtitle lines342
Mean anchor coverage0.96
Songs with every line anchored9 / 18 (every line got a time)
Processing time63–90 s per song, CPU-only faster-whisper

Running it on CPU is deliberate, not a limitation I worked around: all 18 tracks were aligned on a Core i9-13900K (in the Windows VM that runs this pipeline), because the GPU stays reserved for the generation pass that produces these songs, and 63–90 s of faster-whisper int8 per track is cheap enough that queuing the aligner behind a diffusion job would cost more than the CPU time saves.

Each player below scrolls its lyrics, keeps the current line centred, and has a .srt download next to it if you want to burn the subtitles into a video or open them in VLC.

Tiny Borrowed Shoes album cover
Tiny Borrowed Shoes · Celtic folk · 21 lines · 139 s.srt
Mist Over Skellig album cover
Mist Over Skellig · Celtic folk · 18 lines · 193 s.srt
Willow by the Silver Stream album cover
Willow by the Silver Stream · Celtic folk · 19 lines · 197 s.srt
Clockwork Starlight album cover
Clockwork Starlight · Japanese healing · 21 lines · 160 s.srt
Waltz of the Dandelion album cover
Waltz of the Dandelion · Japanese healing · 18 lines · 179 s.srt
Whale in the Milky Way album cover
Whale in the Milky Way · Japanese healing · 20 lines · 164 s.srt
Puddle Jumping Parade album cover
Puddle Jumping Parade · Japanese cute · 21 lines · 105 s.srt
Strawberry Cloud Bakery album cover
Strawberry Cloud Bakery · Japanese cute · 19 lines · 100 s.srt
Captain Marshmallow album cover
Captain Marshmallow · Japanese cute · 22 lines · 95 s.srt
Lanterns on the Lotus Pond album cover
Lanterns on the Lotus Pond · Guofeng fusion · 19 lines · 131 s.srt
Wings of the Kunpeng album cover
Wings of the Kunpeng · Guofeng fusion · 19 lines · 213 s.srt
The Puppeteer's Thread album cover
The Puppeteer's Thread · Guofeng fusion · 19 lines · 221 s.srt
Paper Airplane in September album cover
Paper Airplane in September · Acoustic folk · 18 lines · 128 s.srt
Porridge and Evening Smoke album cover
Porridge and Evening Smoke · Acoustic folk · 17 lines · 144 s.srt
Green Train to the Sea album cover
Green Train to the Sea · Acoustic folk · 17 lines · 108 s.srt
Sanctuary of the Dawn album cover
Sanctuary of the Dawn · Choral · 19 lines · 182 s.srt
Look Upon Your Path album cover
Look Upon Your Path · Choral · 17 lines · 122 s.srt
Bells of the Winter Star album cover
Bells of the Winter Star · Choral · 18 lines · 177 s.srt
💼 Work With Me

Looking for high-impact engineering work or custom systems?

From front-end web apps down to edge proxies and container automation. Open to serious freelance client projects and senior engineering roles.

Lee Teong Hoe

Lee Teong Hoe · Mr Hoelee

Full-stack developer & DevOps engineer based in Malaysia. I run my own production infrastructure and build web systems for small businesses — this blog is the living record.

Hiring? Profile & stack · Malaysian SME? Website design & development ↗ · All posts