Subtitles as LLM input: the timestamps are the smallest problem.

A caption track looks like a clean transcript because it is written in the same language. It is not. It is a timing structure: short fragments cut for on-screen reading, produced by a speech recogniser working a rolling window. Feed it to a model as-is and you are teaching the model to repeat itself, to answer in fragments, and to treat [Applause] as vocabulary.

What the model actually needs

• Sentence-sized units. Retrieval, summarisation and fine-tuning all assume a chunk is a coherent thought. A cue is a screen-width decision, not a thought.
• No repeated text. A rolling transcript triples the same words; a model learns the repetition as a pattern.
• No timing tokens. 00:00:12 is not content. It costs tokens, pollutes quotes, and occasionally gets answered back.
• A noise policy. Applause, music and laughter markers are real information in some tasks and pure noise in others — but that decision has to be made deliberately, not inherited from the file.

Problem one: auto captions repeat themselves

Automatic captions are generated with a rolling window: each new event contains the tail of the previous one plus a few new words. On screen it looks like a line being extended. In a file it is the same sentence written two or three times, each copy slightly longer than the last.

and that is why we
and that is why we rebuilt the intro
why we rebuilt the intro in a weekend

and that is why we
rebuilt the intro
in a weekend

The right-hand column is what our extraction produces: the download path de-duplicates rolling overlap on automatic tracks and only drops adjacent duplicate lines on manually uploaded ones, because a human captioner is not repeating themselves on purpose. If you have ever pasted a raw auto-transcript into a prompt and got an answer that stuttered, this is why.

Problem two: a cue is not a sentence

Even after de-duplication, an automatic track is a column of two-to-five-word fragments, because each one had to fit on screen for the length of time it is displayed. Merging them back into sentences needs three rules, and all three matter:

This is the step people skip, and it is the one that decides whether your dataset reads like speech or like teleprompter fragments. A regex cannot do it: joining every cue into one paragraph destroys the boundaries you still need for retrieval.

Problem three: bracketed noise and missing words

• Bracketed markers. [Applause], [Music], [Laughter] are inserted by the recogniser. Keep them if you are studying audience reaction; strip them if you are building a text corpus.
• No punctuation at all in long stretches. Fast speech, technical vocabulary and heavy accents all reduce the recogniser's confidence, and confidence is what produces punctuation.
• Systematic mishearing. Names, product names and numbers are wrong in consistent ways. A model trained on them learns the wrong spelling — correct the entities you care about before, not after.
• Rolling overlap that survives. If your source is a third-party dump rather than a fresh extraction, de-duplication may never have happened. Check the first twenty lines for a repeating tail.

Chunking once the text is clean

A pipeline that holds up

1. Pick the creator track over the automatic oneAlmost every quality problem downstream is cheaper to fix at this step. A creator-uploaded track arrives punctuated, correctly spelled and already sentence-shaped; an automatic track arrives as fragments you will spend hours repairing.What to do: Check which tracks the video publishes before extracting. If both exist, take the manual one — even if it is shorter, it is the one a human wrote.
2. Take plain text for the words, and the timing only for the gapsSRT and VTT are timing structures; every line carries two clock values you strip before the text is usable. But the timestamps are also the only record of silence — so decide once whether your merge needs gaps, and do not hand-parse SRT for text it already lost.What to do: Use TXT as the text source: one line per cue, a blank line between, no timing at all. If the merge needs silence breaks, save the SRT from the same extraction and read the gaps from it — the pair costs one extra call, not a second download.
3. Merge fragments into sentences, then stopJoin on sentence-ending punctuation first, break on a length cap second — and if you kept the SRT, break on gaps longer than about two seconds as well. Over-merging is worse than under-merging: one 3,000-word chunk retrieves badly and fine-tunes worse.What to do: Keep the merge rules in code, not in a prompt. Re-run them when the source track changes, and store the sentence boundaries — you will need them again for citations.
4. Strip noise, fix entities, record what you didDecide explicitly whether [Music] stays, normalise whitespace, and correct the names that matter for your domain. Then write down the decisions — a dataset whose cleaning steps are undocumented is one nobody can reproduce.What to do: Keep a one-page cleaning log next to the data: track kind, rules applied, entity fixes, and the date. Reproducibility is the difference between a dataset and a pile of text.

Rights, before you build anything at scale

A caption track is a transcript of somebody's speech, and speech is protected. Being technically able to extract it is not the same as being allowed to build a corpus from it. The safe perimeter is narrow and worth stating plainly: your own videos, videos explicitly licensed for reuse, and material you have permission to use. Downloading other people's captions in bulk to train or publish a model is a copyright and terms-of-service problem, not a technical one — and no amount of cleaning changes that.

For research on a handful of videos, read them one at a time and keep the citations. For anything larger, get permission first. This is also the more durable choice: a dataset you can defend is one you can ship.