Streaming recognition and offline refinement: balancing latency and accuracy

Live captions pull in two directions. The text has to appear as soon as the words are spoken, and once a sentence is finished it should be as accurate as possible. Streaming models are good at the first, offline models at the second. We use both: streaming recognition produces text while you speak, and at the end of each sentence the audio of that sentence goes to an offline model for refinement, and the refined result replaces the original text. This article covers how the two results merge, how the interface transitions between them, and where things can go wrong and how we fall back.

1. Why latency and accuracy pull against each other

A streaming model has to produce text before the sentence is finished, so it can only rely on what it has already heard plus a very short look ahead. But many words are only settled by what follows: homophones depend on the next words, numbers depend on the unit, and since Thai has no spaces, even where a word ends depends on what comes after. An offline model reads the whole sentence at once, and most of these ambiguities disappear with the extra context.

So "faster" and "more accurate" come down to the same choice: how long to wait before committing. Wait less and text appears sooner, with less to base the decision on. Wait longer and there is more evidence, but the user also stares at a blank screen for longer. Streaming recognition chooses not to wait: it gives a provisional result first, and allows it to be changed later.

Latency also has to be looked at in two parts. First-token latency is the time from the start of speech to the first character on screen. Settling latency is the time from the end of a sentence until it becomes text that will not change. Refinement only affects the second one and never slows the first token.

2. Three kinds of text: partial, final and refined

• Partial: everything heard so far in the current sentence. Each new partial replaces the previous one as a whole, it is not appended, and it can be empty. Characters already on screen can be changed.
• Final: after the speaker pauses for about two seconds, the server settles the sentence. The connection stays open and waits for the next one.
• Refined result: the browser keeps the audio it has just sent. When a final arrives, it cuts out that sentence, turns it into a WAV file and sends it to the offline recognition model, the same one used for file transcription, which returns text with punctuation.

How "that sentence" is cut out is an implementation detail. We do not ask the server for the audio. Locally we look for the first silence of 1.5 seconds or longer, and cut from the first frame with sound up to that point, with a little margin on both sides. The rest of the audio is kept for the next sentence.

• partial: ผมจะส่งรายงานให้
• final: ผมจะส่งรายงานให้ภายในวันสุก
• refined: ผมจะส่งรายงานให้ภายในวันศุกร์
• meaning: I will send you the report by Friday.

In this example "ศุกร์" (Friday) and "สุก" (ripe) sound the same. While the streaming model has no more context after the word, either spelling can come out. With the whole sentence available, the offline model finds it easier to choose the right one.

3. Merge rules

1. A partial replaces the whole current line. It is never accumulated.
2. Each final becomes a numbered sentence, added after the existing sentences.
3. A refinement replaces by sentence number, not by position. When it returns, later sentences may already be on screen, and several refinements may be in flight at once. Each result changes only its own sentence, so it does not matter which one comes back first.
4. If refinement has no result, the streaming text stays. No request is sent when the audio is shorter than 0.25 seconds. If the response is empty, the request fails, the daily limit is reached, or nothing has come back after 12 seconds, the streaming text also stays. Refinement is an improvement, not a dependency.
5. For screen readers, a settled sentence is read out once, in a separate live region, when it arrives. Later replacements by refinement are not read again, so nobody hears the same sentence twice.

4. The interface transition: from blue to the main color

Three states, three looks:

• The sentence being spoken is blue and ends with a blinking caret.
• A sentence that is final but whose refinement has not returned is still blue.
• After the refinement comes back, or is given up on, the text is replaced and the color fades to the main color over about 0.8 seconds.

Color expresses how certain the text is, not what kind of sentence it is. When readers see blue, they know it may change. When they see the main color, they know it will not. If text is swapped quietly with no signal, readers think they misread it.

Only the color moves, not the layout. The caption area shows just the latest few lines, so older sentences scroll out of view naturally, and the text that gets changed is always the newest one or two sentences, which is where the reader is already looking. When the system is set to reduce motion, the caret stops blinking.

5. Where it goes wrong

• Client and server disagree on sentence boundaries. The server settles a sentence using its own silence detection, while the client slices audio at 1.5 seconds of silence. If a sentence has a longer pause in the middle, or the server cuts in the middle of continuous speech, the audio sent for refinement may include a bit too much or too little. A safer approach is to align using the sentence start time returned by the server. That requires first confirming that the two clocks still agree after a reconnection, so for now we cut by pauses.
• Long continuous speech. Only the most recent 30 seconds of audio are kept locally. For a longer sentence, the beginning does not go into refinement, and the refined text may miss the start of the sentence.
• Very quiet speech. Frames below the noise threshold do not count as sound, so the whole sentence may be treated as silence. There is no audio to refine, and the streaming text stays.
• Interrupted connection. When the server closes the connection normally, the web page reconnects automatically, up to three times in a row. When the connection drops abnormally, it does not reconnect and shows a network error straight away.
• Trial time limit. When it is reached, the server signals it with a dedicated close code, and the page shows a specific explanation instead of a vague "something went wrong".

6. How to measure it

First-token latency is counted like this:

first-token latency
= time the first non-empty
  partial arrives
− start of the first frame
  with sound

The timing is done in the browser, so it covers three parts: capture buffering, the network and server processing. Capture uses frames of 2048 samples, so a frame's length depends on the browser's sample rate: 128 milliseconds at 16 kHz, about 43 milliseconds at 48 kHz. Refinement time is also a browser-side round trip: encoding the WAV, uploading, recognizing and returning. The processing time reported by the server itself is shorter.

To measure what refinement gains, score the two results separately: the final text as it was settled, and the text after refinement, each with CER on the same test set. The difference between them is the improvement refinement brings. Looking only at the accuracy of the final captions cannot show this. The numbers, test sets and hardware are on the public benchmarks page.

DA

DataArc Team