Thai, English and Chinese in one sentence: why code-switching is hard to recognize
In meetings, live streams and customer service calls in Thailand, it is entirely ordinary for a sentence to contain an English brand name, an abbreviation or a Chinese personal name. For the speaker this is no obstacle at all. For speech recognition it is a separate kind of problem. This article explains where errors in mixed-language speech usually appear, and how to score them so that a different way of writing a word is not counted as a mishearing.
1. What mixed speech is and why it is hard
Code-switching means one speaker alternating between several languages in the same stretch of speech. When it happens inside a sentence, such as an English product name embedded in a Thai sentence, it is called intra-sentential switching, and it is the hardest kind.
The difficulty comes from three sides:
- Data. Most training audio has one language per recording. The model rarely sees the moment a switch happens, so it does not know what can follow a Thai word.
- Pronunciation. Few consonants can end a syllable in Thai, so English words are re-pronounced to fit Thai rules once they are borrowed. For example, ball is written บอล and the final letter is read as an n sound. English words spoken by Thai speakers do not sound like recordings of native speakers, so what the model learned from English data cannot be copied over directly.
- Script. The model does not only have to hear which word it is, it also has to decide which script to write it in. The same word can be written in Latin letters or in Thai, both are in use, and the reference answer will contain only one of them.
2. Where errors usually appear
Brand and product names
Words such as Shopee, LINE and Grab are written in Latin letters in product interfaces, and often in Thai in casual chat, for example ช้อปปี้ and ไลน์. The model may output either form, or a string of Thai words that sound close but have nothing to do with the brand. It is worse when the brand is also an ordinary word.
Spelling of loanwords
Some loanwords have a settled Thai spelling. Almost nobody writes คอมพิวเตอร์ as computer. Others have no settled form: "meeting" is written as meeting by some people and as มีตติ้ง by others. The second kind is where most disagreements in evaluation come from.
Chinese personal names and place names
When a Chinese name sits in a Thai sentence, the speaker may read it in Mandarin or in a dialect such as Teochew. Either way, the sound does not map one to one onto the characters: 李明 and 李铭 are pronounced exactly the same, and the audio alone cannot tell them apart. That takes context or a name list.
Language boundaries
Errors usually appear in the one or two words before and after a switch. When the switched part is longer, the model may also get the language wrong for the whole stretch and write a short English phrase as Thai words that sound similar.
Numbers, abbreviations and units
5G, KPI, "สองล้าน" (two million) and "2 ล้าน" are spoken and written differently, and Thai digits ๑๒๓ are mixed with Arabic digits 123. The more languages involved, the more ways there are to write the same thing.
Tones
Thai and Chinese both have tones, but the tone systems differ. A Chinese word inserted into a Thai sentence is sometimes spoken with Thai intonation, which weakens the tone cue. The model then leans more on context, and context is thinnest exactly at the switch.
3. Two common designs and where each fails
There are two common ways to handle multilingual speech.
One is to identify the language first and then hand the audio to a model for that language. Switching between sentences is no problem for it: this sentence is Thai, the next is English, and each goes its own way. Switching inside a sentence does not work, because language identification works on a whole sentence or even a longer stretch. An English word inside a sentence is recognized as Thai and written as Thai words that sound similar.
The other is one model that supports several languages, with a vocabulary that holds several scripts. It can switch inside a sentence, but words from different languages compete in the same vocabulary. When the training data has few examples with switching, the model tends to stay in the base language of the sentence and pull the embedded word toward it.
Both designs need to be checked with a test set that contains switching. A good score on a single-language test set does not imply a good score on mixed speech.
4. Scoring: WER and CER are not enough
Thai has no spaces, so WER depends on the tokenizer, as we explained in another post on Thai word segmentation and CER. The same is true of Chinese. English is normally counted by word, and counting by character exaggerates the differences. In a line of mixed text no single unit suits all three scripts.
The standard approach is mixed error rate (MER): segment each script using its natural unit into a single sequence of tokens, then compute edit distance across the whole sequence.
MER = (S + D + I) / NS, D and I are the numbers of substituted, deleted and inserted tokens, and N is the number of tokens in the reference. Tokens are cut like this:
- Thai: by character, with each vowel and tone mark counted as one.
- Chinese: by character.
- English letters and digits: a continuous run counts as one word.
Here is an example of this convention.
- Reference: ผมคุยกับ 李明 เรื่อง Shopee
- Hypothesis A: ผมคุยกับ 李名 เรื่อง shopee. After normalization only one token differs, 名 against 明, so 1 error. Case does not count.
- Hypothesis B: ผมคุยกับ 李明 เรื่อง ช้อปปี้. The sound is completely right, but the brand is written in Thai, so 7 errors: 1 substitution and 6 insertions.
- Meaning: I talked with Li Ming about Shopee.
Hypothesis B shows why the script convention has to be fixed first. An English word takes only 1 token in the reference, so misspelling a letter costs 1. Writing it in Thai instead takes 7 characters and costs 7, even though what was heard is the same.
MER also has a blind spot. Its denominator is mostly Thai characters, and English words and Chinese names make up a small share. The overall MER can look low while proper nouns are wrong in many places. So besides the overall MER, report the error rate per language, and an error rate computed only on tokens at the points where the language switches.
5. Normalization before scoring
Normalization applies the same treatment to the reference and the hypothesis, and the rules are written before the evaluation:
- Unicode is normalized with NFKC, full-width letters become half-width, and Thai diacritic ordering follows [Thai diacritic order rules].
- English is lowercased. Punctuation is stripped for core lexical scoring (after normalizing full-width and half-width forms when punctuation is evaluated separately).
- Chinese characters are unified to a single script variant (Simplified or Traditional).
- Number forms are unified: Thai digits and Arabic digits count as the same, and for spoken and written numbers, one form is chosen in the annotation guideline.
The most important part is the rule for loanwords, which we suggest writing as three lines:
- Brand names, product names and abbreviations keep their original Latin letters.
- Loanwords with an established Thai spelling are written in Thai, for example คอมพิวเตอร์.
- Chinese personal names and place names are written in Chinese characters.
The alias list contains only the spellings the guideline explicitly allows. It has to be fixed before the evaluation is run, used unchanged for every system, and published with the results. The alias list is the easiest place to tune a score quietly, and aliases added or removed afterwards do not count as normalization.
6. Building the test set
- Use real conversations rather than reading from a word list. Brand names read aloud are pronounced too carefully and are far from real use.
- Annotate against one guideline, with two people annotating each recording. Resolve disagreements by revising the guideline, not case by case.
- For every sentence, record which languages are used, how many times it switches, and the type of switch: one word embedded, a phrase embedded, or a whole sentence. Report by these groups, so that you can see whether errors rise with more switching and which kind of switch is hardest.
- Brand names and personal names in the test set must not enter the training data, and must not enter any word list used for tuning.
The size, version and test date of our in-house mixed-speech set are on the public benchmarks page.
7. What we do
In the product, English brand names and Chinese personal names inside a sentence are output in the language the speaker actually used, and are not forced into Thai transliteration. For evaluation, the main metric on the mixed-speech set is MER. The test set, the normalization rules and the calculation method are on the public benchmarks page.