BENCHMARKS

Public benchmarks

Evaluated on the official Thai test splits of FLEURS, Common Voice 17 and GigaSpeech 2, a 500-sample real-world ASR benchmark, and a 100-utterance Thai TTS evaluation calibrated with per-annotator Z-score normalization.

Last updated Nov 11, 2026 · Models DataArc-ASR-TH · DataArc-TTS-TH

Core benchmark overview

DataArc-ASR-TH
92.79%
FLEURS · Common Voice 17 · GigaSpeech 2
ElevenLabs Scribe v2
91.68%
FLEURS · Common Voice 17 · GigaSpeech 2
Google Chirp v3
91.44%
FLEURS · Common Voice 17 · GigaSpeech 2

Public test suite speech recognition accuracy table
Test set	Language	Metric	DataArc-ASR-TH	ElevenLabs Scribe v2	Google Chirp v3
FLEURS · Common Voice 17 · GigaSpeech 2	Thai	Accuracy higher is better	92.79%	91.68%	91.44%

Accuracy: higher is better (%). Combined score across the official Thai test splits of FLEURS, Common Voice 17 and GigaSpeech 2. Competitors were evaluated via their public APIs with default settings. Updated Nov 11, 2026.

DataArc-TTS-TH
4.41 / 5
Thai subjective test set · 100 sentences
ElevenLabs v3
4.33 / 5
Thai subjective test set · 100 sentences
Google Gemini TTS
4.29 / 5
Thai subjective test set · 100 sentences

Thai speech synthesis quality overview table
Test set	Language	Metric	DataArc-TTS-TH	ElevenLabs v3	Google Gemini TTS
Thai subjective test set · 100 sentences	Thai	Weighted score higher is better	4.41	4.33	4.29
Thai subjective test set · 100 sentences	Thai	MOS higher is better	4.39	4.32	4.27

Weighted score and MOS are 5-point blind evaluations from 4 native Thai speakers, calibrated via independent per-annotator Z-score normalization before averaging (higher is better). Updated Nov 11, 2026; see the full multi-model breakdown below.

IN-HOUSE DATASET · 500 SAMPLES

In-house dataset deep-dive (500 real-world samples)

Scored with our unified evaluation script (cal_wer_cer.py: PyThaiNLP newmm word segmentation + regex normalization for numbers, currencies and dates) across 10 business domains and code-switching scenarios.

1. Core metrics across four mainstream ASR engines (500 samples)

Holistic metrics comparison across four mainstream ASR engines
Evaluation dimension	Google Gemini 3.5 Transcribe	DataArc-ASR-TH-async	DataArc-ASR-TH-streaming	Cartesia ink-whisper (Cloud API)	Comparison & analysis
Median CER (P50)	2.82%	3.68%	7.46%	17.06%	Gemini 3.5 ranks first in median precision, with 50% of clips at or below 2.82% CER
Mean CER (Macro CER)	6.23%	5.88%	9.46%	19.27%	DataArc-ASR-TH-async and Gemini sit in the top tier, well ahead of Cartesia
P90 CER	16.74%	14.68%	23.69%	35.96%	DataArc-ASR-TH-async and Gemini remain resilient on long-tail clips with background noise
Median WER (P50)	5.00%	5.26%	11.76%	30.00%	Gemini achieves the best word-level median accuracy (5.00%)
Mean WER (Macro WER)	8.03%	7.42%	12.50%	31.46%	DataArc-ASR-TH-async and Gemini perform closely on macro word error rate
Zero-error clips (0 CER)	233 / 500 (46.6%)	192 / 500 (38.4%)	100 / 500 (20.0%)	36 / 500 (7.2%)	Nearly half of Gemini's transcripts have zero character errors
Excellent clips (CER ≤ 5%)	300 / 500 (60.0%)	278 / 500 (55.6%)	185 / 500 (37.0%)	72 / 500 (14.4%)	60% of clips are transcribed with virtually no noticeable errors
Good clips (CER ≤ 10%)	386 / 500 (77.2%)	400 / 500 (80.0%)	330 / 500 (66.0%)	130 / 500 (26.0%)	The vast majority of samples fall into the production-ready range
Recognition success rate	500 / 500 (100.0%)	500 / 500 (100.0%)	500 / 500 (100.0%)	500 / 500 (100.0%)	All 500 utterances completed without timeouts or dropped streams
Average response latency	4.64 s (P50: 4.57s)	Full-clip upload (1–3s)	44.03 ms (first-token)	0.26 s (P50: 0.23s)	Gemini runs non-streaming LLM inference with higher end-to-end latency
Single-request RTF (P50)	0.96	0.18	0.06	0.05	DataArc-ASR-TH-streaming and Cartesia deliver ultra-fast real-time factors
Deployment & operations	Cloud API (managed)	Self-hosted GPU (13.4 GB VRAM)	Self-hosted GPU (2.3 GB) / CPU	Cloud API (managed)	Gemini and Cartesia require zero local GPU infrastructure

2. Median CER (P50) breakdown by business domain

Median P50 CER comparison across 10 business domains
Business domain	Samples	Gemini 3.5 (P50)	DataArc-ASR-TH-async (P50)	DataArc-ASR-TH-streaming (P50)	Cartesia (P50)	Domain traits & notes
Technology	56	2.27%	6.28%	11.63%	11.51%	Strong Gemini result: accurately captures English model loanwords such as `430i Coupe`, `LCI`, `M440i`
Legal & Government	4	2.00%	2.67%	3.83%	13.53%	Formal vocabulary; both Gemini and DataArc-ASR-TH-async reach very high precision
Finance	147	2.67%	2.63%	5.56%	20.00%	High accuracy on dense Thai currency figures and numeric expressions
Daily Conversation	202	3.80%	4.51%	7.58%	16.77%	Gemini's contextual inference handles casual elision better than DataArc-ASR-TH-async
Entertainment	8	1.06%	3.12%	6.32%	11.59%	Strong semantic recovery from Gemini despite background music and effects
Travel & Culture	7	0.00%	3.90%	9.09%	22.08%	100% hit rate on Thai place names and cultural proper nouns
Other (Creator & Media)	54	0.00%	4.21%	7.50%	16.95%	Short-video voiceovers reach 0.00% median character error rate
Interview & Podcast	4	5.57%	3.61%	5.83%	18.29%	Solid transcription quality across long-form conversational turns
Education	6	8.93%	4.57%	8.58%	9.19%	Steady speaking pace with clean overall readability
News	5	9.86%	2.50%	5.26%	18.42%	DataArc-ASR-TH-async retains a clear advantage on standard broadcast news delivery

3. Code-switching scenario performance

Character error rate on pure Thai vs Thai-English code-switching
Data subset	Samples	Gemini 3.5 CER (Mean / P50)	DataArc-ASR-TH-async CER (Mean)	DataArc-ASR-TH-streaming CER (Mean)	Key takeaway
Pure Thai (Non Code-Switching)	350	4.91% / 0.00%	5.40%	8.77%	Gemini achieves 0.00% median CER across pure Thai clips
Thai-English (Code-Switching)	150	9.29% / 7.00%	7.00%	11.08%	Strong handling of embedded English loanwords and technical abbreviations

THAI TTS BENCHMARK · Z-SCORE NORMALIZED

Thai speech synthesis (TTS) subjective benchmark (Z-score normalized)

Blind 5-point subjective evaluation across 100 Thai utterances per model rated by 4 native speakers. Scores are independently Z-score normalized per annotator and dimension to remove individual leniency/strictness bias, then mapped back to the global baseline (ordered by weighted_score descending).

Thai TTS Z-score normalized multi-dimension subjective benchmark table
#	Model	Weighted	MOS	Text fidelity20%	Clarity20%	Speaking rate5%	Prosody25%	Emotion20%	Pauses & breath10%
1	DataArc-TTS-THOurs	4.41	4.39	4.53	4.72	4.23	4.12	4.35	4.33
2	ElevenLabs v3	4.33	4.32	4.61	4.54	4.27	3.88	4.33	4.30
3	Google Gemini TTS	4.29	4.27	4.56	4.54	4.04	4.04	4.23	4.16
4	Google Chirp 3 HD	4.07	4.03	4.52	4.61	3.66	3.77	3.73	3.85
5	CosyVoice 2	3.72	3.76	3.73	3.71	3.96	3.58	3.80	3.80
6	Thonburian	3.72	3.74	3.00	3.31	4.04	4.17	4.17	3.72

Independent Z-score normalization & dimension weights

Individual raters systematically score higher or lower than peers. Each rater's raw scores are standardized per dimension, rescaled to the combined global mean (3.9316) and standard deviation (0.7789) on the 5-point scale, and averaged across all 4 annotators:

z_score    = (score - annotator_mean) / annotator_std
norm_score = z_score × global_std + global_mean
final      = mean(norm_score across 4 annotators)

Weighted score dimension breakdown

Prosody25%
Text fidelity20%
Clarity20%
Emotional consistency20%
Pauses & breath10%
Speaking rate5%

Annotator raw score distribution & global baseline

Per-annotator raw weighted score mean and standard deviation
Annotator	Raw mean	Raw std
Annotator #1	3.3906	0.6214
Annotator #2	4.1689	0.6788
Annotator #3	3.4169	0.8041
Annotator #4	4.4966	0.3900
Global baseline	3.9316	0.7789

Raw 5-point weighted_score distribution per native-speaker annotator before Z-score calibration to the global baseline.

METHODOLOGY

Methodology

Data
ASR public benchmarks use the official Thai test splits of FLEURS, Common Voice 17 and GigaSpeech 2; the ASR in-house benchmark uses 500 human-annotated real-world clips across 10 domains (350 pure Thai, 150 Thai-English code-switched); the TTS benchmark uses 100 Thai utterances per model, all excluded from training.
Text normalization and rater calibration
ASR transcripts pass through Unicode NFKC normalization, canonical Thai vowel/tone ordering, regex normalization for numbers, currencies and dates, and PyThaiNLP newmm segmentation. TTS subjective ratings use independent per-annotator Z-score normalization rescaled to the global baseline to eliminate individual rater bias.
Metrics
ASR reports combined recognition accuracy (%) on the public suite and P50, Macro and P90 CER/WER on the 500-sample suite. TTS reports 5-point Z-score normalized weighted scores (combining prosody, text fidelity, clarity, emotional consistency, pauses & breath, and speaking rate) alongside MOS.
Reproduction
All models are evaluated on identical inputs under the same scoring pipeline. Download the raw CSV below for the public ASR suite, the 500-sample in-house ASR tables, and the full TTS Z-score normalized scores and annotator baselines.

Raw results CSV Why we use CER for Thai