vee
3.1 KiB
Speech pipeline benchmark — 2026-08-28
Environment
- Windows / Python 3.13
sherpa-onnx==1.13.6, CPU provider, 8 ASR threads- SenseVoiceSmall full precision,
language=zh,use_itn=true - Silero VAD: threshold 0.45, min speech 0.16 s, min silence 0.65 s
- 48 text regressions（including reported field failures plus ratio/coordination matrices）
- 16 capability-alignment cases replayed 100 times（1600 samples）
- 43 catalog capability definitions × 5 resolver projections（215 subcases）
- 10 local 16 kHz mono Windows SAPI WAVs covering filler, target refinement, cross-domain correction, normal multi-command, number/direction correction, negation, unrelated speech, fast speech and slow speech
The WAV set is synthetic and deliberately kept under ignored artifacts/tmp; it is useful for
repeatability but is not a substitute for TAKSTAR V2 recordings in the real cabin.
Before / after
| Metric | Before | After |
|---|---|---|
| Raw exact transcription (10 WAV) | 8/10 | 8/10 |
| Expected understanding text (same 10 cases) | 2/10 | 10/10 |
| Intent exact match | no structured speech Intent | 10/10 |
| Device / action / parameter exact | no structured speech metrics | 10/10 / 10/10 / 10/10 |
| Correction success | unsafe text-only heuristics | 4/4 |
| False command activation | 1 observed wrong-direction activation | 0/2 no-command cases |
| Severe wrong device / wrong number | not reported | 0 / 0 |
| Duplicate transcript rate | not observable (single callback) | 0% |
| ASR inference P50 / P95 | 80 / 110 ms | 79.7 / 110.5 ms |
| Semantic P50 / P95 | 0.03 ms (old regex only) | 0.33 / 0.56 ms（1600 capability samples） |
| Command resolution P50 / P95 | not available | 0.03 / 0.06 ms |
| Steady end-to-end P50 / P95 | not instrumented | 95.3 / 134.1 ms |
| Cold first request | not instrumented end-to-end | 1400.9 ms |
Raw exact stays at 80% because two files differ only in acoustic homophone/punctuation output;
raw_text is intentionally not rewritten. The semantic and business metrics operate on separate
fields.
Full precision vs int8 probe
On sherpa-onnx's official Chinese test WAV, both files produced the same transcript in this one probe:
| Model | First call including load | Warm P50 | Warm P95 |
|---|---|---|---|
model.onnx |
1514.9 ms | 90.9 ms | 92.7 ms |
model.int8.onnx |
889.6 ms | 77.4 ms | 81.8 ms |
This is insufficient acoustic coverage to prove equal accuracy, so the accuracy-first default
remains full precision. The model file stays configurable through SPEECH_MODEL_FILE.
VAD tuning probe
Thresholds 0.35 / 0.45 / 0.55 and end silence 0.45 / 0.65 / 0.85 s were replayed against the available Chinese test WAV. All combinations found one segment; threshold 0.45 preserved the complete utterance boundary. The selected 0.65 s server endpoint, 0.25 s pre-roll and 0.35 s post-roll are conservative for short commands. Real cabin noise and long-pause recordings remain required before claiming production calibration.
Reproduce current metrics with:
.venv/Scripts/python tools/benchmark_speech_pipeline.py \
--audio-dir artifacts/tmp/speech-regression-audio