# Speech pipeline benchmark — 2026-08-28

## Environment

- Windows / Python 3.13
- `sherpa-onnx==1.13.6`, CPU provider, 8 ASR threads
- SenseVoiceSmall full precision, `language=zh`, `use_itn=true`
- Silero VAD: threshold 0.45, min speech 0.16 s, min silence 0.65 s
- 48 text regressions（including reported field failures plus ratio/coordination matrices）
- 16 capability-alignment cases replayed 100 times（1600 samples）
- 43 catalog capability definitions × 5 resolver projections（215 subcases）
- 10 local 16 kHz mono Windows SAPI WAVs covering filler, target refinement,
  cross-domain correction, normal multi-command, number/direction correction,
  negation, unrelated speech, fast speech and slow speech

The WAV set is synthetic and deliberately kept under ignored `artifacts/tmp`; it is useful for
repeatability but is not a substitute for TAKSTAR V2 recordings in the real cabin.

## Before / after

| Metric | Before | After |
|---|---:|---:|
| Raw exact transcription (10 WAV) | 8/10 | 8/10 |
| Expected understanding text (same 10 cases) | 2/10 | 10/10 |
| Intent exact match | no structured speech Intent | 10/10 |
| Device / action / parameter exact | no structured speech metrics | 10/10 / 10/10 / 10/10 |
| Correction success | unsafe text-only heuristics | 4/4 |
| False command activation | 1 observed wrong-direction activation | 0/2 no-command cases |
| Severe wrong device / wrong number | not reported | 0 / 0 |
| Duplicate transcript rate | not observable (single callback) | 0% |
| ASR inference P50 / P95 | 80 / 110 ms | 79.7 / 110.5 ms |
| Semantic P50 / P95 | 0.03 ms (old regex only) | 0.33 / 0.56 ms（1600 capability samples） |
| Command resolution P50 / P95 | not available | 0.03 / 0.06 ms |
| Steady end-to-end P50 / P95 | not instrumented | 95.3 / 134.1 ms |
| Cold first request | not instrumented end-to-end | 1400.9 ms |

Raw exact stays at 80% because two files differ only in acoustic homophone/punctuation output;
`raw_text` is intentionally not rewritten. The semantic and business metrics operate on separate
fields.

## Full precision vs int8 probe

On sherpa-onnx's official Chinese test WAV, both files produced the same transcript in this one
probe:

| Model | First call including load | Warm P50 | Warm P95 |
|---|---:|---:|---:|
| `model.onnx` | 1514.9 ms | 90.9 ms | 92.7 ms |
| `model.int8.onnx` | 889.6 ms | 77.4 ms | 81.8 ms |

This is insufficient acoustic coverage to prove equal accuracy, so the accuracy-first default
remains full precision. The model file stays configurable through `SPEECH_MODEL_FILE`.

## VAD tuning probe

Thresholds 0.35 / 0.45 / 0.55 and end silence 0.45 / 0.65 / 0.85 s were replayed against the
available Chinese test WAV. All combinations found one segment; threshold 0.45 preserved the
complete utterance boundary. The selected 0.65 s server endpoint, 0.25 s pre-roll and 0.35 s
post-roll are conservative for short commands. Real cabin noise and long-pause recordings remain
required before claiming production calibration.

Reproduce current metrics with:

```bash
.venv/Scripts/python tools/benchmark_speech_pipeline.py \
  --audio-dir artifacts/tmp/speech-regression-audio
```