vee
68 lines
3.1 KiB
Markdown
68 lines
3.1 KiB
Markdown
# Speech pipeline benchmark — 2026-08-28
|
||
|
||
## Environment
|
||
|
||
- Windows / Python 3.13
|
||
- `sherpa-onnx==1.13.6`, CPU provider, 8 ASR threads
|
||
- SenseVoiceSmall full precision, `language=zh`, `use_itn=true`
|
||
- Silero VAD: threshold 0.45, min speech 0.16 s, min silence 0.65 s
|
||
- 48 text regressions（including reported field failures plus ratio/coordination matrices）
|
||
- 16 capability-alignment cases replayed 100 times（1600 samples）
|
||
- 43 catalog capability definitions × 5 resolver projections（215 subcases）
|
||
- 10 local 16 kHz mono Windows SAPI WAVs covering filler, target refinement,
|
||
cross-domain correction, normal multi-command, number/direction correction,
|
||
negation, unrelated speech, fast speech and slow speech
|
||
|
||
The WAV set is synthetic and deliberately kept under ignored `artifacts/tmp`; it is useful for
|
||
repeatability but is not a substitute for TAKSTAR V2 recordings in the real cabin.
|
||
|
||
## Before / after
|
||
|
||
| Metric | Before | After |
|
||
|---|---:|---:|
|
||
| Raw exact transcription (10 WAV) | 8/10 | 8/10 |
|
||
| Expected understanding text (same 10 cases) | 2/10 | 10/10 |
|
||
| Intent exact match | no structured speech Intent | 10/10 |
|
||
| Device / action / parameter exact | no structured speech metrics | 10/10 / 10/10 / 10/10 |
|
||
| Correction success | unsafe text-only heuristics | 4/4 |
|
||
| False command activation | 1 observed wrong-direction activation | 0/2 no-command cases |
|
||
| Severe wrong device / wrong number | not reported | 0 / 0 |
|
||
| Duplicate transcript rate | not observable (single callback) | 0% |
|
||
| ASR inference P50 / P95 | 80 / 110 ms | 79.7 / 110.5 ms |
|
||
| Semantic P50 / P95 | 0.03 ms (old regex only) | 0.33 / 0.56 ms（1600 capability samples） |
|
||
| Command resolution P50 / P95 | not available | 0.03 / 0.06 ms |
|
||
| Steady end-to-end P50 / P95 | not instrumented | 95.3 / 134.1 ms |
|
||
| Cold first request | not instrumented end-to-end | 1400.9 ms |
|
||
|
||
Raw exact stays at 80% because two files differ only in acoustic homophone/punctuation output;
|
||
`raw_text` is intentionally not rewritten. The semantic and business metrics operate on separate
|
||
fields.
|
||
|
||
## Full precision vs int8 probe
|
||
|
||
On sherpa-onnx's official Chinese test WAV, both files produced the same transcript in this one
|
||
probe:
|
||
|
||
| Model | First call including load | Warm P50 | Warm P95 |
|
||
|---|---:|---:|---:|
|
||
| `model.onnx` | 1514.9 ms | 90.9 ms | 92.7 ms |
|
||
| `model.int8.onnx` | 889.6 ms | 77.4 ms | 81.8 ms |
|
||
|
||
This is insufficient acoustic coverage to prove equal accuracy, so the accuracy-first default
|
||
remains full precision. The model file stays configurable through `SPEECH_MODEL_FILE`.
|
||
|
||
## VAD tuning probe
|
||
|
||
Thresholds 0.35 / 0.45 / 0.55 and end silence 0.45 / 0.65 / 0.85 s were replayed against the
|
||
available Chinese test WAV. All combinations found one segment; threshold 0.45 preserved the
|
||
complete utterance boundary. The selected 0.65 s server endpoint, 0.25 s pre-roll and 0.35 s
|
||
post-roll are conservative for short commands. Real cabin noise and long-pause recordings remain
|
||
required before claiming production calibration.
|
||
|
||
Reproduce current metrics with:
|
||
|
||
```bash
|
||
.venv/Scripts/python tools/benchmark_speech_pipeline.py \
|
||
--audio-dir artifacts/tmp/speech-regression-audio
|
||
```
|