feat: integrate Chinese speech understanding and capability routing

2026-08-28 15:19:44 +08:00
parent 2529b4896a
commit aa12135b22
39 changed files with 6864 additions and 56 deletions
@@ -0,0 +1,67 @@
+# Speech pipeline benchmark — 2026-08-28
+
+## Environment
+
+- Windows / Python 3.13
+- `sherpa-onnx==1.13.6`, CPU provider, 8 ASR threads
+- SenseVoiceSmall full precision, `language=zh`, `use_itn=true`
+- Silero VAD: threshold 0.45, min speech 0.16 s, min silence 0.65 s
+- 48 text regressions（including reported field failures plus ratio/coordination matrices）
+- 16 capability-alignment cases replayed 100 times（1600 samples）
+- 43 catalog capability definitions × 5 resolver projections（215 subcases）
+- 10 local 16 kHz mono Windows SAPI WAVs covering filler, target refinement,
+  cross-domain correction, normal multi-command, number/direction correction,
+  negation, unrelated speech, fast speech and slow speech
+
+The WAV set is synthetic and deliberately kept under ignored `artifacts/tmp`; it is useful for
+repeatability but is not a substitute for TAKSTAR V2 recordings in the real cabin.
+
+## Before / after
+
+| Metric | Before | After |
+|---|---:|---:|
+| Raw exact transcription (10 WAV) | 8/10 | 8/10 |
+| Expected understanding text (same 10 cases) | 2/10 | 10/10 |
+| Intent exact match | no structured speech Intent | 10/10 |
+| Device / action / parameter exact | no structured speech metrics | 10/10 / 10/10 / 10/10 |
+| Correction success | unsafe text-only heuristics | 4/4 |
+| False command activation | 1 observed wrong-direction activation | 0/2 no-command cases |
+| Severe wrong device / wrong number | not reported | 0 / 0 |
+| Duplicate transcript rate | not observable (single callback) | 0% |
+| ASR inference P50 / P95 | 80 / 110 ms | 79.7 / 110.5 ms |
+| Semantic P50 / P95 | 0.03 ms (old regex only) | 0.33 / 0.56 ms（1600 capability samples） |
+| Command resolution P50 / P95 | not available | 0.03 / 0.06 ms |
+| Steady end-to-end P50 / P95 | not instrumented | 95.3 / 134.1 ms |
+| Cold first request | not instrumented end-to-end | 1400.9 ms |
+
+Raw exact stays at 80% because two files differ only in acoustic homophone/punctuation output;
+`raw_text` is intentionally not rewritten. The semantic and business metrics operate on separate
+fields.
+
+## Full precision vs int8 probe
+
+On sherpa-onnx's official Chinese test WAV, both files produced the same transcript in this one
+probe:
+
+| Model | First call including load | Warm P50 | Warm P95 |
+|---|---:|---:|---:|
+| `model.onnx` | 1514.9 ms | 90.9 ms | 92.7 ms |
+| `model.int8.onnx` | 889.6 ms | 77.4 ms | 81.8 ms |
+
+This is insufficient acoustic coverage to prove equal accuracy, so the accuracy-first default
+remains full precision. The model file stays configurable through `SPEECH_MODEL_FILE`.
+
+## VAD tuning probe
+
+Thresholds 0.35 / 0.45 / 0.55 and end silence 0.45 / 0.65 / 0.85 s were replayed against the
+available Chinese test WAV. All combinations found one segment; threshold 0.45 preserved the
+complete utterance boundary. The selected 0.65 s server endpoint, 0.25 s pre-roll and 0.35 s
+post-roll are conservative for short commands. Real cabin noise and long-pause recordings remain
+required before claiming production calibration.
+
+Reproduce current metrics with:
+
+```bash
+.venv/Scripts/python tools/benchmark_speech_pipeline.py \
+  --audio-dir artifacts/tmp/speech-regression-audio
+```