merge: integrate remote speech capabilities into feature/N7Web
This commit is contained in:
minghanchen
@@ -0,0 +1,67 @@
|
||||
# Speech pipeline benchmark — 2026-08-28
|
||||
|
||||
## Environment
|
||||
|
||||
- Windows / Python 3.13
|
||||
- `sherpa-onnx==1.13.6`, CPU provider, 8 ASR threads
|
||||
- SenseVoiceSmall full precision, `language=zh`, `use_itn=true`
|
||||
- Silero VAD: threshold 0.45, min speech 0.16 s, min silence 0.65 s
|
||||
- 48 text regressions（including reported field failures plus ratio/coordination matrices）
|
||||
- 16 capability-alignment cases replayed 100 times（1600 samples）
|
||||
- 43 catalog capability definitions × 5 resolver projections（215 subcases）
|
||||
- 10 local 16 kHz mono Windows SAPI WAVs covering filler, target refinement,
|
||||
cross-domain correction, normal multi-command, number/direction correction,
|
||||
negation, unrelated speech, fast speech and slow speech
|
||||
|
||||
The WAV set is synthetic and deliberately kept under ignored `artifacts/tmp`; it is useful for
|
||||
repeatability but is not a substitute for TAKSTAR V2 recordings in the real cabin.
|
||||
|
||||
## Before / after
|
||||
|
||||
| Metric | Before | After |
|
||||
|---|---:|---:|
|
||||
| Raw exact transcription (10 WAV) | 8/10 | 8/10 |
|
||||
| Expected understanding text (same 10 cases) | 2/10 | 10/10 |
|
||||
| Intent exact match | no structured speech Intent | 10/10 |
|
||||
| Device / action / parameter exact | no structured speech metrics | 10/10 / 10/10 / 10/10 |
|
||||
| Correction success | unsafe text-only heuristics | 4/4 |
|
||||
| False command activation | 1 observed wrong-direction activation | 0/2 no-command cases |
|
||||
| Severe wrong device / wrong number | not reported | 0 / 0 |
|
||||
| Duplicate transcript rate | not observable (single callback) | 0% |
|
||||
| ASR inference P50 / P95 | 80 / 110 ms | 79.7 / 110.5 ms |
|
||||
| Semantic P50 / P95 | 0.03 ms (old regex only) | 0.33 / 0.56 ms（1600 capability samples） |
|
||||
| Command resolution P50 / P95 | not available | 0.03 / 0.06 ms |
|
||||
| Steady end-to-end P50 / P95 | not instrumented | 95.3 / 134.1 ms |
|
||||
| Cold first request | not instrumented end-to-end | 1400.9 ms |
|
||||
|
||||
Raw exact stays at 80% because two files differ only in acoustic homophone/punctuation output;
|
||||
`raw_text` is intentionally not rewritten. The semantic and business metrics operate on separate
|
||||
fields.
|
||||
|
||||
## Full precision vs int8 probe
|
||||
|
||||
On sherpa-onnx's official Chinese test WAV, both files produced the same transcript in this one
|
||||
probe:
|
||||
|
||||
| Model | First call including load | Warm P50 | Warm P95 |
|
||||
|---|---:|---:|---:|
|
||||
| `model.onnx` | 1514.9 ms | 90.9 ms | 92.7 ms |
|
||||
| `model.int8.onnx` | 889.6 ms | 77.4 ms | 81.8 ms |
|
||||
|
||||
This is insufficient acoustic coverage to prove equal accuracy, so the accuracy-first default
|
||||
remains full precision. The model file stays configurable through `SPEECH_MODEL_FILE`.
|
||||
|
||||
## VAD tuning probe
|
||||
|
||||
Thresholds 0.35 / 0.45 / 0.55 and end silence 0.45 / 0.65 / 0.85 s were replayed against the
|
||||
available Chinese test WAV. All combinations found one segment; threshold 0.45 preserved the
|
||||
complete utterance boundary. The selected 0.65 s server endpoint, 0.25 s pre-roll and 0.35 s
|
||||
post-roll are conservative for short commands. Real cabin noise and long-pause recordings remain
|
||||
required before claiming production calibration.
|
||||
|
||||
Reproduce current metrics with:
|
||||
|
||||
```bash
|
||||
.venv/Scripts/python tools/benchmark_speech_pipeline.py \
|
||||
--audio-dir artifacts/tmp/speech-regression-audio
|
||||
```
|
||||
@@ -0,0 +1,116 @@
|
||||
# 中文语音控制链路
|
||||
|
||||
## 整改前的实际链路
|
||||
|
||||
```text
|
||||
getUserMedia
|
||||
-> ScriptProcessor(4096)
|
||||
-> 浏览器 RMS 阈值与前后裁剪
|
||||
-> 单个 16 kHz PCM WAV
|
||||
-> POST /api/speech/transcribe
|
||||
-> 单个 sherpa-onnx OfflineStream
|
||||
-> SenseVoiceSmall 单次 final decode
|
||||
-> regex 文本清理
|
||||
-> 直接覆盖输入框
|
||||
-> 用户提交后进入既有 IntentIR / capability / CAN 链路
|
||||
```
|
||||
|
||||
旧实现没有 AudioWorklet、ASR Web Worker、WASM、streaming partial、VAD segment、
|
||||
chunk overlap 或 segment assembler。因此观测到的“空调打开打……靠背靠座椅靠背”
|
||||
不是 partial/final 被前端重复 append，而是 SenseVoice 单次 decode 已返回该 raw 文本；
|
||||
旧正则层只是尝试掩盖 acoustic false start，且没有结构化 cancelled intent。
|
||||
|
||||
## 整改后的链路
|
||||
|
||||
```text
|
||||
Microphone (优先 TAKSTAR V2)
|
||||
-> AudioContext（跨 utterance 复用）
|
||||
-> AudioWorklet（旧环境 ScriptProcessor fallback）
|
||||
-> 自适应 RMS endpoint + 350 ms 前置 / 450 ms 后置采集
|
||||
-> 16 kHz PCM WAV + utteranceId
|
||||
-> Silero VAD（0.25 s pre-roll / 0.35 s post-roll，重叠先合并）
|
||||
-> SenseVoiceSmall zh + ITN（每个 VAD segment 一个 OfflineStream）
|
||||
-> final ASR callback
|
||||
-> TranscriptAssembler（replace / commit / order / idempotency / overlap）
|
||||
-> rawText
|
||||
-> UnderstandingTranscriptProcessor
|
||||
-> cleanText + active commands + cancelledFragments + uncertain
|
||||
-> 页面展示与提交安全门
|
||||
-> 用户确认后进入既有 IntentIR / capability / CAN 链路
|
||||
```
|
||||
|
||||
模型、tokens、VAD native session 和浏览器 AudioContext 都只初始化一次。服务端请求以
|
||||
`utteranceId + SHA-256(audio)` 幂等：相同音频返回缓存，不同音频复用同一 ID 返回 409。
|
||||
|
||||
## 数据边界
|
||||
|
||||
- `raw_text`：SenseVoice 原始证据，只去模型标签和多余空白，永不被语义文本覆盖。
|
||||
- `clean_text`：用于输入框显示的理解式文本。
|
||||
- `commands`：与工程 capability catalog 对齐的 active Intent 预览。
|
||||
- `cancelled_fragments`：否定、自我修正、false start 或显式撤销的证据及 cancelled Intent。
|
||||
- `uncertain`：参数越界、缺少方向/单位/设备、拼接冲突等不能安全执行的情况。
|
||||
- `execution_eligible`：存在 active commands、无 uncertain、无校验错误时才为 true。
|
||||
|
||||
语音 API 本身不执行任何能力。页面会拒绝直接提交仍带有原始 uncertain 标记的语音结果；
|
||||
用户编辑后才解除该标记。
|
||||
|
||||
## Capability 驱动的语义链路
|
||||
|
||||
语音层不再维护可执行设备表。能力事实按以下优先级读取：
|
||||
|
||||
1. `domains/*/config/capabilities.yaml` 与
|
||||
`config/native_can_capabilities/*.yaml`；
|
||||
2. `CapabilityRegistry` 统一加载 catalog；
|
||||
3. `NativeCanRuntime.supported_capabilities`、domain runtime、seat compiler 和
|
||||
mode profile registry 确认可执行性与 executor binding；
|
||||
4. `CapabilityResolver` 只把中文槽位或既有一句话解析 action 适配到上述事实。
|
||||
|
||||
`UnderstandingTranscriptProcessor` 会优先复用
|
||||
`application/native_can_intent.py` 已有的确定性解析结果，所以驾驶台默认目标、CH1
|
||||
主驾能力和网页一句话编排保持一致。`vocabulary.py` 只包含 alias、同义词、颜色与
|
||||
scope 词法映射，不声明车辆支持哪些能力。
|
||||
|
||||
每个 command 同时保留 `target_scope`、`property`、`operation`、`value`、`unit`、
|
||||
`qualifiers`、IntentIR action、executor binding 和 executor payload。`clean_text` 从
|
||||
这些结构化 command 渲染；unsupported、ambiguous、incomplete 不会产生假的
|
||||
executor payload，且 UI 会阻止整条原始语音直接提交。
|
||||
|
||||
当前代码证据中，前大灯只有反馈和只读 UI，没有控制 capability；因此“打开前大灯”
|
||||
为 `recognized_unsupported`。零重力是 `config/modes/vehicles/denza_n7/zero_gravity.yaml`
|
||||
定义的副驾四轴 mode profile，不是主驾单一 capability；明确要求主驾时返回
|
||||
`recognized_unsupported`。
|
||||
|
||||
能力矩阵始终从 catalog 动态生成：
|
||||
|
||||
```bash
|
||||
python tools/report_capability_matrix.py --format markdown
|
||||
```
|
||||
|
||||
## Debug trace
|
||||
|
||||
生产默认关闭。设置 `SPEECH_DEBUG=1` 后，每条 JSON 日志包含：
|
||||
|
||||
- `utteranceId`；
|
||||
- `segmentId`、sequence、开始/结束时间、音频 overlap；
|
||||
- raw ASR final、进入拼接器前文本、拼接后文本；
|
||||
- clean text、active/cancelled Intent、uncertain reason；
|
||||
- audio、VAD、每段 ASR、assembly、semantic、recovery、total 耗时。
|
||||
|
||||
`SPEECH_DEBUG_RESPONSE=1` 才把 trace 放进 API 响应。浏览器使用 `?speechDebug=1`
|
||||
查看采集实现、设备、阈值、utteranceId 和服务器结果。
|
||||
|
||||
## 默认 VAD 与识别配置
|
||||
|
||||
| 配置 | 默认值 |
|
||||
|---|---:|
|
||||
| language | `zh` |
|
||||
| use_itn | `true` |
|
||||
| Silero threshold | `0.45` |
|
||||
| min speech | `0.16 s` |
|
||||
| min silence | `0.65 s` |
|
||||
| max speech | `12.0 s` |
|
||||
| pre-roll | `0.25 s` |
|
||||
| post-roll | `0.35 s` |
|
||||
|
||||
参数可通过 `SPEECH_VAD_*` 调整。真实 TAKSTAR V2 车内噪声录音仍需要继续扩展 audio
|
||||
regression suite；没有真实证据时不自动切换到 int8，也不自动下载 Recovery 模型。
|
||||
Reference in New Issue
Block a user