Skip to content

feat(asr): support additional Bailian speech models - #866

Merged
H-Chris233 merged 6 commits into
Open-Less:betafrom
LauraGPT:codex/funasr-bailian-model-routing
Aug 2, 2026
Merged

feat(asr): support additional Bailian speech models#866
H-Chris233 merged 6 commits into
Open-Less:betafrom
LauraGPT:codex/funasr-bailian-model-routing

Conversation

@LauraGPT

@LauraGPT LauraGPT commented Jul 26, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

User description

Closes #850

Summary

  • route Bailian model families to classic realtime, Qwen realtime, synchronous multimodal, or asynchronous transcription clients
  • support Fun-ASR, Fun-ASR-MTL, Paraformer file transcription, Qwen3-ASR Flash/filetrans, and 8 kHz Fun-ASR realtime models
  • align desktop and mobile routing, provider validation, model hints, localized UI, and documentation
  • implement temporary OSS upload, async task polling, model-specific response parsing, cancellation-aware QA flows, and dynamic long-recording timeouts
  • harden credential and anonymous requests against redirects, untrusted transfer URLs, SSRF, and stalled requests

Verification

  • cargo check --tests
  • cargo test --lib: 773 passed (serial full-suite run)
  • npm run build: 2445 modules transformed
  • npm test: 26 frontend tests passed
  • git diff --check

Notes

DashScope temporary OSS upload is documented as a development and low-concurrency path. The added documentation calls out that limitation for users.

PR Type

Enhancement, Tests, Documentation

Description

  • Relax Bailian ASR whitelist via model-family routing

  • Add async transcription with temporary OSS upload and polling

  • Support 8k realtime models through 16k-to-8k downsampling

  • Align mobile routing, validation, docs, and localized hints

Diagram Walkthrough

flowchart LR
  A["User selects Bailian model"] --> B{"Model prefix?"}
  B -- "*-realtime / 8k realtime" --> C["Classic WebSocket"]
  B -- "qwen3-asr-flash-realtime*" --> D["Qwen Realtime WebSocket"]
  B -- "fun-asr-flash* / qwen3-asr-flash" --> E["Synchronous Multimodal"]
  B -- "fun-asr / fun-asr-mtl / paraformer" --> F["Async Transcription + Poll"]
Loading

File Walkthrough
Relevant files
Enhancement
6 files
bailian.rs
Add 8 kHz model support with PCM downsampling                       
+137/-7 
dashscope_multimodal.rs
Implement async transcription and model-family routing     
+784/-14
providers.rs
Extend model picker and route-aware validation                     
+100/-36
coordinator.rs
Relax whitelist and align desktop/mobile routing                 
+66/-29 
dictation.rs
Use model-aware dynamic DashScope timeouts                             
+1/-1     
ProvidersSection.tsx
Show protocol hints and extend mobile Bailian UI                 
+18/-8   
Error handling
1 files
qa_session.rs
Make DashScope transcription cancellation-aware                   
+34/-21 
Localization
5 files
en.ts
Add sync/async ASR hints in English                                           
+2/-1     
ja.ts
Add sync/async ASR hints in Japanese                                         
+2/-1     
ko.ts
Add sync/async ASR hints in Korean                                             
+2/-1     
zh-CN.ts
Add sync/async ASR hints in Simplified Chinese                     
+2/-1     
zh-TW.ts
Add sync/async ASR hints in Traditional Chinese                   
+2/-1     
Documentation
1 files
bailian-asr-models.md
Document Bailian model families and endpoints                       
+32/-0   

@github-actions

github-actions Bot commented Jul 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

(Review updated until commit ca04810)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

850 - Partially compliant

Compliant requirements:

  • 支持 fun-asr / fun-asr-mtl / paraformer 异步文件转写（新建异步客户端 + OSS 上传 + 轮询 + 下载）。
  • 支持 qwen3-asr-flash 同步 Multimodal 识别。
  • 支持 fun-asr-flash-8k-realtime / paraformer-8k-realtime / sensevoice-8k-realtime，通过降采样和 sample_rate=8000 走经典 WebSocket。
  • fun-asr-flash-* 改为前缀匹配。
  • 模型下拉列表扩充，并加入区分三类协议的文案提示。
  • 移动端与桌面端路由对齐。
  • 增加了覆盖路由、请求体、超时、重定向/SSRF 的单元测试。

Non-compliant requirements:

  • qwen3-asr-flash-filetrans 仍被 protocol_for_model 显式拒绝（返回 None），未满足“用户手填 qwen3-asr-flash-filetrans 时不再被拒绝”的验收标准；PR 说明其与本地录音的临时 OSS 上传链路不兼容，需要公网 URL，因此没有实现。

Requires further human verification:

  • 需要人工确认 DashScope 临时 OSS 上传 + 异步任务轮询在真实 API 上的可用性（文档注明为开发/低并发路径）。
  • 需要确认 fun-asr-2025-11-07、fun-asr-2025-08-25 等新增下拉模型 ID 是否真实存在于百炼官方模型列表。
  • qwen3-asr-flash-filetrans 不支持属于有意取舍，需要产品确认是否接受该临时限制。
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Misleading Model Hint

bailianModelProtocol classifies qwen3-asr-flash-filetrans* as 'sync' because it doesn't match the realtime or async branches, but the backend protocol_for_model explicitly rejects these models. The UI will show the "synchronous recording" hint for a model that cannot be used, and saving it fails with a backend rejection. Add an explicit unsupported branch (or return a value the hint can render as unsupported) to keep the hint consistent with the backend route.

 
function bailianModelProtocol(model: string): 'realtime' | 'sync' | 'async' {
  const m = model.trim();
  if (!m || m.includes('realtime')) return 'realtime';
  // qwen3-asr-flash-filetrans 仅接受公网 URL，暂不支持（后端 protocol_for_model
  // 显式拒绝），前端不再归为 async 提示。
  if (m === 'fun-asr'
    || m.startsWith('fun-asr-') && !m.startsWith('fun-asr-flash')
    || m.startsWith('paraformer')) return 'async';
  // 其余（fun-asr-flash-*、qwen3-asr-flash、qwen-audio-3.0-asr-flash）为同步录音模型。
  return 'sync';
}

@LauraGPT

LauraGPT commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up on the PR-Agent 8 kHz tail note: the decimator intentionally keeps source indices 0, 2, 4, ... . Therefore, when a stream ends after three complete 16 kHz samples, the buffered third sample is the next kept 8 kHz sample rather than half of an output pair.

Signed commit a58630b extracts that tail flush into a named helper and adds a regression proving [1, 2, 3] -> [1, 3] at end of stream. Fresh verification passed cargo check --tests and all 773 Rust tests serially. (Two unrelated localhost connection-pool tests raced in one parallel run; both passed in isolation, and the full serial run is 773/773.)

@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit a58630b

@H-Chris233
fix(asr): address review findings on Bailian model routing (Open-Less…
853442c 
…#866)

- route paraformer-8k-realtime / sensevoice-8k-realtime to classic WebSocket,
  matching the 8k downsample naming space in bailian.rs
- downsample 16k->8k with pairwise averaging (first-order low-pass) instead of
  pure decimation to reduce aliasing; drop the lone tail sample on flush
- cap async provider validation at 120s (60s poll) instead of 660s so the
  validate button cannot block for nearly 11 minutes
- join async transcript segments with a space so latin-script words do not
  get glued together
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 853442c

@H-Chris233
fix(asr): drop qwen3-asr-flash-filetrans and harden async polling (Op…
bc9830d 
…en-Less#866)

- qwen3-asr-flash-filetrans only accepts public URLs per DashScope docs; it
  is incompatible with the local-recording temporary OSS + oss:// path, so
  reject it in protocol_for_model / routing / picker / docs instead of
  shipping a model that would silently fail on real recordings
- simplify async_transcription_body / extract_async_result_url to the
  single fun-asr results shape
- add bounded retry (exponential backoff, deadline-capped) to async task
  polling and result download so one transient network blip does not
  discard an entire recording
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit bc9830d

@H-Chris233
Merge branch 'beta' into codex/funasr-bailian-model-routing
2b0c0e8 
Resolve conflicts with Open-Less#876 (qwen-audio-3.0-asr-flash batch support):
- route qwen-audio-3.0-asr-flash through protocol_for_model as
  DashScopeBatchProtocol::Multimodal (streaming variants stay rejected)
- drop is_supported_model in favor of protocol_for_model everywhere
- keep PR's async transcription / polling / 8k downsample changes intact
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 2b0c0e8

@H-Chris233
docs(asr): list qwen-audio-3.0-asr-flash as a synchronous model (Open…
ca04810 
…-Less#866)

Merge with beta Open-Less#876 brought qwen-audio-3.0-asr-flash into the model
picker and protocol routing; keep the model reference doc in sync and
note that -streaming variants are not supported.
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit ca04810

@H-Chris233 H-Chris233 self-assigned this Aug 2, 2026
@H-Chris233

H-Chris233 commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

不要动这个PR，等CI跑过我就合并

@H-Chris233
H-Chris233 merged commit 8f49971 into Open-Less:beta Aug 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

See More

@H-Chris233 H-Chris233

Labels

Review effort 4/5

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[asr] 放宽百炼 ASR 模型白名单，支持官方文档中的更多模型

2 participants

@LauraGPT @H-Chris233