StreamSense-Serve 4.0 Router
This repository contains the trained, auditable routing artifact used by
StreamSense-Serve 4.0. It is not a copy of the
Qwen or BGE base-model weights. The artifact stores four lightweight decision branches and their
jointly calibrated thresholds:
- safety risk;
- lightweight-evidence uncertainty;
- cross-modal conflict;
- visual dependency.
The runtime escalates to visual enhancement when any branch crosses its threshold. Risk and
visual dependency use frozen BAAI/bge-small-zh-v1.5 embeddings followed by logistic heads;
uncertainty and conflict use physical signal and cross-modal features. The BGE revision is pinned
inside router_artifact.json and must be downloaded under its upstream terms.
Files
router_artifact.json: coefficients, normalizers, frozen encoder revision, four thresholds,
training-data digest, and grouped-OOF metrics;router_test_metrics.json: results on 576 physical media cases from eight unseen semantic
groups, including baselines, condition metrics, robustness slices, and group bootstrap;e2e_summary.json: formal 32-video × three-system result, including answer, citation, visual-call,
and total-backend-call rates;e2e_diagnostics.json: group-bootstrap confidence intervals, paired system differences, and
failure-mode counts;SHA256SUMS: integrity hashes for the published artifacts;MODEL_CARD.md: this card in a standalone filename for local archives.
Measured result
| Metric | Value |
|---|
| Development semantic groups | 24 |
| Grouped OOF folds | 4 |
| OOF escalation recall | 95.88% |
| Held-out semantic groups | 8 |
| Held-out physical cases | 576 |
| Held-out escalation recall | 94.81% |
| Held-out risk recall | 98.61% |
| Visual-enhancement rate | 94.97% |
The held-out split is deliberately difficult: 520/576 cases are oracle-positive, so even a
perfect router could avoid visual enhancement on at most 9.72% of this split. The measured 5.03%
reduction is not a production-traffic cost estimate.
The separate formal end-to-end run used 32 real MP4 cases sampled evenly from eight untouched
semantic groups and executed learned, always-visual, and never-visual systems once each (96 system
records). Quality pass rates were 65.63%, 68.75%, and 78.13%, respectively. The preregistered H5
hypothesis therefore failed: in this test, OCR already resolved the two visual groups and the
free-text VLM evidence introduced state inconsistencies instead of net quality. This negative
result is retained because it defines the next work item: non-OCR visual tasks and structured VLM
evidence. Latency values in this run are not comparable across systems because ASR/OCR outputs
were memoized in execution order.
Reproduce
1python -m pip install -e ".[semantic-router]"
2python scripts/evaluate_v4_router.py \
3 --features /path/to/features.jsonl \
4 --router router_artifact.json \
5 --output reproduced_metrics.json \
6 --bootstrap-iterations 1000
The complete media builder, Faster-Whisper/RapidOCR extraction, group-isolated OOF training, and
end-to-end Qwen-VL/EvidenceAgent evaluation are in the GitHub repository. See the
implementation status,
release notes and failure diary,
and
learning guide.
Intended use and limitations
This artifact is an engineering benchmark router for evidence-grounded audio/video assistants.
It was trained on 32 generated Chinese semantic scenarios expanded into 2,304 physical corruption
conditions. It is not validated for medical decisions, surveillance, identity recognition, or
autonomous safety actions. The corpus uses one synthetic voice family and controlled visual
templates; accents, spontaneous speech, natural meetings, and deployment priors require separate
evaluation.
The endpoint experiment uses one text EvidenceAgent call per case and adds one image-bearing VLM
call only when the router escalates. Therefore visual-enhancement rate does not mean that all
non-escalated cases avoid every large-model call.
