Merge/funnel panel by MajestyHenius · Pull Request #6 · OpenSQZ/OpenGlass

MajestyHenius · 2026-09-02T08:51:17Z

Change Type

Bug fix (link issue number #xxx if any)
New feature
Doc update

Description

Describe the purpose of this PR.

Checklist

Code self-tested
Docs updated
Tests added

- panel: four cumulative ESP32 chains (basic chat / voice control / frame selection / full anti-hallucination) plus a criteria selector; machine paths move to runtime.local.json; TLS certs are self-signed on first launch - esp32_runtime: --rotate, /v1/realtime protocol adapter, reject prompts no longer block the capture loop, every captured frame reaches the first-person view - cam_pipeline_v2: report orientation-classifier load failure instead of silently falling back to the early heuristic - docs: dependency list and setup steps in both READMEs

MengzhangLI · 2026-09-03T09:48:50Z

+            frame_due = time.monotonic() - last_frame_sent >= self.config.image_resend_s
+            if (
+                frame.jpeg
+                and frame_age_ms <= self.config.image_max_age_s * 1000


[P1] A rejected batch leaves the previous accepted frame active. This loop can then resend that stale image every 0.5 s for up to 30 s, so the model may keep seeing a frame the funnel no longer accepts. Please invalidate or consume the accepted frame when a new batch is rejected.

According to the MiniCPM-o paper, what the model emits in the current chunk corresponds to frames from ~1.5 s earlier. Dropping the accepted frame on reject would strip the visual context out from under an utterance that is still mid-sentence and was generated from frames that did pass the funnel. "Current frame rejected" does not mean "current output untrustworthy" — there is a latency in between.

Rather than invalidating on reject, the fix belongs in the timing model: the staleness bound should be derived from the model latency and the reject cadence together. But you're right, those values need to be reconsidered carefully.

MengzhangLI · 2026-09-03T09:50:36Z

+                    except Exception as e:
+                        LOG.warning("[强制措施] 播 wav 失败: %s", e)
+                    try:
+                        await harness.send({"type": "funnel.resume", "reason": "hint_done"})


[P1] This unconditional resume can override a user STOP received while the safety hint is playing. Please resume only if this task still owns the pause (for example, by checking a control generation/token), otherwise leave the user's stopped state unchanged.

Fixed. The hint now records gate.stop_count when it sends funnel.stop and only resumes if that value is unchanged; a user STOP arriving mid-hint bumps the counter and the resume is skipped. The broader timing design — how long prompts may hold the audio channel, whether the funnel keeps judging after a user STOP, and how the staleness bound relates to the model's ~1.5 s latency — still needs rework.

MengzhangLI · 2026-09-03T09:51:31Z

+        while True:
+            try:
+                await self.manager.start_initial()
+                return


[P1] This loop returns as soon as the initial session connects, so it never observes a later disconnect. The upstream video session closes after about 300 seconds, leaving the process healthy-looking but the stream dead. Please monitor the session and reconnect on close/error.

MengzhangLI · 2026-09-03T09:52:23Z

+        },
+        "esp32_select": {
+            "label": "③ 质量筛选",
+            "tail": "demo_funnel",


[P1] Chains ②/③/④ all use the same demo_funnel process name. After switching chains, this alive check skips startup, so the new funnel arguments are not applied and the UI can disagree with the running mode. Restart when the chain configuration changes, or use distinct process identities.

Fixed. The panel now records the configuration fingerprint a tail process was started with — (chain, funnel level, criteria, device, prompt) — and restarts it when any of those change.

MengzhangLI · 2026-09-03T09:52:41Z

+# ============================================================
+
+class WebUIServer:
+    def __init__(self, port: int = 8080, host: str = "0.0.0.0",


[P1] Binding to 0.0.0.0 exposes the live view, session metadata, video, and raw user/AI WAV routes to the local network without authentication. Please default to 127.0.0.1 and require explicit, authenticated opt-in for LAN access.

Changed the default to loopback. The server now binds ["127.0.0.1", "::1"] when a loopback host is requested, falling back to a single address if IPv6 is unavailable. LAN access still requires explicitly passing a non-loopback --web-ui-host, which now logs what that exposes.

MengzhangLI · 2026-09-03T09:53:08Z

 cd OpenGlass
 python -m pip install -r runtime/openglass_omni/requirements.txt
+
+python -m pip install -r extensions/requirements-phase-b.txt   # voice control + CV funnel (harness chains only; skip if you just run basic chat)


[P1] The Quick Start cannot be followed from a clean clone: this file is actually at extensions/assistive_harness/phase_b/requirements-phase-b.txt. Also, the MiniCPM-o-Demo command above uses master, but that repository uses main, so the clone fails. Please fix both READMEs and the panel dependency hint.

Fixed the path — it's extensions/assistive_harness/phase_b/requirements-phase-b.txt, corrected in both READMEs and in the panel's dependency hint.

On the branch: MiniCPM-o-Demo does default to main and is fixed. Note the adjacent llama.cpp-omni line still needs master, that one was left alone.

MengzhangLI · 2026-09-03T09:53:23Z

+  obstacle_avoidance:
+    # Experimental: enabled only for stationary, supervised browser validation.
+    # It is not a mobility-safety guarantee and must not be used for navigation.
+    enabled: true


[P1] This experimental safety feature is documented as stationary and supervised only, but it is enabled by default and can be triggered by phrases such as “前面安全吗”. Please default it to off and avoid directional navigation advice unless a separate safety review approves it.

MengzhangLI · 2026-09-03T09:53:39Z

+        if self._is_explicit_command(compact, self.stop_phrases):
+            return RouteResult(ControlIntent.STOP_SPEECH, reason="explicit stop command")
+        # STOP wins if a transcript contains both anchors.
+        if self._contains_command(compact, self.stop_embedded_phrases):


[P2] Substring matching makes negated or quoted phrases actionable. For example, “请不要停一下”, “我没有说停一下”, or asking how to translate “重新开始” can trigger STOP/RESET. Please require a bounded command or add negation/quotation handling.

MengzhangLI · 2026-09-03T09:54:01Z

+        self._web_ui = None
+        if web_ui_port and WebUIServer is not None:
+            try:
+                self._web_ui = WebUIServer(


[P2] WebUIServer is created without a stop_callback. As a result, POST /api/stop can report success and disable the button without setting the runtime stop event or closing/saving the session. Please wire this to the runtime stop path.

Wired. WebUIServer already supported stop_callback; it just wasn't passed. It now receives self._stop_evt.set, so POST /api/stop sets the runtime stop event and the session finalizes properly.

MengzhangLI · 2026-09-03T09:54:26Z

+        )
+        self.manager.harness = self.telemetry
+
+    async def test_stop_and_resume_flush_pc_output_and_ack(self) -> None:


[Test coverage] Local checks: compileall passed; 30/31 unit tests passed (1 skipped because YOLO weights were unavailable). The suite does not cover chain switching, realtime reconnect after session close, or STOP racing with a safety announcement. Please add regression tests for these cases before merging.

- README: correct requirements-phase-b.txt path (both languages) and the MiniCPM-o-Demo clone branch (master -> main); llama.cpp-omni stays on master - panel: restart the tail process when the chain configuration changes. Chains 2/3/4 share the demo_funnel process name, so neither the cross-chain kill nor the alive check would restart it and the previous funnel arguments stayed in effect. Now keyed on a (chain, funnel level, criteria, device, prompt) fingerprint, which also covers device/criteria changes within a chain. - bridge_ui: default to loopback instead of 0.0.0.0. Binds both 127.0.0.1 and ::1 so the panel's embedded http://localhost:8080 view keeps working on hosts that resolve localhost to IPv6. Non-loopback binds now log what they expose. - esp32_runtime: pass stop_callback to WebUIServer so POST /api/stop actually stops the runtime and finalizes the session; add --web-ui-host - esp32_runtime: the reject hint records gate.stop_count before pausing and only resumes if it still owns the pause, so a user STOP during the hint is no longer overridden

MajestyHenius added 2 commits September 2, 2026 10:25

feat: harness + CV funnel (from feature/esp32-ocr)

075360f

MengzhangLI reviewed Sep 3, 2026

View reviewed changes

Uh oh!

Conversation

MajestyHenius commented Sep 2, 2026

Change Type

Description

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants