Guide Dog for DSH, powered by MiniMax English | 简体中文 A dynamic Cordis plugin that gives DeepSeek Harness multimodal superpowers through the mmx CLI (MiniMax): - Eyes for DeepSeek — MiniMax VLM (guide_dog_vision / guide_dog_inspect) describes images, so a model with no native vision input (e.g. DeepSeek) can still review frontend designs, figures, screenshots, and generated images. - Hands for generation — images (image-01), video (MiniMax-H3 / Hailuo), speech (MiniMax TTS), music (music-3.0), text (MiniMax-M3), and web search. - Web UI preview & playback — every generated file is served same-origin at /guide-dog/media/<file> and rendered inline in the conversation tool cards (<img>, <audio controls>, <video controls>), plus a Guide Dog settings page with auth status and a speak tester. - Skill integration — guide_dog_speak reuses your existing audio-conversation and speech-mmx skill pipelines (text transform, CJK auto-detect, per-language voices, host playback), and falls back to raw mmx speech synthesize when the skill scripts are absent. - Automatic invocation — a mounted system-prompt section (guide-dog-vision, order 110) tells the agent to auto-invoke the inspection tools for any job needing visual checks, especially when the active model cannot see images. - Call mode (Phase 2, shipped) — hands-free, real-time voice conversation in the web UI: VAD / push-to-talk turn-taking, streaming sentence-level TTS with barge-in, consensus-first protection for write commands, progress announcements, and a unified floating dual-pill UI at the composer with zh/en i18n (details in "Phase 2 — call mode" below). - Accessibility mode (Phase 3, planned) — an a11y config block is reserved (auto-narration, vision-cloud, summary-first); accessibility features are next on the roadmap and will be tested and rolled out after the call-mode shakeout. ## Featured Guide Dog is featured in dsh-recommend, a community-curated plugin directory for DSH. It passed the project's certification review and carries the gold certified badge at the top of this README; the score badge updates automatically on every registry sync. ## Files | File | Purpose | |---|---| | plugin-host.js | Host half — source of record (tools, RPC, media route, prompt section, voice mode) | | plugin-client.js | Client half — source of record (tool cards, settings page, voice cluster) | | bundle/ | Static web-profile bundle generated from the two halves (deploy/convert_bundle.py) | | deploy/ | convert_bundle.py (source → bundle) and publish.py (bundle → ~/.dsh/dsh-guide-dog + web profile registration) | | package.json | Root manifest declaring the dsh.bundle (makes the repo installable via dsh plugin add) | | README.md | This file | | README.zh-CN.md | Simplified-Chinese version of this README | ## Install Install straight from this repository with the standard DSH plugin command — the root package.json declares the dsh.bundle manifest, so the profile's pnpm layer resolves the package itself (no npm publish required): sh dsh plugin --profile web add github:AtropinolTT/dsh-guide-dog ## Deploy (static web-profile bundle — current) 1. Edit the source of record: plugin-host.js / plugin-client.js. 2. python3 deploy/convert_bundle.py — regenerate bundle/lib/. 3. python3 deploy/publish.py — copy to ~/.dsh/dsh-guide-dog, idempotently register in ~/.dsh/profiles/web (dependency link + bundles entry + node_modules symlink), remove the superseded autoload bundle. 4. Restart DSH (dsh web) — bundles are parsed at startup. No dynamic plugin, no approval cards, no per-session instances: after a DSH restart the tools and voice UI come back with the profile itself. Full details and pitfalls in the "Restart recovery" section below. plugin-source.js is a legacy dynamic-era artifact (both halves concatenated); kept for reference, not used by the current deploy flow. ## Tools | Tool | Args | Returns | |---|---|---| | guide_dog_speak | text, voice (auto), speed, language, playOnHost | {ok, url, voice, bytes} mp3 | | guide_dog_image | prompt, aspectRatio, n, width, height, seed, promptOptimizer, watermark | {ok, urls[], files[]} | | guide_dog_video | prompt, model (MiniMax-H3 default), image, subjectImage, duration, ratio | {ok, url, taskId} mp4 (polls until done) | | guide_dog_vision | image, prompt | {ok, answer} VLM description | | guide_dog_inspect | image, focus (general/frontend/figure/screenshot/ocr), prompt | {ok, answer, focus} structured review | | guide_dog_voices | language | {ok, voices[]} | | guide_dog_music | prompt, lyrics, instrumental, vocals, genre, mood, model | {ok, url} mp3 | | guide_dog_text | message, system, model, maxTokens, temperature | {ok, text} | | guide_dog_search | q | {ok, results[]} (max 10) | * required ## Auto-invoke contract (visual checks) While the plugin runs, a system-prompt section instructs the agent: - For visual checks (frontend design review, figure/plot/chart generation, screenshots, UI mockups, generated-image QA) it MUST call guide_dog_inspect (structured) or guide_dog_vision (general) on the produced image file before finalizing — never claim to have seen an image it has not inspected. - Generated media is served to the user at /guide-dog/media/<file>; the agent must include the returned url fields so the user can preview. - Speech requests route to guide_dog_speak. Example visual-check flow on DeepSeek: 1. (agent) create figure/screenshot file, e.g. chart.png 2. (agent) guide_dog_inspect { image: "chart.png", focus: "figure" } → structured review of axes/labels/readability/encoding 3. (agent) iterate the figure, re-inspect, then finalize with the url 4. (user) previews chart.png in the web UI card ## Media store & serving - Media lives in ~/.dsh/guide-dog/.guide-dog/media — the global store under GLOBAL_ROOT = ~/.dsh/guide-dog (one instance for the whole web profile since 2026-08-16; no longer the per-workspace sandbox root — see "Restart recovery" below). - Served by a same-origin prefix route /guide-dog/media with: - extension allowlist (jpg/jpeg/png/gif/webp/mp3/wav/m4a/ogg/mp4/webm), - basename-only lookup + traversal guard, - Accept-Ranges: bytes with real byte-range responses (video seeking), - 404/405/413/416 as appropriate. - .index.json keeps metadata (prompt, voice, ts, kind) for the settings gallery (guide-dog/list-media RPC). A corrupt index is rebuilt from the directory. - Files persist across plugin restarts; stopping/removing the plugin only removes the runtime registrations, never the files. ## Skill integration (audio-conversation / speech-mmx) guide_dog_speak honors the exact pipeline of your two skills: 1. ~/.agents/skills/audio-conversation/scripts/transform.py — markdown/code/URL stripping (falls back to a built-in JS transform when absent). 2. CJK auto-detect → per-language voice defaults (English_Trustworthy_Man / Chinese (Mandarin)_Gentle_Youth), same as the skill env contract. Explicit voice overrides; language boosts accents. 3. ~/.agents/skills/speech-mmx/scripts/mmx_tts.py speak --input … --out … (falls back to mmx speech synthesize). 4. Browser playback via the returned mp3 URL. With playOnHost: true the host speakers play it too — one file at a time (previous playback is terminated first), mirroring the skill's latest-only rule. Env vars of the skills that still apply when set in the dsh process environment: AUDIO_CONVERSATION_VOICE(_EN/_ZH), AUDIO_CONVERSATION_SPEED, AUDIO_CONVERSATION_DIR, AUDIO_CONVERSATION_NO_PLAY, AUDIO_CONVERSATION_KEEP_FILES, TTS_GEN. Turn files keep the turn-NNN.mp3 naming convention. ## Settings page Settings → Guide Dog (id guide-dog): - Auth — mmx auth status result with the key masked (sk-c…xxxx); never logged in full. - Voice mode — global default on/off radios (per-session override lives on the small speaker button at the input's bottom-left). - Voice input — STT engine select (whisper / sherpa / minimax), recognition language (auto/zh/en), input device select (defaults to the system default), and auto-send-after-recognition checkbox. - STT — faster-whisper availability + version/python, and the whisper model select (base/small). - Speak tester — text + voice selector (from guide-dog/voices), plays the mp3 in the browser. ## Phase 1 — voice mode & voice input ### Feature list - Voice mode (host event-driven) — a host session/event listener watches assistant/message events, extracts the reply text (event.data.content blocks with type === 'text'), checks whether voice mode is effective for that session (session override else global default), and enqueues the TTS result ({url, key}) or error into a per-session voiceQueue. The client polls the queue every second and plays it with a module-level Audio object, or shows a bottom-right toast + beep for 6s. - Voice cluster — conversation.input.left entry guide-dog-voice (order 30) at the input box's bottom-left, themed with DSH tokens (--dsw-alias-*), inheriting the app font: - small speaker icon — click toggles the per-session voice-mode override (guide-dog/set-config with voiceMode.sessions); hover tooltip shows "Voice mode: on/off · Global default: on/off". - language dropdown — recognition language detection (auto/zh/en). - mic icon — record → transcribe → insert (feather-style SVG; recording state pulses red with a second counter). - Session-scoped playback — playback runs on a module-level Audio object, so switching sessions never replays or interrupts it: the current clip plays to the end unless a new playback task (a fresh queue entry from any session) overrides it. - Mic voice input — the mic in the cluster: MediaRecorder with 1s timeslices, live second counter, maxSeconds auto-stop, language from the dropdown, and transcribe via guide-dog/transcribe. Recognized text is inserted into the input box with inputActions.setDraft(text) (auto-send via inputActions.submit() when configured). Error states: mic_denied, no_device, empty_speech, stt_failed, stt_timeout, engine_unavailable, insert_failed (never silent). - Recorder page — sandboxed clients that cannot record in-page get a 🎙 Open recorder page link to the standalone /guide-dog/recorder page (GET serves a self-contained HTML recorder; POST /guide-dog/transcribe-upload accepts raw audio/webm, 20 MB cap, and runs the same transcribeImpl). - Settings controls — the Phase 1 config blocks above, backed by guide-dog/get-config / guide-dog/set-config / guide-dog/status. ### config.json schema Lives at ~/.dsh/guide-dog/.guide-dog/config.json (auto-created from defaults; all keys optional, deep-merged over the defaults): json { "voiceMode": { "default": false, "sessions": { "<sessionId>": true } }, "voiceInput": { "autoSend": false, "engine": "whisper", "language": "auto", "maxSeconds": 60, "whisper": { "python": "python3", "model": "small" } }, "tts": { "voiceEn": "English_expressive_narrator", "voiceZh": "Chinese (Mandarin)_Gentle_Youth", "speed": 0.95, "format": "mp3" } } - voiceMode.sessions maps a session id to a boolean override; default is the fallback. The speaker button at the input's bottom-left toggles the current session's override. - voiceInput.engine: whisper (only engine implemented; sherpa/minimax are reserved — selecting them returns engine_unavailable). - voiceInput.maxSeconds forces the mic recording to stop. ### STT engine (faster-whisper) The whisper engine shells out to a bundled Python script (.guide-dog/scripts/whisper_transcribe.py) using faster-whisper: pip install faster-whisper # needs Python 3.8+; installs torch cpu wheels python3 -c "import faster_whisper; print(faster_whisper.__version__)" The host probes availability at startup and writes the result to .guide-dog/status.json (whisperAvailable, whisperVersion, whisperPython), shown in the Settings → STT row. Model choices: base (fast) / small (accurate); first run downloads the model weights. ### Verification node --check plugin-host.js && node --check plugin-client.js # syntax curl -s http://127.0.0.1:3080/guide-dog/recorder | head -5 # recorder page serves HTML curl -s -X POST http://127.0.0.1:3080/guide-dog/api/guide-dog/status \ -H 'content-type: application/json' -d '{}' | head -5 # status RPC (compat layer) cat ~/.dsh/guide-dog/.guide-dog/status.json # whisper probe result Manual checks (after deploy): click the speaker button (voice mode on, turns green) → send a message → the assistant reply is spoken automatically; switch sessions mid-playback → the clip continues to the end and is NOT replayed; use the mic button → recognized text appears in the input box; Settings → Guide Dog shows the Voice mode / Voice input / STT blocks. ## Phase 2 — call mode ### Feature list - WebSocket-free dual channel — uplink is a whole-clip POST /guide-dog/call-transcribe (webm/opus, ≤20MB, reuses the Phase 1 transcribeImpl and the local whisper pipeline) → {ok, text, language, durationMs}; downlink is GET /guide-dog/tts-stream over a chunked HTTP stream (the host spawns mmx speech synthesize --stream --format pcm --sample-rate 24000 per sentence and pipes stdout incrementally into res.write; the client reads the stream with fetch().body.getReader() → PCM→WAV → seamless Web Audio scheduling). No new WebSocket protocol surface on the transport layer; browser and CLI reuse the same pipeline. - Automatic VAD + push-to-talk (PTT) — default VAD (call.mode='vad'): MediaRecorder (audio/webm;codecs=opus, 250ms timeslice, continuous recording) + a parallel AnalyserNode energy detector (RMS ≥ vad.threshold; silence for vad.silenceMs ends an utterance; vad.minSpeechMs minimum speech; vad.maxSegmentSeconds per-segment cap) — speak-pause-speak automatically becomes two turns; the panel can switch to ptt push-to-talk (hold the mic to talk, release to send; VAD parameters do not participate in endpointing, only in interruption monitoring). - Consensus-first (core interaction paradigm) — active only when call/a11y is on; typing mode keeps the Phase 1 behavior: a prompt soft constraint (guide_dog_call_consensus systemPrompt variable, conversational wording: understand intent first, ask when unclear, explain before writing/modifying and wait for the user's go-ahead) plus a mechanical hard guarantee (tools/pre-execute waterfall interception: write/edit and destructive-bash heuristics rm/mv/cp/truncate/dd/overwriting-redirect/git push etc. → unconfirmed returns {kind:'deny', reason: 'needs_voice_confirmation'} and the model asks by voice; user confirmation keywords hit → released for this turn; before every execution the host TTS-broadcasts a one-sentence summary built from the tool args (not through the model), then opens a consensus.summaryWindowMs interruption window; speech inside the window aborts the execution — the tool has physically not started). Interceptor failure → deny and announce "consensus check failed" (better to block wrongly than to allow wrongly, spec §6.8). - Progress announcements (minimalist principle, RC10) — only useful information is announced: agent/status (running → "processing"), tools/result (tool name → phrase: write/edit → "modifying files", web_search → "searching the web", guide_dog_image/video/music/speak → "generating media", bash only for destructive commands (the same DESTRUCTIVE_BASH_RE as consensus) → "running a command"; read/grep/glob/ skill/non-destructive bash/unknown tools stay silent), agent/error ("processing failed: "); same-phrase 4s cooldown dedupe (multi-step same-kind operations announced once); >120s without any event during a call → heartbeat "still processing, please wait". Announcements and reply playback share one queue: announcements first (queue head), replies yield; announcements go through the streaming channel (the same WebAudio PCM chain as replies) → constructive serialization on a single player, one after another — overlap is impossible. - Streaming TTS — reply text is split per sentence (stream.sentenceSplit charset `。！？.!?

; stream.maxSentenceCharsforce-truncates over-long sentences) and synthesized sentence by sentence; each sentence gets a fresh one-time token viaguide-dog/tts-token(single-use, 5-minute validity, bound to sessionId); pre-synthesis between sentences (while the current sentence plays, the client requests the next sentence's stream ahead of time and appends decoded frames seamlessly on the playback-time chain — the next sentence continues before the previous one finishes; long replies are read in full with ≤400ms gaps). Measured ~600ms to first byte for short Chinese sentences, satisfying the "first audio <1.5s" criterion. **Only the turn's final message is played (RC13)**: intermediate assistant messages (with tool-call blocks) are not enqueued — playing near-identical text per step was the root cause of "the same content repeated"; intermediate steps are covered by progress announcements. A terminal tool turn (the last message still has tool calls) is covered by the turn/end fallback that plays the buffered text — never silent. - **Barge-in** — VAD detects user speech during playback (≥ vad.interruptMinMs300ms to avoid false triggers) → the browser immediately stops playback and clears the unplayed buffer → the stop is a 10ms fade-out (RC13) —src.stop()hard cuts click at sentence boundaries. The first transcript segment after the interruption goes straight to the current turn via theinterrupt RPC (agent.steer, RC11) instead of queueing as a new turn → abort the current tts-streamfetch → the new speech naturally becomes the next turn (Pipecat InterruptionFrame semantics). - **Voice commands** — call transcriptions that hit the command table (pause, resume, repeat/say-again, slower/faster, look-at-screen [Phase 3 stub]) execute locally and are not submitted to the conversation (stop/continue are consensus confirmation words, not in the command table — they pass through to the agent unchanged);guide-dog/call-commandRPC provides host-side commands such asclear-queue. - **Dual-channel mutual exclusion (RC13)** — text already spoken on the host speakers via guide_dog_speak(playOnHost=true) is not replayed through the voice-mode/call queue channels (consumed on use, the same text blocked once) — eliminates the "host + browser double sound". - **Fault tolerance** — a stream interruption auto-reconnects once (at most one retry per (sid,text) within 5s, no retry on 429; a fresh token per sentence; failure toast "playback interrupted"); STT failure does not submit + beep + toast; TTS failure still lands the text + failure beep + panel error state (never silent); consensus-interceptor failure denies conservatively and announces the reason. Session ownership for call transcription/interruption/polling is captured once when the call starts (RC13) — multi-session switching no longer cross-talks. ### RC14 fixes (2026-08-17): announcement content selection + queue tail-truncation + progress dedupe + double-play pinpointing - **Announcement sanitization (sanitizeSpeechText, F1)** — before enqueueing, reply text goes through a markdown/URL/emoji strip: title keeps the title, drops the URL; bare URLs (https?://, www.) are removed entirely; leading list/quote markers (-/+//>) and leading ordered-list markers (1. 1、 1)) are stripped; markdown markers such as bold and backticks are stripped; emoji ranges (U+1F000-U+1FAFF etc.) are stripped. Call mode reads only human language — no URLs/**/-/📢 metacharacters, so URL fragments like "thepaper/newsD/weather.com" are no longer read out in pieces. - **Smart sentence splitting (splitSentences, F2)** — in the default separators '。！？.!?
', .follows a smart rule: split only when. is followed by whitespace + an uppercase letter/digit/CJK ('Hello. Next'→ 2 sentences;'8.17 的上海'stays 1; dots inside URLs are never split). The Chinese separator set gains；;so…； no longer cuts one Chinese reply into two halves. - **Queue cap 40 with tail truncation (VOICE_QUEUE_MAX, F3)** — raised 10→40, **drop from the tail to keep content**: on overflow while (q.length > VOICE_QUEUE_MAX) q.pop()(first-in content wins; the old splice(0, …)head-removal strategy cut the main content first while keeping URL fragments). announce/hb progress still usepop()(unshifted to the queue head, progress first). - **30s progress-phrase dedupe window (F4)** —announce's progressDedupe cooldown extended 4s→30s: web_search results ~4.3s apart no longer announce "searching the web" three times. TheprogressDedupefunction body is untouched;repro-progress.jssemantics preserved. - **Dual-channel mutual exclusion by sanitized text (F5)** — wasHostSpoken/markHostSpokenuniformly use sanitizeSpeechText-processed text as the key: all three wasHostSpoken call sites (downlink, turn-end flush, voice-mode) match on the sanitized key;speakImplregisters both the raw and the sanitized key after a successfulplayOnHost (double key) so any downstream channel dedupes correctly. Known edge: when transform.py rewrites the text the two keys may differ slightly (acceptable edge). - **Diagnostic instrumentation (F6, one-shot retest to pinpoint "reading twice")** — zero behavior change, logs only: - host ([gd-host], visible in the DSH terminal): enqueue from=downlink|turnend|voice-mode|consensus|announce|heartbeat n=... qlen=..., shift key=... remain=..., skip host-spoken sid=... text=..., QUEUE-DUP text=.... - client ([gd], browser DevTools): playStreamEntry ... times=... (accumulates perentry.key || entry.texton each play), PLAY-SUMMARY key=N | ...(when the queue empties, summarizes all current counts then clears). - Retest calibration (basis for the RC15 direction):QUEUE-DUP→ host double-enqueue;PLAY-SUMMARY key=2→ client double-play; neither but still twice → tts-stream double audio write;enqueue from= same source twice with the same text → event replay. ### RC15 fixes (2026-08-17): persistent player + gesture unlock + failed-entry requeue + event-replay dedupe - **Persistent voice player (playVoiceEntry`, F1)* — voice-mode playback changed from "new Audio() + temporar