Bilibili

dsh-bilibili 中文 | English A DeepSeek Harness tool plugin that gives agents a bilibili_extract tool. Send a Bilibili link and the agent extracts the video's text information (transcript / comments / danmaku), captures keyframes on demand, and produces a summary. > This plugin bundles no third-party binaries or models; the open-source projects and services it invokes are listed in THIRD_PARTY_NOTICES.md. --- ## ✨ Features - Full text extraction: metadata, complete timestamped transcript (long transcripts are truncated with a full-text time index), hot comments (with replies), danmaku (top repeated messages + density-peak timeline samples — opening spam no longer dominates); videos without a subtitle track are auto-transcribed — Bijian ASR by default (the same anonymous capability behind Bilibili's "live AI subtitles", 24h cache), or switch to local engines — sherpa-onnx (Chinese, SenseVoice) or whisper.cpp — fully offline; one failing source never breaks the rest (each degrades to empty with a note); - Optional frame vision descriptions: vision-less main models can still "see" frames — send each frame to a local Ollama / llama.cpp backend (Qwen3-VL 2B/4B/8B tiers) or any OpenAI-compatible vision API for a text description; the report cites images only when needed; - Automatic frame selection (picture-driven only): scene-change detection (sampled pass beyond 20 minutes) + even-interval backfill, 5s dedupe; no keyword guessing — deciding "which transcript moments are incomplete and need visuals" is semantic analysis, left to the main agent's two-pass prompts; - Sharp-frame preference: within ±1.5s of each target time, FFmpeg blurdetect scores every frame and the sharpest one wins — motion-blurred animation entrances and fade frames are skipped; - Two-pass workflow: the agent reads the transcript first (instant, zero download), then requests frames with explicit timestamps — each frame is captioned with its nearby subtitle; the agent reports needed moments in a fixed [建议抓帧] mm:ss format; 24h video cache reuse across passes; - Replaceable output template: a concise shareable summary template is bundled; summaryTemplate can point to any custom template file; - Download-first capture: the video is downloaded locally before frame extraction (≤30 min / ≤800MB), with automatic fallback to remote per-frame extraction; - Robust: exponential backoff on Bilibili 412 rate limits; login-required subtitles are detected with a SESSDATA hint; ffmpeg runs pipe-free, so it works in any environment. --- ## 🚀 Installation Prerequisites: Node 18+, ffmpeg on PATH, pnpm. sh # Option 1: install from GitHub (recommended) dsh plugin --profile web add git+https://github.com/CZX2244/dsh-bilibili # Option 2: local directory (development; link mode applies changes instantly) dsh plugin --profile web add ./dsh-bilibili # Restart the web profile (dsh web); bilibili_extract becomes available in new sessions After installation the tool joins the agent's toolchain: when the user sends a Bilibili link (bilibili.com/video/BV..., a b23.tv short link, or a bare BV id), the model can call it for analysis. The plugin also registers bilibili_login (QR-code login) and bilibili_doctor (environment check, below). --- ## 🩺 Environment check (first-run probe) Before the first extraction the plugin runs a three-layer environment probe automatically; the report is cached for about an hour (invalidated when relevant config changes). It reports only what is missing, so nothing gets installed twice and no bandwidth is wasted: | Layer | Checks | Behavior on failure | |---|---|---| | Local dependencies | ffmpeg / whisper-cli / sherpa-onnx binaries, model and tokens files, vision endpoint, output dir writability | ffmpeg missing → video download and frame capture are skipped (no more 800 MB download before a raw spawn error); a not-ready local ASR engine is skipped before downloading the audio stream | | Config | asrProvider validity, sherpa/whisper required fields, visionBaseUrl resolution | Invalid/missing items are marked error/warn with fix hints | | Cloud | Bilibili main API, Bijian ASR, login service reachability; SESSDATA validity (verified via nav isLogin, not just file existence) | Unreachable/expired items are marked unreachable/warn; an unreachable Bilibili main API is flagged as "plugin unusable" (local ASR engines get their audio stream from playurl too) | - The report is attached to every extraction result (an Environment check section); the agent relays the fix hints to the user, or a single line when everything passes; - bilibili_doctor re-checks on demand; refresh=true forces a fresh probe (use after installing/updating dependencies or changing config); - Probes only check existence/reachability: no real ASR tasks are submitted and no media is downloaded; a failing check degrades, never blocks extraction; - The report never contains raw credentials (SESSDATA validity only). --- ## 🎨 Custom output templates The output format is a replaceable part of the plugin: the tool provides the data (transcript / frames / danmaku / comments), the template decides what it looks like. - Bundled default: templates/summary.md — a concise "time-saver" summary (one-sentence takeaway → timestamped key points → worth-watching segments → shareable closing lines); - Bundled alternative: templates/timeline.md — a generic time-axis format (title → hook → timestamped sections with inline points → image anchors → conclusion); add images only when they help, never force them; - Switch templates: set summaryTemplate: 'C:/path/my-template.md' in the config; - Change the default: edit templates/summary.md directly in the plugin directory; - An invalid custom path falls back to the bundled template, so the tool never breaks because of a template. For richer output formats (study notes, review tables, timelines, Q&A cards, etc.), install the companion skill bilibili-video-analyzer (an A–K format catalog) and the agent picks by user intent. --- ## 🧠 Recommended workflow (two-pass; prompt-driven is the primary path) Frame selection is not keyword guessing — the main agent uses the analysis prompts injected into the system prompt to read the transcript and decide which moments are incomplete without the picture. The built-in "content-completeness check" teaches the agent five gap types — dangling reference / conclusion without data / unspoken operations / silent demos / visual comparisons — and asks it to report them explicitly: ① bilibili_extract(url, extract_frames: false) # text only: instant, zero download ② the agent scans the transcript for information gaps and picks the moments that need visuals ③ the agent lists them as [建议抓帧] mm:ss reason, then calls again with timestamps — the plugin aligns each moment to the most changed frame within ±4s (FFmpeg scene detection), then picks the sharpest frame within ±1.5s (blurdetect): semantic targeting → picture refinement → sharpness gate ④ the agent decides which images to cite in the report using description/citation_hint (or read_image) > Fallback: only a single call (no timestamps) uses automatic selection — purely picture-driven: scene changes (sampled pass beyond 20 min) + even-interval backfill. That's insurance for "the model skipped the two-pass flow", with no keyword guessing. > Precision note: captured frame times may differ from the requested time by up to about ±0.5s (fast seek depends on keyframe spacing), and more after scene alignment and sharp-frame refinement — the exact requested moment is shown per frame as 原始请求 in the report. --- ## 🔧 Configuration Defaults live in cordis.patch.yml; override any field in $DSH_HOME/profiles/web/cordis.patch.yml (later layers win per row): yaml - override: - id: bilibili config: sessdata: '' # optional Bilibili SESSDATA (logged-in subtitles / more comments) commentLimit: 20 # max comments to fetch maxFrames: 6 # max keyframes extractFrames: true # false = text-only mode downloadVideo: true # download video locally before capture (recommended) keepVideo: false # true = keep downloaded video files maxVideoMinutes: 30 # videos longer than this are captured remotely per frame maxDownloadMb: 800 # download size cap (MB) quality: 32 # 16=360p 32=480p 64=720p 80=1080p detectScenes: true # scene-change detection (sampled pass beyond 20 min) sceneThreshold: 0.4 # scene threshold 0-1, higher = stricter sharpFrames: true # sharp-frame preference: blurdetect picks the sharpest frame within ±1.5s asrProvider: 'bcut' # ASR engine: bcut (default) | sherpa-onnx (Chinese) | whisper-local | auto | none sherpaBin: '' # sherpa-onnx-offline binary path sherpaModel: '' # sherpa model onnx path (SenseVoice/Paraformer) sherpaModelType: 'sense-voice' # sense-voice | paraformer | zipformer2-ctc sherpaTokens: '' # sherpa tokens.txt path sherpaThreads: 0 # sherpa CPU threads (0 = auto) whisperBin: 'whisper-cli' # whisper.cpp binary (PATH or absolute path) whisperModel: 'medium' # small / medium / large-v3, or a ggml-*.bin path whisperModelDir: '' # model dir; empty = <whisperBin dir>/models whisperLanguage: 'zh' # transcription language whisperThreads: 0 # whisper CPU threads (0 = auto) visionProvider: 'none' # frame vision: none (default) | ollama | llama-cpp | openai-compatible visionBaseUrl: '' # vision endpoint; empty + ollama = http://localhost:11434/v1 visionModel: 'medium' # low(2B) / medium(4B) / high(8B), or an explicit model name visionApiKey: '' # cloud vision API key (empty for local) visionPrompt: '' # vision prompt (empty = built-in per-model default) visionPromptByModel: {} # per-model prompt overrides (explicit model name / low / medium) visionMaxFrames: 6 # max frames to describe (aligned with maxFrames) framesDir: '' # frame output dir; empty = system temp/dsh-bilibili/<bvid> summaryTemplate: '' # template path; empty = bundled templates/summary.md timeoutMs: 300000 # overall tool timeout (ms) ### Local ASR transcription (optional; sherpa-onnx recommended for Chinese) No-subtitle videos default to Bijian (zero config, anonymous, China-friendly). To go fully offline or when Bijian fails, switch to a local engine. The plugin ships no models — only the interface; models and binaries are downloaded by the user (no API keys, quotas, or fees involved). #### Recommended: sherpa-onnx (Chinese, SenseVoice) Bilibili is mostly Chinese content, and SenseVoice beats Whisper on Chinese accuracy while being faster and smaller; official models are hosted on ModelScope (fast in China). | Tier | Recommended model | Size (approx) | For | |------|-------------------|---------------|-----| | Low | SenseVoiceSmall (int8) | ~230 MB | low-end machines | | Mid | SenseVoiceSmall (fp32) | ~900 MB | mainstream (recommended) | | High | Paraformer-large | ~2.5 GB | high-end / maximum accuracy | Steps: 1. Download the sherpa-onnx-offline binary for your OS from sherpa-onnx; 2. Download a model (model.onnx + tokens.txt) — SenseVoice models are on ModelScope or the sherpa-onnx model list; 3. Set asrProvider: 'sherpa-onnx' and fill sherpaBin / sherpaModel / sherpaTokens (sherpaModelType defaults to sense-voice). #### Alternative: whisper.cpp (general / English) | Tier | whisperModel | Model file | Size | For | |------|--------------|------------|------|-----| | Low | small | ggml-small.bin | ~466 MB | low-end / fast drafts | | Mid | medium | ggml-medium.bin | ~1.5 GB | mainstream | | High | large-v3 | ggml-large-v3.bin | ~3 GB | high-end | 1. Download whisper-cli from whisper.cpp; 2. Download the matching ggml-*.bin model into a models/ directory; 3. Set asrProvider: 'whisper-local' and fill whisperBin / whisperModel. > Note: asrProvider: 'auto' falls back in order Bijian → sherpa-onnx → whisper-local. For Chinese, use at least medium (whisper) or pick SenseVoice (sherpa) directly. sherpa-onnx CLI flags vary slightly across versions — check --help of your build if something errors. ### 🔍 Frame vision descriptions (optional) When the main model has no vision, enable this feature: each captured frame is sent to a vision model and returned with a description field, so the main model can decide which images to cite in the report — cite only when visual confirmation matters (charts / UIs / demo details); pure talking-head frames are not cited. Off by default; a vision failure never breaks the main flow (frame paths are still returned). Local (recommended): install Ollama and pull a model — no keys, offline, free: | Tier | visionModel | Ollama model | RAM (approx) | For | |------|-------------|--------------|--------------|-----| | Low | low | qwen3-vl:2b | ~2 GB | ultra low-end | | Mid | medium (default) | qwen3-vl:4b | ~4 GB | low-end to mainstream (recommended) | | High | high | qwen3-vl:8b | ~6-8 GB | mainstream, best quality | Larger models can be passed as explicit names (e.g. qwen3-vl:32b) — they are just no longer a default tier. Mid-tier alternative MiniCPM-V 4.0 (OpenBMB, 2026; officially claims to surpass GPT-4.1-mini, runs on phones; official GGUF/int4 releases — check the official library for its Ollama tag). visionModel also accepts explicit model names (Ollama tags or cloud model ids). Other non-Qwen models (verified on the Ollama library, 2026-08): minicpm-v:8b (OpenBMB MiniCPM-V 2.6, strong Chinese OCR), moondream (1.9B, English-first), gemma3n (Google, English-first). Kimi-VL / InternVL / GLM-4V are not in the official Ollama library — use community GGUFs via llama.cpp or cloud OpenAI-compatible APIs (e.g. Moonshot / Zhipu). MiniCPM-V 4.0's official GGUF works on the llama-cpp route. > Selection rationale: this task is understanding frame content + emitting a citation hint, not OCR transcription — the weights are on Chinese scene understanding and instruction-following, so the default tiers use the Qwen3-VL family (consistent behavior, shared prompts); MiniCPM-V 4.0 for maximum edge efficiency. llama.cpp (local alternative): run llama-server with a vision GGUF (model + mmproj); it exposes an OpenAI-compatible API, and visionModel is simply the --alias you set at launch — matching the alias to a tier keyword reuses the tier config directly: sh llama-server -m qwen3-vl-8b-q4_k_m.gguf --mmproj mmproj-qwen3-vl-8b.gguf --port 8080 --alias qwen3-vl:8b # plugin config: visionProvider: 'llama-cpp' + visionModel: 'medium' The three tiers are Qwen3-VL-first: low Qwen3-VL-2B, mid Qwen3-VL-4B (default), high Qwen3-VL-8B. Verified GGUFs: official Qwen/Qwen3-VL-4B/8B-Thinking-GGUF (with mmproj), community unsloth/Qwen3-VL-4B-Instruct-GGUF, etc. If your llama.cpp build doesn't support the Qwen3-VL architecture yet, fall back to Qwen2.5-VL-7B-Instruct-GGUF (official on ModelScope). llama.cpp also supports MiniCPM-V (incl. 4.0), InternVL, GLM-4V, LLaVA, gemma3n, moondream2, and more. Cloud: any OpenAI-compatible endpoint via visionProvider: 'openai-compatible' + visionBaseUrl + visionModel + visionApiKey. Single-frame description doesn't need flagship multimodal models — budget tiers suffice: GLM-4V-Flash (free quota for Chinese) / GPT-4o-mini / SiliconFlow Qwen-VL. Per-model prompts: every built-in prompt's task is understanding the frame's content (what's happening, what's shown) — visible text is paraphrased as key points only, never transcribed. The plugin picks prompts automatically per model (MiniCPM-V family gets a dedicated prompt, moondream2 gets English, small low-tier models get a shorter prompt); override with visionPrompt (global) or visionPromptByModel (per explicit model name or low/medium tier). Citation quality gate: every vision description must end with a single line 「配图建议：适合/不适合」 (suitable = clear, informative, helps readers understand; unsuitable = talking head, blurry, or uninformative). Frames carry a citation_hint field, and reports cite only suitable frames, at most 1-2 per section. 2B measured results (2026-08, llama.cpp b10428 + Qwen3-VL-2B-Instruct-Q4_K_M, 16-thread CPU, 7 ground-truth test images): 100% citation-tail stability; chart values (120/240/180/300) and poster numbers (32% / 5M / three rounds) matched exactly; talking-head frames correctly marked unsuitable; 4-9s per frame. The low-tier short prompt was tuned from these live runs (anti-hallucination + citation criteria). > Note: describing several frames on a CPU takes tens of seconds to minutes (faster on GPU); visionMaxFrames caps the count. --- ## 📁 Project structure dsh-bilibili/ ├── lib/ │ ├── index.js # Cordis plugin entry: tool registration + system prompt + config schema │ ├── extractor.js # extraction layer: Bilibili API + downloads + scene detection + ffmpeg capture │ ├── keyframes.js # pure functions: automatic frame selection (picture-driven), time formatting │ └── format.js # pure functions: extraction result → model-facing text digest ├── templates/summary.md # bundled default output template (replaceable) ├── test/ # unit tests (node --test) ├── cordis.patch.yml # bundle patch layer (recognized by the plugin system) └── package.json # dsh.bundle.patch declaration + peer dependencies --- ## 🔌 Plugin standard This plugin follows the DeepSeek Harness plugin standard: the npm package declares dsh.bundle.patch → dsh plugin add reconciles it into dsh.profile.bundles → the Cordis loader mounts it after a profile restart. See the deepseek-harness repo for the standard. --- ## 🛠️ Local development (link mode) dsh plugin add installs local directories via link: (changes take effect immediately). Because ESM resolves dependencies by real path, the plugin directory needs a junction pointing at the profile's node_modules: ```powershell New-Item -ItemType Junction -Path ".

ode_modules@deepseek-ai" -Target "$env:USERPROFILE\.dsh\profiles ode_modules\@deepseek-ai" ``` Restart the web profile after changes. --- ## ⚠️ Limitations - Multi-part videos: only part 1 is handled for now; - Videos without subtitles are transcribed via Bijian ASR by default (local sherpa-onnx / whisper.cpp available); transcripts may contain recognition errors and are labeled as such in the result; - Bijian ASR is an anonymous endpoint and may rate-limit rapid repeated calls (returns errors); for frequent/stable transcription preferasrProvider: 'auto'` or a local sherpa-onnx setup; - Frame images are not auto-cleaned (so the model can read_image anytime) — that costs disk space; - Node's fetch doesn't read system proxy env vars; proxied networks are pending support; - Scene detection switches to a sampled pass beyond 20 minutes (full decode cost). --- ## 📄 License MIT