dsh-plugin-judge · Plugin Value Auditor for DeepSeek Harness > Judge whether a DeepSeek Harness plugin is worth using: pre-install review, > post-install audit, and re-audit reminders when the model changes. 中文文档 ## The problem this plugin solves Plugins fall into two natures: - Capability plugins give the model new abilities (new tools, skills, MCP servers, services, commands) — like giving the model "an extra hand". - Constraint plugins impose rules and guardrails on the model (injecting system prompt sections, personas, AGENTS.md rules, intercepting agent/pre-step / tools/pre-execute, restrict / guard on tools, or complete: true wholesale prompt replacement) — like putting the model "in a cage". Constraint plugins carry a hidden cost: the model will follow the rules faithfully (compliance looks great), but it performs worse on the task you actually care about — rules crowd the context and suppress the model's own judgment, putting a low ceiling over a capable model. Worse, as models get upgraded, many "patch rules for the old model" turn from patches into ceilings, and the plugin becomes obsolete or even harmful. So a plugin's value is not a property of the plugin alone — it is a binary relation of "plugin × current model". That is exactly what this plugin evaluates. ## Features | Feature | When | How | | --- | --- | --- | | Pre-install review | Before installing | /plugin-audit <github:owner/repo or npm:pkg> command, the judge_plugin tool, or the settings-page input. Fetches the plugin source → static scan + LLM judge → a "worth installing?" verdict | | Post-install audit | Already installed | Enumerates installed bundles from the profile → scans and classifies each (capability / constraint / cosmetic / hybrid) → constraint-risk score → settings-page report panel | | Model-switch reminders | When the model changes | Watches the default model → flags plugins whose verdict depended on the old model → overlay popup reminder to re-audit | ## How it judges (two layers combined) 1. Rule heuristics (free, deterministic): scans the plugin source and its injected content for signals, weights them into a 0–100 constraint-risk score, and classifies the plugin as capability / constraint / cosmetic / hybrid. 2. LLM judge (fine-grained, on demand): hands the scan result, the injected content summary, and the current model's identity to a model and asks whether these rules help or suppress this specific model, returning a verdict with reasons. ## Usage sh # Install (requires an existing web profile) dsh plugin --profile web add dsh-plugin-judge # Or install from source dsh plugin --profile web add <path-to-this-repo> After installing and restarting the harness: - Type /plugin-audit github:some/repo in chat for a pre-install review; - Or let the model call the judge_plugin tool directly; - The settings panel gains a Plugin Judge page: scan reports for every installed plugin, a pre-install review input, and audit history; - After a model switch, if any plugins need re-auditing, a floating reminder pops up on the page. ## Design ### Data flow Installed plugins: profile manifest (~/.dsh/profiles/<profile>/package.json → dsh.profile.bundles) + each package's package.json / lib source (read locally via node:fs) Candidate plugins: package.json + entry source fetched from npm registry / jsDelivr / GitHub raw (Node fetch) Heuristic scan: capability signals (+) vs constraint signals (−), weighted → class + risk score LLM judge: llm.stream (generative verdict, carrying the current model identity) → verdict + reasons Model watch: agentDefaultModel.currentSelection() polling (fallback: agent/request waterfall observation) Persistence: ~/.dsh/plugin-judge/audits.json (audit history + model fingerprint) Bridge: Host ctx.connection.rpc.handle('/plugin-judge', …) ← Client rpc.call polling Reminders: Host produces pending-reminders → Client polls → shell.overlay popup Reports: Client settings.section "Plugin Judge" page + judge_plugin tool card ### Signal table (heuristics) Capability signals: tools.register / defineTool / skills / MCP / webServer.register / registerFetchProvider / commands.register / provide( … Constraint signals: systemPrompt.section / ctx.systemPrompt / persona / writing AGENTS.md / agent/pre-step / tools/pre-execute / tools/execute waterfall listeners / restrict( / guard( / complete: true wholesale replacement / strong-directive density (must, never, always, MUST, NEVER, 必须、不得、禁止) / fixed-template instructions (strictly follow, exact format, 严格按照、按以下格式) … ### Layout judgePlugin/ ├── package.json # dsh.bundle.patch + dsh.client.inject + exports (Node >= 18) ├── cordis.patch.yml # insert: plugin-judge ├── lib/ │ └── index.js # Host half: inventory / source fetching / heuristic scan / LLM judge / │ # model watch & reminders / tool & command / connection RPC bridge / persistence └── client/ └── client.js # Client half (__ModuleLoader__ format, no build step): # settings.section report panel + shell.overlay model-switch reminder popup ### Disclaimer Verdicts are advisory opinions produced by heuristics plus a model judge — not a security audit. Installing any plugin runs third-party code on your machine with your own permissions; read the source before you install.