Vision Toolkit

If you use DeepSeek or another text-only model in DeepSeek Harness (DSH), you may have run into the same problems: the model cannot see a screenshot, generic descriptions miss the point, buttons have no usable coordinates, and a rebuilt page can look “close enough” without a way to measure the remaining difference. 🏆 This project is the first comprehensive vision-tool plugin in the DeepSeek Harness ecosystem: it was initiated before internal beta and built during the beta with reference to [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit). > **Original work:** The system and division of responsibilities behind these visual tools, together with the `vision-skills` Skill, were personally created and continuously refined by the author through long-term real-world use and repeated iteration. ## Highlights - **Paste and use it immediately.** Paste an image in DSH Web and the text-only route switches to its `(Vision Toolkit)` variant without manual path copying or model changes. - **A seamless image workflow.** Native thumbnails, session history, and workspace paths stay intact; Web can preview artifacts and Headless can continue using the same structured results. - **One command to install.** The built-in free Gemini 3.7 Flash vision service is ready after installation, with no API key required. - **Built-in free vision.** The shared service works immediately after installation with a quota of **300 images per machine per day**. - **Vision guided by intent.** The agent extracts evidence for the task at hand, such as “Where is the error?” or “Where is the button?”, instead of returning a generic caption. - **A complete screenshot-to-verification loop.** Reference images, HTML screenshots, difference regions, and pixel comparison work together for UI restoration. [`agent-vision-toolkit`](https://github.com/Anionex/agent-vision-toolkit) gives an agent more than image captions: it can read, locate, crop, trace, rebuild, and verify visual work. DSH Vision Toolkit is its native DeepSeek Harness integration, bringing that workflow into Web and Headless Profiles. This project has two layers: 1. **Visual tools and a Skill:** the agent learns when to inspect, ground, OCR, crop, trace, or compare pixels. 2. **Native DSH integration:** those capabilities live inside Profiles, sessions, Settings, Artifacts, and the Web UI, with a free Gemini 3.7 Flash vision service ready after installation. > **Install and use it immediately.** The default setup includes a free Gemini 3.7 Flash vision service and requires no API key. Cropping, pixel diffing, color analysis, foreground extraction, SVG tracing, and HTML screenshots run locally without spending vision API requests. ```sh dsh plugin --profile web add @anionex/dsh-vision-toolkit ``` **Upstream toolkit:** [Anionex/agent-vision-toolkit](https://github.com/Anionex/agent-vision-toolkit) · **Project website:** [agent-vision.anionex.me](https://agent-vision.anionex.me)

Table of contents

- [Highlights](#highlights) - [Recent updates](#recent-updates) - [Who it is for](#who-it-is-for) - [See it in action](#see-it-in-action) - [Quick start: three steps](#quick-start-three-steps) - [Common workflows](#common-workflows) - [Toolbox](#toolbox) - [Configuration and limits](#configuration-and-limits) - [Troubleshooting](#troubleshooting) - [Development and community](#development-and-community)

## Recent updates - **2026-08-16 · Windows Python:** Added Microsoft Store Python support, fixing first-time isolated-runtime setup failures for affected Windows users. - **2026-08-17 · Free vision upgrade:** Switched the built-in no-key service to Gemini 3.7 Flash and fixed Qwen/Gemini bounding-box coordinate order. - **2026-08-16 · Better free vision:** Switched the built-in no-key service to Groq Qwen3.6, improving image understanding without adding setup steps. - **2026-08-16 · Image paste:** Text-only routes now switch to a `(Vision Toolkit)` variant and keep a workspace path, fixing blocked pastes and images that could not be reused later. - **2026-08-16 · More shared capacity:** Expanded the free service capacity to reduce peak-time `429` responses. - **2026-08-16 · Real model test:** Added a full image-request test in Settings, fixing the false confidence caused by a successful `/models` request to a model that still cannot process images. ## Who it is for | The problem | What Vision Toolkit delivers | |---|---| | **A text-only model cannot see a screenshot** | Paste an image in DSH Web; the plugin obtains visual evidence and returns the task-relevant parts to the text model | | **The description is long but misses the point** | Ask “Where is the error?” or “What color is the submit button?” and receive an answer focused on that question | | **The model knows an element exists but cannot act on it** | Get original-image pixel coordinates and an optional labeled or numbered preview | | **Long-screenshot OCR skips or duplicates lines** | Split and audit the image while preserving Markdown, chunks, manifests, and resumable run state | | **UI restoration is judged by feel** | Compare the reference and implementation screenshots to get a difference percentage, ranked regions, a heatmap, and JSON | | **Screenshot assets cannot be reused** | Produce a crop, transparent PNG, color palette, or editable SVG instead of stopping at prose | ## See it in action ### Paste an image directly into DSH

*Paste an image into the conversation. A text-only model can switch to its `Vision Toolkit` variant and inspect the image in the context of the user's question.* ### Screenshot to editable page

*Left: the reference screenshot. Right: an editable HTML/CSS result. The result can continue into screenshot rendering and pixel comparison instead of ending as an image description.* ### Sketch to working interface

*Left: a hand-drawn reference. Right: the working interface reconstructed from it.* ### Turn “looks close” into a verifiable result The repository includes a reproducible UI-restoration example: the agent renders the reference and implementation, then uses difference regions, a heatmap, and a JSON report to guide the next correction.

## Quick start: three steps ### 1. Install ```sh dsh plugin --profile web add @anionex/dsh-vision-toolkit ``` You can install it into a Headless Profile too: ```sh dsh plugin --profile headless add @anionex/dsh-vision-toolkit ``` ### 2. Restart and check it Restart a running Web Profile, then open **Settings → Vision Toolkit**. The free provider is already configured; run **Test vision model** to confirm it is reachable. The first start prepares an isolated runtime: the plugin prefers a system Python 3.11+; when none is found, it downloads a hash-verified standalone Python (about 35 MB) from a pinned release source on first use. A normal installation does not require an `agent-vision-toolkit` source checkout or a local path setting. ### 3. Paste an image and describe the outcome you want Paste a screenshot into the conversation or place an image in the session workspace, then invoke `/vision-skills`. For example: ```text Inspect this screenshot. Explain the error and tell me what to fix first. Find the login button in the top-right corner, return original pixel coordinates, and make a boxed preview. Crop this icon and convert it to SVG. Rebuild the page from reference.png. After each pass, render it and run a pixel diff until the major differences are gone. ``` ## Common workflows | Task | Recommended workflow | |---|---| | Image Q&A or screenshot debugging | Inspect → answer around the current question → locate details when needed | | Find a button, icon, or text region | Ground the target → return pixel box → create a labeled preview | | Extract an icon from a screenshot | Ground → crop → trace to SVG | | Read a long webpage screenshot | Split → OCR → merge Markdown → audit boundaries | | Recreate a page or component | Reference → implementation → HTML screenshot → pixel diff → iterate | | Extract brand visuals | Crop region → analyze dominant colors → extract foreground → export transparent PNG | ## Toolbox The plugin provides 10 tools that can be called independently or composed into a workflow: | Tool | Best question to ask | Main result | |---|---|---| | `vision_glance` | “What is happening in this image?” | Focused answer, description, OCR, or multi-image comparison | | `vision_ground` | “Where is the thing I need?” | Original pixel coordinates and optional boxed preview | | `vision_detect` | “Which buttons, icons, or elements are present?” | Numbered element inventory, coordinates, and optional preview | | `vision_crop` | “Extract this region as its own image” | PNG or JPEG crop | | `vision_trace` | “Turn this shape into an editable vector” | SVG | | `vision_pixel_diff` | “Where does the implementation differ from the reference?” | Difference percentage, ranked regions, heatmap, and JSON | | `vision_long_screenshot_ocr` | “Read this entire long screenshot” | Markdown, chunks, manifest, and audit output | | `vision_extract_foreground` | “Remove the background from this subject” | Transparent PNG | | `vision_dominant_colors` | “Which colors dominate this area?” | Palette or ranked candidate colors | | `vision_html_screenshot` | “Render this local page at an exact viewport or capture the full page” | PNG and optional CSS `pageHeight` | Coordinates always use original-image pixels in `x1,y1,x2,y2` form, so grounding output can feed directly into cropping, tracing, or later automation. For a long HTML document, pass `fullPage=true`. The requested width and height remain the layout viewport, while the resulting PNG covers the complete document and reports `pageHeight` in CSS pixels. ## How it works The plugin keeps image understanding and deterministic local image processing in one Agent workflow. Expand the flow below for the implementation boundary.

Architecture and image-input behavior

```mermaid flowchart LR Image["Screenshot or local HTML"] --> Skill["vision-skills Skill"] Skill --> Agent["Text agent selects a task"] Agent --> Vision["Use a vision model when image understanding is needed"] Agent --> Local["Run crop, SVG, and pixel work locally"] Vision --> Result["Answer, OCR, coordinates"] Local --> Artifact["PNG, SVG, heatmap, JSON"] Result --> Session["Continue reasoning and acting"] Artifact --> Session ``` The visual capabilities come from a packaged, pinned `agent-vision-toolkit` snapshot. The DSH plugin handles installation, session-scoped tool exposure, Credentials, path checks, cancellation, timeouts, result files, and Web presentation. The runtime never fetches upstream `main` in the background. The bundled `vision-skills` Skill is the DSH adapter of the upstream `vision-tools` Skill: its `SKILL.md` plus all five upstream playbooks. Tool names, argument syntax, Artifact delivery, progressive exposure, and DSH path/lifecycle boundaries are adapted; the upstream tool-selection rules, coarse-to-fine method, and task SOPs remain intact. The exact upstream Skill commit, source hashes, adapted hashes, and reviewable adapter patch are recorded in `assets/skill/UPSTREAM.json` and `patches/vision-tools-dsh.patch`. For routes that DSH positively identifies as text-only, the plugin registers a sibling ` (Vision Toolkit)` variant. By default, pasting an image in DSH Web switches to that variant and gives the model both a reusable workspace path and a visual description focused on the current task.

## Configuration and limits ### Built-in free service The default setup uses: ```text Base URL: https://vision.anionex.me/v1 Model: gemini-3.7-flash API Key: https://agent-vision.anionex.me (filled automatically) ``` Requests that still use the previous `qwen/qwen3.6-27b` model name remain compatible and are routed to the Qwen backend. This is a shared zero-configuration entry point, not an unlimited private endpoint. Request safeguards include: | Limit | Current value | |---|---:| | Daily quota | 300 images per machine per day | | Images per request | Up to 5 | | Image size | 4 MiB per image | | Decoded pixels | 20,000,000 per image | | Output | Up to 4,096 tokens per request | These safeguards prevent unusually large requests from monopolizing memory or request time. When shared capacity is reached, the service returns a readable `429` response with `Retry-After` instead of collapsing into an unexplained model failure. Existing clients that still send `api_key="free"` remain compatible. ### Bring your own vision model For higher quotas, private endpoints, or another model, change the provider in **Settings → Vision Toolkit** and store the API key as a DSH Credential. Settings stores the Credential reference and never reads the saved secret back into the browser. **Step-by-step Groq tutorial:** [Get a free Groq API key and use Qwen3.6-27B for image understanding](docs/groq-qwen3.6-vision.md). It includes screenshots for account/API-key setup, the exact Vision Toolkit settings, and working cURL and Python examples. You can also configure a Profile patch: ```yaml - id: vision-toolkit config: provider: baseUrl: https://api.example.com/v1 credential: MY_VISION_KEY model: your-vision-model protocol: openai ``` OpenAI Chat Completions-compatible endpoints and Anthropic Messages are supported. The Web Settings panel exposes the full provider, runtime, timeout, image-limit, and image-input-variant configuration. For a trusted internal endpoint that uses a self-signed certificate or MITM proxy, start the DSH process with `VISION_SSL_VERIFY=0`. The plugin forwards that value to the isolated Python runtime; certificate verification remains enabled when the variable is unset or has any other value. The false values `false`, `off`, `no`, `none`, and `disabled` are also accepted, case-insensitively. ### Requirements - A DeepSeek Harness Web or Headless Profile. - Node.js `^22.19.0` or `>=24.0.0`. - Python 3.11+ is usually not needed in advance: the plugin prefers a system Python and otherwise downloads a pinned standalone Python 3.13 automatically, preparing its own isolated environment. Only that first automatic download needs network access. - Only `vision_html_screenshot` requires Chrome, Chromium, or Edge. - Inputs must be PNG, JPEG, GIF, or WebP files in the session workspace, the platform temporary directory, or an explicitly allowed directory. ### Configure the Python runtime By default the plugin picks a system Python 3.11+, or downloads a standalone Python when none is found; most users never need to configure this section. The rest is for advanced setups where automatic discovery fails, a specific interpreter is required, or an external runtime is used. The packaged `managed` runtime creates its own isolated virtual environment. `runtime.python` selects the Python executable used to bootstrap or refresh that environment; it does not replace the managed environment with the interpreter's global site-packages. Set it when automatic discovery fails or when several Python installations exist. The override is also used by `runtime.mode: external`. Python 3.11 or newer is required; the automatically downloaded standalone Python is 3.13.15 and, like a system interpreter, is only used to bootstrap the isolated environment. Without an override, the plugin tries `python3` then `python` on macOS/Linux, and `python`, `py -3`, then `python3` on Windows, before falling back to the standalone download. A configured value is passed as one executable name or path, not as a shell command with arguments, so use `py` (not `py -3`) for the Windows launcher; use an absolute path when you need a specific version. Configure it in the Profile patch: ```yaml - id: vision-toolkit config: runtime: # macOS/Linux system Python python: python3 # Or a project-local environment: # python: /absolute/path/to/project/.venv/bin/python # Windows venv (forward slashes also work in YAML): # python: C:/Users/you/project/.venv/Scripts/python.exe # Windows launcher, when its default Python is 3.11+: # python: py ``` For a managed runtime, create the project-local interpreter and point `runtime.python` at it. The plugin installs the locked dependencies into its own managed cache, so installing the lockfile into this bootstrap environment is optional: ```sh python3 --version # must report 3.11 or newer uv venv .venv --python 3.13 ``` For `runtime.mode: external`, install the locked dependencies using the `runtime/requirements.lock` from the **DSH Vision Toolkit plugin** checkout, then point `runtime.agentVisionToolkitPath` at a separate exact `agent-vision-toolkit` snapshot. The packaged `vendor/agent-vision-toolkit` directory is such a snapshot when it has not been modified: ```sh uv pip install --python .venv/bin/python \ -r /absolute/path/to/dsh-vision-toolkit/runtime/requirements.lock ``` ```yaml - id: vision-toolkit config: runtime: mode: external python: /absolute/path/to/dsh-vision-toolkit/.venv/bin/python agentVisionToolkitPath: /absolute/path/to/dsh-vision-toolkit/vendor/agent-vision-toolkit ``` On Windows, use `py -3 --version` for the version check and `.venv\Scripts\python.exe` plus `runtime\requirements.lock` in the corresponding commands: ```powershell py -3 --version # must report 3.11 or newer uv venv .venv --python 3.13 # External mode only; use the plugin checkout's absolute lockfile path: uv pip install --python .venv\Scripts\python.exe -r C:\absolute\path\to\dsh-vision-toolkit\runtime\requirements.lock ``` Point `runtime.python` at the same interpreter, save the Profile patch, and restart the Web Profile. Then open **Settings → Vision Toolkit**: the Runtime panel should show the resolved interpreter and Python version, and **Run health check** plus **Test vision model** should complete without the Python-version error. A final smoke test is to place a PNG/JPEG in the session workspace and call `vision_glance`. The path fence automatically allows the session workspace and the platform temporary directory. On macOS/Linux the temporary root is `/tmp`. On Windows it is `TEMP`, then `TMP`, with the operating-system fallback if neither is set; model-generated `/tmp/...` paths are translated to that Windows directory before the normal realpath fence runs. No `allowedDirs` entry is needed for these platform temporary paths. Use `allowedDirs` only for additional trusted input roots outside the workspace and platform temporary directory: ```yaml - id: vision-toolkit config: allowedDirs: # macOS/Linux example - /srv/vision-inputs # Windows example (use this instead on Windows) # - D:/vision-inputs ``` `allowedDirs` is an input allowlist, not the managed runtime cache. The managed runtime keeps its own files under `$DSH_HOME/cache/dsh-vision-toolkit` (or `~/.dsh/cache/dsh-vision-toolkit` when `DSH_HOME` is unset); that directory does not need to be added. Environment-variable forms such as `$env:TEMP` and `%TEMP%` are not expanded inside `allowedDirs`, so configure extra roots with real absolute paths.

Install, upgrade, disable, and uninstall

```sh dsh plugin --profile web update @anionex/dsh-vision-toolkit dsh plugin --profile web remove @anionex/dsh-vision-toolkit ``` If you are migrating from the retired `@dsh-external/dsh-vision-toolkit`, remove the old package first and install `@anionex/dsh-vision-toolkit`. To disable the bundle temporarily, set this in the Profile patch: ```yaml - id: vision-toolkit disabled: true ``` Restart the Web Profile and refresh the page after enabling or upgrading the Web plugin.

### Plugin updates In **Settings → Vision Toolkit**, **Check for updates** queries the Profile's npm registry. For a direct registry installation, **Update and restart** installs only the exact version you confirmed, verifies it, and restarts an explicitly opted-in POSIX Web process on a fixed `--port`. Local/workspace/file/git/URL installs, Windows, dynamic ports, read-only Profiles, and manager-owned processes remain check-only. The updater revalidates the Profile before mutation, snapshots the original manifest and lockfile, and holds a token-owned cross-process lock. The current Web process exits only after the restart helper confirms that the backup is readable and the lock handoff succeeded. When the Profile was already operational, the replacement must report both the ta