dsh-frontier-repro
Primary signals in. Auditable reproduction out.
中文 · Download · Star the project
An evidence-first frontier AI radar and reproduction gate for [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness). This is not another generic news reader, arXiv search engine, or paper summarizer. It covers the missing path from a curated primary-source signal to a versioned release evidence bundle and then to an auditable reproduction decision. Executed attempts are retained—including failures and negative results—and a successful exact/scaled claim requires commands, artifacts, measured claim results, and independent verifier evidence. | Discover what matters | Build evidence, not hype | Reproduce without overclaiming | |---|---|---| | Curated arXiv, first-party lab, GitHub, Hugging Face, hardware-vendor, verified blog, and optional X signals. | Cross-source release bundles retain immutable revisions, provenance, missing evidence, and substantive change history. | Claim-level protocols preserve commands, artifacts, metrics, resources, verifiers, failures, and negative results. | > [!TIP] > If this makes a frontier release easier to trust or reproduce, [give the repository a star](https://github.com/JimchengChina/dsh-frontier-repro/stargazers). It helps more DSH users find the project. ## What is different Existing DSH plugins already cover literature discovery/full text (`dsh-ai4scholar`, `dsh-literature`), guided paper reading and notes (`dsh-paper-workshop`), general RSS reading (`dsh-news`), scientific-claim adjudication (`dsh-research-plugins`), and replayable experiment execution (`dsh-science-workbench`). This plugin does not duplicate those surfaces. It adds: - a single timeline across arXiv, first-party lab releases, official model artifacts, verified personal blogs, and opt-in X API timelines; - explicit identity provenance for person sources; - explainable ranking rather than an opaque quality label; - a mode-specific evidence matrix for exact, scaled, and behavioral reproduction; - a hard separation between “prerequisites documented” and “reproduction executed”; - run evidence that rejects unsupported success claims. - Hugging Face paper-to-artifact expansion and immutable GitHub commit evidence; - immutable arXiv versions and Hugging Face model/dataset repository SHAs; - persistent source health with volume, structure, failure-streak, and staleness alerts; - conservative lab-scoped clustering into Frontier Release Evidence Bundles, with corroboration and explicit missing evidence; - capability/evaluation/license diffs, `firstSeenAt` / `lastSeenAt` / `supersedesDigest` version chains, and substantive-only watchlists; - a first-party GitHub organization adapter for new repositories, releases, tags, immutable commits, and release-asset digests; - claim-level `execute_existing`, `partial_reimplementation`, and `from_scratch_replication` protocols; - preserved multi-attempt resource records (GPU/CPU/VRAM/time/cost/data scale/job URL) and mandatory verifiers for success; - a file-only Hugging Face Trackio logbook scaffold instead of a duplicate experiment UI; - serialized, journaled collection batches with guarded LIFO rollback; - an explicit evidence dependency graph and canonical reproduction handoff manifest. See [docs/research.md](docs/research.md) for the overlap audit and [docs/architecture.md](docs/architecture.md) for the Cordis spatiotemporal mapping. Corporate/personnel announcements, placeholder titles, missing-date records from date-contracted sources, and personal-life posts are filtered before persistence. Sources may declare category allow/deny lists and known boilerplate titles. Recent Hugging Face model cards are inspected only to discover paper, code, data, and evaluation links; full card text is not archived. arXiv records retain both a stable paper id and the observed immutable `vN`. GitHub repositories and public Hugging Face models/datasets are resolved to full SHAs, while mutable popularity counts remain context only. ## Built-in sources - arXiv categories `cs.AI`, `cs.CL`, `cs.LG`, `cs.CV`, `cs.RO`, and `cs.SE` through the official Atom API. - OpenAI News; Anthropic Newsroom, Research, and Engineering; Google DeepMind News; MiniMax Research; Kimi Blog; DeepSeek API News and Transparency; and Z.ai model release notes. - NVIDIA Technical Blog, AMD ROCm Blog, and the filtered Intel Artificial Intelligence News feed for training, inference, accelerator, compiler, and benchmark signals. - Verified DeepSeek, Moonshot AI, MiniMax, and Z.ai Hugging Face organizations for model artifacts, plus the official DeepSeek GitHub organization for repository/release/tag changes. - Sam Altman's blog and Jack Clark's Import AI. - Sam Altman, Dario Amodei, and Demis Hassabis on X, through X API v2 only. No unverified DeepSeek-founder or GLM-person account is preloaded. Add a custom person source only with a first-party `identityEvidenceUrl`. ## Tools | Tool | Purpose | |---|---| | `frontier_repro_status` | Source, corpus, credential, and persisted source-health status without network access | | `frontier_repro_collect` | Refresh curated sources, persist, dedupe, and rank signals | | `frontier_repro_events` | List versioned cross-source release evidence bundles and substantive watch changes | | `frontier_repro_bundle` | Inspect one bundle, predecessor chain, source records, claims, and all attempts | | `frontier_repro_watch` | Add/remove/acknowledge release watches by substantive digest | | `frontier_repro_search` | Search the local corpus only | | `frontier_repro_revert_collection` | Safely undo the latest live collection batch | | `frontier_repro_get` | Full provenance, artifacts, assessment, and run history | | `frontier_repro_assess` | Evidence gate for exact/scaled/behavioral reproduction | | `frontier_repro_assess_claims` | Claim-level gate for execute/partial/from-scratch modes and exact/scaled/toy equivalence | | `frontier_repro_graph` | Deterministic source/artifact/requirement/run dependency graph | | `frontier_repro_record_result` | Persist executed commands, artifacts, metrics, deviations, and verdict | | `frontier_repro_record_attempt` | Preserve one claim-level attempt, resources, verifier, and honest outcome | | `frontier_repro_manifest` | Canonical reproduction handoff manifest with SHA-256 integrity | | `frontier_repro_trackio_scaffold` | Export a local-first Trackio logbook scaffold without executing or publishing | ## Install Requires Node.js `^22.19` or `>=24` and DeepSeek Harness `0.1.0-rc.7` or newer within the `0.1.x` line. The in-app X credential card uses the plugin-settings extension introduced in rc.7. ```sh dsh plugin --profile web add "https://github.com/JimchengChina/dsh-frontier-repro/releases/latest/download/dsh-frontier-repro.tgz" ``` For source development, clone the repository, run `pnpm install`, and add its absolute path instead. Headless profiles can install the same release with `--profile headless`. Restart the selected profile. The default corpus is `$DSH_HOME/frontier-repro/index.json`. During installation, the plugin reminds you that X API access is optional. In DSH Web, open **Settings → Plugins → Plugin configuration → Frontier Repro** to save an X API bearer token. If you do not configure one, X sources are disabled and skipped by default; arXiv, official lab blogs, GitHub, Hugging Face, and all other sources continue normally. Tagged releases attach a verified package tarball and SHA-256 file. Storefronts can use `https://github.com/JimchengChina/dsh-frontier-repro/releases/latest/download/dsh-frontier-repro.tgz` without running a source build. ## X access Use **Settings → Plugins → Plugin configuration → Frontier Repro** in DSH Web, set `X_BEARER_TOKEN` in the launch environment, or write the same reference through the DSH credentials store. The token is resolved for every collection and never enters config, the corpus, or tool output. Without X API access, default collection skips X sources and every other source continues. Explicitly requesting an X source still returns its exact missing condition. No HTML scraping fallback is used. ## Configuration ```yaml - id: frontier-repro config: defaultDays: 90 defaultLimit: 20 maxRecords: 1000 maxCollections: 20 requestTimeoutMs: 20000 maxResponseBytes: 5242880 pageConcurrency: 3 githubEnrichLimit: 8 huggingFaceEnrichLimit: 20 githubTokenEnv: GITHUB_TOKEN xBearerTokenEnv: X_BEARER_TOKEN # storagePath: /absolute/path/index.json # sourceFile: /absolute/path/sources.json promptGuidance: true promptOrder: 145 ``` `sourceFile` is trusted local administrator configuration. Tool arguments never accept arbitrary source URLs. Custom sources may declare `allowCategories`, `denyCategories`, `requirePublishedAt`, `boilerplateTitles`, and `healthStaleAfterDays`. A `github_org` source declares an organization plus bounded `releaseRepoLimit` / `releasesPerRepo`; person sources require a name, role, and first-party identity evidence. `GITHUB_TOKEN` is optional and only raises public API limits; X API access remains explicitly required for X sources. See [sources.example.json](sources.example.json) and [docs/source-policy.md](docs/source-policy.md). ## P1 workflow: release bundle to verified attempt 1. Collect first-party signals, then select a release with `frontier_repro_events` / `frontier_repro_bundle`. 2. Watch it. Later collections set `changed_since_watch` only when capability claims, evaluation evidence, licenses, or immutable artifacts changed; acknowledge after review. 3. Define independently testable claims with primary evidence and choose `execute_existing`, `partial_reimplementation`, or `from_scratch_replication`, plus `exact`, `scaled`, or `toy` equivalence. 4. Record every attempt. `passed` requires all required claim metrics and a passing verifier with identity/evidence. A toy pass is always stored as `toy_only`, never `reproduced`. 5. Export the Trackio scaffold to visualize the preserved attempts locally or, only when intended, in a Hugging Face Space. The scaffold does not execute the experiment or change verdicts. ## Verification ```sh pnpm run verify node scripts/smoke.mjs pnpm run smoke:plugin node scripts/smoke.mjs openai-news google-deepmind-news zai-release-notes ``` The tests cover parsing, enrichment, event clustering/versioning, GitHub organization releases, watch rollback safety, claim grading, multi-attempt/verifier enforcement, Trackio export, digests, lifecycle serialization, transaction rollback, evidence topology, manifests, storage, and registration/disposal through the real DSH `ToolRuntime`. `smoke.mjs` exercises source adapters directly; `smoke:plugin` runs collection through the real tool runtime. Both perform read-only requests and never call X. ## Limitations - First-party sites change. Source failures are isolated and reported; successful records still persist. - Source-health alerts indicate collection drift or staleness, not that an upstream publication is false or abandoned. - OpenAI RSS is fetchable while article pages may block automated clients. The feed record remains usable and a selected page can be verified with DSH's browser/web capability. - A primary source proves provenance, not correctness. Ranking is discovery priority, not a truth score. - X requires the user's own API entitlement and budget; the plugin does not bypass access controls. - Anonymous GitHub API requests have low rate limits. Configure the optional token or reduce `githubEnrichLimit`; Hugging Face pinning can be bounded with `huggingFaceEnrichLimit`. A pinning failure is reported and does not erase the source record. - Release clustering is deliberately conservative and lab-scoped. Unrecognized model names remain one-record bundles rather than being merged by vague semantic similarity. - A Trackio scaffold is an export. The plugin does not install Python packages, run jobs, upload to Spaces, or treat dashboard logging as verification. - The gate checks evidence completeness and consistency, not whether every submitted evidence statement is true. - Manifest integrity is a deterministic content digest, not an identity signature or remote artifact checksum. - Scheduling is deliberately left to existing DSH schedule/cron plugins. ## License MIT