测试通过和功能真的能用之间隔着好几层容易被忽略的陷阱——mock 太多会验证不到真实链路,覆盖率只证明代码跑过、不证明行为正确,而 CI 矩阵如果全靠 YAML 手写会迅速失控成没人敢改的黑箱。DeepSeek Harness 用一套分层的 Vitest 测试体系 一套用 TypeScript 编写的门禁编排器(scripts/run-gates.ts)来同时压住这几个问题,本篇逐层拆开这套体系,并用docs/testing.md里的原文措辞讲清楚背后的测试哲学。课程写作时docs/testing.md的 Tiers 一节列的是五层,现在实测已经变成七层——新增了 Owner-local expected output 和 Performance benchmarks 两层,本文已按当前实际内容更新;这类层数/门禁数量的具体断言建议读者去仓库里cat docs/testing.md核对一遍最新版本,不要死记本文的数字。学习目标理解 Vitest 分层测试(unit / coverage / e2e / expected / bench / snapshot / web,共七层)各自验证什么、为什么不能合并成一层。能引用docs/testing.md里几条核心测试哲学的准确措辞,并理解它们各自对应的具体工程手段(而不是空洞的口号)。理解 GitHub Actionsci.yml里all-checks-passed汇总门禁背后 required jobs 的设计意图,以及为什么要用scripts/run-gates.ts而不是纯 YAML 去编排具体检查项。理解 lefthook 本地 hook 和 CI 完整矩阵之间快反馈 vs 全覆盖的分工关系。知道.gitlab-ci.yml在这套体系里承担的专门角色,以及它为什么和 GitHub Actions 分离。背景与设计动机如果只跑一层能过就行的单元测试,会掉进什么坑?最典型的是绿色的单测,坏掉的产品——一个组件的所有子函数都被 mock 到位、单测全绿,但真实的组装路径(Loader 装配、进程间协议、构建产物的模块解析)从来没有被跑过一次,直到用户手上才炸。反过来,如果每次提交都跑一遍真实调用 LLM API 的端到端测试,本地开发和无密钥的贡献者会被彻底挡在门外。这两个极端都不可接受,所以测试必须分层,每一层验证不同的真实性维度,而不是简单地叠加更多断言。CI 编排同样面临一个类似的取舍:全部写成 GitHub Actions YAML 的run步骤,意味着这次到底跑了哪些检查、依赖关系是什么、并发度怎么控制这些逻辑分散在几十个 job 里,改一次门禁要在 YAML 里到处改脏活。DeepSeek Harness 把检查项的依赖图、并发调度、失败汇总这部分逻辑写成了一个真正的 TypeScript 程序(scripts/run-gates.ts),YAML 里的每个 job 只负责调用这个程序的某个具名模式(ci-static、ci-coverage等),把编排这件事从声明式配置降级成了普通的程序设计问题——可以写类型、可以写测试、可以本地直接跑。核心机制详解Vitest 七层测试体系docs/testing.md用Tiers一节把测试体系拆成七层(课程写作时是五层,新增了 expected 和 bench 两层),每层对应一个独立的 vitest config 文件:# docs/testing.md当前版本节选,已按新增的两层做了删减聚焦) - **Unit** (pnpm run test): vitest over package and example specs under their tests/** directories plus repository script specs under scripts/**/*.spec.ts; tests stay with the code area they exercise. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). - **Coverage gate** (pnpm run test:coverage): the gating run, per-file 100% on packages/*/*/src. An uncovered line is often dead code the gate flags for deletion, not a missing test to bolt on. - **Real-API e2e** (pnpm run test:e2e): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (EXA_API_KEY, PERPLEXITY_API_KEY, …); each suite self-skips without its key so keyless CI stays green. - **Owner-local expected output** (pnpm run test:expected)新增: keyless assembled CLI/process expectations without a recorded-session round trip. Drivers use *.expected.e2e.ts beside tests/expected/; CI runs built exports. - **Performance benchmarks** (pnpm run test:bench; required Linux PR gate node 24 / benchmarks)新增: benchmarks/ groups user-path gates. It builds libraries and workers; timed code runs under plain Node, never TSX. Synthetic inputs enforce time, heap, and scaling budgets. - **Snapshot** (pnpm run test:snapshot): a top-level scenarios highest recorded parent generation supplies user input and model replay, then serves as the expected persisted result. Process scenarios start through dsh: headless owns one-shot behavior, the SDK owns persistent control, ACP owns automation-protocol behavior, and Web retains browser/ARIA evidence. - **Web browser snapshot** (pnpm run test:web; required Linux PR gate): Chromium compares session-driven output under snapshots/web/ and UI-only output under apps/web/tests/expected/. CI forces read-only DSH_SNAPSHOTreplay, never writing expected outputs.对应关系是vitest.config.ts(unit coverage 共用同一份配置,只是 coverage 多传--coverage)、vitest.e2e.config.ts(real-API)、vitest.expected.config.ts(owner-local expected output,新增)、vitest.bench.config.ts(performance benchmarks,新增,对应新的顶层benchmarks/workspace 成员)、vitest.snapshot.config.ts(keyless 快照)、vitest.web.config.ts(浏览器快照)。新增的两层里,expected解决的是snapshot 需要录制模型回放、成本较高和e2e 需要真实 API key之间的中间地带——不需要密钥也不需要录制会话,直接断言构建产物 CLI/进程的确定性输出;bench则是把性能预算(耗时、堆内存、随规模变化的增长曲线)第一次变成了正式的、会挡 PR 的 CI 门禁,而不只是包内自查的诊断脚本(.perf.ts)。每一层解决的是不同问题:Unit关心的是代码区域自身的正确性——放在被测代码旁边的tests/**目录里,而不是集中式的顶层测试目录。仓库要求每个 registry 都要有 HMR 安全性测试:卸载贡献插件的 fiber,断言清理逻辑正确执行——这直接对应 Cordis 插件系统的热重载语义。Coverage gate是真正卡 CI 的门禁,要求packages/*/*/src下逐文件100% 行覆盖率。vitest.config.ts里能看到这个阈值配置是硬编码的:// vitest.config.ts当前版本;阈值现在包在一个 coveragePartitionMode 判断里, // 因为覆盖率门禁现在支持拆分成多个并发分区跑,分区模式下不需要在每个分区里 // 单独判定 100% 阈值,汇总阶段才做最终判定 coverage: { provider: v8, include: [packages/*/*/src/**/*.{ts,tsx}], exclude: [ packages/*/*/src/types.ts, packages/*/*/src/bin.ts, packages/*/*/src/worker.ts, // ... ], // 100% or it doesnt merge (docs/testing.md: excessive tests are welcome). // Per-file so a well-covered big file cant subsidize a bare one. thresholds: coveragePartitionMode ? undefined : { perFile: true, statements: 100, branches: 100, functions: 100, lines: 100, }, },perFile: true是关键设计——它防止一个测得很充分的大文件把一个几乎没测的小文件的平均分拉上去。豁免机制也是显式且有记录的,比如pwsh-local相关文件的覆盖率豁免直接写在配置里,并附带探测逻辑:// vitest.config.ts // pwsh-locals run/start/lifecycle suites self-skip without a real pwsh // (executor.spec.ts hasPwsh), leaving this file far below per-file 100% // on pwsh-less hosts; the exemption keeps those hosts green while CI // runners ship pwsh and still enforce the full bar. const pwshCoverageExclusions spawnSync(resolvePwshPath(), [...]).status 0 ? [] : [packages/shell/pwsh-local/src/index.ts, packages/shell/pwsh-sandbox/src/**/*.ts]这条豁免不是我们懒得测,而是显式声明这个文件的覆盖率依赖一个本地环境不一定具备的外部程序(真实的pwsh),豁免只在探测不到pwsh时生效,CI 跑在真装了pwsh的 runner 上时依然强制满覆盖率。Real-API e2e是是否真的能用的最后一道防线。vitest.e2e.config.ts里能看到测试超时被放宽到 120 秒、重试 2 次:// vitest.e2e.config.ts test: { ... // Real model calls: generous timeouts, and retries for transient flakes // (the shared internal key hits concurrency quotas). No coverage — the // unit suites own the coverage gate. testTimeout: 120_000, hookTimeout: 30_000, retry: 2, fileParallelism: e2eMaxWorkers 1, maxWorkers: e2eMaxWorkers, },Snapshot和Web browser snapshot分别验证协议/展示层的外部行为和浏览器渲染出来的真实像素/DOM。test:web在 CI 里被固定成只读回放模式(DSH_SNAPSHOTreplay),永不在 CI 里写期望输出——录制和刷新只能在本地做,每一次 diff 都要求人工审查。几条测试哲学,以及它们对应的具体机制docs/testing.md里的措辞相当克制,但每一句背后都指向仓库里可以验证的具体机制,不是空洞的原则宣言。We are DeepSeek — do not ration real-API tests.完整上下文是:A no-key test proves plumbing; only a with-key run proves the agent works against a real model. Cover file-writing prompts, multi-turn conversations, tool use, and mid-stream cancellation. Highest-value aresmoke teststhat boot the real example, send one prompt, and check the world — they catch the green unit tests, broken product class that mocks cannot ( postmortem 0001).这句话背后有一个真实的踩坑记录——docs/postmortem/0001-acp-default-export-drops-inject.md——说明这条原则不是理论推导,而是从一次真实事故里总结出来的。Prefer the real implementation over a mock.原文强调只在昂贵或不确定的边界上打 mock,举例用的具体测试工具函数在课程写作之后换过一次,但原则完全没变:Mock only the expensive or non-deterministic boundary (LLM adapter, network, clock); keep everything downstream real. A hand-rolled stand-in proves the bridge moves bytes, not that the shipping tool behaves as asserted. Bridge tool-call tests keep the real tool registry and pipeline behind the scripted mock model:makeBridgeHarness()mounts the loop, session store, tool registry, and JSONL persistence with aMockAdapteras the only mock (packages/acp/acp/tests/harness.ts).也就是说测试里唯一被替身的是模型会返回什么 token这个不确定的边界(MockAdapter),循环本身、会话存储、工具注册表、JSONL 持久化全都是真实代码路径。Verify the world, not the self-report.这条直接约束断言的写法:An e2e assertion re-runs the command or re-reads the file externally; a keyword probe on the agents own output lets a cheating agent pass. Assert untouched files are byte-identical.意思是不要去检查agent 自己说它写完了文件,而要真的重新读一次那个文件、重新跑一次那个命令去验证——这条规则专门用来防止模型输出里恰好带对了关键词,但实际什么都没做这种假阳性。Test the real entry path.这条把真实细化到了发布出去的那份构建产物本身:Real entry path means the published artifact: a packagebinruns builtlib/bin.jsunder plainnode, exposing failures tsx masks (settle races, module resolution, swallowed load failures). ... Keep the built-artifact smokes green (packages/examples/*/tests/built-bin.e2e.ts,packages/ptc-runtime/ptc-runtime-node/tests/built-lib.e2e.ts), and assert a genuinely-missing config exits non-zero.这段引用本身有一处值得留意的细节:docs/testing.md原文举的例子里仍然写着packages/examples/*/tests/built-bin.e2e.ts,但packages/examples这个分类目录在第 01 篇讲到的那次重构里已经被移除了——实测这个测试现在的真实路径是apps/cli/tests/built-bin.e2e.ts。这说明连仓库自己的文档在快速迭代下也会出现代码已经挪了、文档里的例子还没跟上的滞后,是个提醒:读到具体路径类的例子时,养成用find/grep核实一遍的习惯,比死记文档原文更可靠。apps/cli/tests/built-bin.e2e.ts、packages/ptc-runtime/ptc-runtime-node/tests/built-lib.e2e.ts这类测试专门跑构建产物而不是源码——因为tsx直接跑.ts源码时,某些只在真实模块解析、真实 worker/子进程边界才会出现的问题会被悄悄掩盖掉。这类跑构建产物的built-lib.e2e.ts测试现在分布得更广了,packages/experimental/*(webworker-packer、inspector、agent-team)、packages/lsp/lsp-stdio、packages/api/remotes下都能找到同名文件——不再局限于课程写作时点名的那一两个包,是这条测试原则被更广泛套用的结果(此处packages/examples分类目录、code-runtime分类目录都已经不存在了,参考第 01 篇)。GitHub Actions:required jobs 与all-checks-passed.github/workflows/ci.yml现在有 720 行左右(课程写作时是 900 多行——行数变少不代表检查变少,而是自托管 runner 的失败转移逻辑被重构、注释也更集中了),最终的汇总门禁逻辑依然很简单——all-checks-passed这一个 job 声明了 PR 真正必须全绿的 job 清单,但清单本身变长了:# .github/workflows/ci.yml当前版本 all-checks-passed: name: all checks passed # This bookkeeping-only verdict must not depend on custom-pool # provisioning — and under Linux failover it follows the same selector as # the worker jobs it aggregates, so a standard-hosted outage cannot strand # the branch-protection verdict either. ... needs: [node-24, node-24-coverage, node-24-bench, node-24-consumers, node-compat, python-sdk, python-runtime, windows-build, windows-native-tests] steps: - name: Fail if any needed job did not succeed if: contains(needs.*.result, failure) || contains(needs.*.result, cancelled) || contains(needs.*.result, skipped) ... - name: All checks passed和课程写作时相比,needs列表有两处变化:新增了node-24-bench(对应上一节新增的 Performance benchmarks 门禁层),原来的单个windowsjob 拆成了windows-build和windows-native-tests两个独立 required job(另有一个不在 required 列表里的观察窗windows-coverage,跑 PR 但不参与合并判定)。其余维度含义不变:node-24(node 24 / static,静态检查)、node-24-coverage(node 24 / coverage,覆盖率门禁)、node-24-consumers(node 24 / snapshots and artifacts,快照 构建产物验证 Playwright 浏览器快照)、node-compat(Node 双矩阵兼容性)、python-sdk(无密钥 SDK 套件)、python-runtime(单文件可执行体的发布形态 smoke)。job 内的新注释还透露了一个课程写作时没有的细节:CI 现在有一套显式的自托管失败转移开关(DSH_CI_FAILOVER_LINUX),all-checks-passed这个纯汇总 job 也要跟着同一个开关重新定位到备用 runner 池(vm-backup/blacksmith),避免自托管池故障时连这个 PR 到底算不算全绿这个判断本身都卡住。其余像ci-master.yml里的windows(Wine 演练,原称windows-native)、serial-linux-selfhosted、serial-macos、serial-windows这些 job 依然是自愈演练,不在 required 列表里——它们在每次 master push 上重复验证自托管失败转移池的可用性,不参与 PR 是否可合并的判断。scripts/run-gates.ts:用程序而不是 YAML 编排门禁每个 CI job 的具体检查内容不是写在 YAML 里,而是通过一个统一的 CLI 委托给scripts/run-gates.ts:// scripts/run-gates.ts export type Mode | ci-primary | ci-linux-primary | ci-static | ci-lint-contracts-ready | ci-coverage | ci-bench | ci-snapshot | ci-artifacts | ci-consumers | ci-windows-blocking | ci-windows-complete | ci-windows-observational-ready | node-compat | check-all | hygiene | doc-sync | doc-quick export function gatesForMode(selected: Mode): Gate[] { switch (selected) { case ci-primary: return ciPrimaryGates() case ci-static: return ciStaticGates({ ownsBuild: false }) case ci-coverage: return coverageGates() // ... } }ci-static模式展开后是一组具体的检查项组合:// scripts/run-gates.ts function ciStaticGates(options: { ownsBuild: boolean }): Gate[] { return [ ...ciSharedStaticGates(), ...options.ownsBuild ? [pnpmScript(build, build)] : [], ...docSyncLeafGates({ /* ... */ }), pnpmScript(module-graph, verify-module-graph, { label: module graph }), pnpmScript(knip, knip), ] }每个Gate是一个带id、command、needs(依赖的其他 gate)、allowFailure的普通对象,调度器runGates()用这份依赖图做有界并发调度(DSH_GATE_CONCURRENCY环境变量可覆盖并发度)。这样设计的好处是显而易见的:门禁的依赖关系、并发策略、失败汇总逻辑全部是可以本地单独运行、单独调试、写单测覆盖的 TypeScript 代码,而不是散落在 YAMLrun:步骤里的 shell 片段拼接。改一次覆盖率门禁应该在哪些检查之后跑只需要改needs数组,不需要理解 GitHub Actions 的needs:/矩阵语法怎么和自定义 shell 脚本纠缠在一起。lefthook:本地快反馈,CI 全覆盖lefthook.yml开头的注释直接说明了分工原则:# lefthook.yml # Git hooks (lefthook). Keep these local checkpoints fast; CI owns the full # repository-wide gate matrix. # Install: node scripts/install-lefthook.mjs (runs automatically via postinstall).pre-commit只做与本次改动直接相关、能在几秒内跑完的检查——翻译配对校验(仅校验 staged 的*.i18n.yaml)、archived agent notes 格式、对 staged 文件跑 oxlint 自动修复、第三方 notice 自动重生成、空白检查、vendor manifest 一致性守卫:# lefthook.yml pre-commit: jobs: - name: lint (staged) glob: *.{ts,tsx,mts,cts,mjs} exclude: [vendor/*/src/**] run: node_modules/.bin/tsx scripts/run-oxlint.ts --config .oxlintrc.staged.json --fix --no-error-on-unmatched-pattern {staged_files} stage_fixed: true - name: vendor manifest guard run: scripts/check-vendor-manifest.shpre-push则只跑一项——完整的pnpm run typecheck,这是本地能负担、又能拦住类型错误才推上去被 CI 打回来这种最常见浪费的最后一道防线。真正的全仓库 100% 覆盖率Windows Wine 矩阵Playwright 浏览器快照这些开销大、跑得慢的检查,全部只在 CI 里跑——本地 hook 从不试图复制它们。.gitlab-ci.yml:专职 Python wheel 发布仓库同时维护一份.gitlab-ci.yml,和 GitHub Actions 的职责完全不重叠——它只在打python-v*格式的 tag 时触发,分build/publish两个 stage,产出并发布 Python SDK 的 wheel 包(SDK wheel 五个平台的 runtime wheel:linux x64/arm64、macOS arm64/x64、windows x64——课程写作时还只有三个平台):# .gitlab-ci.yml workflow: rules: - if: $CI_COMMIT_TAG ~ /^python-v[0-9]\.[0-9]\.[0-9](-[0-9A-Za-z.])?$/ - when: never stages: - build - publish它验证 tag 版本号与package.json一致,构建单文件可执行体,用smoke-python-runtime.py做发布形态的 smoke 测试,最后用twine upload发布——这条流水线和 GitHub Actions 的node-*/windows/python-sdk等 job 是互补关系:GitHub Actions 管每次 PR/push 的正确性门禁,GitLab CI 管打 tag 触发的一次性发布动作。常见问题/易踩坑把覆盖率不足当成要补测试,而不先检查是不是死代码——docs/testing.md明确提醒An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on。忘记test:web在 CI 里是只读回放模式——本地跑test:web:refresh才能刷新快照,直接在 CI 环境变量里改DSH_SNAPSHOT不会生效,而且这类改动一定要求人工审查每一个 diff。以为本地 lefthook 全绿就等于 CI 会全绿——lefthook 故意只覆盖本地能快速验证的子集,pre-push只跑 typecheck,完整的覆盖率门禁、Windows 矩阵、快照回归都只存在于 CI。小结DeepSeek Harness 的测试体系用七层 Vitest 配置(课程写作时是五层,后来新增了 expected 和 bench 两层)分别验证代码区域自身正确(unit)、逐文件真的被跑过(coverage)、对着真实模型真的能用(e2e)、构建产物的确定性输出没有偏差(expected)、关键路径的性能预算没有退化(bench)、外部契约没有意外变化(snapshot)、浏览器渲染出来的真实效果没有回归(web),每一层都对应docs/testing.md里一条可验证、有真实事故背书的测试哲学,而不是抽象的质量口号。CI 层面,scripts/run-gates.ts把门禁的依赖图和调度逻辑从 YAML 里抽出来变成普通 TypeScript 程序,GitHub Actions 只负责触发具名模式并在all-checks-passed里汇总 required jobs(现在多了新的 bench 门禁,Windows 也拆成了 build/native-tests 两个独立 job,汇总 job 本身还要跟着自托管失败转移开关重新定位);lefthook 承担本地几秒内能跑完的快反馈,和 CI 的完整矩阵形成互补而非重复;.gitlab-ci.yml则专职处理与 PR 门禁完全无关的 Python wheel 发布流程(runtime wheel 的平台覆盖已经从三个扩展到五个)。
