From 502ea5eca8dcac56bea4014e9f1cc57afda0c63c Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 17 Oct 2026 15:37:20 +0000 Subject: [PATCH] Clarify ownership of upstream integration tests --- docs/policy/testing.md | 41 +++++++++++++++++++++++++++++++++++++--- docs/roadmap/upgrades.md | 21 +++++++++++++++----- 2 files changed, 54 insertions(+), 8 deletions(-) diff --git a/docs/policy/testing.md b/docs/policy/testing.md index d0f16c6..aed917c 100644 --- a/docs/policy/testing.md +++ b/docs/policy/testing.md @@ -2,7 +2,7 @@ **Base Policy Version:** 1.3 -**Last Updated**: 2026-10-07 +**Last Updated**: 2026-10-17 **Upstream Source:** [testing.md](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/testing.md) @@ -70,7 +70,7 @@ implemented: Use external-package consumer tests for facade semantics and real backend integration where correctness depends on that interaction. Internal package -tests own focused mechanisms. Do not retest the dependencies' complete suites. +tests own focused mechanisms. Boundary checks protect selected architectural constraints; behavioral evidence protects invocation semantics and ownership through conversions. @@ -79,7 +79,7 @@ concurrency guarantees only if deliberately selected. Coverage targets, race testing, static analysis, example gates, and platform matrices require their own risk justification; another library's workflow does not establish them here. -To the extent the following features are present in the application, prioritize tests for: +For behavior owned by CommandKit, prioritize tests for: 1. Public and package-level contracts. 2. Domain rules and important invariants. @@ -106,6 +106,41 @@ application's guarantees. Account for protection already provided at other layer Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere. +## Upstream integration testing ownership + +CommandKit tests the interfaces and adapters through which it exposes upstream +APIs. ConfigKit, PromptKit, and other integrated libraries own tests establishing +the correctness of their domain functionality. Trust their public contracts; +CommandKit tests must not independently establish whether those libraries +implement their claimed domain functionality. This applies to partial feature +checks as well as complete suites. + +Integration assertions should establish CommandKit-owned behavior: + +- CLI choices, typed inputs, configuration selection, and context reach the + appropriate upstream API with the intended values. +- CommandKit performs its selected coordination and preserves returned data, + diagnostics, and supported error identity across adaptation. +- CommandKit's reports and invocation outcomes correctly represent the returned + evidence, including presence, null, empty, and redacted values where applicable. +- Command availability, input/state lifetimes, cancellation handling, and output + delivery follow CommandKit's contracts. + +For example, test that a namespace choice becomes the intended native listing +filter, rather than testing which resources PromptKit considers members of a +namespace. Test faithful presentation of ConfigKit's returned source evidence, +rather than its destination resolution, version checking, or sensitivity +classification. Test request forwarding and returned-error handling, rather than +whether PromptKit correctly detects missing artifacts or invalid templates. + +A real upstream engine or result can be a useful fixture for a focused +integration test. Its use does not change assertion ownership: the test should +detect incorrect CommandKit adaptation, not independently establish upstream +domain correctness. Use the simplest suitable fixture or test double; this +boundary does not require new mocking interfaces or prohibit real dependencies +in tests. CommandKit's own parsing, registration, coordination, presentation, +and outcome behavior remain within its testing scope. + ## Choose the right test boundary Choose the narrowest stable boundary that expresses the behavior clearly and diff --git a/docs/roadmap/upgrades.md b/docs/roadmap/upgrades.md index 822e94d..b234e8c 100644 --- a/docs/roadmap/upgrades.md +++ b/docs/roadmap/upgrades.md @@ -283,11 +283,22 @@ does not require a CommandKit version to mirror any upstream version. ### Evidence for the eventual implementation -The implementation plan should select proportionate evidence for native source -placement/version disclosure, direct prompt selection, absent/empty labels, -preserved requirements behavior, optional catalog availability, scoped/default -and explicit listing filters, errors, and output delivery. Use maintained offline -consumers and existing release-launcher coverage where applicable. +The implementation plan should test CommandKit-owned interfaces and adaptation +under the [upstream testing ownership policy](../policy/testing.md#upstream-integration-testing-ownership). +Verify that command inputs and context reach the intended native APIs, including +ID-only prompt requests and default/explicit listing filters. Verify faithful +representation of upstream-returned source evidence, optional labels, inventories, +diagnostics, and errors, together with CommandKit's optional command availability, +invocation lifetimes, outcome classification, and output delivery. + +These checks establish CommandKit's access to the capabilities. They should not +test whether ConfigKit correctly places or composes sources, checks versions, +or classifies sensitivity, nor whether PromptKit correctly resolves namespaces, +discovers requirements, or validates concrete requests. Those domain tests belong +to the upstream libraries. Real upstream fixtures and maintained offline +consumers can exercise CommandKit's boundary without duplicating native feature +test matrices. Existing release-launcher coverage likewise owns CommandKit's +tool invocation, not ReleaseKit's publication algorithms. Verify all affected modules both in the local workspace and with their own readonly dependency graphs. Temporary replacements for unpublished CommandKit