Plan Notarius credential forwarding

docs/roadmap/implementation.md

@@ -1,225 +1,220 @@
-# ConfigKit v0.3.0 Implementation Plan
+# Notarius Credential Forwarding Implementation Plan
 
-Status: Complete — 2026-09-16
+Status: Not started
 
-Implement the [feature roadmap](configkit-v0.3.0.md) in sequential stage order.
-Each stage is a bounded assignment for a gpt-5.6-terra coding agent. Mark it
-complete only after its acceptance checks pass, and record the commands and
-results beside its status. This replaces the completed original ConfigKit
-migration plan; that history remains in Git.
+Implement the [feature roadmap](notarius-credential-forwarding.md) in the order
+below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent.
+Record a short completion note and executed checks beside each stage's status;
+do not mark it complete until its acceptance criteria pass. The completed
+ConfigKit upgrade plan is preserved in Git history.
 
-## Working rules and fixed decisions
+## Instructions and fixed decisions
 
 - Read `docs/development.md`, all documents under `docs/policy/`, the feature
-  roadmap, and the stage-specific sources before editing.
-- Scope is the ConfigKit upgrade and the identified configuration-discovery test
-  isolation gaps. Do not implement the separate Notarius credential-forwarding
-  proposal, remove other roadmaps, or refactor unrelated command infrastructure.
-- Keep production configuration semantics unchanged: root-relative pipeline
-  paths, current schema identifiers, imports/profiles, presence-aware defaults,
-  historical ownership checks, effective-only secret resolution, safe inspection,
-  and domain expansion. Do not adopt `Resolution.InputInfo` or
-  `Resolution.Source` without a separate feature requirement.
-- Use the released ConfigKit v0.3.0 root API, not internal packages or sibling
-  checkout state. No replace directive, workspace dependency, vendoring, parser
-  fallback, or additional base load.
-- Preserve the Go 1.26 baseline and Linux amd64/arm64 plus best-effort macOS arm64
-  targets. Do not change CI toolchains or upstream repositories in this work.
-- Tests use real configuration/discovery logic and temporary files. Synthetic
-  credentials only; no real system configuration, cloud services, or adapter
-  executables. Do not mutate global search paths in the tests being repaired.
-- Keep each stage buildable. Diagnose unexpected failures rather than skipping
-  tests, weakening invariants, or changing snapshots indiscriminately.
-- Update canonical documentation only for implemented changes. Published release
-  notes, including v2.0.0, are immutable. No tag, push, release, or CI polling is
-  authorized by this plan.
+  roadmap, and each stage's listed sources. Inspect callers and existing tests
+  before editing; file lists are starting points, not edit allowlists.
+- Implement one optional `notarius.llm_api_key` field. It contains an environment
+  variable name, not a value. Omission and null mean no credential forwarding.
+  Every surviving non-null reference resolves during ordinary configuration
+  loading, even when extract is disabled, omitted from the range, or resumable.
+- Reuse ConfigKit v0.3.0 and invocation-scoped secret bootstrap. Do not upgrade
+  dependencies, modify upstream repositories, add flags, infer provider settings,
+  or add a general environment map. Keep unrelated work out of scope.
+- Keep the credential opaque across configuration and stage orchestration:
+  both `NotariusConfig.LLMAPIKey` and `notarius.RunRequest.LLMAPIKey` use
+  `*configkit.Secret`. This is the adapter's explicit typed-credential boundary
+  contract, using only ConfigKit's public root API. It avoids another credential
+  abstraction and avoids introducing raw-secret string fields in stage requests.
+  Only the Notarius adapter calls `Value()` to construct the child environment.
+- A supplied adapter credential must already be resolved. Reject unresolved or
+  empty programmatic references before process launch with a value-free error.
+  Do not resolve again from the process environment or silently drop an invalid
+  credential. Nil remains supported by all existing callers and fakes.
+- The adapter forwards `EnvName()` unchanged through `EnvOverrides` and
+  includes that name in `SensitiveEnvNames`. Do not widen the shared inherited
+  environment or hardcode a provider-specific destination variable.
+- Neither credential name nor value participates in extract's semantic resume
+  fingerprint. Safe configuration projections include at most the name:
+  renaming can change the effective digest, value rotation cannot. Do not bump
+  the extract fingerprint version or add digest-wide invalidation.
+- Never serialize or log a raw subprocess request/environment map. Tests use
+  disposable synthetic credentials and assertions that do not print values on
+  failure. Environment-mutating tests are nonparallel and restore state.
+- Retain the roadmap's downstream responsibility boundary: redact captured
+  streams, but do not scan/rewrite arbitrary Notarius files or claim protection
+  against encoded secrets or a malicious executable.
+- Each stage must compile and keep existing tests green. Stage 1's configuration
+  support is an intermediate state, not a releasable completed feature.
+- No release, tag, push, upstream changes, or CI polling are part of this plan.
+  Preserve published release notes and unrelated roadmaps.
 
-## Stage 1 — Isolate command configuration discovery
+## Stage 1 — Add typed configuration and safe projections
 
-Status: Complete — 2026-09-16
+Status: Not started
 
-Completion: Added an invocation-local pipeline discovery dependency with the
-production resolver as its default, then moved the affected missing-config tests
-onto real resolver calls over temporary candidate paths. The command-boundary
-coverage verifies successful default selection reaches the configured loader and
-that discovery failures stop before bootstrap, typed loading, or execution.
+Read: `internal/config/config.go`, `credential_projection.go`,
+`effective_projection.go`, `effective_pipeline.go`, `effective_digest.go`,
+`pipeline_configkit.go`, `pipeline_normalize.go`, `validate_notarius.go`,
+`notarius_test.go`, `load_validate_test.go`, and `pipeline_configkit_test.go`.
+Also read `docs/config.md` and `docs/internal/configuration.md`.
 
-Checks:
-
-- `go test ./internal/app -run 'TestResolvePipelineConfigPath|TestExecute.*Missing|TestExecuteSessionInitMissingDefaultConfig'` — pass.
-- `GOWORK=off go test -race ./internal/app` — pass.
-- `GOWORK=off go test ./...` — pass with ConfigKit v0.1.2.
-
-Purpose: Make missing-config command tests deterministic before changing the
-dependency, while exercising the real discovery logic.
-
-Read:
-`internal/app/command_dependencies.go`, `commands.go`, `config_loader.go`,
-`pipeline_config_path.go`, `pipeline_config_path_test.go`,
-`analyze_artifacts_commands_test.go`, `commands_test.go`,
-`operator_helpers_test.go`; configuration discovery in `docs/cli.md` and
-`docs/config.md`.
-
-Implementation:
-
-1. Add a private `resolvePipelineConfigPath func(string) (string, error)` field
-   to `commandDependencies`. Normalize a nil value to the existing
-   `resolvePipelineConfigPath` function. Keep that function's production default
-   candidates and explicit-path behavior unchanged.
-2. In `loadPipelineConfigWithDependencies`, call the normalized dependency
-   instead of the free function. Discovery must still precede secret bootstrap
-   and typed configuration loading. Preserve the selected path and all existing
-   error propagation. Verify all command-loading paths continue through this
-   shared owner; do not add per-command discovery implementations.
-3. Repair the analyze and publish missing-config tests and the
-   `run missing config uses defaults` case in `TestExecuteMissingRequiredFlags`.
-   Call `executeWithDependencies` with a discovery closure that invokes
-   `resolvePipelineConfigPathWithCandidates` on an invocation-owned list of
-   missing paths under `t.TempDir()`. Use fresh dependencies per invocation.
-   A small shared test helper is appropriate for this repeated fixture.
-   Do not merely inject a prewritten discovery error or add an explicit
-   `--config` argument: these tests must still exercise default discovery.
-4. Convert `TestExecuteSessionInitMissingDefaultConfigReportsSearchedPaths`
-   to the same seam, removing its global search-path mutation. Change
-   `TestResolvePipelineConfigPathWithCandidatesMissing` to temporary missing
-   paths rather than assumed-nonexistent absolute host paths.
-   Inspect adjacent missing-config tests and apply the same treatment where
-   needed; do not turn this into a general rewrite of unrelated successful
-   command fixtures.
-5. Preserve the existing resolver tests for explicit precedence and first
-   existing candidate. Add one representative command-boundary case using a
-   temporary existing default pipeline and the real resolver to prove that
-   successful discovery reaches the configured loader with that selected path.
-   Stop at a controlled loader boundary; do not initialize runtime services.
-   In a representative missing-path case, guard the loader/bootstrap/executor
-   boundaries so accidental continuation after discovery failure fails the test.
-6. Assert nonzero exit status, command context, a short missing-config diagnostic,
-   and the controlled searched path. Avoid complete prose snapshots. Keep all
-   test fixtures outside real system configuration and secrets directories.
+1. Add `LLMAPIKey *configkit.Secret` to `NotariusConfig`, with
+   `yaml:"llm_api_key,omitempty" json:"llm_api_key,omitempty"`. Leave its default
+   nil. Do not add a new environment lookup, codec, or secret-validation framework;
+   ConfigKit already owns loaded reference syntax and resolution.
+2. Add `NotariusConfig.MarshalYAML` in `credential_projection.go`, following
+   the existing Audita/S3 pattern: preserve all other Notarius fields and their
+   tags, and project the credential through `secretReferenceName`.
+   This protects direct YAML serialization as well as effective inspection.
+   JSON uses ConfigKit's safe secret representation. Preserve existing behavior
+   of nil stage sections, reference/output maps, and logical path normalization.
+3. Verify that the existing effective projection, schema-derived metadata paths,
+   copy/normalization helpers, source reporting, and digest capture carry the new
+   field correctly. Make only narrowly necessary changes. Do not add another
+   projection or provenance store, and do not mutate or resolve the secret in
+   normalization.
+4. Extend package-boundary tests for:
+   - omitted/null references; a valid custom reference; missing, empty, and
+     invalid reference names rejected without exposing values;
+   - selected profiles replacing or nulling an unavailable base reference,
+     while references in disabled Notarius configuration still resolve;
+   - safe effective YAML/JSON and direct pipeline YAML containing the reference
+     name, never its value; nil references retain the previous projection shape;
+   - correct root/profile provenance for the new field;
+   - unchanged safe digest after rotating a value under the same name, and
+     observable projection/diff identity changes after renaming the reference.
+   Reuse existing credential-projection and profile regression test owners.
+   Avoid asserting the entire configuration or complete diagnostic wording.
 
 Acceptance:
 
-- `go test ./internal/app -run 'TestResolvePipelineConfigPath|TestExecute.*Missing|TestExecuteSessionInitMissingDefaultConfig'`
-- `go test -race ./internal/app`
-- `go test ./...` on the existing ConfigKit v0.1.2 dependency.
-- The repaired tests own their candidate paths and do not depend on installed
-  configuration being absent. Do not create/remove an actual system config to
-  demonstrate this. Ordinary `Execute` still uses the production defaults.
+- `GOWORK=off go test ./internal/config`
+- `GOWORK=off go test ./...`
+- Existing examples still load without adding new required credentials.
+- No adapter delivery yet, but all serialization paths for the new field are safe.
 
-## Stage 2 — Adopt the released dependency and version diagnostics
+## Stage 2 — Deliver the selected credential and register redaction
 
-Status: Complete — 2026-09-16
+Status: Not started
 
-Completion: Upgraded the normal module graph from ConfigKit v0.1.2 to the
-released v0.3.0 without production-code migration. Party schema version `1`
-remains invalid through Narratio's unsupported-version check, while a boolean
-case preserves malformed-kind coverage. Existing credential, ownership,
-inspection, provenance, digest, bootstrap, and maintained-example regressions
-pass unchanged.
+Read: `internal/adapters/notarius/runner.go`, `subprocess.go`, `fake.go`,
+`subprocess_test.go`; `internal/adapters/subprocess/run.go`,
+`diagnostics.go`, `run_test.go`, `execution_test.go`;
+`internal/stage/extract.go`, `extract_test.go`; and
+`docs/internal/adapters.md`, `docs/internal/stage-extract.md`.
 
-Checks:
-
-- `GOWORK=off go mod tidy -diff` — pass.
-- `GOWORK=off go mod verify` — pass.
-- `GOWORK=off go test ./internal/config ./internal/app` — pass.
-- `GOWORK=off go test -race ./internal/config ./internal/app` — pass.
-- Targeted preserved-behavior regression selections in `internal/config` and
-  `internal/app` — pass.
-
-Purpose: Upgrade without altering the set of valid Narratio configurations.
-
-Read: `go.mod`, `go.sum`, `internal/config/party.go`,
-`party_test.go`, `pipeline_configkit.go`, `pipeline_configkit_test.go`,
-`document_configkit.go`, and ConfigKit's tagged v0.2.0/v0.3.0 release notes
-and public API documentation. The downloaded module is authoritative; a sibling
-checkout is reference material only.
-
-Implementation:
-
-1. Run `GOWORK=off go get gitea.maximumdirect.net/eric/ConfigKit@v0.3.0`
-   and `GOWORK=off go mod tidy`. Review the dependency diff; this task does not
-   authorize unrelated module or Go-version upgrades.
-2. Change the integer-version case in
-   `TestParsePartyCanonicalRejectsInvalidDocuments` from a `version.type`
-   expectation to unsupported-version rejection. Rename it to identify an
-   unsupported integer version. Preserve `PartySchemaVersion =
-   "narratio.party.v1"` and the existing application checker.
-3. Retain successful canonical-party coverage and ordinary-string strictness.
-   Add one boolean schema-version case if no equivalent malformed-kind coverage
-   exists, so the suite distinguishes an unsupported identifier from an invalid
-   version kind. Do not duplicate ConfigKit's numeric grammar matrix.
-4. Run existing profile credential replacement/null-removal, hidden duplicate
-   ordinary/family definition, safe inspection, provenance, digest, bootstrap,
-   and maintained-example tests unchanged. In particular, keep the
-   `Metadata.HistoricalPaths()` check and the single effective pipeline load.
-   No production migration is expected beyond Stage 1's dependency seam.
-5. Investigate any additional behavior difference against the roadmap before
-   changing production policy. Do not introduce private-engine imports or restore
-   old parsing to preserve incidental error wording.
+1. Add the optional opaque field to `notarius.RunRequest` and document that it
+   is an already-resolved reference. Pass `notariusConfig.LLMAPIKey` when extract
+   constructs the request. Do not add credentials to stage metadata, receipts,
+   output definitions, manifests, reference arguments, or generated files.
+   Keep fakes compatible and avoid dumping credential-bearing requests in tests.
+2. Validate any non-nil request credential before subprocess launch. Obtain the
+   name and value only at this adapter boundary; a non-nil unresolved reference
+   is an error, not an instruction to reread the environment.
+3. In `SubprocessRunner.Run`, supply exactly the selected credential override
+   and its sensitive name to the shared subprocess request. Nil adds no override
+   or sensitive name. Preserve arguments, reference ordering, timeout, working
+   directory, log/receipt destinations, process lifecycle, and bundle validation.
+4. Extend the extract request test to show the selected resolved reference
+   reaches the adapter without adding raw values to result metadata. Construct
+   resolved test credentials through the real loader, not reflection or private
+   ConfigKit state.
+5. Extend adapter tests for nil, resolved custom name, and unresolved reference.
+   Assert custom override/name wiring and unchanged argv without printing the
+   environment map. A caller changing the parent environment after loading must
+   not change the resolved credential delivered in this request.
+6. Extend the existing real subprocess test using a controlled executable:
+   - check that the selected value arrives under exactly the configured name,
+     and an unrelated synthetic parent credential is absent;
+   - report only success/failure of those checks, not an environment dump;
+   - emit the selected canary through stdout/stderr in success and nonzero-exit
+     cases, and assert its absence from persisted captured streams and returned
+     errors, with the existing redaction marker present;
+   - keep a valid successful bundle/receipt fixture; put any stdout canary in an
+     ignorable JSON field rather than corrupting the expected receipt protocol.
+   Use a custom name outside the built-in sensitive-name list. Verify actual
+   redaction on disk, not only the request's `SensitiveEnvNames`.
+7. Reuse shared subprocess timeout/cancellation and streaming-boundary coverage.
+   Do not duplicate its whole test matrix or add a new redactor. If adapter
+   control flow bypasses that shared path, add a focused regression for it.
 
 Acceptance:
 
-- `GOWORK=off go mod tidy -diff` and `GOWORK=off go mod verify`.
-- `GOWORK=off go test ./internal/config ./internal/app`.
-- `GOWORK=off go test -race ./internal/config ./internal/app`.
-- Valid configurations and effective secret handling remain unchanged; integer
-  party versions still fail. No temporary module file, test overlay, local
-  replacement, or skipped failing case is needed.
+- `GOWORK=off go test ./internal/adapters/notarius ./internal/adapters/subprocess ./internal/stage`
+- `GOWORK=off go test -race ./internal/adapters/notarius ./internal/stage`
+- Delivery is opt-in and uses the loaded value. Known and custom names receive
+  the same captured-stream protection, including subprocess failure diagnostics.
 
-## Stage 3 — Document and verify the completed adoption
+## Stage 3 — Verify command lifecycle, inspection, and resume invariants
 
-Status: Complete — 2026-09-16
+Status: Not started
 
-Completion: Narratio now uses released ConfigKit v0.3.0. The implementation
-adds invocation-local pipeline discovery for isolated command tests and adjusts
-the integer party-version expectation to Narratio's unsupported-version
-diagnostic; it requires no operator configuration migration. The internal
-command-dependency inventory documents the new seam. No unreleased Narratio
-release note exists, and the immutable v2.0.0 note was not changed.
+Read: `internal/app/secrets_bootstrap.go`, `secrets_bootstrap_test.go`,
+`config_commands_test.go`, `config_loader.go`, `extract_lifecycle_test.go`,
+`semantic_resume_test.go`; `internal/stage/semantic_contracts_delivery.go`,
+`semantic_contracts_delivery_test.go`, `prepare.go`, and their focused tests.
 
-Checks:
+1. Extend existing command credential fixtures with the Notarius reference.
+   Exercise real pipeline loading/bootstrap with a controlled pipeline path:
+   conventional and explicit secret directories, environment precedence, and
+   an existing empty environment variable remaining an error rather than being
+   replaced from a file. Reuse established fixture helpers and avoid a second
+   exhaustive bootstrap suite.
+2. Prove public config validation accepts the resolved field and rejects an
+   unavailable one before any adapter execution. Exercise show, sources, and
+   diff with a custom reference name and a synthetic canary: reference names and
+   ownership are visible, raw values are absent, and profile null removal works.
+3. Cover prepared/resolved pipeline diagnostics with the Notarius canary using
+   the actual persistence path. Assert that manifests and Narratio-generated
+   configuration contain no credential bytes. Do not assert sanitation of
+   arbitrary files authored directly by the downstream fixture.
+4. Extend extract semantic fingerprint and resume tests to cover credential
+   addition/removal, reference-name changes, and value rotation after valid load.
+   The semantic fingerprint and resume eligibility must remain unchanged.
+   Verify a previously successful extraction is reused without invoking Notarius;
+   retain normal selected-reference validation at command loading.
+5. Keep effective digest/provenance distinct from semantic resume identity.
+   Do not change the runner's invalidation relation, fingerprint version, or
+   schema to satisfy these tests. No blanket cache key may include credentials.
 
-- `GOWORK=off go mod tidy -diff` — pass.
-- `GOWORK=off go mod verify` — pass.
-- `GOWORK=off go test -count=1 ./...` — pass.
-- `GOWORK=off go test -race -count=1 ./...` — pass.
-- `GOWORK=off go vet ./...` — pass.
-- `GOWORK=off go build ./...` — pass.
-- `GOWORK=off go test ./internal/doccheck` — pass.
-- `GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$'` — pass.
-- `git diff --check` — pass.
-- `gofmt -l` on the changed Go files — no output.
-- `scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR` — pass with
-  Linux amd64, Linux arm64, and macOS arm64 assets. The macOS result is
-  cross-build evidence only, not native runtime validation.
+Acceptance:
 
-Purpose: Establish a clean, reproducible handoff for later release preparation.
+- `GOWORK=off go test ./internal/app ./internal/config ./internal/stage`
+- `GOWORK=off go test -race ./internal/app ./internal/config ./internal/stage`
+- Tests do not depend on real system configuration, installed Notarius, or a
+  provider. No secret value is printed even if a non-disclosure assertion fails.
+- Credential changes alone do not reexecute a completed extraction or stale
+  dependent artifacts.
 
-Read: `docs/development.md`, `docs/internal/configuration.md`,
-`docs/internal/overview.md`, `docs/config.md`, `docs/release.md`,
-`scripts/build-release-assets.sh`, and `scripts/check-release-candidate.sh`.
+## Stage 4 — Document the operator contract and validate the whole feature
 
-Implementation:
+Status: Not started
 
-1. Update internal command-dependency documentation to include the new
-   invocation-local discovery seam where the dependency inventory is described.
-   Update other current documentation only if a claim is actually affected.
-   The unchanged pipeline path policy and party identifier need no migration
-   instructions. Do not rewrite historical version pins in completed audit records.
-2. Record in this plan's completion notes the released dependency, diagnostic
-   adjustment, test-isolation repair, and absence of operator migration.
-   If a genuinely unreleased release note already exists, update it. There is
-   currently no such note: v2.0.0 is already tagged. Do not invent a release
-   version or modify v2.0.0; transfer this summary into the new note when a
-   separate release task chooses its version.
-3. Run all checks below against the normal module files, with no overlays or
-   skipped tests. Review the final diff for scope, formatting, whitespace,
-   credentials, and accidental dependency changes. Record results and limitations.
-4. Mark the implementation stages complete and update the feature roadmap's
-   status only after acceptance succeeds. Do not delete roadmap/audit records
-   or implement the adjacent Notarius roadmap.
+Read: `docs/config.md`, `docs/cli.md`, `docs/integrations/notarius.md`,
+`docs/internal/configuration.md`, `docs/internal/adapters.md`,
+`docs/internal/stage-extract.md`, `docs/troubleshooting.md`,
+`examples/README.md`, `docs/release.md`, and the feature roadmap.
 
-Required checks:
+1. Update canonical documentation with the field's type, absence/null behavior,
+   environment-name matching with Notarius/PromptKit, existing bootstrap
+   precedence, and the load-time requirement even when extraction is reusable.
+   State the redaction/downstream responsibility boundary and operational-only
+   resume treatment in their appropriate owners; link rather than duplicate.
+   Do not advertise new CLI flags or automatic provider-name discovery.
+2. Add a maintained, secret-free example of the optional reference in an
+   appropriate existing Notarius example. Keep ordinary examples runnable without
+   real keys: if a fixture actively declares the reference, its example test must
+   supply and restore a synthetic environment value. Never add real secret files.
+3. Preserve immutable release notes, including v2.0.0. Update an existing
+   unreleased note if one exists; otherwise record a concise release-ready summary
+   here for the separately authorized release task. Do not choose/tag a new
+   version merely to complete this stage.
+4. Run the repository checks below against normal module files. Inspect the diff
+   for unintended dependency, platform, CLI, schema, and fingerprint changes.
+   Mark the roadmap and all stages complete only after acceptance succeeds.
+   Retain other roadmap and audit files.
+
+Required validation:
 
 ```sh
 GOWORK=off go mod tidy -diff
@@ -233,20 +228,13 @@ GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$'
 git diff --check
 ```
 
-Also check formatting with `gofmt -l` on changed Go files. Use
-`scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR`, where
-`OUTPUT_DIR` is an empty absolute directory created with `mktemp -d`, to
-exercise the maintained Linux amd64/arm64 and macOS arm64 build path without
-choosing or publishing a release. Report macOS as cross-build evidence only.
-
-Acceptance: All checks pass without access to an installed Narratio configuration
-or real credentials. The implementation no longer needs evaluation-only
-workarounds. The later release task must create its note, commit/push its candidate,
-and run the normal release-candidate checker; this plan does not claim those
-publication steps are complete.
+Check changed Go files with `gofmt -l`. Run
+`scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR` with an empty absolute
+temporary directory created by `mktemp -d`. All three maintained targets must
+build; report macOS cross-compilation as such, not native runtime validation.
+No live model calls, tag publication, or CI monitoring are required.
 
 ## Open Questions
 
-None. Dependency version, configuration behavior, discovery seam, test scope,
-and validation gates are decided. Release version selection is deliberately
-outside this implementation's scope and does not block the work.
+None. The operator contract, optionality, delivery boundary, redaction scope,
+and resume policy are settled. Release version selection is a separate task.