narratio/docs/roadmap/implementation.md

# Implementation Plan: Audita v2 and Shared Provider Credentials

Status: Complete (2026-09-25).

## Scope and working instructions

Implement [the Audita roadmap](audita.md). The baseline is Narratio commit
`70f1dbf` (v2.2.1), with ConfigKit v0.4.0 and Audita v2.0.0 as the target
upstream contracts. This replaces the completed checkpoint-refactoring plan,
which remains in Git history.

Complete Stage 1 through Stage 8 in order, one bounded implementation prompt per
stage for gpt-5.6-terra. Record completion and actual verification results under
each stage. These instructions apply to the implementation sprint; the planning
pass changes documentation only. Intermediate stages are not release candidates.
Do not add compatibility code to make intermediate states look like a supported
release.

Before each stage, read `AGENTS.md`, `docs/development.md`, and all policies
under `docs/policy/`: agents, architecture, documentation, testing, and release.
Read this plan's selected design and that stage's preparation. Use Go 1.26 and graph-first code
discovery; inspect current files when graph results are incomplete. Preserve
unrelated changes. Test at behavioral owners rather than replicating every case
across configuration, adapter, stage, and command tests.

The user approved a clean cutover: no Audita v1 runner, deprecated aliases,
compatibility shims, generated configuration conversion, or silently ignored
old settings. Do not broaden that decision into a purge of unrelated legacy
code or historical session data.

## Selected design

### ConfigKit and Narratio overlay terminology

Upgrade ConfigKit from v0.3.1 to v0.4.0 before adapter work. Rename its
`Options.Profile`, `LayerProfile`, and any `CodeProfile*` uses to their overlay
equivalents. Selected pipeline files move from `profiles/<name>.yml` to
`overlays/<name>.yml`; no fallback directory or alias is permitted.

Rename Narratio's pipeline-selection flag to `--overlay` everywhere it is
accepted, including convenience commands and configuration inspection. Rename
application-owned selection types/helpers, help, errors, and current inspection
labels consistently. `config diff` still takes two positional selections,
now described as overlays. Preserve omission, explicit empty, and `default`
as base selection; do not add a persistent default selector or new precedence.

Rename newly written informational provenance to overlay terminology, including
`selected_overlay` in manifest configuration provenance. Do not add a dual-field
reader, automatically rewrite historical manifests, or introduce a manifest-version
rejection solely for this informational rename. Existing strict typed decoding
rejects retired provenance fields. Document backed-up operator migration of
that informational member without changing execution/checkpoint state; it does
not become resume authority. Preserve
semantic fingerprints and effective-value digest rules when values are unchanged.

Only Narratio pipeline configuration selection is renamed. Keep model/profile
fields such as Scriptorium `profile_id`, Audita `proposal_profile` and
`validation_profile`, and Notarius `config_profile` unchanged. Explicit imports
from directories named `profiles` remain legal; only selected pipeline overlay
discovery changes. Do not globally replace the word “profile.”

### Configuration and credentials

- Both `pipeline.audita.credentials` and `pipeline.notarius.credentials` are
  lists of ConfigKit secret references. Use `[]configkit.Secret`; each element
  is an actual reference rather than a nullable pointer. Existing ConfigKit
  validation owns name/value resolution. Omitted, null, or empty lists mean no
  additional credential forwarding.
- Define each key once through existing environment/secrets-directory bootstrap.
  Forward it under its original name to each adapter whose list includes it.
  No alias mappings, provider registry, downstream config inspection, extra
  bootstrap, or inheritance of the entire environment.
- Repeated names need no new rejection rule: forwarding the same captured
  reference once suffices. Arbitrary valid variable names remain supported;
  provider names in the roadmap are examples, not an allowlist.
- Add a narrow shared helper in `internal/adapters/subprocess` converting resolved
  references to `EnvOverrides` and `SensitiveEnvNames`. It checks the existing
  resolved/nonempty requirement for direct adapter callers but never resolves
  secrets or rereads environment values. Errors expose no values. Both adapters
  use it; the ConfigKit upgrade is owned by Stage 1, not this helper. No new
  credential service is needed.
- Configuration projections and diagnostic metadata contain reference names
  only. Register all forwarded names for existing captured-stream redaction.
  Downstream tools remain responsible for files they write themselves.
- Remove both singular `llm_api_key` fields. For Audita also remove `base_url`,
  `model`, `validation_model`, `total_llm_concurrency`,
  `proposal_llm_concurrency`, `validation_llm_concurrency`,
  `work_dir_retention`, and `report`, including defaults and projections.
- Retain Audita `binary`, `timeout`, `config_path`, `transcript_description`,
  `output_schema`, and `modules`. Add optional string selectors `overlay`,
  `proposal_profile`, and `validation_profile`: empty/unset means no CLI
  override. Audita owns downstream file/profile validation and native config
  discovery. Narratio overlay names are not implicitly Audita overlay names.
- Preserve nil versus non-nil empty `modules` through loading, projection,
  cloning, composition, adapter defaults, and request overrides. Nil omits the
  flag; empty passes `--modules` with an empty argument; nonempty joins in order
  with repetitions intact. A nil request override uses the configured selection.
  Avoid clones that collapse empty slices to nil. YAML null follows unset
  behavior. Other adapters' module semantics do not change.

### Invocation, reports, and failure

Use `process INPUT --glossary PATH --output PATH` plus selected surviving
overrides. Production passes no reporting or debug switches, including
`--debug`, `--reporting=false`, `--report-dir`, or `--work-dir`.
Remove required `WorkDir` and legacy `ReportPath` request/result plumbing.
Document production external-config defaults without forcibly overriding or
rejecting deliberate external reporting/debug choices.

Replace Audita's long positional config-values constructor with typed runner
configuration. Keep timeout parsing in composition and subprocess cancellation
through the existing runner. No version probe or automatic config-validation
subprocess is required.

The corrected transcript remains the only canonical polish output, with existing
identity, path, validation, and bounded file handling. Discover exactly
`errors.yml`, `recoveries.yml`, and `report.md` beside the requested output.
Represent available paths in a small typed report bundle in `PolishResult`.
Fixed names belong to the adapter; do not parse stderr or recursively scan.

Reports are diagnostics, not a new correctness gate:

- Bound reads of present reports using the existing named 16 MiB report limit;
  retain confined regular-file handling and errors identifying Audita, the file,
  and limit. No new smaller limits or report schema/status/count validation is
  necessary when Narratio only preserves bytes; Markdown needs no parser.
- Missing files are not independently fatal: external configuration can disable
  reports and early failures may produce none. Default-reporting success tests
  must nevertheless demonstrate capture of all three files.
- On process failure, preserve available diagnostics and the original error;
  report-inspection errors must not hide it. Never promote an output merely
  because it exists following a nonzero exit.
- Exit zero with a valid transcript may include recoverable errors or no model
  corrections. Accept it without a fail-on-recovery or minimum-correction rule.

Keep reports next to the run-local transcript, not in a second canonical
destination. Register present report paths in the existing `StageResult.Logs`
diagnostic collection alongside stdout/stderr, with safe bundle-path metadata.
Reuse current manifest-driven diagnostic publication rather than adding a
manifest schema or new publish source IDs. Physical placement does not define
canonical artifact identity. Existing diagnostic retention/cleanup rules apply.

On failure, retain run-local files and expose the output/report directory in
contextual errors. Do not add runner-wide failed-result persistence merely to
register those files. Do not fabricate legacy JSON reports or delete historical
reports. Normal successful stage-record replacement removes obsolete current
output references.

Generated invocation YAML becomes `audita.generated.v2`, recording actual
arguments, selections, paths, and credential names, never values. This remains
a diagnostic invocation record, not an Audita configuration file.

### Resume boundary

Bump only polish semantic-config version from 1 to 2. Record config path,
overlay/proposal/validation selectors, ordered modules with presence preserved,
transcript description, output schema, and existing output identity. Remove old
endpoint/model fields. Existing runner comparison reconsiders obsolete evidence
when polish is selected; do not synthesize equivalent fingerprints or expand
normal dependency/selection invalidation.

Credentials, binary path, timeout, and diagnostics are not semantic keys. Do not
hash downstream config/profile contents; operators force polish after edits
behind unchanged selections. Notarius credential migration does not change
extraction fingerprints, references, force, resume, or chunk-cache behavior.

## Stage 1: Adopt ConfigKit v0.4.0 and pipeline overlays

Status: Complete (2026-09-25).

Verification: `go test ./internal/config ./internal/app ./internal/manifest`,
`go test ./...`, `go test ./internal/doccheck`,
`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and
`git diff --check` passed.

Preparation: ConfigKit v0.4.0 release notes and selected-overlay migration guide
in `../ConfigKit/docs/`; Narratio `internal/config/pipeline_configkit.go`,
`artifact_catalog.go`, `load.go`, `effective_digest.go`; CLI flag parsing and
config commands in `internal/app/`; configuration provenance in
`internal/manifest/manifest.go`; current config/CLI docs and pipeline examples.

1. Update `go.mod`/`go.sum` to ConfigKit v0.4.0 using normal module tooling;
   do not leave a local replacement or change unrelated dependencies.
   Replace removed ConfigKit APIs in loading, ownership, and artifact catalog
   inspection. Account for source layer `overlay` and diagnostic prefix
   `overlay.*` wherever they are consumed or asserted.
2. Rename Narratio's pipeline selector to `--overlay` across all commands and
   aliases, load options, pair loading, inspection, provenance, and tests.
   Preserve explicit-presence handling through convenience commands and existing
   base/default selection. Remove `--profile` rather than retaining an alias.
   Apply the selected design's informational manifest-provenance treatment.
3. Move maintained selected pipeline files and generated fixture paths to
   `overlays/`. Update current canonical documentation, policy terminology,
   examples, and user-visible migration guidance with the directory/flag break.
   Do not rewrite historical release notes, rename downstream model profiles,
   or prohibit ordinary explicit imports from a directory named `profiles`.
4. Adapt existing config/app tests to prove base and named-overlay selection,
   imported defaults plus overrides, secret replacement/removal, source attribution
   through character-family expansion, config diff, and convenient-command
   forwarding. Add focused checks that the retired flag is rejected and the old
   directory is not a discovery fallback. Do not freeze incidental error wording.
5. Verify unchanged effective values retain digest and stage-reuse behavior;
   only diagnostic source labels/paths and selected provenance terminology change.
   Test current provenance round trips and strict rejection of unknown/retired
   provenance fields. Document deliberate operator migration; do not add a
   compatibility decoder, semantic invalidation rule, or schema-version gate.
6. Run focused and full tests before proceeding so failures from the dependency
   and terminology change are separated from adapter migrations. No Audita or
   Notarius credential/CLI behavior changes in this stage.

Verify: `go test ./internal/config ./internal/app ./internal/manifest`,
`go test ./...`, `go test ./internal/doccheck`,
`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and
`git diff --check`.

Done when Narratio uses ConfigKit v0.4.0, exposes pipeline overlays consistently,
has no old selector/directory fallback, and preserves configuration semantics and
downstream model-profile concepts.

## Stage 2: Add shared resolved-credential forwarding

Status: Complete (2026-09-25).

Verification: `go test ./internal/adapters/subprocess`,
`go test -race ./internal/adapters/subprocess`, and `git diff --check` passed.

Preparation: `internal/adapters/subprocess/run.go`, its environment/redaction
helpers and tests; `internal/adapters/notarius/subprocess.go`;
`internal/config/credential_projection.go`; the pinned ConfigKit secret API.

1. Add the narrow helper described above. Return an independent environment map
   and selected sensitive names. No logging/serialization of values and no
   changes to minimal child environment inheritance.
2. Test multiple custom names, empty lists, repeated names, unresolved references,
   and captured values after the parent environment changes. Reuse existing
   split/overlapping-stream redaction coverage rather than repeating its matrix.
3. Leave production callers unchanged until their migrations.

Verify: `go test ./internal/adapters/subprocess` and
`go test -race ./internal/adapters/subprocess`.

Done when both adapters can reuse one small conversion mechanism without adding
a resolver, provider policy, or ambient-secret inheritance.

## Stage 3: Migrate Notarius to credential lists

Status: Complete (2026-09-25).

Verification: `go test ./internal/config ./internal/adapters/notarius ./internal/stage ./internal/app`,
`go test -race ./internal/adapters/notarius`, and `git diff --check` passed.

Preparation: `internal/config/config.go`, `credential_projection.go`, loading/
inspection tests; `internal/adapters/notarius/runner.go`, `subprocess.go`;
`internal/stage/extract.go` and its request construction;
`docs/integrations/notarius.md` and secret handling in `docs/config.md`.

1. Replace the singular field with the typed list across configuration,
   projection, request types, stage wiring, and adapter execution. Use Stage 2's
   helper and register all selected names for redaction.
2. Update name-only metadata and config-show/diff/effective projections. Test
   real ConfigKit loading, overlay list replacement, and empty-list clearing.
   Preserve bootstrap precedence and existing load-time reference resolution,
   including references outside the selected stage range.
3. Migrate maintained examples, fixtures, and current Notarius configuration/
   integration documentation for this field. Ordinary strict decoding rejects
   the old field; do not add a compatibility parser.
4. Cover multi-key forwarding, unselected-key exclusion, empty selection,
   captured-value behavior, and custom-name redaction on success/failure.
   Keep extraction cache/reference/output tests as regression protection,
   not occasions to redesign extraction.

Verify: `go test ./internal/config ./internal/adapters/notarius ./internal/stage ./internal/app`
and `go test -race ./internal/adapters/notarius`.

Done when Notarius uses credential lists end to end and no active singular
credential path remains.

## Stage 4: Cut Audita configuration and adapter over to v2

Status: Complete (2026-09-25).

Verification: `go test ./internal/config ./internal/adapters/audita ./internal/stage ./internal/app`,
`go test -race ./internal/adapters/audita`, `go test ./...`, and
`git diff --check` passed.

Preparation: Audita v2.0.0 `docs/migrations/v2.md`, `docs/cli.md`,
`docs/integrations/subprocess.md`, and `docs/integrations/output-files.md` in
`../audita/`; Narratio `internal/adapters/audita/`,
`internal/app/runner.go` (`buildDefaultAuditaRunner`), and config types,
defaults, validation, and credential projection.

1. Replace the Audita configuration surface as selected above, updating defaults,
   validation, safe projection, examples, and config fixtures. Prove nil/empty
   modules survive real configuration loading and overlay overrides.
2. Migrate runner construction, selectors, and credential forwarding. Remove
   retired flags, fixed credential alias, required work-directory plumbing, and
   old JSON-report validation. Add the typed report bundle.
3. Implement adjacent report discovery, bounded reads, failure results, and
   generated invocation v2. Share metadata construction between success/failure
   instead of maintaining duplicate field maps.
4. Rewrite the Audita subprocess fixture to model v2 argument acceptance and
   output placement. Cover omitted/selected selectors, ordered/repeated/empty
   modules, no report/debug switches, malformed/missing transcript, ordinary
   reports, externally disabled reporting, recoverable-error success, and
   nonzero exit after transcript creation. Cover multiple custom credential names
   and redaction; preserve timeout/cancellation coverage at its existing owner.
5. Make mechanical polish and app-fixture adjustments needed to compile: remove
   obsolete request/metadata fields and old report materialization. Stage 5 owns
   complete diagnostic integration. Remove retired fingerprint members as needed
   to compile; Stage 6 completes the replacement semantic contract. These are
   bounded call-site changes, not permission to introduce transitional shims.

Verify: `go test ./internal/config ./internal/adapters/audita ./internal/stage ./internal/app`
and `go test -race ./internal/adapters/audita`.

Done when configuration composes a v2-only adapter and subprocess tests prove its
credential/transcript/report boundary. Stages 5 and 6 complete diagnostic and
semantic integration; all stages are required before release.

## Stage 5: Integrate polish reports with diagnostic lifecycle

Status: Complete (2026-09-25).

Verification: `go test ./internal/stage ./internal/app ./internal/manifest ./internal/artifacts`,
`go test -race ./internal/stage ./internal/app`, and `git diff --check` passed.

Preparation: `internal/stage/polish.go`, `polish_test.go`, `run_local.go`,
`stage.go`, `publish_sources.go`; result application in `internal/app/runner.go`;
publish/restore diagnostic tests; `docs/internal/stage-polish.md`.

1. Finish stage integration: materialize only the canonical polished transcript;
   register available adjacent reports as diagnostics plus safe metadata and
   existing logs/generated invocation references. Remove old `audita_report`
   production and stale stage metadata.
2. Use existing authoritative-path checks for the requested transcript and fixed
   adjacent report paths. Do not accept arbitrary fake-returned paths, scan
   directories, or copy reports to a second destination. Preserve workspace
   handling/modes.
3. On execution failure, preserve canonical transcript authority and run-local
   diagnostics, with actionable directory/log context and original error identity.
   Test failure after output creation.
4. Adapt the fake and app helper. With real manifest/store collaborators, prove
   successful promotion, report references, absence of report canonical outputs,
   and current-run diagnostic publication. Preserve restore's existing scope of
   declared committed transcript/artifact objects: run-archive diagnostics do not
   become automatically restored canonical objects. Test that report references
   do not create new required canonical restore targets. Document remote diagnostic
   inspection separately; no new diagnostic-restore feature is authorized.
5. Leave historical files untouched. Update tests expecting a second canonical
   polish output without introducing a manifest-format migration.

Verify: `go test ./internal/stage ./internal/app ./internal/manifest ./internal/artifacts`
and `go test -race ./internal/stage ./internal/app`.

Done when reports use existing diagnostic lifecycle and failed execution cannot
promote an incidental transcript.

## Stage 6: Complete semantic evidence and resume behavior

Status: Complete (2026-09-25).

Verification: `go test ./internal/stage ./internal/app`,
`go test -race ./internal/stage ./internal/app`, and `git diff --check` passed.

Preparation: `internal/stage/semantic_contracts_refinement.go`, semantic
fingerprint tests, app runner semantic comparison tests, and extraction semantic
configuration tests (locate their current owners through graph discovery).

1. Implement polish evidence version 2 and the selected payload. Remove sorting;
   preserve nil/empty distinction in payload encoding and copies.
2. Test changes from module order/repetition/presence and external selectors.
   Credentials and execution-only settings do not change evidence. Avoid
   assertions on incidental digest literals.
3. Exercise old-version evidence through runner resume behavior: selected polish
   is reconsidered and normal dependent invalidation applies. Downstream-only
   bounded execution does not unexpectedly invoke polish. Unchanged new evidence
   reuses valid completed output.
4. Confirm credential-list changes leave extraction semantic/cache behavior
   unchanged. No downstream file-content scanning or eager fingerprint migration.

Verify: `go test ./internal/stage ./internal/app` and
`go test -race ./internal/stage ./internal/app`.

Done when selections affect reuse accurately without blanket credential/config
invalidation or changes to unselected-stage execution.

## Stage 7: Complete operator migration documentation and examples

Status: Complete (2026-09-25).

Verification: `go test ./internal/doccheck`,
`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and
`git diff --check` passed.

Preparation: documentation policy; `docs/config.md`, `docs/operations.md`,
`docs/troubleshooting.md`, both integration guides,
`docs/internal/adapters.md`, `docs/internal/stage-polish.md`, and pipeline examples.

1. Complete current-behavior documentation as implementation lands. Configuration
   owns fields/defaults; integrations own external contracts; operations owns
   placement and migration; internal docs explain ownership and link onward.
2. Provide an old-to-new migration table: endpoints/capacity move to Audita
   backends, models to profiles, role selection to selectors, and credentials to
   provider-named lists. Remove report/retention settings. Explain Audita's
   all-configured-backend credential resolution and Notarius's profile lookup.
3. Describe preparing and validating Audita v2 config offline with its documented
   command before switching executables. Link upstream examples rather than
   building a downstream configuration generator. Explain ordinary reports,
   recoverable-error success, and force after edits behind unchanged selectors.
4. Production examples omit modules when upstream defaults are intended;
   `modules: []` deliberately disables correction modules. Production external
   Audita config keeps reports enabled/debug disabled, without Narratio flag
   overrides. Testing-profile debug integration remains deferred.
5. Document the configuration-breaking clean cutover for release-version
   consideration. Do not edit historical release records, tag a release, or
   remove unrelated roadmaps.

Verify: `go test ./internal/doccheck`,
`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and
`git diff --check`.

Done when operators can migrate using valid examples without obsolete fields,
implicit credential aliases, or retired CLI/report expectations.

## Stage 8: Verify assembled behavior and audit scope

Status: Complete (2026-09-25).

Verification: the offline app-level shared/Audita-only credential scenario
passed with config bootstrap, both subprocess fixtures, safe inspection/logging,
and ordinary polish reports. The tagged Audita v2.0.0 CLI validated its minimal
configuration and produced a transcript plus all three reports with an empty
module sequence and no provider request. Active-reference and scope audits found
no removed production fields or compatibility path; remaining old names occur in
migration guidance, explicit-rejection checks, or historical fixtures. `go test
./...`, `go test -race ./...`, `go vet ./...`, `go build ./...`, `go test
./internal/doccheck`, `go test ./internal/config -run
'^TestExamplesLoadAndValidate$'`, and `git diff --check` passed.

Preparation: cumulative diff, this plan, feature roadmap, app configuration/
subprocess integration tests, and all policies.

1. Adapt or add one representative offline app-level scenario showing one
   provider-named secret source supplying both adapters and another key selected
   only for Audita. Use real config/bootstrap/composition with subprocess fixtures.
   Prove safe inspection/logging and successful polish reports. Leave exhaustive
   flag/redaction cases at lower layers.
2. Where the tagged Audita checkout is available, smoke-test CLI acceptance and
   ordinary transcript/reports offline with an explicit minimal v2 config,
   unauthenticated backend/profile fixtures, and an empty module sequence.
   No provider requests or production credentials. This is optional local evidence,
   not a default test dependent on sibling checkouts or installed tools.
3. Audit active references to removed fields/flags/aliases; historical release
   notes and completed roadmaps are not compatibility code. Review tests
   collectively for duplication while retaining consequential failure coverage.
4. Confirm no provider-name restrictions, duplicate-name rejection, downstream
   content hashing, fail-on-recovery rule, mandatory explicit config path, debug
   requirement, live-service gate, pipeline-order change, or Notarius cache change
   slipped in. Do not require missing reports when disabled externally. Surface
   additional policy choices rather than encoding them silently in tests.
5. Run every gate below and record actual results. Mark complete only after all
   stages and gates pass. No release, tag, push, or unrelated cleanup in this plan.

```sh
go test ./...
go test -race ./...
go vet ./...
go build ./...
go test ./internal/doccheck
go test ./internal/config -run '^TestExamplesLoadAndValidate$'
git diff --check
```

Done when the assembled v2 integration and shared credentials meet the roadmap,
verification passes, and no compatibility machinery remains within scope.

## Open Questions

None requiring a product decision before implementation. The user settled the
clean cutover and shared provider-named lists. Reports use existing diagnostics
without new content-based success criteria; debug-profile work remains deferred.
Release version selection belongs to the later release procedure.