narratio/docs/roadmap/implementation.md

Implementation Plan: Audita v2 and Shared Provider Credentials

Status: Not started.

Scope and working instructions

Implement the Audita roadmap. The baseline is Narratio commit 70f1dbf (v2.2.1), with Audita v2.0.0 as the upstream contract. This replaces the completed checkpoint-refactoring plan, which remains in Git history.

Complete Stage 1 through Stage 7 in order, one bounded implementation prompt per stage for gpt-5.6-terra. Record completion and actual verification results under each stage. These instructions apply to the implementation sprint; the planning pass changes documentation only. Intermediate stages are not release candidates. Do not add compatibility code to make intermediate states look like a supported release.

Before each stage, read AGENTS.md, docs/development.md, and all three policies under docs/policy/: architecture, documentation, and testing. Read this plan's selected design and that stage's preparation. Use Go 1.26 and graph-first code discovery; inspect current files when graph results are incomplete. Preserve unrelated changes. Test at behavioral owners rather than replicating every case across configuration, adapter, stage, and command tests.

The user approved a clean cutover: no Audita v1 runner, deprecated aliases, compatibility shims, generated configuration conversion, or silently ignored old settings. Do not broaden that decision into a purge of unrelated legacy code or historical session data.

Selected design

Configuration and credentials

• Both pipeline.audita.credentials and pipeline.notarius.credentials are lists of ConfigKit secret references. Use []configkit.Secret; each element is an actual reference rather than a nullable pointer. Existing ConfigKit validation owns name/value resolution. Omitted, null, or empty lists mean no additional credential forwarding.
• Define each key once through existing environment/secrets-directory bootstrap. Forward it under its original name to each adapter whose list includes it. No alias mappings, provider registry, downstream config inspection, extra bootstrap, or inheritance of the entire environment.
• Repeated names need no new rejection rule: forwarding the same captured reference once suffices. Arbitrary valid variable names remain supported; provider names in the roadmap are examples, not an allowlist.
• Add a narrow shared helper in internal/adapters/subprocess converting resolved references to EnvOverrides and SensitiveEnvNames. It checks the existing resolved/nonempty requirement for direct adapter callers but never resolves secrets or rereads environment values. Errors expose no values. Both adapters use it; no dependency upgrade or credential service is needed.
• Configuration projections and diagnostic metadata contain reference names only. Register all forwarded names for existing captured-stream redaction. Downstream tools remain responsible for files they write themselves.
• Remove both singular llm_api_key fields. For Audita also remove base_url, model, validation_model, total_llm_concurrency, proposal_llm_concurrency, validation_llm_concurrency, work_dir_retention, and report, including defaults and projections.
• Retain Audita binary, timeout, config_path, transcript_description, output_schema, and modules. Add optional string selectors overlay, proposal_profile, and validation_profile: empty/unset means no CLI override. Audita owns downstream file/profile validation and native config discovery. Narratio profile names are not implicitly Audita overlay names.
• Preserve nil versus non-nil empty modules through loading, projection, cloning, composition, adapter defaults, and request overrides. Nil omits the flag; empty passes --modules with an empty argument; nonempty joins in order with repetitions intact. A nil request override uses the configured selection. Avoid clones that collapse empty slices to nil. YAML null follows unset behavior. Other adapters' module semantics do not change.

Invocation, reports, and failure

Use process INPUT --glossary PATH --output PATH plus selected surviving overrides. Production passes no reporting or debug switches, including --debug, --reporting=false, --report-dir, or --work-dir. Remove required WorkDir and legacy ReportPath request/result plumbing. Document production external-config defaults without forcibly overriding or rejecting deliberate external reporting/debug choices.

Replace Audita's long positional config-values constructor with typed runner configuration. Keep timeout parsing in composition and subprocess cancellation through the existing runner. No version probe or automatic config-validation subprocess is required.

The corrected transcript remains the only canonical polish output, with existing identity, path, validation, and bounded file handling. Discover exactly errors.yml, recoveries.yml, and report.md beside the requested output. Represent available paths in a small typed report bundle in PolishResult. Fixed names belong to the adapter; do not parse stderr or recursively scan.

Reports are diagnostics, not a new correctness gate:

• Bound reads of present reports using the existing named 16 MiB report limit; retain confined regular-file handling and errors identifying Audita, the file, and limit. No new smaller limits or report schema/status/count validation is necessary when Narratio only preserves bytes; Markdown needs no parser.
• Missing files are not independently fatal: external configuration can disable reports and early failures may produce none. Default-reporting success tests must nevertheless demonstrate capture of all three files.
• On process failure, preserve available diagnostics and the original error; report-inspection errors must not hide it. Never promote an output merely because it exists following a nonzero exit.
• Exit zero with a valid transcript may include recoverable errors or no model corrections. Accept it without a fail-on-recovery or minimum-correction rule.

Keep reports next to the run-local transcript, not in a second canonical destination. Register present report paths in the existing StageResult.Logs diagnostic collection alongside stdout/stderr, with safe bundle-path metadata. Reuse current manifest-driven diagnostic publication rather than adding a manifest schema or new publish source IDs. Physical placement does not define canonical artifact identity. Existing diagnostic retention/cleanup rules apply.

On failure, retain run-local files and expose the output/report directory in contextual errors. Do not add runner-wide failed-result persistence merely to register those files. Do not fabricate legacy JSON reports or delete historical reports. Normal successful stage-record replacement removes obsolete current output references.

Generated invocation YAML becomes audita.generated.v2, recording actual arguments, selections, paths, and credential names, never values. This remains a diagnostic invocation record, not an Audita configuration file.

Resume boundary

Bump only polish semantic-config version from 1 to 2. Record config path, overlay/proposal/validation selectors, ordered modules with presence preserved, transcript description, output schema, and existing output identity. Remove old endpoint/model fields. Existing runner comparison reconsiders obsolete evidence when polish is selected; do not synthesize equivalent fingerprints or expand normal dependency/selection invalidation.

Credentials, binary path, timeout, and diagnostics are not semantic keys. Do not hash downstream config/profile contents; operators force polish after edits behind unchanged selections. Notarius credential migration does not change extraction fingerprints, references, force, resume, or chunk-cache behavior.

Stage 1: Add shared resolved-credential forwarding

Status: Not started.

Preparation: internal/adapters/subprocess/run.go, its environment/redaction helpers and tests; internal/adapters/notarius/subprocess.go; internal/config/credential_projection.go; the pinned ConfigKit secret API.

1. Add the narrow helper described above. Return an independent environment map and selected sensitive names. No logging/serialization of values and no changes to minimal child environment inheritance.
2. Test multiple custom names, empty lists, repeated names, unresolved references, and captured values after the parent environment changes. Reuse existing split/overlapping-stream redaction coverage rather than repeating its matrix.
3. Leave production callers unchanged until their migrations.

Verify: go test ./internal/adapters/subprocess and go test -race ./internal/adapters/subprocess.

Done when both adapters can reuse one small conversion mechanism without adding a resolver, provider policy, or ambient-secret inheritance.

Stage 2: Migrate Notarius to credential lists

Status: Not started.

Preparation: internal/config/config.go, credential_projection.go, loading/ inspection tests; internal/adapters/notarius/runner.go, subprocess.go; internal/stage/extract.go and its request construction; docs/integrations/notarius.md and secret handling in docs/config.md.

1. Replace the singular field with the typed list across configuration, projection, request types, stage wiring, and adapter execution. Use Stage 1's helper and register all selected names for redaction.
2. Update name-only metadata and config-show/diff/effective projections. Test real ConfigKit loading, profile list replacement, and empty-list clearing. Preserve bootstrap precedence and existing load-time reference resolution, including references outside the selected stage range.
3. Migrate maintained examples, fixtures, and current Notarius configuration/ integration documentation for this field. Ordinary strict decoding rejects the old field; do not add a compatibility parser.
4. Cover multi-key forwarding, unselected-key exclusion, empty selection, captured-value behavior, and custom-name redaction on success/failure. Keep extraction cache/reference/output tests as regression protection, not occasions to redesign extraction.

Verify: go test ./internal/config ./internal/adapters/notarius ./internal/stage ./internal/app and go test -race ./internal/adapters/notarius.

Done when Notarius uses credential lists end to end and no active singular credential path remains.

Stage 3: Cut Audita configuration and adapter over to v2

Status: Not started.

Preparation: Audita v2.0.0 docs/migrations/v2.md, docs/cli.md, docs/integrations/subprocess.md, and docs/integrations/output-files.md in ../audita/; Narratio internal/adapters/audita/, internal/app/runner.go (buildDefaultAuditaRunner), and config types, defaults, validation, and credential projection.

1. Replace the Audita configuration surface as selected above, updating defaults, validation, safe projection, examples, and config fixtures. Prove nil/empty modules survive real configuration loading and profile overrides.
2. Migrate runner construction, selectors, and credential forwarding. Remove retired flags, fixed credential alias, required work-directory plumbing, and old JSON-report validation. Add the typed report bundle.
3. Implement adjacent report discovery, bounded reads, failure results, and generated invocation v2. Share metadata construction between success/failure instead of maintaining duplicate field maps.
4. Rewrite the Audita subprocess fixture to model v2 argument acceptance and output placement. Cover omitted/selected selectors, ordered/repeated/empty modules, no report/debug switches, malformed/missing transcript, ordinary reports, externally disabled reporting, recoverable-error success, and nonzero exit after transcript creation. Cover multiple custom credential names and redaction; preserve timeout/cancellation coverage at its existing owner.
5. Make mechanical polish and app-fixture adjustments needed to compile: remove obsolete request/metadata fields and old report materialization. Stage 4 owns complete diagnostic integration. Remove retired fingerprint members as needed to compile; Stage 5 completes the replacement semantic contract. These are bounded call-site changes, not permission to introduce transitional shims.

Verify: go test ./internal/config ./internal/adapters/audita ./internal/stage ./internal/app and go test -race ./internal/adapters/audita.

Done when configuration composes a v2-only adapter and subprocess tests prove its credential/transcript/report boundary. Stages 4 and 5 are required before release.

Stage 4: Integrate polish reports with diagnostic lifecycle

Status: Not started.

Preparation: internal/stage/polish.go, polish_test.go, run_local.go, stage.go, publish_sources.go; result application in internal/app/runner.go; publish/restore diagnostic tests; docs/internal/stage-polish.md.

1. Finish stage integration: materialize only the canonical polished transcript; register available adjacent reports as diagnostics plus safe metadata and existing logs/generated invocation references. Remove old audita_report production and stale stage metadata.
2. Use existing authoritative-path checks for the requested transcript and fixed adjacent report paths. Do not accept arbitrary fake-returned paths, scan directories, or copy reports to a second destination. Preserve workspace handling/modes.
3. On execution failure, preserve canonical transcript authority and run-local diagnostics, with actionable directory/log context and original error identity. Test failure after output creation.
4. Adapt the fake and app helper. With real manifest/store collaborators, prove successful promotion, report references, absence of report canonical outputs, and current-run diagnostic publication. Preserve restore's existing scope of declared committed transcript/artifact objects: run-archive diagnostics do not become automatically restored canonical objects. Test that report references do not create new required canonical restore targets. Document remote diagnostic inspection separately; no new diagnostic-restore feature is authorized.
5. Leave historical files untouched. Update tests expecting a second canonical polish output without introducing a manifest-format migration.

Verify: go test ./internal/stage ./internal/app ./internal/manifest ./internal/artifacts and go test -race ./internal/stage ./internal/app.

Done when reports use existing diagnostic lifecycle and failed execution cannot promote an incidental transcript.

Stage 5: Complete semantic evidence and resume behavior

Status: Not started.

Preparation: internal/stage/semantic_contracts_refinement.go, semantic fingerprint tests, app runner semantic comparison tests, and extraction semantic configuration tests (locate their current owners through graph discovery).

1. Implement polish evidence version 2 and the selected payload. Remove sorting; preserve nil/empty distinction in payload encoding and copies.
2. Test changes from module order/repetition/presence and external selectors. Credentials and execution-only settings do not change evidence. Avoid assertions on incidental digest literals.
3. Exercise old-version evidence through runner resume behavior: selected polish is reconsidered and normal dependent invalidation applies. Downstream-only bounded execution does not unexpectedly invoke polish. Unchanged new evidence reuses valid completed output.
4. Confirm credential-list changes leave extraction semantic/cache behavior unchanged. No downstream file-content scanning or eager fingerprint migration.

Verify: go test ./internal/stage ./internal/app and go test -race ./internal/stage ./internal/app.

Done when selections affect reuse accurately without blanket credential/config invalidation or changes to unselected-stage execution.

Stage 6: Complete operator migration documentation and examples

Status: Not started.

Preparation: documentation policy; docs/config.md, docs/operations.md, docs/troubleshooting.md, both integration guides, docs/internal/adapters.md, docs/internal/stage-polish.md, and pipeline examples.

1. Complete current-behavior documentation as implementation lands. Configuration owns fields/defaults; integrations own external contracts; operations owns placement and migration; internal docs explain ownership and link onward.
2. Provide an old-to-new migration table: endpoints/capacity move to Audita backends, models to profiles, role selection to selectors, and credentials to provider-named lists. Remove report/retention settings. Explain Audita's all-configured-backend credential resolution and Notarius's profile lookup.
3. Describe preparing and validating Audita v2 config offline with its documented command before switching executables. Link upstream examples rather than building a downstream configuration generator. Explain ordinary reports, recoverable-error success, and force after edits behind unchanged selectors.
4. Production examples omit modules when upstream defaults are intended; modules: [] deliberately disables correction modules. Production external Audita config keeps reports enabled/debug disabled, without Narratio flag overrides. Testing-profile debug integration remains deferred.
5. Document the configuration-breaking clean cutover for release-version consideration. Do not edit historical release records, tag a release, or remove unrelated roadmaps.

Verify: go test ./internal/doccheck, go test ./internal/config -run '^TestExamplesLoadAndValidate$', and git diff --check.

Done when operators can migrate using valid examples without obsolete fields, implicit credential aliases, or retired CLI/report expectations.

Stage 7: Verify assembled behavior and audit scope

Status: Not started.

Preparation: cumulative diff, this plan, feature roadmap, app configuration/ subprocess integration tests, and all policies.

1. Adapt or add one representative offline app-level scenario showing one provider-named secret source supplying both adapters and another key selected only for Audita. Use real config/bootstrap/composition with subprocess fixtures. Prove safe inspection/logging and successful polish reports. Leave exhaustive flag/redaction cases at lower layers.
2. Where the tagged Audita checkout is available, smoke-test CLI acceptance and ordinary transcript/reports offline with an explicit minimal v2 config, unauthenticated backend/profile fixtures, and an empty module sequence. No provider requests or production credentials. This is optional local evidence, not a default test dependent on sibling checkouts or installed tools.
3. Audit active references to removed fields/flags/aliases; historical release notes and completed roadmaps are not compatibility code. Review tests collectively for duplication while retaining consequential failure coverage.
4. Confirm no provider-name restrictions, duplicate-name rejection, downstream content hashing, fail-on-recovery rule, mandatory explicit config path, debug requirement, live-service gate, pipeline-order change, or Notarius cache change slipped in. Do not require missing reports when disabled externally. Surface additional policy choices rather than encoding them silently in tests.
5. Run every gate below and record actual results. Mark complete only after all stages and gates pass. No release, tag, push, or unrelated cleanup in this plan.

go test ./...
go test -race ./...
go vet ./...
go build ./...
go test ./internal/doccheck
go test ./internal/config -run '^TestExamplesLoadAndValidate$'
git diff --check

Done when the assembled v2 integration and shared credentials meet the roadmap, verification passes, and no compatibility machinery remains within scope.

Open Questions

None requiring a product decision before implementation. The user settled the clean cutover and shared provider-named lists. Reports use existing diagnostics without new content-based success criteria; debug-profile work remains deferred. Release version selection belongs to the later release procedure.