Plan Notarius configuration profile support

docs/roadmap/feature-request-notarius-config-profile.yml

@@ -0,0 +1,102 @@
+---
+title: Select a Notarius configuration profile through Narratio
+status: proposed
+consumer: rakestraw.dnd Ansible collection and its pipeline role
+
+use_case: >-
+  The Ansible pipeline role deploys Narratio and Notarius as independently
+  configured applications. Operators select Narratio's testing profile to
+  develop prompts and other pipeline assets with inexpensive models while
+  retaining production settings in the base configuration. That selection
+  currently cannot select a Notarius configuration profile, leaving extraction
+  on its production settings.
+
+desired_outcome: >-
+  Narratio's testing overlay can explicitly select Notarius's testing overlay.
+  Notarius owns the resulting model choices and any other testing-specific
+  settings. The operator selects testing once when invoking Narratio, without
+  editing production files or separately invoking Notarius.
+
+rationale: >-
+  Both applications already use ConfigKit configuration overlays. Selecting
+  the downstream overlay preserves each application's configuration ownership
+  and supports future Notarius testing adjustments without adding a Narratio
+  configuration field for every downstream setting. This is explicit profile
+  composition, not a requirement for arbitrary recursive orchestration.
+
+requested_interface:
+  field: pipeline.notarius.config_profile
+  value: Optional Notarius ConfigKit configuration profile name
+  supplied: >-
+    Invoke Notarius with --config-profile and the selected name when extraction
+    executes. The setting is available in Narratio's base configuration and
+    its ordinary configuration overlays.
+  omitted_or_null: >-
+    Do not supply a profile-selection flag. Preserve existing Notarius
+    configuration selection behavior.
+  downstream_semantics: >-
+    Retain Notarius's existing selector semantics, including the default
+    selector for its base configuration. Notarius resolves profiles relative
+    to its own configuration root, not Narratio's root.
+  explicit_selection: >-
+    Do not automatically forward Narratio's profile name. The selected
+    Notarius profile may have the same name or a different name.
+  distinction: >-
+    This selects a ConfigKit configuration overlay through --config-profile,
+    not a PromptKit LLM profile through --llm-profile. The Notarius overlay
+    can itself select the desired LLM profiles using native Notarius settings.
+
+illustrative_consumer_configuration:
+  note: >-
+    These are separate files managed by the Ansible suite, not a proposed
+    combined configuration document. The Notarius pipeline and LLM profile
+    names are illustrative; the referenced LLM profile must be available to
+    Notarius.
+  narratio_profiles_testing_yml: |
+    notarius:
+      config_profile: testing
+  notarius_profiles_testing_yml: |
+    pipelines:
+      dnd-session:
+        llm_profile: gemini-flash-latest
+  operator_workflow: >-
+    Select --profile testing on the ordinary Narratio invocation. Narratio
+    selects Notarius's testing configuration overlay when it runs extraction;
+    other Narratio testing overrides continue to apply normally.
+
+observable_behavior:
+  - >-
+    Notarius owns overlay discovery, composition, validation, and the effective
+    downstream settings. A missing or invalid selected overlay must not silently
+    fall back to production or be reported as successful extraction.
+  - >-
+    Changing the selected Notarius configuration profile must not silently
+    reuse Narratio's completed extraction from the previous selection.
+    Existing downstream invalidation behavior applies when extraction changes.
+  - >-
+    Editing Notarius overlay contents behind an unchanged selector retains
+    the existing operator responsibility to force extraction when needed.
+    This request does not require inspecting or tracking private Notarius files.
+  - >-
+    Existing credential forwarding remains independently configurable through
+    notarius.llm_api_key, including in the Narratio testing overlay when the
+    selected Notarius settings require a different credential reference.
+
+consumer_follow_up: >-
+  After Narratio releases this interface, the standalone Narratio Ansible role
+  can expose an optional variable and omit the field when unset. The suite role
+  can deploy Notarius's testing overlay and select it from Narratio's testing
+  overlay, then distribute the changes through the collection. These consumer
+  updates are separate from this upstream feature request.
+
+scope_boundaries:
+  - No Notarius change is currently required; v0.7.0 supports --config-profile.
+  - No separate Narratio-side Notarius LLM-profile selector is requested.
+  - No automatic profile-name propagation or downstream configuration rewriting.
+  - No new credential discovery, copying, or provider-name remapping.
+  - This request specifies consumer behavior, not internal Narratio code changes.
+
+references:
+  narratio_configuration: ../config.md
+  narratio_notarius_integration: ../integrations/notarius.md
+  notarius_cli: https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.0/docs/cli.md

docs/roadmap/implementation.md

@@ -0,0 +1,248 @@
+# Implementation Plan: Notarius Configuration Profile Selection
+
+Status: Ready for implementation; all stages pending.
+
+## Scope and Working Instructions
+
+Implement the [feature roadmap](notarius-config-profile.md), whose authority is
+the [consumer request](feature-request-notarius-config-profile.yml). Complete
+the numbered stages in order. Each stage is a bounded task for a gpt-5.6-terra
+coding agent. Do not treat completion of an intermediate stage as completion of
+the feature. Record completed stages and actual verification results here.
+
+Before each stage, read `docs/development.md`, all documents under
+`docs/policy/`, the feature roadmap, and that stage's preparation list. Follow
+AGENTS.md discovery guidance. Use existing collaborators and fixtures; do not
+build a new configuration, cache, or profile-management subsystem. Tests must
+remain offline and independent of installed Notarius binaries and credentials.
+Do not change dependencies, implement Ansible changes, tag, or release as part
+of this plan.
+
+## Implementation Decisions
+
+These choices resolve implementation details without adding restrictions to
+the feature request:
+
+- Represent the authored field as `ConfigProfile *string` with YAML/JSON name
+  `config_profile` and `omitempty`, in both typed configuration and its safe
+  YAML projection. Nil means omitted/null. A non-nil pointer preserves an
+  explicitly supplied empty string as well as `default` and named selections.
+- Carry the same optional value in the Notarius adapter request. Nil emits no
+  flag; non-nil emits `--config-profile` followed by the exact supplied string
+  as a separate argument, including an empty string. Do not trim, lowercase,
+  substitute Narratio's profile, impose a name regex, or inspect profile files.
+  Notarius owns validation and selector interpretation.
+- For reuse evidence only, use one small stage-local helper that maps nil,
+  `""`, and exact `"default"` to the empty identity; other values remain
+  unchanged. Both extraction fingerprints use it. This does not mutate the
+  authored field or change CLI forwarding. Do not attempt to recreate the
+  downstream parser's complete normalization rules.
+- Add `ConfigProfile string` with `json:"config_profile,omitempty"` to both
+  existing fingerprint payloads, populated only where extraction is enabled.
+  Preserve existing field ordering and the current semantic fingerprint version
+  (1). With no non-base selector the serialized payload remains unchanged;
+  a named selection adds evidence that old records cannot already possess.
+  No dual-reader, manifest migration, new durable metadata field, or broad
+  fingerprint-version bump is needed. This is a minimal additive encoding,
+  not a permanent promise to freeze fingerprint bytes across future releases.
+- Profile changes trigger the existing non-resumable/invalidation paths, not
+  force mode. Preserve configured resume/cache controls, force precedence,
+  credentials, output contracts, and stage-range behavior. Do not run extraction
+  outside the operator's selected range or add a separate testing workspace.
+- Preserve the existing division of ownership: private downstream file changes
+  behind an unchanged selector require operator-directed force when needed.
+  No downstream configuration validation subprocess or runtime version gate
+  is added.
+
+## Stage 1: Add Configuration and Adapter Support
+
+Status: Pending.
+
+Preparation:
+
+- `docs/config.md` (composition, profiles, Notarius, credentials).
+- `docs/integrations/notarius.md` (invocation and failure behavior).
+- `internal/config/config.go`, `credential_projection.go`,
+  `pipeline_configkit.go`, `pipeline_normalize.go`, and `notarius_test.go`.
+- `internal/adapters/notarius/runner.go`, `subprocess.go`, `fake.go`, and
+  `subprocess_test.go`.
+
+Work:
+
+1. Add the optional field to `NotariusConfig` and the `safeNotarius` projection,
+   including its returned value. Let existing ConfigKit composition handle
+   overrides and null; do not add a default or resolve this name as a path.
+2. Add the optional field to `notarius.RunRequest` and append the argument pair
+   when non-nil in `SubprocessRunner.Run`, before `--json`. Leave subprocess
+   environment construction and error propagation unchanged.
+3. Audit any explicit Notarius configuration/request copies for the new field.
+   Update actual omissions without introducing a generic cloning framework.
+4. Extend real configuration-load tests to cover a base selection, an overlay
+   replacing it with a differently named downstream selection, and an overlay
+   clearing it with null. Cover omitted, empty, and `default` values compactly.
+   Verify the safe effective-YAML output retains a supplied value, including
+   explicit empty, and round-trips without leaking credential values. Reuse
+   existing credential tests rather than duplicating their full matrix.
+5. Extend adapter invocation tests for omitted and supplied selections,
+   including empty and `default`. Assert the argument/value boundary rather
+   than imposing new ordering requirements on unrelated arguments. Reuse the
+   existing nonzero-exit test with a selected profile to prove there is no
+   fallback or successful receipt processing after failure.
+
+Verification:
+
+```sh
+go test ./internal/config ./internal/adapters/notarius
+```
+
+Completion: Configuration composition and safe inspection preserve the field,
+and the adapter forwards it correctly. The executing stage is connected in
+Stage 2 together with reuse tracking, so it cannot start using untracked profile
+selections as an intermediate implementation.
+
+## Stage 2: Wire Extraction and Profile-Sensitive Reuse
+
+Status: Pending.
+
+Preparation:
+
+- `docs/internal/stage-extract.md`.
+- `internal/stage/extract.go`, `extract_resume.go`,
+  `semantic_contracts_delivery.go`, and their focused tests.
+- In particular, `TestExtractStageResolvesManifestInputAndBuildsExactRequest`,
+  `TestExtractSemanticConfigSensitivityAndOperationalExclusions`, and
+  `TestExtractStageResumeValidationRequiresFreshEvidence`.
+
+Work:
+
+1. Pass `cfg.ConfigProfile` to the adapter request when extraction executes.
+   Keep the field independent of credential forwarding and force overrides.
+2. Implement the reuse-identity helper and additive fingerprint fields described
+   above. Update both `extractSemanticConfig` and `fingerprintDocument`, using
+   the same identity in semantic comparison and `extractionFingerprint`.
+   Keep disabled extraction insensitive to unused downstream settings.
+3. Extend the existing stage request test to prove the authored selection
+   reaches the adapter. Cover ordinary and forced execution sufficiently to
+   show that profile forwarding does not alter existing cache-control precedence.
+4. Extend semantic-sensitivity coverage: base to named, named to different named,
+   and named to base change evidence; unchanged selection does not. Verify
+   nil/empty/`default` share base evidence and disabled settings remain inert.
+5. Exercise the focused resume validator with an actual valid recorded bundle:
+   it remains reusable for the same selection and becomes non-resumable for a
+   changed selection. This guards the second fingerprint path independently
+   of the application's semantic comparison.
+6. Check the additive encoding against the prior no-profile payload shape once
+   in focused coverage or a recorded development check. Avoid whole-manifest
+   snapshots and a permanent historical compatibility harness; retain existing
+   input/output/reference sorting and all unrelated fingerprint fields.
+
+Verification:
+
+```sh
+go test ./internal/stage ./internal/adapters/notarius ./internal/config
+```
+
+Completion: Selected profiles reach actual extraction, and both existing reuse
+checks account for the selection without forcing downstream cache refresh or
+changing the manifest schema.
+
+## Stage 3: Verify Application Lifecycle Behavior
+
+Status: Pending.
+
+Preparation:
+
+- `internal/app/extract_lifecycle_test.go`, `runner_test.go`, and
+  `run_control_test.go`.
+- Existing fixtures and tests for changed extraction outcomes, failure/retry,
+  and independent render/extract invalidation.
+
+Work:
+
+1. Extend the existing lifecycle fixtures rather than building another harness.
+   Run a successful extraction with a configured profile, then verify ordinary
+   reuse for that same profile. Change only the downstream selection and verify
+   that extraction executes again with the new value, upstream transcripts and
+   render remain current, and succeeded dependents are invalidated/rerun through
+   the existing runner policy. Include returning from a named selection to base.
+2. Cover the changed-profile failure boundary with the existing fake runner:
+   return an ordinary downstream error and verify extraction is failed, the
+   previous result is not advertised as current success, and dependents are not
+   silently reused. Reuse existing retry coverage where it already proves the
+   recovery behavior. Do not assert Notarius-specific diagnostic wording.
+3. Ensure one configuration-to-execution path proves explicit composition:
+   Narratio's testing overlay can select a differently named Notarius overlay;
+   without the field, selecting a Narratio profile does not populate the
+   downstream request. Use real configuration loading and the existing injected
+   runner, not a production provider or sibling repository dependency.
+4. Review existing operational-exclusion and force tests for continued validity.
+   Do not broaden invalidation to the full effective configuration digest or
+   Narratio's profile label, and do not automatically change cache options.
+
+Verification:
+
+```sh
+go test ./internal/app ./internal/stage
+```
+
+Completion: Tests demonstrate the requested workflow, including selection
+changes and failure, across the real Narratio configuration/runner boundaries.
+Notarius's own overlay loader and cache internals remain outside these tests.
+
+## Stage 4: Document the Feature and Validate the Completed Change
+
+Status: Pending.
+
+Preparation: Canonical documentation owners listed in the feature roadmap,
+`examples/README.md`, and the existing development-focused operations guidance.
+
+Work:
+
+1. Document the optional field in `docs/config.md`, including omitted/null,
+   explicit base selectors, and independence from Narratio's profile and
+   `notarius.llm_api_key`. Do not add profile-name restrictions.
+2. Update the invocation in `docs/integrations/notarius.md` to include the
+   optional flag and identify Notarius v0.7.0 as a known supporting version,
+   not an unconditional version requirement. Preserve the distinction from
+   `--llm-profile` and ordinary downstream failure handling.
+3. Update `docs/internal/stage-extract.md` to describe selection-sensitive reuse
+   and retain its distinction from private file content and operational settings.
+   Make clear this is an explicit downstream setting, not blanket invalidation
+   by Narratio's profile label or effective digest.
+4. Add concise development workflow guidance in `docs/operations.md`: explicit
+   testing-overlay composition; different downstream names are allowed;
+   switching selection triggers extraction when that stage is in scope; edits
+   behind an unchanged selector may need force; testing is not a separate
+   workspace. Preserve the existing distinction between extract+analyze and
+   analyze-only regeneration.
+5. Update the maintained annotated pipeline example to show the optional field
+   without requiring an externally installed testing overlay (use a commented
+   named-selection example). Do not make previously runnable examples depend
+   on new downstream files. Link to canonical owners instead of duplicating
+   full contracts or adding new top-level Narratio flags.
+6. Review the final diff against every observable behavior and scope boundary
+   in the consumer request. Mark the implementation stages and roadmap status
+   accurately after verification; do not remove the original request or alter
+   historical release notes.
+
+Run the repository checks with the maintained Go 1.26 toolchain:
+
+```sh
+go test ./...
+go test -race ./...
+go vet ./...
+go build ./...
+go test ./internal/doccheck
+go test ./internal/config -run '^TestExamplesLoadAndValidate$'
+git diff --check
+```
+
+Completion: The requested interface, failure and reuse behavior, examples, and
+canonical documentation agree; all checks pass. No release is part of this
+implementation sprint.
+
+## Open Questions
+
+None. The consumer request resolves the product behavior. The choices above
+settle implementation details without adding new operator restrictions,
+downstream configuration ownership, or cache guarantees.

docs/roadmap/notarius-config-profile.md

@@ -0,0 +1,151 @@
+# Notarius Configuration Profile Selection
+
+Status: Proposed; not implemented.
+
+## Goal and Authority
+
+Implement the consumer behavior in the
+[feature request](feature-request-notarius-config-profile.yml): let a Narratio
+configuration overlay explicitly select a Notarius configuration overlay.
+Operators can select testing once in Narratio while each application retains
+ownership of its own model choices and other settings.
+
+This roadmap describes the target state and required change areas, not an
+implementation sequence; the [implementation plan](implementation.md) owns
+the ordered work and implementation decisions. The feature request is the scope
+authority; the mechanisms suggested below do not establish additional
+user-facing contracts.
+
+## Underlying Issue
+
+Narratio can select its own testing profile, but extraction currently invokes
+Notarius without a configuration-profile selector. Consequently, a Narratio
+testing invocation cannot explicitly select the independently deployed Notarius
+testing overlay through pipeline configuration.
+
+Adding individual downstream model or tuning fields would duplicate Notarius's
+configuration responsibilities. Passing its existing profile selector provides
+the requested flexibility without that duplication.
+
+## Targeted End State
+
+### Explicit Configuration Composition
+
+`pipeline.notarius.config_profile` is an optional Notarius configuration-profile
+name, usable in Narratio's base configuration and ordinary overlays. For
+example, Narratio's `profiles/testing.yml` may contain:
+
+```yaml
+notarius:
+  config_profile: testing
+```
+
+Selecting Narratio's `--profile testing` then selects that explicit downstream
+profile whenever extraction executes. The two profile names need not match.
+Narratio does not automatically forward its own profile name.
+
+When supplied, the selection is passed through Notarius's `--config-profile`
+argument. Omitted or null values add no profile-selection flag and preserve
+existing downstream selection behavior. Notarius's selector semantics remain
+authoritative, including empty and `default` selections for the composed base
+configuration. Narratio does not introduce a stricter profile-name policy.
+
+This field selects a ConfigKit configuration overlay, not a PromptKit profile
+through `--llm-profile`. The Notarius overlay owns any resulting LLM-profile
+selection. Notarius resolves overlays relative to its own configuration root;
+Narratio neither resolves those files against its own root nor rewrites them.
+
+### Execution, Reuse, and Failure
+
+A changed downstream profile selection must not silently reuse Narratio's
+completed extraction from the previous selection. Existing dependent-stage
+invalidation applies when extraction changes; the feature does not change the
+pipeline order or transcript-generation behavior.
+
+Notarius continues to own downstream overlay discovery, composition, validation,
+and cache compatibility. A missing or invalid selected overlay must fail
+extraction through the existing subprocess error path, without silently falling
+back to production or reporting successful extraction.
+
+Existing resume and chunk-cache controls, including force precedence, remain
+in place. Profile selection does not itself mean forced regeneration or cache
+deletion. Narratio's decision to execute extraction and Notarius's decision to
+reuse compatible work remain separate responsibilities.
+
+Changes to private Notarius configuration behind an unchanged selector retain
+the existing operator responsibility to force extraction when needed. There is
+no new downstream file inspection or content-tracking requirement.
+
+Credential forwarding remains independently configured through
+`notarius.llm_api_key`. A Narratio testing overlay may select a different
+credential reference alongside the downstream profile; no new credential
+discovery or provider-name mapping is introduced.
+
+## Required Change Areas
+
+### Configuration and Inspection
+
+Extend the typed Notarius configuration and its existing safe effective-YAML
+projection to carry the optional field. Preserve the ordinary configuration
+composition behavior for base values, overlay overrides, and null values.
+Configuration inspection must reflect the selected field without exposing
+credential values. No replacement configuration loader or downstream preflight
+loader is needed.
+
+### Stage and Adapter Boundary
+
+Carry the selection from resolved configuration through the extraction request
+to the subprocess adapter. Construct the optional CLI flag using the existing
+argument-vector mechanism, alongside the current configuration path, references,
+credential forwarding, and cache controls. Retain normal Notarius error handling.
+
+### Extraction Reuse Evidence
+
+Account for the downstream selection in the existing extraction reuse evidence.
+The current semantic-configuration and invocation fingerprints are the relevant
+change points. Keep their treatment of selection consistent so that a profile
+change cannot bypass the required rerun and downstream invalidation.
+
+This roadmap does not require a new manifest schema, separate profile-indexed
+workspaces, historical per-profile result retention, or an upgrade-wide forced
+rerun.
+
+### Verification and Documentation
+
+Use focused behavioral coverage for configuration composition and inspection,
+explicit argument forwarding and omission, changed-selection reuse decisions,
+and ordinary subprocess failure handling. Reuse existing test coverage where
+it already protects unchanged credentials, force behavior, and invalidation;
+do not duplicate Notarius's complete profile-validation or cache test suites.
+
+When implemented, update the canonical
+[configuration reference](../config.md),
+[Notarius integration contract](../integrations/notarius.md),
+[extract internals](../internal/stage-extract.md), and relevant maintained
+examples. The development workflow in [Operations](../operations.md) should
+explain explicit profile composition and the existing need to force extraction
+after private downstream edits. Clarify that profile selection does not create
+an isolated testing workspace or separate current extraction state.
+
+## Compatibility and Scope Boundaries
+
+Notarius v0.7.0 already documents `--config-profile`; no upstream change is
+required. Document the capability needed when the option is used, without adding
+an unconditional runtime version gate for existing unconfigured deployments.
+
+This feature adds no Narratio CLI selector, automatic profile propagation,
+Notarius LLM-profile selector, downstream configuration rewriting, new
+credential mechanism, or generalized recursive orchestration. Existing output
+and receipt contracts remain unchanged.
+
+The consumer's Ansible role and suite updates are follow-up work outside this
+repository. They can deploy downstream overlays and populate this field after
+the interface is released.
+
+## Scope Review
+
+No stronger requirements or new invariants are proposed beyond the feature
+request and existing repository contracts. In particular, minimizing one-time
+upgrade invalidation and avoiding reruns between equivalent base selectors are
+implementation considerations, not additional acceptance guarantees imposed
+by this roadmap. No user decisions are currently required before planning.