Prepare the coordinated suite upgrade sprint

docs/roadmap/features/0022-coordinated-suite-upgrade.md

@@ -0,0 +1,247 @@
+# Feature 0022: Integrate the Reviewed Application Upgrades
+
+Status: ready
+
+## Summary
+
+Bring the four reviewed standalone upgrades into the D&D pipeline role,
+collection, and deployment consumer as one compatible configuration set.
+This completes the agreed
+[application upgrade sequence](../application-upgrade-sequence.md).
+
+## Problem or Opportunity
+
+The standalone roles now support the new releases, but the suite intentionally
+retains its previous pins and configuration. Audita still receives v1 settings;
+Narratio and Notarius testing selection still uses retired profile terminology.
+Updating executable versions alone would break the deployed configuration.
+
+## Scope
+
+- Adopt the reviewed application/role versions together.
+- Provide Audita v2 backend configuration and suite-owned PromptKit profiles.
+- Migrate Narratio credential forwarding and Narratio/Notarius testing overlays.
+- Preserve D&D processing, artifact composition, storage/publishing, shared
+  filesystem, and operator-access behavior.
+- Update the collection snapshot, public dependency manifests, consumer pins,
+  operator guidance, and focused offline integration coverage.
+
+## Non-goals
+
+- Deploying to the operational controller or `pipeline-dnd` host, running real
+  transcripts, or calling inference, WhisperX, or object-storage services.
+- New models, prompts, artifact families, extraction lanes, or processing policy
+  beyond migration choices expressly approved here.
+- Automatic manifest conversion, cache deletion, forced stage reruns, broad
+  directory cleanup, compatibility aliases, or runtime version/integrity probes.
+- Changing generic standalone-role ownership or adding production validation.
+- A newly tagged collection release or replacement of published 0.1.0 assets.
+  This feature updates the public development compatibility set; formal release
+  packaging remains a separate decision.
+
+## Target End State
+
+### Coordinated Versions and Distribution
+
+| Application | Suite version | Reviewed standalone role revision |
+| --- | --- | --- |
+| Scriptorium | v0.15.0 | `b174c8c` |
+| Audita | v2.0.1 | `c3dd59f` |
+| Notarius | v0.8.1 | `6aef48b` |
+| Narratio | v3.0.0 | `3c0700b` |
+| Seriatim | v1.5.0, unchanged | existing v0.1.0 role pin |
+
+Dependency manifests use full immutable revisions resolved from these reviewed
+commits. The pipeline role remains authoritative; the collection imports its
+completed revision through the established public Gitea subtree workflow, never
+by editing `roles/pipeline` directly. The collection continues bundling the
+pipeline role and declaring standalone role dependencies separately.
+
+Relevant reviewed commits must be published to their existing public origins
+before downstream manifests/subtree imports depend on them. Publication of the
+required Git commits is part of the eventual implementation workflow, not this
+roadmap-writing pass. Do not rewrite tags or publish unrelated changes.
+
+The consumer keeps its minimal playbook and host-vars organization. Its role
+and collection pins select the same compatibility set. No controller install
+or remote playbook execution is an acceptance gate.
+
+### Audita Backend and Profile Ownership
+
+The pipeline role supplies `audita_backends`, profile selection, and schema-2
+settings to the standalone Audita role. The pipeline owns the missing profile
+assets under `/usr/local/etc/audita/profiles/`, using its runtime owner/group and
+configuration modes. This directory contains PromptKit LLM profiles, not
+ConfigKit overlays; it must not be renamed to `overlays`.
+
+Approved suite interface:
+
+- `dnd_pipeline_audita_backends`: native backend map, defaulting to one backend
+  named `dnd`, endpoint `https://inference.ai.rakestrawhome.com/v1`, credential
+  reference `AUDITA_LLM_API_KEY`, and shared concurrency `4`. Omit queue capacity
+  to inherit the native default.
+- `dnd_pipeline_audita_profiles`: a native profile mapping keyed by file/profile
+  ID, with complete `dnd-proposal` and `dnd-validation` entries. Each selects
+  backend `dnd`, model `google/gemma-4-31b-it`, and per-attempt
+  `timeout_seconds: 900`. Other model controls inherit native behavior.
+- `dnd_pipeline_audita_llm_proposal_profile` and
+  `dnd_pipeline_audita_llm_validation_profile` select those respective IDs.
+
+Render each supplied profile as one safely serialized native mapping with its
+matching ID. Document profile-map keys as filename-safe IDs. Do not introduce
+per-provider templates, profile merging, or discovery of external assets.
+Callers can compose native maps in inventory and replace the supplied maps.
+As with existing managed secrets, removing a map entry does not imply pruning
+arbitrary files; only selected profiles participate in normal execution.
+
+Preserve the ordered module sequence, `audita-v1` output, chunking values,
+thresholds, context description, and normalization limits. Convert normalization
+duration defaults from `3s`, `3s`, and `240s` to numeric `3.0`, `3.0`, and
+`240.0` seconds. Preserve meaningful setting names where native semantics are
+unchanged; update argument types accordingly.
+
+Remove retired per-role endpoint/model/credential/concurrency/retry variables
+in favor of the native backend/profile interface. Document their replacements;
+do not infer a conversion from arbitrary old inventory. Remove diagnostics
+retention; leave debug disabled through native defaults. Retain the existing
+diagnostics work-directory choice where still supported. Reporting follows
+Audita's native configuration; Narratio consumes its ordinary output contract.
+
+Production and testing continue using the same Audita endpoint and model.
+No separate Audita testing overlay is needed for an identical configuration.
+
+### Credential Lists and Overlay Selection
+
+Replace the suite's singular Narratio Notarius credential variable with
+`dnd_pipeline_narratio_notarius_credentials`, default `[OPENROUTER_API_KEY]`.
+Add `dnd_pipeline_narratio_audita_credentials`, default `[AUDITA_LLM_API_KEY]`.
+Pass both through the reviewed Narratio role's native list variables. These
+lists contain names, never values, and can be replaced with `[]`. Operators
+overriding Audita backends must explicitly list all referenced credentials;
+the suite does not inspect configurations to discover names.
+
+Preserve `dnd_pipeline_secrets`, existing secret names, Vault references,
+protected secret directory/files, and ordinary world-readable nonsecret
+configuration. No credential rotation, secret relocation, or forwarding logic
+is required in Ansible.
+
+Deploy the suite-owned testing files at:
+
+- `/usr/local/etc/narratio/overlays/testing.yml`
+- `/usr/local/etc/notarius/overlays/testing.yml`
+
+Narratio testing sets `notarius.config_overlay: testing`; production omits the
+selector. Notarius remains enabled in both modes. Its production pipeline
+continues selecting `gpt-terra-latest`, testing selects `gpt-luna-latest`, and
+the existing caller-overridable pipeline map/ID behavior remains intact.
+Scriptorium testing still selects `gemini-flash-latest`; production continues
+using the prompt assets' selections. Retain separate artifact/family fragments.
+
+Use Narratio `--overlay testing` and Notarius `--config-overlay testing` in
+tests and operator examples. Deploy the new files declaratively; leave ignored
+legacy `profiles/testing.yml` files alone and document that they are obsolete.
+Do not delete `profiles/` directories or conflate them with PromptKit profiles.
+
+### Runtime Paths and Existing Suite Behavior
+
+Keep Narratio's explicit downstream binaries/config paths, output handoffs,
+Notarius `/var/cache/notarius` roots and resume behavior, shared writable
+directories/ACLs, service user, operator enrollment, and S3/publishing choices.
+Audita profiles are configuration assets, not shared output directories.
+No new runtime storage architecture is required.
+
+### Retry and Timeout Policy
+
+Retain the previously selected per-attempt Audita profile timeouts of 900
+seconds while inheriting the new applications' retry,
+repair, and whole-operation defaults. Expose nullable suite pass-throughs for
+`audita_llm_transient_retries`, `audita_llm_repair_attempts`,
+`audita_llm_operation_timeout`, `notarius_promptkit_transient_retries`, and
+`scriptorium_transient_retries`, prefixed with `dnd_pipeline_` in the public
+suite interface. Null means omit; explicit zero is preserved where valid.
+
+Do not mechanically translate old `max_retries: 3` into both transport retries
+and structural repairs. Preserve explicitly authored Notarius module retries;
+they are a separate behavior. Keep Narratio stage timeouts unchanged. Explain
+the nested budgets and the changed semantics, not a guarantee of identical
+provider cost or timing.
+
+## Repository Ownership
+
+- `ansible-role-dnd-pipeline`: suite defaults, argument specs, dependency wiring,
+  Audita profile assets, testing overlays, documentation, and integration tests.
+- `ansible-collection-dnd`: imported pipeline snapshot, dependency manifest,
+  collection tests, and compatibility guidance.
+- `rakestraw-ansible`: matching dependency pins and deployment guidance; adjust
+  host variables only where required. Preserve Vault content and unrelated
+  inventory. Keep the hosts-and-roles playbook minimal.
+- Standalone role repositories: reviewed commits are reused/published, not
+  redesigned. Application and campaign/prompt repositories are read-only
+  contracts/assets unless a concrete incompatibility is raised for discussion.
+- Workspace: roadmap, later staged plan, decisions, progress, and review.
+
+## Constraints and Settled Decisions
+
+Follow the [Ansible](../../ansible.md), [testing](../../testing.md), and
+[planning](../../planning.md) policies, [MVP roadmap](../mvp.md), and
+[upgrade sequence](../application-upgrade-sequence.md). New D&D policy remains
+in the pipeline role; optional generic application defaults remain null.
+
+## Simplest Viable Implementation
+
+Update dependency variables/pins and existing templates, add an ordinary
+Audita profile directory task and one looped template deployment, and change
+testing destinations to `overlays/`. Reuse existing declarative convergence,
+secret delivery, and collection assembly. No production validation, state
+fingerprints, migration framework, or generated credential inference is needed.
+
+## Acceptance Criteria
+
+- Pipeline and collection installations produce a compatible set of the four
+  upgraded applications while Seriatim remains unchanged.
+- Native offline inspection/validation accepts Audita settings and selected
+  profiles, Notarius base/testing configurations, Scriptorium configuration,
+  and Narratio base/testing composition with synthetic credential inputs.
+  Direct Audita validation receives credential environment variables explicitly:
+  it does not bootstrap Narratio's secrets directory on its own.
+- Focused cases prove caller-supplied backend/profile configuration and
+  credential lists, testing selection, and existing important composition and
+  path handoffs. Assert supplied values, not fixed copies of today's defaults.
+- Immediate repeated convergence is clean. Existing meaningful shared-user,
+  cache, artifact-composition, and installed-collection behavior remains covered
+  without expanding into upstream module/runtime regression matrices.
+- Collection build/install and consumer syntax checks use the established
+  public dependency manifests. No sibling application/role checkout is needed
+  by an external consumer. Required referenced commits exist on public Gitea.
+- Consumer inventory retains comments/Vault references for unknown secrets;
+  no actual keys or operational deployments are required for completion.
+- Guidance covers breaking inventory fields, new CLI selectors, obsolete
+  testing-file locations, and upstream manifest migration. Back up and migrate
+  local provenance without deleting progress or editing immutable snapshots;
+  document normal named-selector invalidation rather than forcing reruns.
+
+## Open Questions
+
+None. The user approved deferring to application defaults wherever possible,
+with nullable inventory overrides, and confirmed retaining the existing
+900-second Audita per-attempt policy. Do not freeze native retry, repair, or
+operation-deadline defaults in suite variables. Operators should note that an
+attempt may consume the entire logical-operation budget; these are separate
+controls, not additive timeout guarantees.
+
+## Dependencies and Follow-up Work
+
+Features 0018–0021 provide the reviewed standalone revisions. This roadmap is
+approved; execution is defined in the [implementation plan](../implementation.md).
+Final review precedes feature closeout. Actual deployment,
+session migration, live processing, and a formal tagged release are separate
+operator actions, not completion gates for this feature.
+
+## Upstream References
+
+- [Audita v2 migration](https://gitea.maximumdirect.net/eric/audita/src/tag/v2.0.1/docs/migrations/v2.md)
+- [Audita configuration](https://gitea.maximumdirect.net/eric/audita/src/tag/v2.0.1/docs/config.md)
+- [Narratio v3 release](https://gitea.maximumdirect.net/eric/narratio/src/tag/v3.0.0/docs/releases/v3.0.0.md)
+- [Narratio operations and migration](https://gitea.maximumdirect.net/eric/narratio/src/tag/v3.0.0/docs/operations.md)
+- [Notarius v0.8.0 migration](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.8.0/docs/releases/v0.8.0.md)
+- [Scriptorium v0.14.0 migration](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.14.0/docs/releases/v0.14.0.md)