Plan the Narratio Scriptorium credential update

docs/roadmap/features/0023-narratio-scriptorium-credentials.md

@@ -0,0 +1,164 @@
+# Feature 0023: Adopt Narratio v3.1.0 and Forward Scriptorium Credentials
+
+Status: ready — roadmap and implementation plan approved
+
+## Summary
+
+Update the standalone Narratio role and D&D pipeline to Narratio v3.1.0 and
+expose its Scriptorium credential list. Deliver the coordinated change through
+the collection and existing deployment consumer without deploying to a host.
+
+## Problem or Opportunity
+
+The operator's deployment to `pipeline-dnd` failed during trim bounds generation
+with Scriptorium reporting HTTP 401. Narratio v3.0.0 filtered provider keys out
+of the child environment without offering a public Scriptorium credential
+selection. Analyze had the same forwarding gap. Provisioning a key through
+Ansible alone could not repair that boundary.
+
+Upstream has released explicit Scriptorium credential forwarding in v3.1.0.
+The original report is preserved in Narratio commit `9389f69`;
+the upstream change is `44a3186`, with release documentation at `40cbf42`.
+These are planning references, not substitutes for the released application tag.
+
+## Scope
+
+- Adopt Narratio v3.1.0 in the standalone role and suite version selections.
+- Expose its native credential-list field through the standalone role and
+  supply overrideable D&D defaults through the pipeline role.
+- Refresh the Narratio role dependency, public pipeline subtree snapshot,
+  collection dependency manifest, and existing consumer pins together.
+- Extend existing configuration/composition coverage and document operator
+  credential requirements and normal retry behavior.
+
+## Non-goals
+
+- Changing upstream applications, other application versions, models, prompts,
+  profile selections, or the existing filesystem and credential-storage policy.
+- Adding a credential loader, wrapper, environment file, provider discovery,
+  or per-stage credential-selection interface.
+- Live inference, deployment to a controller or `pipeline-dnd`, credential
+  verification against providers, or modifying Vault contents.
+- Publishing new Ansible release tags or collection release artifacts.
+- Reimplementing upstream credential forwarding, redaction, or checkpoint
+  behavior in Ansible or its test suites.
+
+## Target End State
+
+### Standalone Narratio role
+
+`narratio_version` defaults to `v3.1.0`. The optional public variable
+`narratio_scriptorium_credentials` defaults to null, is documented as a list
+of strings, and renders `scriptorium.credentials` in the native root pipeline
+file only when non-null. An explicit empty list renders `credentials: []`;
+nonempty lists preserve the caller's exact credential names. Include the field
+in the enclosing Scriptorium-section condition so it works independently of
+other optional Scriptorium variables. No provider-specific default belongs in
+the standalone role.
+
+### D&D pipeline policy
+
+`dnd_pipeline_narratio_version` selects `v3.1.0`.
+`dnd_pipeline_narratio_scriptorium_credentials` defaults to
+`[OPENROUTER_API_KEY]` and is passed through the existing Narratio dependency
+variables. Inventory can replace the list, including with `[]`, or set null
+to omit the field through the standalone role's interface.
+
+The testing overlay inherits this list; no duplicate testing-specific setting
+is needed for the currently selected profiles. The existing Notarius reference
+to the same key remains independent. Operators selecting different providers
+can supply the corresponding names through the ordinary inventory interface.
+
+Credential bytes continue to come from Narratio's process environment or its
+existing secrets mechanism. `dnd_pipeline_secrets` remains an optional managed
+file-provisioning mechanism, not the only valid credential source. No new secret
+file or migration is needed where `OPENROUTER_API_KEY` is already provisioned.
+
+### Upstream contract and operator handoff
+
+Narratio calls the field `pipeline.scriptorium.credentials` internally; the
+root pipeline YAML uses `scriptorium: {credentials: [...]}` without an extra
+`pipeline:` wrapper. The list applies to trim, analyze (including character
+families), and render-debug calls. It forwards names and resolved values without
+overriding Scriptorium's own provider/profile selection.
+
+Upstream resolves selected names to nonempty values at configuration load,
+including for commands that do not execute Scriptorium. Tests and operator
+guidance must account for that existing contract. Omitted/null/empty lists
+forward nothing; a native overlay replaces the full list. After upgrading and
+provisioning the selected key, retry failed trim/analyze normally; do not force
+successful work solely because a credential was added or rotated. See the
+upstream references below for the authoritative behavior.
+
+## Repository Ownership
+
+- `ansible-role-narratio`: version default, public variable, argument spec,
+  template, documentation, and focused configuration tests.
+- `ansible-role-dnd-pipeline`: suite version/credential defaults, dependency
+  wiring and pin, documentation, and composition coverage.
+- `ansible-collection-dnd`: exact public pipeline subtree import, matching
+  dependency manifest, and installed-artifact validation.
+- `rakestraw-ansible`: updated consumer dependency pins and concise upgrade
+  guidance only; preserve its minimal playbook, inventory, and Vault content.
+- Workspace: feature tracking, later staged plan, and review evidence.
+- Narratio and the remaining applications/assets: read-only upstream contracts.
+
+## Constraints and Settled Decisions
+
+Follow the [Ansible](../../ansible.md), [testing](../../testing.md), and
+[planning](../../planning.md) policies and the [MVP architecture](../mvp.md).
+Preserve generic application roles versus opinionated suite policy. Reuse the
+existing declarative source-build and template tasks; add no production checks.
+Required dependency commits must be public before external consumer use; subtree
+imports use public Gitea, not sibling paths. Do not alter unrelated pins.
+
+## Simplest Viable Implementation
+
+Add one nullable standalone configuration variable and its ordinary template,
+argument-spec, and documentation entries. Add one suite default and dependency
+mapping. Advance the Narratio version and coordinated dependency pins, then
+reuse the existing collection import and consumer refresh workflow. No new
+production task or persistent state is needed.
+
+## Acceptance Criteria
+
+- The standalone role and assembled suite build the released v3.1.0 source.
+  Any version assertion explicitly supplies the version under test rather than
+  freezing the current default.
+- Representative explicit credential names reach the deployed native
+  configuration; null omission and explicit `[]` are supported. Extend existing
+  cases rather than adding a separate exhaustive matrix.
+- Suite composition preserves an explicitly supplied credential list in base
+  and testing configurations. Use synthetic credential values for offline
+  native validation; do not assert today's default list as a permanent invariant.
+- The real role and installed collection pass their existing relevant
+  validation, including immediate repeated convergence with no changes.
+  Credential-forwarding internals and provider authentication remain upstream
+  responsibilities; no live service or new subprocess simulation is required.
+- Collection and consumer manifests resolve the coordinated public revisions;
+  consumer host listing/syntax checks work without deployment or Vault
+  decryption. Leave the existing hosts-and-roles playbook unchanged.
+- Guidance explains load-time credential requirements, existing environment or
+  file provisioning, dependency refresh, and ordinary retry after the fix.
+
+## Dependencies and Follow-up Work
+
+The public Narratio v3.1.0 tag was verified on 2026-09-29 at
+`40cbf4243838b0a1db6caf4c8f790d914ccb96cc`, matching the reviewed release
+documentation and configuration contract. The release prerequisite is satisfied.
+Use that release rather than substituting a moving branch.
+
+A [staged implementation plan](../implementation.md) is approved for execution.
+Publishing required Ansible dependency commits is explicit in that plan's scope;
+this planning pass does not push anything. Operational deployment and a paid/live smoke run
+remain separate operator activities, not feature-completion gates.
+
+## Open Questions
+
+None identified.
+
+## Upstream References
+
+- [Narratio v3.1.0 release notes](../../../../narratio/docs/releases/v3.1.0.md)
+- [Scriptorium credentials](../../../../narratio/docs/config.md#scriptorium-credentials)
+- [401 troubleshooting](../../../../narratio/docs/troubleshooting.md#scriptorium-returns-http-401-during-trim-or-analyze)

docs/roadmap/implementation.md

@@ -1,13 +1,259 @@
-# Implementation Plan
+# Implementation Plan 0023: Narratio Scriptorium Credentials
 
-No implementation plan is currently active.
+Status: ready — approved for implementation
 
-[Feature 0022](features/0022-coordinated-suite-upgrade.md) was closed on
-2026-09-29 after implementation, independent review, and resolution of all
-three findings. Its roadmap and [remediation record](remediation.md) retain
-the completion evidence; Git history preserves the completed staged plan.
+Target executor: GPT-5.6-Terra
 
-The [application upgrade sequence](application-upgrade-sequence.md) is complete.
-Discuss and select the next unit of work before preparing a new feature
-roadmap or staged implementation plan. Operational deployment and a new tagged
-release are not authorized by this notice.
+Feature roadmap: [Feature 0023](features/0023-narratio-scriptorium-credentials.md)
+
+## Objective
+
+Deploy Narratio v3.1.0 through the existing source-build roles and expose its
+Scriptorium credential list. The roadmap owns scope and rationale. This plan
+does not deploy to an operational controller or `pipeline-dnd`.
+
+## Execution Rules and Starting State
+
+- Execute stages in numeric order. Before each stage, reread its listed policies
+  (paths relative to the workspace), applicable local instructions, and inspect
+  working trees. Preserve unrelated changes. Commit validated stages separately.
+- Use workspace `.venv/bin` tooling; harnesses select it themselves. A missing
+  PATH entry is not a missing Ansible controller.
+- Integration harnesses use sudo and mutate the local development installation.
+  Read setup/cleanup and confirm local paths are disposable before running them.
+  Do not contact operational hosts, inference, transcription, or storage services.
+  Public dependency downloads are expected. Report unsafe local mutation as a
+  specific validation limit rather than silently running on another host.
+- Add no production tasks, state probes, wrappers, credential loaders, or
+  application self-tests. Reuse ordinary source-build and template convergence.
+- Approval of this plan authorizes normal publication of the changed Narratio
+  role, pipeline role, and collection commits to their `origin/main` branches
+  for downstream public dependencies. No force pushes or tags. Do not publish
+  consumer/workspace commits or unrelated work. Resolve any unrelated unpublished
+  commits before pushing a branch that would include them.
+- The public Narratio v3.1.0 tag was verified on 2026-09-29 at
+  `40cbf4243838b0a1db6caf4c8f790d914ccb96cc`, matching the reviewed contract.
+  Install the released tag, not a moving branch.
+- Reviewed starting pins: Narratio role `3c0700b`, pipeline `37b1cf6`, collection
+  `b3bc045`. Recheck actual heads; these identify context, not frozen assertions.
+  The consumer already has local commits; preserve them. Record full resulting
+  SHAs at each stage rather than inventing future commit IDs.
+- Scope: Narratio role, pipeline role, collection, consumer pins/guidance, and
+  workspace tracking. Other roles, upstream applications, assets, Vault content,
+  and operational deployments remain unchanged.
+
+## Stage 1: Expose credentials in the standalone Narratio role
+
+### Required Policy Review
+
+Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, `docs/ansible.md`,
+and `docs/testing.md`, plus applicable repository-local instructions.
+
+### Repository and Preconditions
+
+`ansible-role-narratio`; v3.1.0 must be fetchable. Review the released upstream
+`docs/config.md#scriptorium-credentials` and `docs/releases/v3.1.0.md`.
+
+### Required Changes
+
+1. Set `narratio_version: v3.1.0`. Add
+   `narratio_scriptorium_credentials: null` and declare an optional list of
+   strings in `meta/argument_specs.yml`.
+2. Extend the Scriptorium section predicate in `templates/pipeline.yml.j2` and
+   render `credentials` with `to_json` only when the variable `is not none`.
+   Root YAML uses `scriptorium:`, without a new `pipeline:` wrapper. The field
+   must render even when all other optional Scriptorium variables are null.
+3. Document null omission, explicit `[]`, arbitrary provider reference names,
+   load-time resolution, and environment/secrets provisioning. Keep provider
+   policy out of the standalone role.
+4. Extend existing tests: baseline explicitly omits this field; representative
+   configuration supplies two synthetic names and checks that exact list;
+   the existing empty-list case supplies and expects `[]`. Reuse existing
+   synthetic names/values and pass them explicitly into native validation as
+   the Narratio account. Cover credential-only section rendering within a
+   focused configuration case, not a new exhaustive fixture matrix.
+5. Include the new variable in baseline reset inputs so the existing v2.1.0
+   version-selection/recovery cases omit unsupported configuration. Preserve
+   those cases. Explicitly request v3.1.0 in the representative case and verify
+   its reported version; do not assert the role's current default version.
+
+### Validation
+
+From this repository run `bash -n tests/integration`, `./tests/integration`,
+and `git diff --check`. The existing harness includes syntax/lint, native
+configuration validation, real convergence, and clean repeats. Require supplied
+list preservation, omission/empty-list semantics, the requested binary version,
+and zero changes on immediate reapplication. Do not simulate upstream provider
+requests or retest internal credential forwarding.
+
+### Completion and Commit
+
+Commit `Support Scriptorium credentials in Narratio`. Publish the validated
+commit to the public role origin, verify with a remote lookup, and record its
+full SHA for Stage 2.
+
+## Stage 2: Configure credentials in the D&D pipeline
+
+### Required Policy Review
+
+Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, `docs/ansible.md`,
+and `docs/testing.md`, plus applicable repository-local instructions.
+
+### Repository and Preconditions
+
+`ansible-role-dnd-pipeline`; Stage 1 is validated and public.
+
+### Required Changes
+
+1. Pin Stage 1's exact SHA in `requirements.yml`. Set
+   `dnd_pipeline_narratio_version: v3.1.0`. Add
+   `dnd_pipeline_narratio_scriptorium_credentials: [OPENROUTER_API_KEY]`, its
+   optional list-of-strings argument specification, and the direct mapping to
+   `narratio_scriptorium_credentials` in `meta/main.yml`.
+2. Let the testing overlay inherit the base list. Preserve ordinary inventory
+   replacement, `[]`, and null pass-through semantics; add no duplicate overlay
+   default or per-stage setting.
+3. Document that Notarius and Scriptorium can use the same provisioned key but
+   have independent lists. Selected keys must resolve at load time even for
+   inspection. Managed secret files are optional. Explain normal retry after
+   upgrading, without forcing successful stages.
+4. Extend the existing custom-credential fixture with an explicitly supplied
+   Scriptorium list and synthetic values. Verify its presence in effective base
+   and testing configurations, using existing native inspection and secret
+   delivery. Do not assert the baseline default list or add subprocess stubs.
+   Preserve the baseline's existing synthetic OpenRouter key.
+5. Leave other application versions, model/profile choices, tasks, secrets
+   policy, and dependency pins unchanged.
+
+### Validation
+
+Run `bash -n tests/integration`, `./tests/integration`, and `git diff --check`.
+Require native base/testing acceptance and the existing immediate clean-repeat
+behavior, without live service calls.
+
+### Completion and Commit
+
+Commit `Forward Scriptorium credentials in the D&D suite`. Publish and record
+the validated public pipeline SHA for Stage 3.
+
+## Stage 3: Assemble the updated collection
+
+### Required Policy Review
+
+Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, `docs/ansible.md`,
+and `docs/testing.md`, plus applicable repository-local instructions.
+
+### Repository and Preconditions
+
+`ansible-collection-dnd`; Stage 2's exact source revision is public.
+
+### Required Changes
+
+1. Import the recorded full SHA through the existing public workflow:
+
+   ```sh
+   git subtree pull --prefix=roles/pipeline \
+     https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \
+     <full-published-pipeline-sha> --squash
+   ```
+
+   Replace the placeholder; never edit the subtree directly or import a sibling
+   path. Keep the normal subtree merge/squash commits.
+2. Update the collection-owned `requirements.yml` Narratio role pin to Stage 1.
+   Preserve other pins and collection metadata version; this is not a new tagged
+   release. Update any current development compatibility guidance as needed.
+3. Adjust collection-owned fixtures only if synthetic credentials are missing.
+   The baseline already supplies `OPENROUTER_API_KEY`. Do not duplicate detailed
+   role tests or add assertions that freeze suite defaults.
+
+### Validation
+
+Run `bash -n tests/integration`, `./tests/integration`, and `git diff --check`.
+The harness builds/installs the artifact in isolated Ansible paths, resolves
+public dependencies, validates base/testing configuration, and checks repeats.
+Compare the imported `HEAD:roles/pipeline` Git tree with the source commit's
+tree; they must match exactly.
+
+### Completion and Commit
+
+Commit collection-owned changes as `Ship the Narratio credential forwarding
+update`. Publish the validated collection head and record its full SHA.
+
+## Stage 4: Refresh the consumer without deploying
+
+### Required Policy Review
+
+Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, `docs/ansible.md`,
+and `docs/testing.md`, plus applicable repository-local instructions.
+
+### Repository and Preconditions
+
+`rakestraw-ansible`; Stage 3 is validated and public. Preserve pre-existing
+local commits and unrelated requirements.
+
+### Required Changes
+
+1. Update only Narratio role and `rakestraw.dnd` pins in `requirements.yml` to
+   the full Stage 1 and Stage 3 SHAs.
+2. Update `docs/pipeline-dnd.md`'s explicit refresh SHA and source guidance links.
+   Explain v3.1.0 credential selection and ordinary retry. Keep forced refresh
+   instructions for development snapshots with unchanged collection metadata
+   and preserve prior migration guidance.
+3. Leave playbook, host variables, and Vault unchanged: the suite selects the
+   existing Notarius key. No real keys or wrappers are needed.
+
+### Validation
+
+Using workspace `.venv/bin/ansible-galaxy`, install the D&D dependencies and
+the pinned collection into fresh `mktemp -d` paths. Reuse the collection manifest
+for that dependency set rather than installing unrelated maximumdirect projects.
+Set `ANSIBLE_ROLES_PATH` and `ANSIBLE_COLLECTIONS_PATH` to the disposable paths.
+From the consumer repository, using its existing inventory configuration, run:
+
+```sh
+../.venv/bin/ansible-playbook playbooks/pipeline/dnd.yml --list-hosts
+../.venv/bin/ansible-playbook playbooks/pipeline/dnd.yml --syntax-check
+git diff --check
+```
+
+Require only `pipeline-dnd` in the host list and successful dependency/syntax
+resolution. These checks do not deploy or require provider keys/Vault decryption.
+Do not add ansible-doc output, live host access, or API success as a gate.
+
+### Completion and Commit
+
+Commit `Use Narratio with Scriptorium credential forwarding`. Leave consumer
+commits local; record validation and the selected collection revision.
+
+## Stage 5: Record implementation results for review
+
+### Required Policy Review
+
+Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, and
+`docs/planning.md`.
+
+### Repository and Preconditions
+
+Workspace root; Stages 1–4 are completed and validated.
+
+### Required Changes and Validation
+
+Update Feature 0023 and `docs/status.md` to `in_progress`, implementation complete
+and awaiting independent review. Record exact commits, public versus local
+status, validation commands/results, and deviations. Add concise stage evidence
+below. Run `git diff --check` and `scripts/status`; confirm changed repositories
+are clean after their commits. Do not mark the feature done or retire the plan.
+
+### Completion and Commit
+
+Commit `Record the Scriptorium credential update handoff`. Leave workspace
+commits local. Report unverified behavior explicitly: offline checks do not prove
+the operator's actual key is valid or that live trim succeeds.
+
+## Progress Notes
+
+Planning only; no implementation stages executed.
+
+## Open Questions
+
+None.

docs/status.md

@@ -4,14 +4,12 @@ Last updated: 2026-09-29
 
 Current milestone: Operational D&D pipeline deployment
 
-Current focus: Feature 0022 is complete, including all three review
-remediations. The application upgrade sequence is complete; no implementation
-plan is active. The corrected pipeline and collection revisions are public;
-the consumer commits remain local. No operational deployment or tagged release
-was performed. Recommended next activity: a separately authorized operational
-upgrade and production/testing smoke run, followed by a coordinated tagged
-release if desired. Broader downgrade/removal policy remains unselected design
-work, not a blocker to the completed features.
+Current focus: Feature 0023's roadmap and staged implementation plan are approved
+and ready for the implementation sprint. Narratio v3.1.0 is released; its public tag was
+verified. The feature wires Scriptorium credentials through the standalone role
+and D&D suite to address the operator's reported trim HTTP 401.
+Feature 0022 and the application upgrade sequence remain complete. Broader
+downgrade/removal policy remains unselected design work.
 See the [upgrade sequence](roadmap/application-upgrade-sequence.md) and
 [feature evidence](roadmap/features/0022-coordinated-suite-upgrade.md).
 
@@ -49,6 +47,7 @@ See the [upgrade sequence](roadmap/application-upgrade-sequence.md) and
 | F0021 | Adopt Narratio v3.0.0 | Narratio role and workspace | done | Role `3c0700b`, following `a28f22c` and `45f6836`, passed independent review with no findings; [feature roadmap](roadmap/features/0021-narratio-v3-adoption.md) records evidence; suite pins unchanged |
 | F0022 | Integrate the reviewed application upgrades | Pipeline role, collection, consumer, and workspace | done | Closed after review and resolution of all three findings; public pipeline `37b1cf6`, public collection `b3bc045`, local consumer `15f0f4e`; [remediation evidence](roadmap/remediation.md) |
 | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` |
+| F0023 | Adopt Narratio v3.1.0 and forward Scriptorium credentials | Narratio role, pipeline role, collection, consumer, and workspace | ready | Roadmap and [implementation plan](roadmap/implementation.md) approved; v3.1.0 tag verified; ready for implementation |
 | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review |
 
 ## Updating This File