diff --git a/docs/roadmap/features/0023-narratio-scriptorium-credentials.md b/docs/roadmap/features/0023-narratio-scriptorium-credentials.md new file mode 100644 index 0000000..63d281c --- /dev/null +++ b/docs/roadmap/features/0023-narratio-scriptorium-credentials.md @@ -0,0 +1,164 @@ +# Feature 0023: Adopt Narratio v3.1.0 and Forward Scriptorium Credentials + +Status: ready — roadmap and implementation plan approved + +## Summary + +Update the standalone Narratio role and D&D pipeline to Narratio v3.1.0 and +expose its Scriptorium credential list. Deliver the coordinated change through +the collection and existing deployment consumer without deploying to a host. + +## Problem or Opportunity + +The operator's deployment to `pipeline-dnd` failed during trim bounds generation +with Scriptorium reporting HTTP 401. Narratio v3.0.0 filtered provider keys out +of the child environment without offering a public Scriptorium credential +selection. Analyze had the same forwarding gap. Provisioning a key through +Ansible alone could not repair that boundary. + +Upstream has released explicit Scriptorium credential forwarding in v3.1.0. +The original report is preserved in Narratio commit `9389f69`; +the upstream change is `44a3186`, with release documentation at `40cbf42`. +These are planning references, not substitutes for the released application tag. + +## Scope + +- Adopt Narratio v3.1.0 in the standalone role and suite version selections. +- Expose its native credential-list field through the standalone role and + supply overrideable D&D defaults through the pipeline role. +- Refresh the Narratio role dependency, public pipeline subtree snapshot, + collection dependency manifest, and existing consumer pins together. +- Extend existing configuration/composition coverage and document operator + credential requirements and normal retry behavior. + +## Non-goals + +- Changing upstream applications, other application versions, models, prompts, + profile selections, or the existing filesystem and credential-storage policy. +- Adding a credential loader, wrapper, environment file, provider discovery, + or per-stage credential-selection interface. +- Live inference, deployment to a controller or `pipeline-dnd`, credential + verification against providers, or modifying Vault contents. +- Publishing new Ansible release tags or collection release artifacts. +- Reimplementing upstream credential forwarding, redaction, or checkpoint + behavior in Ansible or its test suites. + +## Target End State + +### Standalone Narratio role + +`narratio_version` defaults to `v3.1.0`. The optional public variable +`narratio_scriptorium_credentials` defaults to null, is documented as a list +of strings, and renders `scriptorium.credentials` in the native root pipeline +file only when non-null. An explicit empty list renders `credentials: []`; +nonempty lists preserve the caller's exact credential names. Include the field +in the enclosing Scriptorium-section condition so it works independently of +other optional Scriptorium variables. No provider-specific default belongs in +the standalone role. + +### D&D pipeline policy + +`dnd_pipeline_narratio_version` selects `v3.1.0`. +`dnd_pipeline_narratio_scriptorium_credentials` defaults to +`[OPENROUTER_API_KEY]` and is passed through the existing Narratio dependency +variables. Inventory can replace the list, including with `[]`, or set null +to omit the field through the standalone role's interface. + +The testing overlay inherits this list; no duplicate testing-specific setting +is needed for the currently selected profiles. The existing Notarius reference +to the same key remains independent. Operators selecting different providers +can supply the corresponding names through the ordinary inventory interface. + +Credential bytes continue to come from Narratio's process environment or its +existing secrets mechanism. `dnd_pipeline_secrets` remains an optional managed +file-provisioning mechanism, not the only valid credential source. No new secret +file or migration is needed where `OPENROUTER_API_KEY` is already provisioned. + +### Upstream contract and operator handoff + +Narratio calls the field `pipeline.scriptorium.credentials` internally; the +root pipeline YAML uses `scriptorium: {credentials: [...]}` without an extra +`pipeline:` wrapper. The list applies to trim, analyze (including character +families), and render-debug calls. It forwards names and resolved values without +overriding Scriptorium's own provider/profile selection. + +Upstream resolves selected names to nonempty values at configuration load, +including for commands that do not execute Scriptorium. Tests and operator +guidance must account for that existing contract. Omitted/null/empty lists +forward nothing; a native overlay replaces the full list. After upgrading and +provisioning the selected key, retry failed trim/analyze normally; do not force +successful work solely because a credential was added or rotated. See the +upstream references below for the authoritative behavior. + +## Repository Ownership + +- `ansible-role-narratio`: version default, public variable, argument spec, + template, documentation, and focused configuration tests. +- `ansible-role-dnd-pipeline`: suite version/credential defaults, dependency + wiring and pin, documentation, and composition coverage. +- `ansible-collection-dnd`: exact public pipeline subtree import, matching + dependency manifest, and installed-artifact validation. +- `rakestraw-ansible`: updated consumer dependency pins and concise upgrade + guidance only; preserve its minimal playbook, inventory, and Vault content. +- Workspace: feature tracking, later staged plan, and review evidence. +- Narratio and the remaining applications/assets: read-only upstream contracts. + +## Constraints and Settled Decisions + +Follow the [Ansible](../../ansible.md), [testing](../../testing.md), and +[planning](../../planning.md) policies and the [MVP architecture](../mvp.md). +Preserve generic application roles versus opinionated suite policy. Reuse the +existing declarative source-build and template tasks; add no production checks. +Required dependency commits must be public before external consumer use; subtree +imports use public Gitea, not sibling paths. Do not alter unrelated pins. + +## Simplest Viable Implementation + +Add one nullable standalone configuration variable and its ordinary template, +argument-spec, and documentation entries. Add one suite default and dependency +mapping. Advance the Narratio version and coordinated dependency pins, then +reuse the existing collection import and consumer refresh workflow. No new +production task or persistent state is needed. + +## Acceptance Criteria + +- The standalone role and assembled suite build the released v3.1.0 source. + Any version assertion explicitly supplies the version under test rather than + freezing the current default. +- Representative explicit credential names reach the deployed native + configuration; null omission and explicit `[]` are supported. Extend existing + cases rather than adding a separate exhaustive matrix. +- Suite composition preserves an explicitly supplied credential list in base + and testing configurations. Use synthetic credential values for offline + native validation; do not assert today's default list as a permanent invariant. +- The real role and installed collection pass their existing relevant + validation, including immediate repeated convergence with no changes. + Credential-forwarding internals and provider authentication remain upstream + responsibilities; no live service or new subprocess simulation is required. +- Collection and consumer manifests resolve the coordinated public revisions; + consumer host listing/syntax checks work without deployment or Vault + decryption. Leave the existing hosts-and-roles playbook unchanged. +- Guidance explains load-time credential requirements, existing environment or + file provisioning, dependency refresh, and ordinary retry after the fix. + +## Dependencies and Follow-up Work + +The public Narratio v3.1.0 tag was verified on 2026-09-29 at +`40cbf4243838b0a1db6caf4c8f790d914ccb96cc`, matching the reviewed release +documentation and configuration contract. The release prerequisite is satisfied. +Use that release rather than substituting a moving branch. + +A [staged implementation plan](../implementation.md) is approved for execution. +Publishing required Ansible dependency commits is explicit in that plan's scope; +this planning pass does not push anything. Operational deployment and a paid/live smoke run +remain separate operator activities, not feature-completion gates. + +## Open Questions + +None identified. + +## Upstream References + +- [Narratio v3.1.0 release notes](../../../../narratio/docs/releases/v3.1.0.md) +- [Scriptorium credentials](../../../../narratio/docs/config.md#scriptorium-credentials) +- [401 troubleshooting](../../../../narratio/docs/troubleshooting.md#scriptorium-returns-http-401-during-trim-or-analyze) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 520abf9..8cdddec 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,13 +1,259 @@ -# Implementation Plan +# Implementation Plan 0023: Narratio Scriptorium Credentials -No implementation plan is currently active. +Status: ready — approved for implementation -[Feature 0022](features/0022-coordinated-suite-upgrade.md) was closed on -2026-09-29 after implementation, independent review, and resolution of all -three findings. Its roadmap and [remediation record](remediation.md) retain -the completion evidence; Git history preserves the completed staged plan. +Target executor: GPT-5.6-Terra -The [application upgrade sequence](application-upgrade-sequence.md) is complete. -Discuss and select the next unit of work before preparing a new feature -roadmap or staged implementation plan. Operational deployment and a new tagged -release are not authorized by this notice. +Feature roadmap: [Feature 0023](features/0023-narratio-scriptorium-credentials.md) + +## Objective + +Deploy Narratio v3.1.0 through the existing source-build roles and expose its +Scriptorium credential list. The roadmap owns scope and rationale. This plan +does not deploy to an operational controller or `pipeline-dnd`. + +## Execution Rules and Starting State + +- Execute stages in numeric order. Before each stage, reread its listed policies + (paths relative to the workspace), applicable local instructions, and inspect + working trees. Preserve unrelated changes. Commit validated stages separately. +- Use workspace `.venv/bin` tooling; harnesses select it themselves. A missing + PATH entry is not a missing Ansible controller. +- Integration harnesses use sudo and mutate the local development installation. + Read setup/cleanup and confirm local paths are disposable before running them. + Do not contact operational hosts, inference, transcription, or storage services. + Public dependency downloads are expected. Report unsafe local mutation as a + specific validation limit rather than silently running on another host. +- Add no production tasks, state probes, wrappers, credential loaders, or + application self-tests. Reuse ordinary source-build and template convergence. +- Approval of this plan authorizes normal publication of the changed Narratio + role, pipeline role, and collection commits to their `origin/main` branches + for downstream public dependencies. No force pushes or tags. Do not publish + consumer/workspace commits or unrelated work. Resolve any unrelated unpublished + commits before pushing a branch that would include them. +- The public Narratio v3.1.0 tag was verified on 2026-09-29 at + `40cbf4243838b0a1db6caf4c8f790d914ccb96cc`, matching the reviewed contract. + Install the released tag, not a moving branch. +- Reviewed starting pins: Narratio role `3c0700b`, pipeline `37b1cf6`, collection + `b3bc045`. Recheck actual heads; these identify context, not frozen assertions. + The consumer already has local commits; preserve them. Record full resulting + SHAs at each stage rather than inventing future commit IDs. +- Scope: Narratio role, pipeline role, collection, consumer pins/guidance, and + workspace tracking. Other roles, upstream applications, assets, Vault content, + and operational deployments remain unchanged. + +## Stage 1: Expose credentials in the standalone Narratio role + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, `docs/ansible.md`, +and `docs/testing.md`, plus applicable repository-local instructions. + +### Repository and Preconditions + +`ansible-role-narratio`; v3.1.0 must be fetchable. Review the released upstream +`docs/config.md#scriptorium-credentials` and `docs/releases/v3.1.0.md`. + +### Required Changes + +1. Set `narratio_version: v3.1.0`. Add + `narratio_scriptorium_credentials: null` and declare an optional list of + strings in `meta/argument_specs.yml`. +2. Extend the Scriptorium section predicate in `templates/pipeline.yml.j2` and + render `credentials` with `to_json` only when the variable `is not none`. + Root YAML uses `scriptorium:`, without a new `pipeline:` wrapper. The field + must render even when all other optional Scriptorium variables are null. +3. Document null omission, explicit `[]`, arbitrary provider reference names, + load-time resolution, and environment/secrets provisioning. Keep provider + policy out of the standalone role. +4. Extend existing tests: baseline explicitly omits this field; representative + configuration supplies two synthetic names and checks that exact list; + the existing empty-list case supplies and expects `[]`. Reuse existing + synthetic names/values and pass them explicitly into native validation as + the Narratio account. Cover credential-only section rendering within a + focused configuration case, not a new exhaustive fixture matrix. +5. Include the new variable in baseline reset inputs so the existing v2.1.0 + version-selection/recovery cases omit unsupported configuration. Preserve + those cases. Explicitly request v3.1.0 in the representative case and verify + its reported version; do not assert the role's current default version. + +### Validation + +From this repository run `bash -n tests/integration`, `./tests/integration`, +and `git diff --check`. The existing harness includes syntax/lint, native +configuration validation, real convergence, and clean repeats. Require supplied +list preservation, omission/empty-list semantics, the requested binary version, +and zero changes on immediate reapplication. Do not simulate upstream provider +requests or retest internal credential forwarding. + +### Completion and Commit + +Commit `Support Scriptorium credentials in Narratio`. Publish the validated +commit to the public role origin, verify with a remote lookup, and record its +full SHA for Stage 2. + +## Stage 2: Configure credentials in the D&D pipeline + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, `docs/ansible.md`, +and `docs/testing.md`, plus applicable repository-local instructions. + +### Repository and Preconditions + +`ansible-role-dnd-pipeline`; Stage 1 is validated and public. + +### Required Changes + +1. Pin Stage 1's exact SHA in `requirements.yml`. Set + `dnd_pipeline_narratio_version: v3.1.0`. Add + `dnd_pipeline_narratio_scriptorium_credentials: [OPENROUTER_API_KEY]`, its + optional list-of-strings argument specification, and the direct mapping to + `narratio_scriptorium_credentials` in `meta/main.yml`. +2. Let the testing overlay inherit the base list. Preserve ordinary inventory + replacement, `[]`, and null pass-through semantics; add no duplicate overlay + default or per-stage setting. +3. Document that Notarius and Scriptorium can use the same provisioned key but + have independent lists. Selected keys must resolve at load time even for + inspection. Managed secret files are optional. Explain normal retry after + upgrading, without forcing successful stages. +4. Extend the existing custom-credential fixture with an explicitly supplied + Scriptorium list and synthetic values. Verify its presence in effective base + and testing configurations, using existing native inspection and secret + delivery. Do not assert the baseline default list or add subprocess stubs. + Preserve the baseline's existing synthetic OpenRouter key. +5. Leave other application versions, model/profile choices, tasks, secrets + policy, and dependency pins unchanged. + +### Validation + +Run `bash -n tests/integration`, `./tests/integration`, and `git diff --check`. +Require native base/testing acceptance and the existing immediate clean-repeat +behavior, without live service calls. + +### Completion and Commit + +Commit `Forward Scriptorium credentials in the D&D suite`. Publish and record +the validated public pipeline SHA for Stage 3. + +## Stage 3: Assemble the updated collection + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, `docs/ansible.md`, +and `docs/testing.md`, plus applicable repository-local instructions. + +### Repository and Preconditions + +`ansible-collection-dnd`; Stage 2's exact source revision is public. + +### Required Changes + +1. Import the recorded full SHA through the existing public workflow: + + ```sh + git subtree pull --prefix=roles/pipeline \ + https://gitea.maximumdirect.net/eric/ansible-role-dnd-pipeline.git \ + --squash + ``` + + Replace the placeholder; never edit the subtree directly or import a sibling + path. Keep the normal subtree merge/squash commits. +2. Update the collection-owned `requirements.yml` Narratio role pin to Stage 1. + Preserve other pins and collection metadata version; this is not a new tagged + release. Update any current development compatibility guidance as needed. +3. Adjust collection-owned fixtures only if synthetic credentials are missing. + The baseline already supplies `OPENROUTER_API_KEY`. Do not duplicate detailed + role tests or add assertions that freeze suite defaults. + +### Validation + +Run `bash -n tests/integration`, `./tests/integration`, and `git diff --check`. +The harness builds/installs the artifact in isolated Ansible paths, resolves +public dependencies, validates base/testing configuration, and checks repeats. +Compare the imported `HEAD:roles/pipeline` Git tree with the source commit's +tree; they must match exactly. + +### Completion and Commit + +Commit collection-owned changes as `Ship the Narratio credential forwarding +update`. Publish the validated collection head and record its full SHA. + +## Stage 4: Refresh the consumer without deploying + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, `docs/ansible.md`, +and `docs/testing.md`, plus applicable repository-local instructions. + +### Repository and Preconditions + +`rakestraw-ansible`; Stage 3 is validated and public. Preserve pre-existing +local commits and unrelated requirements. + +### Required Changes + +1. Update only Narratio role and `rakestraw.dnd` pins in `requirements.yml` to + the full Stage 1 and Stage 3 SHAs. +2. Update `docs/pipeline-dnd.md`'s explicit refresh SHA and source guidance links. + Explain v3.1.0 credential selection and ordinary retry. Keep forced refresh + instructions for development snapshots with unchanged collection metadata + and preserve prior migration guidance. +3. Leave playbook, host variables, and Vault unchanged: the suite selects the + existing Notarius key. No real keys or wrappers are needed. + +### Validation + +Using workspace `.venv/bin/ansible-galaxy`, install the D&D dependencies and +the pinned collection into fresh `mktemp -d` paths. Reuse the collection manifest +for that dependency set rather than installing unrelated maximumdirect projects. +Set `ANSIBLE_ROLES_PATH` and `ANSIBLE_COLLECTIONS_PATH` to the disposable paths. +From the consumer repository, using its existing inventory configuration, run: + +```sh +../.venv/bin/ansible-playbook playbooks/pipeline/dnd.yml --list-hosts +../.venv/bin/ansible-playbook playbooks/pipeline/dnd.yml --syntax-check +git diff --check +``` + +Require only `pipeline-dnd` in the host list and successful dependency/syntax +resolution. These checks do not deploy or require provider keys/Vault decryption. +Do not add ansible-doc output, live host access, or API success as a gate. + +### Completion and Commit + +Commit `Use Narratio with Scriptorium credential forwarding`. Leave consumer +commits local; record validation and the selected collection revision. + +## Stage 5: Record implementation results for review + +### Required Policy Review + +Read `AGENTS.md`, `docs/orientation.md`, `docs/conventions.md`, and +`docs/planning.md`. + +### Repository and Preconditions + +Workspace root; Stages 1–4 are completed and validated. + +### Required Changes and Validation + +Update Feature 0023 and `docs/status.md` to `in_progress`, implementation complete +and awaiting independent review. Record exact commits, public versus local +status, validation commands/results, and deviations. Add concise stage evidence +below. Run `git diff --check` and `scripts/status`; confirm changed repositories +are clean after their commits. Do not mark the feature done or retire the plan. + +### Completion and Commit + +Commit `Record the Scriptorium credential update handoff`. Leave workspace +commits local. Report unverified behavior explicitly: offline checks do not prove +the operator's actual key is valid or that live trim succeeds. + +## Progress Notes + +Planning only; no implementation stages executed. + +## Open Questions + +None. diff --git a/docs/status.md b/docs/status.md index 6c33641..b384f0f 100644 --- a/docs/status.md +++ b/docs/status.md @@ -4,14 +4,12 @@ Last updated: 2026-09-29 Current milestone: Operational D&D pipeline deployment -Current focus: Feature 0022 is complete, including all three review -remediations. The application upgrade sequence is complete; no implementation -plan is active. The corrected pipeline and collection revisions are public; -the consumer commits remain local. No operational deployment or tagged release -was performed. Recommended next activity: a separately authorized operational -upgrade and production/testing smoke run, followed by a coordinated tagged -release if desired. Broader downgrade/removal policy remains unselected design -work, not a blocker to the completed features. +Current focus: Feature 0023's roadmap and staged implementation plan are approved +and ready for the implementation sprint. Narratio v3.1.0 is released; its public tag was +verified. The feature wires Scriptorium credentials through the standalone role +and D&D suite to address the operator's reported trim HTTP 401. +Feature 0022 and the application upgrade sequence remain complete. Broader +downgrade/removal policy remains unselected design work. See the [upgrade sequence](roadmap/application-upgrade-sequence.md) and [feature evidence](roadmap/features/0022-coordinated-suite-upgrade.md). @@ -49,6 +47,7 @@ See the [upgrade sequence](roadmap/application-upgrade-sequence.md) and | F0021 | Adopt Narratio v3.0.0 | Narratio role and workspace | done | Role `3c0700b`, following `a28f22c` and `45f6836`, passed independent review with no findings; [feature roadmap](roadmap/features/0021-narratio-v3-adoption.md) records evidence; suite pins unchanged | | F0022 | Integrate the reviewed application upgrades | Pipeline role, collection, consumer, and workspace | done | Closed after review and resolution of all three findings; public pipeline `37b1cf6`, public collection `b3bc045`, local consumer `15f0f4e`; [remediation evidence](roadmap/remediation.md) | | A0001 | Audit completed standalone-role test suites | workspace and standalone roles | done | Findings recorded in `docs/roadmap/test-suite-audit-findings.md` | +| F0023 | Adopt Narratio v3.1.0 and forward Scriptorium credentials | Narratio role, pipeline role, collection, consumer, and workspace | ready | Roadmap and [implementation plan](roadmap/implementation.md) approved; v3.1.0 tag verified; ready for implementation | | R0001 | Simplify completed standalone-role test suites | workspace and standalone roles | done | Role commits `e08c63c`, `d2d0b9d`, and `cfecf37` passed implementation and independent review | ## Updating This File