ansible-workspace-dnd/docs/roadmap/implementation.md

Implementation Plan 0011: Managed Pipeline Credentials and Notarius Reuse

Status: ready

Target executor: GPT-5.6-Terra

Feature roadmap: docs/roadmap/features/0011-managed-pipeline-credential-delivery.md

Active plan location: docs/roadmap/implementation.md

Objective

Upgrade the generic Narratio role to v2.1.0 and expose its native Notarius credential and cache controls, then have the D&D pipeline role deliver shared provider credentials through Narratio's conventional secrets directory, coordinate the Audita and Notarius credential references, and request compatible Notarius checkpoint reuse.

The feature roadmap remains authoritative for scope, rationale, target end state, and acceptance criteria. Decisions 0001 and 0002 remain authoritative for direct Ansible convergence and the shared-runtime-group authorization boundary.

Execution Rules

• Execute stages in numeric order.
• Before each stage, confirm its preconditions and the working-tree state of its owning repository.
• Before each stage, reread every document listed in that stage's Required Policy Review. Do not rely on policy recalled from an earlier stage.
• Preserve unrelated and pre-existing changes. Stop if an owning repository is unexpectedly dirty at a stage boundary.
• Keep ansible-role-narratio generic. It exposes Narratio's native optional fields but does not select D&D credentials, manage secret files, or own suite cache policy.
• Keep credential delivery and cross-application policy in ansible-role-dnd-pipeline. Do not modify an application source repository or move suite policy into the Audita, Notarius, or Scriptorium roles.
• Never place a real credential in source, documentation, examples, test fixtures, command output, or commits. Tests may use distinctive synthetic values and must prove that those values do not appear in captured output.
• Treat a dnd_pipeline_secrets key as untrusted path input. Validate the complete mapping before any key is interpolated into a destination path. Accept only environment-variable names matching ^[A-Za-z_][A-Za-z0-9_]*$; accept only nonempty strings or null as values.
• Apply no_log: true to every production task that evaluates, writes, or removes mapping entries. Do not expose a loop label containing a credential value.
• Use direct module-owned convergence: one fixed secrets-directory task, one validation loop, one copy loop for non-null entries, and one file loop for explicit null entries. Do not add preliminary stat, directory scans, recursive ownership repair, checksums, handlers, wrappers, manifests, state files, backups, or post-write validation tasks.
• An omitted credential key is unmanaged. Do not mirror the mapping onto the directory, purge unlisted files, or infer that an empty mapping removes anything.
• Keep ordinary application YAML secret-free. It may contain only credential reference names.
• Leave Narratio's notarius.chunk_cache unset in the pipeline role. The deployed Notarius root configuration remains the sole owner of ordinary auto chunk-plan reuse, while Narratio owns its native forced-refresh behavior.
• Test role-owned behavior through public role inputs and native application commands. Do not test Ansible module guarantees, inspect module results, assert task order, snapshot complete default documents, contact a live provider, or execute the application pipeline.
• Run every stage's validation before committing it. Commit each completed stage when it leaves its owning repository valid, using the plain-English commit intent specified by that stage.
• Do not begin a dependent stage after an earlier stage fails validation.
• The pipeline integration suite installs role dependencies from canonical Git origins. The Stage 1 Narratio-role commit must therefore be reachable from its canonical origin before Stage 2 begins. If a fresh Galaxy installation cannot resolve it, stop after Stage 1 and report the commit hash for publication; do not substitute a branch, local role path, or weakened test.

Starting State and Preconditions

• The workspace control repository contains the reviewed Feature 0011 roadmap and this implementation plan in a clean planning commit on main.

• Features 0001 through 0010 and the standalone-role test-suite simplification are complete.

• The implementation repositories are clean on main at these reviewed starting commits:

  Repository	Starting commit
  ansible-role-narratio	eeb1105590e638c1f46b42c98a5dbbce4ddeea86
  ansible-role-dnd-pipeline	bbf01e107a5e4e70a2abd6c696bfdb9ccc38bd7e

• The Narratio role matches origin/main. The pipeline role is intentionally ahead of its canonical origin and must not be reset or replaced.

• ../narratio is clean at immutable tag v2.1.0, commit c45e042462b77a8873b4215edc40e3840149db34.

• Narratio v2.1.0 adds optional notarius.llm_api_key, notarius.resume, and notarius.chunk_cache fields without requiring migration of existing v2 configuration. Its conventional secrets bootstrap loads immediate regular files from a sibling secrets/ directory, gives an existing process environment value precedence, and requires every configured typed reference to resolve during configuration loading.

• Notarius v0.7.0 is already selected by the suite, records checkpoints under the persistent FHS cache root, and leaves ordinary chunk-plan mode at auto.

• PromptKit documents OPENROUTER_API_KEY as the credential name for the OpenRouter backend selected by the suite's production Notarius profile.

• The workspace .venv provides Ansible Core 2.21.3 and ansible-lint 26.8.0.

• The local integration environment remains an accepted proxy for Ubuntu 26.04, supports task-level privilege escalation, and provides ordinary POSIX group and ACL behavior.

Repository Scope

Repositories that change:

• ansible-role-narratio: Narratio v2.1.0 selection, three nullable native Notarius fields, generic documentation, and focused integration coverage.
• ansible-role-dnd-pipeline: compatible Narratio role and application pins, credential delivery, Audita and Notarius reference coordination, checkpoint reuse, consumer documentation, and focused integration coverage.
• ansible-workspace-dnd: implementation progress and review handoff only after the role stages are complete.

Repositories that may be inspected but must remain unchanged:

• ../narratio, which is the read-only v2.1.0 application authority.
• ../notarius, ../audita, ../seriatim, ../scriptorium, and PromptKit, which may be consulted for current native contracts.
• ansible-role-notarius, ansible-role-audita, ansible-role-seriatim, and ansible-role-scriptorium.
• ansible-collection-dnd, all asset repositories, rakestraw-ansible, and every other repository in workspace.yml.

Stage 1: Upgrade the Generic Narratio Role

Objective

Make the standalone role install Narratio v2.1.0 and expose the release's three optional Notarius fields without adding suite-specific credential or cache policy.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/roadmap/features/0011-managed-pipeline-credential-delivery.md
• docs/decisions/0001-minimal-module-driven-ansible.md
• ../narratio/docs/releases/v2.1.0.md
• ../narratio/docs/config.md
• ../narratio/docs/integrations/notarius.md
• ../narratio/examples/pipeline.full.annotated.yml
• ansible-role-narratio/defaults/main.yml
• ansible-role-narratio/meta/argument_specs.yml
• ansible-role-narratio/templates/pipeline.yml.j2
• ansible-role-narratio/tests/integration
• ansible-role-narratio/tests/representative.yml
• ansible-role-narratio/tests/verify-baseline.yml
• ansible-role-narratio/tests/verify-config.yml
• ansible-role-narratio/README.md

Preconditions

• ansible-role-narratio is clean at its stated starting commit.
• Its existing tests/integration entry point passes before modification.
• The upstream files listed above remain at the reviewed immutable v2.1.0 tag.

Owning Repository

ansible-role-narratio

Required Changes

1. Advance narratio_version from v2.0.0 to v2.1.0. Do not change the established source checkout, build condition, metadata convergence, or application-account task flow.
2. Add these optional public variables beside the existing Notarius variables, each defaulted to null: narratio_notarius_llm_api_key, narratio_notarius_resume, and narratio_notarius_chunk_cache.
3. Add matching optional argument-specification entries. Use str for the credential reference and chunk-cache mode and bool for resume. Do not duplicate Narratio's native enum or secret-resolution validation in the role.
4. Extend the existing notarius template section and its parent-emission condition. Render llm_api_key, resume, and chunk_cache only when their corresponding variable is not none; preserve intentional false for resume and serialize every value with the template's established filter.
5. Leave all three fields absent when null. Do not create a secrets directory, read an environment variable, accept a raw credential value, or select a Notarius provider in this standalone role.
6. Update the README's version, optional-variable list, native Notarius field discussion, and validation summary. Explain that the credential field is an environment-variable name and that credential delivery remains the caller's responsibility. Describe resume and chunk-cache as native application choices without assigning D&D defaults.
7. Update the integration suite to explicitly select v2.1.0 for the existing version-selection assertion. Continue testing version selection rather than asserting that v2.1.0 remains the default.
8. Extend the representative configuration case with a synthetic environment reference, resume: false as the falsey-value case, and one valid explicit chunk-cache mode. Supply a synthetic value for that named environment variable only when running Narratio's native validation or safe inspection; never put a credential value in managed YAML.
9. Observe the three representative fields and baseline null omission through the repository's existing focused configuration checks. Retain native narratio config validate coverage and immediate clean repeated convergence. Do not test ConfigKit's resolution precedence or Narratio's subprocess forwarding implementation.

Validation

Run from ansible-role-narratio:

tests/integration
git diff --check
git status --short

Expected result: syntax and production-profile lint pass; an explicitly selected v2.1.0 binary reports that version; baseline null values remain omitted; representative credential-name, false resume, and chunk-cache values load through Narratio's native configuration interface; immediate repeats report changed=0 and failed=0; and only intended role, test, and README files are modified.

Completion Criteria

• Defaults, argument specification, template, README, and tests expose one coherent generic interface for the three v2.1.0 fields.
• The role owns no credential value, secrets directory, D&D provider choice, or suite cache policy.
• Tests exercise the public configuration interface without snapshotting the complete default document or re-testing application internals.
• The repository is valid and the stage is committed.
• The commit is reachable from the canonical Narratio-role origin before Stage 2 begins.

Commit

Commit intent: Upgrade the Narratio role to version 2.1.0

Stage 2: Implement Pipeline Credential Delivery and Reuse Policy

Objective

Add the pipeline role's production credential-delivery tasks, coordinate its Audita and Notarius references, and enable Narratio checkpoint reuse through the reviewed Stage 1 interface.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/roadmap/features/0011-managed-pipeline-credential-delivery.md
• docs/decisions/0001-minimal-module-driven-ansible.md
• docs/decisions/0002-shared-readable-pipeline-filesystem-policy.md
• the committed Stage 1 changes in ansible-role-narratio
• ../narratio/docs/releases/v2.1.0.md
• ../narratio/docs/config.md
• ../narratio/docs/integrations/audita.md
• ../narratio/docs/integrations/notarius.md
• ../narratio/docs/integrations/scriptorium.md
• ansible-role-dnd-pipeline/defaults/main.yml
• ansible-role-dnd-pipeline/meta/argument_specs.yml
• ansible-role-dnd-pipeline/meta/main.yml
• ansible-role-dnd-pipeline/requirements.yml
• ansible-role-dnd-pipeline/tasks/main.yml
• ansible-role-dnd-pipeline/tests/converge.yml
• ansible-role-dnd-pipeline/tests/integration

Preconditions

• Stage 1 is committed, validated, and clean.
• The Stage 1 commit is reachable from the canonical ansible-role-narratio Git origin. Stop and report the hash if a fresh Galaxy installation cannot resolve it.
• ansible-role-dnd-pipeline is clean at its stated starting commit or an expected reviewed descendant.
• Its existing tests/integration entry point passes before modification.

Owning Repository

ansible-role-dnd-pipeline

Required Changes

1. Set dnd_pipeline_narratio_version: v2.1.0 and pin the rakestraw.narratio entry in requirements.yml to the exact Stage 1 commit. Retain the canonical Git URL and all unrelated dependency pins.
2. Add dnd_pipeline_secrets: {} as a required, defaulted dictionary in the public interface. Its keys are credential filenames/environment-variable names; values are either nonempty strings to converge or null to remove the exact named file.
3. Add required, defaulted dnd_pipeline_narratio_notarius_llm_api_key: OPENROUTER_API_KEY as a string and dnd_pipeline_narratio_notarius_resume: true as a boolean. Do not add a suite variable for Narratio's optional chunk-cache override.
4. Change the suite default for dnd_pipeline_audita_llm_validation_api_key_env from AUDITA_VALIDATION_LLM_API_KEY to AUDITA_LLM_API_KEY. Retain the proposal and validation fields as independently overrideable public variables; do not add an equality assertion or promise split-key forwarding.
5. Pass dnd_pipeline_audita_llm_proposal_api_key_env through Narratio's narratio_audita_llm_api_key interface. Pass the new Notarius reference and resume variables through narratio_notarius_llm_api_key and narratio_notarius_resume. Do not pass narratio_notarius_chunk_cache; Notarius's root auto setting remains in control for ordinary runs.
6. Before any task interpolates a mapping key into a path, validate every dnd_pipeline_secrets | dict2items entry in one ansible.builtin.assert loop. Require a string key matching ^[A-Za-z_][A-Za-z0-9_]*$ and a value that is either null or a nonempty string. Apply no_log: true to the entire validation task. Do not normalize, trim, rename, or infer keys or values.
7. Create /usr/local/etc/narratio/secrets with one ordinary ansible.builtin.file task using the selected runtime user and group and fixed mode 0750. Do not recursively alter existing entries.
8. Converge non-null mapping entries with one ansible.builtin.copy loop to /usr/local/etc/narratio/secrets/<key>, using the selected runtime user and group and fixed mode 0640. Write only the supplied value; do not append metadata, encode it, back it up, or reread it. Apply no_log: true.
9. Converge null entries to absence with one ansible.builtin.file loop over the same mapping. Remove only the exact validated named file and apply no_log: true. An empty mapping and omitted keys must leave existing directory entries untouched.
10. Keep production convergence to those four operations: validation, directory creation, non-null copy, and explicit-null removal. Do not add stat, find, commands, handlers, environment files, shell profiles, launchers, services, providers, rotation, or role-private state.
11. Adapt the existing integration driver's baseline arguments to supply distinctive synthetic values for AUDITA_LLM_API_KEY and OPENROUTER_API_KEY through dnd_pipeline_secrets, with both ambient variables explicitly unset. Preserve isolated dependency installation and cleanup restoration.
12. Extend the existing native Narratio effective-configuration check to observe the two public credential reference names and resume: true while confirming that Narratio's chunk_cache override is absent. Retain the native Notarius check that its root chunk-plan mode is auto, the service and existing-group-member validation paths, and the baseline clean repeat.

Validation

Run from ansible-role-dnd-pipeline:

tests/integration
git diff --check
git status --short

Expected result: isolated role installation resolves the exact Stage 1 commit; syntax and production-profile lint pass; the role converges the synthetic credential mapping without displaying its values; Narratio natively loads the conventional secrets directory as both the service account and the existing-group-member fixture; its safe effective configuration names the Audita and Notarius references and enables resume without a chunk-cache override; Notarius retains auto; the baseline repeat reports changed=0 and failed=0; and only intended pipeline-role files are modified.

Completion Criteria

• The suite installs Narratio v2.1.0 through the exact reviewed role revision.
• One public mapping owns only explicitly named credential files, including explicit null removal, behind complete pre-path validation and secret-safe task output.
• Audita's suite defaults use one typed reference, Notarius uses OPENROUTER_API_KEY, and ordinary extraction requests checkpoint reuse.
• The pipeline does not duplicate Notarius's chunk-plan policy or add a second credential-loading mechanism.
• The existing integration suite passes and the repository is committed.

Commit

Commit intent: Add managed credentials to the D&D pipeline

Stage 3: Document and Exercise the Credential Contract

Objective

Document the pipeline's credential and reuse policy and add the smallest behavioral coverage for validation, ownership semantics, explicit removal, redaction, authorization, and immediate idempotence.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/roadmap/features/0011-managed-pipeline-credential-delivery.md
• docs/decisions/0002-shared-readable-pipeline-filesystem-policy.md
• the committed Stage 1 and Stage 2 changes
• ../narratio/docs/config.md
• ../narratio/docs/cli.md
• ../narratio/docs/integrations/notarius.md
• ansible-role-dnd-pipeline/README.md
• ansible-role-dnd-pipeline/tests/integration
• ansible-role-dnd-pipeline/tests/converge.yml

Preconditions

• Stages 1 and 2 are committed, validated, and clean.
• The pipeline's isolated dependency installation resolves the Stage 1 commit.
• The deployed baseline can be restored through the integration driver's existing cleanup path.

Owning Repository

ansible-role-dnd-pipeline

Required Changes

1. Update the README's compatible Narratio version, Audita validation credential default, public variable documentation, and obsolete statements that API-key management and checkpoint reuse are deferred.
2. Document dnd_pipeline_secrets as an empty-by-default mapping with valid environment-variable keys, nonempty string values for presence, null for exact removal, and omission for unmanaged entries. Explain that the role neither mirrors nor purges the directory.
3. Include an Ansible Vault inventory pattern that references encrypted variables without showing a real credential. State that Vault is a recommended consumer workflow, not functionality implemented by the role.
4. Document the fixed conventional path, 0750 directory and 0640 file modes, selected runtime ownership, runtime-group authorization meaning, process-environment precedence, and the absence of wrappers, services, or parallel loaders. Make clear that ordinary configuration remains world-readable and secret-free while the secrets subtree does not grant world access.
5. Document the coordinated AUDITA_LLM_API_KEY default, the typed OPENROUTER_API_KEY Notarius reference, Scriptorium's existing profile-owned reference behavior, ordinary checkpoint resume, Notarius's root-owned auto chunk cache, and Narratio's forced refresh behavior.
6. Add one representative expected-failure case for an invalid mapping key and one for an empty non-null value. Require each convergence to fail before any candidate destination is managed. Keep task output secret-safe, do not assert a task name or other implementation trace, and do not add an exhaustive validation matrix.
7. Exercise explicit-null removal with a uniquely named synthetic credential: first manage it, select its name through the public Notarius reference variable, and confirm native configuration loading succeeds. Then converge the same mapping entry to null with the relevant ambient variable unset. Observe unavailability through an expected failure from Narratio's native configuration loader rather than by inspecting a module result or file metadata.
8. Require an immediate identical repeat of the explicit-null state to report changed=0 and failed=0, then restore the baseline credential mapping so cleanup and later tests remain deterministic.
9. Retain native configuration validation as the service account and the existing-group-member fixture. Add at most one representative native negative check showing that an identity outside the runtime group cannot load the credential-backed pipeline; do not build a permission matrix or assert numeric filesystem metadata.
10. Capture convergence and native inspection output and fail if either distinctive synthetic credential value appears. Names may appear in safe effective configuration; values must not.
11. Retain the existing baseline, testing-profile, selected-version, caller-composed Notarius pipeline, shared-cache, immutable local asset, and clean-repeat coverage. Do not contact an LLM, WhisperX, storage, or publishing endpoint; do not execute extraction or test ConfigKit, Narratio, Notarius, or trusted Ansible modules independently.

Validation

Run from ansible-role-dnd-pipeline:

tests/integration
git diff --check
git status --short

Expected result: isolated dependency installation, syntax checking, and production-profile lint pass; valid synthetic credentials load through the conventional directory for authorized identities; invalid names and empty values fail safely; explicit null makes only the selected credential unavailable through native loading and converges cleanly on repeat; no synthetic value appears in captured output; all existing representative paths continue to pass; and documentation matches the public interface.

Completion Criteria

• Consumer documentation completely describes the mapping, Vault usage, authorization boundary, credential flows, cache policy, and exclusions.
• Tests protect role-owned validation, explicit ownership semantics, redaction, authorized use, and clean repetition without testing module or application internals.
• The default suite and representative explicit-null state each converge cleanly on immediate repeat.
• The full integration suite passes and the repository is committed.

Commit

Commit intent: Document and test pipeline credential delivery

Stage 4: Record the Implementation for Review

Objective

Record the completed implementation and validation evidence in the workspace without declaring the feature complete before independent review.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/planning.md
• docs/roadmap/features/0011-managed-pipeline-credential-delivery.md
• docs/roadmap/implementation.md
• docs/status.md

Preconditions

• Stages 1 through 3 are committed and their owning repositories are clean.
• Both complete integration entry points pass at the committed revisions.
• The workspace control repository is clean at the planning commit.

Owning Repository

ansible-workspace-dnd

Required Changes

1. Update this plan's Progress Notes with the three implementation commit hashes and concise validation results.
2. Update docs/status.md so Feature 0011 is in_progress, its evidence names the implementation commits, and the current focus is independent review of the completed implementation.
3. Keep the feature roadmap ready; do not mark Feature 0011 done, clean the durable roadmap, or create or revise docs/roadmap/remediation.md. Those actions belong to independent review and closeout.
4. Do not change implementation repositories or unrelated workspace policy in this stage.

Validation

Run from the workspace root:

git diff --check
./scripts/status
git status --short

Expected result: the status report identifies only the intended workspace tracking edits before commit; both implementation repositories are clean; and the dashboard and progress notes point to the exact completed stage commits.

Completion Criteria

• Implementation and validation evidence is traceable across repositories.
• Feature 0011 remains awaiting independent review rather than being marked complete.
• The workspace control repository is valid and the stage is committed.

Commit

Commit intent: Record the pipeline credential implementation

Final Validation and Handoff

After Stage 4, rerun both owning repositories' complete integration entry points:

cd ansible-role-narratio && tests/integration
cd ../ansible-role-dnd-pipeline && tests/integration

Then, from the workspace root, run:

git diff --check
./scripts/status

The final handoff must:

• identify the Stage 1 through Stage 4 commit hashes and confirm every changed repository has a clean working tree;
• confirm syntax and production-profile lint pass in both changed roles;
• confirm the standalone role installs an explicitly selected Narratio v2.1.0, exposes all three nullable native fields, validates representative values, and immediately converges cleanly;
• confirm the pipeline installs the exact immutable Narratio-role commit, loads managed synthetic credentials for authorized identities, rejects bad mapping entries before path use, removes an explicitly null entry, suppresses credential values, requests Notarius checkpoint reuse, leaves ordinary chunk-plan mode with Notarius, and immediately converges cleanly;
• confirm no upstream application, asset, collection, unrelated standalone role, real credential, external service, or real human account was modified;
• summarize any deviation, blocker, or deferred work, including any pause required to publish the Stage 1 dependency commit;
• report any production complexity beyond the three generic Narratio fields, suite variables and dependency wiring, and the four direct credential tasks, together with the acceptance criterion that required it; and
• leave Feature 0011 in_progress for independent review rather than marking it done.

Progress Notes

• Stage 1 complete: 292d8669e7aad5cfe80b0c469d22b6177e8cf0f1 (Upgrade the Narratio role to version 2.1.0). The Narratio integration suite passed syntax and production-profile lint, explicit v2.1.0 selection, native representative validation, redaction of the synthetic test value, and immediate clean baseline and representative repeats. The commit is published on the canonical Narratio-role origin.