ansible-workspace-dnd/docs/roadmap/implementation.md

# Implementation Plan 0011: Managed Pipeline Credentials and Notarius Reuse

Status: ready

Target executor: GPT-5.6-Terra

Feature roadmap:
`docs/roadmap/features/0011-managed-pipeline-credential-delivery.md`

Active plan location: `docs/roadmap/implementation.md`

## Objective

Upgrade the generic Narratio role to v2.1.0 and expose its native Notarius
credential and cache controls, then have the D&D pipeline role deliver shared
provider credentials through Narratio's conventional secrets directory,
coordinate the Audita and Notarius credential references, and request
compatible Notarius checkpoint reuse.

The feature roadmap remains authoritative for scope, rationale, target end
state, and acceptance criteria. Decisions 0001 and 0002 remain authoritative
for direct Ansible convergence and the shared-runtime-group authorization
boundary.

## Execution Rules

- Execute stages in numeric order.
- Before each stage, confirm its preconditions and the working-tree state of
  its owning repository.
- Before each stage, reread every document listed in that stage's Required
  Policy Review. Do not rely on policy recalled from an earlier stage.
- Preserve unrelated and pre-existing changes. Stop if an owning repository is
  unexpectedly dirty at a stage boundary.
- Keep `ansible-role-narratio` generic. It exposes Narratio's native optional
  fields but does not select D&D credentials, manage secret files, or own
  suite cache policy.
- Keep credential delivery and cross-application policy in
  `ansible-role-dnd-pipeline`. Do not modify an application source repository
  or move suite policy into the Audita, Notarius, or Scriptorium roles.
- Never place a real credential in source, documentation, examples, test
  fixtures, command output, or commits. Tests may use distinctive synthetic
  values and must prove that those values do not appear in captured output.
- Treat a `dnd_pipeline_secrets` key as untrusted path input. Validate the
  complete mapping before any key is interpolated into a destination path.
  Accept only environment-variable names matching
  `^[A-Za-z_][A-Za-z0-9_]*$`; accept only nonempty strings or null as values.
- Apply `no_log: true` to every production task that evaluates, writes, or
  removes mapping entries. Do not expose a loop label containing a credential
  value.
- Use direct module-owned convergence: one fixed secrets-directory task, one
  validation loop, one copy loop for non-null entries, and one file loop for
  explicit null entries. Do not add preliminary `stat`, directory scans,
  recursive ownership repair, checksums, handlers, wrappers, manifests, state
  files, backups, or post-write validation tasks.
- An omitted credential key is unmanaged. Do not mirror the mapping onto the
  directory, purge unlisted files, or infer that an empty mapping removes
  anything.
- Keep ordinary application YAML secret-free. It may contain only credential
  reference names.
- Leave Narratio's `notarius.chunk_cache` unset in the pipeline role. The
  deployed Notarius root configuration remains the sole owner of ordinary
  `auto` chunk-plan reuse, while Narratio owns its native forced-refresh
  behavior.
- Test role-owned behavior through public role inputs and native application
  commands. Do not test Ansible module guarantees, inspect module results,
  assert task order, snapshot complete default documents, contact a live
  provider, or execute the application pipeline.
- Run every stage's validation before committing it. Commit each completed
  stage when it leaves its owning repository valid, using the plain-English
  commit intent specified by that stage.
- Do not begin a dependent stage after an earlier stage fails validation.
- The pipeline integration suite installs role dependencies from canonical Git
  origins. The Stage 1 Narratio-role commit must therefore be reachable from
  its canonical origin before Stage 2 begins. If a fresh Galaxy installation
  cannot resolve it, stop after Stage 1 and report the commit hash for
  publication; do not substitute a branch, local role path, or weakened test.

## Starting State and Preconditions

- The workspace control repository contains the reviewed Feature 0011 roadmap
  and this implementation plan in a clean planning commit on `main`.
- Features 0001 through 0010 and the standalone-role test-suite simplification
  are complete.
- The implementation repositories are clean on `main` at these reviewed
  starting commits:

  | Repository | Starting commit |
  | --- | --- |
  | `ansible-role-narratio` | `eeb1105590e638c1f46b42c98a5dbbce4ddeea86` |
  | `ansible-role-dnd-pipeline` | `bbf01e107a5e4e70a2abd6c696bfdb9ccc38bd7e` |

- The Narratio role matches `origin/main`. The pipeline role is intentionally
  ahead of its canonical origin and must not be reset or replaced.
- `../narratio` is clean at immutable tag `v2.1.0`, commit
  `c45e042462b77a8873b4215edc40e3840149db34`.
- Narratio v2.1.0 adds optional `notarius.llm_api_key`, `notarius.resume`, and
  `notarius.chunk_cache` fields without requiring migration of existing v2
  configuration. Its conventional secrets bootstrap loads immediate regular
  files from a sibling `secrets/` directory, gives an existing process
  environment value precedence, and requires every configured typed reference
  to resolve during configuration loading.
- Notarius v0.7.0 is already selected by the suite, records checkpoints under
  the persistent FHS cache root, and leaves ordinary chunk-plan mode at
  `auto`.
- PromptKit documents `OPENROUTER_API_KEY` as the credential name for the
  OpenRouter backend selected by the suite's production Notarius profile.
- The workspace `.venv` provides Ansible Core 2.21.3 and ansible-lint 26.8.0.
- The local integration environment remains an accepted proxy for Ubuntu
  26.04, supports task-level privilege escalation, and provides ordinary POSIX
  group and ACL behavior.

## Repository Scope

Repositories that change:

- `ansible-role-narratio`: Narratio v2.1.0 selection, three nullable native
  Notarius fields, generic documentation, and focused integration coverage.
- `ansible-role-dnd-pipeline`: compatible Narratio role and application pins,
  credential delivery, Audita and Notarius reference coordination, checkpoint
  reuse, consumer documentation, and focused integration coverage.
- `ansible-workspace-dnd`: implementation progress and review handoff only
  after the role stages are complete.

Repositories that may be inspected but must remain unchanged:

- `../narratio`, which is the read-only v2.1.0 application authority.
- `../notarius`, `../audita`, `../seriatim`, `../scriptorium`, and PromptKit,
  which may be consulted for current native contracts.
- `ansible-role-notarius`, `ansible-role-audita`,
  `ansible-role-seriatim`, and `ansible-role-scriptorium`.
- `ansible-collection-dnd`, all asset repositories, `rakestraw-ansible`, and
  every other repository in `workspace.yml`.

## Stage 1: Upgrade the Generic Narratio Role

### Objective

Make the standalone role install Narratio v2.1.0 and expose the release's
three optional Notarius fields without adding suite-specific credential or
cache policy.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/roadmap/features/0011-managed-pipeline-credential-delivery.md`
- `docs/decisions/0001-minimal-module-driven-ansible.md`
- `../narratio/docs/releases/v2.1.0.md`
- `../narratio/docs/config.md`
- `../narratio/docs/integrations/notarius.md`
- `../narratio/examples/pipeline.full.annotated.yml`
- `ansible-role-narratio/defaults/main.yml`
- `ansible-role-narratio/meta/argument_specs.yml`
- `ansible-role-narratio/templates/pipeline.yml.j2`
- `ansible-role-narratio/tests/integration`
- `ansible-role-narratio/tests/representative.yml`
- `ansible-role-narratio/tests/verify-baseline.yml`
- `ansible-role-narratio/tests/verify-config.yml`
- `ansible-role-narratio/README.md`

### Preconditions

- `ansible-role-narratio` is clean at its stated starting commit.
- Its existing `tests/integration` entry point passes before modification.
- The upstream files listed above remain at the reviewed immutable v2.1.0 tag.

### Owning Repository

`ansible-role-narratio`

### Required Changes

1. Advance `narratio_version` from `v2.0.0` to `v2.1.0`. Do not change the
   established source checkout, build condition, metadata convergence, or
   application-account task flow.
2. Add these optional public variables beside the existing Notarius variables,
   each defaulted to null:
   `narratio_notarius_llm_api_key`, `narratio_notarius_resume`, and
   `narratio_notarius_chunk_cache`.
3. Add matching optional argument-specification entries. Use `str` for the
   credential reference and chunk-cache mode and `bool` for resume. Do not
   duplicate Narratio's native enum or secret-resolution validation in the
   role.
4. Extend the existing `notarius` template section and its parent-emission
   condition. Render `llm_api_key`, `resume`, and `chunk_cache` only when their
   corresponding variable `is not none`; preserve intentional `false` for
   resume and serialize every value with the template's established filter.
5. Leave all three fields absent when null. Do not create a secrets directory,
   read an environment variable, accept a raw credential value, or select a
   Notarius provider in this standalone role.
6. Update the README's version, optional-variable list, native Notarius field
   discussion, and validation summary. Explain that the credential field is
   an environment-variable name and that credential delivery remains the
   caller's responsibility. Describe resume and chunk-cache as native
   application choices without assigning D&D defaults.
7. Update the integration suite to explicitly select v2.1.0 for the existing
   version-selection assertion. Continue testing version selection rather than
   asserting that v2.1.0 remains the default.
8. Extend the representative configuration case with a synthetic environment
   reference, `resume: false` as the falsey-value case, and one valid explicit
   chunk-cache mode. Supply a synthetic value for that named environment
   variable only when running Narratio's native validation or safe inspection;
   never put a credential value in managed YAML.
9. Observe the three representative fields and baseline null omission through
   the repository's existing focused configuration checks. Retain native
   `narratio config validate` coverage and immediate clean repeated
   convergence. Do not test ConfigKit's resolution precedence or Narratio's
   subprocess forwarding implementation.

### Validation

Run from `ansible-role-narratio`:

```text
tests/integration
git diff --check
git status --short
```

Expected result: syntax and production-profile lint pass; an explicitly
selected v2.1.0 binary reports that version; baseline null values remain
omitted; representative credential-name, false resume, and chunk-cache values
load through Narratio's native configuration interface; immediate repeats
report `changed=0` and `failed=0`; and only intended role, test, and README
files are modified.

### Completion Criteria

- Defaults, argument specification, template, README, and tests expose one
  coherent generic interface for the three v2.1.0 fields.
- The role owns no credential value, secrets directory, D&D provider choice,
  or suite cache policy.
- Tests exercise the public configuration interface without snapshotting the
  complete default document or re-testing application internals.
- The repository is valid and the stage is committed.
- The commit is reachable from the canonical Narratio-role origin before
  Stage 2 begins.

### Commit

Commit intent: `Upgrade the Narratio role to version 2.1.0`

## Stage 2: Implement Pipeline Credential Delivery and Reuse Policy

### Objective

Add the pipeline role's production credential-delivery tasks, coordinate its
Audita and Notarius references, and enable Narratio checkpoint reuse through
the reviewed Stage 1 interface.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/roadmap/features/0011-managed-pipeline-credential-delivery.md`
- `docs/decisions/0001-minimal-module-driven-ansible.md`
- `docs/decisions/0002-shared-readable-pipeline-filesystem-policy.md`
- the committed Stage 1 changes in `ansible-role-narratio`
- `../narratio/docs/releases/v2.1.0.md`
- `../narratio/docs/config.md`
- `../narratio/docs/integrations/audita.md`
- `../narratio/docs/integrations/notarius.md`
- `../narratio/docs/integrations/scriptorium.md`
- `ansible-role-dnd-pipeline/defaults/main.yml`
- `ansible-role-dnd-pipeline/meta/argument_specs.yml`
- `ansible-role-dnd-pipeline/meta/main.yml`
- `ansible-role-dnd-pipeline/requirements.yml`
- `ansible-role-dnd-pipeline/tasks/main.yml`
- `ansible-role-dnd-pipeline/tests/converge.yml`
- `ansible-role-dnd-pipeline/tests/integration`

### Preconditions

- Stage 1 is committed, validated, and clean.
- The Stage 1 commit is reachable from the canonical
  `ansible-role-narratio` Git origin. Stop and report the hash if a fresh
  Galaxy installation cannot resolve it.
- `ansible-role-dnd-pipeline` is clean at its stated starting commit or an
  expected reviewed descendant.
- Its existing `tests/integration` entry point passes before modification.

### Owning Repository

`ansible-role-dnd-pipeline`

### Required Changes

1. Set `dnd_pipeline_narratio_version: v2.1.0` and pin the
   `rakestraw.narratio` entry in `requirements.yml` to the exact Stage 1
   commit. Retain the canonical Git URL and all unrelated dependency pins.
2. Add `dnd_pipeline_secrets: {}` as a required, defaulted dictionary in the
   public interface. Its keys are credential filenames/environment-variable
   names; values are either nonempty strings to converge or null to remove the
   exact named file.
3. Add required, defaulted
   `dnd_pipeline_narratio_notarius_llm_api_key: OPENROUTER_API_KEY` as a string
   and `dnd_pipeline_narratio_notarius_resume: true` as a boolean. Do not add a
   suite variable for Narratio's optional chunk-cache override.
4. Change the suite default for
   `dnd_pipeline_audita_llm_validation_api_key_env` from
   `AUDITA_VALIDATION_LLM_API_KEY` to `AUDITA_LLM_API_KEY`. Retain the
   proposal and validation fields as independently overrideable public
   variables; do not add an equality assertion or promise split-key forwarding.
5. Pass `dnd_pipeline_audita_llm_proposal_api_key_env` through Narratio's
   `narratio_audita_llm_api_key` interface. Pass the new Notarius reference and
   resume variables through `narratio_notarius_llm_api_key` and
   `narratio_notarius_resume`. Do not pass
   `narratio_notarius_chunk_cache`; Notarius's root `auto` setting remains in
   control for ordinary runs.
6. Before any task interpolates a mapping key into a path, validate every
   `dnd_pipeline_secrets | dict2items` entry in one `ansible.builtin.assert`
   loop. Require a string key matching `^[A-Za-z_][A-Za-z0-9_]*$` and a value
   that is either null or a nonempty string. Apply `no_log: true` to the entire
   validation task. Do not normalize, trim, rename, or infer keys or values.
7. Create `/usr/local/etc/narratio/secrets` with one ordinary
   `ansible.builtin.file` task using the selected runtime user and group and
   fixed mode `0750`. Do not recursively alter existing entries.
8. Converge non-null mapping entries with one `ansible.builtin.copy` loop to
   `/usr/local/etc/narratio/secrets/<key>`, using the selected runtime user and
   group and fixed mode `0640`. Write only the supplied value; do not append
   metadata, encode it, back it up, or reread it. Apply `no_log: true`.
9. Converge null entries to absence with one `ansible.builtin.file` loop over
   the same mapping. Remove only the exact validated named file and apply
   `no_log: true`. An empty mapping and omitted keys must leave existing
   directory entries untouched.
10. Keep production convergence to those four operations: validation,
    directory creation, non-null copy, and explicit-null removal. Do not add
    `stat`, `find`, commands, handlers, environment files, shell profiles,
    launchers, services, providers, rotation, or role-private state.
11. Adapt the existing integration driver's baseline arguments to supply
    distinctive synthetic values for `AUDITA_LLM_API_KEY` and
    `OPENROUTER_API_KEY` through `dnd_pipeline_secrets`, with both ambient
    variables explicitly unset. Preserve isolated dependency installation and
    cleanup restoration.
12. Extend the existing native Narratio effective-configuration check to
    observe the two public credential reference names and `resume: true` while
    confirming that Narratio's `chunk_cache` override is absent. Retain the
    native Notarius check that its root chunk-plan mode is `auto`, the service
    and existing-group-member validation paths, and the baseline clean repeat.

### Validation

Run from `ansible-role-dnd-pipeline`:

```text
tests/integration
git diff --check
git status --short
```

Expected result: isolated role installation resolves the exact Stage 1
commit; syntax and production-profile lint pass; the role converges the
synthetic credential mapping without displaying its values; Narratio natively
loads the conventional secrets directory as both the service account and the
existing-group-member fixture; its safe effective configuration names the
Audita and Notarius references and enables resume without a chunk-cache
override; Notarius retains `auto`; the baseline repeat reports `changed=0` and
`failed=0`; and only intended pipeline-role files are modified.

### Completion Criteria

- The suite installs Narratio v2.1.0 through the exact reviewed role revision.
- One public mapping owns only explicitly named credential files, including
  explicit null removal, behind complete pre-path validation and secret-safe
  task output.
- Audita's suite defaults use one typed reference, Notarius uses
  `OPENROUTER_API_KEY`, and ordinary extraction requests checkpoint reuse.
- The pipeline does not duplicate Notarius's chunk-plan policy or add a second
  credential-loading mechanism.
- The existing integration suite passes and the repository is committed.

### Commit

Commit intent: `Add managed credentials to the D&D pipeline`

## Stage 3: Document and Exercise the Credential Contract

### Objective

Document the pipeline's credential and reuse policy and add the smallest
behavioral coverage for validation, ownership semantics, explicit removal,
redaction, authorization, and immediate idempotence.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/roadmap/features/0011-managed-pipeline-credential-delivery.md`
- `docs/decisions/0002-shared-readable-pipeline-filesystem-policy.md`
- the committed Stage 1 and Stage 2 changes
- `../narratio/docs/config.md`
- `../narratio/docs/cli.md`
- `../narratio/docs/integrations/notarius.md`
- `ansible-role-dnd-pipeline/README.md`
- `ansible-role-dnd-pipeline/tests/integration`
- `ansible-role-dnd-pipeline/tests/converge.yml`

### Preconditions

- Stages 1 and 2 are committed, validated, and clean.
- The pipeline's isolated dependency installation resolves the Stage 1 commit.
- The deployed baseline can be restored through the integration driver's
  existing cleanup path.

### Owning Repository

`ansible-role-dnd-pipeline`

### Required Changes

1. Update the README's compatible Narratio version, Audita validation
   credential default, public variable documentation, and obsolete statements
   that API-key management and checkpoint reuse are deferred.
2. Document `dnd_pipeline_secrets` as an empty-by-default mapping with valid
   environment-variable keys, nonempty string values for presence, null for
   exact removal, and omission for unmanaged entries. Explain that the role
   neither mirrors nor purges the directory.
3. Include an Ansible Vault inventory pattern that references encrypted
   variables without showing a real credential. State that Vault is a
   recommended consumer workflow, not functionality implemented by the role.
4. Document the fixed conventional path, `0750` directory and `0640` file
   modes, selected runtime ownership, runtime-group authorization meaning,
   process-environment precedence, and the absence of wrappers, services, or
   parallel loaders. Make clear that ordinary configuration remains
   world-readable and secret-free while the secrets subtree does not grant
   world access.
5. Document the coordinated `AUDITA_LLM_API_KEY` default, the typed
   `OPENROUTER_API_KEY` Notarius reference, Scriptorium's existing
   profile-owned reference behavior, ordinary checkpoint resume, Notarius's
   root-owned `auto` chunk cache, and Narratio's forced refresh behavior.
6. Add one representative expected-failure case for an invalid mapping key
   and one for an empty non-null value. Require each convergence to fail before
   any candidate destination is managed. Keep task output secret-safe, do not
   assert a task name or other implementation trace, and do not add an
   exhaustive validation matrix.
7. Exercise explicit-null removal with a uniquely named synthetic credential:
   first manage it, select its name through the public Notarius reference
   variable, and confirm native configuration loading succeeds. Then converge
   the same mapping entry to null with the relevant ambient variable unset.
   Observe unavailability through an expected failure from Narratio's native
   configuration loader rather than by inspecting a module result or file
   metadata.
8. Require an immediate identical repeat of the explicit-null state to report
   `changed=0` and `failed=0`, then restore the baseline credential mapping so
   cleanup and later tests remain deterministic.
9. Retain native configuration validation as the service account and the
   existing-group-member fixture. Add at most one representative native
   negative check showing that an identity outside the runtime group cannot
   load the credential-backed pipeline; do not build a permission matrix or
   assert numeric filesystem metadata.
10. Capture convergence and native inspection output and fail if either
    distinctive synthetic credential value appears. Names may appear in safe
    effective configuration; values must not.
11. Retain the existing baseline, testing-profile, selected-version,
    caller-composed Notarius pipeline, shared-cache, immutable local asset,
    and clean-repeat coverage. Do not contact an LLM, WhisperX, storage, or
    publishing endpoint; do not execute extraction or test ConfigKit,
    Narratio, Notarius, or trusted Ansible modules independently.

### Validation

Run from `ansible-role-dnd-pipeline`:

```text
tests/integration
git diff --check
git status --short
```

Expected result: isolated dependency installation, syntax checking, and
production-profile lint pass; valid synthetic credentials load through the
conventional directory for authorized identities; invalid names and empty
values fail safely; explicit null makes only the selected credential
unavailable through native loading and converges cleanly on repeat; no
synthetic value appears in captured output; all existing representative paths
continue to pass; and documentation matches the public interface.

### Completion Criteria

- Consumer documentation completely describes the mapping, Vault usage,
  authorization boundary, credential flows, cache policy, and exclusions.
- Tests protect role-owned validation, explicit ownership semantics,
  redaction, authorized use, and clean repetition without testing module or
  application internals.
- The default suite and representative explicit-null state each converge
  cleanly on immediate repeat.
- The full integration suite passes and the repository is committed.

### Commit

Commit intent: `Document and test pipeline credential delivery`

## Stage 4: Record the Implementation for Review

### Objective

Record the completed implementation and validation evidence in the workspace
without declaring the feature complete before independent review.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/planning.md`
- `docs/roadmap/features/0011-managed-pipeline-credential-delivery.md`
- `docs/roadmap/implementation.md`
- `docs/status.md`

### Preconditions

- Stages 1 through 3 are committed and their owning repositories are clean.
- Both complete integration entry points pass at the committed revisions.
- The workspace control repository is clean at the planning commit.

### Owning Repository

`ansible-workspace-dnd`

### Required Changes

1. Update this plan's Progress Notes with the three implementation commit
   hashes and concise validation results.
2. Update `docs/status.md` so Feature 0011 is `in_progress`, its evidence names
   the implementation commits, and the current focus is independent review of
   the completed implementation.
3. Keep the feature roadmap `ready`; do not mark Feature 0011 `done`, clean the
   durable roadmap, or create or revise `docs/roadmap/remediation.md`. Those
   actions belong to independent review and closeout.
4. Do not change implementation repositories or unrelated workspace policy in
   this stage.

### Validation

Run from the workspace root:

```text
git diff --check
./scripts/status
git status --short
```

Expected result: the status report identifies only the intended workspace
tracking edits before commit; both implementation repositories are clean; and
the dashboard and progress notes point to the exact completed stage commits.

### Completion Criteria

- Implementation and validation evidence is traceable across repositories.
- Feature 0011 remains awaiting independent review rather than being marked
  complete.
- The workspace control repository is valid and the stage is committed.

### Commit

Commit intent: `Record the pipeline credential implementation`

## Final Validation and Handoff

After Stage 4, rerun both owning repositories' complete integration entry
points:

```text
cd ansible-role-narratio && tests/integration
cd ../ansible-role-dnd-pipeline && tests/integration
```

Then, from the workspace root, run:

```text
git diff --check
./scripts/status
```

The final handoff must:

- identify the Stage 1 through Stage 4 commit hashes and confirm every changed
  repository has a clean working tree;
- confirm syntax and production-profile lint pass in both changed roles;
- confirm the standalone role installs an explicitly selected Narratio v2.1.0,
  exposes all three nullable native fields, validates representative values,
  and immediately converges cleanly;
- confirm the pipeline installs the exact immutable Narratio-role commit,
  loads managed synthetic credentials for authorized identities, rejects bad
  mapping entries before path use, removes an explicitly null entry, suppresses
  credential values, requests Notarius checkpoint reuse, leaves ordinary
  chunk-plan mode with Notarius, and immediately converges cleanly;
- confirm no upstream application, asset, collection, unrelated standalone
  role, real credential, external service, or real human account was modified;
- summarize any deviation, blocker, or deferred work, including any pause
  required to publish the Stage 1 dependency commit;
- report any production complexity beyond the three generic Narratio fields,
  suite variables and dependency wiring, and the four direct credential tasks,
  together with the acceptance criterion that required it; and
- leave Feature 0011 `in_progress` for independent review rather than marking
  it `done`.

## Progress Notes

- Planning complete; implementation has not begun.