Prepare the managed credential implementation sprint

docs/roadmap/features/0011-managed-pipeline-credential-delivery.md

@@ -0,0 +1,392 @@
+# Feature 0011: Managed Pipeline Credentials and Notarius Reuse
+
+Status: ready
+
+## Summary
+
+Upgrade the generic Narratio role and D&D pipeline to Narratio v2.1.0, manage
+provider credentials through Narratio's conventional ConfigKit secrets
+directory, and use the release's typed Notarius credential and checkpoint-reuse
+controls. Credential values remain outside ordinary application YAML and are
+readable only by the selected runtime user and group. Ordinary pipeline runs
+reuse compatible Notarius checkpoints, while forced extraction retains
+Narratio's native refresh behavior.
+
+## Problem or Opportunity
+
+The pipeline role currently writes credential reference names but manages no
+credential values. A caller must prepare the invoking environment through an
+external mechanism before a real run can reach the selected model providers.
+
+The installed Narratio v2.0.0 release also cannot forward a selected provider
+credential to Notarius. Its Notarius child receives a restricted environment,
+and the v2.0 configuration contract exposes no typed credential field. The
+pipeline records Notarius checkpoints but cannot ask Notarius to reuse them.
+
+Narratio v2.1.0, immutable tag `v2.1.0` at `c45e042`, resolves both application
+contract gaps. It adds optional `notarius.llm_api_key`, `notarius.resume`, and
+`notarius.chunk_cache` fields without requiring migration of existing v2
+configuration. The credential field resolves through the existing process
+environment or secrets-directory bootstrap, forwards only the exact selected
+name and value, and participates in the application's existing redaction
+boundary. Ordinary extraction may request checkpoint reuse; forced extraction
+omits resume and refreshes the active chunk plan.
+
+The suite separately configures Audita proposal and validation with different
+environment names even though both use the same endpoint and model. Narratio's
+Audita boundary accepts one typed credential. This feature therefore
+coordinates both suite-default Audita sections on `AUDITA_LLM_API_KEY` while
+retaining their existing independently overrideable variables.
+
+## Governing Policies
+
+- [The Ansible house style](../../ansible.md) governs the direct desired-state
+  implementation, native configuration interface, and ordinary module
+  convergence.
+- [The testing house style](../../testing.md) governs synthetic, native,
+  behavior-oriented validation without live provider calls or filesystem
+  metadata assertions.
+- [The MVP roadmap](../mvp.md) assigns pipeline-wide secret delivery,
+  application-version coordination, and cache policy to the suite role.
+- [Decision 0002](../../decisions/0002-shared-readable-pipeline-filesystem-policy.md)
+  defines the selected runtime group as the authorization boundary and fixes
+  `0750` directory and `0640` file modes for shared credential material.
+- [Feature 0009](0009-shared-pipeline-filesystem-and-execution.md) establishes
+  the service-user and existing-group-member invocation model.
+- [Feature 0010](0010-dnd-notarius-extraction-integration.md) establishes the
+  Notarius pipeline, FHS cache roots, automatic chunk-plan reuse, and checkpoint
+  recording extended by this feature.
+- Narratio's immutable v2.1.0 release documentation and current configuration
+  and Notarius integration references are authoritative for the application
+  fields, resolution semantics, forced-run behavior, and redaction contract.
+
+## Scope
+
+- Advance the standalone Narratio role's selected application release from
+  v2.0.0 to v2.1.0 without changing its established source-build task flow.
+- Expose the three new native optional fields through nullable, field-specific
+  role variables: `narratio_notarius_llm_api_key`,
+  `narratio_notarius_resume`, and `narratio_notarius_chunk_cache`.
+- Keep the standalone role generic. All three variables default to null and
+  are omitted unless a caller selects them, leaving Narratio's native defaults
+  in control.
+- Advance the pipeline role's compatible Narratio application version to
+  v2.1.0 and pin its Narratio role dependency to the reviewed revision that
+  provides the new interface.
+- Use `/usr/local/etc/narratio/secrets`, the conventional directory adjacent
+  to `/usr/local/etc/narratio/pipeline.yml`, as the suite's sole managed
+  credential location.
+- Create that directory as the selected pipeline runtime user and group with
+  mode `0750`.
+- Add one public `dnd_pipeline_secrets` mapping. Keys are environment-variable
+  names. A non-null string value means that the named credential file must be
+  present with that content; null means that the specifically named managed
+  credential file must be absent. An omitted key is unmanaged.
+- Default `dnd_pipeline_secrets` to an empty mapping so the role never invents
+  credentials and remains usable with externally prepared process environment
+  values.
+- Validate selected keys as environment-variable names and reject empty
+  non-null values before constructing a destination path.
+- Write non-null credential files as the selected runtime user and group with
+  mode `0640`, and remove only names explicitly assigned null.
+- Suppress credential values from Ansible task output during validation,
+  creation, updates, and removal.
+- Keep ordinary application configuration limited to non-secret credential
+  reference names.
+- Coordinate Audita proposal and validation on the suite-default
+  `AUDITA_LLM_API_KEY` name and pass that name through the standalone Narratio
+  role's existing `narratio_audita_llm_api_key` interface.
+- Add `dnd_pipeline_narratio_notarius_llm_api_key`, default it to
+  `OPENROUTER_API_KEY` for Notarius's selected production profile, and pass it
+  through Narratio's typed Notarius credential field.
+- Add `dnd_pipeline_narratio_notarius_resume`, default it to true, and pass it
+  through Narratio so ordinary extraction requests reuse of the checkpoints
+  already recorded by the suite's Notarius configuration.
+- Expose Narratio's native `notarius.chunk_cache` in the standalone role but do
+  not duplicate it behind a suite variable or set it from the pipeline role.
+  The suite already configures Notarius's ordinary chunk-plan mode as `auto`;
+  Narratio v2.1.0 independently owns its forced-extraction refresh override.
+- Preserve Scriptorium's application-owned `api_key_env` profile selection.
+  Narratio's secrets bootstrap supplies the selected environment value and the
+  existing Scriptorium adapter passes that named credential to its child.
+- Document the mapping interface, Ansible Vault inventory pattern, process
+  environment precedence, shared-group authorization boundary, explicit null
+  removal, application credential flow, and checkpoint-reuse policy.
+- Add focused integration coverage using distinctive synthetic values and
+  native Narratio configuration loading. No default test contacts a model
+  provider.
+
+## Non-goals
+
+- Storing raw credentials in Narratio, Audita, Notarius, or Scriptorium YAML.
+- Providing real credentials, unencrypted example values, or a default secret
+  mapping.
+- Implementing Ansible Vault, choosing a vault password mechanism, or requiring
+  one specific inventory encryption workflow.
+- Integrating an external secret manager, cloud vault, hardware-backed key
+  store, or dynamic credential broker.
+- Making the pipeline role authoritative for files omitted from
+  `dnd_pipeline_secrets`, purging the whole directory, or recursively changing
+  pre-existing content.
+- Adding a launcher script, shell profile, environment file, systemd unit,
+  scheduler, or persistent service merely to inject credentials.
+- Passing arbitrary caller-defined environment variables through Narratio's
+  subprocess boundary.
+- Managing human accounts or their membership in the runtime group.
+- Running Audita, Notarius, or Scriptorium against a live endpoint during the
+  default integration suites.
+- Testing Notarius checkpoint reuse end to end in the Ansible role. Narratio
+  and Notarius own that behavior; the role validates the selected native
+  configuration and immediate Ansible convergence.
+- Setting Narratio's optional `notarius.chunk_cache` field redundantly when
+  the deployed Notarius configuration already selects `auto`.
+- Changing Narratio's native forced-extraction refresh policy.
+- Rotating credentials, retaining prior values, validating provider
+  authorization, or recovering deliberately corrupted secret files.
+- Adding storage or publishing credentials before the suite selects a storage
+  or publishing feature.
+- Modifying any upstream application source.
+
+## Target End State
+
+The standalone Narratio role installs v2.1.0 by default and can render all
+three new Notarius fields when a caller supplies them. It owns no credential
+files and selects no D&D-specific credential or cache policy.
+
+The pipeline role owns this conventional restricted subtree:
+
+```text
+/usr/local/etc/narratio/
+├── pipeline.yml
+└── secrets/                  narratio:narratio 0750
+    ├── AUDITA_LLM_API_KEY    narratio:narratio 0640
+    └── OPENROUTER_API_KEY    narratio:narratio 0640
+```
+
+The filenames are suite-selected reference names, never credential values.
+The selected runtime user and group remain configurable through the existing
+pipeline interface.
+
+A consumer may keep encrypted values in inventory and select explicit removal
+without embedding any value in ordinary configuration:
+
+```yaml
+dnd_pipeline_secrets:
+  AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
+  OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
+  RETIRED_PROVIDER_KEY: null
+```
+
+The role owns only these three named entries for this invocation. It converges
+the first two to regular files, removes the third, and leaves every omitted
+name alone. The actual `vault_dnd_*` values belong in encrypted inventory and
+must not appear in examples, plans, test logs, commits, or ordinary generated
+configuration.
+
+When Narratio runs without `--secrets-dir`, it discovers the sibling directory
+and loads its immediate regular files before resolving the pipeline. An
+already-set process environment value wins over the corresponding file. The
+base and `testing` profiles use the same delivery mechanism; profiles select
+models and application profiles, not a second credential store.
+
+The suite deliberately uses `narratio:narratio`, directory mode `0750`, and
+file mode `0640` rather than Narratio's general single-user recommendation of
+`0700` and `0600`. Membership in the selected runtime group is an explicit
+authorization to invoke this pipeline with its shared provider credentials.
+No world access is granted. Deployments that do not trust every group member
+must use separate process-environment or `--secrets-dir` provisioning and are
+outside this shared-operator policy.
+
+Audita's proposal and validation configuration both reference
+`AUDITA_LLM_API_KEY` by default. Narratio resolves that typed reference and
+passes the value explicitly to the Audita child. Callers may still override
+the existing Audita reference variables, but a custom split-key design also
+requires an application boundary capable of forwarding both values and is not
+promised by this feature.
+
+Narratio's effective Notarius configuration names `OPENROUTER_API_KEY`, or a
+caller-selected replacement, through `notarius.llm_api_key`. Narratio resolves
+the named secret and forwards only that exact name and value to Notarius. It
+does not infer a provider from Notarius configuration or forward unrelated
+credentials.
+
+Ordinary extraction sets `notarius.resume: true`. Notarius continues to own
+checkpoint recording, identity, and storage beneath the configured
+`/var/cache/notarius/checkpoints` root. Narratio requests reuse but does not
+inspect the cache. The pipeline leaves Narratio's `notarius.chunk_cache` field
+unset because the Notarius root configuration already selects `auto` at the
+native owner. Forced extraction follows Narratio v2.1.0 behavior: resume is
+omitted and the active chunk plan is refreshed without deleting unrelated
+cache content.
+
+Scriptorium continues to select credential names through its PromptKit
+profiles. The Narratio process receives those names from the secret bootstrap,
+and its existing adapter forwards the selected profile credential.
+
+## Repository Ownership
+
+- `../narratio` at immutable tag `v2.1.0` is the read-only application
+  authority for secrets bootstrap, typed Notarius credential forwarding,
+  checkpoint reuse, forced refresh, and subprocess redaction.
+- `ansible-role-narratio` advances the application release and exposes the
+  three native Notarius fields through its generic field-specific interface.
+  It does not manage secret files or select suite policy.
+- `ansible-role-dnd-pipeline` owns the conventional secrets directory, managed
+  credential mapping, suite-level Audita and Notarius reference coordination,
+  checkpoint-reuse selection, consumer documentation, and behavioral tests.
+- `ansible-workspace-dnd` owns this roadmap, the settled MVP policy, progress,
+  and independent review record.
+- The Audita, Notarius, and Scriptorium application roles remain unchanged;
+  they continue to own only their native configuration interfaces.
+- The collection, asset, remaining application, and reference repositories
+  remain unchanged.
+
+## Constraints and Settled Decisions
+
+- Raw credential bytes never enter ordinary application YAML, Git-tracked
+  examples, workspace plans, or committed test fixtures.
+- The role manages secret values only through `ansible.builtin.copy` content
+  or an equally direct secret-safe module operation with `no_log: true`.
+- The conventional directory path is fixed because automatic Narratio
+  discovery depends on its relationship to the fixed root pipeline document.
+- Credential filenames must match the environment-variable grammar accepted
+  by ConfigKit. Validation exists to prevent a caller-controlled key from
+  becoming a path outside the secrets directory.
+- Empty credential values are invalid. Null means explicit absence and is
+  distinct from an empty string.
+- An omitted mapping key is outside the role's ownership. Exact-directory
+  mirroring and deletion of unlisted files are intentionally excluded.
+- The shared runtime group is an authorization group. Its members may read the
+  managed credentials; all other local users are denied access.
+- The suite's group-readable credential policy deliberately differs from
+  Narratio's general single-user mode recommendation. ConfigKit does not
+  enforce those recommended modes, and the selected policy grants no world
+  access.
+- Application adapters, rather than Ansible, own the secure transfer of loaded
+  values to child processes. Ansible does not create wrappers or reconstruct
+  the subprocess environment.
+- Notarius's root configuration remains the single owner of ordinary
+  chunk-plan mode. Narratio's native `resume` field expresses the distinct
+  request to load compatible recorded checkpoints.
+- Production convergence performs no provider call, credential validity test,
+  cache probe, content checksum bookkeeping, backup, or post-write reread.
+
+## Simplest Viable Implementation Shape
+
+The standalone Narratio role needs only a release update and three nullable
+field-specific variables rendered through its existing template. It remains
+generic and does not own credential files or suite policy.
+
+The pipeline role needs one empty credential mapping, two Narratio Notarius
+policy variables, coordinated Audita reference defaults, and ordinary
+dependency wiring. Credential convergence consists of one directory task, one
+validation loop, one copy loop for non-null entries, and one file loop for
+explicit null removals. Both secret-bearing loops suppress their output.
+
+Documentation and focused native integration coverage complete the public
+interface. No state probe, handler, template assembly, manifest, wrapper,
+directory scan, or role-private bookkeeping is required. The validation loop
+is deliberate: the mapping key becomes a filesystem path, so rejecting
+anything other than a valid environment-variable name protects the declared
+destination boundary.
+
+## Rationale and Alternatives
+
+Narratio's native ConfigKit bootstrap is preferable to a shell wrapper or
+service-specific environment file because every pipeline-loading command uses
+the same application-owned behavior, whether invoked by the service account or
+an authorized human operator. It also preserves the application's existing
+environment-precedence and redaction rules.
+
+A mapping is more ergonomic for inventory than a list of file objects because
+the native identity of each entry is already its environment-variable name.
+Null provides explicit removal without making an empty mapping destructive or
+requiring a filesystem scan. Omitted files remain available for consumers that
+combine Ansible-managed credentials with another provisioning mechanism.
+
+Per-user `0700` directories would follow Narratio's general recommendation but
+would require separate keys or an additional invocation convention for every
+authorized human. The selected shared group already defines who may execute
+the suite with shared state, so group-readable and world-inaccessible
+credentials provide the intended operational model with less duplication.
+
+Selecting Narratio checkpoint reuse completes the suite's earlier decision to
+enable every available Notarius cache. Duplicating `chunk_cache: auto` in both
+Narratio and Notarius configuration would add no ordinary behavior, so the
+pipeline leaves the optional Narratio override unset and relies on each
+application's documented ownership boundary.
+
+## Acceptance Criteria
+
+- The standalone Narratio role installs an explicitly selected v2.1.0 and can
+  render representative non-null and falsey values for
+  `narratio_notarius_llm_api_key`, `narratio_notarius_resume`, and
+  `narratio_notarius_chunk_cache` while omitting null values.
+- The pipeline role selects Narratio v2.1.0 and pins the reviewed standalone
+  role revision that provides its native fields.
+- The pipeline role creates `/usr/local/etc/narratio/secrets` with the selected
+  runtime ownership and suite credential-directory mode.
+- `dnd_pipeline_secrets` defaults to an empty mapping and accepts valid
+  environment names with nonempty string or null values.
+- Invalid names and empty non-null values fail before any destination outside
+  the declared secret directory can be constructed.
+- Non-null entries converge to regular files with the selected runtime
+  ownership and mode `0640`; null entries converge to absence; omitted names
+  remain unmanaged.
+- Credential values are suppressed from Ansible output and never appear in
+  application YAML, generated examples, native effective-configuration output,
+  test logs, or committed fixtures.
+- Audita proposal and validation use `AUDITA_LLM_API_KEY` by default, and
+  Narratio's typed Audita field names that reference so the adapter explicitly
+  supplies the loaded value.
+- Narratio's effective pipeline selects `OPENROUTER_API_KEY` through the typed
+  Notarius field and selects checkpoint resume for ordinary extraction.
+- Narratio's optional chunk-cache override remains absent from the suite
+  configuration while Notarius's effective root configuration continues to
+  select `auto`.
+- Scriptorium continues to receive the credential named by its selected
+  profile through the existing adapter behavior.
+- A synthetic pipeline integration invocation with the relevant ambient
+  variables unset loads credentials through the conventional directory and
+  succeeds at native offline configuration validation as the service account
+  and one existing-group-member fixture.
+- One representative explicit-null transition makes the selected synthetic
+  credential unavailable through the native bootstrap boundary without
+  inspecting module results or file metadata.
+- A distinctive synthetic secret never appears in captured integration output.
+- The baseline and representative credential applications each report
+  `changed=0` and `failed=0` on an immediate identical repeat.
+- Syntax checking, production-profile lint, and the complete standalone
+  Narratio and pipeline integration entry points pass.
+- No live provider, WhisperX service, storage backend, model call, or
+  application cache execution occurs in default tests.
+
+## Dependencies and Follow-up Work
+
+- Features 0005, 0006, 0008, 0009, and 0010 are complete prerequisites.
+- Narratio v2.1.0 at immutable tag `c45e042` satisfies the former typed
+  Notarius credential and cache-control prerequisite.
+- The completed upstream feature request was removed from Narratio's active
+  roadmap during its v2.1.0 release cleanup; the release reference now owns the
+  delivered behavior.
+- The standalone Narratio-role commit produced by this feature must be
+  available from its canonical Git origin before the pipeline's isolated
+  dependency test can use it.
+- Collection assembly should follow this feature so the first assembled
+  collection packages a pipeline with complete credential delivery and cache
+  reuse.
+- Storage and publishing policy may later add S3 or other credential reference
+  names to consumer inventory without changing the file-delivery mechanism.
+- Operator selection, group-membership management, services, scheduling,
+  upgrades, removal, and external secret-manager integration remain separate
+  features.
+
+## Potential Future Hardening
+
+- Integrate a dedicated external secret manager or short-lived credential
+  broker.
+- Support distinct per-operator secret directories for deployments that do
+  not treat runtime-group membership as shared credential authorization.
+- Add explicit credential rotation orchestration and provider-side validity
+  checks.
+- Add platform-specific mandatory-access-control policy for credential files.