ansible-workspace-dnd/docs/roadmap/features/0011-managed-pipeline-credential-delivery.md

# Feature 0011: Managed Pipeline Credentials and Notarius Reuse

Status: ready

## Summary

Upgrade the generic Narratio role and D&D pipeline to Narratio v2.1.0, manage
provider credentials through Narratio's conventional ConfigKit secrets
directory, and use the release's typed Notarius credential and checkpoint-reuse
controls. Credential values remain outside ordinary application YAML and are
readable only by the selected runtime user and group. Ordinary pipeline runs
reuse compatible Notarius checkpoints, while forced extraction retains
Narratio's native refresh behavior.

## Problem or Opportunity

The pipeline role currently writes credential reference names but manages no
credential values. A caller must prepare the invoking environment through an
external mechanism before a real run can reach the selected model providers.

The installed Narratio v2.0.0 release also cannot forward a selected provider
credential to Notarius. Its Notarius child receives a restricted environment,
and the v2.0 configuration contract exposes no typed credential field. The
pipeline records Notarius checkpoints but cannot ask Notarius to reuse them.

Narratio v2.1.0, immutable tag `v2.1.0` at `c45e042`, resolves both application
contract gaps. It adds optional `notarius.llm_api_key`, `notarius.resume`, and
`notarius.chunk_cache` fields without requiring migration of existing v2
configuration. The credential field resolves through the existing process
environment or secrets-directory bootstrap, forwards only the exact selected
name and value, and participates in the application's existing redaction
boundary. Ordinary extraction may request checkpoint reuse; forced extraction
omits resume and refreshes the active chunk plan.

The suite separately configures Audita proposal and validation with different
environment names even though both use the same endpoint and model. Narratio's
Audita boundary accepts one typed credential. This feature therefore
coordinates both suite-default Audita sections on `AUDITA_LLM_API_KEY` while
retaining their existing independently overrideable variables.

## Governing Policies

- [The Ansible house style](../../ansible.md) governs the direct desired-state
  implementation, native configuration interface, and ordinary module
  convergence.
- [The testing house style](../../testing.md) governs synthetic, native,
  behavior-oriented validation without live provider calls or filesystem
  metadata assertions.
- [The MVP roadmap](../mvp.md) assigns pipeline-wide secret delivery,
  application-version coordination, and cache policy to the suite role.
- [Decision 0002](../../decisions/0002-shared-readable-pipeline-filesystem-policy.md)
  defines the selected runtime group as the authorization boundary and fixes
  `0750` directory and `0640` file modes for shared credential material.
- [Feature 0009](0009-shared-pipeline-filesystem-and-execution.md) establishes
  the service-user and existing-group-member invocation model.
- [Feature 0010](0010-dnd-notarius-extraction-integration.md) establishes the
  Notarius pipeline, FHS cache roots, automatic chunk-plan reuse, and checkpoint
  recording extended by this feature.
- Narratio's immutable v2.1.0 release documentation and current configuration
  and Notarius integration references are authoritative for the application
  fields, resolution semantics, forced-run behavior, and redaction contract.

## Scope

- Advance the standalone Narratio role's selected application release from
  v2.0.0 to v2.1.0 without changing its established source-build task flow.
- Expose the three new native optional fields through nullable, field-specific
  role variables: `narratio_notarius_llm_api_key`,
  `narratio_notarius_resume`, and `narratio_notarius_chunk_cache`.
- Keep the standalone role generic. All three variables default to null and
  are omitted unless a caller selects them, leaving Narratio's native defaults
  in control.
- Advance the pipeline role's compatible Narratio application version to
  v2.1.0 and pin its Narratio role dependency to the reviewed revision that
  provides the new interface.
- Use `/usr/local/etc/narratio/secrets`, the conventional directory adjacent
  to `/usr/local/etc/narratio/pipeline.yml`, as the suite's sole managed
  credential location.
- Create that directory as the selected pipeline runtime user and group with
  mode `0750`.
- Add one public `dnd_pipeline_secrets` mapping. Keys are environment-variable
  names. A non-null string value means that the named credential file must be
  present with that content; null means that the specifically named managed
  credential file must be absent. An omitted key is unmanaged.
- Default `dnd_pipeline_secrets` to an empty mapping so the role never invents
  credentials and remains usable with externally prepared process environment
  values.
- Validate selected keys as environment-variable names and reject empty
  non-null values before constructing a destination path.
- Write non-null credential files as the selected runtime user and group with
  mode `0640`, and remove only names explicitly assigned null.
- Suppress credential values from Ansible task output during validation,
  creation, updates, and removal.
- Keep ordinary application configuration limited to non-secret credential
  reference names.
- Coordinate Audita proposal and validation on the suite-default
  `AUDITA_LLM_API_KEY` name and pass that name through the standalone Narratio
  role's existing `narratio_audita_llm_api_key` interface.
- Add `dnd_pipeline_narratio_notarius_llm_api_key`, default it to
  `OPENROUTER_API_KEY` for Notarius's selected production profile, and pass it
  through Narratio's typed Notarius credential field.
- Add `dnd_pipeline_narratio_notarius_resume`, default it to true, and pass it
  through Narratio so ordinary extraction requests reuse of the checkpoints
  already recorded by the suite's Notarius configuration.
- Expose Narratio's native `notarius.chunk_cache` in the standalone role but do
  not duplicate it behind a suite variable or set it from the pipeline role.
  The suite already configures Notarius's ordinary chunk-plan mode as `auto`;
  Narratio v2.1.0 independently owns its forced-extraction refresh override.
- Preserve Scriptorium's application-owned `api_key_env` profile selection.
  Narratio's secrets bootstrap supplies the selected environment value and the
  existing Scriptorium adapter passes that named credential to its child.
- Document the mapping interface, Ansible Vault inventory pattern, process
  environment precedence, shared-group authorization boundary, explicit null
  removal, application credential flow, and checkpoint-reuse policy.
- Add focused integration coverage using distinctive synthetic values and
  native Narratio configuration loading. No default test contacts a model
  provider.

## Non-goals

- Storing raw credentials in Narratio, Audita, Notarius, or Scriptorium YAML.
- Providing real credentials, unencrypted example values, or a default secret
  mapping.
- Implementing Ansible Vault, choosing a vault password mechanism, or requiring
  one specific inventory encryption workflow.
- Integrating an external secret manager, cloud vault, hardware-backed key
  store, or dynamic credential broker.
- Making the pipeline role authoritative for files omitted from
  `dnd_pipeline_secrets`, purging the whole directory, or recursively changing
  pre-existing content.
- Adding a launcher script, shell profile, environment file, systemd unit,
  scheduler, or persistent service merely to inject credentials.
- Passing arbitrary caller-defined environment variables through Narratio's
  subprocess boundary.
- Managing human accounts or their membership in the runtime group.
- Running Audita, Notarius, or Scriptorium against a live endpoint during the
  default integration suites.
- Testing Notarius checkpoint reuse end to end in the Ansible role. Narratio
  and Notarius own that behavior; the role validates the selected native
  configuration and immediate Ansible convergence.
- Setting Narratio's optional `notarius.chunk_cache` field redundantly when
  the deployed Notarius configuration already selects `auto`.
- Changing Narratio's native forced-extraction refresh policy.
- Rotating credentials, retaining prior values, validating provider
  authorization, or recovering deliberately corrupted secret files.
- Adding storage or publishing credentials before the suite selects a storage
  or publishing feature.
- Modifying any upstream application source.

## Target End State

The standalone Narratio role installs v2.1.0 by default and can render all
three new Notarius fields when a caller supplies them. It owns no credential
files and selects no D&D-specific credential or cache policy.

The pipeline role owns this conventional restricted subtree:

```text
/usr/local/etc/narratio/
├── pipeline.yml
└── secrets/                  narratio:narratio 0750
    ├── AUDITA_LLM_API_KEY    narratio:narratio 0640
    └── OPENROUTER_API_KEY    narratio:narratio 0640
```

The filenames are suite-selected reference names, never credential values.
The selected runtime user and group remain configurable through the existing
pipeline interface.

A consumer may keep encrypted values in inventory and select explicit removal
without embedding any value in ordinary configuration:

```yaml
dnd_pipeline_secrets:
  AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
  OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"
  RETIRED_PROVIDER_KEY: null
```

The role owns only these three named entries for this invocation. It converges
the first two to regular files, removes the third, and leaves every omitted
name alone. The actual `vault_dnd_*` values belong in encrypted inventory and
must not appear in examples, plans, test logs, commits, or ordinary generated
configuration.

When Narratio runs without `--secrets-dir`, it discovers the sibling directory
and loads its immediate regular files before resolving the pipeline. An
already-set process environment value wins over the corresponding file. The
base and `testing` profiles use the same delivery mechanism; profiles select
models and application profiles, not a second credential store.

The suite deliberately uses `narratio:narratio`, directory mode `0750`, and
file mode `0640` rather than Narratio's general single-user recommendation of
`0700` and `0600`. Membership in the selected runtime group is an explicit
authorization to invoke this pipeline with its shared provider credentials.
No world access is granted. Deployments that do not trust every group member
must use separate process-environment or `--secrets-dir` provisioning and are
outside this shared-operator policy.

Audita's proposal and validation configuration both reference
`AUDITA_LLM_API_KEY` by default. Narratio resolves that typed reference and
passes the value explicitly to the Audita child. Callers may still override
the existing Audita reference variables, but a custom split-key design also
requires an application boundary capable of forwarding both values and is not
promised by this feature.

Narratio's effective Notarius configuration names `OPENROUTER_API_KEY`, or a
caller-selected replacement, through `notarius.llm_api_key`. Narratio resolves
the named secret and forwards only that exact name and value to Notarius. It
does not infer a provider from Notarius configuration or forward unrelated
credentials.

Ordinary extraction sets `notarius.resume: true`. Notarius continues to own
checkpoint recording, identity, and storage beneath the configured
`/var/cache/notarius/checkpoints` root. Narratio requests reuse but does not
inspect the cache. The pipeline leaves Narratio's `notarius.chunk_cache` field
unset because the Notarius root configuration already selects `auto` at the
native owner. Forced extraction follows Narratio v2.1.0 behavior: resume is
omitted and the active chunk plan is refreshed without deleting unrelated
cache content.

Scriptorium continues to select credential names through its PromptKit
profiles. The Narratio process receives those names from the secret bootstrap,
and its existing adapter forwards the selected profile credential.

## Repository Ownership

- `../narratio` at immutable tag `v2.1.0` is the read-only application
  authority for secrets bootstrap, typed Notarius credential forwarding,
  checkpoint reuse, forced refresh, and subprocess redaction.
- `ansible-role-narratio` advances the application release and exposes the
  three native Notarius fields through its generic field-specific interface.
  It does not manage secret files or select suite policy.
- `ansible-role-dnd-pipeline` owns the conventional secrets directory, managed
  credential mapping, suite-level Audita and Notarius reference coordination,
  checkpoint-reuse selection, consumer documentation, and behavioral tests.
- `ansible-workspace-dnd` owns this roadmap, the settled MVP policy, progress,
  and independent review record.
- The Audita, Notarius, and Scriptorium application roles remain unchanged;
  they continue to own only their native configuration interfaces.
- The collection, asset, remaining application, and reference repositories
  remain unchanged.

## Constraints and Settled Decisions

- Raw credential bytes never enter ordinary application YAML, Git-tracked
  examples, workspace plans, or committed test fixtures.
- The role manages secret values only through `ansible.builtin.copy` content
  or an equally direct secret-safe module operation with `no_log: true`.
- The conventional directory path is fixed because automatic Narratio
  discovery depends on its relationship to the fixed root pipeline document.
- Credential filenames must match the environment-variable grammar accepted
  by ConfigKit. Validation exists to prevent a caller-controlled key from
  becoming a path outside the secrets directory.
- Empty credential values are invalid. Null means explicit absence and is
  distinct from an empty string.
- An omitted mapping key is outside the role's ownership. Exact-directory
  mirroring and deletion of unlisted files are intentionally excluded.
- The shared runtime group is an authorization group. Its members may read the
  managed credentials; all other local users are denied access.
- The suite's group-readable credential policy deliberately differs from
  Narratio's general single-user mode recommendation. ConfigKit does not
  enforce those recommended modes, and the selected policy grants no world
  access.
- Application adapters, rather than Ansible, own the secure transfer of loaded
  values to child processes. Ansible does not create wrappers or reconstruct
  the subprocess environment.
- Notarius's root configuration remains the single owner of ordinary
  chunk-plan mode. Narratio's native `resume` field expresses the distinct
  request to load compatible recorded checkpoints.
- Production convergence performs no provider call, credential validity test,
  cache probe, content checksum bookkeeping, backup, or post-write reread.

## Simplest Viable Implementation Shape

The standalone Narratio role needs only a release update and three nullable
field-specific variables rendered through its existing template. It remains
generic and does not own credential files or suite policy.

The pipeline role needs one empty credential mapping, two Narratio Notarius
policy variables, coordinated Audita reference defaults, and ordinary
dependency wiring. Credential convergence consists of one directory task, one
validation loop, one copy loop for non-null entries, and one file loop for
explicit null removals. Both secret-bearing loops suppress their output.

Documentation and focused native integration coverage complete the public
interface. No state probe, handler, template assembly, manifest, wrapper,
directory scan, or role-private bookkeeping is required. The validation loop
is deliberate: the mapping key becomes a filesystem path, so rejecting
anything other than a valid environment-variable name protects the declared
destination boundary.

## Rationale and Alternatives

Narratio's native ConfigKit bootstrap is preferable to a shell wrapper or
service-specific environment file because every pipeline-loading command uses
the same application-owned behavior, whether invoked by the service account or
an authorized human operator. It also preserves the application's existing
environment-precedence and redaction rules.

A mapping is more ergonomic for inventory than a list of file objects because
the native identity of each entry is already its environment-variable name.
Null provides explicit removal without making an empty mapping destructive or
requiring a filesystem scan. Omitted files remain available for consumers that
combine Ansible-managed credentials with another provisioning mechanism.

Per-user `0700` directories would follow Narratio's general recommendation but
would require separate keys or an additional invocation convention for every
authorized human. The selected shared group already defines who may execute
the suite with shared state, so group-readable and world-inaccessible
credentials provide the intended operational model with less duplication.

Selecting Narratio checkpoint reuse completes the suite's earlier decision to
enable every available Notarius cache. Duplicating `chunk_cache: auto` in both
Narratio and Notarius configuration would add no ordinary behavior, so the
pipeline leaves the optional Narratio override unset and relies on each
application's documented ownership boundary.

## Acceptance Criteria

- The standalone Narratio role installs an explicitly selected v2.1.0 and can
  render representative non-null and falsey values for
  `narratio_notarius_llm_api_key`, `narratio_notarius_resume`, and
  `narratio_notarius_chunk_cache` while omitting null values.
- The pipeline role selects Narratio v2.1.0 and pins the reviewed standalone
  role revision that provides its native fields.
- The pipeline role creates `/usr/local/etc/narratio/secrets` with the selected
  runtime ownership and suite credential-directory mode.
- `dnd_pipeline_secrets` defaults to an empty mapping and accepts valid
  environment names with nonempty string or null values.
- Invalid names and empty non-null values fail before any destination outside
  the declared secret directory can be constructed.
- Non-null entries converge to regular files with the selected runtime
  ownership and mode `0640`; null entries converge to absence; omitted names
  remain unmanaged.
- Credential values are suppressed from Ansible output and never appear in
  application YAML, generated examples, native effective-configuration output,
  test logs, or committed fixtures.
- Audita proposal and validation use `AUDITA_LLM_API_KEY` by default, and
  Narratio's typed Audita field names that reference so the adapter explicitly
  supplies the loaded value.
- Narratio's effective pipeline selects `OPENROUTER_API_KEY` through the typed
  Notarius field and selects checkpoint resume for ordinary extraction.
- Narratio's optional chunk-cache override remains absent from the suite
  configuration while Notarius's effective root configuration continues to
  select `auto`.
- Scriptorium continues to receive the credential named by its selected
  profile through the existing adapter behavior.
- A synthetic pipeline integration invocation with the relevant ambient
  variables unset loads credentials through the conventional directory and
  succeeds at native offline configuration validation as the service account
  and one existing-group-member fixture.
- One representative explicit-null transition makes the selected synthetic
  credential unavailable through the native bootstrap boundary without
  inspecting module results or file metadata.
- A distinctive synthetic secret never appears in captured integration output.
- The baseline and representative credential applications each report
  `changed=0` and `failed=0` on an immediate identical repeat.
- Syntax checking, production-profile lint, and the complete standalone
  Narratio and pipeline integration entry points pass.
- No live provider, WhisperX service, storage backend, model call, or
  application cache execution occurs in default tests.

## Dependencies and Follow-up Work

- Features 0005, 0006, 0008, 0009, and 0010 are complete prerequisites.
- Narratio v2.1.0 at immutable tag `c45e042` satisfies the former typed
  Notarius credential and cache-control prerequisite.
- The completed upstream feature request was removed from Narratio's active
  roadmap during its v2.1.0 release cleanup; the release reference now owns the
  delivered behavior.
- The standalone Narratio-role commit produced by this feature must be
  available from its canonical Git origin before the pipeline's isolated
  dependency test can use it.
- Collection assembly should follow this feature so the first assembled
  collection packages a pipeline with complete credential delivery and cache
  reuse.
- Storage and publishing policy may later add S3 or other credential reference
  names to consumer inventory without changing the file-delivery mechanism.
- Operator selection, group-membership management, services, scheduling,
  upgrades, removal, and external secret-manager integration remain separate
  features.

## Potential Future Hardening

- Integrate a dedicated external secret manager or short-lived credential
  broker.
- Support distinct per-operator secret directories for deployments that do
  not treat runtime-group membership as shared credential authorization.
- Add explicit credential rotation orchestration and provider-side validity
  checks.
- Add platform-specific mandatory-access-control policy for credential files.