ansible-workspace-dnd/docs/roadmap/features/0011-managed-pipeline-credential-delivery.md

Feature 0011: Managed Pipeline Credentials and Notarius Reuse

Status: done

Summary

Upgrade the generic Narratio role and D&D pipeline to Narratio v2.1.0, manage provider credentials through Narratio's conventional ConfigKit secrets directory, and use the release's typed Notarius credential and checkpoint-reuse controls. Credential values remain outside ordinary application YAML and are readable only by the selected runtime user and group. Ordinary pipeline runs reuse compatible Notarius checkpoints, while forced extraction retains Narratio's native refresh behavior.

Problem or Opportunity

The pipeline role currently writes credential reference names but manages no credential values. A caller must prepare the invoking environment through an external mechanism before a real run can reach the selected model providers.

The installed Narratio v2.0.0 release also cannot forward a selected provider credential to Notarius. Its Notarius child receives a restricted environment, and the v2.0 configuration contract exposes no typed credential field. The pipeline records Notarius checkpoints but cannot ask Notarius to reuse them.

Narratio v2.1.0, immutable tag v2.1.0 at c45e042, resolves both application contract gaps. It adds optional notarius.llm_api_key, notarius.resume, and notarius.chunk_cache fields without requiring migration of existing v2 configuration. The credential field resolves through the existing process environment or secrets-directory bootstrap, forwards only the exact selected name and value, and participates in the application's existing redaction boundary. Ordinary extraction may request checkpoint reuse; forced extraction omits resume and refreshes the active chunk plan.

The suite separately configures Audita proposal and validation with different environment names even though both use the same endpoint and model. Narratio's Audita boundary accepts one typed credential. This feature therefore coordinates both suite-default Audita sections on AUDITA_LLM_API_KEY while retaining their existing independently overrideable variables.

Governing Policies

• The Ansible house style governs the direct desired-state implementation, native configuration interface, and ordinary module convergence.
• The testing house style governs synthetic, native, behavior-oriented validation without live provider calls or filesystem metadata assertions.
• The MVP roadmap assigns pipeline-wide secret delivery, application-version coordination, and cache policy to the suite role.
• Decision 0002 defines the selected runtime group as the authorization boundary and fixes 0750 directory and 0640 file modes for shared credential material.
• Feature 0009 establishes the service-user and existing-group-member invocation model.
• Feature 0010 establishes the Notarius pipeline, FHS cache roots, automatic chunk-plan reuse, and checkpoint recording extended by this feature.
• Narratio's immutable v2.1.0 release documentation and current configuration and Notarius integration references are authoritative for the application fields, resolution semantics, forced-run behavior, and redaction contract.

Scope

• Advance the standalone Narratio role's selected application release from v2.0.0 to v2.1.0 without changing its established source-build task flow.
• Expose the three new native optional fields through nullable, field-specific role variables: narratio_notarius_llm_api_key, narratio_notarius_resume, and narratio_notarius_chunk_cache.
• Keep the standalone role generic. All three variables default to null and are omitted unless a caller selects them, leaving Narratio's native defaults in control.
• Advance the pipeline role's compatible Narratio application version to v2.1.0 and pin its Narratio role dependency to the reviewed revision that provides the new interface.
• Use /usr/local/etc/narratio/secrets, the conventional directory adjacent to /usr/local/etc/narratio/pipeline.yml, as the suite's sole managed credential location.
• Create that directory as the selected pipeline runtime user and group with mode 0750.
• Add one public dnd_pipeline_secrets mapping from environment-variable names to string file contents. Each supplied entry means that the named credential file must be present with that content; an omitted key is unmanaged.
• Default dnd_pipeline_secrets to an empty mapping so the role never invents credentials and remains usable with externally prepared process environment values.
• Validate selected keys as environment-variable names and values as strings before constructing a destination path.
• Write supplied credential files as the selected runtime user and group with mode 0640.
• Suppress credential values from Ansible task output during validation, creation, and updates.
• Keep ordinary application configuration limited to non-secret credential reference names.
• Coordinate Audita proposal and validation on the suite-default AUDITA_LLM_API_KEY name and pass that name through the standalone Narratio role's existing narratio_audita_llm_api_key interface.
• Add dnd_pipeline_narratio_notarius_llm_api_key, default it to OPENROUTER_API_KEY for Notarius's selected production profile, and pass it through Narratio's typed Notarius credential field.
• Add dnd_pipeline_narratio_notarius_resume, default it to true, and pass it through Narratio so ordinary extraction requests reuse of the checkpoints already recorded by the suite's Notarius configuration.
• Expose Narratio's native notarius.chunk_cache in the standalone role but do not duplicate it behind a suite variable or set it from the pipeline role. The suite already configures Notarius's ordinary chunk-plan mode as auto; Narratio v2.1.0 independently owns its forced-extraction refresh override.
• Preserve Scriptorium's application-owned api_key_env profile selection. Narratio's secrets bootstrap supplies the selected environment value and the existing Scriptorium adapter passes that named credential to its child.
• Document the mapping interface, Ansible Vault inventory pattern, process environment precedence, shared-group authorization boundary, application credential flow, and checkpoint-reuse policy.
• Add focused integration coverage using distinctive synthetic values and native Narratio configuration loading. No default test contacts a model provider.

Non-goals

• Storing raw credentials in Narratio, Audita, Notarius, or Scriptorium YAML.
• Providing real credentials, unencrypted example values, or a default secret mapping.
• Implementing Ansible Vault, choosing a vault password mechanism, or requiring one specific inventory encryption workflow.
• Integrating an external secret manager, cloud vault, hardware-backed key store, or dynamic credential broker.
• Making the pipeline role authoritative for files omitted from dnd_pipeline_secrets, purging the whole directory, or recursively changing pre-existing content.
• Adding a launcher script, shell profile, environment file, systemd unit, scheduler, or persistent service merely to inject credentials.
• Passing arbitrary caller-defined environment variables through Narratio's subprocess boundary.
• Managing human accounts or their membership in the runtime group.
• Running Audita, Notarius, or Scriptorium against a live endpoint during the default integration suites.
• Testing Notarius checkpoint reuse end to end in the Ansible role. Narratio and Notarius own that behavior; the role validates the selected native configuration and immediate Ansible convergence.
• Setting Narratio's optional notarius.chunk_cache field redundantly when the deployed Notarius configuration already selects auto.
• Changing Narratio's native forced-extraction refresh policy.
• Rotating credentials, retaining prior values, validating provider authorization, or recovering deliberately corrupted secret files.
• Adding storage or publishing credentials before the suite selects a storage or publishing feature.
• Modifying any upstream application source.

Target End State

The standalone Narratio role installs v2.1.0 by default and can render all three new Notarius fields when a caller supplies them. It owns no credential files and selects no D&D-specific credential or cache policy.

The pipeline role owns this conventional restricted subtree:

/usr/local/etc/narratio/
├── pipeline.yml
└── secrets/                  narratio:narratio 0750
    ├── AUDITA_LLM_API_KEY    narratio:narratio 0640
    └── OPENROUTER_API_KEY    narratio:narratio 0640

The filenames are suite-selected reference names, never credential values. The selected runtime user and group remain configurable through the existing pipeline interface.

A consumer may keep encrypted values in inventory without embedding any value in ordinary configuration:

dnd_pipeline_secrets:
  AUDITA_LLM_API_KEY: "{{ vault_dnd_audita_llm_api_key }}"
  OPENROUTER_API_KEY: "{{ vault_dnd_openrouter_api_key }}"

The role owns only the supplied named entries for this invocation and leaves every omitted name alone. The actual vault_dnd_* values belong in encrypted inventory and must not appear in examples, plans, test logs, commits, or ordinary generated configuration.

When Narratio runs without --secrets-dir, it discovers the sibling directory and loads its immediate regular files before resolving the pipeline. An already-set process environment value wins over the corresponding file. The base and testing profiles use the same delivery mechanism; profiles select models and application profiles, not a second credential store.

The suite deliberately uses narratio:narratio, directory mode 0750, and file mode 0640 rather than Narratio's general single-user recommendation of 0700 and 0600. Membership in the selected runtime group is an explicit authorization to invoke this pipeline with its shared provider credentials. No world access is granted. Deployments that do not trust every group member must use separate process-environment or --secrets-dir provisioning and are outside this shared-operator policy.

Audita's proposal and validation configuration both reference AUDITA_LLM_API_KEY by default. Narratio resolves that typed reference and passes the value explicitly to the Audita child. Callers may still override the existing Audita reference variables, but a custom split-key design also requires an application boundary capable of forwarding both values and is not promised by this feature.

Narratio's effective Notarius configuration names OPENROUTER_API_KEY, or a caller-selected replacement, through notarius.llm_api_key. Narratio resolves the named secret and forwards only that exact name and value to Notarius. It does not infer a provider from Notarius configuration or forward unrelated credentials.

Ordinary extraction sets notarius.resume: true. Notarius continues to own checkpoint recording, identity, and storage beneath the configured /var/cache/notarius/checkpoints root. Narratio requests reuse but does not inspect the cache. The pipeline leaves Narratio's notarius.chunk_cache field unset because the Notarius root configuration already selects auto at the native owner. Forced extraction follows Narratio v2.1.0 behavior: resume is omitted and the active chunk plan is refreshed without deleting unrelated cache content.

Scriptorium continues to select credential names through its PromptKit profiles. The Narratio process receives those names from the secret bootstrap, and its existing adapter forwards the selected profile credential.

Repository Ownership

• ../narratio at immutable tag v2.1.0 is the read-only application authority for secrets bootstrap, typed Notarius credential forwarding, checkpoint reuse, forced refresh, and subprocess redaction.
• ansible-role-narratio advances the application release and exposes the three native Notarius fields through its generic field-specific interface. It does not manage secret files or select suite policy.
• ansible-role-dnd-pipeline owns the conventional secrets directory, managed credential mapping, suite-level Audita and Notarius reference coordination, checkpoint-reuse selection, consumer documentation, and behavioral tests.
• ansible-workspace-dnd owns this roadmap, the settled MVP policy, progress, and independent review record.
• The Audita, Notarius, and Scriptorium application roles remain unchanged; they continue to own only their native configuration interfaces.
• The collection, asset, remaining application, and reference repositories remain unchanged.

Constraints and Settled Decisions

• Raw credential bytes never enter ordinary application YAML, Git-tracked examples, workspace plans, or committed test fixtures.
• The role manages secret values only through ansible.builtin.copy content or an equally direct secret-safe module operation with no_log: true.
• The conventional directory path is fixed because automatic Narratio discovery depends on its relationship to the fixed root pipeline document.
• Credential filenames must match the environment-variable grammar accepted by ConfigKit. Validation exists to prevent a caller-controlled key from becoming a path outside the secrets directory.
• Credential values are written as supplied strings. ConfigKit and the typed application reference own any nonempty-value requirement when a credential is selected for use.
• An omitted mapping key is outside the role's ownership. Exact-directory mirroring and deletion of unlisted files are intentionally excluded.
• The shared runtime group is an authorization group. Its members may read the managed credentials; all other local users are denied access.
• The suite's group-readable credential policy deliberately differs from Narratio's general single-user mode recommendation. ConfigKit does not enforce those recommended modes, and the selected policy grants no world access.
• Application adapters, rather than Ansible, own the secure transfer of loaded values to child processes. Ansible does not create wrappers or reconstruct the subprocess environment.
• Notarius's root configuration remains the single owner of ordinary chunk-plan mode. Narratio's native resume field expresses the distinct request to load compatible recorded checkpoints.
• Production convergence performs no provider call, credential validity test, cache probe, content checksum bookkeeping, backup, or post-write reread.

Simplest Viable Implementation Shape

The standalone Narratio role needs only a release update and three nullable field-specific variables rendered through its existing template. It remains generic and does not own credential files or suite policy.

The pipeline role needs one empty credential mapping, two Narratio Notarius policy variables, coordinated Audita reference defaults, and ordinary dependency wiring. Credential convergence consists of one directory task, one validation loop, and one copy loop for supplied entries. Both secret-bearing loops suppress their output.

Documentation and focused native integration coverage complete the public interface. No state probe, handler, template assembly, manifest, wrapper, directory scan, or role-private bookkeeping is required. The validation loop is deliberate: the mapping key becomes a filesystem path, so rejecting anything other than a valid environment-variable name protects the declared destination boundary.

Rationale and Alternatives

Narratio's native ConfigKit bootstrap is preferable to a shell wrapper or service-specific environment file because every pipeline-loading command uses the same application-owned behavior, whether invoked by the service account or an authorized human operator. It also preserves the application's existing environment-precedence and redaction rules.

A mapping is more ergonomic for inventory than a list of file objects because the native identity of each entry is already its environment-variable name. Omitted files remain unmanaged so consumers can combine Ansible-managed credentials with another provisioning mechanism. Credential removal and directory mirroring remain outside this feature until an operator-facing lifecycle requirement is expressly selected.

Per-user 0700 directories would follow Narratio's general recommendation but would require separate keys or an additional invocation convention for every authorized human. The selected shared group already defines who may execute the suite with shared state, so group-readable and world-inaccessible credentials provide the intended operational model with less duplication.

Selecting Narratio checkpoint reuse completes the suite's earlier decision to enable every available Notarius cache. Duplicating chunk_cache: auto in both Narratio and Notarius configuration would add no ordinary behavior, so the pipeline leaves the optional Narratio override unset and relies on each application's documented ownership boundary.

Acceptance Criteria

• The standalone Narratio role installs an explicitly selected v2.1.0 and can render representative non-null and falsey values for narratio_notarius_llm_api_key, narratio_notarius_resume, and narratio_notarius_chunk_cache while omitting null values.
• The pipeline role selects Narratio v2.1.0 and pins the reviewed standalone role revision that provides its native fields.
• The pipeline role creates /usr/local/etc/narratio/secrets with the selected runtime ownership and suite credential-directory mode.
• dnd_pipeline_secrets defaults to an empty mapping and accepts valid environment names with string values.
• Invalid names and non-string values fail before any destination outside the declared secret directory can be constructed.
• Supplied entries converge to regular files with the selected runtime ownership and mode 0640; omitted names remain unmanaged.
• Credential values are suppressed from Ansible output and never appear in application YAML, generated examples, native effective-configuration output, test logs, or committed fixtures.
• Audita proposal and validation use AUDITA_LLM_API_KEY by default, and Narratio's typed Audita field names that reference so the adapter explicitly supplies the loaded value.
• Narratio's effective pipeline selects OPENROUTER_API_KEY through the typed Notarius field and selects checkpoint resume for ordinary extraction.
• Narratio's optional chunk-cache override remains absent from the suite configuration while Notarius's effective root configuration continues to select auto.
• Scriptorium continues to receive the credential named by its selected profile through the existing adapter behavior.
• A synthetic pipeline integration invocation with the relevant ambient variables unset loads credentials through the conventional directory and succeeds at native offline configuration validation as the service account and one existing-group-member fixture.
• A distinctive synthetic secret never appears in captured integration output.
• The baseline credential application reports changed=0 and failed=0 on an immediate identical repeat.
• Syntax checking, production-profile lint, and the complete standalone Narratio and pipeline integration entry points pass.
• No live provider, WhisperX service, storage backend, model call, or application cache execution occurs in default tests.

Dependencies and Follow-up Work

• Features 0005, 0006, 0008, 0009, and 0010 are complete prerequisites.
• Narratio v2.1.0 at immutable tag c45e042 satisfies the former typed Notarius credential and cache-control prerequisite.
• The completed upstream feature request was removed from Narratio's active roadmap during its v2.1.0 release cleanup; the release reference now owns the delivered behavior.
• The standalone Narratio-role commit produced by this feature must be available from its canonical Git origin before the pipeline's isolated dependency test can use it.
• Collection assembly should follow this feature so the first assembled collection packages a pipeline with complete credential delivery and cache reuse.
• Storage and publishing policy may later add S3 or other credential reference names to consumer inventory without changing the file-delivery mechanism.
• Operator selection, group-membership management, services, scheduling, upgrades, removal, and external secret-manager integration remain separate features.

Potential Future Hardening

• Integrate a dedicated external secret manager or short-lived credential broker.
• Support distinct per-operator secret directories for deployments that do not treat runtime-group membership as shared credential authorization.
• Add explicit credential rotation orchestration and provider-side validity checks.
• Add an explicit credential-removal or directory-lifecycle contract if an operator need is identified.
• Add platform-specific mandatory-access-control policy for credential files.