ansible-workspace-dnd/docs/roadmap/features/0017-notarius-testing-overlay.md

Feature 0017: Notarius Testing Overlay

Status: done

Summary

Selecting Narratio's testing profile also selects a separately managed Notarius testing configuration overlay. Operators can develop D&D assets with economical extraction settings without modifying production configuration or invoking Notarius separately.

Problem or Opportunity

The suite's existing testing overlay changes Scriptorium selections but leaves Notarius on its production configuration. Narratio v2.2.0 now provides the explicit pipeline.notarius.config_profile interface requested by this consumer. Notarius v0.7.1 retains the corresponding --config-profile option and updates PromptKit and its provider catalogs. This feature adopts v0.7.1 as its baseline.

Scope

• Expose the optional native selector in the standalone Narratio role.
• Update standalone Notarius and suite version defaults to v0.7.1, with matching current-version documentation and collection distribution.
• Deploy a Notarius ConfigKit testing overlay through the suite role.
• Explicitly select that overlay from Narratio's existing testing overlay.
• Select gpt-terra-latest for production and gpt-luna-latest for testing, while retaining inventory overrides for both choices.
• Document and validate the composed configuration, then distribute the updated suite through the collection and refresh the consumer dependency pin.

Non-goals

• Further upstream application changes or a Narratio-side LLM-profile override.
• Automatically propagating profile names between applications.
• Disabling Notarius, caching, publishing, or other pipeline stages for testing.
• New credentials, cache layouts, other production tuning, or deployment automation.
• Live controller/target deployment or paid model calls as completion gates.

Target End State

Standalone Narratio Interface

narratio_notarius_config_profile defaults to null. The role omits notarius.config_profile when null and serializes explicitly supplied values, including an empty string. Documentation and argument specifications expose the field without duplicating or narrowing Narratio's selector semantics.

Explicit Suite Composition

The suite leaves the base selector unset. Its existing Narratio profiles/testing.yml includes:

notarius:
  config_profile: testing

The suite deploys /usr/local/etc/notarius/profiles/testing.yml. That file overlays the selected suite pipeline's llm_profile, using dnd_pipeline_narratio_notarius_pipeline_id as the pipeline key and a new dnd_pipeline_notarius_testing_llm_profile variable as the value. It does not copy the full pipeline definition. ConfigKit performs the ordinary merge.

The default suite pipeline currently inherits its LLM selection at pipeline level. Caller-supplied binding-level selections retain Notarius's native precedence; this feature does not rewrite arbitrary caller pipelines or promise that a pipeline-level default overrides explicit binding choices.

Production and Testing LLM Defaults

The suite's base dnd-session pipeline selects llm_profile: gpt-terra-latest instead of dnd-extraction. Callers can continue overriding that choice through the existing dnd_pipeline_notarius_pipelines mapping. The new dnd_pipeline_notarius_testing_llm_profile defaults to gpt-luna-latest.

Both IDs are supplied by the OpenRouter catalog v2.20260922.1 selected by Notarius v0.7.1. Use those native catalog profiles directly; no operator profile asset or profile-source setting is needed for these defaults. Retain the existing OPENROUTER_API_KEY credential reference. These are deliberately moving latest selections, not promises of immutable underlying model versions. This production-model policy belongs to the suite, not the standalone role.

Notarius stays enabled and retains its configured caching, output contracts, references, and production defaults outside this overlay. Existing Scriptorium testing overrides remain intact. Credentials continue through the existing typed reference and suite secrets interfaces; no credential values enter YAML.

Configuration files and directories use the suite's existing owner, group, and mode variables. The selector and overlay add no new runtime identity or filesystem-permission policy.

Repository Ownership

• ansible-role-narratio: generic optional selector, documentation, and tests.
• ansible-role-notarius: v0.7.1 installation default and corresponding documentation; existing profile-source settings remain sufficient.
• ansible-role-dnd-pipeline: testing policy, overlay assets, and composition.
• ansible-collection-dnd: imported pipeline snapshot and compatible dependency manifest. Its current Narratio-role v0.1.0 pin must advance to include the new public variable; changing only the sibling role is insufficient.
• rakestraw-ansible: matching dependency pins and concise operator guidance.
• Workspace: roadmap, implementation plan, progress, and review.
• Narratio and Notarius applications: read-only authoritative contracts. The standalone Notarius role already exposes profile-source settings and does not need D&D-specific overlay policy.

Constraints and Settled Decisions

Follow the MVP roadmap, Ansible house style, and testing house style. Narratio v2.2.0 is already selected by the suite; Notarius v0.7.1 is the selected target. Consult these tags, not only newer application checkouts.

Notarius v0.7.1 Compatibility

The upstream release notes describe unchanged CLI, configuration, artifact, and subprocess contracts. The application-owned dnd-extraction fallback retains its settings and no configuration migration is required. Independently of that upstream behavior, the user has selected gpt-terra-latest as this suite's new production default.

Profile selection must use the catalogs selected by v0.7.1 (PromptKit v0.10.0). The removed deepseek-3-2, deepseek-4-flash, and deepseek-4-pro catalog IDs must not be assumed available or silently replaced by similarly named profiles. This is deployment guidance, not a request for an Ansible catalog validator.

Existing checkpoints with the previous profile-source identity are not reused; source-keyed chunk plans and existing output artifacts are unaffected. Document this upstream upgrade behavior without deleting caches or forcing Narratio extraction automatically.

Each application owns its configuration. Narratio selects the downstream overlay; Notarius resolves and validates it. Existing application behavior owns extraction reuse when selections change and explicit-force handling for changes behind a stable profile name. Do not add Ansible state tracking for either.

Simplest Viable Implementation

Add one nullable role field and its conditional template output. Extend the suite's existing directory/template management to deploy the Notarius overlay and extend the existing Narratio testing template with the explicit selector. Use native serialization and ordinary file/template modules. Select the two existing catalog IDs rather than deploying duplicate profile definitions. No assembly facts, preflight state probes, or generic overlay-management framework is required.

Rationale and Alternatives

A downstream ConfigKit overlay can own model choices and later Notarius tuning without expanding Narratio's configuration surface for each setting. Direct LLM-profile forwarding would solve only the model-selection case. Automatic profile propagation would couple unrelated profile names and is not selected.

Acceptance Criteria

• The standalone role omits an unset selector and preserves explicitly supplied representative values, including the empty-string base selector.
• Native configuration inspection shows Narratio testing selecting Notarius testing, and Notarius testing selecting the requested LLM profile. Base configuration retains its production selection.
• A representative inventory-supplied testing selection is supported. Tests exercise behavior rather than asserting that today's default model stays fixed.
• Notarius's deployed-version validation accepts the composed pipeline and selected profile. No paid inference is required.
• Existing integration coverage remains valid, and immediate repeated role convergence reports no changes. Do not independently retest file-module ownership, modes, or template-file existence.
• The built collection works with its declared public role dependencies; no local sibling checkout is needed by consumers. Subtree imports follow the documented public-Gitea workflow, and published tags remain immutable.

Dependencies and Follow-up Work

The upstream selector shipped in Narratio v2.2.0; the downstream baseline is now Notarius v0.7.1. Git history preserves the completed implementation plan. Future testing-specific Notarius settings belong in its overlay but are not implicitly included in this feature.

Completion and Review Evidence

Completed and published in Narratio role 330afe7, Notarius role a82a83d, pipeline role c56a942, collection ce5b838, and consumer 0ced48c.

Implementation reported passing full integration suites for both changed application roles, the pipeline, and the collection, including clean repeated convergence. Independent review found no code-quality issues, architecture gaps, or unnecessarily strengthened requirements requiring remediation.

The review independently verified collection build and public dependency installation, syntax and shell checks, native configuration validation, consistent dependency pins, and exact subtree/source agreement. It reviewed the recorded integration results without rerunning host-mutating suites. Operational deployment and paid inference were neither required nor performed as part of this feature review.

Open Questions

None. The user selected gpt-luna-latest for testing and gpt-terra-latest for production; both are available in the catalog selected by Notarius v0.7.1.