200 lines
9.9 KiB
Markdown
200 lines
9.9 KiB
Markdown
# Feature 0017: Notarius Testing Overlay
|
|
|
|
Status: done
|
|
|
|
## Summary
|
|
|
|
Selecting Narratio's `testing` profile also selects a separately managed
|
|
Notarius `testing` configuration overlay. Operators can develop D&D assets
|
|
with economical extraction settings without modifying production configuration
|
|
or invoking Notarius separately.
|
|
|
|
## Problem or Opportunity
|
|
|
|
The suite's existing testing overlay changes Scriptorium selections but leaves
|
|
Notarius on its production configuration. Narratio v2.2.0 now provides the
|
|
explicit `pipeline.notarius.config_profile` interface requested by this consumer.
|
|
Notarius v0.7.1 retains the corresponding `--config-profile` option and updates
|
|
PromptKit and its provider catalogs. This feature adopts v0.7.1 as its baseline.
|
|
|
|
## Scope
|
|
|
|
- Expose the optional native selector in the standalone Narratio role.
|
|
- Update standalone Notarius and suite version defaults to v0.7.1, with
|
|
matching current-version documentation and collection distribution.
|
|
- Deploy a Notarius ConfigKit testing overlay through the suite role.
|
|
- Explicitly select that overlay from Narratio's existing testing overlay.
|
|
- Select `gpt-terra-latest` for production and `gpt-luna-latest` for testing,
|
|
while retaining inventory overrides for both choices.
|
|
- Document and validate the composed configuration, then distribute the
|
|
updated suite through the collection and refresh the consumer dependency pin.
|
|
|
|
## Non-goals
|
|
|
|
- Further upstream application changes or a Narratio-side LLM-profile override.
|
|
- Automatically propagating profile names between applications.
|
|
- Disabling Notarius, caching, publishing, or other pipeline stages for testing.
|
|
- New credentials, cache layouts, other production tuning, or deployment automation.
|
|
- Live controller/target deployment or paid model calls as completion gates.
|
|
|
|
## Target End State
|
|
|
|
### Standalone Narratio Interface
|
|
|
|
`narratio_notarius_config_profile` defaults to `null`. The role omits
|
|
`notarius.config_profile` when null and serializes explicitly supplied values,
|
|
including an empty string. Documentation and argument specifications expose
|
|
the field without duplicating or narrowing Narratio's selector semantics.
|
|
|
|
### Explicit Suite Composition
|
|
|
|
The suite leaves the base selector unset. Its existing Narratio
|
|
`profiles/testing.yml` includes:
|
|
|
|
```yaml
|
|
notarius:
|
|
config_profile: testing
|
|
```
|
|
|
|
The suite deploys `/usr/local/etc/notarius/profiles/testing.yml`. That file
|
|
overlays the selected suite pipeline's `llm_profile`, using
|
|
`dnd_pipeline_narratio_notarius_pipeline_id` as the pipeline key and a new
|
|
`dnd_pipeline_notarius_testing_llm_profile` variable as the value. It does not
|
|
copy the full pipeline definition. ConfigKit performs the ordinary merge.
|
|
|
|
The default suite pipeline currently inherits its LLM selection at pipeline
|
|
level. Caller-supplied binding-level selections retain Notarius's native
|
|
precedence; this feature does not rewrite arbitrary caller pipelines or promise
|
|
that a pipeline-level default overrides explicit binding choices.
|
|
|
|
### Production and Testing LLM Defaults
|
|
|
|
The suite's base `dnd-session` pipeline selects `llm_profile: gpt-terra-latest`
|
|
instead of `dnd-extraction`. Callers can continue overriding that choice through
|
|
the existing `dnd_pipeline_notarius_pipelines` mapping. The new
|
|
`dnd_pipeline_notarius_testing_llm_profile` defaults to `gpt-luna-latest`.
|
|
|
|
Both IDs are supplied by the OpenRouter catalog `v2.20260922.1` selected by
|
|
Notarius v0.7.1. Use those native catalog profiles directly; no operator profile
|
|
asset or profile-source setting is needed for these defaults. Retain the
|
|
existing `OPENROUTER_API_KEY` credential reference. These are deliberately
|
|
moving `latest` selections, not promises of immutable underlying model versions.
|
|
This production-model policy belongs to the suite, not the standalone role.
|
|
|
|
Notarius stays enabled and retains its configured caching, output contracts,
|
|
references, and production defaults outside this overlay. Existing Scriptorium
|
|
testing overrides remain intact. Credentials continue through the existing
|
|
typed reference and suite secrets interfaces; no credential values enter YAML.
|
|
|
|
Configuration files and directories use the suite's existing owner, group,
|
|
and mode variables. The selector and overlay add no new runtime identity or
|
|
filesystem-permission policy.
|
|
|
|
## Repository Ownership
|
|
|
|
- `ansible-role-narratio`: generic optional selector, documentation, and tests.
|
|
- `ansible-role-notarius`: v0.7.1 installation default and corresponding
|
|
documentation; existing profile-source settings remain sufficient.
|
|
- `ansible-role-dnd-pipeline`: testing policy, overlay assets, and composition.
|
|
- `ansible-collection-dnd`: imported pipeline snapshot and compatible dependency
|
|
manifest. Its current Narratio-role `v0.1.0` pin must advance to include the
|
|
new public variable; changing only the sibling role is insufficient.
|
|
- `rakestraw-ansible`: matching dependency pins and concise operator guidance.
|
|
- Workspace: roadmap, implementation plan, progress, and review.
|
|
- Narratio and Notarius applications: read-only authoritative contracts.
|
|
The standalone Notarius role already exposes profile-source settings and
|
|
does not need D&D-specific overlay policy.
|
|
|
|
## Constraints and Settled Decisions
|
|
|
|
Follow the [MVP roadmap](../mvp.md), [Ansible house style](../../ansible.md),
|
|
and [testing house style](../../testing.md). Narratio v2.2.0 is already selected
|
|
by the suite; Notarius v0.7.1 is the selected target. Consult these tags, not
|
|
only newer application checkouts.
|
|
|
|
### Notarius v0.7.1 Compatibility
|
|
|
|
The [upstream release notes](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.1/docs/releases/v0.7.1.md)
|
|
describe unchanged CLI, configuration, artifact, and subprocess contracts.
|
|
The application-owned `dnd-extraction` fallback retains its settings and no
|
|
configuration migration is required. Independently of that upstream behavior,
|
|
the user has selected `gpt-terra-latest` as this suite's new production default.
|
|
|
|
Profile selection must use the catalogs selected by v0.7.1 (PromptKit v0.10.0).
|
|
The removed `deepseek-3-2`, `deepseek-4-flash`, and `deepseek-4-pro` catalog IDs
|
|
must not be assumed available or silently replaced by similarly named profiles.
|
|
This is deployment guidance, not a request for an Ansible catalog validator.
|
|
|
|
Existing checkpoints with the previous profile-source identity are not reused;
|
|
source-keyed chunk plans and existing output artifacts are unaffected. Document
|
|
this upstream upgrade behavior without deleting caches or forcing Narratio
|
|
extraction automatically.
|
|
|
|
Each application owns its configuration. Narratio selects the downstream
|
|
overlay; Notarius resolves and validates it. Existing application behavior owns
|
|
extraction reuse when selections change and explicit-force handling for changes
|
|
behind a stable profile name. Do not add Ansible state tracking for either.
|
|
|
|
## Simplest Viable Implementation
|
|
|
|
Add one nullable role field and its conditional template output. Extend the
|
|
suite's existing directory/template management to deploy the Notarius overlay
|
|
and extend the existing Narratio testing template with the explicit selector.
|
|
Use native serialization and ordinary file/template modules. Select the two
|
|
existing catalog IDs rather than deploying duplicate profile definitions. No assembly facts,
|
|
preflight state probes, or generic overlay-management framework is required.
|
|
|
|
## Rationale and Alternatives
|
|
|
|
A downstream ConfigKit overlay can own model choices and later Notarius tuning
|
|
without expanding Narratio's configuration surface for each setting. Direct
|
|
LLM-profile forwarding would solve only the model-selection case. Automatic
|
|
profile propagation would couple unrelated profile names and is not selected.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- The standalone role omits an unset selector and preserves explicitly supplied
|
|
representative values, including the empty-string base selector.
|
|
- Native configuration inspection shows Narratio testing selecting Notarius
|
|
testing, and Notarius testing selecting the requested LLM profile. Base
|
|
configuration retains its production selection.
|
|
- A representative inventory-supplied testing selection is supported. Tests
|
|
exercise behavior rather than asserting that today's default model stays fixed.
|
|
- Notarius's deployed-version validation accepts the composed pipeline and
|
|
selected profile. No paid inference is required.
|
|
- Existing integration coverage remains valid, and immediate repeated role
|
|
convergence reports no changes. Do not independently retest file-module
|
|
ownership, modes, or template-file existence.
|
|
- The built collection works with its declared public role dependencies; no
|
|
local sibling checkout is needed by consumers. Subtree imports follow the
|
|
documented public-Gitea workflow, and published tags remain immutable.
|
|
|
|
## Dependencies and Follow-up Work
|
|
|
|
The upstream selector shipped in Narratio v2.2.0; the downstream baseline is
|
|
now Notarius v0.7.1. Git history preserves the completed implementation plan.
|
|
Future testing-specific Notarius settings belong in its overlay but are not
|
|
implicitly included in this feature.
|
|
|
|
## Completion and Review Evidence
|
|
|
|
Completed and published in Narratio role `330afe7`, Notarius role `a82a83d`,
|
|
pipeline role `c56a942`, collection `ce5b838`, and consumer `0ced48c`.
|
|
|
|
Implementation reported passing full integration suites for both changed
|
|
application roles, the pipeline, and the collection, including clean repeated
|
|
convergence. Independent review found no code-quality issues, architecture gaps,
|
|
or unnecessarily strengthened requirements requiring remediation.
|
|
|
|
The review independently verified collection build and public dependency
|
|
installation, syntax and shell checks, native configuration validation,
|
|
consistent dependency pins, and exact subtree/source agreement. It reviewed
|
|
the recorded integration results without rerunning host-mutating suites.
|
|
Operational deployment and paid inference were neither required nor performed
|
|
as part of this feature review.
|
|
|
|
## Open Questions
|
|
|
|
None. The user selected `gpt-luna-latest` for testing and `gpt-terra-latest`
|
|
for production; both are available in the catalog selected by Notarius v0.7.1.
|