Add the Notarius testing overlay and production model defaults

2026-09-22 18:52:39 +00:00
parent 8ceb3f4941
commit c56a9425d3
8 changed files with 127 additions and 45 deletions
--- a/README.md
+++ b/README.md
@@ -18,7 +18,8 @@ ansible-galaxy collection install -r collections/requirements.yml
 The dependency manifest installs the roles under their canonical names:
 `rakestraw.seriatim`, `rakestraw.audita`, `rakestraw.notarius`,
 `rakestraw.scriptorium`, and `rakestraw.narratio`. It selects the compatible
-`v0.1.0` release of each role. This standalone role is named
+`v0.1.0` releases where compatible and public commit pins for Narratio and
+Notarius when the suite needs newer interfaces. This standalone role is named
 `rakestraw.dnd_pipeline`.

 ```yaml
@@ -62,7 +63,7 @@ The compatible application versions are selected through these variables:
 | --- | --- |
 | `dnd_pipeline_audita_version` | `v0.9.1` |
 | `dnd_pipeline_narratio_version` | `v2.2.0` |
-| `dnd_pipeline_notarius_version` | `v0.7.0` |
+| `dnd_pipeline_notarius_version` | `v0.7.1` |
 | `dnd_pipeline_seriatim_version` | `v1.5.0` |
 | `dnd_pipeline_scriptorium_version` | `v0.13.0` |
 | `dnd_pipeline_manage_build_dependencies` | `true` |
@@ -146,6 +147,7 @@ run-specific output path.
 | `dnd_pipeline_narratio_render_enabled` | `true` |
 | `dnd_pipeline_narratio_render_format` | `markdown` |
 | `dnd_pipeline_narratio_testing_scriptorium_profile` | `gemini-flash-latest` |
+| `dnd_pipeline_notarius_testing_llm_profile` | `gpt-luna-latest` |

 `dnd_pipeline_audita_output_schema` is also the Narratio Audita-adapter schema;
 it is the shared cross-application boundary. Other Narratio-local controls,
@@ -222,14 +224,25 @@ suite-owned base fragments in this fixed order:
 └── profiles/testing.yml
 ```

-The `testing` profile is an overlay, not a second configuration tree. It only
-sets the Scriptorium `profile_id` for the four artifact sources and the one
-family. Its default, `gemini-flash-latest`, is an inexpensive development
+Notarius follows the corresponding base-and-overlay layout:
+
+```text
+/usr/local/etc/notarius/
+├── config.yml
+├── pipelines/dnd-session.yml
+└── profiles/testing.yml
+```
+
+The `testing` profile is an overlay, not a second configuration tree. It sets
+the Scriptorium `profile_id` for the four artifact sources and the one family,
+and explicitly selects Notarius's separately managed `testing` overlay. The
+Scriptorium default, `gemini-flash-latest`, is an inexpensive development
 choice, not a compatibility guarantee. Replace it through inventory when a
 different testing profile is appropriate:

 ```yaml
 dnd_pipeline_narratio_testing_scriptorium_profile: endpoint-profile
+dnd_pipeline_notarius_testing_llm_profile: alternate-economy-profile
 ```

 ### Notarius extraction policy
@@ -237,6 +250,8 @@ dnd_pipeline_narratio_testing_scriptorium_profile: endpoint-profile
 The Notarius root configuration imports the fixed
 `pipelines/dnd-session.yml` fragment. The suite owns that fragment at
 `/usr/local/etc/notarius/pipelines/dnd-session.yml`; callers do not edit it.
+The separate `/usr/local/etc/notarius/profiles/testing.yml` overlay changes only
+the selected suite pipeline's LLM profile.

 | Variable | Default policy |
 | --- | --- |
@@ -244,6 +259,7 @@ The Notarius root configuration imports the fixed
 | `dnd_pipeline_notarius_cache_root` | `/var/cache/notarius` |
 | `dnd_pipeline_notarius_cache_chunk_plans_mode` | `auto` |
 | `dnd_pipeline_notarius_cache_checkpoints_enabled` | `true` |
+| `dnd_pipeline_notarius_testing_llm_profile` | `gpt-luna-latest` |
 | `dnd_pipeline_narratio_notarius_pipeline_id` | `dnd-session` |
 | `dnd_pipeline_narratio_notarius_llm_api_key` | `OPENROUTER_API_KEY` |
 | `dnd_pipeline_narratio_notarius_resume` | `true` |
@@ -252,7 +268,11 @@ The Notarius root configuration imports the fixed

 The pipeline, reference, and output values are complete native mappings.
 Inventory can replace them or compose additions with Jinja's `combine` filter;
-a replacement pipeline map requires a matching selected pipeline ID.
+a replacement pipeline map requires a matching selected pipeline ID. The base
+`dnd-session` pipeline selects `gpt-terra-latest`, while the testing overlay
+selects `gpt-luna-latest`. Both are moving catalog IDs rather than immutable
+model promises. Pipeline and binding-level overrides retain Notarius's native
+precedence; the suite does not rewrite caller-supplied bindings.

 Narratio materializes the `party`, `players`, and `glossary` references for each
 invocation. No campaign filesystem path appears in Notarius configuration. The
@@ -270,10 +290,13 @@ Notarius profile and ordinary extraction requests checkpoint reuse with
 `resume: true`. Chunk plans use Notarius's root-owned `auto` policy; the
 pipeline deliberately leaves Narratio's optional `chunk_cache` override unset.
 The `chunk-plans` and `checkpoints` paths derive from the cache root and are
-created by Notarius during use. Base and `testing` configurations select the
-same `dnd-extraction` profile and Notarius pipeline; the testing overlay changes
-only Scriptorium profiles, so a real testing run may invoke Notarius. The
-integration suite performs offline configuration inspection only.
+created by Notarius during use. Notarius v0.7.1 selects PromptKit v0.10.0 and
+OpenRouter catalog v2.20260922.1, which supplies both suite-selected IDs.
+Existing checkpoints with the previous profile-source identity are not reused;
+source-keyed chunk plans and output artifacts remain compatible. No cache
+deletion or forced extraction is performed by this role. A real testing run may
+invoke Notarius; the integration suite performs offline configuration inspection
+only and makes no provider request.

 ### Inspecting configuration

--- a/defaults/main.yml
+++ b/defaults/main.yml
@@ -1,7 +1,7 @@
 ---
 dnd_pipeline_audita_version: v0.9.1
 dnd_pipeline_narratio_version: v2.2.0
-dnd_pipeline_notarius_version: v0.7.0
+dnd_pipeline_notarius_version: v0.7.1
 dnd_pipeline_seriatim_version: v1.5.0
 dnd_pipeline_scriptorium_version: v0.13.0
 dnd_pipeline_manage_build_dependencies: true
@@ -58,7 +58,7 @@ dnd_pipeline_audita_diagnostics_work_dir: /tmp/audita
 dnd_pipeline_audita_diagnostics_retention: auto
 dnd_pipeline_notarius_pipelines:
  dnd-session:
-    llm_profile: dnd-extraction
+    llm_profile: gpt-terra-latest
    input:
      module: seriatim
    chunk:
@@ -207,6 +207,7 @@ dnd_pipeline_notarius_pipelines:
 dnd_pipeline_notarius_cache_root: /var/cache/notarius
 dnd_pipeline_notarius_cache_chunk_plans_mode: auto
 dnd_pipeline_notarius_cache_checkpoints_enabled: true
+dnd_pipeline_notarius_testing_llm_profile: gpt-luna-latest
 dnd_pipeline_narratio_workspace_cleanup_after_publish: true
 dnd_pipeline_narratio_storage_backend: s3
 dnd_pipeline_narratio_storage_s3_bucket: rakestraw-assets
--- a/meta/argument_specs.yml
+++ b/meta/argument_specs.yml
@@ -221,6 +221,10 @@ argument_specs:
        type: bool
        required: true
        description: Whether Notarius records checkpoints.
+      dnd_pipeline_notarius_testing_llm_profile:
+        type: str
+        required: true
+        description: Notarius LLM profile selected by the testing overlay.
      dnd_pipeline_narratio_workspace_cleanup_after_publish:
        type: bool
        required: true
--- a/requirements.yml
+++ b/requirements.yml
@@ -11,7 +11,7 @@ roles:
  - name: rakestraw.notarius
    src: https://gitea.maximumdirect.net/eric/ansible-role-notarius.git
    scm: git
-    version: v0.1.0
+    version: a82a83dfc23bb80a59f45f94f78d6754c47285e7
  - name: rakestraw.scriptorium
    src: https://gitea.maximumdirect.net/eric/ansible-role-scriptorium.git
    scm: git
@@ -19,4 +19,4 @@ roles:
  - name: rakestraw.narratio
    src: https://gitea.maximumdirect.net/eric/ansible-role-narratio.git
    scm: git
-    version: v0.1.0
+    version: 330afe7856da94b2402a75447f4b60aba767843f
--- a/tasks/main.yml
+++ b/tasks/main.yml
@@ -146,14 +146,17 @@
    - artifact-families
    - profiles

- name: Create the Notarius pipeline configuration directory
+- name: Create Notarius configuration directories
  become: true
  ansible.builtin.file:
-    path: /usr/local/etc/notarius/pipelines
+    path: "/usr/local/etc/notarius/{{ item }}"
    state: directory
    owner: "{{ dnd_pipeline_runtime_user }}"
    group: "{{ dnd_pipeline_runtime_group }}"
    mode: "{{ dnd_pipeline_config_directory_mode }}"
+  loop:
+    - pipelines
+    - profiles

 - name: Deploy the Notarius D&D pipeline configuration
  become: true
@@ -164,6 +167,15 @@
    group: "{{ dnd_pipeline_runtime_group }}"
    mode: "{{ dnd_pipeline_config_file_mode }}"

+- name: Deploy the Notarius testing overlay
+  become: true
+  ansible.builtin.template:
+    src: notarius-testing.yml.j2
+    dest: /usr/local/etc/notarius/profiles/testing.yml
+    owner: "{{ dnd_pipeline_runtime_user }}"
+    group: "{{ dnd_pipeline_runtime_group }}"
+    mode: "{{ dnd_pipeline_config_file_mode }}"
+
 - name: Deploy Narratio D&D configuration fragments
  become: true
  ansible.builtin.template:
--- a/templates/notarius-testing.yml.j2
+++ b/templates/notarius-testing.yml.j2
@@ -0,0 +1,3 @@
+pipelines:
+  {{ dnd_pipeline_narratio_notarius_pipeline_id | to_json }}:
+    llm_profile: {{ dnd_pipeline_notarius_testing_llm_profile | to_json }}
--- a/templates/testing.yml.j2
+++ b/templates/testing.yml.j2
@@ -1,3 +1,5 @@
+notarius:
+  config_profile: testing
 scriptorium:
  artifacts:
    session_recap:
--- a/tests/integration
+++ b/tests/integration
@@ -44,6 +44,8 @@ baseline_args=(
  -e 'dnd_pipeline_narratio_storage_s3_secret_access_key=OBJECT_STORAGE_KEY'
  -e '{"dnd_pipeline_narratio_workspace_cleanup_after_publish":false,"dnd_pipeline_narratio_storage_s3_force_path_style":false,"dnd_pipeline_narratio_cache_s3_audio":false,"dnd_pipeline_narratio_publish_enabled":true,"dnd_pipeline_narratio_publish_upload_run":true}'
  -e '{"dnd_pipeline_narratio_publish_outputs":[{"source":"narratio.transcript.final","required":true},{"source":"narratio.artifact.session_recap","required":true}]}'
+  -e 'dnd_pipeline_narratio_testing_scriptorium_profile=endpoint-profile'
+  -e 'dnd_pipeline_notarius_testing_llm_profile=gpt-terra-latest'
 )

 cleanup() {
@@ -91,9 +93,14 @@ run_converge "$baseline_log" "${baseline_args[@]}"
 /usr/local/bin/audita config validate --config /usr/local/etc/audita/config.yml
 /usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml \
  --pipeline dnd-session
+/usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml \
+  --config-profile testing --pipeline dnd-session
 notarius_effective_config="$test_tmp/notarius-effective.yml"
 /usr/local/bin/notarius config effective --config /usr/local/etc/notarius/config.yml \
  > "$notarius_effective_config"
+notarius_testing_effective_config="$test_tmp/notarius-testing-effective.yml"
+/usr/local/bin/notarius config effective --config /usr/local/etc/notarius/config.yml \
+  --config-profile testing > "$notarius_testing_effective_config"
 grep -Fq 'mode: auto' "$notarius_effective_config"
 grep -Fq 'step: describe-session' "$notarius_effective_config"
 /usr/local/bin/scriptorium inspect profile --config /usr/local/etc/scriptorium/config.yml \
@@ -108,6 +115,10 @@ narratio_effective_config="$test_tmp/narratio-effective.yml"
 /usr/bin/sudo -u narratio /usr/local/bin/narratio config show \
  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs \
  > "$narratio_effective_config"
+narratio_testing_effective_config="$test_tmp/narratio-testing-effective.yml"
+/usr/bin/sudo -u narratio /usr/local/bin/narratio config show \
+  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs --profile testing \
+  > "$narratio_testing_effective_config"
 grep -Fq 'enabled: true' "$narratio_effective_config"
 grep -Fq 'llm_api_key: AUDITA_LLM_API_KEY' "$narratio_effective_config"
 grep -Fq 'pipeline_id: dnd-session' "$narratio_effective_config"
@@ -116,13 +127,16 @@ grep -Fq 'resume: true' "$narratio_effective_config"
 ! grep -Fq 'chunk_cache:' "$narratio_effective_config"
 grep -Fq 'party: narratio.input.party' "$narratio_effective_config"
 grep -Fq 'lane_id: item-registry' "$narratio_effective_config"
-"$workspace_venv/bin/python" - "$narratio_effective_config" <<'PY'
+"$workspace_venv/bin/python" - \
+  "$narratio_effective_config" "$narratio_testing_effective_config" <<'PY'
 import sys

 import yaml

 with open(sys.argv[1], encoding="utf-8") as config_file:
    config = yaml.safe_load(config_file)
+with open(sys.argv[2], encoding="utf-8") as config_file:
+    testing_config = yaml.safe_load(config_file)

 assert config["workspace"]["cleanup_after_publish"] is False
 assert config["storage"] == {
@@ -154,11 +168,21 @@ assert config["publish"] == {
        },
    ],
 }
+assert "config_profile" not in config["notarius"]
+assert testing_config["notarius"]["config_profile"] == "testing"
+assert all(
+    artifact["profile_id"] == "endpoint-profile"
+    for artifact in testing_config["scriptorium"]["artifacts"].values()
+)
+assert any(
+    artifact_id.startswith("character_meta_analysis_")
+    for artifact_id in testing_config["scriptorium"]["artifacts"]
+)
 PY
-! grep -Fq "$synthetic_audita_api_key" "$baseline_log" "$narratio_effective_config" "$notarius_effective_config"
-! grep -Fq "$synthetic_notarius_api_key" "$baseline_log" "$narratio_effective_config" "$notarius_effective_config"
-! grep -Fq "$synthetic_object_storage_key_id" "$baseline_log" "$narratio_effective_config" "$notarius_effective_config"
-! grep -Fq "$synthetic_object_storage_key" "$baseline_log" "$narratio_effective_config" "$notarius_effective_config"
+! grep -Fq "$synthetic_audita_api_key" "$baseline_log" "$narratio_effective_config" "$narratio_testing_effective_config" "$notarius_effective_config" "$notarius_testing_effective_config"
+! grep -Fq "$synthetic_notarius_api_key" "$baseline_log" "$narratio_effective_config" "$narratio_testing_effective_config" "$notarius_effective_config" "$notarius_testing_effective_config"
+! grep -Fq "$synthetic_object_storage_key_id" "$baseline_log" "$narratio_effective_config" "$narratio_testing_effective_config" "$notarius_effective_config" "$notarius_testing_effective_config"
+! grep -Fq "$synthetic_object_storage_key" "$baseline_log" "$narratio_effective_config" "$narratio_testing_effective_config" "$notarius_effective_config" "$notarius_testing_effective_config"
 /usr/bin/sudo -u narratio /usr/local/bin/narratio config artifacts \
  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs | \
  awk -F '\t' '$1 == "session_recap" && $4 == "true" { found = 1 } END { exit !found }'
@@ -168,9 +192,6 @@ PY
 /usr/bin/sudo -u narratio /usr/local/bin/narratio config artifacts \
  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs | \
  awk -F '\t' 'index($1, "character_meta_analysis_") == 1 && $4 == "false" && $7 == "character_meta_analysis" { found = 1 } END { exit !found }'
-/usr/bin/sudo -u narratio /usr/local/bin/narratio config diff default testing \
-  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs | \
-  awk -F '\t' '$1 != "changed" || $2 !~ /\.profile_id$/ { invalid = 1 } END { exit invalid || NR == 0 }'
 /usr/bin/sudo -u "$operator_user" /usr/local/bin/narratio config validate \
  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs
 /usr/bin/sudo -u "$operator_user" /usr/local/bin/notarius config validate \
@@ -218,21 +239,6 @@ if run_converge "$invalid_key_log" "${baseline_args[@]}" \
  exit 1
 fi

-testing_profile_log="$test_tmp/testing-profile.log"
-run_converge "$testing_profile_log" "${baseline_args[@]}" \
-  -e 'dnd_pipeline_narratio_testing_scriptorium_profile=endpoint-profile'
-/usr/bin/sudo -u narratio /usr/local/bin/narratio config validate \
-  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs --profile testing
-/usr/bin/sudo -u narratio /usr/local/bin/narratio config diff default testing \
-  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs | \
-  grep -F '"endpoint-profile"'
-/usr/local/bin/scriptorium inspect profile --config /usr/local/etc/scriptorium/config.yml \
-  --profile-dir "$profile_dir" --profile endpoint-profile
-testing_profile_repeat_log="$test_tmp/testing-profile-repeat.log"
-run_converge "$testing_profile_repeat_log" "${baseline_args[@]}" \
-  -e 'dnd_pipeline_narratio_testing_scriptorium_profile=endpoint-profile'
-require_clean_recap "$testing_profile_repeat_log"
-
 selected_version_log="$test_tmp/selected-version.log"
 run_converge "$selected_version_log" "${baseline_args[@]}" \
  -e 'dnd_pipeline_seriatim_version=v1.4.0'
@@ -242,24 +248,55 @@ run_converge "$selected_version_repeat_log" "${baseline_args[@]}" \
  -e 'dnd_pipeline_seriatim_version=v1.4.0'
 require_clean_recap "$selected_version_repeat_log"

-pipeline_group_one='{"archive-session":{"llm_profile":"dnd-extraction","input":{"module":"seriatim"},"artifacts":{"spells":{"extract":{"module":"dnd/spells"},"normalize":{"module":"dnd/spells"}}}}}'
-pipeline_group_two='{"research-session":{"llm_profile":"dnd-extraction","input":{"module":"seriatim"},"artifacts":{"spells":{"extract":{"module":"dnd/spells"},"normalize":{"module":"dnd/spells"}}}}}'
-alternate_config="{\"notarius_pipeline_group_one\":$pipeline_group_one,\"notarius_pipeline_group_two\":$pipeline_group_two,\"dnd_pipeline_notarius_pipelines\":\"{{ notarius_pipeline_group_one | combine(notarius_pipeline_group_two) }}\",\"dnd_pipeline_narratio_notarius_pipeline_id\":\"archive-session\",\"dnd_pipeline_notarius_cache_chunk_plans_mode\":\"bypass\"}"
+pipeline_group_one='{"archive-session":{"llm_profile":"gpt-luna-latest","input":{"module":"seriatim"},"artifacts":{"spells":{"extract":{"module":"dnd/spells"},"normalize":{"module":"dnd/spells"}}}}}'
+pipeline_group_two='{"research-session":{"llm_profile":"gpt-luna-latest","input":{"module":"seriatim"},"artifacts":{"spells":{"extract":{"module":"dnd/spells"},"normalize":{"module":"dnd/spells"}}}}}'
+alternate_config="{\"notarius_pipeline_group_one\":$pipeline_group_one,\"notarius_pipeline_group_two\":$pipeline_group_two,\"dnd_pipeline_notarius_pipelines\":\"{{ notarius_pipeline_group_one | combine(notarius_pipeline_group_two) }}\",\"dnd_pipeline_narratio_notarius_pipeline_id\":\"archive-session\",\"dnd_pipeline_notarius_testing_llm_profile\":\"gpt-terra-latest\",\"dnd_pipeline_notarius_cache_chunk_plans_mode\":\"bypass\"}"
 alternate_log="$test_tmp/alternate.log"
 run_converge "$alternate_log" "${baseline_args[@]}" -e "$alternate_config"
 /usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml \
  --pipeline archive-session
+/usr/local/bin/notarius config validate --config /usr/local/etc/notarius/config.yml \
+  --config-profile testing --pipeline archive-session
 alternate_narratio_config="$test_tmp/alternate-narratio.yml"
 /usr/bin/sudo -u narratio /usr/local/bin/narratio config show \
  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs \
  > "$alternate_narratio_config"
 grep -Fq 'pipeline_id: archive-session' "$alternate_narratio_config"
+alternate_testing_narratio_config="$test_tmp/alternate-testing-narratio.yml"
+/usr/bin/sudo -u narratio /usr/local/bin/narratio config show \
+  --config /usr/local/etc/narratio/pipeline.yml --campaign centaurs --profile testing \
+  > "$alternate_testing_narratio_config"
 alternate_effective_config="$test_tmp/alternate-effective.yml"
 /usr/local/bin/notarius config effective --config /usr/local/etc/notarius/config.yml \
  > "$alternate_effective_config"
-grep -Fq 'archive-session:' "$alternate_effective_config"
-grep -Fq 'research-session:' "$alternate_effective_config"
-grep -Fq 'mode: bypass' "$alternate_effective_config"
+alternate_testing_effective_config="$test_tmp/alternate-testing-effective.yml"
+/usr/local/bin/notarius config effective --config /usr/local/etc/notarius/config.yml \
+  --config-profile testing > "$alternate_testing_effective_config"
+"$workspace_venv/bin/python" - \
+  "$alternate_narratio_config" "$alternate_testing_narratio_config" \
+  "$alternate_effective_config" "$alternate_testing_effective_config" <<'PY'
+import sys
+
+import yaml
+
+with open(sys.argv[1], encoding="utf-8") as config_file:
+    narratio_base = yaml.safe_load(config_file)
+with open(sys.argv[2], encoding="utf-8") as config_file:
+    narratio_testing = yaml.safe_load(config_file)
+with open(sys.argv[3], encoding="utf-8") as config_file:
+    notarius_base = yaml.safe_load(config_file)
+with open(sys.argv[4], encoding="utf-8") as config_file:
+    notarius_testing = yaml.safe_load(config_file)
+
+assert narratio_base["notarius"]["pipeline_id"] == "archive-session"
+assert "config_profile" not in narratio_base["notarius"]
+assert narratio_testing["notarius"]["pipeline_id"] == "archive-session"
+assert narratio_testing["notarius"]["config_profile"] == "testing"
+assert notarius_base["pipelines"]["archive-session"]["llm_profile"] == "gpt-luna-latest"
+assert notarius_testing["pipelines"]["archive-session"]["llm_profile"] == "gpt-terra-latest"
+assert notarius_testing["pipelines"]["research-session"]["llm_profile"] == "gpt-luna-latest"
+assert notarius_base["cache"]["chunk_plans"]["mode"] == "bypass"
+PY
 alternate_repeat_log="$test_tmp/alternate-repeat.log"
 run_converge "$alternate_repeat_log" "${baseline_args[@]}" -e "$alternate_config"
 require_clean_recap "$alternate_repeat_log"