ansible-workspace-dnd/docs/roadmap/implementation.md

Implementation Plan 0010: D&D Notarius Extraction Integration

Status: ready

Target executor: GPT-5.6-Terra

Feature roadmap: docs/roadmap/features/0010-dnd-notarius-extraction-integration.md

Active plan location: docs/roadmap/implementation.md

Objective

Upgrade the generic Notarius role to the v0.7.0 configuration contract, then have the D&D pipeline role deploy the maintained ten-lane extraction pipeline, enable it through Narratio, bind prepared campaign references, accept all ten lane contracts, and provision the shared FHS cache policy established by Feature 0009.

The feature roadmap remains authoritative for scope, rationale, target end state, and acceptance criteria. Decisions 0001 and 0002 remain authoritative for direct Ansible convergence and the shared-readable filesystem boundary.

Execution Rules

• Execute stages in numeric order.
• Before each stage, confirm its preconditions and the working-tree state of its owning repository.
• Before each stage, reread every document listed in that stage's Required Policy Review. Do not rely on policy recalled from an earlier stage.
• Preserve unrelated and pre-existing changes. Stop if an owning repository is unexpectedly dirty at a stage boundary.
• Keep ansible-role-notarius generic. ConfigKit import support is a native application interface; the D&D pipeline, campaign bindings, cache policy, and Narratio coordination belong only to ansible-role-dnd-pipeline.
• Use the Notarius v0.7.0 maintained complete pipeline as the topology and contract authority. Remove external path references, but retain generated artifact handoffs and canonical version 5 binding objects.
• Trust the existing application-role templates and Ansible modules. Do not add production validation commands, preliminary probes, manual comparisons, handlers, checksums, manifests, state files, or corrective permission tasks.
• Create only the shared Notarius cache root. Notarius owns creation of its configured chunk-plans and checkpoints children during actual use.
• Keep the existing Narratio testing overlay unchanged so it inherits the enabled Notarius stage and dnd-extraction workload profile.
• Do not add credentials, a PromptKit profile file, campaign paths, a global Notarius output or debug root, checkpoint resume, extraction execution, model calls, storage, publishing, services, schedules, or user-membership management.
• Test public behavior through the installed applications' native inspection and validation interfaces. Do not snapshot complete configuration files, parse production task results, inspect filesystem metadata, or reproduce application and Ansible validators in assertions.
• Run every stage's validation before committing it. Commit each completed stage when it leaves the owning repository valid, using the plain-English commit intent specified by that stage.
• Do not begin a dependent stage after an earlier stage fails validation.
• The pipeline integration suite installs role dependencies from their canonical Git origins. The Stage 1 Notarius-role commit must therefore be reachable from its canonical origin before Stage 2 begins. If it is not, stop after Stage 1 and report the commit hash for publication; do not use a branch, local path, or weakened dependency test as a substitute.

Starting State and Preconditions

• The workspace control repository contains the reviewed Feature 0010 roadmap and this implementation plan on main in a clean planning commit descended from f91871ba0f90da19592eb56824f5abb9e7868285.

• Features 0001 through 0009 and the standalone-role test-suite simplification are complete.

• The implementation repositories are clean on main at these reviewed starting commits:

  Repository	Starting commit
  ansible-role-notarius	e9ae2dc34023e1e8462bf0991de62b97d8c0e0ee
  ansible-role-dnd-pipeline	b15fe8dcf57a53784bc3b0b8bf04aa97d39edb3f

• The Notarius-role starting commit matches origin/main. The pipeline role is intentionally ahead of its canonical origin and must not be reset or replaced.

• ../notarius is clean at the immutable v0.7.0 tag, 4e77725b24cb62bd1f96ee31b4d17b4a2ed68ba7.

• ../narratio is clean at the immutable v2.0.0 tag, ed4623bc415901134c765fe9cf06d59aa7891500.

• Notarius v0.7.0 preserves the subprocess receipt and extraction-artifact contracts consumed by Narratio v2.0.0 while requiring Notarius configuration version 5.

• The workspace .venv provides Ansible Core 2.21.3 and ansible-lint 26.8.0.

• The local integration environment remains an accepted proxy for Ubuntu 26.04, supports task-level privilege escalation, and provides ordinary POSIX ACL behavior.

Repository Scope

Repositories that change:

• ansible-role-notarius: v0.7.0 selection, configuration version 5, nullable native ConfigKit imports, generic documentation, and focused integration coverage.
• ansible-role-dnd-pipeline: compatible Notarius role and application pins, opinionated D&D pipeline configuration, Narratio integration, shared cache root, documentation, and focused integration coverage.
• ansible-workspace-dnd: implementation progress and review handoff only after the role stages are complete.

Repositories that may be inspected but must remain unchanged:

• ../notarius and ../narratio, which are read-only application authorities.
• ansible-role-narratio, whose generic Notarius adapter already exposes all fields required by this feature.
• ansible-role-audita, ansible-role-seriatim, ansible-role-scriptorium, and ansible-collection-dnd.
• assets/campaigns, assets/narratio-prompts, and assets/configs.
• rakestraw-ansible and every other repository in workspace.yml.

Stage 1: Upgrade the Generic Notarius Role

Status: complete (14a38e04e35ef524d6a97fa7fb5a04981a0a2161)

Objective

Make the standalone role install Notarius v0.7.0, render configuration version 5, and expose the application's native ConfigKit import list without adding D&D-specific policy.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/roadmap/features/0010-dnd-notarius-extraction-integration.md
• docs/decisions/0001-minimal-module-driven-ansible.md
• ../notarius/docs/releases/v0.7.0.md
• ../notarius/docs/config.md
• ../notarius/docs/cli.md
• ../notarius/examples/dnd-complete.config.yml
• ../notarius/examples/pipelines/dnd-session.yml
• ansible-role-notarius/defaults/main.yml
• ansible-role-notarius/meta/argument_specs.yml
• ansible-role-notarius/templates/config.yml.j2
• ansible-role-notarius/tests/integration
• ansible-role-notarius/tests/verify-config.yml
• ansible-role-notarius/README.md

Preconditions

• ansible-role-notarius is clean at its stated starting commit.
• The repository's existing tests/integration entry point passes before modification.
• The v0.7.0 upstream files listed above remain at the reviewed immutable tag.

Owning Repository

ansible-role-notarius

Required Changes

1. Advance notarius_version from v0.6.0 to v0.7.0 and notarius_config_version from 4 to 5. Do not change the established source-build task flow or other generic defaults.
2. Add notarius_configkit_imports: null as one optional public variable and add a matching argument-specification entry using a list of native mapping objects. Do not create field aliases for import entries or add role-owned merge behavior.
3. Render a top-level configkit.imports section only when notarius_configkit_imports is not none. Serialize the supplied list with an established Ansible/Jinja data filter; do not manually construct YAML or alter notarius_pipelines.
4. Keep every other optional field governed by its current null-omission behavior. The role manages only its root configuration and must not create any caller-selected imported fragment.
5. Update the README's version, configuration version, variable list, and configuration discussion. Explain that ConfigKit imports are caller-owned native composition inputs and that relative imports resolve from the managed root configuration directory. Do not mention the D&D suite's fixed fragment or cache policy as a standalone default.
6. Migrate the integration fixture from configuration version 4 shorthand to canonical version 5 module-binding objects. Use a test-owned imported fragment to exercise notarius_configkit_imports; fixture creation is test setup and must not become a production role task.
7. Exercise an explicitly selected notarius_version: v0.7.0, caller-side composition of two small pipeline maps, an intentional falsey cache value, a representative cache override, and the existing caller-supplied Go executable/missing-binary recovery behavior. Use v0.7.0 for cases that need a configuration-compatible executable.
8. Validate the representative imported pipeline with notarius config validate --pipeline <id> and observe composition and the cache override through notarius config effective or another native inspection command. Remove direct managed-file assertions from tests/verify-config.yml; delete that playbook if it has no remaining behavioral purpose.
9. Retain an immediate identical repeat that reports changed=0 and failed=0. Do not assert that v0.7.0 remains the role default, test ConfigKit itself, or add production validation and state inspection.

Validation

Run from ansible-role-notarius:

tests/integration
git diff --check
git status --short

Expected result: syntax and production-profile lint pass; the explicitly selected v0.7.0 binary reports its selected version; native Notarius commands validate and inspect the representative composed version 5 configuration; missing-binary recovery still works with the caller's Go executable; all immediate repeats report changed=0 and failed=0; and only intended role, test, and README files are modified.

Completion Criteria

• Defaults, argument specification, template, README, and tests expose one coherent generic ConfigKit import interface.
• The root document is version 5 and no D&D pipeline or caller-owned fragment is embedded in the standalone role.
• Tests observe native application behavior and do not snapshot defaults or re-test Ansible module guarantees.
• The repository is valid and the stage is committed.

Commit

Commit intent: Upgrade the Notarius role to version 0.7.0

Stage 2: Implement the Suite Notarius Integration

Status: complete (90893ae)

Objective

Add the pipeline role's complete D&D Notarius configuration, Narratio wiring, and shared cache convergence using the reviewed standalone-role interface.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/roadmap/features/0010-dnd-notarius-extraction-integration.md
• docs/decisions/0001-minimal-module-driven-ansible.md
• docs/decisions/0002-shared-readable-pipeline-filesystem-policy.md
• the committed Stage 1 changes in ansible-role-notarius
• ../notarius/docs/releases/v0.7.0.md
• ../notarius/docs/config.md
• ../notarius/docs/consumers/dnd-pipeline.md
• ../notarius/examples/dnd-complete.config.yml
• ../notarius/examples/pipelines/dnd-session.yml
• ../narratio/docs/integrations/notarius.md
• ../narratio/examples/pipeline.full.annotated.yml
• ansible-role-dnd-pipeline/defaults/main.yml
• ansible-role-dnd-pipeline/meta/argument_specs.yml
• ansible-role-dnd-pipeline/meta/main.yml
• ansible-role-dnd-pipeline/requirements.yml
• ansible-role-dnd-pipeline/tasks/main.yml
• ansible-role-dnd-pipeline/templates/testing.yml.j2
• ansible-role-dnd-pipeline/tests/integration

Preconditions

• Stage 1 is committed, validated, and clean.
• The Stage 1 commit is reachable from the canonical ansible-role-notarius Git origin. Stop and report the hash if a fresh Git or Galaxy fetch cannot resolve it.
• ansible-role-dnd-pipeline is clean at its stated starting commit or an expected reviewed descendant.
• Its existing tests/integration entry point passes before modification.

Owning Repository

ansible-role-dnd-pipeline

Required Changes

1. Set dnd_pipeline_notarius_version: v0.7.0 and pin the rakestraw.notarius entry in requirements.yml to the exact Stage 1 commit. Retain the canonical Git URL and every unrelated dependency pin.
2. Add these required, defaulted suite variables and matching argument specifications: dnd_pipeline_notarius_pipelines as a complete native map, dnd_pipeline_notarius_cache_root as a path, dnd_pipeline_notarius_cache_chunk_plans_mode as a string, dnd_pipeline_notarius_cache_checkpoints_enabled as a boolean, dnd_pipeline_narratio_notarius_pipeline_id as a string, dnd_pipeline_narratio_notarius_references as a complete native map, and dnd_pipeline_narratio_notarius_outputs as a complete native map.
3. Default the pipeline map to one dnd-session entry adapted from the Notarius v0.7.0 maintained fragment. Preserve its embedded dnd-extraction profile ID, Seriatim input, dnd/scenes chunker with two retries, three ordered steps, ten lanes, canonical object bindings, generated artifact handoffs, JSON chunk map, and nine-lane evidence-context selection with a three-unit window.
4. Remove every external path reference from the adapted pipeline: omit the example's party, glossary, and binding-local spell_catalog paths. Retain only generated artifact references between steps. Do not install a PromptKit profile file or set global output/debug roots.
5. Default the selected pipeline ID to dnd-session. Default the Narratio reference map to party: narratio.input.party, players: narratio.input.players, and glossary: narratio.input.glossary; do not add spell_catalog until a deployed campaign supplies that optional input.
6. Default the Narratio output map to all ten exact lane, media type, schema ID, schema version, and module-key descriptors recorded in the feature roadmap and Narratio's maintained example. Keep both maps replaceable as whole native values through normal inventory precedence.
7. Default the cache root to /var/cache/notarius, chunk-plan mode to auto, and checkpoint recording to true. Derive /var/cache/notarius/chunk-plans and /var/cache/notarius/checkpoints in dependency wiring rather than adding independent suite variables.
8. Pass the fixed relative import pipelines/dnd-session.yml, selected cache settings, and derived child paths to rakestraw.notarius. Leave its root-level notarius_pipelines unset so the standalone role continues to own only the root document.
9. Pass narratio_notarius_enabled: true, the selected pipeline ID, /usr/local/etc/notarius working directory, and the reference/output maps to rakestraw.narratio. Retain the established binary and configuration paths, leave its native three-hour timeout in control, and do not change the Narratio role or testing-overlay template.
10. Create /var/cache/notarius directly as the selected runtime user and group with mode 2775 before ACL convergence. Add that root to both existing access and default ACL loops; do not create its child stores or add a second ACL implementation.
11. Create /usr/local/etc/notarius/pipelines directly with the suite configuration owner, group, and directory mode. Add one template that serializes only pipelines: dnd_pipeline_notarius_pipelines to dnd-session.yml with the suite file metadata. Do not duplicate the standalone root template or manually render every nested map field.
12. Keep production convergence limited to the existing role dependencies, one cache-root file operation, the existing ACL loops, one configuration directory operation, and one template operation. Do not add commands, stat, handlers, validation tasks, state bookkeeping, secrets, extraction, or retention behavior.

Validation

Run from ansible-role-dnd-pipeline:

tests/integration
git diff --check
git status --short

Expected result: isolated dependency installation resolves the exact Stage 1 commit; syntax and production-profile lint pass; the five applications install successfully; Notarius natively validates the imported dnd-session pipeline; Narratio validates the assembled base and testing configurations; the baseline repeat reports changed=0 and failed=0; and only intended pipeline-role files are modified.

Completion Criteria

• The suite installs Notarius v0.7.0 through the exact reviewed role revision.
• The complete path-free D&D pipeline is owned by one suite fragment and is imported through the standalone role's public interface.
• Narratio enables and selects the pipeline with the three prepared campaign references and all ten exact accepted outputs.
• The FHS cache root and inherited access policy use direct module-owned convergence; child stores remain application-owned.
• The unchanged testing overlay inherits Notarius enablement and pipeline selection.
• The repository is valid and the stage is committed.

Commit

Commit intent: Integrate Notarius extraction into the D&D pipeline

Stage 3: Document and Exercise the Integration Contract

Objective

Document the suite's Notarius policy and add the smallest behavioral coverage that proves native composition, Narratio coordination, shared access, customization, and immediate idempotence.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/ansible.md
• docs/testing.md
• docs/roadmap/features/0010-dnd-notarius-extraction-integration.md
• docs/decisions/0002-shared-readable-pipeline-filesystem-policy.md
• the committed Stage 1 and Stage 2 changes
• ../notarius/docs/cli.md
• ../notarius/docs/consumers/dnd-pipeline.md
• ../narratio/docs/cli.md
• ../narratio/docs/integrations/notarius.md
• ansible-role-dnd-pipeline/README.md
• ansible-role-dnd-pipeline/tests/integration
• ansible-role-dnd-pipeline/tests/converge.yml

Preconditions

• Stages 1 and 2 are committed, validated, and clean.
• The pipeline's isolated dependency installation resolves the Stage 1 commit.
• The deployed baseline can be restored through the existing integration driver's cleanup path.

Owning Repository

ansible-role-dnd-pipeline

Required Changes

1. Update the README's compatible-version table to Notarius v0.7.0 and replace the statement that Notarius remains disabled with the implemented extraction behavior.
2. Document the seven new suite variables, the fixed root import and fragment path, caller-side map replacement/composition with Jinja combine, the prepared-reference boundary, all ten accepted output sources, and the optional future spell_catalog extension. Distinguish native maps from field-specific scalar controls.
3. Document /var/cache/notarius, auto chunk plans, recorded-but-not-resumed checkpoints, derived child paths, 2775 setgid/default-ACL policy, and use by either the service account or an existing member of the runtime group. Retain the world-readable non-secret policy and the deferred restricted credential boundary.
4. Document that base and testing configurations use the same dnd-extraction profile and pipeline. Do not claim that testing avoids Notarius model calls during a real run; integration tests perform only offline configuration inspection.
5. Extend the baseline integration path to run notarius config validate --pipeline dnd-session and inspect notarius config effective. Observe one generated artifact handoff and one cache setting from the native effective output without reading the managed YAML file or enumerating every pipeline option.
6. Use Narratio's native config validate for both centaurs and dilfs in the base configuration and at least one campaign under testing. Use config show or another native inspection surface to observe Notarius enablement, the selected pipeline, one prepared reference, and one output descriptor. Keep the existing check that the testing overlay changes only its intended Scriptorium profile selections.
7. As the existing uniquely named operator fixture in the runtime group, run one native Notarius validation or inspection command against the same root and imported fragment used by the service account. Add one actual service/operator read-write exchange beneath /var/cache/notarius, or move the existing representative exchange there, to exercise the new shared root without parsing ACLs or mode bits. Do not add an exhaustive root or identity matrix.
8. Add one alternate convergence case that supplies two small version 5 pipeline maps and composes them into dnd_pipeline_notarius_pipelines with Jinja combine, selects a matching alternate pipeline ID, and overrides one cache choice. Observe the two pipeline IDs and effective cache choice through native Notarius inspection, then require one immediate identical repeat with changed=0 and failed=0.
9. Retain the baseline clean repeat, immutable local asset inputs, isolated dependency installation, cleanup safety, and existing representative application checks. Restore the default deployment during cleanup.
10. Do not run extraction, contact an LLM or WhisperX endpoint, snapshot a complete rendered document, assert every nested pipeline field, test filesystem metadata, or reproduce Notarius, Narratio, ConfigKit, PromptKit, or Ansible behavior.

Validation

Run from ansible-role-dnd-pipeline:

tests/integration
git diff --check
git status --short

Expected result: isolated role and collection installation, syntax checking, and production-profile lint pass; native Notarius validation and effective inspection prove composition and representative policy; Narratio validates both campaigns and the testing profile; a supported group member can inspect the same Notarius configuration and exchange representative cache content; the combined-map/cache override is observable; all immediate repeats report changed=0 and failed=0; and documentation matches the public interface.

Completion Criteria

• Documentation describes the implemented suite policy, public customization boundary, shared cache, testing behavior, and deliberate exclusions.
• Tests protect role-owned integration behavior without freezing complete defaults or testing module internals.
• Both deployed campaigns validate, and the base and testing configurations retain the enabled Notarius stage.
• Baseline and representative override applications converge cleanly on their immediate repeats.
• The repository is valid and the stage is committed.

Commit

Commit intent: Document and test the D&D Notarius integration

Stage 4: Record the Implementation for Review

Objective

Record the completed implementation and validation evidence in the workspace without declaring the feature complete before independent review.

Required Policy Review

Before beginning this stage, read:

• AGENTS.md
• docs/orientation.md
• docs/conventions.md
• docs/planning.md
• docs/roadmap/features/0010-dnd-notarius-extraction-integration.md
• docs/roadmap/implementation.md
• docs/status.md

Preconditions

• Stages 1 through 3 are committed and their owning repositories are clean.
• Both complete integration entry points pass at the committed revisions.
• The workspace control repository is clean at the planning commit.

Owning Repository

ansible-workspace-dnd

Required Changes

1. Update this plan's Progress Notes with the three implementation commit hashes and concise validation results.
2. Update docs/status.md so Feature 0010 is in_progress, its evidence names the implementation commits, and the current focus is independent review of the completed implementation.
3. Keep the feature roadmap ready; do not mark Feature 0010 done, clean the durable roadmap, or create docs/roadmap/remediation.md. Those actions belong to the independent review and closeout workflow.
4. Do not change implementation repositories or unrelated workspace policy in this stage.

Validation

Run from the workspace root:

git diff --check
./scripts/status
git status --short

Expected result: the status report identifies only the intended workspace tracking edits before commit; both implementation repositories are clean; and the dashboard and progress notes point to the exact completed stage commits.

Completion Criteria

• The implementation and validation evidence is traceable across repositories.
• Feature 0010 remains awaiting independent review rather than being marked complete.
• The workspace control repository is valid and the stage is committed.

Commit

Commit intent: Record the D&D Notarius integration implementation

Final Validation and Handoff

After Stage 4, rerun both owning repositories' complete integration entry points:

cd ansible-role-notarius && tests/integration
cd ../ansible-role-dnd-pipeline && tests/integration

Then, from the workspace root, run:

git diff --check
./scripts/status

The final handoff must:

• identify the Stage 1 through Stage 4 commit hashes and confirm each changed repository has a clean working tree;
• confirm syntax and production-profile lint pass in both changed roles;
• confirm the standalone role installs an explicitly selected Notarius v0.7.0, validates and inspects composed version 5 configuration natively, and immediately converges cleanly;
• confirm the pipeline installs the exact immutable Notarius-role commit, validates the complete D&D pipeline and both Narratio campaign resolutions, preserves Notarius under testing, supports the representative shared-user path, and immediately converges cleanly;
• confirm no upstream application, asset, collection, unrelated standalone role, secret, external service, or real human account was modified;
• summarize any deviation, blocker, or deferred work, including any pause required to publish the Stage 1 dependency commit;
• report any production complexity beyond the planned public variables, dependency wiring, one imported fragment, one cache-root task, and extension of the existing ACL loops, together with the acceptance criterion that required it; and
• leave Feature 0010 in_progress for independent review rather than marking it done.

Progress Notes

No implementation stages have begun. Record only concise stage commit hashes, validation results, and blockers here during execution; preserve detailed evidence in the implementation commits and final handoff.