ansible-workspace-dnd/docs/roadmap/implementation.md

# Implementation Plan 0010: D&D Notarius Extraction Integration

Status: ready

Target executor: GPT-5.6-Terra

Feature roadmap:
`docs/roadmap/features/0010-dnd-notarius-extraction-integration.md`

Active plan location: `docs/roadmap/implementation.md`

## Objective

Upgrade the generic Notarius role to the v0.7.0 configuration contract, then
have the D&D pipeline role deploy the maintained ten-lane extraction pipeline,
enable it through Narratio, bind prepared campaign references, accept all ten
lane contracts, and provision the shared FHS cache policy established by
Feature 0009.

The feature roadmap remains authoritative for scope, rationale, target end
state, and acceptance criteria. Decisions 0001 and 0002 remain authoritative
for direct Ansible convergence and the shared-readable filesystem boundary.

## Execution Rules

- Execute stages in numeric order.
- Before each stage, confirm its preconditions and the working-tree state of
  its owning repository.
- Before each stage, reread every document listed in that stage's Required
  Policy Review. Do not rely on policy recalled from an earlier stage.
- Preserve unrelated and pre-existing changes. Stop if an owning repository is
  unexpectedly dirty at a stage boundary.
- Keep `ansible-role-notarius` generic. ConfigKit import support is a native
  application interface; the D&D pipeline, campaign bindings, cache policy,
  and Narratio coordination belong only to `ansible-role-dnd-pipeline`.
- Use the Notarius v0.7.0 maintained complete pipeline as the topology and
  contract authority. Remove external path references, but retain generated
  artifact handoffs and canonical version 5 binding objects.
- Trust the existing application-role templates and Ansible modules. Do not
  add production validation commands, preliminary probes, manual comparisons,
  handlers, checksums, manifests, state files, or corrective permission tasks.
- Create only the shared Notarius cache root. Notarius owns creation of its
  configured `chunk-plans` and `checkpoints` children during actual use.
- Keep the existing Narratio `testing` overlay unchanged so it inherits the
  enabled Notarius stage and `dnd-extraction` workload profile.
- Do not add credentials, a PromptKit profile file, campaign paths, a global
  Notarius output or debug root, checkpoint resume, extraction execution,
  model calls, storage, publishing, services, schedules, or user-membership
  management.
- Test public behavior through the installed applications' native inspection
  and validation interfaces. Do not snapshot complete configuration files,
  parse production task results, inspect filesystem metadata, or reproduce
  application and Ansible validators in assertions.
- Run every stage's validation before committing it. Commit each completed
  stage when it leaves the owning repository valid, using the plain-English
  commit intent specified by that stage.
- Do not begin a dependent stage after an earlier stage fails validation.
- The pipeline integration suite installs role dependencies from their
  canonical Git origins. The Stage 1 Notarius-role commit must therefore be
  reachable from its canonical origin before Stage 2 begins. If it is not,
  stop after Stage 1 and report the commit hash for publication; do not use a
  branch, local path, or weakened dependency test as a substitute.

## Starting State and Preconditions

- The workspace control repository contains the reviewed Feature 0010 roadmap
  and this implementation plan on `main` in a clean planning commit descended
  from `f91871ba0f90da19592eb56824f5abb9e7868285`.
- Features 0001 through 0009 and the standalone-role test-suite simplification
  are complete.
- The implementation repositories are clean on `main` at these reviewed
  starting commits:

  | Repository | Starting commit |
  | --- | --- |
  | `ansible-role-notarius` | `e9ae2dc34023e1e8462bf0991de62b97d8c0e0ee` |
  | `ansible-role-dnd-pipeline` | `b15fe8dcf57a53784bc3b0b8bf04aa97d39edb3f` |

- The Notarius-role starting commit matches `origin/main`. The pipeline role
  is intentionally ahead of its canonical origin and must not be reset or
  replaced.
- `../notarius` is clean at the immutable `v0.7.0` tag,
  `4e77725b24cb62bd1f96ee31b4d17b4a2ed68ba7`.
- `../narratio` is clean at the immutable `v2.0.0` tag,
  `ed4623bc415901134c765fe9cf06d59aa7891500`.
- Notarius v0.7.0 preserves the subprocess receipt and extraction-artifact
  contracts consumed by Narratio v2.0.0 while requiring Notarius
  configuration version 5.
- The workspace `.venv` provides Ansible Core 2.21.3 and ansible-lint 26.8.0.
- The local integration environment remains an accepted proxy for Ubuntu
  26.04, supports task-level privilege escalation, and provides ordinary POSIX
  ACL behavior.

## Repository Scope

Repositories that change:

- `ansible-role-notarius`: v0.7.0 selection, configuration version 5,
  nullable native ConfigKit imports, generic documentation, and focused
  integration coverage.
- `ansible-role-dnd-pipeline`: compatible Notarius role and application pins,
  opinionated D&D pipeline configuration, Narratio integration, shared cache
  root, documentation, and focused integration coverage.
- `ansible-workspace-dnd`: implementation progress and review handoff only
  after the role stages are complete.

Repositories that may be inspected but must remain unchanged:

- `../notarius` and `../narratio`, which are read-only application authorities.
- `ansible-role-narratio`, whose generic Notarius adapter already exposes all
  fields required by this feature.
- `ansible-role-audita`, `ansible-role-seriatim`,
  `ansible-role-scriptorium`, and `ansible-collection-dnd`.
- `assets/campaigns`, `assets/narratio-prompts`, and `assets/configs`.
- `rakestraw-ansible` and every other repository in `workspace.yml`.

## Stage 1: Upgrade the Generic Notarius Role

### Objective

Make the standalone role install Notarius v0.7.0, render configuration version
5, and expose the application's native ConfigKit import list without adding
D&D-specific policy.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/roadmap/features/0010-dnd-notarius-extraction-integration.md`
- `docs/decisions/0001-minimal-module-driven-ansible.md`
- `../notarius/docs/releases/v0.7.0.md`
- `../notarius/docs/config.md`
- `../notarius/docs/cli.md`
- `../notarius/examples/dnd-complete.config.yml`
- `../notarius/examples/pipelines/dnd-session.yml`
- `ansible-role-notarius/defaults/main.yml`
- `ansible-role-notarius/meta/argument_specs.yml`
- `ansible-role-notarius/templates/config.yml.j2`
- `ansible-role-notarius/tests/integration`
- `ansible-role-notarius/tests/verify-config.yml`
- `ansible-role-notarius/README.md`

### Preconditions

- `ansible-role-notarius` is clean at its stated starting commit.
- The repository's existing `tests/integration` entry point passes before
  modification.
- The v0.7.0 upstream files listed above remain at the reviewed immutable tag.

### Owning Repository

`ansible-role-notarius`

### Required Changes

1. Advance `notarius_version` from `v0.6.0` to `v0.7.0` and
   `notarius_config_version` from `4` to `5`. Do not change the established
   source-build task flow or other generic defaults.
2. Add `notarius_configkit_imports: null` as one optional public variable and
   add a matching argument-specification entry using a list of native mapping
   objects. Do not create field aliases for import entries or add role-owned
   merge behavior.
3. Render a top-level `configkit.imports` section only when
   `notarius_configkit_imports is not none`. Serialize the supplied list with
   an established Ansible/Jinja data filter; do not manually construct YAML or
   alter `notarius_pipelines`.
4. Keep every other optional field governed by its current null-omission
   behavior. The role manages only its root configuration and must not create
   any caller-selected imported fragment.
5. Update the README's version, configuration version, variable list, and
   configuration discussion. Explain that ConfigKit imports are caller-owned
   native composition inputs and that relative imports resolve from the
   managed root configuration directory. Do not mention the D&D suite's fixed
   fragment or cache policy as a standalone default.
6. Migrate the integration fixture from configuration version 4 shorthand to
   canonical version 5 module-binding objects. Use a test-owned imported
   fragment to exercise `notarius_configkit_imports`; fixture creation is test
   setup and must not become a production role task.
7. Exercise an explicitly selected `notarius_version: v0.7.0`, caller-side
   composition of two small pipeline maps, an intentional falsey cache value,
   a representative cache override, and the existing caller-supplied Go
   executable/missing-binary recovery behavior. Use v0.7.0 for cases that need
   a configuration-compatible executable.
8. Validate the representative imported pipeline with
   `notarius config validate --pipeline <id>` and observe composition and the
   cache override through `notarius config effective` or another native
   inspection command. Remove direct managed-file assertions from
   `tests/verify-config.yml`; delete that playbook if it has no remaining
   behavioral purpose.
9. Retain an immediate identical repeat that reports `changed=0` and
   `failed=0`. Do not assert that v0.7.0 remains the role default, test
   ConfigKit itself, or add production validation and state inspection.

### Validation

Run from `ansible-role-notarius`:

```text
tests/integration
git diff --check
git status --short
```

Expected result: syntax and production-profile lint pass; the explicitly
selected v0.7.0 binary reports its selected version; native Notarius commands
validate and inspect the representative composed version 5 configuration;
missing-binary recovery still works with the caller's Go executable; all
immediate repeats report `changed=0` and `failed=0`; and only intended role,
test, and README files are modified.

### Completion Criteria

- Defaults, argument specification, template, README, and tests expose one
  coherent generic ConfigKit import interface.
- The root document is version 5 and no D&D pipeline or caller-owned fragment
  is embedded in the standalone role.
- Tests observe native application behavior and do not snapshot defaults or
  re-test Ansible module guarantees.
- The repository is valid and the stage is committed.

### Commit

Commit intent: `Upgrade the Notarius role to version 0.7.0`

## Stage 2: Implement the Suite Notarius Integration

### Objective

Add the pipeline role's complete D&D Notarius configuration, Narratio wiring,
and shared cache convergence using the reviewed standalone-role interface.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/roadmap/features/0010-dnd-notarius-extraction-integration.md`
- `docs/decisions/0001-minimal-module-driven-ansible.md`
- `docs/decisions/0002-shared-readable-pipeline-filesystem-policy.md`
- the committed Stage 1 changes in `ansible-role-notarius`
- `../notarius/docs/releases/v0.7.0.md`
- `../notarius/docs/config.md`
- `../notarius/docs/consumers/dnd-pipeline.md`
- `../notarius/examples/dnd-complete.config.yml`
- `../notarius/examples/pipelines/dnd-session.yml`
- `../narratio/docs/integrations/notarius.md`
- `../narratio/examples/pipeline.full.annotated.yml`
- `ansible-role-dnd-pipeline/defaults/main.yml`
- `ansible-role-dnd-pipeline/meta/argument_specs.yml`
- `ansible-role-dnd-pipeline/meta/main.yml`
- `ansible-role-dnd-pipeline/requirements.yml`
- `ansible-role-dnd-pipeline/tasks/main.yml`
- `ansible-role-dnd-pipeline/templates/testing.yml.j2`
- `ansible-role-dnd-pipeline/tests/integration`

### Preconditions

- Stage 1 is committed, validated, and clean.
- The Stage 1 commit is reachable from the canonical
  `ansible-role-notarius` Git origin. Stop and report the hash if a fresh Git
  or Galaxy fetch cannot resolve it.
- `ansible-role-dnd-pipeline` is clean at its stated starting commit or an
  expected reviewed descendant.
- Its existing `tests/integration` entry point passes before modification.

### Owning Repository

`ansible-role-dnd-pipeline`

### Required Changes

1. Set `dnd_pipeline_notarius_version: v0.7.0` and pin the
   `rakestraw.notarius` entry in `requirements.yml` to the exact Stage 1
   commit. Retain the canonical Git URL and every unrelated dependency pin.
2. Add these required, defaulted suite variables and matching argument
   specifications:
   `dnd_pipeline_notarius_pipelines` as a complete native map,
   `dnd_pipeline_notarius_cache_root` as a path,
   `dnd_pipeline_notarius_cache_chunk_plans_mode` as a string,
   `dnd_pipeline_notarius_cache_checkpoints_enabled` as a boolean,
   `dnd_pipeline_narratio_notarius_pipeline_id` as a string,
   `dnd_pipeline_narratio_notarius_references` as a complete native map, and
   `dnd_pipeline_narratio_notarius_outputs` as a complete native map.
3. Default the pipeline map to one `dnd-session` entry adapted from the
   Notarius v0.7.0 maintained fragment. Preserve its embedded
   `dnd-extraction` profile ID, Seriatim input, `dnd/scenes` chunker with two
   retries, three ordered steps, ten lanes, canonical object bindings,
   generated artifact handoffs, JSON chunk map, and nine-lane evidence-context
   selection with a three-unit window.
4. Remove every external `path` reference from the adapted pipeline: omit the
   example's `party`, `glossary`, and binding-local `spell_catalog` paths.
   Retain only generated `artifact` references between steps. Do not install a
   PromptKit profile file or set global output/debug roots.
5. Default the selected pipeline ID to `dnd-session`. Default the Narratio
   reference map to `party: narratio.input.party`,
   `players: narratio.input.players`, and
   `glossary: narratio.input.glossary`; do not add `spell_catalog` until a
   deployed campaign supplies that optional input.
6. Default the Narratio output map to all ten exact lane, media type, schema
   ID, schema version, and module-key descriptors recorded in the feature
   roadmap and Narratio's maintained example. Keep both maps replaceable as
   whole native values through normal inventory precedence.
7. Default the cache root to `/var/cache/notarius`, chunk-plan mode to `auto`,
   and checkpoint recording to `true`. Derive
   `/var/cache/notarius/chunk-plans` and
   `/var/cache/notarius/checkpoints` in dependency wiring rather than adding
   independent suite variables.
8. Pass the fixed relative import `pipelines/dnd-session.yml`, selected cache
   settings, and derived child paths to `rakestraw.notarius`. Leave its
   root-level `notarius_pipelines` unset so the standalone role continues to
   own only the root document.
9. Pass `narratio_notarius_enabled: true`, the selected pipeline ID,
   `/usr/local/etc/notarius` working directory, and the reference/output maps
   to `rakestraw.narratio`. Retain the established binary and configuration
   paths, leave its native three-hour timeout in control, and do not change the
   Narratio role or testing-overlay template.
10. Create `/var/cache/notarius` directly as the selected runtime user and
    group with mode `2775` before ACL convergence. Add that root to both
    existing access and default ACL loops; do not create its child stores or
    add a second ACL implementation.
11. Create `/usr/local/etc/notarius/pipelines` directly with the suite
    configuration owner, group, and directory mode. Add one template that
    serializes only `pipelines: dnd_pipeline_notarius_pipelines` to
    `dnd-session.yml` with the suite file metadata. Do not duplicate the
    standalone root template or manually render every nested map field.
12. Keep production convergence limited to the existing role dependencies,
    one cache-root file operation, the existing ACL loops, one configuration
    directory operation, and one template operation. Do not add commands,
    `stat`, handlers, validation tasks, state bookkeeping, secrets, extraction,
    or retention behavior.

### Validation

Run from `ansible-role-dnd-pipeline`:

```text
tests/integration
git diff --check
git status --short
```

Expected result: isolated dependency installation resolves the exact Stage 1
commit; syntax and production-profile lint pass; the five applications install
successfully; Notarius natively validates the imported `dnd-session` pipeline;
Narratio validates the assembled base and testing configurations; the baseline
repeat reports `changed=0` and `failed=0`; and only intended pipeline-role
files are modified.

### Completion Criteria

- The suite installs Notarius v0.7.0 through the exact reviewed role revision.
- The complete path-free D&D pipeline is owned by one suite fragment and is
  imported through the standalone role's public interface.
- Narratio enables and selects the pipeline with the three prepared campaign
  references and all ten exact accepted outputs.
- The FHS cache root and inherited access policy use direct module-owned
  convergence; child stores remain application-owned.
- The unchanged testing overlay inherits Notarius enablement and pipeline
  selection.
- The repository is valid and the stage is committed.

### Commit

Commit intent: `Integrate Notarius extraction into the D&D pipeline`

## Stage 3: Document and Exercise the Integration Contract

### Objective

Document the suite's Notarius policy and add the smallest behavioral coverage
that proves native composition, Narratio coordination, shared access,
customization, and immediate idempotence.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/ansible.md`
- `docs/testing.md`
- `docs/roadmap/features/0010-dnd-notarius-extraction-integration.md`
- `docs/decisions/0002-shared-readable-pipeline-filesystem-policy.md`
- the committed Stage 1 and Stage 2 changes
- `../notarius/docs/cli.md`
- `../notarius/docs/consumers/dnd-pipeline.md`
- `../narratio/docs/cli.md`
- `../narratio/docs/integrations/notarius.md`
- `ansible-role-dnd-pipeline/README.md`
- `ansible-role-dnd-pipeline/tests/integration`
- `ansible-role-dnd-pipeline/tests/converge.yml`

### Preconditions

- Stages 1 and 2 are committed, validated, and clean.
- The pipeline's isolated dependency installation resolves the Stage 1 commit.
- The deployed baseline can be restored through the existing integration
  driver's cleanup path.

### Owning Repository

`ansible-role-dnd-pipeline`

### Required Changes

1. Update the README's compatible-version table to Notarius v0.7.0 and replace
   the statement that Notarius remains disabled with the implemented
   extraction behavior.
2. Document the seven new suite variables, the fixed root import and fragment
   path, caller-side map replacement/composition with Jinja `combine`, the
   prepared-reference boundary, all ten accepted output sources, and the
   optional future `spell_catalog` extension. Distinguish native maps from
   field-specific scalar controls.
3. Document `/var/cache/notarius`, `auto` chunk plans, recorded-but-not-resumed
   checkpoints, derived child paths, `2775` setgid/default-ACL policy, and use
   by either the service account or an existing member of the runtime group.
   Retain the world-readable non-secret policy and the deferred restricted
   credential boundary.
4. Document that base and `testing` configurations use the same
   `dnd-extraction` profile and pipeline. Do not claim that testing avoids
   Notarius model calls during a real run; integration tests perform only
   offline configuration inspection.
5. Extend the baseline integration path to run
   `notarius config validate --pipeline dnd-session` and inspect
   `notarius config effective`. Observe one generated artifact handoff and one
   cache setting from the native effective output without reading the managed
   YAML file or enumerating every pipeline option.
6. Use Narratio's native `config validate` for both `centaurs` and `dilfs` in
   the base configuration and at least one campaign under `testing`. Use
   `config show` or another native inspection surface to observe Notarius
   enablement, the selected pipeline, one prepared reference, and one output
   descriptor. Keep the existing check that the testing overlay changes only
   its intended Scriptorium profile selections.
7. As the existing uniquely named operator fixture in the runtime group, run
   one native Notarius validation or inspection command against the same root
   and imported fragment used by the service account. Add one actual
   service/operator read-write exchange beneath `/var/cache/notarius`, or move
   the existing representative exchange there, to exercise the new shared
   root without parsing ACLs or mode bits. Do not add an exhaustive root or
   identity matrix.
8. Add one alternate convergence case that supplies two small version 5
   pipeline maps and composes them into
   `dnd_pipeline_notarius_pipelines` with Jinja `combine`, selects a matching
   alternate pipeline ID, and overrides one cache choice. Observe the two
   pipeline IDs and effective cache choice through native Notarius inspection,
   then require one immediate identical repeat with `changed=0` and
   `failed=0`.
9. Retain the baseline clean repeat, immutable local asset inputs, isolated
   dependency installation, cleanup safety, and existing representative
   application checks. Restore the default deployment during cleanup.
10. Do not run extraction, contact an LLM or WhisperX endpoint, snapshot a
    complete rendered document, assert every nested pipeline field, test
    filesystem metadata, or reproduce Notarius, Narratio, ConfigKit, PromptKit,
    or Ansible behavior.

### Validation

Run from `ansible-role-dnd-pipeline`:

```text
tests/integration
git diff --check
git status --short
```

Expected result: isolated role and collection installation, syntax checking,
and production-profile lint pass; native Notarius validation and effective
inspection prove composition and representative policy; Narratio validates
both campaigns and the testing profile; a supported group member can inspect
the same Notarius configuration and exchange representative cache content;
the combined-map/cache override is observable; all immediate repeats report
`changed=0` and `failed=0`; and documentation matches the public interface.

### Completion Criteria

- Documentation describes the implemented suite policy, public customization
  boundary, shared cache, testing behavior, and deliberate exclusions.
- Tests protect role-owned integration behavior without freezing complete
  defaults or testing module internals.
- Both deployed campaigns validate, and the base and testing configurations
  retain the enabled Notarius stage.
- Baseline and representative override applications converge cleanly on their
  immediate repeats.
- The repository is valid and the stage is committed.

### Commit

Commit intent: `Document and test the D&D Notarius integration`

## Stage 4: Record the Implementation for Review

### Objective

Record the completed implementation and validation evidence in the workspace
without declaring the feature complete before independent review.

### Required Policy Review

Before beginning this stage, read:

- `AGENTS.md`
- `docs/orientation.md`
- `docs/conventions.md`
- `docs/planning.md`
- `docs/roadmap/features/0010-dnd-notarius-extraction-integration.md`
- `docs/roadmap/implementation.md`
- `docs/status.md`

### Preconditions

- Stages 1 through 3 are committed and their owning repositories are clean.
- Both complete integration entry points pass at the committed revisions.
- The workspace control repository is clean at the planning commit.

### Owning Repository

`ansible-workspace-dnd`

### Required Changes

1. Update this plan's Progress Notes with the three implementation commit
   hashes and concise validation results.
2. Update `docs/status.md` so Feature 0010 is `in_progress`, its evidence names
   the implementation commits, and the current focus is independent review of
   the completed implementation.
3. Keep the feature roadmap `ready`; do not mark Feature 0010 `done`, clean the
   durable roadmap, or create `docs/roadmap/remediation.md`. Those actions
   belong to the independent review and closeout workflow.
4. Do not change implementation repositories or unrelated workspace policy in
   this stage.

### Validation

Run from the workspace root:

```text
git diff --check
./scripts/status
git status --short
```

Expected result: the status report identifies only the intended workspace
tracking edits before commit; both implementation repositories are clean; and
the dashboard and progress notes point to the exact completed stage commits.

### Completion Criteria

- The implementation and validation evidence is traceable across repositories.
- Feature 0010 remains awaiting independent review rather than being marked
  complete.
- The workspace control repository is valid and the stage is committed.

### Commit

Commit intent: `Record the D&D Notarius integration implementation`

## Final Validation and Handoff

After Stage 4, rerun both owning repositories' complete integration entry
points:

```text
cd ansible-role-notarius && tests/integration
cd ../ansible-role-dnd-pipeline && tests/integration
```

Then, from the workspace root, run:

```text
git diff --check
./scripts/status
```

The final handoff must:

- identify the Stage 1 through Stage 4 commit hashes and confirm each changed
  repository has a clean working tree;
- confirm syntax and production-profile lint pass in both changed roles;
- confirm the standalone role installs an explicitly selected Notarius v0.7.0,
  validates and inspects composed version 5 configuration natively, and
  immediately converges cleanly;
- confirm the pipeline installs the exact immutable Notarius-role commit,
  validates the complete D&D pipeline and both Narratio campaign resolutions,
  preserves Notarius under `testing`, supports the representative shared-user
  path, and immediately converges cleanly;
- confirm no upstream application, asset, collection, unrelated standalone
  role, secret, external service, or real human account was modified;
- summarize any deviation, blocker, or deferred work, including any pause
  required to publish the Stage 1 dependency commit;
- report any production complexity beyond the planned public variables,
  dependency wiring, one imported fragment, one cache-root task, and extension
  of the existing ACL loops, together with the acceptance criterion that
  required it; and
- leave Feature 0010 `in_progress` for independent review rather than marking
  it `done`.

## Progress Notes

No implementation stages have begun. Record only concise stage commit hashes,
validation results, and blockers here during execution; preserve detailed
evidence in the implementation commits and final handoff.