# Feature 0010: D&D Notarius Extraction Integration Status: done ## Summary Enable Narratio's Notarius stage with the maintained ten-lane D&D extraction pipeline. The pipeline role will supply the opinionated Notarius pipeline, bind campaign-owned inputs through Narratio's invocation boundary, and declare the exact lane contracts Narratio may accept as extraction sources. It uses the service ownership and shared-access foundation established by Feature 0009. ## Problem or Opportunity Features 0006 through 0008 install and configure the five applications, deploy the D&D content, and define the transcript-processing and Scriptorium workflow. Narratio still leaves its Notarius stage disabled, and Notarius's managed configuration contains no pipeline. A run can therefore produce a cleaned and trimmed transcript but cannot yet extract the structured D&D registries and events implemented by Notarius. The uploaded test-machine configuration demonstrates the intended multi-step pipeline, but it embeds campaign-specific filesystem paths and explicit runtime roots. The current Notarius v0.7.0 complete example and Narratio 2.0 integration boundary provide the maintained replacement. Narratio 2.0 names Notarius v0.6.0 as its subprocess baseline, while Notarius v0.7.0 expressly preserves that subprocess receipt and extraction-artifact contract. The Notarius upgrade therefore changes deployment configuration without requiring a Narratio adapter change. Narratio must prepare invocation-local campaign references and own the run-specific output staging directory; generated registry and event handoffs remain internal to Notarius. Notarius v0.7.0 also replaces configuration version 4 with a strict, ConfigKit-backed version 5 contract and makes cache access policy a deployment responsibility. ## Governing Policies - [The Ansible house style](../../ansible.md) governs the suite-owned public interface, direct desired-state configuration, and reliance on the standalone roles' native templates. - [The testing house style](../../testing.md) governs focused native contract validation and immediate clean repeated convergence. - [The MVP roadmap](../mvp.md) assigns opinionated D&D policy and cross-application coordination to the pipeline role while keeping the standalone application roles generic. - [Decision 0001](../../decisions/0001-minimal-module-driven-ansible.md) requires module-owned convergence without redundant inspection or supplemental state. - [Decision 0002](../../decisions/0002-shared-readable-pipeline-filesystem-policy.md) defines the service ownership, shared operator access, world-readable content, and restricted credential boundary. - [Feature 0008](0008-opinionated-dnd-processing-configuration.md) establishes the trimmed transcript, ConfigKit layout, and `testing` overlay into which this feature integrates. - [Feature 0009](0009-shared-pipeline-filesystem-and-execution.md) establishes the runtime identity, configuration ownership interfaces, and inherited shared-access policy extended by this feature. ## Scope - Update the standalone Notarius role for Notarius v0.7.0 and configuration version 5 while preserving its generic, independently usable interface. - Advance the pipeline role's selected Notarius application version to v0.7.0 and pin its Notarius role dependency to the reviewed revision that provides the version 5 interface. - Expose Notarius's native `configkit.imports` field through one optional `notarius_configkit_imports` role variable; the role continues to own only its root configuration and does not manage caller-selected fragments. - Supply one complete `dnd-session` Notarius pipeline as deliberate D&D suite policy in a suite-managed imported fragment at `/usr/local/etc/notarius/pipelines/dnd-session.yml`. - Adapt the maintained Notarius v0.7.0 complete D&D pipeline using canonical version 5 module-binding objects, including: - the embedded `dnd-extraction` PromptKit workload profile; - Seriatim input and scene-based chunking; - the ordered `describe-session`, `extract-events`, and `track-enemies` steps; - all ten maintained D&D artifact lanes and their generated handoffs; and - JSON output with the chunk map and selected evidence context. - Remove every campaign-specific filesystem reference from the Notarius pipeline definition. - Enable Narratio's Notarius stage and select `dnd-session`. - Bind `party`, `players`, and `glossary` to Narratio's prepared campaign sources at invocation time. - Define Narratio acceptance descriptors for all ten maintained lane outputs. - Make the pipeline map, Narratio reference map, and Narratio output map ordinary suite variables so callers can compose or replace native mappings through inventory precedence. - Make the selected pipeline ID an ordinary suite variable and keep it coordinated with the default pipeline map. - Enable Notarius caching to the maximum currently available extent: use `auto` chunk-plan caching and enable checkpoint recording. Narratio does not yet request checkpoint reuse with Notarius's `--resume` flag. - Provision `/var/cache/notarius` as a shared FHS cache root owned by `narratio:narratio` with mode `2775`, and configure Notarius to create its chunk-plan and checkpoint stores beneath that root. - Apply Feature 0009's established access and default ACL policy to the new cache root so content created by either supported invocation identity remains group-writable and world-readable. - Keep Notarius enabled under the existing `testing` overlay. Testing and production select the same Notarius pipeline and workload profile until the suite chooses a distinct testing profile strategy. - Document the reference boundary, lane-to-source mapping, runtime-state policy, production/testing behavior, and supported caller customization. - Extend focused integration coverage for native Notarius validation, Narratio base/testing resolution, representative reference and output wiring, and immediate clean repeated convergence. ## Non-goals - Changing any application's native configuration fields or duplicating a standalone role's root configuration template in the pipeline role. - Making the standalone Notarius role manage imported fragments; fragment deployment remains the responsibility of the caller that selects an import. - Copying the uploaded test-machine configuration verbatim. - Installing a filesystem PromptKit profile for Notarius. Notarius v0.7.0 already embeds the selected production `dnd-extraction` fallback profile. - Defining a second Notarius pipeline solely to select an inexpensive testing model, or otherwise adding profile inheritance that Notarius does not natively provide. - Binding a spell catalog before the campaign registry declares one. - Publishing, rendering, or otherwise consuming the new extraction sources in Scriptorium artifacts. - Selecting a default campaign. - Delivering API keys, provider credentials, environment files, or a general secret-management mechanism. - Running Notarius extraction, making model calls, or processing a complete transcript during integration tests. - Managing output retention, cache cleanup, or debug retention. - Adding checkpoint-resume control to Narratio or changing Narratio's subprocess invocation to pass `--resume` or `--config-profile`. - Installing a systemd service, timer, scheduler, or other unattended invocation mechanism. - Selecting human operator accounts or changing their membership in the `narratio` group. - Revising the general configuration ownership, dependency ordering, ACL dependency, or existing shared roots established by Feature 0009. - Delivering or populating the future restricted `secrets/` directory. - Modifying upstream application source, embedded Notarius prompts, schemas, modules, codecs, or validators. - Collection assembly, role releases, storage/publishing, upgrade/removal behavior, or end-to-end production validation. ## Target End State The standalone Notarius role continues to own the version 5 root document at `/usr/local/etc/notarius/config.yml`. Its public interface exposes the native `configkit.imports` list, and the pipeline role selects the explicit relative import `pipelines/dnd-session.yml`. The pipeline role owns that fragment and renders the complete `dnd_pipeline_notarius_pipelines` map into it, producing one native `dnd-session` definition by default. The definition follows the Notarius v0.7.0 maintained complete D&D topology: ```text Seriatim transcript -> dnd/scenes chunking -> describe-session item-registry npc-registry location-registry scene-descriptions -> extract-events item-occurrences spells combat-turns npc-occurrences location-occurrences -> track-enemies enemy-events -> JSON bundle, chunk map, and evidence context ``` Feature 0009 already makes application configuration service-owned and world-readable. The Notarius root document and new suite fragment retain that `narratio:narratio`, `0755` directory, and `0644` file policy through the standalone role's established ownership and mode interface. The `extract-events` step consumes the four generated outputs from `describe-session` under their declared internal reference slots. The `track-enemies` step consumes the generated NPC registry and scene descriptions from `describe-session` and the generated combat turns and NPC occurrences from `extract-events`. Those handoffs are part of the Notarius pipeline and never appear in Narratio's CLI reference map. Narratio's effective base configuration enables Notarius, selects the managed configuration and `dnd-session`, and supplies three campaign-owned references: | Notarius selector | Prepared Narratio source | | --- | --- | | `party` | `narratio.input.party` | | `players` | `narratio.input.players` | | `glossary` | `narratio.input.glossary` | Narratio snapshots those inputs for each invocation and passes their absolute paths with `--reference`. No campaign path appears in Notarius configuration. The optional `spell_catalog` binding is absent because neither deployed campaign currently declares `inputs.spell_catalog_file`. A caller that adds that campaign input can extend the suite reference map with `spell_catalog: narratio.input.spell_catalog` without changing either application role. Narratio accepts all ten configured lanes using these exact v1 contracts and registers each accepted payload as `narratio.extraction.`: | Narratio output key | Lane ID | Schema ID | Module key | | --- | --- | --- | --- | | `item_registry` | `item-registry` | `notarius.dnd.item_registry` | `dnd/item-registry` | | `npc_registry` | `npc-registry` | `notarius.dnd.npc_registry` | `dnd/npc-registry` | | `location_registry` | `location-registry` | `notarius.dnd.location_registry` | `dnd/location-registry` | | `scene_descriptions` | `scene-descriptions` | `notarius.dnd.scene_descriptions` | `dnd/scene-descriptions` | | `item_occurrences` | `item-occurrences` | `notarius.dnd.item_occurrences` | `dnd/item-occurrences` | | `spells` | `spells` | `notarius.dnd.spells` | `dnd/spells` | | `combat_turns` | `combat-turns` | `notarius.dnd.combat_turns` | `dnd/combat-turns` | | `npc_occurrences` | `npc-occurrences` | `notarius.dnd.npc_occurrences` | `dnd/npc-occurrences` | | `location_occurrences` | `location-occurrences` | `notarius.dnd.location_occurrences` | `dnd/location-occurrences` | | `enemy_events` | `enemy-events` | `notarius.dnd.enemy_events` | `dnd/enemy-events` | Every entry requires `media_type: application/json` and `schema_version: v1`. Narratio validates the received descriptor before registering the source; it does not infer contracts from filenames. Narratio supplies its run-specific staging root through Notarius's `--output-dir` override, so the suite does not set Notarius's global `output.directory`. Reconstructible Notarius state uses the FHS cache layout: ```text /var/cache/notarius/ ├── chunk-plans/ └── checkpoints/ ``` The pipeline role creates the root as `narratio:narratio` mode `2775`, sets chunk-plan mode to `auto`, enables checkpoint recording, and configures both child paths. Notarius creates cache entries with conventional maximum creation modes and leaves their effective modes to deployment provisioning, process umask, setgid inheritance, and default ACLs. The suite applies a default ACL that retains owner and group write access and world read/traverse access for new cache content. Narratio and its Notarius subprocess may run as the dedicated `narratio` account or as a human member of the `narratio` group. Chunk plans are reused automatically; checkpoints are recorded but are not reused because Narratio does not pass `--resume`. The debug root remains unset. The selected Notarius process working directory is `/usr/local/etc/notarius`, matching the managed configuration location; no relative profile or reference paths depend on it. The cache root receives the same setgid and default-ACL policy that Feature 0009 applies to Narratio's existing shared roots. Non-secret content created beneath it remains writable by the shared group and readable by other local users. The suite does not create or populate a `secrets/` directory. The base Narratio configuration and the `testing` ConfigKit overlay both run the extraction stage using Notarius's embedded `dnd-extraction` profile. The testing overlay continues to override only the Scriptorium profile selections; it does not disable or restate Notarius. A later feature may select a distinct inexpensive Notarius testing profile without changing the pipeline topology. ## Repository Ownership - `ansible-role-dnd-pipeline` owns the D&D pipeline map and imported fragment, Narratio integration mappings, Notarius dependency-variable wiring, the new cache root and its inherited shared-access policy, testing-overlay choice, consumer documentation, and suite integration coverage. - `ansible-role-notarius` owns the generic Notarius v0.7.0 installation, version 5 root configuration, optional ConfigKit import interface, focused role documentation, and role-local integration coverage. It does not own or render the suite's imported D&D fragment. - `ansible-role-audita`, `ansible-role-narratio`, and `ansible-role-scriptorium` remain unchanged. The Narratio role owns its generic Notarius adapter fields and ConfigKit-aware root configuration. - `../notarius` is the read-only authority for configuration version 5, the maintained D&D pipeline, embedded profile, module handoffs, and lane contracts. - `../narratio` is the read-only authority for prepared campaign sources, subprocess invocation, output acceptance, and profile overlay behavior. - `assets/configs/notarius/config.yml` is an operational reference only. Its useful topology is adapted, while its campaign paths and runtime roots are not copied. - `assets/campaigns` remains unchanged and continues to own the campaign registry inputs that Narratio prepares. - `ansible-workspace-dnd` owns this roadmap, the later implementation plan, settled MVP decisions, and aggregate status. ## Constraints and Settled Decisions - This feature advances the standalone role default and pipeline application selection from Notarius v0.6.0 to v0.7.0. Narratio remains at v2.0.0. - Narratio v2.0.0 documents Notarius v0.6.0 as its invocation baseline, but Notarius v0.7.0 preserves the subprocess receipt and extraction-artifact contracts. Version 5 configuration migration is confined to the Notarius role and the suite-owned Notarius fragment. - The pipeline dependency manifest must pin the reviewed standalone Notarius role revision that implements this v0.7.0/configuration-version-5 contract. - Feature 0009 has already established the runtime identity, dependency order, standalone ownership interfaces, configuration modes, `acl` package, and pinned `ansible.posix` dependency used here. - The pipeline role relies on the standalone roles for application installation and both root configuration documents. It manages only the suite-owned Notarius and Narratio fragments selected through those public interfaces. - The standalone Notarius role defaults to configuration version 5 and exposes `notarius_configkit_imports` as the complete native import list. A null value omits the `configkit` section and leaves composition with Notarius. - The pipeline role passes the fixed import `pipelines/dnd-session.yml` to the standalone role and manages that imported file as suite-owned configuration. It does not pass the pipeline map into the standalone role's root-level `notarius_pipelines` field. - `/var/cache/notarius` uses the Feature 0009 runtime identity, mode `2775`, and access/default ACL entries equivalent to `u::rwx`, `g::rwx`, `o::r-x`, and `m::rwx`. - `dnd_pipeline_notarius_pipelines` is a complete native map whose default contains `dnd-session`. Callers may replace it or compose additional pipeline definitions with Jinja's `combine` filter in inventory. - `dnd_pipeline_narratio_notarius_pipeline_id` defaults to `dnd-session`. Callers that replace that key are responsible for selecting a matching ID. - `dnd_pipeline_narratio_notarius_references` is the complete native Narratio reference map. It defaults to `party`, `players`, and `glossary`; its values are prepared source IDs, never filesystem paths. - `dnd_pipeline_narratio_notarius_outputs` is the complete native Narratio output map. It defaults to all ten lanes because the suite claims to consume the complete maintained D&D extraction workflow. - `dnd_pipeline_notarius_cache_root` defaults to `/var/cache/notarius`, `dnd_pipeline_notarius_cache_chunk_plans_mode` defaults to `auto`, and `dnd_pipeline_notarius_cache_checkpoints_enabled` defaults to `true`. They are intentional, inventory-overridable suite policy. The two application cache directories derive from that root rather than becoming independent suite variables. - The pipeline defaults to `llm_profile: dnd-extraction` and relies on Notarius's embedded fallback. It does not set `promptkit.profile_file`, `promptkit.profile_dir`, a local backend, or credentials. - The pipeline uses the version 5 `input: {module: seriatim}` binding, `dnd/scenes` chunking with two retries, and the three ordered steps documented above. Extract, merge, and normalize bindings use canonical objects with `module:`; retries and generated reference handoffs match the maintained Notarius complete example. - JSON output enables `include_chunk_map` and evidence context with a three-unit window. Evidence context includes item occurrences, item registry, location registry, location occurrences, NPC registry, spells, combat turns, NPC occurrences, and enemy events, matching the maintained example. - Notarius pipeline YAML contains no external file references. Narratio owns all campaign input materialization for the orchestrated path. - Narratio's Notarius binary and configuration paths remain the established `/usr/local/bin/notarius` and `/usr/local/etc/notarius/config.yml`. - The suite uses the runtime user and group selected by Feature 0009 for the Notarius cache and configuration fragment. - Narratio's Notarius working directory is explicitly `/usr/local/etc/notarius`; the timeout inherits Narratio's native three-hour default because the suite has selected no different policy. - Narratio owns per-run Notarius output staging. The suite provisions `/var/cache/notarius` for the shared `narratio` identity, and Notarius owns its child cache stores. The role does not create `/var/lib/notarius`, set a global output root, or provision a debug root. - Checkpoint enablement records reusable state but does not cause Narratio to reuse it. Resume remains an explicit Notarius invocation action and requires a later Narratio integration decision. - The production configuration enables Notarius. The existing `testing` overlay inherits that enabled stage and the same `dnd-extraction` profile; it does not override Notarius merely to avoid model calls. - All maps use native application shapes and remain replaceable through normal inventory precedence. The suite does not create field-by-field aliases for every nested lane or module option. - Directory and ACL tasks declare only the shared roots required by the applications, using ordinary state-management modules with task-level privilege escalation. No `stat`, command, handler, checksum, manifest, or production validation task accompanies them. ## Simplest Viable Implementation The production changes remain direct configuration management: 1. Update the standalone Notarius role's version and configuration-version defaults and add the nullable native ConfigKit import field to defaults, argument specification, root template, and documentation. Cover that public behavior with focused native validation and a clean repeat. 2. Advance the pipeline's Notarius application selection and standalone-role dependency pin, then add the selected Notarius pipeline map, shared FHS cache policy, and three suite-level Narratio integration mappings to pipeline-role defaults and the argument specification. 3. Pass the fixed import and cache choices to `rakestraw.notarius`, and pass Notarius enablement, selected ID, working directory, references, and outputs to `rakestraw.narratio`. 4. Create the Notarius pipeline-fragment directory and cache root with ordinary `ansible.builtin.file` tasks, render the suite-owned pipeline map into one imported fragment, and extend Feature 0009's access/default ACL task to the new root. 5. Update both role READMEs and extend focused integration coverage with native configuration validation, effective-configuration inspection, inspection as a supported group member, and clean repeats. The standalone roles' template operations continue to own both root configuration files. The pipeline role adds one Notarius pipeline-fragment template but does not duplicate the root document or change the existing Narratio testing-overlay template for this feature. ## Rationale and Alternatives Using Notarius's maintained complete example preserves the upstream module topology and contract while keeping suite policy explicit. Maintaining a smaller custom pipeline would make the role responsible for deciding which D&D artifacts no longer matter. Copying the uploaded file would instead preserve obsolete host paths and bypass Narratio's verified reference boundary. Rendering one native map into a ConfigKit-imported fragment keeps configuration ownership clear and preserves the inventory-composition workflow already selected for Notarius. It also follows Notarius v0.7.0's maintained composition shape without making the generic role own D&D content. Field-specific variables for every lane, step, and retry would create a large parallel schema with little consumer value. One complete map is easier to replace, combine, validate, and compare with Notarius documentation. `/var/cache/notarius` follows the filesystem hierarchy's intended location for reconstructible application state and is preferable to the uploaded `/var/lib/notarius/cache` layout. Assigning the root to the dedicated `narratio` account and group gives unattended and human-initiated runs one reusable cache. Mode `2775` preserves the shared group, and the default ACL prevents an invoking account's umask from accidentally removing group write or world read access from new content. Notarius v0.7.0 respects this deployment policy rather than replacing it with application-owned private modes. Narratio's staging root remains separate because output bundles are run results, not cache data. The `testing` overlay cannot natively replace a nested Notarius pipeline's `llm_profile`; that configuration lives in a separate application file. Duplicating the full pipeline or managing a second Notarius root configuration would add substantial configuration drift for one model choice. Testing therefore inherits production extraction for now; a distinct testing strategy can be selected later if it becomes a concrete need. ## Acceptance Criteria ### Required Configuration Behavior - The pipeline role installs a Notarius configuration that natively validates the `dnd-session` pipeline without provider credentials or a model call. - The pipeline role selects Notarius v0.7.0 and its dependency manifest pins the reviewed standalone-role revision that supplies the version 5 interface. - The standalone Notarius role supports an explicitly selected v0.7.0, emits configuration version 5 with its compatible defaults, and can render a caller-supplied native ConfigKit import list. - The root Notarius configuration imports the suite-owned `pipelines/dnd-session.yml` fragment, and that fragment contains the complete pipeline map selected through inventory. - The pipeline contains the maintained Seriatim input, D&D scene chunker, three ordered steps, ten lanes, generated handoffs, JSON chunk map, and selected evidence-context policy. - The Notarius configuration contains no campaign-specific reference paths, PromptKit operator-profile path, fixed output root, or debug root. - The Notarius configuration selects `auto` chunk-plan caching, enables checkpoints, and uses `/var/cache/notarius/chunk-plans` and `/var/cache/notarius/checkpoints` respectively. - The deployed cache root is `narratio:narratio` mode `2775`; its access and default ACLs preserve group write and world read/traverse access without depending on Notarius to enforce entry modes. Checkpoints are recorded; the roadmap does not claim Narratio reuses them. - The Notarius root configuration and suite-owned imported fragment retain Feature 0009's `narratio:narratio`, `0755` directory, and `0644` file policy. No secret value appears in either file. - Narratio's effective base configuration enables Notarius, selects `dnd-session`, and uses the installed binary, configuration, and working directory. - Narratio maps `party`, `players`, and `glossary` to the three documented prepared campaign sources and contains no default `spell_catalog` binding. - Narratio declares all ten documented output entries with exact lane, media, schema, version, and module identities. - Narratio natively validates the effective base configuration for either deployed campaign without a model call. - `narratio config show` or the equivalent native inspection surface exposes the enabled stage, selected pipeline, representative reference, and representative output contract. - The effective `testing` profile keeps Notarius enabled with the same pipeline and profile while retaining the existing Scriptorium testing overrides. - A caller can compose an additional Notarius pipeline into `dnd_pipeline_notarius_pipelines` and select it through the corresponding Narratio pipeline-ID variable without changing the role. - A caller can add the optional spell-catalog source by extending the native Narratio reference map after providing that campaign input. - An immediate repeated role application with unchanged inputs reports no changes. - The production role contains no new configuration probes, validation commands, runtime state beyond the required cache root, state bookkeeping, duplicated root templates, secrets, extraction invocation, storage, or publishing behavior. ### Required Validation - YAML syntax checking and production-profile ansible-lint pass for the role and test playbooks. - The standalone-role integration suite explicitly selects Notarius v0.7.0, validates a representative imported version 5 configuration through the native CLI, and retains immediate clean repeated convergence without asserting that v0.7.0 remains the default forever. - The complete existing pipeline integration entry point continues to install the pinned role combination and validate both deployed campaigns. - A representative human account in the `narratio` group can execute native Notarius configuration inspection through the same files used by the service account. - Notarius natively validates the role-managed `dnd-session` configuration. - Notarius `config effective` exposes the composed imported pipeline and a representative cache override without parsing managed YAML directly. - Narratio natively validates both base and `testing` configurations with an explicitly selected deployed campaign. - Focused native inspection observes a representative generated Notarius handoff and representative Narratio reference and output mappings rather than reproducing the full application validators in shell assertions. - A test explicitly supplies a small additional or replacement Notarius pipeline mapping and observes that selected behavior; it does not assert that `dnd-session` remains the role's current default forever. - A representative test explicitly overrides one cache choice and observes the effective Notarius configuration; it does not freeze all current cache defaults or test Notarius's cache implementation. - The immediate repeat reports `changed=0` and `failed=0`. - Tests do not execute extraction, call an LLM, snapshot complete rendered files, enumerate every nested option as an implementation lock, inspect template task results, test filesystem metadata, or reproduce Ansible, Notarius, Narratio, ConfigKit, or PromptKit behavior. ## Dependencies and Follow-up Work Features 0001 through 0009 and the standalone-role test-suite simplification are complete, so the shared filesystem and execution prerequisite is satisfied. Implementation depends on the current Notarius v0.7.0 maintained D&D configuration and ConfigKit contract, the compatible Narratio v2.0.0 subprocess boundary, campaign sources deployed by Feature 0007, and trimmed transcript policy defined by Feature 0008. The next suite feature should select which extraction sources feed new or existing Scriptorium artifacts, or address pipeline-wide credential delivery if real end-to-end execution requires it first. Later work should cover storage and publishing, optional operator selection and membership management, collection assembly, release pinning, and an end-to-end workflow using actual external services. A separate feature may add inexpensive Notarius testing extraction if a concrete workflow justifies a second operator profile or application configuration. ## Potential Future Hardening - Add explicit cache and debug retention policy once operational use provides concrete retention requirements. - Define credential delivery beneath the restricted `secrets/` subtree. No further Notarius cache-permission feature is required for shared invocation. - Pin content repositories to immutable releases when their upstream projects begin publishing them.