eric/ansible-workspace-dnd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
ansible-workspace-dnd/docs/roadmap/features/0010-dnd-notarius-extraction-integration.md

30 KiB
Raw Permalink Blame History

Feature 0010: D&D Notarius Extraction Integration

Status: done

Summary

Enable Narratio's Notarius stage with the maintained ten-lane D&D extraction pipeline. The pipeline role will supply the opinionated Notarius pipeline, bind campaign-owned inputs through Narratio's invocation boundary, and declare the exact lane contracts Narratio may accept as extraction sources. It uses the service ownership and shared-access foundation established by Feature 0009.

Problem or Opportunity

Features 0006 through 0008 install and configure the five applications, deploy the D&D content, and define the transcript-processing and Scriptorium workflow. Narratio still leaves its Notarius stage disabled, and Notarius's managed configuration contains no pipeline. A run can therefore produce a cleaned and trimmed transcript but cannot yet extract the structured D&D registries and events implemented by Notarius.

The uploaded test-machine configuration demonstrates the intended multi-step pipeline, but it embeds campaign-specific filesystem paths and explicit runtime roots. The current Notarius v0.7.0 complete example and Narratio 2.0 integration boundary provide the maintained replacement. Narratio 2.0 names Notarius v0.6.0 as its subprocess baseline, while Notarius v0.7.0 expressly preserves that subprocess receipt and extraction-artifact contract. The Notarius upgrade therefore changes deployment configuration without requiring a Narratio adapter change. Narratio must prepare invocation-local campaign references and own the run-specific output staging directory; generated registry and event handoffs remain internal to Notarius. Notarius v0.7.0 also replaces configuration version 4 with a strict, ConfigKit-backed version 5 contract and makes cache access policy a deployment responsibility.

Governing Policies

  • The Ansible house style governs the suite-owned public interface, direct desired-state configuration, and reliance on the standalone roles' native templates.
  • The testing house style governs focused native contract validation and immediate clean repeated convergence.
  • The MVP roadmap assigns opinionated D&D policy and cross-application coordination to the pipeline role while keeping the standalone application roles generic.
  • Decision 0001 requires module-owned convergence without redundant inspection or supplemental state.
  • Decision 0002 defines the service ownership, shared operator access, world-readable content, and restricted credential boundary.
  • Feature 0008 establishes the trimmed transcript, ConfigKit layout, and testing overlay into which this feature integrates.
  • Feature 0009 establishes the runtime identity, configuration ownership interfaces, and inherited shared-access policy extended by this feature.

Scope

  • Update the standalone Notarius role for Notarius v0.7.0 and configuration version 5 while preserving its generic, independently usable interface.
  • Advance the pipeline role's selected Notarius application version to v0.7.0 and pin its Notarius role dependency to the reviewed revision that provides the version 5 interface.
  • Expose Notarius's native configkit.imports field through one optional notarius_configkit_imports role variable; the role continues to own only its root configuration and does not manage caller-selected fragments.
  • Supply one complete dnd-session Notarius pipeline as deliberate D&D suite policy in a suite-managed imported fragment at /usr/local/etc/notarius/pipelines/dnd-session.yml.
  • Adapt the maintained Notarius v0.7.0 complete D&D pipeline using canonical version 5 module-binding objects, including:
    • the embedded dnd-extraction PromptKit workload profile;
    • Seriatim input and scene-based chunking;
    • the ordered describe-session, extract-events, and track-enemies steps;
    • all ten maintained D&D artifact lanes and their generated handoffs; and
    • JSON output with the chunk map and selected evidence context.
  • Remove every campaign-specific filesystem reference from the Notarius pipeline definition.
  • Enable Narratio's Notarius stage and select dnd-session.
  • Bind party, players, and glossary to Narratio's prepared campaign sources at invocation time.
  • Define Narratio acceptance descriptors for all ten maintained lane outputs.
  • Make the pipeline map, Narratio reference map, and Narratio output map ordinary suite variables so callers can compose or replace native mappings through inventory precedence.
  • Make the selected pipeline ID an ordinary suite variable and keep it coordinated with the default pipeline map.
  • Enable Notarius caching to the maximum currently available extent: use auto chunk-plan caching and enable checkpoint recording. Narratio does not yet request checkpoint reuse with Notarius's --resume flag.
  • Provision /var/cache/notarius as a shared FHS cache root owned by narratio:narratio with mode 2775, and configure Notarius to create its chunk-plan and checkpoint stores beneath that root.
  • Apply Feature 0009's established access and default ACL policy to the new cache root so content created by either supported invocation identity remains group-writable and world-readable.
  • Keep Notarius enabled under the existing testing overlay. Testing and production select the same Notarius pipeline and workload profile until the suite chooses a distinct testing profile strategy.
  • Document the reference boundary, lane-to-source mapping, runtime-state policy, production/testing behavior, and supported caller customization.
  • Extend focused integration coverage for native Notarius validation, Narratio base/testing resolution, representative reference and output wiring, and immediate clean repeated convergence.

Non-goals

  • Changing any application's native configuration fields or duplicating a standalone role's root configuration template in the pipeline role.
  • Making the standalone Notarius role manage imported fragments; fragment deployment remains the responsibility of the caller that selects an import.
  • Copying the uploaded test-machine configuration verbatim.
  • Installing a filesystem PromptKit profile for Notarius. Notarius v0.7.0 already embeds the selected production dnd-extraction fallback profile.
  • Defining a second Notarius pipeline solely to select an inexpensive testing model, or otherwise adding profile inheritance that Notarius does not natively provide.
  • Binding a spell catalog before the campaign registry declares one.
  • Publishing, rendering, or otherwise consuming the new extraction sources in Scriptorium artifacts.
  • Selecting a default campaign.
  • Delivering API keys, provider credentials, environment files, or a general secret-management mechanism.
  • Running Notarius extraction, making model calls, or processing a complete transcript during integration tests.
  • Managing output retention, cache cleanup, or debug retention.
  • Adding checkpoint-resume control to Narratio or changing Narratio's subprocess invocation to pass --resume or --config-profile.
  • Installing a systemd service, timer, scheduler, or other unattended invocation mechanism.
  • Selecting human operator accounts or changing their membership in the narratio group.
  • Revising the general configuration ownership, dependency ordering, ACL dependency, or existing shared roots established by Feature 0009.
  • Delivering or populating the future restricted secrets/ directory.
  • Modifying upstream application source, embedded Notarius prompts, schemas, modules, codecs, or validators.
  • Collection assembly, role releases, storage/publishing, upgrade/removal behavior, or end-to-end production validation.

Target End State

The standalone Notarius role continues to own the version 5 root document at /usr/local/etc/notarius/config.yml. Its public interface exposes the native configkit.imports list, and the pipeline role selects the explicit relative import pipelines/dnd-session.yml. The pipeline role owns that fragment and renders the complete dnd_pipeline_notarius_pipelines map into it, producing one native dnd-session definition by default. The definition follows the Notarius v0.7.0 maintained complete D&D topology:

Seriatim transcript
  -> dnd/scenes chunking
  -> describe-session
       item-registry
       npc-registry
       location-registry
       scene-descriptions
  -> extract-events
       item-occurrences
       spells
       combat-turns
       npc-occurrences
       location-occurrences
  -> track-enemies
       enemy-events
  -> JSON bundle, chunk map, and evidence context

Feature 0009 already makes application configuration service-owned and world-readable. The Notarius root document and new suite fragment retain that narratio:narratio, 0755 directory, and 0644 file policy through the standalone role's established ownership and mode interface.

The extract-events step consumes the four generated outputs from describe-session under their declared internal reference slots. The track-enemies step consumes the generated NPC registry and scene descriptions from describe-session and the generated combat turns and NPC occurrences from extract-events. Those handoffs are part of the Notarius pipeline and never appear in Narratio's CLI reference map.

Narratio's effective base configuration enables Notarius, selects the managed configuration and dnd-session, and supplies three campaign-owned references:

Notarius selector Prepared Narratio source
party narratio.input.party
players narratio.input.players
glossary narratio.input.glossary

Narratio snapshots those inputs for each invocation and passes their absolute paths with --reference. No campaign path appears in Notarius configuration. The optional spell_catalog binding is absent because neither deployed campaign currently declares inputs.spell_catalog_file. A caller that adds that campaign input can extend the suite reference map with spell_catalog: narratio.input.spell_catalog without changing either application role.

Narratio accepts all ten configured lanes using these exact v1 contracts and registers each accepted payload as narratio.extraction.<output-key>:

Narratio output key Lane ID Schema ID Module key
item_registry item-registry notarius.dnd.item_registry dnd/item-registry
npc_registry npc-registry notarius.dnd.npc_registry dnd/npc-registry
location_registry location-registry notarius.dnd.location_registry dnd/location-registry
scene_descriptions scene-descriptions notarius.dnd.scene_descriptions dnd/scene-descriptions
item_occurrences item-occurrences notarius.dnd.item_occurrences dnd/item-occurrences
spells spells notarius.dnd.spells dnd/spells
combat_turns combat-turns notarius.dnd.combat_turns dnd/combat-turns
npc_occurrences npc-occurrences notarius.dnd.npc_occurrences dnd/npc-occurrences
location_occurrences location-occurrences notarius.dnd.location_occurrences dnd/location-occurrences
enemy_events enemy-events notarius.dnd.enemy_events dnd/enemy-events

Every entry requires media_type: application/json and schema_version: v1. Narratio validates the received descriptor before registering the source; it does not infer contracts from filenames.

Narratio supplies its run-specific staging root through Notarius's --output-dir override, so the suite does not set Notarius's global output.directory. Reconstructible Notarius state uses the FHS cache layout:

/var/cache/notarius/
├── chunk-plans/
└── checkpoints/

The pipeline role creates the root as narratio:narratio mode 2775, sets chunk-plan mode to auto, enables checkpoint recording, and configures both child paths. Notarius creates cache entries with conventional maximum creation modes and leaves their effective modes to deployment provisioning, process umask, setgid inheritance, and default ACLs. The suite applies a default ACL that retains owner and group write access and world read/traverse access for new cache content. Narratio and its Notarius subprocess may run as the dedicated narratio account or as a human member of the narratio group. Chunk plans are reused automatically; checkpoints are recorded but are not reused because Narratio does not pass --resume. The debug root remains unset. The selected Notarius process working directory is /usr/local/etc/notarius, matching the managed configuration location; no relative profile or reference paths depend on it.

The cache root receives the same setgid and default-ACL policy that Feature 0009 applies to Narratio's existing shared roots. Non-secret content created beneath it remains writable by the shared group and readable by other local users. The suite does not create or populate a secrets/ directory.

The base Narratio configuration and the testing ConfigKit overlay both run the extraction stage using Notarius's embedded dnd-extraction profile. The testing overlay continues to override only the Scriptorium profile selections; it does not disable or restate Notarius. A later feature may select a distinct inexpensive Notarius testing profile without changing the pipeline topology.

Repository Ownership

  • ansible-role-dnd-pipeline owns the D&D pipeline map and imported fragment, Narratio integration mappings, Notarius dependency-variable wiring, the new cache root and its inherited shared-access policy, testing-overlay choice, consumer documentation, and suite integration coverage.
  • ansible-role-notarius owns the generic Notarius v0.7.0 installation, version 5 root configuration, optional ConfigKit import interface, focused role documentation, and role-local integration coverage. It does not own or render the suite's imported D&D fragment.
  • ansible-role-audita, ansible-role-narratio, and ansible-role-scriptorium remain unchanged. The Narratio role owns its generic Notarius adapter fields and ConfigKit-aware root configuration.
  • ../notarius is the read-only authority for configuration version 5, the maintained D&D pipeline, embedded profile, module handoffs, and lane contracts.
  • ../narratio is the read-only authority for prepared campaign sources, subprocess invocation, output acceptance, and profile overlay behavior.
  • assets/configs/notarius/config.yml is an operational reference only. Its useful topology is adapted, while its campaign paths and runtime roots are not copied.
  • assets/campaigns remains unchanged and continues to own the campaign registry inputs that Narratio prepares.
  • ansible-workspace-dnd owns this roadmap, the later implementation plan, settled MVP decisions, and aggregate status.

Constraints and Settled Decisions

  • This feature advances the standalone role default and pipeline application selection from Notarius v0.6.0 to v0.7.0. Narratio remains at v2.0.0.
  • Narratio v2.0.0 documents Notarius v0.6.0 as its invocation baseline, but Notarius v0.7.0 preserves the subprocess receipt and extraction-artifact contracts. Version 5 configuration migration is confined to the Notarius role and the suite-owned Notarius fragment.
  • The pipeline dependency manifest must pin the reviewed standalone Notarius role revision that implements this v0.7.0/configuration-version-5 contract.
  • Feature 0009 has already established the runtime identity, dependency order, standalone ownership interfaces, configuration modes, acl package, and pinned ansible.posix dependency used here.
  • The pipeline role relies on the standalone roles for application installation and both root configuration documents. It manages only the suite-owned Notarius and Narratio fragments selected through those public interfaces.
  • The standalone Notarius role defaults to configuration version 5 and exposes notarius_configkit_imports as the complete native import list. A null value omits the configkit section and leaves composition with Notarius.
  • The pipeline role passes the fixed import pipelines/dnd-session.yml to the standalone role and manages that imported file as suite-owned configuration. It does not pass the pipeline map into the standalone role's root-level notarius_pipelines field.
  • /var/cache/notarius uses the Feature 0009 runtime identity, mode 2775, and access/default ACL entries equivalent to u::rwx, g::rwx, o::r-x, and m::rwx.
  • dnd_pipeline_notarius_pipelines is a complete native map whose default contains dnd-session. Callers may replace it or compose additional pipeline definitions with Jinja's combine filter in inventory.
  • dnd_pipeline_narratio_notarius_pipeline_id defaults to dnd-session. Callers that replace that key are responsible for selecting a matching ID.
  • dnd_pipeline_narratio_notarius_references is the complete native Narratio reference map. It defaults to party, players, and glossary; its values are prepared source IDs, never filesystem paths.
  • dnd_pipeline_narratio_notarius_outputs is the complete native Narratio output map. It defaults to all ten lanes because the suite claims to consume the complete maintained D&D extraction workflow.
  • dnd_pipeline_notarius_cache_root defaults to /var/cache/notarius, dnd_pipeline_notarius_cache_chunk_plans_mode defaults to auto, and dnd_pipeline_notarius_cache_checkpoints_enabled defaults to true. They are intentional, inventory-overridable suite policy. The two application cache directories derive from that root rather than becoming independent suite variables.
  • The pipeline defaults to llm_profile: dnd-extraction and relies on Notarius's embedded fallback. It does not set promptkit.profile_file, promptkit.profile_dir, a local backend, or credentials.
  • The pipeline uses the version 5 input: {module: seriatim} binding, dnd/scenes chunking with two retries, and the three ordered steps documented above. Extract, merge, and normalize bindings use canonical objects with module:; retries and generated reference handoffs match the maintained Notarius complete example.
  • JSON output enables include_chunk_map and evidence context with a three-unit window. Evidence context includes item occurrences, item registry, location registry, location occurrences, NPC registry, spells, combat turns, NPC occurrences, and enemy events, matching the maintained example.
  • Notarius pipeline YAML contains no external file references. Narratio owns all campaign input materialization for the orchestrated path.
  • Narratio's Notarius binary and configuration paths remain the established /usr/local/bin/notarius and /usr/local/etc/notarius/config.yml.
  • The suite uses the runtime user and group selected by Feature 0009 for the Notarius cache and configuration fragment.
  • Narratio's Notarius working directory is explicitly /usr/local/etc/notarius; the timeout inherits Narratio's native three-hour default because the suite has selected no different policy.
  • Narratio owns per-run Notarius output staging. The suite provisions /var/cache/notarius for the shared narratio identity, and Notarius owns its child cache stores. The role does not create /var/lib/notarius, set a global output root, or provision a debug root.
  • Checkpoint enablement records reusable state but does not cause Narratio to reuse it. Resume remains an explicit Notarius invocation action and requires a later Narratio integration decision.
  • The production configuration enables Notarius. The existing testing overlay inherits that enabled stage and the same dnd-extraction profile; it does not override Notarius merely to avoid model calls.
  • All maps use native application shapes and remain replaceable through normal inventory precedence. The suite does not create field-by-field aliases for every nested lane or module option.
  • Directory and ACL tasks declare only the shared roots required by the applications, using ordinary state-management modules with task-level privilege escalation. No stat, command, handler, checksum, manifest, or production validation task accompanies them.

Simplest Viable Implementation

The production changes remain direct configuration management:

  1. Update the standalone Notarius role's version and configuration-version defaults and add the nullable native ConfigKit import field to defaults, argument specification, root template, and documentation. Cover that public behavior with focused native validation and a clean repeat.
  2. Advance the pipeline's Notarius application selection and standalone-role dependency pin, then add the selected Notarius pipeline map, shared FHS cache policy, and three suite-level Narratio integration mappings to pipeline-role defaults and the argument specification.
  3. Pass the fixed import and cache choices to rakestraw.notarius, and pass Notarius enablement, selected ID, working directory, references, and outputs to rakestraw.narratio.
  4. Create the Notarius pipeline-fragment directory and cache root with ordinary ansible.builtin.file tasks, render the suite-owned pipeline map into one imported fragment, and extend Feature 0009's access/default ACL task to the new root.
  5. Update both role READMEs and extend focused integration coverage with native configuration validation, effective-configuration inspection, inspection as a supported group member, and clean repeats.

The standalone roles' template operations continue to own both root configuration files. The pipeline role adds one Notarius pipeline-fragment template but does not duplicate the root document or change the existing Narratio testing-overlay template for this feature.

Rationale and Alternatives

Using Notarius's maintained complete example preserves the upstream module topology and contract while keeping suite policy explicit. Maintaining a smaller custom pipeline would make the role responsible for deciding which D&D artifacts no longer matter. Copying the uploaded file would instead preserve obsolete host paths and bypass Narratio's verified reference boundary.

Rendering one native map into a ConfigKit-imported fragment keeps configuration ownership clear and preserves the inventory-composition workflow already selected for Notarius. It also follows Notarius v0.7.0's maintained composition shape without making the generic role own D&D content. Field-specific variables for every lane, step, and retry would create a large parallel schema with little consumer value. One complete map is easier to replace, combine, validate, and compare with Notarius documentation.

/var/cache/notarius follows the filesystem hierarchy's intended location for reconstructible application state and is preferable to the uploaded /var/lib/notarius/cache layout. Assigning the root to the dedicated narratio account and group gives unattended and human-initiated runs one reusable cache. Mode 2775 preserves the shared group, and the default ACL prevents an invoking account's umask from accidentally removing group write or world read access from new content. Notarius v0.7.0 respects this deployment policy rather than replacing it with application-owned private modes. Narratio's staging root remains separate because output bundles are run results, not cache data.

The testing overlay cannot natively replace a nested Notarius pipeline's llm_profile; that configuration lives in a separate application file. Duplicating the full pipeline or managing a second Notarius root configuration would add substantial configuration drift for one model choice. Testing therefore inherits production extraction for now; a distinct testing strategy can be selected later if it becomes a concrete need.

Acceptance Criteria

Required Configuration Behavior

  • The pipeline role installs a Notarius configuration that natively validates the dnd-session pipeline without provider credentials or a model call.
  • The pipeline role selects Notarius v0.7.0 and its dependency manifest pins the reviewed standalone-role revision that supplies the version 5 interface.
  • The standalone Notarius role supports an explicitly selected v0.7.0, emits configuration version 5 with its compatible defaults, and can render a caller-supplied native ConfigKit import list.
  • The root Notarius configuration imports the suite-owned pipelines/dnd-session.yml fragment, and that fragment contains the complete pipeline map selected through inventory.
  • The pipeline contains the maintained Seriatim input, D&D scene chunker, three ordered steps, ten lanes, generated handoffs, JSON chunk map, and selected evidence-context policy.
  • The Notarius configuration contains no campaign-specific reference paths, PromptKit operator-profile path, fixed output root, or debug root.
  • The Notarius configuration selects auto chunk-plan caching, enables checkpoints, and uses /var/cache/notarius/chunk-plans and /var/cache/notarius/checkpoints respectively.
  • The deployed cache root is narratio:narratio mode 2775; its access and default ACLs preserve group write and world read/traverse access without depending on Notarius to enforce entry modes. Checkpoints are recorded; the roadmap does not claim Narratio reuses them.
  • The Notarius root configuration and suite-owned imported fragment retain Feature 0009's narratio:narratio, 0755 directory, and 0644 file policy. No secret value appears in either file.
  • Narratio's effective base configuration enables Notarius, selects dnd-session, and uses the installed binary, configuration, and working directory.
  • Narratio maps party, players, and glossary to the three documented prepared campaign sources and contains no default spell_catalog binding.
  • Narratio declares all ten documented output entries with exact lane, media, schema, version, and module identities.
  • Narratio natively validates the effective base configuration for either deployed campaign without a model call.
  • narratio config show or the equivalent native inspection surface exposes the enabled stage, selected pipeline, representative reference, and representative output contract.
  • The effective testing profile keeps Notarius enabled with the same pipeline and profile while retaining the existing Scriptorium testing overrides.
  • A caller can compose an additional Notarius pipeline into dnd_pipeline_notarius_pipelines and select it through the corresponding Narratio pipeline-ID variable without changing the role.
  • A caller can add the optional spell-catalog source by extending the native Narratio reference map after providing that campaign input.
  • An immediate repeated role application with unchanged inputs reports no changes.
  • The production role contains no new configuration probes, validation commands, runtime state beyond the required cache root, state bookkeeping, duplicated root templates, secrets, extraction invocation, storage, or publishing behavior.

Required Validation

  • YAML syntax checking and production-profile ansible-lint pass for the role and test playbooks.
  • The standalone-role integration suite explicitly selects Notarius v0.7.0, validates a representative imported version 5 configuration through the native CLI, and retains immediate clean repeated convergence without asserting that v0.7.0 remains the default forever.
  • The complete existing pipeline integration entry point continues to install the pinned role combination and validate both deployed campaigns.
  • A representative human account in the narratio group can execute native Notarius configuration inspection through the same files used by the service account.
  • Notarius natively validates the role-managed dnd-session configuration.
  • Notarius config effective exposes the composed imported pipeline and a representative cache override without parsing managed YAML directly.
  • Narratio natively validates both base and testing configurations with an explicitly selected deployed campaign.
  • Focused native inspection observes a representative generated Notarius handoff and representative Narratio reference and output mappings rather than reproducing the full application validators in shell assertions.
  • A test explicitly supplies a small additional or replacement Notarius pipeline mapping and observes that selected behavior; it does not assert that dnd-session remains the role's current default forever.
  • A representative test explicitly overrides one cache choice and observes the effective Notarius configuration; it does not freeze all current cache defaults or test Notarius's cache implementation.
  • The immediate repeat reports changed=0 and failed=0.
  • Tests do not execute extraction, call an LLM, snapshot complete rendered files, enumerate every nested option as an implementation lock, inspect template task results, test filesystem metadata, or reproduce Ansible, Notarius, Narratio, ConfigKit, or PromptKit behavior.

Dependencies and Follow-up Work

Features 0001 through 0009 and the standalone-role test-suite simplification are complete, so the shared filesystem and execution prerequisite is satisfied. Implementation depends on the current Notarius v0.7.0 maintained D&D configuration and ConfigKit contract, the compatible Narratio v2.0.0 subprocess boundary, campaign sources deployed by Feature 0007, and trimmed transcript policy defined by Feature 0008.

The next suite feature should select which extraction sources feed new or existing Scriptorium artifacts, or address pipeline-wide credential delivery if real end-to-end execution requires it first. Later work should cover storage and publishing, optional operator selection and membership management, collection assembly, release pinning, and an end-to-end workflow using actual external services. A separate feature may add inexpensive Notarius testing extraction if a concrete workflow justifies a second operator profile or application configuration.

Potential Future Hardening

  • Add explicit cache and debug retention policy once operational use provides concrete retention requirements.
  • Define credential delivery beneath the restricted secrets/ subtree. No further Notarius cache-permission feature is required for shared invocation.
  • Pin content repositories to immutable releases when their upstream projects begin publishing them.
Reference in New Issue View Git Blame Copy Permalink