Prepare the D&D processing configuration sprint

docs/roadmap/features/0008-opinionated-dnd-processing-configuration.md

@@ -0,0 +1,473 @@
+# Feature 0008: Opinionated D&D Processing Configuration
+
+Status: ready
+
+## Summary
+
+Turn the installed application suite and deployed D&D assets into a coherent
+processing configuration. The pipeline role will supply deliberately
+opinionated Audita and Narratio policy, split Narratio's suite-owned processing
+and artifact definitions into explicit ConfigKit fragments, and provide a
+named `testing` profile for inexpensive prompt and model iteration without
+changing production policy.
+
+## Problem or Opportunity
+
+Features 0006 and 0007 install the five applications, connect their standard
+paths, and deploy valid campaigns, prompts, and schemas. The assembled role
+still leaves most optional application policy unset. Narratio can validate its
+minimal root document and discover content, but it does not yet describe the
+D&D trimming, rendering, recap, rumor, scene-map, or per-character analysis
+workflow.
+
+A current test-machine configuration provides useful evidence of the intended
+workflow, but it cannot be copied. It includes pre-Narratio-2.0 syntax,
+installation-specific storage and inference settings, obsolete paths,
+campaign-specific character entries, and redundant artifact dependencies.
+The current application repositories remain authoritative for supported
+configuration.
+
+Narratio 2.0 also provides two useful composition mechanisms that the suite
+has not yet adopted: additive root imports for independently owned base
+fragments and a fixed `profiles/testing.yml` overlay. Using both will make the
+production policy reviewable and make inexpensive development runs easy to
+select without editing the production configuration.
+
+## Governing Policies
+
+- [The Ansible house style](../../ansible.md) governs the suite-owned public
+  interface, templates, direct desired-state tasks, and omission of
+  unselected application defaults.
+- [The testing house style](../../testing.md) governs native behavioral
+  validation, explicit testing inputs, and clean repeated convergence.
+- [The MVP roadmap](../mvp.md) assigns opinionated cross-application and D&D
+  policy to the pipeline role while keeping standalone application roles
+  generic.
+- [Decision 0001](../../decisions/0001-minimal-module-driven-ansible.md)
+  requires direct module-owned convergence without supplemental state or
+  redundant verification.
+
+## Scope
+
+- Supply the complete reviewed Audita configuration as deliberate D&D suite
+  policy, including its context, schema, module sequence, endpoint, models,
+  credential environment-variable names, timeouts, retries, concurrency,
+  chunking, normalization, thresholds, and diagnostics settings.
+- Configure Narratio's normalization, trimming, and Markdown rendering stages
+  through its current public contract, explicitly pinning only the schemas,
+  identifiers, enablement choices, and paths that form the D&D suite's
+  cross-stage contract.
+- Use the deployed `dnd.session_bounds` prompt for transcript-bound detection.
+- Define each ordinary Narratio Scriptorium artifact in its own managed YAML
+  fragment:
+  - session recap;
+  - hostile rumors;
+  - deranged rumors; and
+  - scene map.
+- Define per-character gameplay analysis as one Narratio artifact-family
+  fragment over `party.characters`, not as campaign-specific artifact entries.
+- Supply only the inputs declared by each current prompt definition.
+- Let each production prompt select its own declared Promptkit default profile
+  unless the roadmap records a concrete reason for a suite override.
+- Manage a valid `profiles/testing.yml` Narratio overlay that selects an
+  inexpensive Scriptorium profile while leaving the base production policy
+  and inherited Audita configuration unchanged.
+- Expose the suite-owned testing-profile choice through normal Ansible
+  inventory precedence.
+- Compose the base fragments through an explicit ordered
+  `narratio_configkit_imports` list in the standalone Narratio role's root
+  configuration.
+- Create and manage the required `conf.d`, `artifacts`,
+  `artifact-families`, and `profiles` directories and their suite-owned files
+  under `/usr/local/etc/narratio`.
+- Document production and testing invocations, the imported layout, public
+  variables, and the boundary with later pipeline features.
+- Extend focused integration coverage for both base and `testing` resolution,
+  representative artifact relationships, party-driven family expansion, and
+  immediate clean repeated convergence.
+
+## Non-goals
+
+- Enabling Narratio's Notarius stage or installing the uploaded Notarius
+  pipeline. Its multi-stage D&D extraction, campaign reference bindings,
+  output descriptors, and runtime directories remain a separate feature.
+- Configuring S3 or other remote storage, publishing outputs, upload behavior,
+  or cleanup-after-publish.
+- Selecting a default campaign. Campaign choice remains an inventory or
+  invocation decision.
+- Delivering API keys, object-storage credentials, environment files, or a
+  general secret-management mechanism.
+- Selecting human operators or managing their Narratio group membership.
+- Adding a scheduler, service, or automated invocation.
+- Running WhisperX, Audita model calls, Scriptorium model calls, Notarius, or
+  a complete transcript during integration tests.
+- Copying the test-machine configuration verbatim or preserving its obsolete
+  Narratio, campaign, Scriptorium, or credential paths.
+- Duplicating prompt content, prompt schemas, built-in Promptkit profiles, or
+  application validation rules in the pipeline role.
+- Modifying the standalone application roles or upstream application source.
+- Collection assembly, role releases, asset release policy, upgrade/removal
+  behavior, or end-to-end production validation.
+
+## Target End State
+
+The standalone Narratio role continues to own
+`/usr/local/etc/narratio/pipeline.yml`. That root document contains the
+installation paths already supplied by the pipeline role and an explicit list
+of suite-owned relative imports. The pipeline role owns the imported files:
+
+```text
+/usr/local/etc/narratio/
+├── pipeline.yml
+├── conf.d/
+│   └── processing.yml
+├── artifacts/
+│   ├── session-recap.yml
+│   ├── hostile-rumors.yml
+│   ├── deranged-rumors.yml
+│   └── scene-map.yml
+├── artifact-families/
+│   └── character-meta-analysis.yml
+└── profiles/
+    └── testing.yml
+```
+
+Each base artifact or family key is declared by exactly one fragment. The
+root uses explicit `path` imports rather than globs; Narratio therefore loads
+only the files selected by current suite policy, even if an obsolete file is
+left on a previously managed host. Imported maps remain additive and no two
+base sources claim the same leaf.
+
+The production configuration uses the deployed prompt definitions and their
+frontier-oriented default profiles. Audita uses the selected
+`https://inference.ai.rakestrawhome.com/v1` endpoint and
+`google/gemma-4-31b-it` proposal and validation model. Narratio's canonical
+processing paths and schemas are suite-owned contracts rather than incidental
+copies of application defaults. Session recap consumes the final trimmed
+transcript and may consume the previous session's recap. Hostile and deranged
+rumors consume the current recap, allowing Narratio to infer their dependency
+without a duplicate `depends_on` declaration. Scene map is configured as an
+available, normally disabled artifact. Character analysis is an available
+party-driven family whose generated members receive `character.name` and
+`character.class_summary` through family member variables.
+
+The installed `testing` profile is a valid overlay selected with
+`--profile testing`. It selects `gemini-flash-latest` for each Scriptorium
+artifact and artifact family while inheriting the base topology, paths,
+prompts, inputs, outputs, and enabled state. Audita inherits the same endpoint
+and model in both modes for now, so the overlay does not restate its base
+configuration. The testing profile neither imports files nor contains
+credentials. Operators can validate, inspect, or run the same session under
+base or testing policy without modifying an Ansible-managed file.
+
+## Repository Ownership
+
+- `ansible-role-dnd-pipeline` owns the suite variables, Narratio fragment and
+  testing-profile templates, direct directory/file convergence, dependency
+  variable wiring, consumer documentation, and integration coverage. It is
+  the only implementation repository expected to change.
+- `ansible-role-narratio` remains unchanged. It owns the root pipeline
+  template and the public `narratio_configkit_imports` and stage variables
+  through which the suite composes it.
+- `ansible-role-audita` remains unchanged. It owns Audita's field-specific
+  configuration interface; the pipeline role selects only intentional D&D
+  policy through that interface.
+- `ansible-role-scriptorium`, `assets/narratio-prompts`, and `../promptkit`
+  remain unchanged authorities for source-directory configuration, prompt
+  contracts, and built-in production/testing profile choices.
+- `../audita` and `../narratio` remain unchanged authorities for current
+  application configuration and native inspection behavior.
+- `ansible-role-notarius` and `../notarius` are read-only inputs for the later
+  D&D extraction feature.
+- `rakestraw-ansible` and the uploaded test-machine configurations are
+  historical and operational references, not application contracts.
+- `ansible-workspace-dnd` owns this roadmap, the later implementation plan,
+  settled MVP decisions, and aggregate status.
+
+## Constraints and Settled Decisions
+
+- Standalone application roles remain generic. Every concrete value added by
+  this feature must represent policy deliberately owned by the D&D suite.
+- Application defaults observed in documentation or the uploaded files are
+  not copied merely to restate them. An explicit suite value must preserve a
+  chosen outcome across upstream default changes.
+- The pipeline explicitly owns a value when another stage, artifact, prompt,
+  or future publishing rule relies on its schema, identifier, enabled state,
+  or path. Behavior confined to Narratio inherits Narratio's application
+  default unless the suite has independently selected that behavior as policy.
+- The pipeline role does not duplicate Narratio's root template. It acts as a
+  caller: the standalone role renders the root and the pipeline role deploys
+  the suite-owned fragments named by that root.
+- Every ordinary artifact and every artifact family has its own source and
+  installed file. General stage policy may share `conf.d/processing.yml`.
+- Imports are explicit relative `path` entries. Globs are not used because
+  the role does not need directory discovery and must not activate stale
+  files accidentally.
+- The imported files cannot contain `configkit`, nested imports, or profile
+  directives. The `testing` overlay cannot import or stack other profiles.
+- Suite-managed configuration directories are `root:narratio` mode `0750`;
+  files are `root:narratio` mode `0640`, matching the established Narratio
+  configuration boundary.
+- The role declares directory and template state directly. It does not
+  inspect files first, validate production configuration during convergence,
+  keep manifests, or track fragment digests.
+- The base artifact definitions omit `profile_id` when the deployed prompt's
+  declared default is the selected production policy.
+- Artifact inputs must match the current prompt contracts:
+  - session recap: required `transcript`, optional `recap`;
+  - hostile and deranged rumors: required `recap`;
+  - scene map: required `transcript`; and
+  - character analysis family: required `transcript`, optional `recap`, plus
+    member variables for `character_name` and `character_class`.
+- A `narratio.artifact.session_recap` input already creates the rumor
+  dependency. The base fragments do not repeat that edge with `depends_on`.
+- Scene map and character analysis are configured but disabled by default;
+  Narratio's explicit artifact selection may still run their concrete entries.
+- The character family uses `for_each: party.characters`, an output path with
+  exactly one `{character_id}` token, and member selectors
+  `character.name` and `character.class_summary`. It contains no hard-coded
+  campaign character names.
+- Production and testing policy never embeds secret values. Credential
+  delivery remains a later feature.
+- The testing profile is a single valid overlay at the native fixed location.
+  It overrides each Scriptorium artifact and family `profile_id` with the
+  suite's `gemini-flash-latest` testing-profile variable rather than
+  duplicating complete definitions. Audita deliberately uses the same base
+  endpoint and models under both configurations.
+- Each fixed artifact or family fragment has one complete native-mapping
+  variable: `dnd_pipeline_narratio_session_recap_artifact`,
+  `dnd_pipeline_narratio_hostile_rumors_artifact`,
+  `dnd_pipeline_narratio_deranged_rumors_artifact`,
+  `dnd_pipeline_narratio_scene_map_artifact`, and
+  `dnd_pipeline_narratio_character_analysis_family`. The file names, imports,
+  and top-level source keys remain fixed suite topology; callers customize the
+  mapping and disable an available source with `enabled: false` rather than
+  removing its managed fragment or import.
+
+## Selected Narratio Coordination Policy
+
+The pipeline role pins the following effective Narratio fields because they
+connect processing stages, prompt contracts, or stable suite outputs. A value
+that currently equals an application default is still intentional suite policy
+when an upstream change would otherwise break that connection.
+
+| Native Narratio field | D&D suite default | Coordination purpose |
+| --- | --- | --- |
+| `pipeline.seriatim.output_schema` | `seriatim-intermediate` | Establishes the merge-stage transcript contract. |
+| `pipeline.audita.output_schema` | `audita-v1` | Matches the Audita configuration and adapter output contract. |
+| `pipeline.normalize.output_path` | `transcripts/final.json` | Provides the canonical normalized transcript source. |
+| `pipeline.normalize.output_schema` | `seriatim-intermediate` | Provides the canonical schema consumed by later stages. |
+| `pipeline.trim.enabled` | `true` | Makes the trimmed transcript part of the selected D&D workflow. |
+| `pipeline.trim.output_path` | `transcripts/final.trimmed.json` | Provides the canonical transcript consumed by D&D artifacts. |
+| `pipeline.trim.bounds.prompt_id` | `dnd.session_bounds` | Selects the deployed D&D bounds prompt. |
+| `pipeline.trim.bounds.transcript_input_name` | `transcript` | Matches that prompt's declared input contract. |
+| `pipeline.trim.bounds.output_path` | `artifacts/session_bounds.json` | Provides a stable bounds artifact location. |
+| `pipeline.render.enabled` | `true` | Selects rendered transcripts as a suite outcome. |
+| `pipeline.render.format` | `markdown` | Selects the suite's rendered transcript format. |
+
+The fixed artifact mappings own these output paths:
+
+| Source | D&D suite output |
+| --- | --- |
+| `session_recap` | `artifacts/session_recap.md` |
+| `hostile_rumors` | `artifacts/hostile_rumors.md` |
+| `deranged_rumors` | `artifacts/deranged_rumors.md` |
+| `scene_map` | `artifacts/scene_map.json` |
+| `character_meta_analysis` family | `artifacts/character_meta_analysis/{character_id}.md` |
+
+The processing fragment omits Narratio-local choices such as reports,
+timeouts, debug rendering, work-directory retention, render title, timestamp,
+segment-ID, and metadata presentation, and the bounds prompt profile. Narratio
+continues to supply those defaults. Likewise, the Narratio Audita adapter does
+not repeat Audita's endpoint, models, module sequence, tuning, or diagnostic
+settings: it uses the pipeline-managed Audita configuration file. The Audita
+output schema is the deliberate exception because it spans that application
+boundary, and one suite value must feed both configurations.
+
+Audita does not choose a fixed final transcript path in this integration.
+Narratio supplies the run-specific `--output` path when invoking Audita and
+then normalizes the result to the canonical path above. The role therefore
+does not invent a separate Audita output-path setting.
+
+## Selected Audita and Testing Policy
+
+The pipeline role deliberately owns all of the following Audita values. They
+are suite defaults, not assertions about Audita's upstream defaults, and each
+remains replaceable through the corresponding suite-prefixed role variable.
+
+| Native Audita field | D&D suite default |
+| --- | --- |
+| `pipeline.modules` | `[glossary, homophones, glossary, spoken_word, grammar]` |
+| `output.schema` | `audita-v1` |
+| `llm.proposal.base_url` | `https://inference.ai.rakestrawhome.com/v1` |
+| `llm.proposal.model` | `google/gemma-4-31b-it` |
+| `llm.proposal.api_key_env` | `AUDITA_LLM_API_KEY` |
+| `llm.proposal.timeout` | `900s` |
+| `llm.proposal.max_retries` | `3` |
+| `llm.validation.base_url` | `https://inference.ai.rakestrawhome.com/v1` |
+| `llm.validation.model` | `google/gemma-4-31b-it` |
+| `llm.validation.api_key_env` | `AUDITA_VALIDATION_LLM_API_KEY` |
+| `llm.validation.timeout` | `900s` |
+| `llm.validation.max_retries` | `3` |
+| `concurrency.total_llm` | `4` |
+| `concurrency.proposal_llm` | `4` |
+| `concurrency.validation_llm` | `4` |
+| `chunking.target_sections` | `8` |
+| `chunking.max_section_tokens` | `6144` |
+| `chunking.min_section_tokens` | `2048` |
+| `normalization.max_segment_gap` | `3s` |
+| `normalization.ellipsis_gap` | `3s` |
+| `normalization.max_segment_duration` | `240s` |
+| `normalization.max_segment_tokens` | `2048` |
+| `thresholds.glossary` | `0.8` |
+| `thresholds.homophones` | `0.8` |
+| `thresholds.spoken_word` | `0.8` |
+| `thresholds.grammar` | `0.8` |
+| `context.description` | `The following transcript was recorded during a Dungeons & Dragons game session. It may contain some unrelated pre-session or post-session banter, in addition to actual D&D gameplay.` |
+| `diagnostics.work_dir` | `/tmp/audita` |
+| `diagnostics.retention` | `auto` |
+
+The repeated `glossary` entry is intentional and preserves the selected
+multi-pass module sequence. Environment-variable names are configuration, not
+secret values; the later secret-delivery feature will provide their contents.
+
+The testing overlay uses one suite variable whose default is
+`gemini-flash-latest` for every Scriptorium artifact and artifact family.
+Callers may replace that profile normally. Audita uses the production endpoint
+and models in testing until the suite deliberately selects alternatives; the
+overlay omits identical Audita fields rather than writing them twice. These
+model choices are expected to evolve rapidly and can be updated as ordinary
+suite policy without changing the fragment architecture.
+
+## Simplest Viable Implementation
+
+The production change should remain a small configuration extension:
+
+1. Add only the suite variables required by the selected Audita policy,
+   selected Narratio coordination policy, fragment mappings, and
+   testing-profile override; align the argument specification.
+2. Pass the Audita choices and explicit Narratio import list through the
+   existing dependency interfaces.
+3. Create the four Narratio configuration subdirectories with one
+   `ansible.builtin.file` loop.
+4. Render the processing fragment, four artifact fragments, one family
+   fragment, and testing overlay with one `ansible.builtin.template` loop.
+5. Update the README and extend the existing integration driver with native
+   base/testing validation and a clean repeat.
+
+No preliminary `stat`, configuration assembly fact, handler, validation
+command, checksum, manifest, or fragment cleanup task is required. Explicit
+imports make an obsolete unreferenced file inert without destructive directory
+reconciliation.
+
+## Rationale and Alternatives
+
+ConfigKit fragments preserve the application's native composition model and
+give each artifact a focused review boundary. Rendering one monolithic
+Narratio mapping through the standalone root variable would produce a valid
+file but would discard that operational and editorial advantage. Reimplementing
+the entire root template in the pipeline role would duplicate the standalone
+role and create competing configuration ownership.
+
+Explicit imports are preferred to globs because the suite already knows every
+managed fragment. A glob would add discovery behavior and could keep a removed
+artifact active through a stale file. Deleting and recreating the directory on
+every run would harm idempotence and caller safety merely to compensate for
+that avoidable ambiguity.
+
+One testing overlay is preferable to a second complete configuration tree.
+Profiles replace only selected leaves after base composition, so topology and
+paths cannot drift between production and development. The selected
+`gemini-flash-latest` default provides immediate value while remaining one
+ordinary role-default change when model economics evolve.
+
+The uploaded Notarius pipeline is concrete and valid, but adding it here would
+combine a ten-lane extraction contract, campaign-reference mapping, output
+descriptor mapping, runtime directories, and model policy with an already
+coherent Narratio/Scriptorium feature. Keeping it separate makes both reviews
+smaller and preserves a clear acceptance boundary.
+
+## Acceptance Criteria
+
+### Required Configuration Behavior
+
+- The pipeline role produces the documented imported configuration layout
+  with one file per artifact and artifact family.
+- The root pipeline document contains the exact explicit relative imports and
+  no glob imports.
+- Base Narratio configuration validates natively for either deployed campaign.
+- The effective base configuration uses the selected Seriatim, Audita,
+  normalization, trimming, bounds, and rendering contracts and the canonical
+  artifact output paths documented above.
+- `narratio config artifacts` reports a valid recap-to-rumor graph and expands
+  the character-analysis family for the selected campaign.
+- The recap, rumor, scene-map, and character-family definitions use only inputs
+  and variables supported by their deployed prompt definitions and Narratio's
+  current source vocabulary.
+- The two rumor dependencies are inferred from their recap input sources and
+  are not duplicated as explicit ordering edges.
+- Scene map and the character-analysis family are available but disabled by
+  default; recap and both rumor artifacts are enabled.
+- Production artifacts inherit their selected profiles from the prompt
+  definitions unless an expressly agreed suite override is documented.
+- Audita's managed configuration contains the selected D&D policy and remains
+  valid through Audita's native validator.
+- The same suite-owned `audita-v1` schema is supplied to Audita's managed
+  configuration and Narratio's Audita adapter; Narratio supplies Audita's
+  run-specific output path rather than relying on a second fixed path.
+- `profiles/testing.yml` is a valid, independently selectable Narratio profile
+  that selects the configured inexpensive Scriptorium profile for every
+  artifact and family and inherits base workflow structure.
+- The testing profile does not repeat Audita configuration while production
+  and testing deliberately share the same Audita endpoint and models.
+- Base and testing configuration can be compared with
+  `narratio config diff default testing` without requiring a model call.
+- All suite-managed configuration remains readable by the `narratio` group and
+  is not made writable by runtime or human operator accounts.
+- An immediate repeated role application with unchanged inputs reports no
+  changes.
+- The role contains no configuration probes, production self-validation,
+  fragment manifests, duplicate root template, secrets, Notarius integration,
+  storage, or publishing behavior.
+
+### Required Validation
+
+- YAML syntax checking and production-profile ansible-lint pass for the role
+  and test playbooks.
+- The complete existing pipeline integration entry point continues to install
+  the pinned role combination and validate asset discovery.
+- Audita natively validates the role-managed D&D configuration.
+- Narratio natively validates both base and `testing` configuration with an
+  explicitly selected deployed campaign.
+- Narratio's native artifact inspection demonstrates one enabled ordinary
+  artifact, one inferred dependency, and representative family expansion.
+- A test explicitly supplies a representative testing Scriptorium profile and
+  observes its effective selection; it does not assert that the same value
+  remains the role's current default.
+- Scriptorium can resolve the representative production and testing profiles
+  through its native inspection interface without making a provider request.
+- The immediate repeat reports `changed=0` and `failed=0`.
+- Tests do not snapshot full rendered files, enumerate every character or
+  artifact, inspect template/file task results, test filesystem metadata, or
+  reproduce ConfigKit, Narratio, Audita, Scriptorium, or Promptkit validation.
+
+## Dependencies and Follow-up Work
+
+Features 0001 through 0007 and the standalone-role test-suite simplification
+are complete. Feature 0008 depends on the current Narratio 2.0 ConfigKit
+contract, the canonical party files deployed by Feature 0007, and prompt
+definitions whose production default profiles are available from Promptkit.
+
+Feature 0009 should adapt the validated D&D Notarius pipeline, remove its
+hard-coded campaign references in favor of Narratio invocation bindings,
+declare its runtime-directory policy, map its accepted outputs into Narratio,
+and enable extraction. Later features should address storage/publishing and
+secret delivery together, operator access, collection assembly, and an
+end-to-end workflow that can use actual external services.
+
+## Potential Future Hardening
+
+- Use ConfigKit source inspection in operational diagnostics to show which
+  managed fragment supplied an effective value.
+- Add a controlled cleanup policy for obsolete suite fragments if accumulated
+  inert files become an operational problem.