From 24238d249e269e2392fc6aff569650365469c79b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 25 Jul 2026 18:59:57 +0000 Subject: [PATCH] Plan scene-aware combat extraction --- docs/roadmap/implementation.md | 412 ++++++++++++++++++ docs/roadmap/scene-aware-combat-extraction.md | 189 ++++++++ 2 files changed, 601 insertions(+) create mode 100644 docs/roadmap/implementation.md create mode 100644 docs/roadmap/scene-aware-combat-extraction.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..803e7f7 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,412 @@ +# Scene-Aware Combat Extraction Implementation Plan + +## Status + +Ready for implementation. Execute each stage in order, with one implementation +prompt per stage. + +## Feature Contract + +The target behavior and policy choices are defined in +[Scene-Aware Combat Extraction](scene-aware-combat-extraction.md). That document +is authoritative when this plan and the feature contract appear to overlap. + +The central rule is strict opt-in: `dnd/combat-turns` may call the LLM only for +an accepted chunk with one exact scene-description record whose `kind` is +`combat`. Exact non-combat matches and missing or mismatched coverage produce an +accepted empty combat-turn list without an LLM call. + +## Required Reading + +Before Stage 1, read: + +- `docs/development.md`; +- every document under `docs/policy/`; +- `docs/roadmap/scene-aware-combat-extraction.md`; +- `docs/internal/pipeline.md`; +- `docs/internal/modules.md`; +- `docs/integrations/dnd-scene-description-artifacts.md`; +- `docs/integrations/dnd-combat-turn-artifacts.md`; and +- the focused code and tests named in the applicable stage. + +Before changing a subsystem in a later stage, reread its focused current +documentation and tests if they have changed since the earlier stage. + +## Global Implementation Constraints + +- Keep scene interpretation under `internal/modules/dnd`; generic framework and + source packages must not import D&D types or interpret scene kinds. +- Use the existing ordered generated-reference mechanism. Do not add implicit + lane discovery, a new dependency syntax, a general DAG, or direct output-file + reads. +- Add `scene_descriptions` only to the combat extractor. The combat normalizer + continues to accept only the optional `npcs` registry. +- Keep both durable v1 schemas unchanged. +- Do not add the scene artifact or a derived classification to the combat + prompt. Do not change combat prompt assets, prompt ordering, prompt inputs, + response schema, or prompt fingerprint. +- Preserve existing combat extraction behavior for eligible combat chunks, + including NPC grounding, retries, validators, candidate mapping, and + normalization. +- Treat the scene artifact as control context, never as combat evidence. Never + copy its references, title, or summary into combat turns. +- Use deterministic, offline tests and existing LLM fakes. Do not call a live + provider or turn probabilistic model output into a correctness gate. +- Test stable behavior at the narrowest owning boundary. Do not add tests for + private helper shape, cache implementation details, or exact diagnostic prose. +- Do not add a framework abstraction unless implementation proves the existing + reference contracts cannot express this feature. No such framework gap is + currently expected. + +## Stage 1 — Add the Domain Scene-Eligibility Reference Boundary + +### Goal + +Create one domain-owned, immutable boundary that decodes an approved +scene-description reference, retains only gating data, and supports both +construction-time external references and operation-time generated references. +This stage must not change combat extraction behavior. + +### Implementation + +1. Add `internal/modules/dnd/scenedescriptions/registry`, following the useful + external/generated resolution pattern in `internal/modules/dnd/npcs/registry` + without copying NPC-specific projection or identity policy. +2. Define these package contracts: + - reference slot: `scene_descriptions`; + - maximum item size: 1 MiB (`1048576` bytes); + - accepted representation: exactly one `application/json` item decoded by + `codec/scenedescriptions`; + - eligibility projection: scene ID, exact `source_ref`, and `kind` only. +3. Decode through the approved scene-description codec. Reject: + - more than one item; + - invalid or non-JSON media types; + - oversized content; + - invalid approved scene JSON; and + - duplicate scene IDs, even if duplicate records are otherwise identical. +4. Do not retain titles, summaries, original bytes, paths, or prompt material in + the resolved eligibility view. Framework reference provenance remains the + owner of raw reference identity. +5. Build an immutable lookup keyed by exact scene ID. Expose a chunk-match result + with three stable states: + - exact: ID, source ID, start unit ID, and end unit ID all match; + - missing: no record has the current chunk ID; + - mismatched: the ID exists but any source-range field differs. + Only the exact state exposes the scene kind. +6. Compute a semantic eligibility digest from a canonical projection sorted by + scene ID. The digest must: + - ignore title, summary, and input array order; and + - change when an ID, source-range field, or kind changes. + Represent an unbound view with the canonical empty projection + `{"scenes":[]}` and its `sha256:` digest so generated-reference construction + has a stable sentinel without pretending the required reference was bound. +7. Provide a resolver that: + - seeds and validates an external materialized item during construction; + - permits an unbound seed because a generated required reference is not + populated until the operation-time handoff; + - resolves an operation-time generated item when present; + - otherwise returns the seeded view; and + - safely reuses resolved immutable views across concurrent extract jobs. +8. Expose only the accessors required by combat extraction: bound state, record + count, eligibility digest, and exact chunk matching. Return defensive values + rather than mutable maps or scene records. + +### Tests + +Add focused package tests that own: + +- approved decoding and exact match classification; +- missing and mismatched match states; +- strict item count, media type, size, codec, and duplicate-ID failures; +- semantic digest stability across prose and ordering changes; +- semantic digest changes for each gating field; +- construction-time seed versus operation-time generated override; and +- immutability and concurrent-safe reuse at the public package boundary where + practical. + +Do not duplicate the scene codec's exhaustive JSON-shape tests. + +### Validation + +Run: + +```sh +go test ./internal/modules/dnd/scenedescriptions/registry +go test ./internal/modules/dnd/codec/scenedescriptions +git diff --check +``` + +Stage 1 is complete when the new package is independently usable and tested but +no production module selects it yet. + +## Stage 2 — Enforce the Required Combat Scene Gate + +### Goal + +Make `dnd/combat-turns` require the scene-description artifact and invoke its +existing LLM path only for exact combat scenes. Update directly affected +configuration fixtures and contract tests in the same stage so the repository +does not retain knowingly invalid combat configurations. + +### Module Contract + +1. In `internal/modules/dnd/extract/combatturns`, add aliases for the + scene-registry slot and size limit and declare a reference slot with: + - name `scene_descriptions`; + - `Required: true`; + - media type `application/json`; + - accepted artifact kind `dnd/scene-description-list`; + - maximum size 1 MiB; and + - a description stating that it gates combat eligibility and is not combat + evidence. +2. Preserve sorted reference-slot ordering and the existing agreement between + `ReferenceSlots()` and `ModuleSpec().ReferenceSlots`. +3. Add a scene resolver to the immutable extractor. Constructor behavior must: + - validate a materialized external scene artifact before execution; + - allow the construction reference set to lack an item for a configured + generated binding; and + - retain no scene prose. +4. Define a stable gating policy identity + `dnd.combat_turns.scene_gate.v1`. +5. Extend manifest metadata with `scene_gate_policy`. For a seeded external + reference only, also report `scene_eligibility_digest` and + `scene_description_count`. Do not publish those operation-time values for a + generated reference in singleton component metadata; framework provenance + owns that handoff. +6. Extend prepared-component checkpoint fingerprints with: + - `scene_gate_policy`; and + - `scene_eligibility`, using the seeded external eligibility digest or the + registry's stable unbound sentinel for a generated binding. + Keep the existing prompt, response-schema, mapping-policy, and NPC-registry + fingerprints unchanged. + +### Extraction Behavior + +In `Extractor.Extract`, retain common request validation through +`shared.PrepareChunkExtraction`, then resolve and apply the scene gate before +building prompt inputs or resolving NPC prompt grounding: + +1. If the operation-time scene reference is absent, return a contextual module + error stating that the required reference is missing. Normal configured runs + should have failed earlier during resolution or required-producer handoff; + this check protects direct module use. +2. For an exact `combat` match, continue through the existing NPC resolution, + prompt construction, LLM request, response mapping, and validation path + without semantic changes. +3. For an exact `narrative`, `recap`, or `meta` match, return + `dnd.CombatTurnList{CombatTurns: []dnd.CombatTurn{}}`. Make no LLM call and + emit no warning. +4. For a missing or mismatched match, return the same non-nil empty list without + an LLM call and attach exactly one content-safe warning: + - scope: `scene_descriptions`; + - reason code: `scene_classification_unavailable`; + - message: explain that no exact classification was available and combat + extraction was skipped, without scene prose or transcript content. +5. The deterministic result is an accepted extract result, not an error or + rejection. It must pass the existing combat shape, source-reference, schema, + merge, and normalization boundaries and must not consume retry attempts. +6. Do not pass `scene_descriptions` into `shared.PromptInputs` or add it to the + structured completion request. The current shared helper already selects + only campaign and transcript inputs; preserve that boundary explicitly. + +### Required Test And Fixture Migration + +1. Update combat extractor test helpers so tests of the existing LLM behavior + supply an exact combat scene artifact for their current chunk. Keep + NPC-specific test intent unchanged. +2. Add a compact table-driven extractor test covering: + - exact combat; + - exact narrative, recap, and meta; + - missing scene ID; + - matching ID with each mismatched range identity; and + - an operation-time missing required reference. + Assert typed output, warning reason where applicable, and whether the fake LLM + was called. Do not assert private helper calls. +3. Update module specification, registration, metadata, and fingerprint tests + for the required slot and new policy identities. Assert that changing only + scene prose does not change the eligibility fingerprint, while changing + kind or range does. +4. Update every in-repository pipeline or programmatic test configuration that + selects `dnd/combat-turns`: + - the maintained complete example must bind the earlier + `scene-descriptions` lane to the later `scene_descriptions` step reference; + - CLI combat contract configurations must bind a compatible external scene + artifact when testing a standalone one-step profile; + - the ordered D&D integration fixture must produce scene descriptions in its + first step and bind them in its consumer step; and + - fake clients used by those integration tests must return an exact combat + scene where the test expects the combat LLM path. +5. Add or update a configuration-resolution assertion proving that a selected + combat extractor without `scene_descriptions` is rejected as an unbound + required slot. +6. Update existing expected lane counts, schemas, request sets, fingerprints, + and provenance counts only where the new scene producer or consumer + materially changes the contract. Preserve the original NPC-grounding + assertions. + +### Validation + +Run: + +```sh +go run ./cmd/notarius config validate \ + --config examples/dnd-complete.config.yml \ + --pipeline dnd-session +go test ./internal/modules/dnd/... +go test ./internal/modules/integration +go test ./internal/cli +git diff --check +``` + +Stage 2 is complete when required-slot resolution is enforced, every combat +extract path obeys the eligibility rule, all affected configurations are valid, +and focused suites pass. + +## Stage 3 — Prove Ordered Handoff, Provenance, And Retry Behavior + +### Goal + +Protect the cross-component behavior that cannot be established by the +scene-registry and extractor package tests alone. Keep this stage lean: reuse +the real codecs, resolver, pipeline, and validators, with a deterministic fake +only at the LLM boundary. + +### Integration Coverage + +1. Extend or refactor the focused ordered D&D integration test so it proves, in + one representative assembled workflow: + - scene descriptions and NPCs normalize in the first step; + - their accepted artifacts are supplied to the intended second-step targets; + - an exact combat scene causes exactly one combat completion request; + - the combat request contains transcript, campaign, and NPC inputs as + configured but no `scene_descriptions` input; + - the combat output and existing NPC canonicalization remain correct; + - the manifest records one generated scene-description reference for the + combat extract target with the expected kind, schema, media type, digest, + and producer identity; and + - manifest and component metadata contain no scene title, summary, transcript + text, or generated artifact payload. +2. Add one focused non-combat ordered-run case using an exact `narrative` + classification. Configure combat retries to a value greater than zero, then + prove: + - no combat completion request occurs; + - one accepted empty combat lane reaches normalized output; + - no retry or rejection is recorded for the deterministic skip; and + - no unavailable-classification warning is emitted for an exact non-combat + match. +3. Add one focused missing-or-mismatched case at the narrowest stable boundary + not already covered by Stage 2. Prove the accepted empty result and + `scene_classification_unavailable` warning reach the runner's durable warning + collection without a combat LLM call. Do not repeat every mismatch variant + at integration scope. + +### Checkpoint And Identity Coverage + +1. Confirm the existing framework-generated-reference dependency fingerprint + already includes the canonical generated scene artifact. Reuse that + mechanism; do not add a second framework fingerprint path. +2. Add focused D&D coverage proving: + - a seeded external reference's component fingerprint changes when eligibility + kind or exact range changes; + - title-only or summary-only changes leave that semantic component + fingerprint unchanged; and + - the generated reference appears in the combat extract checkpoint + dependencies, so changing the generated artifact prevents reuse. +3. Prefer an existing checkpoint identity or prepared-pipeline test boundary. + Do not add a full CLI resume test if the same dependency invalidation is + already credibly protected by the framework checkpoint tests plus the D&D + prepared-pipeline assertion. + +### Validation + +Run: + +```sh +go test ./internal/modules/integration +go test ./internal/framework/pipeline +go test ./internal/cli +go test ./internal/modules/dnd/extract/combatturns +go test ./internal/modules/dnd/scenedescriptions/registry +git diff --check +``` + +Stage 3 is complete when the ordered generated handoff, no-call behavior, +warning propagation, provenance privacy, and checkpoint invalidation are +protected at their stable owning boundaries without redundant end-to-end cases. + +## Stage 4 — Update Canonical Documentation And Finish Validation + +### Goal + +Move the implemented behavior out of future-only documentation and into each +canonical current-behavior owner, then perform repository-wide verification. + +### Documentation + +1. Update `docs/config.md` to own: + - the combat extractor's required `scene_descriptions` slot; + - accepted artifact kind, JSON media type, single-item and 1 MiB limits; + - configuration-time rejection when the slot is unbound; and + - the explicit generated binding used by the complete example. + Update illustrative combat configuration fragments so none show an invalid + unbound combat extractor. +2. Update `docs/operations.md` to explain that the first step produces NPC and + scene-description artifacts and the second step runs combat extraction only + for exact combat scenes. +3. Update `docs/integrations/dnd-combat-turn-artifacts.md` to own the externally + observable eligibility, empty-result, warning, retry, prompt-exclusion, and + provenance behavior. +4. Add a concise downstream-consumer link or summary to + `docs/integrations/dnd-scene-description-artifacts.md` without duplicating + the combat contract. +5. Update `docs/internal/modules.md` with the scene eligibility registry, + construction-time versus operation-time resolution, exact matching, + fingerprint, and content-retention boundaries. Keep generic ordered-handoff + mechanics canonical in `docs/internal/pipeline.md`; update that document only + if implementation changed or clarified a generic mechanism. +6. Update `docs/internal/overview.md` only if its component inventory requires + the new domain package. +7. Remove the completed combat-gating bullet from + `docs/roadmap/future.md`. Preserve the remaining scene-chunking evaluation + and ordered-pipeline guidance. +8. Review `docs/roadmap/scene-aware-combat-extraction.md` for consistency with + the final implementation. Keep it as feature history until a later explicit + retirement request; do not duplicate its implementation sequence there. +9. Verify that README and CLI documentation need no changes. Update them only + if an existing command or linked example became inaccurate. + +### Final Verification + +Run all repository checks: + +```sh +go run ./cmd/notarius config validate \ + --config examples/dnd-minimal.config.yml \ + --pipeline dnd-session +go run ./cmd/notarius config validate \ + --config examples/dnd-complete.config.yml \ + --pipeline dnd-session +go test ./... +go vet ./... +go build ./cmd/notarius +git diff --check +``` + +Also verify: + +- only the two maintained D&D configuration examples remain; +- every current documentation link points to an existing file; +- no current-behavior document describes combat extraction without the required + scene gate; +- no prompt asset or combat response schema changed; and +- no generated build artifact is left in the working tree. + +Stage 4 is complete when documentation ownership is correct, future work no +longer lists the implemented item, both examples validate, and all repository +checks pass. + +## Open Questions + +None. The feature contract and this plan resolve the required behavior, +reference ownership, matching semantics, failure policy, warning policy, +checkpoint identity, documentation ownership, and test boundaries. diff --git a/docs/roadmap/scene-aware-combat-extraction.md b/docs/roadmap/scene-aware-combat-extraction.md new file mode 100644 index 0000000..f9880ac --- /dev/null +++ b/docs/roadmap/scene-aware-combat-extraction.md @@ -0,0 +1,189 @@ +# Scene-Aware Combat Extraction + +## Status + +Proposed near-term scope. + +## Purpose + +Use an accepted D&D scene-description artifact as the eligibility boundary for +combat-turn extraction. The combat LLM should run only for a chunk whose exact +scene record has `kind: combat`; every other chunk should be ignored by the +combat extractor. This work should connect the existing scene-description lane, +ordered generated-reference handoff, and combat-turn lane without expanding the +minimal combat-turn artifact contract or introducing D&D policy into generic +pipeline code. + +## Target Workflow + +The complete D&D pipeline has two ordered steps: + +1. The first step extracts and normalizes NPCs and scene descriptions for the + accepted scene chunks. +2. At the step barrier, the accepted NPC and scene-description artifacts become + generated references. +3. The second step supplies the NPC artifact to its existing consumers and the + scene-description artifact to combat-turn extraction. +4. For each chunk, combat-turn extraction performs its existing LLM extraction + only for an exact `combat` classification. Every other chunk produces a + deterministic empty artifact without an LLM call. + +The scene-description dependency must remain explicit in pipeline +configuration. The combat extractor must not discover another lane implicitly, +read output files directly, inspect chunker-private state, or require generic +chunk materialization to interpret D&D scene classifications. + +## Scene-Description Reference Contract + +The combat extractor should declare a required structured reference slot named +`scene_descriptions`. The slot accepts one approved +`dnd/scene-description-list` artifact using the existing durable scene +description schema. + +The reference may be supplied as: + +- a generated artifact from an earlier ordered step; or +- an external artifact through the existing reference-materialization + boundary. + +External artifacts must be decoded and validated before source parsing or LLM +execution. Generated artifacts must cross the existing typed step-handoff +boundary and be validated before use. A bound artifact that is malformed, +incompatible, or internally inconsistent is an error; it must not be treated as +though the slot were unbound. + +Pipeline resolution must reject combat extraction when this slot is not bound. +In an ordered same-run workflow, failure of the configured scene-description +producer to yield an accepted normalized artifact must fail the run before the +combat consumer step starts, consistent with existing required generated +reference semantics. + +The prepared reference view should be immutable and safe for concurrent +extract jobs. Its metadata and checkpoint identity should be content-safe and +must not expose scene titles, summaries, paths, or source text. + +## Chunk Matching And Gating Policy + +A scene record authorizes combat extraction for the current chunk only when all +of the following are true: + +- the scene ID exactly equals the current accepted chunk ID; +- the scene source ID exactly equals the chunk source ID; +- the scene start and end unit IDs exactly equal the chunk's inclusive source + range; and +- the approved artifact contains exactly one such record. + +The normalized scene-description contract already rejects conflicting IDs and +ranges. The combat extractor must nevertheless require the exact match above at +its own decision boundary rather than relying on array position, range overlap, +title, summary, or inferred chronology. + +An exact match with `kind: combat` performs the existing combat-turn LLM +extraction. An exact match with `narrative`, `recap`, or `meta` returns a typed +`dnd/combat-turn-list` containing an empty `combat_turns` array without making +an LLM call. + +A valid artifact with no exact match, incomplete coverage, or a chunk identity +or range mismatch also returns the deterministic empty artifact. It must not +infer combat eligibility from an overlapping or adjacent scene. Emit a bounded, +content-safe warning for missing or mismatched coverage so operators can +distinguish an intentional non-combat classification from an unavailable exact +classification. The warning must not include scene prose or transcript text. + +This policy relies on the scene contract's mixed-scene precedence: any chunk in +which combat is a substantive central activity is classified as `combat`. +Scene-aware gating must not add a second classification policy. + +## Extraction, Validation, And Provenance + +The deterministic empty result follows the same typed extractor and validator +boundaries as an LLM-produced empty result. It is not a rejection, does not +consume retry budget, and continues through merge and normalization normally. +The durable combat-turn schema remains unchanged. + +Existing combat extraction behavior—including prompt assets, NPC grounding, +candidate mapping, validators, retries, warnings, and normalization—remains +unchanged for chunks classified as combat. The scene-description artifact is +control context only: + +- it must not be added to the combat prompt; +- its title or summary must not become combat evidence; +- its source references must not be copied into combat turns; and +- it must not create, repair, or classify a combat turn. + +Generated-reference provenance and dependency fingerprints should cover the +scene artifact through the existing ordered-handoff machinery. External +references should contribute their existing materialization provenance plus a +component-local semantic fingerprint sufficient to invalidate combat extract +checkpoints when a classification or chunk identity changes. Checkpoint reuse +must never preserve a skipped result after the effective scene classification +changes. + +Run manifests may report bounded module metadata such as the number of approved +scene records. They must not contain scene prose or duplicate the referenced +artifact payload. + +## Configuration And Documentation + +The maintained complete D&D example should bind the normalized +`scene-descriptions` lane from the first step to the `scene_descriptions` slot +in the second step. The minimal example should remain unchanged. + +When implemented, current-behavior documentation should be updated in its +canonical locations: + +- Configuration owns the new selectable reference slot and binding example. +- Operations owns the ordered scene-aware workflow. +- The combat-turn integration contract owns externally observable extraction, + empty-result, and provenance behavior. +- Internal pipeline and module documentation own preparation, handoff, matching, + and checkpoint mechanics. +- `future.md` should remove the completed scene-aware combat item. + +## Quality Expectations + +Tests should protect behavior and architectural boundaries rather than internal +helper shape. Coverage should demonstrate: + +- exact non-combat matches produce accepted empty combat artifacts without an + LLM call; +- exact combat matches retain the existing LLM path; +- an unbound required reference is rejected during pipeline resolution; +- missing and mismatched chunk coverage produces an accepted empty combat + artifact without an LLM call and emits a bounded warning; +- malformed external and generated artifacts fail at the appropriate + preparation or handoff boundary; +- retries are neither consumed nor attempted for deterministic skips; +- scene artifact changes invalidate relevant checkpoint reuse; +- no scene prose is exposed through combat prompts, warnings, metadata, or + manifests; and +- the complete maintained configuration resolves and materializes the intended + ordered dependency. + +Model-output fixtures should remain deterministic test doubles. This feature +does not require live-provider tests or assertions over probabilistic model +quality. + +## Non-Goals + +This scope does not: + +- change the durable scene-description or combat-turn schemas; +- add scene fields to combat-turn artifacts; +- annotate generic chunks with D&D classifications; +- make the combat lane depend implicitly on the scene-description lane; +- skip spell, NPC, NPC-interaction, or scene-description extraction; +- infer combat from scene titles, summaries, overlap, or campaign references; +- introduce arbitrary DAG scheduling or concurrent cross-step execution; +- add prior-run artifact discovery or new reference syntax; or +- add an LLM-backed validator, repair pass, or generic deduplication stage. + +## Completion Criteria + +The scope is complete when an explicitly configured ordered D&D pipeline can +hand an approved scene-description artifact to combat extraction, exact +`combat` scene matches are the only chunks that invoke the combat LLM, every +other chunk produces a deterministic empty result, provenance and checkpoint +identity remain correct, the complete example demonstrates the workflow, and +the canonical current-behavior documentation reflects the implemented +contract.