diff --git a/docs/roadmap/dnd-scene-descriptions.md b/docs/roadmap/dnd-scene-descriptions.md deleted file mode 100644 index acc5c4e..0000000 --- a/docs/roadmap/dnd-scene-descriptions.md +++ /dev/null @@ -1,261 +0,0 @@ -# D&D Scene Descriptions - -Status: Implemented - -The durable output contract is [D&D Scene Description Artifact Contract](../integrations/dnd-scene-description-artifacts.md). - -## Purpose - -Add a focused D&D artifact that describes each accepted scene with a bounded -classification, a concise title, and a concise summary. This is the explicit -home for the small amount of source-bounded synthesis useful to downstream -applications; scene chunking should not remain the long-term owner of those -descriptive fields. - -The feature follows -[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md). -It is intentionally richer than a fact-only occurrence extractor, but every -model-produced field has a specific scene-description purpose and is grounded -by the exact accepted chunk range. - -## Desired End State - -A D&D pipeline that uses accepted scene chunks can add a -`dnd/scene-descriptions` artifact lane. The extractor runs once per accepted -chunk and returns exactly one description for that chunk. The merged, -normalized artifact is an ordered list with one independently identified and -evidenced scene record for every successful chunk extraction. - -Use these production identities: - -| Concern | Identity | -| --- | --- | -| Extractor key | `dnd/scene-descriptions` | -| Normalizer key | `dnd/scene-descriptions` | -| Artifact kind | `dnd/scene-description-list` | -| Durable schema ID | `notarius.dnd.scene_descriptions` | -| Durable schema name | `notarius_dnd_scene_descriptions_v1` | -| Durable schema version | `v1` | -| Durable media type | `application/json` | -| Private prompt ID | `dnd.scene_descriptions` | -| Private response schema ID | `notarius.dnd.scene_descriptions.llm` | -| Private response schema version | `v1` | - -The lane uses the existing typed artifact, append-order merge, codec, -registration, validator-chain, prompt-asset, retry, and output conventions. It -does not introduce a new pipeline stage or a dependency on the durable -chunk-map export. - -## Durable Artifact Contract - -The durable payload is a strict object containing a `scenes` array. The array -is ordered by accepted source position. Each scene contains exactly: - -- `id`: the accepted chunk ID, copied deterministically; -- `source_ref`: the accepted chunk's exact inclusive current-source range; -- `kind`: one value from the closed scene-kind vocabulary; -- `title`: a concise source-grounded label for the scene; and -- `summary`: a concise source-grounded account of the scene's main activity. - -The accepted chunk ID is the scene ID. Do not add a second `chunk_id` field or -generate an unrelated scene identifier: this artifact has exactly one record -per accepted scene chunk, so the chunk identity is already the stable identity -needed for joining with chunk maps and other range-grounded artifacts. - -Every fixed object rejects unknown fields. `id` and `source_ref` are -application-owned evidence and identity fields. The model must not emit or -reproduce chunk IDs, source IDs, unit IDs, or ranges. - -The extractor maps one successful private response into a single-element -durable `scenes` array before extraction validation. It attaches the current -chunk ID and range at that boundary rather than relying on normalization to -manufacture required provenance. - -## Model-Owned Fields - -The private model response contains exactly: - -- `kind`; -- `title`; and -- `summary`. - -It represents one scene object, not a list. An accepted chunk is already the -scene boundary, so the extractor must neither split the chunk nor return -multiple descriptions. It must not return an empty result; inability to -classify or describe the chunk is an extraction failure subject to the -configured retry and rejection policy. - -`title` should be a short, distinguishing phrase rather than a sentence, -chapter number, or generic label such as “Scene.” It may use names and places -established by the transcript or optional campaign references, but must not -invent a proper noun to make the title more colorful. - -`summary` should briefly state the main activity and material transition or -outcome established within the chunk. It must not add analysis, inferred -motives, hidden state, future consequences, relationship claims, or facts from -outside the chunk. Do not require a structured participant list, encounter -outcome, location, mood, or confidence field. - -## Scene-Kind Vocabulary - -Use this closed vocabulary: - -| Kind | Meaning | -| --- | --- | -| `combat` | Active combat materially organizes the scene, including initiative-like exchanges or sustained hostile action. Planning a fight or discussing a completed fight is not combat by itself. | -| `narrative` | Current-session in-world play that is not principally active combat, a prior-session recap, or sustained out-of-character session discussion. This includes exploration, travel, dialogue, investigation, planning in character, and aftermath. | -| `recap` | The scene's organizing purpose is to recount events from a previous session for the table. An in-world character recounting history during current play remains narrative. | -| `meta` | The scene's organizing purpose is sustained out-of-character discussion about the game or session rather than advancing current in-world play. | - -`narrative` is the default category for actual current-session gameplay that -does not meet another definition. The model must choose exactly one kind and -must not emit an uncertainty or mixed category. - -## Residual Mixed-Scene Policy - -Scene chunking should normally separate sustained transitions between kinds. -When an accepted chunk nevertheless contains more than one kind, classify the -activity that best describes the chunk's organizing purpose and main -current-session progression: - -1. use `combat` when active combat is a substantive central activity, even if - the chunk includes brief setup, rules clarification, or immediate aftermath; -2. use `recap` when recounting a previous session is the chunk's primary table - purpose, not merely a brief reminder before current play; -3. use `meta` when sustained out-of-character session discussion is primary and - the chunk contains no more than incidental in-world progression; and -4. use `narrative` for all remaining current-session in-world play. - -Brief table talk, dice resolution, rules clarification, jokes, or administrative -comments are incidental to the enclosing gameplay scene and do not make it -`meta`. A short recollection used to orient current action does not make a scene -`recap`. - -This precedence resolves residual ambiguity for description. It does not alter -the accepted chunk boundary or authorize the extractor to conceal a -demonstrably poor scene plan. - -## Evidence, References, And Prompt Boundary - -The exact accepted chunk range is the evidence for all three model-owned -fields. Do not ask the model for finer-grained source ranges, and do not use -campaign references as evidence. - -The lane requires no generated artifact reference. Existing optional D&D -campaign references may be exposed through the established shared slots when -they help disambiguate names or setting terms in titles and summaries. They -must not introduce events, participants, or claims absent from the current -chunk. Do not require an NPC registry: structured participants remain owned by -NPC and NPC-interaction artifacts. - -Follow the established D&D prompt ordering and prompt-cache boundary: - -1. shared D&D system and identity policy; -2. stable campaign references, including explicit empty placeholders; -3. lane task and classification instructions; and -4. the variable current-chunk transcript last. - -Factor wording genuinely shared with other D&D modules into existing shared -prompt assets. Keep scene-kind definitions and mixed-scene policy in -scene-description-owned assets rather than broadening unrelated extractors. -Do not render the existing shared extraction-evidence asset in this lane: that -asset instructs the model to emit source ranges, while scene identity and the -entire accepted chunk range are attached deterministically by the application. -This is a deliberate exception to the shared extraction prompt sequence, not -permission to duplicate compatible shared wording in lane-owned assets. - -## Merge And Normalization - -Use the existing append-order merge behavior. Normalization remains -deterministic and must: - -- validate every record against the current source document; -- require an exact closed-vocabulary kind without case-folding or repairing an - invalid model value; -- trim surrounding title and summary whitespace without otherwise rewriting - their prose; -- order scenes by source-document position, with accepted chunk ID as the - deterministic tie-breaker; -- remove only completely identical duplicate records; and -- reject conflicting records that reuse one scene ID or exact source range with - different model-owned content. - -Normalization must not combine adjacent scenes, regenerate titles or summaries, -change classifications through heuristics, or infer missing records. Exact -deduplication is defensive cleanup, not semantic reconciliation. - -## Validation And Quality - -Provide the same layered production validation used by other typed D&D -artifacts: - -- valid JSON and strict private/durable shape; -- exactly one private description and one mapped durable scene per chunk; -- closed kind vocabulary; -- non-empty, trimmed title and summary; -- non-empty accepted chunk ID; -- a current-source range exactly equal to the extraction chunk range; -- source-relatedness checks for title and summary; and -- strict durable JSON Schema validation after normalization. - -Shape, identity, enum, and source-range violations are deterministic -rejections. Source-relatedness should initially remain warning-only unless -evaluation establishes a reliable rejection rule. Diagnostics must be bounded -and must not copy transcript or campaign-reference content. - -Evaluate the lane on a small human-reviewed transcript set covering all four -kinds, incidental table talk, brief and sustained recap, combat setup and -aftermath, mixed residual chunks, sparse scenes, proper-noun ambiguity, and -unsupported synthesis. Review classification agreement, title usefulness, -summary fidelity, evidence attachment, and performance on the cost-effective -models Notarius is intended to support. Human evaluation is a development aid, -not an exact-text CI oracle. - -Tests should protect structural contracts, deterministic mapping, -normalization, validation, prompt assembly, registration, and one assembled -production workflow. Do not assert exact generated prose or use brittle prompt -length and shared-prefix change detectors. - -## Relationship To Scene Chunking - -This artifact becomes the intended owner of scene kind, title, and summary. -The existing `dnd/scenes` chunker remains unchanged in this scope. Downstream -consumers should prefer the normalized scene-description artifact over chunk -annotations for kind, title, and summary. - -Removing descriptive fields from the chunker is a separate follow-on scope. It -requires its own review of chunker prompt quality, warning/validator consumers, -cache behavior, and the compatibility implications of changing accepted plan -annotations. Do not couple that migration to the initial artifact lane. - -Likewise, do not make combat extraction conditional on scene kind in this -scope. A later ordered-pipeline feature may consume the accepted normalized -scene-description artifact and no-op combat extraction for non-combat scenes -without making generic chunk materialization depend on D&D classification. - -## Documentation Ownership - -Once implemented, the durable contract belongs in `docs/integrations/`; -selectable keys, optional campaign-reference bindings, and default validator -chains belong in configuration documentation; and prompt, module, validator, -and pipeline mechanics belong in current internal documentation. A maintained -D&D example should demonstrate scene chunking and scene-description extraction -without presenting chunk annotations as the durable description contract. - -## Non-Goals - -This scope does not: - -- change scene boundaries or run a second chunker; -- ask the model for IDs, source ranges, segment numbers, or evidence selection; -- add structured participants, locations, encounters, outcomes, moods, - confidence, or relationship fields; -- duplicate NPC or NPC-interaction extraction; -- require an NPC registry or another generated artifact; -- enrich descriptions from campaign references when the chunk lacks support; -- semantically merge or reconcile adjacent scene descriptions; -- minimize the existing scene chunker in the same change; -- gate combat extraction by scene kind; -- make scene descriptions generated references for later steps; -- change chunk-map export, chunk caching, or plan validation; or -- introduce a DAG, a new pipeline stage, or a new reference mechanism. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index b40d954..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,485 +0,0 @@ -# D&D Scene Descriptions Implementation Plan - -Status: Ready for implementation - -## Objective And Authority - -Implement the accepted target state in -[D&D Scene Descriptions](dnd-scene-descriptions.md): one strict, typed -`dnd/scene-description-list` artifact record for every successfully extracted -accepted scene chunk. The model owns only `kind`, `title`, and `summary`; -Notarius owns the scene ID and exact current-source range. - -This plan is the execution document. The feature roadmap owns intent, durable -policy, classification semantics, and non-goals. If wording here appears to -conflict with that roadmap, preserve the roadmap's contract and update this -plan before proceeding. - -Follow: - -- [Architecture Policy](../policy/architecture.md), especially typed domain - codecs, source-reference ownership, central registration, and the fixed - input/chunk/extract/merge/normalize/output flow; -- [Testing Policy](../policy/testing.md), especially behavioral tests at the - stable owner, offline fake-LLM coverage, and the prohibition on prompt-length - or shared-prefix change-detector tests; and -- [Documentation Policy](../policy/documentation.md), especially one canonical - owner for each contract and updating current-state docs only when behavior is - implemented. - -The repository is pre-release. Use the `v1` identities fixed below; do not add -compatibility aliases, dual schemas, migration adapters, or legacy decoding. - -## Fixed Implementation Decisions - -### Public identities - -Use these values exactly: - -| Concern | Value | -| --- | --- | -| Extractor key | `dnd/scene-descriptions` | -| Normalizer key | `dnd/scene-descriptions` | -| Artifact kind | `dnd/scene-description-list` | -| Durable schema ID | `notarius.dnd.scene_descriptions` | -| Durable schema name | `notarius_dnd_scene_descriptions_v1` | -| Durable schema version | `v1` | -| Durable media type | `application/json` | -| Prompt ID | `dnd.scene_descriptions` | -| Private response-schema key | `dnd_scene_descriptions_llm` | -| Private response-schema ID | `notarius.dnd.scene_descriptions.llm` | -| Private response-schema name | `notarius_dnd_scene_descriptions_llm_v1` | -| Private response-schema version | `v1` | -| Extractor mapping policy | `dnd.scene_descriptions.mapping.v1` | -| Normalizer policy | `dnd.scene_descriptions.normalizer.v1` | - -### Typed and JSON shapes - -Add the following canonical D&D types in `internal/modules/dnd/types.go`: - -```go -type SceneKind string - -const ( - SceneKindCombat SceneKind = "combat" - SceneKindNarrative SceneKind = "narrative" - SceneKindRecap SceneKind = "recap" - SceneKindMeta SceneKind = "meta" -) - -type SceneDescriptionList struct { - Scenes []SceneDescription `json:"scenes"` -} - -type SceneDescription struct { - ID string `json:"id"` - SourceRef source.SourceRef `json:"source_ref"` - Kind SceneKind `json:"kind"` - Title string `json:"title"` - Summary string `json:"summary"` -} -``` - -Also add -`SceneDescriptionListKind contracts.ArtifactKind = -"dnd/scene-description-list"`. - -The durable JSON object contains exactly `scenes`; every scene object contains -exactly `id`, `source_ref`, `kind`, `title`, and `summary`. The private model -response is one object containing exactly `kind`, `title`, and `summary`. It is -not wrapped in `scenes`, cannot be an array, and cannot represent “no result.” -Both schemas reject unknown fields. Both schemas enumerate all four kind -values. Durable strings and private `title` and `summary` use `minLength: 1`; -do not impose arbitrary maximum lengths in schema or deterministic validation. - -### Application-owned mapping - -For each extraction request: - -- obtain transcript material with `shared.ChunkPromptMaterial`; -- prepare optional `players`, `party`, and `glossary` inputs, including the - established explicit empty placeholders, with `shared.PromptInputs`; -- call the private structured prompt once; -- copy `req.Chunk.ID` to the durable `id`; -- copy `req.Chunk.Ref` to the durable `source_ref`; -- copy `kind` without trimming, case-folding, aliasing, or repair; and -- trim surrounding whitespace from `title` and `summary`, then return a - single-element durable `scenes` list. - -An invalid or absent model object is an extraction failure handled by existing -structured-output and retry policy. The extractor must never synthesize a -fallback classification or empty scene list. - -Include the prompt digest, response-schema digest, and mapping policy in -extractor manifest metadata and checkpoint fingerprints, following the combat -turn extractor pattern. The optional campaign references affect the prompt -input identity through existing framework behavior; do not add a generated -artifact reference or an NPC registry slot. - -### Prompt assets and caching - -Create package-owned prompt assets under -`internal/modules/dnd/extract/scenedescriptions/assets/`. Use this exact message -order: - -1. shared `common-dnd-system.md` as the system message; -2. shared `common-dnd-identity.md` as a user message and cache boundary; -3. shared `common-dnd-references.md` as a user message and cache boundary; -4. package-owned `task.md`; -5. package-owned `instructions.md` as the final stable cache boundary; and -6. shared `common-dnd-transcript.md` as the final user message, without cache - control. - -Declare `transcript`, `players`, `party`, and `glossary` inputs with the same -media types and required/optional status used by `dnd/npcs`. Reuse the shared -assets through the existing Scriptorium module/shared filesystem pattern; do -not copy their text into this package. - -Do **not** render `common-dnd-extraction-evidence.md`. Its citation instructions -require the model to emit ranges, which conflicts with the application-owned -whole-chunk evidence contract. Put the scene-kind vocabulary, residual -mixed-scene precedence, title constraints, summary constraints, and the rule -that the response describes exactly the supplied accepted chunk in the -package-owned task/instruction assets. Do not ask for IDs, ranges, source IDs, -participants, confidence, or additional fields. - -### Merge, normalization, and conflicts - -Register the existing append-order merger specialized for -`SceneDescriptionList`. Its append function must preserve nil-versus-present -slice semantics and return independently owned values, matching the other D&D -specializations. - -The normalizer is deterministic and has no reference slots. It must: - -1. reject a nil source, malformed current-source reference, blank ID, invalid - kind, or blank title/summary; -2. trim only surrounding whitespace from title and summary; -3. sort by the source document's start-unit position, then scene ID as the - deterministic tie-breaker; -4. remove only records identical in all five durable fields; -5. reject any repeated scene ID whose remaining fields are not identical; and -6. reject records with the same exact source range and different - `kind`, `title`, or `summary`. - -Two records with different IDs but the same range and identical model-owned -content are not exact duplicates and are not a conflict under this contract; -retain both in deterministic ID order. Do not merge adjacent ranges, reconcile -prose, change kinds heuristically, infer missing scenes, or use chunk -annotations. - -### Validation ownership and default chains - -Add these deterministic validators: - -| Key | Owner | -| --- | --- | -| `extract/dnd/scene-descriptions/shape` | Required list and record fields, exact scene-kind vocabulary, trimmed non-empty strings, and exactly one scene during extraction. | -| `extract/dnd/scene-descriptions/source_refs` | Current-source validity at all stages; at extraction, requires a current chunk and exact equality of both scene ID and source range to that chunk. | -| `extract/dnd/scene-descriptions/source_relatedness` | Advisory lexical grounding for title and summary against the scene's current transcript range only. | -| `normalize/dnd/scene-descriptions/invariants` | Normalized order, exact-duplicate removal, unique/conflict rules, and normalized strings. | - -The shape validator requires a present, non-empty `scenes` list at all stages -and exactly one element when `req.Stage == string(pipeline.StageExtract)`. -Stage-independent record validation rejects blank IDs, unsupported kinds, -blank or untrimmed title/summary, and structurally empty source references. - -The source-reference validator first defers malformed shape to the shape -validator. At extraction it rejects a nil `req.Chunk`, an ID unequal to -`req.Chunk.ID`, or a `source_ref` unequal to `req.Chunk.Ref`; containment is not -sufficient. At normalization, where no current chunk is available, it validates -the reference against `req.Source` but does not attempt to reconstruct chunk -identity. - -The source-relatedness validator is warning-only and runs only after shape and -source-reference validity. For each scene, tokenize the cited current-source -text, title, and summary with `shared.NormalizedTokens`. Ignore tokens shorter -than three Unicode code points and these case-normalized function words: - -`a`, `an`, `and`, `are`, `as`, `at`, `be`, `but`, `by`, `for`, `from`, `had`, -`has`, `have`, `he`, `her`, `him`, `his`, `in`, `into`, `is`, `it`, `its`, -`of`, `on`, `or`, `she`, `that`, `the`, `their`, `them`, `they`, `this`, `to`, -`was`, `were`, `with`. - -Emit one warning scoped to `scenes[i].title` if no remaining title token occurs -in the cited text, and independently one warning scoped to -`scenes[i].summary` if no remaining summary token occurs there. If a field has -no remaining significant token, emit its warning. Never use campaign -references for this check. Bound and safely quote diagnostics through the -existing D&D diagnostic helpers; do not include transcript or reference -content in messages. - -Register these exact default chains: - -```text -extract dnd/scene-descriptions: - generic/valid_json - extract/dnd/scene-descriptions/shape - extract/dnd/scene-descriptions/source_refs - generic/valid_json_schema - extract/dnd/scene-descriptions/source_relatedness - -normalize dnd/scene-descriptions: - generic/valid_json - extract/dnd/scene-descriptions/shape - normalize/dnd/scene-descriptions/invariants - extract/dnd/scene-descriptions/source_refs - generic/valid_json_schema - extract/dnd/scene-descriptions/source_relatedness -``` - -Do not add merge-stage validators, LLM-backed validators, options, generated -references, stage dependencies, chunk-map dependencies, or framework changes. - -## Stage 1: Add The Durable Typed Contract And Codec - -### Work - -1. Extend `internal/modules/dnd/types.go` with the fixed artifact kind, scene - kind, constants, list, and record types. -2. Add `internal/modules/dnd/codec/scenedescriptions/` following the strict - candidate/approved encode/decode pattern in the existing D&D codecs. -3. Embed - `assets/schemas/dnd_scene_descriptions.v1.json` with the exact durable shape - and identities above. -4. Return `scene_count` from codec metadata. -5. Add one representative valid JSON fixture under `testdata/`. - -### Tests - -Add focused codec tests covering schema identity and validity, valid -round-trip, strict unknown/trailing JSON rejection, nil `scenes`, invalid -kinds, blank required strings, malformed source fields, candidate decoding of -semantically invalid but structurally decodable values, metadata, and -independent ownership of encoded/schema bytes where the existing codec -contract requires it. - -Do not duplicate shape-validator and durable JSON Schema coverage exhaustively -in the codec suite. - -### Completion gate - -Run: - -```sh -gofmt -w internal/modules/dnd/types.go internal/modules/dnd/codec/scenedescriptions/*.go -go test ./internal/modules/dnd/codec/scenedescriptions -``` - -The repository must compile with no production registration added yet. - -## Stage 2: Add The Private Model Contract, Prompt, And Extractor - -### Work - -1. Add `internal/modules/dnd/extract/scenedescriptions/` using the established - D&D extractor package layout: assets, model, schema loader, prompt - registration/fingerprints, constructor/options handling, extractor, and - deterministic mapper. -2. Define a private Go response with exactly `kind`, `title`, and `summary`. -3. Embed - `assets/schemas/dnd_scene_descriptions_llm.v1.json` with the fixed strict - private contract. -4. Add `assets/prompts/dnd.scene_descriptions.yaml`, `task.md`, and - `instructions.md` with the fixed message order and policy above. -5. Use `shared.ReferenceSlots` with package-owned descriptions for optional - `players`, `party`, and `glossary` slots. The extractor and its module spec - expose no other slots. -6. Map the private response to exactly one durable record using the current - chunk ID and exact `Chunk.Ref`. - -### Tests - -At the package owners, cover: - -- private schema identity, strict shape, closed enum, and rejection of - application-owned or extra fields; -- prompt and response-schema asset registration and fingerprints; -- prepared message roles/order, compatible shared-asset use, explicit empty - campaign placeholders, one transcript rendering, and transcript-last - placement; -- module spec, capabilities, optional reference slots, option rejection, and - registration; -- request validation and provider failure; -- exact application-owned ID/range mapping; -- title/summary trimming and non-repair of kind; and -- prompt inputs using only current chunk material plus optional references. - -Do not assert prompt byte length, shared-prefix length, exact generated prose, -or provider output quality. - -### Completion gate - -Run: - -```sh -gofmt -w internal/modules/dnd/extract/scenedescriptions/*.go -go test ./internal/modules/dnd/extract/scenedescriptions -``` - -Do not register the extractor in the production D&D family in this stage. - -## Stage 3: Add Deterministic Normalization And Validation - -### Work - -1. Add `internal/modules/dnd/normalize/scenedescriptions/` with the fixed - normalization and conflict behavior. Expose no options or reference slots. -2. Add the four validator packages under - `internal/modules/dnd/validate/scenedescriptions/` using the exact keys and - responsibilities above: - `shape`, `source_refs`, `source_relatedness`, and `invariants`. -3. Give each normalizer/validator a checkpoint policy fingerprint that changes - when its owned deterministic behavior changes. -4. Use shared source and diagnostic helpers where their semantics match. Keep - the scene-specific lexical stopword policy local to the relatedness - validator. - -### Tests - -Cover normalization of whitespace and source ordering; ID tie-breaking; exact -deduplication; both conflict classes; preservation of same-range, -same-content, different-ID records; invalid source/shape/kind handling; input -ownership; cancellation/nil requests; and module registration metadata. - -Cover validators at their stable owners, including: - -- extraction requires exactly one scene; -- extraction ID and range must exactly equal the current chunk; -- normalization validates current-source membership without a chunk; -- shape and source-reference deferral between validators; -- supported and unsupported kinds; -- invariant order, trimming, duplicate, and conflict checks; -- separate bounded title and summary relatedness warnings; -- zero-significant-token warnings; -- transcript-only relatedness even when campaign references contain matching - text; and -- safe bounded diagnostics. - -Use table-driven cases where several inputs exercise one behavior. Do not -replicate complete codec or JSON Schema test matrices in every validator. - -### Completion gate - -Run: - -```sh -gofmt -w internal/modules/dnd/normalize/scenedescriptions/*.go internal/modules/dnd/validate/scenedescriptions/*/*.go -go test ./internal/modules/dnd/normalize/scenedescriptions ./internal/modules/dnd/validate/scenedescriptions/... -``` - -Production registration remains deferred until all leaf packages pass. - -## Stage 4: Compose The Production Lane - -### Work - -1. Update `internal/modules/dnd/register/modules.go` to register the codec, - extractor, append-order specialization, normalizer, typed no-op normalizer, - and prompt/schema assets. -2. Add the list append/clone helper in - `internal/modules/dnd/register/merge.go`. -3. Update `internal/modules/dnd/register/validators.go` to register all four - validators plus typed always-accept and always-reject specializations. -4. Update `internal/modules/dnd/register/chains.go` with the exact extract and - normalize chains above. -5. Extend `internal/modules/dnd/register/register_test.go` to assert the new - keys, artifact-kind specializations, assets, module specs, and exact - production chains. -6. Add one offline assembled production workflow test under `internal/cli/` - using the real family registration/configuration/runner/output path and a - fake structured LLM. It must exercise at least two accepted scene chunks and - prove: - - one model description is mapped to each chunk's ID and exact range; - - append merge and normalization produce source order; - - model-owned fields survive with only specified trimming; - - durable schema identity and output payload are correct; and - - no NPC or generated-artifact reference is required. - -Keep this as one representative assembled test. Do not add equivalent -end-to-end cases in multiple packages. - -### Completion gate - -Run: - -```sh -gofmt -w internal/modules/dnd/register/*.go internal/cli/dnd_scene_descriptions_contract_test.go -go test ./internal/modules/dnd/register ./internal/cli -``` - -At this gate the lane must be selectable through production registration, with -no direct production import outside the D&D family registrar and CLI -composition root. - -## Stage 5: Publish Current Documentation And Maintained Example - -### Work - -Only after Stage 4 is green: - -1. Add `docs/integrations/dnd-scene-description-artifacts.md` as the canonical - durable contract. Document identities, exact JSON shape, scene kinds, - application-owned ID/range mapping, merge/normalization rules, conflict - behavior, advisory relatedness warnings, optional campaign references, and - a copyable lane example. -2. Update `docs/config.md` with the maintained example link; extractor, - normalizer, and validator keys; optional `players`, `party`, and `glossary` - slots; and the exact default chains. -3. Update `docs/internal/modules.md` and `docs/internal/overview.md` for the new - codec/extractor/normalizer/validators and family registration. -4. Update `docs/internal/llm.md` with the prompt manifest and the deliberate - omission of the citation-oriented shared extraction-evidence message. Make - clear that compatible shared messages remain canonical shared assets and the - transcript remains last. -5. Update `docs/internal/pipeline.md` only where its current artifact-lane or - source-attachment inventory requires the new lane; do not describe a new - stage or dependency. -6. Update `docs/integrations/json-output.md` to link the new durable artifact - contract alongside the other typed D&D schemas. -7. Add `examples/dnd-scene-descriptions.config.yml`. It should use - `chunk: dnd/scenes`, one scene-description artifact lane, and - `normalize: dnd/scene-descriptions`. Keep it minimal and do not present - chunk annotations as the durable description artifact. -8. Mark `docs/roadmap/dnd-scene-descriptions.md` as `Status: Implemented`, - add a link to the canonical integration document near the top, and remove - the implemented “Extract D&D Scene Descriptions” entry from - `docs/roadmap/future.md`. Leave chunker minimization and combat gating in - future work. - -### Documentation checks - -Verify every new selectable key, schema identity, enum, validator order, and -example against production code. Check all relative Markdown links and run the -maintained example configuration through config loading or the closest -existing example/config test. Do not document private package mechanics in the -integration contract. - -## Stage 6: Final Quality Gate - -1. Review the diff for accidental changes to `dnd/scenes`, combat gating, - framework stages, generated-reference behavior, or unrelated D&D contracts. -2. Confirm all new JSON objects reject unknown fields and the model is never - asked to return application-owned identity/evidence. -3. Confirm diagnostics contain no transcript or campaign-reference content. -4. Confirm the prompt uses shared assets by reference and that the transcript - is the final message; do not introduce a prompt-length or exact-prefix - detector. -5. Run: - -```sh -gofmt -l . -go test ./... -go vet ./... -git diff --check -``` - -`gofmt -l .` must produce no output. Do not mark the feature roadmap -implemented until all commands pass and the current documentation describes -the shipped behavior. - -## Open Questions - -None. The accepted feature roadmap and the fixed decisions in this plan are -sufficient to implement all stages without further product or architecture -choices. diff --git a/internal/modules/dnd/codec/scenedescriptions/assets/schemas/dnd_scene_descriptions.v1.json b/internal/modules/dnd/codec/scenedescriptions/assets/schemas/dnd_scene_descriptions.v1.json index 2490142..43cdbd5 100644 --- a/internal/modules/dnd/codec/scenedescriptions/assets/schemas/dnd_scene_descriptions.v1.json +++ b/internal/modules/dnd/codec/scenedescriptions/assets/schemas/dnd_scene_descriptions.v1.json @@ -7,6 +7,7 @@ "properties": { "scenes": { "type": "array", + "minItems": 1, "items": { "type": "object", "additionalProperties": false, diff --git a/internal/modules/dnd/codec/scenedescriptions/codec.go b/internal/modules/dnd/codec/scenedescriptions/codec.go index 1f52fbf..de7e578 100644 --- a/internal/modules/dnd/codec/scenedescriptions/codec.go +++ b/internal/modules/dnd/codec/scenedescriptions/codec.go @@ -97,6 +97,9 @@ func validate(value dnd.SceneDescriptionList) error { if value.Scenes == nil { return fmt.Errorf("scenes must be present") } + if len(value.Scenes) == 0 { + return fmt.Errorf("scenes must not be empty") + } for index, scene := range value.Scenes { prefix := fmt.Sprintf("scenes[%d]", index) if strings.TrimSpace(scene.ID) == "" { diff --git a/internal/modules/dnd/codec/scenedescriptions/codec_test.go b/internal/modules/dnd/codec/scenedescriptions/codec_test.go index 22735ec..d0bfb66 100644 --- a/internal/modules/dnd/codec/scenedescriptions/codec_test.go +++ b/internal/modules/dnd/codec/scenedescriptions/codec_test.go @@ -8,6 +8,8 @@ import ( "strings" "testing" + "github.com/santhosh-tekuri/jsonschema/v6" + "gitea.maximumdirect.net/eric/notarius/internal/core/source" "gitea.maximumdirect.net/eric/notarius/internal/framework/contracts" "gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline" @@ -86,6 +88,40 @@ func TestCodecOwnsDurableSchemaAndMetadata(t *testing.T) { } } +func TestDurableSchemaRequiresAtLeastOneScene(t *testing.T) { + schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(New().Schema().JSONSchema)) + if err != nil { + t.Fatal(err) + } + compiler := jsonschema.NewCompiler() + if err := compiler.AddResource("schema.json", schemaDocument); err != nil { + t.Fatal(err) + } + compiled, err := compiler.Compile("schema.json") + if err != nil { + t.Fatal(err) + } + for _, test := range []struct { + name string + content []byte + valid bool + }{ + {name: "one scene", content: mustEncode(t, validList()), valid: true}, + {name: "empty scenes", content: []byte(`{"scenes":[]}`)}, + } { + t.Run(test.name, func(t *testing.T) { + instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(test.content)) + if err != nil { + t.Fatal(err) + } + err = compiled.Validate(instance) + if (err == nil) != test.valid { + t.Fatalf("Validate() error = %v, want valid=%t", err, test.valid) + } + }) + } +} + func TestCodecStrictlyRejectsInvalidDurableBoundaries(t *testing.T) { validJSON := `{"scenes":[{"id":"chunk-000001","source_ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1},"kind":"narrative","title":"Arrival","summary":"The party arrives."}]}` tests := []struct{ name, raw, want string }{ @@ -95,6 +131,7 @@ func TestCodecStrictlyRejectsInvalidDurableBoundaries(t *testing.T) { {"unknown source field", strings.Replace(validJSON, `"end_unit_id":1`, `"end_unit_id":1,"unexpected":true`, 1), "unknown field"}, {"trailing", `{"scenes":[]} {}`, "multiple JSON values"}, {"missing scenes", `{}`, "scenes must be present"}, + {"empty scenes", `{"scenes":[]}`, "scenes must not be empty"}, {"invalid kind", strings.Replace(validJSON, `"kind":"narrative"`, `"kind":"other"`, 1), "kind must be supported"}, {"blank title", strings.Replace(validJSON, `"title":"Arrival"`, `"title":" "`, 1), "title must not be empty"}, {"blank summary", strings.Replace(validJSON, `"summary":"The party arrives."`, `"summary":" "`, 1), "summary must not be empty"}, @@ -138,6 +175,7 @@ func TestCodecRejectsRequiredApprovedValues(t *testing.T) { want string }{ {"nil scenes", dnd.SceneDescriptionList{}, "scenes must be present"}, + {"empty scenes", dnd.SceneDescriptionList{Scenes: []dnd.SceneDescription{}}, "scenes must not be empty"}, {"empty ID", mutate(validList(), func(value *dnd.SceneDescriptionList) { value.Scenes[0].ID = " " }), "id must not be empty"}, {"unsupported kind", mutate(validList(), func(value *dnd.SceneDescriptionList) { value.Scenes[0].Kind = "unsupported" }), "kind must be supported"}, {"empty title", mutate(validList(), func(value *dnd.SceneDescriptionList) { value.Scenes[0].Title = " " }), "title must not be empty"}, @@ -155,6 +193,15 @@ func TestCodecRejectsRequiredApprovedValues(t *testing.T) { } } +func mustEncode(t *testing.T, value dnd.SceneDescriptionList) []byte { + t.Helper() + content, err := json.Marshal(value) + if err != nil { + t.Fatal(err) + } + return content +} + func mutate(value dnd.SceneDescriptionList, change func(*dnd.SceneDescriptionList)) dnd.SceneDescriptionList { change(&value) return value diff --git a/internal/modules/dnd/extract/scenedescriptions/assets/prompts/instructions.md b/internal/modules/dnd/extract/scenedescriptions/assets/prompts/instructions.md index 6e9efc9..d6be286 100644 --- a/internal/modules/dnd/extract/scenedescriptions/assets/prompts/instructions.md +++ b/internal/modules/dnd/extract/scenedescriptions/assets/prompts/instructions.md @@ -1,9 +1,44 @@ -Choose exactly one kind: combat, narrative, recap, or meta. When a chunk is -mixed, use this precedence: combat, then recap, then meta, then narrative. +Choose exactly one kind: -The title should identify the central event or subject. The summary should -describe only what the accepted chunk establishes. Campaign references may -disambiguate names but never add events or lore. +- combat: active combat materially organizes the scene, including + initiative-like exchanges or sustained hostile action. Planning a fight or + discussing a completed fight is not combat by itself. +- narrative: current-session in-world play that is not principally active + combat, a prior-session recap, or sustained out-of-character session + discussion. This includes exploration, travel, dialogue, investigation, + in-character planning, and aftermath. +- recap: the scene's organizing purpose is to recount events from a previous + session for the table. An in-world character recounting history during + current play remains narrative. +- meta: the scene's organizing purpose is sustained out-of-character + discussion about the game or session rather than advancing current in-world + play. + +Narrative is the default for actual current-session gameplay that does not meet +another definition. When the accepted chunk is mixed: + +1. use combat when active combat is a substantive central activity, even with + brief setup, rules clarification, or immediate aftermath; +2. otherwise use recap when recounting a previous session is the chunk's + primary table purpose; +3. otherwise use meta when sustained out-of-character session discussion is + primary and in-world progression is no more than incidental; and +4. use narrative for all remaining current-session in-world play. + +Brief table talk, dice resolution, rules clarification, jokes, or +administrative comments do not make a gameplay scene meta. A short recollection +used to orient current action does not make a scene recap. + +The title must be a short, distinguishing phrase rather than a sentence, +chapter number, or generic label such as "Scene." It may use names and places +established by the transcript or disambiguated by campaign references, but it +must not invent a proper noun. + +The summary must briefly state the main activity and material transition or +outcome established within the accepted chunk. Do not add analysis, inferred +motives, hidden state, future consequences, relationship claims, or facts from +outside the chunk. Campaign references may disambiguate names but never add +events or lore. Do not return identifiers, source identifiers, source ranges, unit identifiers, participants, confidence, or any fields besides kind, title, and summary. diff --git a/internal/modules/dnd/normalize/scenedescriptions/normalizer.go b/internal/modules/dnd/normalize/scenedescriptions/normalizer.go index 5eb5f6e..6e5a410 100644 --- a/internal/modules/dnd/normalize/scenedescriptions/normalizer.go +++ b/internal/modules/dnd/normalize/scenedescriptions/normalizer.go @@ -78,6 +78,10 @@ func normalizeList(input dnd.SceneDescriptionList, doc *source.SourceDocument) ( return dnd.SceneDescriptionList{}, fmt.Errorf("scenes must not be empty") } + unitPositions := make(map[int]int, len(doc.Units)) + for index, unit := range doc.Units { + unitPositions[unit.ID] = index + } output := dnd.SceneDescriptionList{Scenes: make([]dnd.SceneDescription, len(input.Scenes))} for index, scene := range input.Scenes { scene.Title = strings.TrimSpace(scene.Title) @@ -92,8 +96,8 @@ func normalizeList(input dnd.SceneDescriptionList, doc *source.SourceDocument) ( } sort.SliceStable(output.Scenes, func(left, right int) bool { - leftStart, _ := source.UnitIndex(doc, output.Scenes[left].SourceRef.StartUnitID) - rightStart, _ := source.UnitIndex(doc, output.Scenes[right].SourceRef.StartUnitID) + leftStart := unitPositions[output.Scenes[left].SourceRef.StartUnitID] + rightStart := unitPositions[output.Scenes[right].SourceRef.StartUnitID] if leftStart != rightStart { return leftStart < rightStart } @@ -103,6 +107,7 @@ func normalizeList(input dnd.SceneDescriptionList, doc *source.SourceDocument) ( unique := make([]dnd.SceneDescription, 0, len(output.Scenes)) byID := make(map[string]dnd.SceneDescription, len(output.Scenes)) byRange := make(map[source.SourceRef]dnd.SceneDescription, len(output.Scenes)) + seen := make(map[dnd.SceneDescription]struct{}, len(output.Scenes)) for _, scene := range output.Scenes { if previous, ok := byID[scene.ID]; ok && !identical(previous, scene) { return dnd.SceneDescriptionList{}, fmt.Errorf("scene ID %s has conflicting records", diagnostics.Quote(scene.ID)) @@ -110,11 +115,12 @@ func normalizeList(input dnd.SceneDescriptionList, doc *source.SourceDocument) ( if previous, ok := byRange[scene.SourceRef]; ok && !sameModelContent(previous, scene) { return dnd.SceneDescriptionList{}, fmt.Errorf("source range %s has conflicting records", sourceRefLabel(scene.SourceRef)) } - if containsIdentical(unique, scene) { + if _, ok := seen[scene]; ok { continue } byID[scene.ID] = scene byRange[scene.SourceRef] = scene + seen[scene] = struct{}{} unique = append(unique, scene) } output.Scenes = unique @@ -129,15 +135,6 @@ func sameModelContent(left, right dnd.SceneDescription) bool { return left.Kind == right.Kind && left.Title == right.Title && left.Summary == right.Summary } -func containsIdentical(scenes []dnd.SceneDescription, target dnd.SceneDescription) bool { - for _, scene := range scenes { - if identical(scene, target) { - return true - } - } - return false -} - func sourceRefLabel(ref source.SourceRef) string { return fmt.Sprintf("%s:%d-%d", diagnostics.Quote(ref.SourceID), ref.StartUnitID, ref.EndUnitID) } diff --git a/internal/modules/dnd/normalize/scenedescriptions/normalizer_test.go b/internal/modules/dnd/normalize/scenedescriptions/normalizer_test.go index e66c493..dd529d7 100644 --- a/internal/modules/dnd/normalize/scenedescriptions/normalizer_test.go +++ b/internal/modules/dnd/normalize/scenedescriptions/normalizer_test.go @@ -65,6 +65,24 @@ func TestNormalizeRejectsInvalidCandidatesAndConflicts(t *testing.T) { } } +func TestNormalizeUsesDocumentOrderRatherThanNumericUnitID(t *testing.T) { + doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{ + {ID: 20, Text: "The party arrives."}, + {ID: 10, Text: "The party departs."}, + }} + input := dnd.SceneDescriptionList{Scenes: []dnd.SceneDescription{ + scene("later", 10, 10, dnd.SceneKindNarrative, "Departure", "The party departs."), + scene("first", 20, 20, dnd.SceneKindNarrative, "Arrival", "The party arrives."), + }} + result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input, doc)) + if err != nil { + t.Fatalf("Normalize() error = %v", err) + } + if got, want := []string{result.Value.Scenes[0].ID, result.Value.Scenes[1].ID}, []string{"first", "later"}; !reflect.DeepEqual(got, want) { + t.Fatalf("scene order = %#v, want %#v", got, want) + } +} + func TestNormalizerContractAndCancellation(t *testing.T) { if _, err := DecodeOptions(nil); err != nil { t.Fatalf("DecodeOptions(nil) error = %v", err)