notarius/docs/roadmap/implementation.md

D&D Scene Descriptions Implementation Plan

Status: Ready for implementation

Objective And Authority

Implement the accepted target state in D&D Scene Descriptions: one strict, typed dnd/scene-description-list artifact record for every successfully extracted accepted scene chunk. The model owns only kind, title, and summary; Notarius owns the scene ID and exact current-source range.

This plan is the execution document. The feature roadmap owns intent, durable policy, classification semantics, and non-goals. If wording here appears to conflict with that roadmap, preserve the roadmap's contract and update this plan before proceeding.

Follow:

• Architecture Policy, especially typed domain codecs, source-reference ownership, central registration, and the fixed input/chunk/extract/merge/normalize/output flow;
• Testing Policy, especially behavioral tests at the stable owner, offline fake-LLM coverage, and the prohibition on prompt-length or shared-prefix change-detector tests; and
• Documentation Policy, especially one canonical owner for each contract and updating current-state docs only when behavior is implemented.

The repository is pre-release. Use the v1 identities fixed below; do not add compatibility aliases, dual schemas, migration adapters, or legacy decoding.

Fixed Implementation Decisions

Public identities

Use these values exactly:

Concern	Value
Extractor key	dnd/scene-descriptions
Normalizer key	dnd/scene-descriptions
Artifact kind	dnd/scene-description-list
Durable schema ID	notarius.dnd.scene_descriptions
Durable schema name	notarius_dnd_scene_descriptions_v1
Durable schema version	v1
Durable media type	application/json
Prompt ID	dnd.scene_descriptions
Private response-schema key	dnd_scene_descriptions_llm
Private response-schema ID	notarius.dnd.scene_descriptions.llm
Private response-schema name	notarius_dnd_scene_descriptions_llm_v1
Private response-schema version	v1
Extractor mapping policy	dnd.scene_descriptions.mapping.v1
Normalizer policy	dnd.scene_descriptions.normalizer.v1

Typed and JSON shapes

Add the following canonical D&D types in internal/modules/dnd/types.go:

type SceneKind string

const (
    SceneKindCombat    SceneKind = "combat"
    SceneKindNarrative SceneKind = "narrative"
    SceneKindRecap     SceneKind = "recap"
    SceneKindMeta      SceneKind = "meta"
)

type SceneDescriptionList struct {
    Scenes []SceneDescription `json:"scenes"`
}

type SceneDescription struct {
    ID        string           `json:"id"`
    SourceRef source.SourceRef `json:"source_ref"`
    Kind      SceneKind        `json:"kind"`
    Title     string           `json:"title"`
    Summary   string           `json:"summary"`
}

Also add SceneDescriptionListKind contracts.ArtifactKind = "dnd/scene-description-list".

The durable JSON object contains exactly scenes; every scene object contains exactly id, source_ref, kind, title, and summary. The private model response is one object containing exactly kind, title, and summary. It is not wrapped in scenes, cannot be an array, and cannot represent “no result.” Both schemas reject unknown fields. Both schemas enumerate all four kind values. Durable strings and private title and summary use minLength: 1; do not impose arbitrary maximum lengths in schema or deterministic validation.

Application-owned mapping

For each extraction request:

• obtain transcript material with shared.ChunkPromptMaterial;
• prepare optional players, party, and glossary inputs, including the established explicit empty placeholders, with shared.PromptInputs;
• call the private structured prompt once;
• copy req.Chunk.ID to the durable id;
• copy req.Chunk.Ref to the durable source_ref;
• copy kind without trimming, case-folding, aliasing, or repair; and
• trim surrounding whitespace from title and summary, then return a single-element durable scenes list.

An invalid or absent model object is an extraction failure handled by existing structured-output and retry policy. The extractor must never synthesize a fallback classification or empty scene list.

Include the prompt digest, response-schema digest, and mapping policy in extractor manifest metadata and checkpoint fingerprints, following the combat turn extractor pattern. The optional campaign references affect the prompt input identity through existing framework behavior; do not add a generated artifact reference or an NPC registry slot.

Prompt assets and caching

Create package-owned prompt assets under internal/modules/dnd/extract/scenedescriptions/assets/. Use this exact message order:

1. shared common-dnd-system.md as the system message;
2. shared common-dnd-identity.md as a user message and cache boundary;
3. shared common-dnd-references.md as a user message and cache boundary;
4. package-owned task.md;
5. package-owned instructions.md as the final stable cache boundary; and
6. shared common-dnd-transcript.md as the final user message, without cache control.

Declare transcript, players, party, and glossary inputs with the same media types and required/optional status used by dnd/npcs. Reuse the shared assets through the existing Scriptorium module/shared filesystem pattern; do not copy their text into this package.

Do not render common-dnd-extraction-evidence.md. Its citation instructions require the model to emit ranges, which conflicts with the application-owned whole-chunk evidence contract. Put the scene-kind vocabulary, residual mixed-scene precedence, title constraints, summary constraints, and the rule that the response describes exactly the supplied accepted chunk in the package-owned task/instruction assets. Do not ask for IDs, ranges, source IDs, participants, confidence, or additional fields.

Merge, normalization, and conflicts

Register the existing append-order merger specialized for SceneDescriptionList. Its append function must preserve nil-versus-present slice semantics and return independently owned values, matching the other D&D specializations.

The normalizer is deterministic and has no reference slots. It must:

1. reject a nil source, malformed current-source reference, blank ID, invalid kind, or blank title/summary;
2. trim only surrounding whitespace from title and summary;
3. sort by the source document's start-unit position, then scene ID as the deterministic tie-breaker;
4. remove only records identical in all five durable fields;
5. reject any repeated scene ID whose remaining fields are not identical; and
6. reject records with the same exact source range and different kind, title, or summary.

Two records with different IDs but the same range and identical model-owned content are not exact duplicates and are not a conflict under this contract; retain both in deterministic ID order. Do not merge adjacent ranges, reconcile prose, change kinds heuristically, infer missing scenes, or use chunk annotations.

Validation ownership and default chains

Add these deterministic validators:

Key	Owner
extract/dnd/scene-descriptions/shape	Required list and record fields, exact scene-kind vocabulary, trimmed non-empty strings, and exactly one scene during extraction.
extract/dnd/scene-descriptions/source_refs	Current-source validity at all stages; at extraction, requires a current chunk and exact equality of both scene ID and source range to that chunk.
extract/dnd/scene-descriptions/source_relatedness	Advisory lexical grounding for title and summary against the scene's current transcript range only.
normalize/dnd/scene-descriptions/invariants	Normalized order, exact-duplicate removal, unique/conflict rules, and normalized strings.

The shape validator requires a present, non-empty scenes list at all stages and exactly one element when req.Stage == string(pipeline.StageExtract). Stage-independent record validation rejects blank IDs, unsupported kinds, blank or untrimmed title/summary, and structurally empty source references.

The source-reference validator first defers malformed shape to the shape validator. At extraction it rejects a nil req.Chunk, an ID unequal to req.Chunk.ID, or a source_ref unequal to req.Chunk.Ref; containment is not sufficient. At normalization, where no current chunk is available, it validates the reference against req.Source but does not attempt to reconstruct chunk identity.

The source-relatedness validator is warning-only and runs only after shape and source-reference validity. For each scene, tokenize the cited current-source text, title, and summary with shared.NormalizedTokens. Ignore tokens shorter than three Unicode code points and these case-normalized function words:

a, an, and, are, as, at, be, but, by, for, from, had, has, have, he, her, him, his, in, into, is, it, its, of, on, or, she, that, the, their, them, they, this, to, was, were, with.

Emit one warning scoped to scenes[i].title if no remaining title token occurs in the cited text, and independently one warning scoped to scenes[i].summary if no remaining summary token occurs there. If a field has no remaining significant token, emit its warning. Never use campaign references for this check. Bound and safely quote diagnostics through the existing D&D diagnostic helpers; do not include transcript or reference content in messages.

Register these exact default chains:

extract dnd/scene-descriptions:
  generic/valid_json
  extract/dnd/scene-descriptions/shape
  extract/dnd/scene-descriptions/source_refs
  generic/valid_json_schema
  extract/dnd/scene-descriptions/source_relatedness

normalize dnd/scene-descriptions:
  generic/valid_json
  extract/dnd/scene-descriptions/shape
  normalize/dnd/scene-descriptions/invariants
  extract/dnd/scene-descriptions/source_refs
  generic/valid_json_schema
  extract/dnd/scene-descriptions/source_relatedness

Do not add merge-stage validators, LLM-backed validators, options, generated references, stage dependencies, chunk-map dependencies, or framework changes.

Stage 1: Add The Durable Typed Contract And Codec

Work

1. Extend internal/modules/dnd/types.go with the fixed artifact kind, scene kind, constants, list, and record types.
2. Add internal/modules/dnd/codec/scenedescriptions/ following the strict candidate/approved encode/decode pattern in the existing D&D codecs.
3. Embed assets/schemas/dnd_scene_descriptions.v1.json with the exact durable shape and identities above.
4. Return scene_count from codec metadata.
5. Add one representative valid JSON fixture under testdata/.

Tests

Add focused codec tests covering schema identity and validity, valid round-trip, strict unknown/trailing JSON rejection, nil scenes, invalid kinds, blank required strings, malformed source fields, candidate decoding of semantically invalid but structurally decodable values, metadata, and independent ownership of encoded/schema bytes where the existing codec contract requires it.

Do not duplicate shape-validator and durable JSON Schema coverage exhaustively in the codec suite.

Completion gate

Run:

gofmt -w internal/modules/dnd/types.go internal/modules/dnd/codec/scenedescriptions/*.go
go test ./internal/modules/dnd/codec/scenedescriptions

The repository must compile with no production registration added yet.

Stage 2: Add The Private Model Contract, Prompt, And Extractor

Work

1. Add internal/modules/dnd/extract/scenedescriptions/ using the established D&D extractor package layout: assets, model, schema loader, prompt registration/fingerprints, constructor/options handling, extractor, and deterministic mapper.
2. Define a private Go response with exactly kind, title, and summary.
3. Embed assets/schemas/dnd_scene_descriptions_llm.v1.json with the fixed strict private contract.
4. Add assets/prompts/dnd.scene_descriptions.yaml, task.md, and instructions.md with the fixed message order and policy above.
5. Use shared.ReferenceSlots with package-owned descriptions for optional players, party, and glossary slots. The extractor and its module spec expose no other slots.
6. Map the private response to exactly one durable record using the current chunk ID and exact Chunk.Ref.

Tests

At the package owners, cover:

• private schema identity, strict shape, closed enum, and rejection of application-owned or extra fields;
• prompt and response-schema asset registration and fingerprints;
• prepared message roles/order, compatible shared-asset use, explicit empty campaign placeholders, one transcript rendering, and transcript-last placement;
• module spec, capabilities, optional reference slots, option rejection, and registration;
• request validation and provider failure;
• exact application-owned ID/range mapping;
• title/summary trimming and non-repair of kind; and
• prompt inputs using only current chunk material plus optional references.

Do not assert prompt byte length, shared-prefix length, exact generated prose, or provider output quality.

Completion gate

Run:

gofmt -w internal/modules/dnd/extract/scenedescriptions/*.go
go test ./internal/modules/dnd/extract/scenedescriptions

Do not register the extractor in the production D&D family in this stage.

Stage 3: Add Deterministic Normalization And Validation

Work

1. Add internal/modules/dnd/normalize/scenedescriptions/ with the fixed normalization and conflict behavior. Expose no options or reference slots.
2. Add the four validator packages under internal/modules/dnd/validate/scenedescriptions/ using the exact keys and responsibilities above: shape, source_refs, source_relatedness, and invariants.
3. Give each normalizer/validator a checkpoint policy fingerprint that changes when its owned deterministic behavior changes.
4. Use shared source and diagnostic helpers where their semantics match. Keep the scene-specific lexical stopword policy local to the relatedness validator.

Tests

Cover normalization of whitespace and source ordering; ID tie-breaking; exact deduplication; both conflict classes; preservation of same-range, same-content, different-ID records; invalid source/shape/kind handling; input ownership; cancellation/nil requests; and module registration metadata.

Cover validators at their stable owners, including:

• extraction requires exactly one scene;
• extraction ID and range must exactly equal the current chunk;
• normalization validates current-source membership without a chunk;
• shape and source-reference deferral between validators;
• supported and unsupported kinds;
• invariant order, trimming, duplicate, and conflict checks;
• separate bounded title and summary relatedness warnings;
• zero-significant-token warnings;
• transcript-only relatedness even when campaign references contain matching text; and
• safe bounded diagnostics.

Use table-driven cases where several inputs exercise one behavior. Do not replicate complete codec or JSON Schema test matrices in every validator.

Completion gate

Run:

gofmt -w internal/modules/dnd/normalize/scenedescriptions/*.go internal/modules/dnd/validate/scenedescriptions/*/*.go
go test ./internal/modules/dnd/normalize/scenedescriptions ./internal/modules/dnd/validate/scenedescriptions/...

Production registration remains deferred until all leaf packages pass.

Stage 4: Compose The Production Lane

Work

1. Update internal/modules/dnd/register/modules.go to register the codec, extractor, append-order specialization, normalizer, typed no-op normalizer, and prompt/schema assets.
2. Add the list append/clone helper in internal/modules/dnd/register/merge.go.
3. Update internal/modules/dnd/register/validators.go to register all four validators plus typed always-accept and always-reject specializations.
4. Update internal/modules/dnd/register/chains.go with the exact extract and normalize chains above.
5. Extend internal/modules/dnd/register/register_test.go to assert the new keys, artifact-kind specializations, assets, module specs, and exact production chains.
6. Add one offline assembled production workflow test under internal/cli/ using the real family registration/configuration/runner/output path and a fake structured LLM. It must exercise at least two accepted scene chunks and prove:
   • one model description is mapped to each chunk's ID and exact range;
   • append merge and normalization produce source order;
   • model-owned fields survive with only specified trimming;
   • durable schema identity and output payload are correct; and
   • no NPC or generated-artifact reference is required.

Keep this as one representative assembled test. Do not add equivalent end-to-end cases in multiple packages.

Completion gate

Run:

gofmt -w internal/modules/dnd/register/*.go internal/cli/dnd_scene_descriptions_contract_test.go
go test ./internal/modules/dnd/register ./internal/cli

At this gate the lane must be selectable through production registration, with no direct production import outside the D&D family registrar and CLI composition root.

Stage 5: Publish Current Documentation And Maintained Example

Work

Only after Stage 4 is green:

1. Add docs/integrations/dnd-scene-description-artifacts.md as the canonical durable contract. Document identities, exact JSON shape, scene kinds, application-owned ID/range mapping, merge/normalization rules, conflict behavior, advisory relatedness warnings, optional campaign references, and a copyable lane example.
2. Update docs/config.md with the maintained example link; extractor, normalizer, and validator keys; optional players, party, and glossary slots; and the exact default chains.
3. Update docs/internal/modules.md and docs/internal/overview.md for the new codec/extractor/normalizer/validators and family registration.
4. Update docs/internal/llm.md with the prompt manifest and the deliberate omission of the citation-oriented shared extraction-evidence message. Make clear that compatible shared messages remain canonical shared assets and the transcript remains last.
5. Update docs/internal/pipeline.md only where its current artifact-lane or source-attachment inventory requires the new lane; do not describe a new stage or dependency.
6. Update docs/integrations/json-output.md to link the new durable artifact contract alongside the other typed D&D schemas.
7. Add examples/dnd-scene-descriptions.config.yml. It should use chunk: dnd/scenes, one scene-description artifact lane, and normalize: dnd/scene-descriptions. Keep it minimal and do not present chunk annotations as the durable description artifact.
8. Mark docs/roadmap/dnd-scene-descriptions.md as Status: Implemented, add a link to the canonical integration document near the top, and remove the implemented “Extract D&D Scene Descriptions” entry from docs/roadmap/future.md. Leave chunker minimization and combat gating in future work.

Documentation checks

Verify every new selectable key, schema identity, enum, validator order, and example against production code. Check all relative Markdown links and run the maintained example configuration through config loading or the closest existing example/config test. Do not document private package mechanics in the integration contract.

Stage 6: Final Quality Gate

1. Review the diff for accidental changes to dnd/scenes, combat gating, framework stages, generated-reference behavior, or unrelated D&D contracts.
2. Confirm all new JSON objects reject unknown fields and the model is never asked to return application-owned identity/evidence.
3. Confirm diagnostics contain no transcript or campaign-reference content.
4. Confirm the prompt uses shared assets by reference and that the transcript is the final message; do not introduce a prompt-length or exact-prefix detector.
5. Run:

gofmt -l .
go test ./...
go vet ./...
git diff --check

gofmt -l . must produce no output. Do not mark the feature roadmap implemented until all commands pass and the current documentation describes the shipped behavior.

Open Questions

None. The accepted feature roadmap and the fixed decisions in this plan are sufficient to implement all stages without further product or architecture choices.