notarius/docs/roadmap/implementation.md

# D&D Scene Descriptions Implementation Plan

Status: Ready for implementation

## Objective And Authority

Implement the accepted target state in
[D&D Scene Descriptions](dnd-scene-descriptions.md): one strict, typed
`dnd/scene-description-list` artifact record for every successfully extracted
accepted scene chunk. The model owns only `kind`, `title`, and `summary`;
Notarius owns the scene ID and exact current-source range.

This plan is the execution document. The feature roadmap owns intent, durable
policy, classification semantics, and non-goals. If wording here appears to
conflict with that roadmap, preserve the roadmap's contract and update this
plan before proceeding.

Follow:

- [Architecture Policy](../policy/architecture.md), especially typed domain
  codecs, source-reference ownership, central registration, and the fixed
  input/chunk/extract/merge/normalize/output flow;
- [Testing Policy](../policy/testing.md), especially behavioral tests at the
  stable owner, offline fake-LLM coverage, and the prohibition on prompt-length
  or shared-prefix change-detector tests; and
- [Documentation Policy](../policy/documentation.md), especially one canonical
  owner for each contract and updating current-state docs only when behavior is
  implemented.

The repository is pre-release. Use the `v1` identities fixed below; do not add
compatibility aliases, dual schemas, migration adapters, or legacy decoding.

## Fixed Implementation Decisions

### Public identities

Use these values exactly:

| Concern | Value |
| --- | --- |
| Extractor key | `dnd/scene-descriptions` |
| Normalizer key | `dnd/scene-descriptions` |
| Artifact kind | `dnd/scene-description-list` |
| Durable schema ID | `notarius.dnd.scene_descriptions` |
| Durable schema name | `notarius_dnd_scene_descriptions_v1` |
| Durable schema version | `v1` |
| Durable media type | `application/json` |
| Prompt ID | `dnd.scene_descriptions` |
| Private response-schema key | `dnd_scene_descriptions_llm` |
| Private response-schema ID | `notarius.dnd.scene_descriptions.llm` |
| Private response-schema name | `notarius_dnd_scene_descriptions_llm_v1` |
| Private response-schema version | `v1` |
| Extractor mapping policy | `dnd.scene_descriptions.mapping.v1` |
| Normalizer policy | `dnd.scene_descriptions.normalizer.v1` |

### Typed and JSON shapes

Add the following canonical D&D types in `internal/modules/dnd/types.go`:

```go
type SceneKind string

const (
    SceneKindCombat    SceneKind = "combat"
    SceneKindNarrative SceneKind = "narrative"
    SceneKindRecap     SceneKind = "recap"
    SceneKindMeta      SceneKind = "meta"
)

type SceneDescriptionList struct {
    Scenes []SceneDescription `json:"scenes"`
}

type SceneDescription struct {
    ID        string           `json:"id"`
    SourceRef source.SourceRef `json:"source_ref"`
    Kind      SceneKind        `json:"kind"`
    Title     string           `json:"title"`
    Summary   string           `json:"summary"`
}
```

Also add
`SceneDescriptionListKind contracts.ArtifactKind =
"dnd/scene-description-list"`.

The durable JSON object contains exactly `scenes`; every scene object contains
exactly `id`, `source_ref`, `kind`, `title`, and `summary`. The private model
response is one object containing exactly `kind`, `title`, and `summary`. It is
not wrapped in `scenes`, cannot be an array, and cannot represent “no result.”
Both schemas reject unknown fields. Both schemas enumerate all four kind
values. Durable strings and private `title` and `summary` use `minLength: 1`;
do not impose arbitrary maximum lengths in schema or deterministic validation.

### Application-owned mapping

For each extraction request:

- obtain transcript material with `shared.ChunkPromptMaterial`;
- prepare optional `players`, `party`, and `glossary` inputs, including the
  established explicit empty placeholders, with `shared.PromptInputs`;
- call the private structured prompt once;
- copy `req.Chunk.ID` to the durable `id`;
- copy `req.Chunk.Ref` to the durable `source_ref`;
- copy `kind` without trimming, case-folding, aliasing, or repair; and
- trim surrounding whitespace from `title` and `summary`, then return a
  single-element durable `scenes` list.

An invalid or absent model object is an extraction failure handled by existing
structured-output and retry policy. The extractor must never synthesize a
fallback classification or empty scene list.

Include the prompt digest, response-schema digest, and mapping policy in
extractor manifest metadata and checkpoint fingerprints, following the combat
turn extractor pattern. The optional campaign references affect the prompt
input identity through existing framework behavior; do not add a generated
artifact reference or an NPC registry slot.

### Prompt assets and caching

Create package-owned prompt assets under
`internal/modules/dnd/extract/scenedescriptions/assets/`. Use this exact message
order:

1. shared `common-dnd-system.md` as the system message;
2. shared `common-dnd-identity.md` as a user message and cache boundary;
3. shared `common-dnd-references.md` as a user message and cache boundary;
4. package-owned `task.md`;
5. package-owned `instructions.md` as the final stable cache boundary; and
6. shared `common-dnd-transcript.md` as the final user message, without cache
   control.

Declare `transcript`, `players`, `party`, and `glossary` inputs with the same
media types and required/optional status used by `dnd/npcs`. Reuse the shared
assets through the existing Scriptorium module/shared filesystem pattern; do
not copy their text into this package.

Do **not** render `common-dnd-extraction-evidence.md`. Its citation instructions
require the model to emit ranges, which conflicts with the application-owned
whole-chunk evidence contract. Put the scene-kind vocabulary, residual
mixed-scene precedence, title constraints, summary constraints, and the rule
that the response describes exactly the supplied accepted chunk in the
package-owned task/instruction assets. Do not ask for IDs, ranges, source IDs,
participants, confidence, or additional fields.

### Merge, normalization, and conflicts

Register the existing append-order merger specialized for
`SceneDescriptionList`. Its append function must preserve nil-versus-present
slice semantics and return independently owned values, matching the other D&D
specializations.

The normalizer is deterministic and has no reference slots. It must:

1. reject a nil source, malformed current-source reference, blank ID, invalid
   kind, or blank title/summary;
2. trim only surrounding whitespace from title and summary;
3. sort by the source document's start-unit position, then scene ID as the
   deterministic tie-breaker;
4. remove only records identical in all five durable fields;
5. reject any repeated scene ID whose remaining fields are not identical; and
6. reject records with the same exact source range and different
   `kind`, `title`, or `summary`.

Two records with different IDs but the same range and identical model-owned
content are not exact duplicates and are not a conflict under this contract;
retain both in deterministic ID order. Do not merge adjacent ranges, reconcile
prose, change kinds heuristically, infer missing scenes, or use chunk
annotations.

### Validation ownership and default chains

Add these deterministic validators:

| Key | Owner |
| --- | --- |
| `extract/dnd/scene-descriptions/shape` | Required list and record fields, exact scene-kind vocabulary, trimmed non-empty strings, and exactly one scene during extraction. |
| `extract/dnd/scene-descriptions/source_refs` | Current-source validity at all stages; at extraction, requires a current chunk and exact equality of both scene ID and source range to that chunk. |
| `extract/dnd/scene-descriptions/source_relatedness` | Advisory lexical grounding for title and summary against the scene's current transcript range only. |
| `normalize/dnd/scene-descriptions/invariants` | Normalized order, exact-duplicate removal, unique/conflict rules, and normalized strings. |

The shape validator requires a present, non-empty `scenes` list at all stages
and exactly one element when `req.Stage == string(pipeline.StageExtract)`.
Stage-independent record validation rejects blank IDs, unsupported kinds,
blank or untrimmed title/summary, and structurally empty source references.

The source-reference validator first defers malformed shape to the shape
validator. At extraction it rejects a nil `req.Chunk`, an ID unequal to
`req.Chunk.ID`, or a `source_ref` unequal to `req.Chunk.Ref`; containment is not
sufficient. At normalization, where no current chunk is available, it validates
the reference against `req.Source` but does not attempt to reconstruct chunk
identity.

The source-relatedness validator is warning-only and runs only after shape and
source-reference validity. For each scene, tokenize the cited current-source
text, title, and summary with `shared.NormalizedTokens`. Ignore tokens shorter
than three Unicode code points and these case-normalized function words:

`a`, `an`, `and`, `are`, `as`, `at`, `be`, `but`, `by`, `for`, `from`, `had`,
`has`, `have`, `he`, `her`, `him`, `his`, `in`, `into`, `is`, `it`, `its`,
`of`, `on`, `or`, `she`, `that`, `the`, `their`, `them`, `they`, `this`, `to`,
`was`, `were`, `with`.

Emit one warning scoped to `scenes[i].title` if no remaining title token occurs
in the cited text, and independently one warning scoped to
`scenes[i].summary` if no remaining summary token occurs there. If a field has
no remaining significant token, emit its warning. Never use campaign
references for this check. Bound and safely quote diagnostics through the
existing D&D diagnostic helpers; do not include transcript or reference
content in messages.

Register these exact default chains:

```text
extract dnd/scene-descriptions:
  generic/valid_json
  extract/dnd/scene-descriptions/shape
  extract/dnd/scene-descriptions/source_refs
  generic/valid_json_schema
  extract/dnd/scene-descriptions/source_relatedness

normalize dnd/scene-descriptions:
  generic/valid_json
  extract/dnd/scene-descriptions/shape
  normalize/dnd/scene-descriptions/invariants
  extract/dnd/scene-descriptions/source_refs
  generic/valid_json_schema
  extract/dnd/scene-descriptions/source_relatedness
```

Do not add merge-stage validators, LLM-backed validators, options, generated
references, stage dependencies, chunk-map dependencies, or framework changes.

## Stage 1: Add The Durable Typed Contract And Codec

### Work

1. Extend `internal/modules/dnd/types.go` with the fixed artifact kind, scene
   kind, constants, list, and record types.
2. Add `internal/modules/dnd/codec/scenedescriptions/` following the strict
   candidate/approved encode/decode pattern in the existing D&D codecs.
3. Embed
   `assets/schemas/dnd_scene_descriptions.v1.json` with the exact durable shape
   and identities above.
4. Return `scene_count` from codec metadata.
5. Add one representative valid JSON fixture under `testdata/`.

### Tests

Add focused codec tests covering schema identity and validity, valid
round-trip, strict unknown/trailing JSON rejection, nil `scenes`, invalid
kinds, blank required strings, malformed source fields, candidate decoding of
semantically invalid but structurally decodable values, metadata, and
independent ownership of encoded/schema bytes where the existing codec
contract requires it.

Do not duplicate shape-validator and durable JSON Schema coverage exhaustively
in the codec suite.

### Completion gate

Run:

```sh
gofmt -w internal/modules/dnd/types.go internal/modules/dnd/codec/scenedescriptions/*.go
go test ./internal/modules/dnd/codec/scenedescriptions
```

The repository must compile with no production registration added yet.

## Stage 2: Add The Private Model Contract, Prompt, And Extractor

### Work

1. Add `internal/modules/dnd/extract/scenedescriptions/` using the established
   D&D extractor package layout: assets, model, schema loader, prompt
   registration/fingerprints, constructor/options handling, extractor, and
   deterministic mapper.
2. Define a private Go response with exactly `kind`, `title`, and `summary`.
3. Embed
   `assets/schemas/dnd_scene_descriptions_llm.v1.json` with the fixed strict
   private contract.
4. Add `assets/prompts/dnd.scene_descriptions.yaml`, `task.md`, and
   `instructions.md` with the fixed message order and policy above.
5. Use `shared.ReferenceSlots` with package-owned descriptions for optional
   `players`, `party`, and `glossary` slots. The extractor and its module spec
   expose no other slots.
6. Map the private response to exactly one durable record using the current
   chunk ID and exact `Chunk.Ref`.

### Tests

At the package owners, cover:

- private schema identity, strict shape, closed enum, and rejection of
  application-owned or extra fields;
- prompt and response-schema asset registration and fingerprints;
- prepared message roles/order, compatible shared-asset use, explicit empty
  campaign placeholders, one transcript rendering, and transcript-last
  placement;
- module spec, capabilities, optional reference slots, option rejection, and
  registration;
- request validation and provider failure;
- exact application-owned ID/range mapping;
- title/summary trimming and non-repair of kind; and
- prompt inputs using only current chunk material plus optional references.

Do not assert prompt byte length, shared-prefix length, exact generated prose,
or provider output quality.

### Completion gate

Run:

```sh
gofmt -w internal/modules/dnd/extract/scenedescriptions/*.go
go test ./internal/modules/dnd/extract/scenedescriptions
```

Do not register the extractor in the production D&D family in this stage.

## Stage 3: Add Deterministic Normalization And Validation

### Work

1. Add `internal/modules/dnd/normalize/scenedescriptions/` with the fixed
   normalization and conflict behavior. Expose no options or reference slots.
2. Add the four validator packages under
   `internal/modules/dnd/validate/scenedescriptions/` using the exact keys and
   responsibilities above:
   `shape`, `source_refs`, `source_relatedness`, and `invariants`.
3. Give each normalizer/validator a checkpoint policy fingerprint that changes
   when its owned deterministic behavior changes.
4. Use shared source and diagnostic helpers where their semantics match. Keep
   the scene-specific lexical stopword policy local to the relatedness
   validator.

### Tests

Cover normalization of whitespace and source ordering; ID tie-breaking; exact
deduplication; both conflict classes; preservation of same-range,
same-content, different-ID records; invalid source/shape/kind handling; input
ownership; cancellation/nil requests; and module registration metadata.

Cover validators at their stable owners, including:

- extraction requires exactly one scene;
- extraction ID and range must exactly equal the current chunk;
- normalization validates current-source membership without a chunk;
- shape and source-reference deferral between validators;
- supported and unsupported kinds;
- invariant order, trimming, duplicate, and conflict checks;
- separate bounded title and summary relatedness warnings;
- zero-significant-token warnings;
- transcript-only relatedness even when campaign references contain matching
  text; and
- safe bounded diagnostics.

Use table-driven cases where several inputs exercise one behavior. Do not
replicate complete codec or JSON Schema test matrices in every validator.

### Completion gate

Run:

```sh
gofmt -w internal/modules/dnd/normalize/scenedescriptions/*.go internal/modules/dnd/validate/scenedescriptions/*/*.go
go test ./internal/modules/dnd/normalize/scenedescriptions ./internal/modules/dnd/validate/scenedescriptions/...
```

Production registration remains deferred until all leaf packages pass.

## Stage 4: Compose The Production Lane

### Work

1. Update `internal/modules/dnd/register/modules.go` to register the codec,
   extractor, append-order specialization, normalizer, typed no-op normalizer,
   and prompt/schema assets.
2. Add the list append/clone helper in
   `internal/modules/dnd/register/merge.go`.
3. Update `internal/modules/dnd/register/validators.go` to register all four
   validators plus typed always-accept and always-reject specializations.
4. Update `internal/modules/dnd/register/chains.go` with the exact extract and
   normalize chains above.
5. Extend `internal/modules/dnd/register/register_test.go` to assert the new
   keys, artifact-kind specializations, assets, module specs, and exact
   production chains.
6. Add one offline assembled production workflow test under `internal/cli/`
   using the real family registration/configuration/runner/output path and a
   fake structured LLM. It must exercise at least two accepted scene chunks and
   prove:
   - one model description is mapped to each chunk's ID and exact range;
   - append merge and normalization produce source order;
   - model-owned fields survive with only specified trimming;
   - durable schema identity and output payload are correct; and
   - no NPC or generated-artifact reference is required.

Keep this as one representative assembled test. Do not add equivalent
end-to-end cases in multiple packages.

### Completion gate

Run:

```sh
gofmt -w internal/modules/dnd/register/*.go internal/cli/dnd_scene_descriptions_contract_test.go
go test ./internal/modules/dnd/register ./internal/cli
```

At this gate the lane must be selectable through production registration, with
no direct production import outside the D&D family registrar and CLI
composition root.

## Stage 5: Publish Current Documentation And Maintained Example

### Work

Only after Stage 4 is green:

1. Add `docs/integrations/dnd-scene-description-artifacts.md` as the canonical
   durable contract. Document identities, exact JSON shape, scene kinds,
   application-owned ID/range mapping, merge/normalization rules, conflict
   behavior, advisory relatedness warnings, optional campaign references, and
   a copyable lane example.
2. Update `docs/config.md` with the maintained example link; extractor,
   normalizer, and validator keys; optional `players`, `party`, and `glossary`
   slots; and the exact default chains.
3. Update `docs/internal/modules.md` and `docs/internal/overview.md` for the new
   codec/extractor/normalizer/validators and family registration.
4. Update `docs/internal/llm.md` with the prompt manifest and the deliberate
   omission of the citation-oriented shared extraction-evidence message. Make
   clear that compatible shared messages remain canonical shared assets and the
   transcript remains last.
5. Update `docs/internal/pipeline.md` only where its current artifact-lane or
   source-attachment inventory requires the new lane; do not describe a new
   stage or dependency.
6. Update `docs/integrations/json-output.md` to link the new durable artifact
   contract alongside the other typed D&D schemas.
7. Add `examples/dnd-scene-descriptions.config.yml`. It should use
   `chunk: dnd/scenes`, one scene-description artifact lane, and
   `normalize: dnd/scene-descriptions`. Keep it minimal and do not present
   chunk annotations as the durable description artifact.
8. Mark `docs/roadmap/dnd-scene-descriptions.md` as `Status: Implemented`,
   add a link to the canonical integration document near the top, and remove
   the implemented “Extract D&D Scene Descriptions” entry from
   `docs/roadmap/future.md`. Leave chunker minimization and combat gating in
   future work.

### Documentation checks

Verify every new selectable key, schema identity, enum, validator order, and
example against production code. Check all relative Markdown links and run the
maintained example configuration through config loading or the closest
existing example/config test. Do not document private package mechanics in the
integration contract.

## Stage 6: Final Quality Gate

1. Review the diff for accidental changes to `dnd/scenes`, combat gating,
   framework stages, generated-reference behavior, or unrelated D&D contracts.
2. Confirm all new JSON objects reject unknown fields and the model is never
   asked to return application-owned identity/evidence.
3. Confirm diagnostics contain no transcript or campaign-reference content.
4. Confirm the prompt uses shared assets by reference and that the transcript
   is the final message; do not introduce a prompt-length or exact-prefix
   detector.
5. Run:

```sh
gofmt -l .
go test ./...
go vet ./...
git diff --check
```

`gofmt -l .` must produce no output. Do not mark the feature roadmap
implemented until all commands pass and the current documentation describes
the shipped behavior.

## Open Questions

None. The accepted feature roadmap and the fixed decisions in this plan are
sufficient to implement all stages without further product or architecture
choices.