eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden D&D scene description extraction

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-23 23:03:37 +00:00
parent 9614eb540d
commit e1c2f3c202
8 changed files with 118 additions and 763 deletions
Download Patch File Download Diff File Expand all files Collapse all files

485
docs/roadmap/implementation.md
View File

@@ -1,485 +0,0 @@
# D&D Scene Descriptions Implementation Plan
Status: Ready for implementation
## Objective And Authority
Implement the accepted target state in
[D&D Scene Descriptions](dnd-scene-descriptions.md): one strict, typed
`dnd/scene-description-list` artifact record for every successfully extracted
accepted scene chunk. The model owns only `kind`, `title`, and `summary`;
Notarius owns the scene ID and exact current-source range.
This plan is the execution document. The feature roadmap owns intent, durable
policy, classification semantics, and non-goals. If wording here appears to
conflict with that roadmap, preserve the roadmap's contract and update this
plan before proceeding.
Follow:
- [Architecture Policy](../policy/architecture.md), especially typed domain
codecs, source-reference ownership, central registration, and the fixed
input/chunk/extract/merge/normalize/output flow;
- [Testing Policy](../policy/testing.md), especially behavioral tests at the
stable owner, offline fake-LLM coverage, and the prohibition on prompt-length
or shared-prefix change-detector tests; and
- [Documentation Policy](../policy/documentation.md), especially one canonical
owner for each contract and updating current-state docs only when behavior is
implemented.
The repository is pre-release. Use the `v1` identities fixed below; do not add
compatibility aliases, dual schemas, migration adapters, or legacy decoding.
## Fixed Implementation Decisions
### Public identities
Use these values exactly:
| Concern | Value |
| --- | --- |
| Extractor key | `dnd/scene-descriptions` |
| Normalizer key | `dnd/scene-descriptions` |
| Artifact kind | `dnd/scene-description-list` |
| Durable schema ID | `notarius.dnd.scene_descriptions` |
| Durable schema name | `notarius_dnd_scene_descriptions_v1` |
| Durable schema version | `v1` |
| Durable media type | `application/json` |
| Prompt ID | `dnd.scene_descriptions` |
| Private response-schema key | `dnd_scene_descriptions_llm` |
| Private response-schema ID | `notarius.dnd.scene_descriptions.llm` |
| Private response-schema name | `notarius_dnd_scene_descriptions_llm_v1` |
| Private response-schema version | `v1` |
| Extractor mapping policy | `dnd.scene_descriptions.mapping.v1` |
| Normalizer policy | `dnd.scene_descriptions.normalizer.v1` |
### Typed and JSON shapes
Add the following canonical D&D types in `internal/modules/dnd/types.go`:
```go
type SceneKind string
const (
SceneKindCombat SceneKind = "combat"
SceneKindNarrative SceneKind = "narrative"
SceneKindRecap SceneKind = "recap"
SceneKindMeta SceneKind = "meta"
)
type SceneDescriptionList struct {
Scenes []SceneDescription `json:"scenes"`
}
type SceneDescription struct {
ID string `json:"id"`
SourceRef source.SourceRef `json:"source_ref"`
Kind SceneKind `json:"kind"`
Title string `json:"title"`
Summary string `json:"summary"`
}
```
Also add
`SceneDescriptionListKind contracts.ArtifactKind =
"dnd/scene-description-list"`.
The durable JSON object contains exactly `scenes`; every scene object contains
exactly `id`, `source_ref`, `kind`, `title`, and `summary`. The private model
response is one object containing exactly `kind`, `title`, and `summary`. It is
not wrapped in `scenes`, cannot be an array, and cannot represent “no result.”
Both schemas reject unknown fields. Both schemas enumerate all four kind
values. Durable strings and private `title` and `summary` use `minLength: 1`;
do not impose arbitrary maximum lengths in schema or deterministic validation.
### Application-owned mapping
For each extraction request:
- obtain transcript material with `shared.ChunkPromptMaterial`;
- prepare optional `players`, `party`, and `glossary` inputs, including the
established explicit empty placeholders, with `shared.PromptInputs`;
- call the private structured prompt once;
- copy `req.Chunk.ID` to the durable `id`;
- copy `req.Chunk.Ref` to the durable `source_ref`;
- copy `kind` without trimming, case-folding, aliasing, or repair; and
- trim surrounding whitespace from `title` and `summary`, then return a
single-element durable `scenes` list.
An invalid or absent model object is an extraction failure handled by existing
structured-output and retry policy. The extractor must never synthesize a
fallback classification or empty scene list.
Include the prompt digest, response-schema digest, and mapping policy in
extractor manifest metadata and checkpoint fingerprints, following the combat
turn extractor pattern. The optional campaign references affect the prompt
input identity through existing framework behavior; do not add a generated
artifact reference or an NPC registry slot.
### Prompt assets and caching
Create package-owned prompt assets under
`internal/modules/dnd/extract/scenedescriptions/assets/`. Use this exact message
order:
1. shared `common-dnd-system.md` as the system message;
2. shared `common-dnd-identity.md` as a user message and cache boundary;
3. shared `common-dnd-references.md` as a user message and cache boundary;
4. package-owned `task.md`;
5. package-owned `instructions.md` as the final stable cache boundary; and
6. shared `common-dnd-transcript.md` as the final user message, without cache
control.
Declare `transcript`, `players`, `party`, and `glossary` inputs with the same
media types and required/optional status used by `dnd/npcs`. Reuse the shared
assets through the existing Scriptorium module/shared filesystem pattern; do
not copy their text into this package.
Do **not** render `common-dnd-extraction-evidence.md`. Its citation instructions
require the model to emit ranges, which conflicts with the application-owned
whole-chunk evidence contract. Put the scene-kind vocabulary, residual
mixed-scene precedence, title constraints, summary constraints, and the rule
that the response describes exactly the supplied accepted chunk in the
package-owned task/instruction assets. Do not ask for IDs, ranges, source IDs,
participants, confidence, or additional fields.
### Merge, normalization, and conflicts
Register the existing append-order merger specialized for
`SceneDescriptionList`. Its append function must preserve nil-versus-present
slice semantics and return independently owned values, matching the other D&D
specializations.
The normalizer is deterministic and has no reference slots. It must:
1. reject a nil source, malformed current-source reference, blank ID, invalid
kind, or blank title/summary;
2. trim only surrounding whitespace from title and summary;
3. sort by the source document's start-unit position, then scene ID as the
deterministic tie-breaker;
4. remove only records identical in all five durable fields;
5. reject any repeated scene ID whose remaining fields are not identical; and
6. reject records with the same exact source range and different
`kind`, `title`, or `summary`.
Two records with different IDs but the same range and identical model-owned
content are not exact duplicates and are not a conflict under this contract;
retain both in deterministic ID order. Do not merge adjacent ranges, reconcile
prose, change kinds heuristically, infer missing scenes, or use chunk
annotations.
### Validation ownership and default chains
Add these deterministic validators:
| Key | Owner |
| --- | --- |
| `extract/dnd/scene-descriptions/shape` | Required list and record fields, exact scene-kind vocabulary, trimmed non-empty strings, and exactly one scene during extraction. |
| `extract/dnd/scene-descriptions/source_refs` | Current-source validity at all stages; at extraction, requires a current chunk and exact equality of both scene ID and source range to that chunk. |
| `extract/dnd/scene-descriptions/source_relatedness` | Advisory lexical grounding for title and summary against the scene's current transcript range only. |
| `normalize/dnd/scene-descriptions/invariants` | Normalized order, exact-duplicate removal, unique/conflict rules, and normalized strings. |
The shape validator requires a present, non-empty `scenes` list at all stages
and exactly one element when `req.Stage == string(pipeline.StageExtract)`.
Stage-independent record validation rejects blank IDs, unsupported kinds,
blank or untrimmed title/summary, and structurally empty source references.
The source-reference validator first defers malformed shape to the shape
validator. At extraction it rejects a nil `req.Chunk`, an ID unequal to
`req.Chunk.ID`, or a `source_ref` unequal to `req.Chunk.Ref`; containment is not
sufficient. At normalization, where no current chunk is available, it validates
the reference against `req.Source` but does not attempt to reconstruct chunk
identity.
The source-relatedness validator is warning-only and runs only after shape and
source-reference validity. For each scene, tokenize the cited current-source
text, title, and summary with `shared.NormalizedTokens`. Ignore tokens shorter
than three Unicode code points and these case-normalized function words:
`a`, `an`, `and`, `are`, `as`, `at`, `be`, `but`, `by`, `for`, `from`, `had`,
`has`, `have`, `he`, `her`, `him`, `his`, `in`, `into`, `is`, `it`, `its`,
`of`, `on`, `or`, `she`, `that`, `the`, `their`, `them`, `they`, `this`, `to`,
`was`, `were`, `with`.
Emit one warning scoped to `scenes[i].title` if no remaining title token occurs
in the cited text, and independently one warning scoped to
`scenes[i].summary` if no remaining summary token occurs there. If a field has
no remaining significant token, emit its warning. Never use campaign
references for this check. Bound and safely quote diagnostics through the
existing D&D diagnostic helpers; do not include transcript or reference
content in messages.
Register these exact default chains:
```text
extract dnd/scene-descriptions:
generic/valid_json
extract/dnd/scene-descriptions/shape
extract/dnd/scene-descriptions/source_refs
generic/valid_json_schema
extract/dnd/scene-descriptions/source_relatedness
normalize dnd/scene-descriptions:
generic/valid_json
extract/dnd/scene-descriptions/shape
normalize/dnd/scene-descriptions/invariants
extract/dnd/scene-descriptions/source_refs
generic/valid_json_schema
extract/dnd/scene-descriptions/source_relatedness
```
Do not add merge-stage validators, LLM-backed validators, options, generated
references, stage dependencies, chunk-map dependencies, or framework changes.
## Stage 1: Add The Durable Typed Contract And Codec
### Work
1. Extend `internal/modules/dnd/types.go` with the fixed artifact kind, scene
kind, constants, list, and record types.
2. Add `internal/modules/dnd/codec/scenedescriptions/` following the strict
candidate/approved encode/decode pattern in the existing D&D codecs.
3. Embed
`assets/schemas/dnd_scene_descriptions.v1.json` with the exact durable shape
and identities above.
4. Return `scene_count` from codec metadata.
5. Add one representative valid JSON fixture under `testdata/`.
### Tests
Add focused codec tests covering schema identity and validity, valid
round-trip, strict unknown/trailing JSON rejection, nil `scenes`, invalid
kinds, blank required strings, malformed source fields, candidate decoding of
semantically invalid but structurally decodable values, metadata, and
independent ownership of encoded/schema bytes where the existing codec
contract requires it.
Do not duplicate shape-validator and durable JSON Schema coverage exhaustively
in the codec suite.
### Completion gate
Run:
```sh
gofmt -w internal/modules/dnd/types.go internal/modules/dnd/codec/scenedescriptions/*.go
go test ./internal/modules/dnd/codec/scenedescriptions
```
The repository must compile with no production registration added yet.
## Stage 2: Add The Private Model Contract, Prompt, And Extractor
### Work
1. Add `internal/modules/dnd/extract/scenedescriptions/` using the established
D&D extractor package layout: assets, model, schema loader, prompt
registration/fingerprints, constructor/options handling, extractor, and
deterministic mapper.
2. Define a private Go response with exactly `kind`, `title`, and `summary`.
3. Embed
`assets/schemas/dnd_scene_descriptions_llm.v1.json` with the fixed strict
private contract.
4. Add `assets/prompts/dnd.scene_descriptions.yaml`, `task.md`, and
`instructions.md` with the fixed message order and policy above.
5. Use `shared.ReferenceSlots` with package-owned descriptions for optional
`players`, `party`, and `glossary` slots. The extractor and its module spec
expose no other slots.
6. Map the private response to exactly one durable record using the current
chunk ID and exact `Chunk.Ref`.
### Tests
At the package owners, cover:
- private schema identity, strict shape, closed enum, and rejection of
application-owned or extra fields;
- prompt and response-schema asset registration and fingerprints;
- prepared message roles/order, compatible shared-asset use, explicit empty
campaign placeholders, one transcript rendering, and transcript-last
placement;
- module spec, capabilities, optional reference slots, option rejection, and
registration;
- request validation and provider failure;
- exact application-owned ID/range mapping;
- title/summary trimming and non-repair of kind; and
- prompt inputs using only current chunk material plus optional references.
Do not assert prompt byte length, shared-prefix length, exact generated prose,
or provider output quality.
### Completion gate
Run:
```sh
gofmt -w internal/modules/dnd/extract/scenedescriptions/*.go
go test ./internal/modules/dnd/extract/scenedescriptions
```
Do not register the extractor in the production D&D family in this stage.
## Stage 3: Add Deterministic Normalization And Validation
### Work
1. Add `internal/modules/dnd/normalize/scenedescriptions/` with the fixed
normalization and conflict behavior. Expose no options or reference slots.
2. Add the four validator packages under
`internal/modules/dnd/validate/scenedescriptions/` using the exact keys and
responsibilities above:
`shape`, `source_refs`, `source_relatedness`, and `invariants`.
3. Give each normalizer/validator a checkpoint policy fingerprint that changes
when its owned deterministic behavior changes.
4. Use shared source and diagnostic helpers where their semantics match. Keep
the scene-specific lexical stopword policy local to the relatedness
validator.
### Tests
Cover normalization of whitespace and source ordering; ID tie-breaking; exact
deduplication; both conflict classes; preservation of same-range,
same-content, different-ID records; invalid source/shape/kind handling; input
ownership; cancellation/nil requests; and module registration metadata.
Cover validators at their stable owners, including:
- extraction requires exactly one scene;
- extraction ID and range must exactly equal the current chunk;
- normalization validates current-source membership without a chunk;
- shape and source-reference deferral between validators;
- supported and unsupported kinds;
- invariant order, trimming, duplicate, and conflict checks;
- separate bounded title and summary relatedness warnings;
- zero-significant-token warnings;
- transcript-only relatedness even when campaign references contain matching
text; and
- safe bounded diagnostics.
Use table-driven cases where several inputs exercise one behavior. Do not
replicate complete codec or JSON Schema test matrices in every validator.
### Completion gate
Run:
```sh
gofmt -w internal/modules/dnd/normalize/scenedescriptions/*.go internal/modules/dnd/validate/scenedescriptions/*/*.go
go test ./internal/modules/dnd/normalize/scenedescriptions ./internal/modules/dnd/validate/scenedescriptions/...
```
Production registration remains deferred until all leaf packages pass.
## Stage 4: Compose The Production Lane
### Work
1. Update `internal/modules/dnd/register/modules.go` to register the codec,
extractor, append-order specialization, normalizer, typed no-op normalizer,
and prompt/schema assets.
2. Add the list append/clone helper in
`internal/modules/dnd/register/merge.go`.
3. Update `internal/modules/dnd/register/validators.go` to register all four
validators plus typed always-accept and always-reject specializations.
4. Update `internal/modules/dnd/register/chains.go` with the exact extract and
normalize chains above.
5. Extend `internal/modules/dnd/register/register_test.go` to assert the new
keys, artifact-kind specializations, assets, module specs, and exact
production chains.
6. Add one offline assembled production workflow test under `internal/cli/`
using the real family registration/configuration/runner/output path and a
fake structured LLM. It must exercise at least two accepted scene chunks and
prove:
- one model description is mapped to each chunk's ID and exact range;
- append merge and normalization produce source order;
- model-owned fields survive with only specified trimming;
- durable schema identity and output payload are correct; and
- no NPC or generated-artifact reference is required.
Keep this as one representative assembled test. Do not add equivalent
end-to-end cases in multiple packages.
### Completion gate
Run:
```sh
gofmt -w internal/modules/dnd/register/*.go internal/cli/dnd_scene_descriptions_contract_test.go
go test ./internal/modules/dnd/register ./internal/cli
```
At this gate the lane must be selectable through production registration, with
no direct production import outside the D&D family registrar and CLI
composition root.
## Stage 5: Publish Current Documentation And Maintained Example
### Work
Only after Stage 4 is green:
1. Add `docs/integrations/dnd-scene-description-artifacts.md` as the canonical
durable contract. Document identities, exact JSON shape, scene kinds,
application-owned ID/range mapping, merge/normalization rules, conflict
behavior, advisory relatedness warnings, optional campaign references, and
a copyable lane example.
2. Update `docs/config.md` with the maintained example link; extractor,
normalizer, and validator keys; optional `players`, `party`, and `glossary`
slots; and the exact default chains.
3. Update `docs/internal/modules.md` and `docs/internal/overview.md` for the new
codec/extractor/normalizer/validators and family registration.
4. Update `docs/internal/llm.md` with the prompt manifest and the deliberate
omission of the citation-oriented shared extraction-evidence message. Make
clear that compatible shared messages remain canonical shared assets and the
transcript remains last.
5. Update `docs/internal/pipeline.md` only where its current artifact-lane or
source-attachment inventory requires the new lane; do not describe a new
stage or dependency.
6. Update `docs/integrations/json-output.md` to link the new durable artifact
contract alongside the other typed D&D schemas.
7. Add `examples/dnd-scene-descriptions.config.yml`. It should use
`chunk: dnd/scenes`, one scene-description artifact lane, and
`normalize: dnd/scene-descriptions`. Keep it minimal and do not present
chunk annotations as the durable description artifact.
8. Mark `docs/roadmap/dnd-scene-descriptions.md` as `Status: Implemented`,
add a link to the canonical integration document near the top, and remove
the implemented “Extract D&D Scene Descriptions” entry from
`docs/roadmap/future.md`. Leave chunker minimization and combat gating in
future work.
### Documentation checks
Verify every new selectable key, schema identity, enum, validator order, and
example against production code. Check all relative Markdown links and run the
maintained example configuration through config loading or the closest
existing example/config test. Do not document private package mechanics in the
integration contract.
## Stage 6: Final Quality Gate
1. Review the diff for accidental changes to `dnd/scenes`, combat gating,
framework stages, generated-reference behavior, or unrelated D&D contracts.
2. Confirm all new JSON objects reject unknown fields and the model is never
asked to return application-owned identity/evidence.
3. Confirm diagnostics contain no transcript or campaign-reference content.
4. Confirm the prompt uses shared assets by reference and that the transcript
is the final message; do not introduce a prompt-length or exact-prefix
detector.
5. Run:
```sh
gofmt -l .
go test ./...
go vet ./...
git diff --check
```
`gofmt -l .` must produce no output. Do not mark the feature roadmap
implemented until all commands pass and the current documentation describes
the shipped behavior.
## Open Questions
None. The accepted feature roadmap and the fixed decisions in this plan are
sufficient to implement all stages without further product or architecture
choices.