notarius/docs/roadmap/implementation.md

# Implementation Plan: Checkpoint 6 D&D Spells Extractor

## Status

This is a staged implementation plan for
[`6-dnd-spells-extractor.md`](6-dnd-spells-extractor.md). It is intended for an
LLM coding agent to follow stage by stage.

This plan implements only checkpoint 6. Do not add D&D item, NPC, combat, or
encounter extraction; do not add broad D&D rules validation; do not add a
`notarius run` command; and do not make core framework packages D&D-specific.

## Policy Context

Follow:

- [`docs/policy/architecture.md`](../policy/architecture.md)
- [`docs/policy/documentation.md`](../policy/documentation.md)

Required boundaries:

- keep D&D-specific artifact semantics, prompt data shaping, and response
  interpretation inside `internal/modules/extract/dnd/spells`;
- keep source-format details inside input modules and do not depend on concrete
  Seriatim package types from the spells extractor;
- keep core source, runner, pipeline, LLM, prompt, config, and validator
  packages source-agnostic and domain-agnostic;
- use embedded prompt and schema assets instead of inline prompt/schema strings;
- register the extractor through the existing extractor registry;
- use flat capability strings in module metadata;
- keep future or planned behavior in `docs/roadmap/` until implemented.

## Global Implementation Decisions

- Add no new third-party dependency.
- Use `dnd/spells` as the stable extractor module key.
- Put concrete extractor code under `internal/modules/extract/dnd/spells`.
- Use `dnd.spell_cast` as the artifact type and `v1` as the schema version.
- Use `dnd.spell_casts` as the extractor-provided capability.
- The extractor module spec must require `chunks` and `source.transcript`.
  It must not require `transcript.speaker` or `transcript.timestamps`; those
  metadata values may be included in prompts when present, but extraction must
  not depend on Seriatim-specific helper APIs.
- Keep durable artifact payloads focused on spell data:

```go
type SpellCast struct {
    Caster               string `json:"caster"`
    Spell                string `json:"spell"`
    Effect               string `json:"effect"`
    NarrativeDescription string `json:"narrative_description"`
}
```

- The LLM response model must include source references so the extractor can
  populate `artifacts.ArtifactCandidate.SourceRefs`, but source references must
  live in the generic artifact envelope for durable pipeline output rather than
  being duplicated inside the `SpellCast` payload.
- `caster` means the in-world character or creature casting the spell. Do not
  use table speaker/player as the durable caster field. Source-unit `speaker`
  metadata may be rendered as optional prompt context when present.
- Use `source.SourceRef` JSON field names exactly as the core type defines
  them: `source_id`, `start_unit_id`, and `end_unit_id`.
- Trim text fields before marshaling candidate payloads. Do not silently trim
  source-reference IDs; invalid source refs must be rejected by validation.
- `Extractor.Validators()` must return module-owned deterministic validators by
  default. Do not require pipeline profiles to configure validators explicitly
  for the default checkpoint behavior.
- Treat an empty `spell_casts` response array as a successful empty extraction.
  Treat a missing or null `spell_casts` field as malformed structured output.
- Use synthetic D&D transcript fixture text only. Do not include private
  campaign transcript content.

## Stage 1: Domain Model, Module Skeleton, And Registry Metadata

### Goal

Create the D&D spells extractor package, define the artifact and response
models, and make the extractor discoverable through the existing extractor
registry without calling an LLM yet.

### Files To Add Or Update

- `internal/modules/extract/dnd/spells/extractor.go`
- `internal/modules/extract/dnd/spells/model.go`
- `internal/modules/extract/dnd/spells/registry_test.go`

### Required API

Add:

```go
package spells

const Key = "dnd/spells"
const ArtifactType = "dnd.spell_cast"
const SchemaVersion = "v1"

type SpellCast struct {
    Caster               string `json:"caster"`
    Spell                string `json:"spell"`
    Effect               string `json:"effect"`
    NarrativeDescription string `json:"narrative_description"`
}

type Extractor struct{}

func New() *Extractor
func (e *Extractor) Key() string
func (e *Extractor) ArtifactType() string
func (e *Extractor) SchemaVersion() string
func (e *Extractor) Validators() []contracts.Validator
func (e *Extractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error)
func ModuleSpec() pipeline.ModuleSpec
func Register(registry *pipeline.ExtractorRegistry) error
```

Add unexported response structs in the same package:

```go
type extractionResponse struct {
    SpellCasts []spellCastResponse `json:"spell_casts"`
}

type spellCastResponse struct {
    Caster               string             `json:"caster"`
    Spell                string             `json:"spell"`
    Effect               string             `json:"effect"`
    NarrativeDescription string             `json:"narrative_description"`
    SourceRefs           []source.SourceRef `json:"source_refs"`
}
```

### Required Behavior

- `New()` returns a non-nil extractor.
- `ModuleSpec()` returns defensive slices with stage `pipeline.StageExtract`,
  key `dnd/spells`, requires `chunks` and `source.transcript`, and provides
  `dnd.spell_casts`.
- `Register()` calls `ExtractorRegistry.RegisterWithSpec(ModuleSpec(), ...)`.
- `Register(nil)` returns an error from the registry path rather than panicking.
- `Extract()` may return a clear not-yet-implemented error in this stage only.
- `Validators()` returns nil in this stage only; later stages must replace it
  with the real validator chain.

### Required Tests

- `New()` returns an extractor whose key, artifact type, and schema version
  match the constants.
- `ModuleSpec()` uses extract stage and declares the required capabilities.
- Caller mutation of `ModuleSpec().Provides` or `ModuleSpec().Requires` does not
  affect later calls.
- `Register()` makes the extractor buildable from an `ExtractorRegistry`.
- Registry lookup returns the spells module spec.
- `Register(nil)` returns an error containing extractor registry context.

### Validation

Run:

```sh
gofmt -w internal/modules/extract/dnd/spells
go test ./internal/modules/extract/dnd/spells
go test ./...
```

## Stage 2: Structured Response Schema Asset

### Goal

Add the structured response schema used by the D&D spells extractor and register
it through the existing LLM schema registry.

### Files To Add Or Update

- `internal/framework/llm/assets/schemas/dnd_spells.v1.json`
- `internal/framework/llm/schema_registry.go`
- `internal/framework/llm/schema_registry_test.go`
- `internal/modules/extract/dnd/spells/schema_test.go`

### Schema Decisions

Register:

- response schema key: `dnd_spells`
- schema ID: `notarius.dnd.spells`
- schema version: `v1`
- response schema name: `notarius_dnd_spells_v1`
- asset path: `assets/schemas/dnd_spells.v1.json`

The JSON schema must require a top-level object:

```json
{
  "spell_casts": [
    {
      "caster": "Aria",
      "spell": "Cure Wounds",
      "effect": "Heals an injured ally.",
      "narrative_description": "Aria casts Cure Wounds after the fight.",
      "source_refs": [
        {
          "source_id": "session-alpha",
          "start_unit_id": "seg-001",
          "end_unit_id": "seg-002"
        }
      ]
    }
  ]
}
```

Required schema constraints:

- `additionalProperties: false` at every object level.
- top-level `spell_casts` is required and must be an array.
- `spell_casts` may be empty.
- each spell cast requires non-empty `caster`, `spell`, `effect`, and
  `narrative_description`.
- each spell cast requires `source_refs` with `minItems: 1`.
- each source ref requires non-empty `source_id`, `start_unit_id`, and
  `end_unit_id`.

### Required Tests

- `llm.LookupResponseSchema(llm.DNDSpellsSchemaKey)` succeeds.
- Registered schema list remains sorted and now includes the D&D spells schema.
- The schema content is valid JSON and mutation-safe through registry lookups.
- The spells package can look up the schema key it will use during extraction.
- The schema diagnostics map omits raw schema content.

### Validation

Run:

```sh
gofmt -w internal/framework/llm internal/modules/extract/dnd/spells
go test ./internal/framework/llm
go test ./internal/modules/extract/dnd/spells
go test ./...
```

## Stage 3: Prompt Assets And Prompt Rendering

### Goal

Add embedded prompt assets for D&D spell extraction and render deterministic
system/user messages from generic source chunks.

### Files To Add Or Update

- `internal/framework/prompt/assets/dnd/spells/system.md`
- `internal/framework/prompt/assets/dnd/spells/user.md`
- `internal/framework/prompt/registry.go`
- `internal/framework/prompt/registry_test.go`
- `internal/framework/prompt/render_test.go`
- `internal/modules/extract/dnd/spells/prompt.go`
- `internal/modules/extract/dnd/spells/prompt_test.go`

### Prompt Decisions

Register:

- prompt ID: `dnd.spells`
- prompt constant: `DNDSpellsPromptID`
- prompt version: `v1`
- embedded directory: `assets/dnd/spells`

The system prompt must:

- include the shared prompt hardening text via `{{ hardening }}`;
- identify the task as extracting D&D spell casts only;
- state that the model must not infer spells not supported by the provided
  source text;
- state that source references must use source-unit IDs exactly as provided.

The user prompt must render:

- source document ID;
- chunk ID and chunk index when a chunk is present;
- source units in their existing order;
- each source unit's ID and text;
- selected metadata only as optional context, using generic labels. Include
  `speaker`, `start`, and `end` when present without importing Seriatim helper
  APIs.

Add a module-local prompt data builder that accepts
`contracts.ExtractionRequest` and returns template data. It must not mutate the
request, source document, chunk, units, or metadata maps.

### Required Tests

- Prompt metadata lookup succeeds for `dnd.spells`.
- Registered prompt metadata remains sorted.
- Rendering includes hardening text.
- Rendering includes source ID, chunk ID, unit IDs, unit text, and optional
  speaker/timestamp metadata when present.
- Rendering fails clearly if required template data is missing.
- Module-local prompt data construction works with a generic source chunk and
  does not depend on concrete Seriatim package helpers.

### Validation

Run:

```sh
gofmt -w internal/framework/prompt internal/modules/extract/dnd/spells
go test ./internal/framework/prompt
go test ./internal/modules/extract/dnd/spells
go test ./...
```

## Stage 4: Extractor LLM Call And Candidate Mapping

### Goal

Implement `Extractor.Extract` so a source chunk can flow through prompt
rendering, structured LLM completion, and candidate creation.

### Files To Add Or Update

- `internal/modules/extract/dnd/spells/extractor.go`
- `internal/modules/extract/dnd/spells/model.go`
- `internal/modules/extract/dnd/spells/extractor_test.go`

### Required Behavior

`Extract()` must:

- reject nil extractor, nil context, canceled context, nil source, nil chunk,
  empty chunk units, and nil LLM client with clear `dnd spells` error context;
- render the `dnd.spells` prompt using the stage 3 prompt data builder;
- look up the stage 2 response schema;
- call `req.LLMClient.CompleteStructured` with:
  - `StageName: Key`;
  - two messages, system then user;
  - `ResponseSchemaName: schema.Name`;
  - `ResponseSchema: schema.JSONSchema`;
- decode into `extractionResponse`;
- reject a nil `SpellCasts` slice as malformed structured output;
- return no candidates for an empty `SpellCasts` slice;
- preserve response order when creating candidates;
- trim `caster`, `spell`, `effect`, and `narrative_description` before
  marshaling the `SpellCast` payload;
- copy response source refs into `ArtifactCandidate.SourceRefs`;
- leave `ExtractorKey`, `ArtifactType`, `SchemaVersion`, and `Index` empty or
  zero so the runner's existing candidate normalization remains authoritative;
- wrap LLM, prompt, schema lookup, response validation, and JSON marshal errors
  with useful `dnd spells` context.

Do not call validators from `Extract()`. Validation belongs to the runner's
validator phase.

### Required Tests

Use a fake `contracts.StructuredLLMClient`.

- Successful extraction returns one candidate with expected payload fields,
  source refs, response schema name, schema JSON, prompt messages, and stage
  name.
- Empty `spell_casts: []` returns no candidates and no error.
- Missing or null `spell_casts` is rejected as malformed structured output.
- LLM client errors are wrapped with D&D spells context.
- Nil source, nil chunk, empty chunk units, nil LLM client, nil context, and
  canceled context are rejected.
- Multiple response spell casts produce candidates in response order.
- Candidate source refs are copied so later mutation of the fake response does
  not mutate returned candidates.

### Validation

Run:

```sh
gofmt -w internal/modules/extract/dnd/spells
go test ./internal/modules/extract/dnd/spells
go test ./...
```

## Stage 5: Deterministic Validator Chain

### Goal

Add module-owned deterministic validators for spell payload shape, required
fields, and source-reference grounding.

### Files To Add Or Update

- `internal/modules/extract/dnd/spells/validator.go`
- `internal/modules/extract/dnd/spells/validator_test.go`
- `internal/modules/extract/dnd/spells/extractor.go`

### Validator Decisions

Add two validators:

1. `ShapeValidator`
   - `Name()` returns `dnd/spells/shape`.
   - Rejects malformed JSON payloads with reason code `invalid_payload`.
   - Rejects blank `caster`, `spell`, `effect`, or `narrative_description`
     with reason code `missing_required_field`.
   - Approves candidates with valid payload shape and required fields.

2. `SourceRefValidator`
   - `Name()` returns `dnd/spells/source_refs`.
   - Rejects candidates with no source refs using reason code
     `missing_source_ref`.
   - Rejects any invalid source ref using reason code `invalid_source_ref` and
     the `source.ValidateRef` error message.
   - Approves candidates whose source refs all validate against the request
     source document.

`Extractor.Validators()` must return `ShapeValidator{}` followed by
`SourceRefValidator{}`. Return a fresh slice each time.

Both validators must:

- satisfy `contracts.Validator`;
- return one decision for every candidate;
- preserve candidate indexes in decisions;
- return an error, not rejection decisions, when called with nil source only if
  source-reference validation cannot run. Shape validation does not require a
  source document.

### Required Tests

- `Extractor.Validators()` returns the two validators in the required order and
  is mutation-safe.
- Each validator approves a valid candidate.
- Shape validator rejects malformed payload JSON.
- Shape validator rejects each blank required field.
- Source ref validator rejects missing refs.
- Source ref validator rejects unknown source IDs, unknown unit IDs, and
  reversed unit ranges.
- Both validators return one decision per candidate and preserve indexes.
- Approved decisions use `validate.Approved`; rejected decisions use the reason
  codes listed above.

### Validation

Run:

```sh
gofmt -w internal/modules/extract/dnd/spells
go test ./internal/modules/extract/dnd/spells
go test ./...
```

## Stage 6: Pipeline Resolution And Runner Integration

### Goal

Prove Seriatim input can flow through the existing runner into the real D&D
spells extractor, fake downstream infrastructure, and the extractor-owned
validators.

### Files To Add Or Update

- `internal/modules/extract/dnd/spells/config_test.go`
- `internal/modules/extract/dnd/spells/runner_test.go`
- `internal/modules/extract/dnd/spells/testdata/seriatim_spell_session.json`
- `internal/modules/extract/dnd/spells/testdata/pipeline.yml`

### Required Test Catalog

Build test-only catalogs and registries with:

- Seriatim input registered through `seriatim.Register`;
- a fake chunker registered as `fake/chunk`, requiring `source.transcript` and
  providing `chunks`;
- D&D spells extractor registered through `spells.Register`;
- `pipeline.AppendOrderMerger` registered as `appendorder`, requiring
  `dnd.spell_casts`;
- `pipeline.NoopNormalizer` registered as `noop`;
- a fake `json` output encoder registered as output stage.

Do not add real chunk or output modules for this checkpoint.

### YAML Fixture

Use a synthetic pipeline fixture:

```yaml
version: 1
pipelines:
  dnd-spells-fixture:
    input: seriatim
    chunk: fake/chunk
    artifacts:
      spells:
        extract: dnd/spells
        merge: appendorder
        normalize: noop
    output: json
```

### Required Tests

- `config.ParseFileConfigYAML` and `Config.ApplyFileConfig` load the fixture.
- `Config.Resolve` succeeds with pipeline ID `dnd-spells-fixture` and the
  test-only catalog.
- The resolved artifact lane ID is `spells` and extractor module is
  `dnd/spells`.
- The resolved pipeline digest is non-empty and stable across repeated
  resolution.
- Removing `source.transcript` from the Seriatim module spec causes resolution
  to fail with a missing capability error for `dnd/spells`.
- Removing `dnd.spell_casts` from the extractor spec causes resolution to fail
  with a missing capability error for `appendorder`.
- Selecting an unknown `--only` lane still fails through existing resolution
  behavior.
- Runner success path:
  - parse the Seriatim fixture;
  - use a fake LLM that returns at least two valid spell casts;
  - assert approved artifacts preserve response order;
  - assert approved payloads contain spell data;
  - assert approved artifact source refs validate with `source.ValidateRef`;
  - assert the manifest records input module `seriatim`, extractor
    `dnd/spells`, lane `spells`, and validation status `approved`.
- Runner rejection path:
  - fake LLM returns a spell cast with an invalid source ref;
  - runner completes with rejected artifacts and validation status `rejected`;
  - rejection contains validator `dnd/spells/source_refs` and reason code
    `invalid_source_ref`.
- Runner malformed-output path:
  - fake LLM returns or reports malformed structured output;
  - runner returns an extraction error with D&D spells context and failed
    validation status.

### Validation

Run:

```sh
gofmt -w internal/modules/extract/dnd/spells
go test ./internal/modules/extract/dnd/spells
go test ./internal/modules/input/seriatim
go test ./internal/framework/pipeline
go test ./...
```

## Stage 7: Documentation And Final Verification

### Goal

Document the implemented D&D spells extraction contract without describing
unimplemented extractors or a `notarius run` command.

### Files To Add Or Update

- `docs/integrations/dnd-spells.md`
- `docs/roadmap/6-dnd-spells-extractor.md`

### Required Documentation

Create `docs/integrations/dnd-spells.md` as implemented-behavior documentation
with:

- module key, artifact type, schema version, prompt ID, and response schema key;
- accepted source expectations: generic source document/chunk with transcript
  capability supplied by pipeline resolution;
- spell payload fields;
- source-reference behavior: LLM response includes refs, durable artifact
  output carries refs in the generic artifact envelope;
- default deterministic validators and rejection reason codes;
- declared required/provided capabilities;
- note that item, NPC, combat, encounter, broad D&D rules, and CLI `run`
  workflows are not implemented by this checkpoint.

Update `docs/roadmap/6-dnd-spells-extractor.md` only if implementation reveals
a real scope or policy correction. Keep implementation staging in this file,
not in the feature roadmap.

### Final Validation

Run:

```sh
gofmt -w internal/modules/extract/dnd/spells internal/framework/llm internal/framework/prompt
go test ./...
go build ./cmd/notarius
rm -f ./notarius
```

### Done Criteria

- `go test ./...` passes.
- `go build ./cmd/notarius` passes.
- The D&D spells extractor is registered through the extractor registry.
- Pipeline-profile resolution can select a `spells` artifact lane using
  extractor module `dnd/spells`.
- Seriatim minimal transcript JSON can flow through the runner into the D&D
  spells extractor in tests.
- Spell artifacts include valid generic source references in the artifact
  envelope.
- Invalid source references are rejected by the extractor-owned validator chain.
- D&D concepts do not appear in core runner, source, pipeline, config, prompt,
  LLM, or validator contracts.
- No CLI `run` behavior is documented or implemented in this checkpoint.

## Open Questions

None. This plan chooses the checkpoint-6 behavior needed to implement the
feature without requiring additional product decisions.