175 lines
7.7 KiB
Markdown
175 lines
7.7 KiB
Markdown
# D&D Validation-Boundary Alignment Implementation Plan
|
|
|
|
Status: Proposed.
|
|
|
|
Implement this plan in order. It repairs the immediate combat extraction
|
|
failure, makes D&D validation diagnostics consistently domain-owned, and then
|
|
aligns the spell and NPC private response boundaries with the same policy.
|
|
|
|
Do not change durable artifact schemas, artifact kinds, public Go types,
|
|
framework retry behavior, checkpoint formats, or the scene chunker. Existing
|
|
v1 private-schema identities may be corrected in place because Notarius has not
|
|
been run in production. Changed prompt and schema content will invalidate
|
|
development checkpoints through existing fingerprints.
|
|
|
|
## Cross-Stage Decisions
|
|
|
|
Private LLM schemas own the transport envelope: required fields, JSON types,
|
|
nullability, array/object shape, and unknown-field rejection. Deterministic
|
|
domain validators own semantic rules: supported enum values, nonblank values,
|
|
required non-empty collections, positive and resolvable source units,
|
|
catalog/identity policy, and normalized invariants.
|
|
|
|
For typed D&D artifacts, `generic/valid_json` remains first as a representation
|
|
sanity check. All deterministic domain validators that can reject a candidate
|
|
run next. `generic/valid_json_schema` runs after them as a durable-schema
|
|
backstop, followed by warning-only relatedness validators. This order gives
|
|
expected candidate failures bounded domain reason codes while retaining a final
|
|
check that typed encoding conforms to the durable contract.
|
|
|
|
Do not expose raw JSON Schema errors or candidate values through the generic
|
|
validator. Do not add tests that require particular words or phrases to remain
|
|
in prompt prose.
|
|
|
|
## Stage 1: Repair Combat Extraction
|
|
|
|
### Changes
|
|
|
|
- Update the combat extraction instructions to enumerate the complete allowed
|
|
values:
|
|
- `turn_kind`: `turn`, `reaction`, `legendary_action`, `lair_action`,
|
|
`other`;
|
|
- action `category`: `attack`, `spell`, `movement`, `item`,
|
|
`ability_check`, `saving_throw`, `condition`, `other`.
|
|
- Keep the private combat schema structurally permissive and the durable schema
|
|
strict. Do not restore enum, minimum, `minLength`, or `minItems` constraints
|
|
to the private schema.
|
|
- Reorder the combat extraction default chain to:
|
|
`generic/valid_json`, combat shape, combat source references,
|
|
`generic/valid_json_schema`, combat source relatedness.
|
|
- Reorder the combat normalization default chain to:
|
|
`generic/valid_json`, combat shape, normalized invariants, combat source
|
|
references, `generic/valid_json_schema`, combat source relatedness.
|
|
- Preserve configured validator overrides as authoritative; only production
|
|
default composition changes.
|
|
|
|
### Tests
|
|
|
|
- Add an assembled combat pipeline case whose raw LLM response contains an
|
|
unsupported turn kind or action category. Exhausted retries must produce a
|
|
non-fatal `invalid_combat_turn_shape` rejection owned by the combat shape
|
|
validator, not `json_schema_invalid` or a framework error.
|
|
- Retain coverage that a later valid retry succeeds and discarded-attempt
|
|
warnings/rejections do not become durable.
|
|
- Update registrar contract tests to assert the new extraction and
|
|
normalization order.
|
|
- Rely on behavioral enum/schema tests and prompt fingerprint coverage; do not
|
|
add prompt-word change-detector tests.
|
|
|
|
### Completion Check
|
|
|
|
Run `go test ./internal/modules/dnd/extract/combatturns
|
|
./internal/modules/dnd/validate/combatturns/... ./internal/modules/dnd/register
|
|
./internal/modules/integration` and `git diff --check`.
|
|
|
|
## Stage 2: Make D&D Validator Ordering Consistent
|
|
|
|
### Changes
|
|
|
|
- Move `generic/valid_json_schema` behind all rejecting domain validators in
|
|
every spell and NPC extraction and normalization default chain:
|
|
- spell extraction/normalization: shape, catalog, source references, schema,
|
|
then source relatedness;
|
|
- NPC extraction: shape, source references, schema, then source relatedness;
|
|
- NPC normalization: shape, identity, source references, schema, then source
|
|
relatedness.
|
|
- Keep `generic/valid_json` first and warning-only source relatedness last.
|
|
- Do not change validator implementations, reason codes, warning promotion,
|
|
retry counts, or user-provided chain order.
|
|
- Document the default-chain policy in the pipeline/module internals: domain
|
|
validators diagnose expected semantic failures and the generic schema
|
|
validator is the final rejecting representation backstop.
|
|
|
|
### Tests
|
|
|
|
- Update production registrar tests for every affected chain.
|
|
- Add one representative spell and NPC assembled rejection proving that a
|
|
domain-invalid but encodable candidate is attributed to the owning domain
|
|
validator rather than the generic schema validator. Do not duplicate each
|
|
validator package's existing case matrix at integration level.
|
|
- Confirm explicitly configured validator chains retain their exact configured
|
|
order.
|
|
|
|
### Completion Check
|
|
|
|
Run `go test ./internal/modules/dnd/register ./internal/modules/integration
|
|
./internal/framework/pipeline` and `git diff --check`.
|
|
|
|
## Stage 3: Align Spell and NPC Private Response Boundaries
|
|
|
|
### Changes
|
|
|
|
- Revise the existing v1 spell and NPC private schemas in place:
|
|
- retain required fields, JSON types, array/object structure,
|
|
`additionalProperties: false`, and omission of framework-assigned fields;
|
|
- remove `minLength`, `minItems`, and positive-number `minimum` constraints;
|
|
- leave the durable spell and NPC schemas unchanged.
|
|
- Replace `shared.UnitRef` in the private spell and NPC response DTOs with
|
|
integer candidates so zero and negative unit IDs survive decoding and mapping
|
|
into `source.SourceRef` for deterministic source validation.
|
|
- Update canonicalization and ordering helpers to operate on candidate integers
|
|
without repairing invalid values. Valid positive IDs retain current output,
|
|
ordering, and exact-deduplication behavior; invalid ranges remain available
|
|
to validators.
|
|
- Keep malformed JSON, missing/unknown fields, wrong JSON types, and
|
|
non-integer source IDs as LLM-boundary errors.
|
|
- Update LLM/module internals and the spell/NPC integration contracts to state
|
|
the structural-private/semantic-validator ownership boundary.
|
|
|
|
### Tests
|
|
|
|
- For each private schema, prove structurally valid candidates with blank
|
|
strings, empty required collections, and nonpositive unit IDs pass the
|
|
private schema, while missing fields, unknown fields, and wrong JSON types do
|
|
not.
|
|
- Through raw-JSON LLM fakes, prove semantic values survive decoding and mapping
|
|
without repair.
|
|
- Add representative assembled cases showing:
|
|
- blank or empty spell/NPC fields are rejected by the appropriate shape
|
|
validator;
|
|
- nonpositive or nonexistent unit IDs are rejected by the appropriate source
|
|
validator; and
|
|
- exhausted validation retries remain non-fatal rejected outputs.
|
|
- Preserve existing valid mapping, source-position ordering, deduplication,
|
|
catalog, identity, checkpoint-fingerprint, and durable codec tests.
|
|
|
|
### Completion Check
|
|
|
|
Run `go test ./internal/modules/dnd/extract/spells
|
|
./internal/modules/dnd/extract/npcs ./internal/modules/dnd/validate/spells/...
|
|
./internal/modules/dnd/validate/npcs/... ./internal/modules/integration` and
|
|
`git diff --check`.
|
|
|
|
## Final Verification
|
|
|
|
Run:
|
|
|
|
```text
|
|
git diff --check
|
|
go test ./...
|
|
go vet ./...
|
|
go build ./cmd/notarius
|
|
go test -race ./internal/modules/dnd/... ./internal/framework/pipeline ./internal/cli ./internal/modules/integration
|
|
```
|
|
|
|
Review current-behavior documentation for stale statements that private spell,
|
|
NPC, or combat schemas own semantic validation. Confirm the scene schema and
|
|
prompt remain unchanged: their enumerations are explicitly communicated and
|
|
scene-plan construction has a distinct structural mapping boundary.
|
|
|
|
## Open Questions
|
|
|
|
None. The stages above define the validation ownership, default ordering,
|
|
compatibility policy, diagnostic behavior, and test boundaries required for
|
|
implementation.
|