Prepare the v0.6.0 release
This commit is contained in:
84
docs/releases/v0.6.0.md
Normal file
84
docs/releases/v0.6.0.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Notarius v0.6.0
|
||||
|
||||
This release improves the reliability and ergonomics of unattended,
|
||||
subprocess-driven D&D extraction pipelines.
|
||||
|
||||
## Summary
|
||||
|
||||
Notarius now gives models more precise, semantically useful correction guidance
|
||||
when deterministic or model-backed validation rejects an otherwise structured
|
||||
candidate. Semantic reconciliation retries identify candidates with compact,
|
||||
request-local numbers, preserve valid candidates when a proposal cannot be
|
||||
repaired, and report bounded process warnings when reconciliation is
|
||||
incomplete. D&D extraction also canonicalizes safely resolvable reversed source
|
||||
ranges and applies more consistent schema, prompt, and validator constraints,
|
||||
improving successful operation with smaller models without weakening evidence
|
||||
validation.
|
||||
|
||||
An optional LLM-backed combat-semantics validator can review whether scene
|
||||
descriptions correctly consolidate combat and apply the `combat` kind. For
|
||||
subprocess callers, command-line references now follow the pipeline-scoped
|
||||
configuration model: a shared reference can be supplied once and automatically
|
||||
reach every compatible selected target, while lane- and binding-specific forms
|
||||
remain available for exceptions.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- The meaning of an unqualified `--reference slot=path` or
|
||||
`--without-reference slot` selector has changed. It now applies to every
|
||||
selected pipeline target that declares the slot instead of requiring exactly
|
||||
one matching target. Review callers that relied on ambiguity rejection or a
|
||||
unique implicit target; the current selector contract is documented in the
|
||||
[CLI reference](../cli.md).
|
||||
- The stage-wide `merge.slot` CLI reference shorthand has been removed. Use
|
||||
`lane.merge.slot` for an exact merge binding, `lane.slot` for all compatible
|
||||
bindings in one lane, or an unqualified `slot` for pipeline scope.
|
||||
- Command-line file references cannot replace, remove, or coexist with a
|
||||
generated artifact handoff for the same concrete target and slot. Resolution
|
||||
reports the conflict so the caller can narrow or remove the selector.
|
||||
- The private LLM-facing semantic-reconciliation contract now uses contiguous
|
||||
`candidate_number` values rather than application identities. This is not a
|
||||
durable or operator-configurable contract and requires no operator action.
|
||||
- Durable D&D artifact schemas, the `notarius.run-result.v2` receipt,
|
||||
`notarius.warnings.v2`, and `notarius.diagnostics.v1` are unchanged from
|
||||
`v0.5.0`.
|
||||
|
||||
## Upgrade
|
||||
|
||||
1. Update subprocess commands to provide shared inputs once with
|
||||
`--reference slot=path`, and convert any stage-wide `merge.slot` selector to
|
||||
an explicit supported scope. The [subprocess consumer guide](../consumers/subprocess.md)
|
||||
and [complete D&D consumer guide](../consumers/dnd-pipeline.md) show the
|
||||
current invocation pattern.
|
||||
2. Review CLI reference overrides that overlap generated same-run references;
|
||||
narrow or remove an external selector rather than attempting to replace the
|
||||
generated handoff.
|
||||
3. Run `notarius config validate --config <path> --pipeline <id>` for every
|
||||
deployed pipeline configuration.
|
||||
4. Optionally add
|
||||
`extract/dnd/scene-descriptions/combat_semantics` to the scene-description
|
||||
validator chain where the deployment wants LLM-backed combat-scene review;
|
||||
see the [configuration reference](../config.md).
|
||||
5. Perform a representative run with the deployed model profile and inspect
|
||||
the machine-readable receipt, warnings, and diagnostics before promotion.
|
||||
|
||||
## Changes
|
||||
|
||||
- Added feedback-aware, module-requested retry support for normalizers while
|
||||
keeping retry budgets bounded and preserving attempt diagnostics.
|
||||
- Reworked semantic reconciliation around compact, contiguous candidate
|
||||
numbers, explicit namespace guidance, complete proposal validation, and
|
||||
model-facing retry feedback that omits opaque internal identities.
|
||||
- Preserved candidates separately when an exhausted semantic proposal cannot
|
||||
be safely applied, with a bounded process warning describing the fallback.
|
||||
- Canonicalized safely resolvable reversed D&D source-reference endpoints
|
||||
before deterministic coverage validation and clarified the shared evidence
|
||||
prompt.
|
||||
- Improved D&D schemas, prompts, and validators with closed-value constraints,
|
||||
contextual correction guidance, shared source-range coverage logic, and
|
||||
clearer item-holder transition rules.
|
||||
- Added the optional D&D scene-description combat-semantics validator, shared
|
||||
combat policy assets, evaluation fixtures, and retry-path coverage.
|
||||
- Made unqualified CLI references pipeline-scoped, added hierarchical lane and
|
||||
exact-binding selectors, defined deterministic override precedence, and
|
||||
documented first-class subprocess use.
|
||||
@@ -1,258 +0,0 @@
|
||||
# Pipeline-Scoped CLI References
|
||||
|
||||
## Purpose
|
||||
|
||||
Make command-line reference binding match the pipeline-level mental model used
|
||||
by configuration and by subprocess callers. A caller should be able to supply
|
||||
each shared external reference once, while retaining explicit syntax for the
|
||||
less common case in which one lane or one stage binding needs a different
|
||||
source.
|
||||
|
||||
## Motivation And Current Problem
|
||||
|
||||
Notarius already accepts repeatable `--reference selector=path` and
|
||||
`--without-reference selector` flags. The current unqualified form,
|
||||
`--reference slot=path`, succeeds only when exactly one selected chunk, extract,
|
||||
merge, or normalize target declares that slot. If several targets consume a
|
||||
shared reference such as `party`, `glossary`, or `spell_catalog`, the command is
|
||||
rejected as ambiguous and the caller must repeat stage-qualified bindings.
|
||||
|
||||
That behavior is internally precise but does not match the public configuration
|
||||
model. A reference declared at pipeline scope is shared with every compatible
|
||||
target, while target-local configuration provides the exceptional override.
|
||||
It is therefore surprising for the least-qualified CLI syntax to mean "find one
|
||||
unique target" rather than "supply this reference to the pipeline."
|
||||
|
||||
The mismatch is especially costly for subprocess use. The primary complete D&D
|
||||
workflow expects an orchestrator to provide one transcript, one output root,
|
||||
and a small collection of campaign reference files. Requiring the orchestrator
|
||||
to know and enumerate every internal consumer of those shared files couples it
|
||||
to lane composition, makes commands needlessly long, and creates maintenance
|
||||
work whenever another compatible D&D module is added.
|
||||
|
||||
## Target CLI Contract
|
||||
|
||||
`--reference` remains repeatable, but selector qualification expresses scope:
|
||||
|
||||
| Form | Target scope |
|
||||
| --- | --- |
|
||||
| `slot=path` | Every selected pipeline target that declares `slot`. |
|
||||
| `chunk.slot=path` | The selected chunk binding. |
|
||||
| `lane.slot=path` | Every extract, merge, or normalize binding in `lane` that declares `slot`. |
|
||||
| `lane.extract.slot=path` | The extract binding in `lane`. |
|
||||
| `lane.merge.slot=path` | The merge binding in `lane`. |
|
||||
| `lane.normalize.slot=path` | The normalize binding in `lane`. |
|
||||
|
||||
The current stage-wide `merge.slot=path` shorthand is removed. A merge-specific
|
||||
override must name its lane as `lane.merge.slot=path`; this keeps the grammar
|
||||
hierarchical and avoids another uniqueness-dependent selector.
|
||||
|
||||
`--without-reference` uses the same selector forms without `=path`. It removes
|
||||
matching external bindings only. It never removes a generated artifact handoff,
|
||||
and resolution continues to reject a missing required reference.
|
||||
|
||||
Broad selectors are expected to match multiple compatible targets. They fail
|
||||
when they match no selected target, when the named lane is not selected, or
|
||||
when a stage-qualified target does not declare the slot. Errors should identify
|
||||
the selector and relevant scope without requiring callers to understand private
|
||||
resolver structures.
|
||||
|
||||
## Precedence And Conflict Policy
|
||||
|
||||
Command-line references are operational overrides and take precedence over
|
||||
external file references supplied at pipeline, step, lane, or binding scope in
|
||||
configuration. Among CLI selectors that affect the same concrete target:
|
||||
|
||||
1. an exact `lane.stage.slot` or `chunk.slot` selector wins over a lane-scoped
|
||||
selector;
|
||||
2. a lane-scoped selector wins over a pipeline-scoped selector; and
|
||||
3. the last occurrence wins among selectors with equal scope and action.
|
||||
|
||||
This ordering lets an orchestrator provide shared defaults once and express
|
||||
only genuine exceptions:
|
||||
|
||||
```sh
|
||||
--reference party=/refs/party.txt \
|
||||
--reference npc-registry.party=/refs/npc-party-context.txt
|
||||
```
|
||||
|
||||
Binding and unbinding the same concrete target at the same specificity is a
|
||||
configuration error rather than an argument-order-dependent result. A
|
||||
more-specific unbind may carve an exception out of a broader binding, and a
|
||||
more-specific binding may restore an exception to a broader unbind.
|
||||
|
||||
Generated artifact references remain a distinct source form. A CLI file
|
||||
reference must not silently replace, remove, or coexist with a generated
|
||||
handoff for the same concrete target and slot. Resolution fails with a
|
||||
target-specific conflict and directs the caller to narrow or remove the CLI
|
||||
selector. Same-run generated registry and eligibility handoffs in the complete
|
||||
D&D pipeline therefore remain controlled by pipeline composition.
|
||||
|
||||
CLI reference paths continue to resolve relative to the process working
|
||||
directory and retain CLI provenance. Subprocess guidance must recommend
|
||||
absolute paths. The reference source is auxiliary context and does not affect
|
||||
the generated prompt session identifier.
|
||||
|
||||
## Subprocess Target State
|
||||
|
||||
Passing shared references is a first-class part of the documented subprocess
|
||||
workflow. The complete D&D invocation should have this shape:
|
||||
|
||||
```sh
|
||||
notarius run dnd-session \
|
||||
--config /absolute/path/to/notarius.yml \
|
||||
--input /absolute/path/to/transcripts/final.trimmed.json \
|
||||
--output-dir /absolute/path/to/notarius-output \
|
||||
--reference party=/absolute/path/to/references/party.txt \
|
||||
--reference players=/absolute/path/to/references/players.txt \
|
||||
--reference glossary=/absolute/path/to/references/glossary.txt \
|
||||
--reference spell_catalog=/absolute/path/to/references/spells.json \
|
||||
--json
|
||||
```
|
||||
|
||||
Only references actually available to and desired by the deployment need to be
|
||||
passed. The selected modules continue to determine accepted slot names, media
|
||||
types, requiredness, and size limits. The orchestrator passes references as
|
||||
separate argument-vector elements rather than building a shell command, captures
|
||||
standard output and standard error separately, checks exit status, and uses the
|
||||
JSON receipt's `output_directory` to locate the generated run bundle.
|
||||
|
||||
The documentation must clearly distinguish:
|
||||
|
||||
- the input file supplied by `--input`;
|
||||
- the output *root* supplied by `--output-dir` and the run-specific output path
|
||||
returned in the receipt;
|
||||
- external file references supplied by `--reference`; and
|
||||
- generated references produced and consumed within the configured ordered
|
||||
pipeline.
|
||||
|
||||
## Required Implementation Changes
|
||||
|
||||
### CLI selector model and parsing
|
||||
|
||||
- Replace the unique-target interpretation of unqualified and lane-qualified
|
||||
selectors with explicit pipeline, lane, chunk, and exact-binding scopes.
|
||||
- Remove parsing and help text for the stage-wide `merge.slot` form.
|
||||
- Preserve repeatable flag handling, non-empty selector/path checks, supported
|
||||
stage names, and syntax-error exit classification.
|
||||
- Represent selector scope explicitly enough that precedence and diagnostics do
|
||||
not depend on inferring intent from empty fields.
|
||||
|
||||
### Target expansion and precedence
|
||||
|
||||
- Refactor CLI reference resolution to return every compatible selected target
|
||||
for a broad selector rather than requiring uniqueness.
|
||||
- Resolve overlapping bind and unbind requests into one deterministic action
|
||||
per concrete target and slot using the precedence policy above.
|
||||
- Produce exact `pipeline.ReferenceBinding` and `pipeline.ReferenceUnbind`
|
||||
values after CLI scoping is resolved. Keep broad-selector policy at the CLI
|
||||
boundary rather than adding CLI grammar or D&D knowledge to the generic
|
||||
pipeline framework.
|
||||
- Preserve CLI provenance and the existing rule that command-line paths are
|
||||
materialized relative to the working directory.
|
||||
- Detect generated-reference conflicts before file access or stage execution
|
||||
and retain required-slot validation after effective bindings are known.
|
||||
- Ensure lane selection is honored for compact pipelines. Explicit ordered
|
||||
pipelines continue to use their complete configured lane set because they do
|
||||
not support `--only`.
|
||||
|
||||
### Public and internal documentation
|
||||
|
||||
- Rewrite the reference-selector section of `docs/cli.md`, which is the
|
||||
canonical owner of flag syntax, selector semantics, precedence, and command
|
||||
errors.
|
||||
- Update `docs/consumers/subprocess.md` so reference-bearing invocation is part
|
||||
of the primary subprocess workflow rather than an unillustrated aside.
|
||||
- Update `docs/consumers/dnd-pipeline.md` with the complete D&D subprocess
|
||||
invocation and explain which references are external versus generated.
|
||||
- Lightly update `docs/config.md` to link the configuration hierarchy to the
|
||||
CLI override contract without duplicating CLI syntax.
|
||||
- Update `docs/internal/cli.md` and, only where necessary, the reference
|
||||
resolution discussion in `docs/internal/pipeline.md` to describe the
|
||||
implemented expansion and precedence boundary.
|
||||
- Do not expand the README quickstart or duplicate complete configuration
|
||||
files in prose. The maintained example configuration remains the canonical
|
||||
copyable pipeline definition.
|
||||
|
||||
No ADR is required. This is a deliberate public CLI usability correction that
|
||||
fits the existing architectural decisions: configuration remains centralized,
|
||||
operational overrides remain explicit, generated handoffs remain part of
|
||||
pipeline composition, and broad CLI syntax is translated into existing exact
|
||||
framework bindings at the application boundary. The CLI and configuration
|
||||
references are the durable owners of the resulting current behavior once it is
|
||||
implemented.
|
||||
|
||||
## Testing And Validation
|
||||
|
||||
Add lean offline behavioral coverage at the CLI contract boundary for:
|
||||
|
||||
- one pipeline-scoped reference reaching multiple compatible chunk and lane
|
||||
targets;
|
||||
- one lane-scoped reference reaching both extract and normalize bindings;
|
||||
- exact-binding and lane-scoped exceptions overriding broader values;
|
||||
- final-occurrence behavior at equal specificity;
|
||||
- broad selectors with no matches and selectors naming unselected lanes or
|
||||
undeclared slots;
|
||||
- bind/unbind specificity and same-specificity conflicts;
|
||||
- rejection of external/generated conflicts and protection of required
|
||||
generated handoffs;
|
||||
- working-directory resolution and CLI provenance for expanded bindings; and
|
||||
- removal of the `merge.slot` shorthand.
|
||||
|
||||
Prefer package-level resolution and representative `RunWithOptions` tests over
|
||||
duplicating the full selector matrix through end-to-end fixtures. Existing
|
||||
tests that encode unique-target ambiguity should be rewritten or removed rather
|
||||
than retained as historical change detectors. Do not assert complete error
|
||||
strings when stable classification and a concise semantic fragment provide
|
||||
sufficient confidence.
|
||||
|
||||
Validation for the completed change should include:
|
||||
|
||||
```sh
|
||||
go test ./internal/cli ./internal/core/config ./internal/framework/pipeline
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
Also validate both maintained example configurations against their selected
|
||||
pipelines, verify every changed documentation link, and manually review the
|
||||
D&D subprocess command against the actual flag parser and complete example
|
||||
configuration.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Passing reference contents directly in command-line arguments or environment
|
||||
variables.
|
||||
- Adding a reference-manifest file, glob syntax, wildcard selectors, or a
|
||||
second pipeline-reference flag.
|
||||
- Allowing the CLI to define undeclared reference slots or bypass module media
|
||||
type, size, UTF-8, requiredness, or materialization validation.
|
||||
- Replacing generated same-run artifacts with external files or changing
|
||||
ordered pipeline dependencies.
|
||||
- Changing configuration-file reference path resolution, prompt-session
|
||||
derivation, output directory allocation, or the run-result receipt.
|
||||
- Adding D&D-specific selector behavior to generic CLI or pipeline packages.
|
||||
- Preserving uniqueness-dependent `slot` or `lane.slot` behavior as an alias.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A subprocess caller can supply `party`, `players`, `glossary`, and
|
||||
`spell_catalog` once each and have every compatible selected D&D target
|
||||
receive the corresponding external file.
|
||||
- `slot=path` has pipeline scope and never fails merely because several selected
|
||||
targets declare the slot.
|
||||
- Lane and exact-binding selectors provide deterministic, documented
|
||||
exceptions with specificity-based precedence.
|
||||
- A broad selector that matches nothing is rejected before source parsing or
|
||||
LLM work.
|
||||
- CLI external references override configured external paths but never silently
|
||||
replace or remove generated artifact references.
|
||||
- All effective CLI bindings retain correct provenance and absolute-path
|
||||
materialization behavior.
|
||||
- `docs/cli.md` owns the exact public syntax, and both subprocess guides present
|
||||
shared CLI references as the primary orchestration workflow.
|
||||
- Current behavior outside `docs/roadmap/` is not documented until the code
|
||||
lands, and no complete example is duplicated outside `examples/`.
|
||||
- Focused tests, repository-wide tests, vetting, build, maintained configuration
|
||||
validation, and documentation-link review pass.
|
||||
@@ -1,240 +0,0 @@
|
||||
# D&D Validation Reliability Hardening
|
||||
|
||||
## Purpose
|
||||
|
||||
Improve the correctness and efficiency of D&D validation retries by giving the
|
||||
producer complete, contextual, semantically useful correction guidance; moving
|
||||
simple closed-value checks into compatible LLM-facing schemas; and removing
|
||||
duplicated source-range containment logic. The work should make one retry more
|
||||
likely to repair every detectable defect without weakening deterministic
|
||||
validation or exposing internal identifiers to a model.
|
||||
|
||||
## Current State And Findings
|
||||
|
||||
The validation architecture is coherent and the major safety boundaries are in
|
||||
place. Validators run through the framework-managed retry protocol, model-facing
|
||||
guidance is distinct from reason codes and operator diagnostics, schemas use
|
||||
provider-compatible required fields, and D&D extraction adapters canonically
|
||||
order safely resolvable source-reference endpoints before validation. The
|
||||
review nevertheless found four opportunities to make that architecture more
|
||||
consistent and effective.
|
||||
|
||||
### Generic correction guidance
|
||||
|
||||
Most rejecting D&D validators produce detailed operator-facing messages but
|
||||
pair them with a fixed, general correction sentence. The framework correctly
|
||||
constructs retry text from `CorrectionGuidance` alone, so the model does not see
|
||||
the internal diagnostic—and therefore often does not learn which records were
|
||||
wrong or how each record must change. Item-occurrence shape validation and the
|
||||
combat-semantics validator already demonstrate the intended contextual pattern.
|
||||
|
||||
Passing the operator message through unchanged is not an acceptable fix.
|
||||
Operator messages may contain Go paths, array indexes that do not reliably map
|
||||
back to canonicalized model output, reason codes, validator terminology, or
|
||||
opaque application identities. Those details are useful for diagnosis but are
|
||||
not appropriate model context.
|
||||
|
||||
### Missing closed-value schema constraints
|
||||
|
||||
Four private LLM response schemas describe categorical fields as unconstrained
|
||||
strings even though their prompts, durable artifact schemas, and deterministic
|
||||
validators define closed value sets:
|
||||
|
||||
- `combat-turns`: `turn_kind`;
|
||||
- `enemy-events`: `kind`;
|
||||
- `npc-occurrences`: `kind`; and
|
||||
- `scene-descriptions`: `kind`.
|
||||
|
||||
This defers an inexpensive structural check until the outer semantic-validation
|
||||
loop. Item and location occurrences already use the preferable private-schema
|
||||
enum pattern.
|
||||
|
||||
### First-defect validation
|
||||
|
||||
Spell shape validation returns after the first invalid cast field, and enemy
|
||||
engagement validation returns after the first duplicated engagement subject.
|
||||
When multiple defects exist, each retry can therefore reveal only one of them.
|
||||
That needlessly consumes the bounded producer-attempt budget and makes
|
||||
correction less reliable for smaller models.
|
||||
|
||||
### Duplicated source-range containment
|
||||
|
||||
Nine extraction source-reference validators independently determine whether a
|
||||
citation belongs to the current chunk. Five validate every source unit spanned
|
||||
by a reference, while four check only the endpoints. Those behaviors are
|
||||
equivalent for today's contiguous materialized chunks, but the duplicated
|
||||
implementations and different apparent contracts create a drift risk.
|
||||
Scene-description validation has a separate exact-one-scene range contract and
|
||||
should remain specialized.
|
||||
|
||||
## Target End State
|
||||
|
||||
- Every producer-correctable D&D rejection supplies bounded contextual guidance
|
||||
that identifies all affected records using transcript-grounded names, source
|
||||
ranges, and response fields as appropriate, explains each semantic defect,
|
||||
and requests one complete corrected replacement.
|
||||
- Model-facing guidance never contains reason codes, validator keys, Go field
|
||||
paths, opaque or hash-derived entity IDs, or raw internal error text.
|
||||
Operator diagnostics remain detailed and separately available.
|
||||
- Every detectable issue in a candidate is collected in one validator pass,
|
||||
subject to the repository's established diagnostic bounds. A retry is not
|
||||
spent merely to reveal the next defect.
|
||||
- Closed categorical fields are constrained by compatible enums in the private
|
||||
LLM schemas. Deterministic typed validators retain the same rules as
|
||||
defense-in-depth and as protection for non-LLM producers and later stages.
|
||||
- One D&D-owned helper implements full-span chunk containment for extraction
|
||||
evidence. All applicable source-reference validators use it and no
|
||||
module-local endpoint-only or coverage implementation remains.
|
||||
- Validator results remain immutable, deterministic, bounded, and suitable for
|
||||
the existing feedback-aware replacement-request protocol. Retry budgets,
|
||||
warning policy, durable artifact schemas, and accepted output semantics do
|
||||
not change.
|
||||
|
||||
## Required Work
|
||||
|
||||
### Contextual correction guidance
|
||||
|
||||
Audit each rejecting D&D validator according to the stage that can actually
|
||||
correct its result:
|
||||
|
||||
- Extraction validators should build model-facing issue descriptions from the
|
||||
candidate and its transcript context. Shape, registry-membership, identity,
|
||||
catalog, engagement, and source-reference validators should name the
|
||||
contextual artifact and cited range where useful, state the invalid value or
|
||||
relationship in plain language, and state the required replacement shape.
|
||||
- LLM-backed NPC, item, and location registry reconciliation should continue to
|
||||
translate its typed proposal issues through the shared category-to-prose
|
||||
renderer, including any domain-specific supplement.
|
||||
- Normalization-only deterministic invariant validators should retain useful
|
||||
operator diagnostics. They should not claim that a producer can directly
|
||||
repair deterministically derived ordering, identity, or normalization state.
|
||||
If such a rejection can reach a feedback-capable producer, guidance must be
|
||||
expressed only in terms of the source candidate that producer controls.
|
||||
- LLM-backed semantic validators may continue using their model explanation
|
||||
when it is bounded and semantically meaningful.
|
||||
|
||||
Introduce a small D&D-shared diagnostic utility only for demonstrated common
|
||||
mechanics such as stable grouping, de-duplication, contextual source-range
|
||||
rendering, and bounded correction aggregation. Domain validators must continue
|
||||
to own the meaning and prose of their rules. The generic pipeline must not gain
|
||||
D&D knowledge, and operator `Message` values must never be mechanically copied
|
||||
into `CorrectionGuidance`.
|
||||
|
||||
Update validator policy fingerprints wherever correction behavior changes so
|
||||
checkpoints created under generic feedback are not reused as though the policy
|
||||
were identical.
|
||||
|
||||
### Private-schema enum guardrails
|
||||
|
||||
Add the existing supported value sets to the four private LLM schemas:
|
||||
|
||||
- `assets/dnd/combat-turns/schemas/dnd_combat_turns_llm.v1.json`;
|
||||
- `assets/dnd/enemy-events/schemas/dnd_enemy_events_llm.v1.json`;
|
||||
- `assets/dnd/npc-occurrences/schemas/dnd_npc_occurrences_llm.v1.json`; and
|
||||
- `assets/dnd/scene-descriptions/schemas/dnd_scene_descriptions_llm.v1.json`.
|
||||
|
||||
Use the exact values owned by the corresponding durable contract and typed
|
||||
domain constants. Keep all object properties required, retain current nullable
|
||||
types where present, and do not introduce `uniqueItems`, optional properties,
|
||||
or conditional schema logic. Revise schema tests that currently accept unknown
|
||||
values, and rely on computed asset fingerprints to invalidate incompatible
|
||||
LLM-output checkpoints. Keep deterministic enum validation in place.
|
||||
|
||||
### Complete per-attempt issue collection
|
||||
|
||||
Refactor spell shape and enemy engagement validation to inspect the complete
|
||||
candidate and collect every detectable violation before returning. Build the
|
||||
operator diagnostic and contextual correction request from the same evaluated
|
||||
issue set while preserving their different audiences. De-duplicate repeated
|
||||
semantic instructions, retain enough contextual identification for every
|
||||
affected record, and use the established bounded diagnostic behavior rather
|
||||
than an unbounded error string.
|
||||
|
||||
Review the surrounding D&D validators while applying the contextual-guidance
|
||||
change. Remove any additional accidental early exits that prevent independent
|
||||
candidate defects from being reported together, but retain immediate returns
|
||||
for request-level prerequisites whose absence makes further inspection unsafe
|
||||
or meaningless.
|
||||
|
||||
### Shared full-span chunk containment
|
||||
|
||||
Add one helper under `internal/modules/dnd/shared` that determines whether a
|
||||
source reference's complete document-ordered span is contained by a chunk. The
|
||||
helper must:
|
||||
|
||||
- validate source identity;
|
||||
- resolve endpoints through `source.DocumentIndex` rather than numeric-ID
|
||||
assumptions;
|
||||
- require every unit in the inclusive span to be present in the chunk;
|
||||
- handle nil or unresolved inputs without panic;
|
||||
- avoid mutating the source, chunk, or reference; and
|
||||
- leave source-reference validity and error wording to the consuming validator.
|
||||
|
||||
Use the helper from the spell, NPC-registry, NPC-occurrence, item-registry,
|
||||
item-occurrence, location-registry, location-occurrence, combat-turn, and
|
||||
enemy-event source-reference validators. Keep the scene-description exact-range
|
||||
validator separate because it enforces a materially different contract.
|
||||
|
||||
## Testing And Documentation
|
||||
|
||||
Add lean offline behavioral coverage at the narrowest stable boundary:
|
||||
|
||||
- correction tests should prove that multiple contextual defects produce one
|
||||
actionable, bounded request and that internal identifiers and diagnostic
|
||||
syntax are absent; they should not snapshot exact prose or message length;
|
||||
- schema tests should prove rejection of representative unsupported categorical
|
||||
values and acceptance of the supported sets without duplicating every
|
||||
provider behavior;
|
||||
- spell-shape and engagement tests should prove that independent defects are
|
||||
reported together;
|
||||
- the shared containment helper should own the full case matrix, including
|
||||
partial spans, non-monotonic unit IDs, wrong sources, unresolved endpoints,
|
||||
and nil inputs; consuming validators need only enough coverage to prove they
|
||||
use the common policy; and
|
||||
- existing validator tests should be simplified when the shared helper makes
|
||||
module-local cases redundant.
|
||||
|
||||
Update `docs/internal/dnd.md` to document the durable conventions for
|
||||
contextual correction guidance, complete issue collection, private-schema enum
|
||||
guardrails, and shared full-span extraction containment. Update
|
||||
`docs/internal/pipeline.md` only if framework behavior changes; the intended
|
||||
work applies its existing contract and should normally require only a link or
|
||||
no change. No new ADR is required because ADR-0014 and the architecture policy
|
||||
already decide the separation between semantic guidance and internal
|
||||
diagnostics. Create an ADR only if implementation requires changing that
|
||||
framework-level decision.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Passing operator diagnostics or internal errors directly to an LLM.
|
||||
- Exposing durable IDs, hashes, reason codes, validator keys, or raw provider
|
||||
responses in correction text.
|
||||
- Changing stage retry counts, PromptKit structural-repair budgets, validator
|
||||
failure policy, or warning classification.
|
||||
- Silently repairing domain-semantic defects or weakening deterministic
|
||||
validators after adding schema guardrails.
|
||||
- Adding provider-sensitive schema constructs beyond simple enums.
|
||||
- Moving D&D-specific behavior into the generic pipeline framework.
|
||||
- Generalizing the scene-description exact-range contract into the shared
|
||||
extraction containment helper.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Every producer-correctable D&D rejection reviewed in this work gives the next
|
||||
attempt all currently detectable, actionable corrections in contextual prose.
|
||||
- No model-facing correction request contains an opaque entity ID, hash,
|
||||
validator key, reason code, Go-style field path, or unfiltered operator error.
|
||||
- Spell shape and enemy engagement validation aggregate independent defects in
|
||||
one pass and keep outputs immutable.
|
||||
- The four private LLM schemas reject unsupported categorical values, retain
|
||||
provider-compatible required-only shapes, and leave durable v1 contracts
|
||||
unchanged.
|
||||
- All nine applicable extraction source-reference validators use one D&D-shared
|
||||
full-span containment policy; scene descriptions retain their specialized
|
||||
exact-range check.
|
||||
- Validator and asset fingerprints change wherever their effective policy
|
||||
changes, preventing reuse of stale checkpoints.
|
||||
- Canonical internal documentation records the conventions future D&D
|
||||
validators must follow.
|
||||
- Focused tests, `go test ./...`, `go vet ./...`, and
|
||||
`go build ./cmd/notarius` pass in a supported development environment.
|
||||
@@ -14,8 +14,9 @@ eligibility, and the separation of actionable process warnings from quality
|
||||
diagnostics. The remaining near-term work applies those completed foundations
|
||||
to domain review and empirical evaluation.
|
||||
|
||||
The active D&D reliability work is defined by
|
||||
[Feedback-Aware Semantic-Reconciliation Retries](semantic-reconciliation-retry-feedback.md).
|
||||
Near-term reliability work should now be selected from the concrete evaluation
|
||||
and extension opportunities below. The retry, validation, and subprocess
|
||||
foundations described above are implemented current behavior.
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
|
||||
@@ -1,95 +0,0 @@
|
||||
# D&D Item-Occurrence Holder Reliability
|
||||
|
||||
## Purpose
|
||||
|
||||
Improve item-occurrence extraction reliability for smaller models by making
|
||||
holder rules easier to follow and semantic retry feedback specific enough to
|
||||
correct a rejected response.
|
||||
|
||||
## Problem
|
||||
|
||||
The item-occurrence prompt correctly defines the permitted `from` and `to`
|
||||
values for every occurrence kind, but presents the rules in dense prose. The
|
||||
private schema requires both nullable fields but cannot express their
|
||||
cross-field relationship. The deterministic shape validator correctly rejects
|
||||
incompatible combinations, yet its model-facing correction guidance only asks
|
||||
for generally valid holder combinations. Notarius therefore resends the
|
||||
defective response without telling the model which contextual records failed or
|
||||
what their corrected holder shape must be.
|
||||
|
||||
## Target State
|
||||
|
||||
- The module instructions present one compact, unambiguous holder matrix:
|
||||
`discovered` uses two nulls; `acquired` uses a null `from` and non-null `to`;
|
||||
`lost` and `consumed` use a non-null `from` and null `to`; and `transferred`
|
||||
uses two distinct named party members.
|
||||
- The private LLM schema continues requiring `quantity`, `from`, and `to` with
|
||||
nullable types, and constrains `kind` to the five supported values. The
|
||||
durable v1 artifact contract remains unchanged.
|
||||
- A holder-combination rejection supplies bounded, semantically meaningful
|
||||
correction guidance that identifies affected occurrences by contextual item
|
||||
name and transcript evidence where useful, states the required JSON
|
||||
null/non-null shape, and requests a complete corrected replacement.
|
||||
- Correction guidance never exposes durable item IDs, hashes, validator keys,
|
||||
reason codes, or other opaque implementation identifiers. It does not rely
|
||||
solely on mapped array indexes, because canonical extraction ordering may
|
||||
differ from the raw response order appended to the retry prompt.
|
||||
- Operator-facing validation messages remain specific and include enough
|
||||
information to distinguish a missing holder from an extra or misplaced
|
||||
holder.
|
||||
- The existing validator remains strict. Holder combinations are not silently
|
||||
rewritten: unlike reversed evidence endpoints, changing a holder can conceal
|
||||
a misclassified discovery, acquisition, loss, consumption, or transfer.
|
||||
|
||||
## Required Work
|
||||
|
||||
1. Rewrite the holder paragraph in
|
||||
`assets/dnd/item-occurrences/prompts/instructions.md` as a compact matrix,
|
||||
retaining the existing classification and currency rules without adding
|
||||
contradictory repetition.
|
||||
2. Add the five supported `kind` values as an enum in the private item-occurrence
|
||||
response schema. Keep every listed field required and keep `quantity`,
|
||||
`from`, and `to` explicitly nullable.
|
||||
3. Refactor the item-occurrence shape validator to build detailed bounded
|
||||
operator issues and separate actionable correction guidance from the same
|
||||
evaluated candidate. Group repeated holder failures where practical while
|
||||
retaining contextual record identification and every distinct correction
|
||||
rule needed by the model.
|
||||
4. Preserve the generic validation retry protocol: the complete original
|
||||
prompt, exact defective assistant response, and one user correction message.
|
||||
Do not add a module-local retry loop.
|
||||
5. Ensure the changed prompt and schema produce new computed fingerprints, and
|
||||
bump the validator policy fingerprint so stale checkpoints cannot be reused
|
||||
across the behavior change. Update internal documentation where needed; the
|
||||
external item-occurrence contract requires no semantic change.
|
||||
6. Add lean offline regression coverage for the holder matrix, schema enum,
|
||||
contextual correction guidance, bounded aggregation, absence of opaque IDs,
|
||||
and a rejected-then-corrected extraction attempt. Avoid exact prose snapshots
|
||||
and live-provider tests.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Automatically deleting, moving, or inferring holder values.
|
||||
- Changing the durable item-occurrence schema or event taxonomy.
|
||||
- Encoding the complete semantic holder matrix through provider-sensitive
|
||||
conditional JSON Schema constructs.
|
||||
- Adding warnings for ordinary semantic rejection or successful correction.
|
||||
- Tuning unrelated D&D extraction prompts or validators.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- The maintained item-occurrence prompt shows the clear holder matrix,
|
||||
and the private schema rejects unsupported kind strings while remaining
|
||||
compatible with strict structured-output providers.
|
||||
- A rejected `acquired` occurrence receives guidance that explicitly requires
|
||||
`from: null` and a non-null `to`; equivalent exact guidance exists for the
|
||||
other four kinds.
|
||||
- Multiple invalid occurrences yield one bounded correction request containing
|
||||
all distinct required fixes and contextual identifiers that can be matched to
|
||||
the appended raw response.
|
||||
- No correction message contains an item ID, hash, validator key, or reason
|
||||
code, and the raw model candidate remains byte-faithful.
|
||||
- Valid candidates and the durable output contract are unchanged; invalid
|
||||
holder combinations still fail after configured retries are exhausted.
|
||||
- Focused tests, `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius`
|
||||
pass.
|
||||
@@ -1,270 +0,0 @@
|
||||
# Semantic Reconciliation Candidate Numbers
|
||||
|
||||
## Purpose
|
||||
|
||||
Make the private semantic-reconciliation protocol easier for smaller models to
|
||||
follow by clearly separating request-local candidate selection from transcript
|
||||
unit addressing. Replace the ambiguous `candidate_id` vocabulary with explicit
|
||||
one-based candidate numbers, declare the valid number range for every request,
|
||||
and ensure both initial and corrective instructions describe the exact response
|
||||
the model can produce.
|
||||
|
||||
## Motivation And Current State
|
||||
|
||||
The domain-neutral semantic-reconciliation core supplies each eligible registry
|
||||
candidate with a contiguous, one-based integer handle. The model returns groups
|
||||
of those handles, deterministic code resolves them to retained typed records,
|
||||
and unsafe groups are discarded without merging their candidates. This is the
|
||||
architecture required by ADR-0013 and correctly prevents model-generated values
|
||||
from becoming durable entity identity.
|
||||
|
||||
The current private JSON vocabulary calls these handles `candidate_id`,
|
||||
`candidate_ids`, and `canonical_candidate_id`. Candidate material and transcript
|
||||
context also contain `start_unit_id`, `end_unit_id`, and transcript unit `id`
|
||||
values. Although the shared protocol says to use only supplied candidate IDs,
|
||||
all of these values are nearby integers and the term “ID” does not clearly
|
||||
communicate which namespace is request-local.
|
||||
|
||||
An observed small-model run exhausted semantic-proposal retries after returning
|
||||
a duplicate group with an unknown candidate member and fewer than two remaining
|
||||
distinct valid members. The safe fallback behaved correctly: the group was
|
||||
omitted and all affected items were preserved separately. The issue pattern is
|
||||
consistent with confusing a transcript or evidence unit ID for a candidate
|
||||
handle, though an ordinary warning intentionally does not retain the exact
|
||||
model response needed to prove that inference.
|
||||
|
||||
The item-specific semantic instructions contain a related mismatch. They tell
|
||||
the model to choose a supplied candidate name as canonical even though the
|
||||
private response can return only the handle of the candidate whose label should
|
||||
be retained.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Use candidate-number vocabulary throughout the private protocol
|
||||
|
||||
The model-visible candidate material will use `candidate_number`. Duplicate
|
||||
groups will use `candidate_numbers` and `canonical_candidate_number`. Matching
|
||||
Go names, internal comments, diagnostics, tests, and current-behavior internal
|
||||
documentation will use “candidate number” where they refer to this private
|
||||
request-local value.
|
||||
|
||||
The rename applies coherently to the generic semantic-reconciliation core and
|
||||
all current consumers: NPC, item, and location registry normalization. It does
|
||||
not change the abstract architectural term “request-local candidate handle” in
|
||||
ADR-0013; a candidate number remains the concrete representation of that
|
||||
handle.
|
||||
|
||||
The old JSON property names will not be accepted as aliases. No compatibility
|
||||
adapter, dual-field schema, migration reader, or legacy prompt will remain.
|
||||
This is a private model contract, not a durable artifact contract.
|
||||
|
||||
### Declare the valid range in each request
|
||||
|
||||
Prepared candidate material will contain an explicit request-level range:
|
||||
|
||||
```json
|
||||
{
|
||||
"candidate_number_range": {
|
||||
"first": 1,
|
||||
"last": 12
|
||||
},
|
||||
"candidates": [
|
||||
{
|
||||
"candidate_number": 1,
|
||||
"label": "Example",
|
||||
"source_refs": []
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`first` and `last` are required integers. For every model invocation, `first`
|
||||
is 1 and `last` is the number of eligible candidates in the prepared request.
|
||||
The candidate list remains contiguous and one-based. Preparation owns this
|
||||
derived declaration and must keep it consistent with the emitted candidate
|
||||
objects and retained mappings.
|
||||
|
||||
The initial shared protocol will state that valid candidate numbers are exactly
|
||||
the declared inclusive range and that the model must use the explicit
|
||||
`candidate_number` attached to each candidate. Transcript unit `id` values and
|
||||
evidence `start_unit_id` and `end_unit_id` values identify source positions;
|
||||
they are never candidate numbers and must never appear in the response merely
|
||||
because they are integers.
|
||||
|
||||
### Make retry guidance request-aware
|
||||
|
||||
When proposal assessment finds a non-positive or unknown member or canonical
|
||||
number, the model-facing correction will include the request’s exact valid
|
||||
inclusive candidate-number range. It will retain response-local duplicate-group
|
||||
ordinals so the model can locate the defective group in the appended prior
|
||||
response. It will explicitly distinguish candidate numbers from transcript and
|
||||
evidence unit IDs.
|
||||
|
||||
Correction guidance must remain bounded and semantically meaningful. It must
|
||||
not expose internal issue categories, reason codes, Go paths, durable entity
|
||||
IDs, source identities, or raw operator errors. The framework will continue to
|
||||
append the exact latest defective response separately under the existing
|
||||
replacement-request protocol; correction prose must not duplicate it.
|
||||
|
||||
### Describe canonical selection in terms of the actual response
|
||||
|
||||
The item-registry normalization instructions will say to set
|
||||
`canonical_candidate_number` to the supplied candidate number whose label is
|
||||
the clearest transcript-supported designation. Equivalent NPC and location
|
||||
instructions must be reviewed for the same mismatch and corrected if present.
|
||||
Domain instructions continue to decide semantic equivalence and preferred
|
||||
canonical labels; the shared protocol continues to own selection mechanics.
|
||||
|
||||
## Target End State
|
||||
|
||||
- Every semantic-reconciliation prompt presents one unambiguous, request-local
|
||||
candidate-number namespace.
|
||||
- Candidate material declares the exact valid inclusive range and assigns every
|
||||
eligible candidate one explicit number in that range.
|
||||
- Transcript and evidence unit identifiers are clearly identified as a
|
||||
separate namespace that is never valid in proposal selection fields.
|
||||
- Structured responses contain only `duplicate_groups`,
|
||||
`candidate_numbers`, and `canonical_candidate_number` at their existing
|
||||
logical levels; the former `candidate_id` property family is rejected.
|
||||
- Proposal assessment remains deterministic, validates exact membership in the
|
||||
prepared candidate set, discards unsafe groups, applies independent safe
|
||||
groups, and preserves affected candidates separately on fallback.
|
||||
- A retry for an invalid selection tells the model which response-local groups
|
||||
need correction and the exact valid candidate-number range without exposing
|
||||
internal diagnostic vocabulary.
|
||||
- NPC, item, and location registry reconciliation use the same generic private
|
||||
protocol, while their typed consolidation and identity rules remain
|
||||
domain-owned.
|
||||
- Durable registry schemas, artifact IDs, evidence ranges, warning policy,
|
||||
retry budgets, and PromptKit structural-repair budgets do not change.
|
||||
|
||||
## Required Work
|
||||
|
||||
### Generic protocol and model
|
||||
|
||||
- Rename the model-facing fields in the generic candidate material, proposal
|
||||
response types, private response schema, shared protocol prompt, and
|
||||
presentation prompt.
|
||||
- Rename internal semantic-reconciliation fields and accessors where they
|
||||
represent request-local candidate numbers, so implementation terminology
|
||||
does not immediately drift back toward ambiguous IDs.
|
||||
- Add the required `candidate_number_range` object to prepared material and
|
||||
derive it from the filtered eligible candidate set.
|
||||
- Preserve the current contiguous one-based assignment, immutable retained
|
||||
mapping, bounded material limits, stable plan ordering, and exact typed
|
||||
resolution behavior.
|
||||
- Keep exact candidate membership as deterministic semantic validation. The
|
||||
shared static response schema may enforce positive integers and structural
|
||||
shape, but it must not pretend to encode a request-specific maximum that it
|
||||
cannot know.
|
||||
|
||||
### Request-aware corrective feedback
|
||||
|
||||
- Make the valid candidate-number range available to the generic correction
|
||||
renderer through owned preparation or reconciliation result state.
|
||||
- Add the exact range and the candidate-number/source-unit distinction to
|
||||
corrections for unknown or non-positive proposal members and canonical
|
||||
selections. Retain existing guidance for repeated members, undersized groups,
|
||||
canonical membership, and overlapping groups.
|
||||
- Preserve stable grouping, de-duplication, response-local group ordinals, byte
|
||||
bounds, and separation between operator diagnostics and model guidance.
|
||||
- Ensure NPC, item, and location normalizers use the generic renderer rather
|
||||
than duplicating range or namespace language. Item-specific currency
|
||||
guidance remains an additive domain rule.
|
||||
|
||||
### D&D semantic instructions
|
||||
|
||||
- Rewrite the item-registry canonical-selection sentence around
|
||||
`canonical_candidate_number` and the selected candidate’s label.
|
||||
- Review the NPC- and location-registry normalization instructions and apply
|
||||
the same correction wherever they describe returning a name rather than
|
||||
selecting its candidate number.
|
||||
- Do not duplicate the generic number-range or namespace rules in D&D-specific
|
||||
files; those rules belong in the mandatory shared protocol.
|
||||
|
||||
### Fingerprints and checkpoints
|
||||
|
||||
- Revise the existing private v1 prompt and schema assets in place; do not add
|
||||
legacy schema variants or compatibility aliases.
|
||||
- Ensure changed protocol, candidate-material, response-schema, correction,
|
||||
and typed-policy behavior participates in the established prompt, asset, and
|
||||
checkpoint fingerprint mechanisms. Explicitly bump a core or local policy
|
||||
fingerprint wherever the effective behavior is not already covered by an
|
||||
asset digest.
|
||||
- A checkpoint produced under the former candidate-ID contract must not be
|
||||
reused as compatible with the candidate-number contract.
|
||||
|
||||
## Testing
|
||||
|
||||
Add lean offline coverage at the narrowest stable boundaries:
|
||||
|
||||
- preparation tests should prove contiguous candidate numbers, an exact
|
||||
matching range declaration, filtering behavior, immutable mappings, and the
|
||||
absence of former model-facing field names;
|
||||
- private-schema tests should accept the candidate-number proposal shape and
|
||||
reject the former candidate-ID fields, missing required properties, unknown
|
||||
properties, non-integer numbers, non-positive numbers, and undersized arrays;
|
||||
- proposal-assessment tests should retain the existing safety matrix using the
|
||||
renamed vocabulary and prove that exact prepared-set membership still owns
|
||||
acceptance;
|
||||
- correction tests should prove that an unknown candidate number produces the
|
||||
exact valid range and namespace distinction without internal issue labels,
|
||||
reason codes, or operator paths;
|
||||
- prompt tests should prove selection of the shared protocol and the corrected
|
||||
domain instruction, without snapshotting complete prose or exact message
|
||||
lengths; and
|
||||
- representative NPC, item, and location normalizer tests should prove safe
|
||||
application, corrective retry, and exhaustion fallback through the renamed
|
||||
private contract.
|
||||
|
||||
Do not add live-provider tests to the default suite. After implementation, a
|
||||
maintainer may repeat the previously problematic small-model workload as
|
||||
observational validation, preferably with an explicitly requested debug bundle
|
||||
if the warning persists.
|
||||
|
||||
## Documentation
|
||||
|
||||
Update `docs/internal/modules.md` as the canonical description of the generic
|
||||
semantic-reconciliation mechanism and `docs/internal/dnd.md` for its three D&D
|
||||
registry consumers. Describe candidate numbers as request-local handles,
|
||||
including the declared range and namespace separation, without repeating the
|
||||
private JSON schema in both documents.
|
||||
|
||||
No new ADR is required. This work reinforces ADR-0012’s prohibition on opaque
|
||||
model-reproduced identity and ADR-0013’s accepted request-local-handle design;
|
||||
it changes only the concrete private vocabulary and reliability presentation.
|
||||
The architecture policy already owns the durable invariant and need not change
|
||||
unless implementation reveals a genuinely new architectural decision.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing durable NPC, item, or location artifact schemas or identifiers.
|
||||
- Asking the model to return candidate labels, evidence ranges, transcript
|
||||
text, source identities, or replacement records.
|
||||
- Accepting both candidate-ID and candidate-number response shapes.
|
||||
- Guessing that an unknown number denotes a transcript unit, list position, or
|
||||
different candidate and silently repairing the proposal.
|
||||
- Weakening deterministic proposal validation or typed domain guards.
|
||||
- Increasing stage retries or PromptKit structural-repair attempts to mask an
|
||||
ambiguous protocol.
|
||||
- Introducing a request-specific dynamic response schema in this work set.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- No model-facing semantic-reconciliation asset or private JSON shape uses the
|
||||
former `candidate_id`, `candidate_ids`, or `canonical_candidate_id` fields.
|
||||
- Candidate material and retry guidance expose the exact valid inclusive
|
||||
candidate-number range for the current request.
|
||||
- Shared instructions explicitly prohibit using transcript unit IDs or evidence
|
||||
range endpoints as candidate numbers.
|
||||
- Item, NPC, and location canonical-selection instructions refer to the actual
|
||||
candidate-number response contract.
|
||||
- Invalid or ambiguous proposals remain safely rejected without false merges;
|
||||
retry exhaustion preserves affected candidates separately and emits the
|
||||
existing process warning.
|
||||
- Durable outputs and public schemas are unchanged, while incompatible private
|
||||
checkpoints are invalidated.
|
||||
- Canonical internal documentation records the implemented terminology and
|
||||
ownership boundaries.
|
||||
- Focused tests, `go test ./...`, `go vet ./...`, and
|
||||
`go build ./cmd/notarius` pass in a supported development environment.
|
||||
@@ -1,117 +0,0 @@
|
||||
# Feedback-Aware Semantic-Reconciliation Retries
|
||||
|
||||
## Purpose
|
||||
|
||||
Improve semantic registry reconciliation for smaller models by turning the
|
||||
existing module-requested retry into a bounded corrective exchange. Preserve
|
||||
the current deterministic safety checks, shared retry budget, and safe fallback
|
||||
when correction is unsuccessful.
|
||||
|
||||
## Problem
|
||||
|
||||
NPC, item, and location registry normalizers can reject unsafe duplicate-group
|
||||
proposals and retain a safe partial result. They currently ask the pipeline for
|
||||
another attempt, but the next attempt receives the original prompt without the
|
||||
defective response or an explanation of the defect. The normalizers also build
|
||||
their retry messages from internal issue-category strings, which are suitable
|
||||
for diagnostics but not useful model instructions. If all attempts are
|
||||
exhausted, the warning reports the omitted-group count without explaining the
|
||||
final defect or confirming that affected candidates were preserved separately.
|
||||
|
||||
## Target State
|
||||
|
||||
- A normalizer retry directive may carry model-facing correction guidance
|
||||
separately from its stable reason code, operator message, and fallback
|
||||
diagnostics.
|
||||
- When the directive includes correction guidance and a valid
|
||||
`single_response_v1` model candidate, the shared producer-attempt state
|
||||
machine reconstructs the complete original prompt and appends the exact
|
||||
defective response as an assistant message followed by one user correction
|
||||
message. Earlier failed attempts do not accumulate.
|
||||
- This correction consumes the existing normalize-stage retry budget. It does
|
||||
not add a module-local loop or alter PromptKit's independent structured-output
|
||||
repair budget.
|
||||
- A retry caused by structurally invalid output remains a fresh stage attempt
|
||||
without semantic feedback when no valid model candidate is available.
|
||||
- Internal issue categories remain available for operator and debug
|
||||
diagnostics, but are never sent to a model. A shared renderer translates
|
||||
every known category into concise prose that explains the defect and the
|
||||
required valid shape.
|
||||
- Item reconciliation appends its domain rule when a proposal attempts to
|
||||
combine different currency denominations or currency with a non-currency
|
||||
item. NPC and location reconciliation use only the shared protocol guidance.
|
||||
- On final exhaustion, safe proposal groups remain applied, unsafe groups
|
||||
remain separate, and one process warning gives a bounded summary of the final
|
||||
defect and explicitly states that the affected candidates were preserved
|
||||
separately.
|
||||
|
||||
## Required Work
|
||||
|
||||
1. Extend `contracts.NormalizeRetry` and its clone, validation, and debug paths
|
||||
with an optional, bounded model-facing correction-guidance field. Keep
|
||||
reason codes and operator messages out of model input. Require guidance to
|
||||
be paired with a valid `single_response_v1` model candidate; preserve the
|
||||
existing feedback-free directive for structural retry cases.
|
||||
2. Extend the generic producer-attempt state machine so a feedback-capable
|
||||
module retry constructs `SemanticCorrection` from the latest exact response
|
||||
and the supplied guidance. Preserve attempt provenance, cancellation,
|
||||
terminal policy, diagnostic handling, byte-faithful response provenance,
|
||||
and the rule that only the latest response is included.
|
||||
3. Add a model-facing renderer to `internal/framework/semanticreconcile`.
|
||||
Implement it with a typed lookup table keyed by `IssueCategory` and an
|
||||
authoritative list of all categories, rather than a positional array.
|
||||
Render response-local duplicate-group numbers in human-friendly form,
|
||||
de-duplicate repeated guidance, enforce a size bound, and return an error for
|
||||
any unmapped category instead of exposing its raw value. Retain
|
||||
`IssueDetails` for operator/debug diagnostics.
|
||||
4. Update the NPC, item, and location registry normalizers to populate the new
|
||||
correction guidance for semantically invalid proposals. Add the item-only
|
||||
currency guidance after the shared prose when typed application rejects a
|
||||
group. Do not add guidance to the invalid-structured-output path when no
|
||||
exact valid candidate exists.
|
||||
5. Improve exhausted-fallback diagnostics across all three registries. Describe
|
||||
the final proposal failure accurately, state that affected candidates were
|
||||
preserved separately, and keep warning samples bounded and free of raw model
|
||||
responses.
|
||||
6. Update the shared semantic-reconciliation and affected normalizer policy
|
||||
fingerprints so checkpoints produced under blind retry behavior are not
|
||||
reused. Record the durable module-retry correction decision in a new ADR,
|
||||
and update the architecture, pipeline, LLM-runtime, and D&D internal docs in
|
||||
their canonical scopes when the behavior is implemented.
|
||||
7. Add lean offline behavioral coverage for corrective module retries, exact
|
||||
latest-response use, feedback-free structural retries, contract validation,
|
||||
complete category-to-prose coverage, domain-specific item guidance, bounded
|
||||
messages, safe final fallback, and warning content. Avoid exact prose
|
||||
snapshots, prompt-length assertions, live providers, and redundant tests of
|
||||
PromptKit internals.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing normalize retry counts, PromptKit repair settings, or pipeline
|
||||
terminal policies.
|
||||
- Changing the reconciliation prompt protocol, response schema, registry
|
||||
artifact schemas, or request-local candidate IDs.
|
||||
- Reintroducing provider-sensitive schema constraints such as `uniqueItems`.
|
||||
- Automatically applying an unsafe duplicate proposal or treating a fallback
|
||||
warning as an extraction-quality warning.
|
||||
- Persisting raw model responses or correction text in ordinary warnings,
|
||||
receipts, manifests, caches, or checkpoints.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A semantic proposal rejected by an NPC, item, or location registry normalizer
|
||||
receives the exact defective response and actionable prose on the next
|
||||
normalize attempt, using only the configured stage retry budget.
|
||||
- Every `IssueCategory` has non-empty model-facing guidance, no raw category or
|
||||
reason-code string reaches the model, and an unmapped future category fails
|
||||
closed.
|
||||
- An invalid item consolidation additionally tells the model to keep currency
|
||||
denominations distinct and not mix currency with non-currency items.
|
||||
- Structural-output failures without a valid model candidate retain the current
|
||||
fresh-retry behavior.
|
||||
- After retry exhaustion, safe consolidations remain applied, affected
|
||||
candidates remain separate, and the single fallback warning explains both
|
||||
the final cause and preservation behavior without including raw model output.
|
||||
- Checkpoint identities change where policy changed, canonical documentation
|
||||
and a new ADR describe the implemented behavior, and focused tests plus
|
||||
`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.
|
||||
@@ -1,96 +0,0 @@
|
||||
# D&D Source-Reference Endpoint Canonicalization
|
||||
|
||||
## Purpose
|
||||
|
||||
Make model-supplied D&D evidence ranges resilient to reversed endpoints without
|
||||
spending producer retries on a losslessly repairable representation error.
|
||||
|
||||
## Problem
|
||||
|
||||
The shared extraction-evidence prompt requires `start_unit_id` and
|
||||
`end_unit_id`, but does not state that start means the earlier unit in
|
||||
transcript order. The private LLM schemas cannot express that cross-field
|
||||
relationship. Current extraction adapters sort and de-duplicate ranges but
|
||||
preserve reversed endpoints, so deterministic source-reference validators
|
||||
reject otherwise useful candidates and may exhaust the producer retry budget.
|
||||
|
||||
## Target State
|
||||
|
||||
- The shared D&D extraction-evidence prompt states that `start_unit_id` is the
|
||||
earlier endpoint and `end_unit_id` is the same or a later endpoint according
|
||||
to transcript order.
|
||||
- The existing LLM-facing and durable `start_unit_id`/`end_unit_id` shapes stay
|
||||
unchanged. No unordered-pair schema or compatibility migration is introduced.
|
||||
- One helper owned by `internal/modules/dnd/shared` orders the endpoints of a
|
||||
source reference using `source.DocumentIndex` positions. It never assumes
|
||||
that numerically smaller unit IDs occur earlier.
|
||||
- The helper swaps endpoints only when the reference belongs to the current
|
||||
source and both endpoint IDs resolve in that source. Forward and single-unit
|
||||
ranges remain unchanged. Wrong-source, missing, non-positive, or otherwise
|
||||
unresolved endpoints remain unchanged for deterministic validators to reject.
|
||||
- Every D&D extraction adapter that consumes model-supplied evidence ranges
|
||||
applies this endpoint operation before its existing range sorting,
|
||||
de-duplication, earliest-evidence calculation, and validation. The covered
|
||||
artifact families are spells, NPC registry, NPC occurrences, item registry,
|
||||
item occurrences, location registry, location occurrences, combat turns, and
|
||||
enemy events.
|
||||
- Successful endpoint swapping is ordinary deterministic canonicalization. It
|
||||
does not emit a warning, quality diagnostic, or retry request.
|
||||
- The exact raw model response remains available through existing model-candidate
|
||||
and debug provenance. Only the typed candidate presented to validators and
|
||||
later stages receives the canonical endpoint order.
|
||||
- Source-reference validators remain strict and continue rejecting reversed
|
||||
ranges that reach them from another producer or indicate an application bug.
|
||||
Normalizers do not become a fallback repair boundary for invalid extraction
|
||||
candidates.
|
||||
|
||||
## Required Work
|
||||
|
||||
1. Clarify endpoint ordering in
|
||||
`assets/dnd/shared/prompts/common-dnd-extraction-evidence.md`, preserving one
|
||||
byte-identical shared instruction for every consuming prompt.
|
||||
2. Extend the shared source-reference ordering utility with a mutation-safe
|
||||
endpoint-ordering operation that uses document position and preserves
|
||||
unresolvable references.
|
||||
3. Integrate the shared operation into all nine model-response-to-candidate
|
||||
mappings listed above. Do not duplicate endpoint comparison logic in the
|
||||
individual artifact packages.
|
||||
4. Update `docs/internal/dnd.md` to distinguish lossless endpoint ordering from
|
||||
malformed-reference validation. External artifact documentation and schemas
|
||||
require no change because their contract already requires ordered ranges.
|
||||
5. Add lean regression coverage for forward, reversed, single-unit,
|
||||
non-monotonic-ID, wrong-source, and unresolved endpoint cases. Add enough
|
||||
adapter-level coverage to prove the shared operation is wired into every
|
||||
affected extraction family without duplicating the helper's full case
|
||||
matrix in each package.
|
||||
6. Update prompt-asset and fingerprint expectations affected by the shared
|
||||
prompt revision, while avoiding exact token-, byte-length-, or prose-snapshot
|
||||
change-detector tests.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing durable source-reference schemas or replacing named endpoints with
|
||||
an unordered LLM response shape.
|
||||
- Weakening source-reference validation or silently repairing missing,
|
||||
out-of-source, or out-of-chunk evidence.
|
||||
- Applying this policy to scene chunk-plan ranges, which are not extraction
|
||||
evidence references and have a separate ownership and validation boundary.
|
||||
- Adding warnings or durable normalization observations for routine endpoint
|
||||
ordering.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A D&D extraction candidate whose two resolvable evidence endpoints are
|
||||
reversed reaches validation as the equivalent forward range without a
|
||||
producer retry.
|
||||
- Ordering follows document position even when unit IDs are not numerically
|
||||
increasing.
|
||||
- Every affected D&D extractor uses the shared implementation; no module-local
|
||||
endpoint-swap implementation remains.
|
||||
- References that cannot be safely ordered remain unchanged and are rejected by
|
||||
the existing deterministic validators where applicable.
|
||||
- Raw model-candidate provenance remains byte-faithful to the provider response.
|
||||
- LLM-facing and durable schemas, validator strictness, module keys, default
|
||||
chains, and output contracts remain unchanged.
|
||||
- Focused package tests, repository-wide tests, `go vet ./...`, and
|
||||
`go build ./cmd/notarius` pass.
|
||||
Reference in New Issue
Block a user