eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Strengthen validation retries and checkpoint safety

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-27 11:50:37 +00:00
parent b2e83bd6e7
commit e6b2ae88d2
73 changed files with 934 additions and 1560 deletions
Download Patch File Download Diff File Expand all files Collapse all files

731
docs/roadmap/implementation.md
View File

@@ -1,731 +0,0 @@
# Feedback-Aware Validation Retry Implementation Plan
## Status
Ready for implementation. This plan implements the target state defined by
[Feedback-Aware Stage Validation Retries](validation-retries.md). Its stages
are ordered dependencies, and each stage is intentionally scoped for one
gpt-5.6-terra implementation prompt.
The feature roadmap owns product intent and durable policy. This document owns
implementation order, concrete boundaries, and stage-level verification. If a
conflict is discovered, preserve the feature roadmap and the repository
policies, stop the affected stage, and revise this plan rather than silently
choosing a different architecture.
## Settled Decisions
The implementation agent must treat these decisions as fixed:
- PromptKit v0.9.0 is the minimum and exact supported PromptKit release for
this work. Notarius uses `RunRequest.AppendedMessages`; it does not create
paired correction manifests or bypass PromptKit preparation and execution.
- A correction request reconstructs the ordinary initial request and appends
exactly two messages: the latest exact producer response as `assistant`,
followed by one deterministic aggregate correction request as `user`.
Earlier correction turns never accumulate.
- A correction-capable producer supplies the exact single LLM response that
directly controlled the candidate. The initial protocol does not synthesize
a model-facing projection of a compound artifact.
- The only supported producer correction protocol is
`single_response_v1`. An empty protocol means unsupported. Configuration
rejects semantic-retry workflows for LLM-backed compound producers or any
other producer unable to satisfy `single_response_v1`.
- The existing producer binding `retries` value is the sole outer stage
budget. PromptKit repair attempts and validator execution retries are
independent budgets.
- Terminal policy is configured as pipeline defaults with field-by-field
producer-binding overrides. Validators never own candidate disposition.
- Application defaults are `fail_run` for exhausted producer structural
failure, `fail_run` for exhausted semantic rejection, and `warn_continue`
for exhausted validator execution failure.
- All applicable validators run sequentially in configured order. Rejections
and validator failures are aggregated; failures are not represented to the
producer as candidate defects.
- Deterministic producers do not consume retries after rejection. An
LLM-backed attempt that took a deterministic fast path and produced no model
response is likewise not correctable for that candidate.
- Known rejected or structurally invalid output never advances. A candidate
accepted under `validator_failure: warn_continue` advances with explicit
incomplete-validation provenance but is not checkpointed.
- The D&D combat-scene validator and broader warning-system redesign remain
out of scope.
## Cross-Stage Implementation Rules
Apply these rules in every stage:
1. Read the feature roadmap, `docs/development.md`, all `docs/policy/`
documents, and the focused current documentation named by the stage before
editing.
2. Inspect current code and tests rather than assuming paths or private helper
layouts. Preserve unrelated work and existing public ordering,
cancellation, scheduler, checkpoint, and sensitive-data invariants.
3. Use transport-neutral types outside `internal/framework/llm`. PromptKit
types must not escape the adapter boundary.
4. Keep correction material attempt-local. Do not put raw assistant responses
or correction text in ordinary errors, warnings, manifests, receipts,
checkpoints, caches, or default debug summaries.
5. Add the smallest durable tests that protect the stage's contracts. Do not
duplicate PromptKit's own prepared-execution, copying, hashing, capacity,
role-validation, or structural-repair test matrix.
6. Use `gofmt` on changed Go files. Run the focused checks listed for the
stage and fix failures before stopping. Do not proceed into the next stage
in the same prompt.
7. Do not delete this plan or the feature roadmap during implementation. They
are retired only after a final review confirms the complete target state.
## Contract Names And Bounds
Use one vocabulary consistently across packages. Exact private helper names
may follow local conventions, but the following contracts and values are not
open for redesign:
- `CorrectionProtocol` has only the empty unsupported value and
`single_response_v1`.
- An application-owned `SemanticCorrection` carries owned
`AssistantResponse []byte` and `UserGuidance string` values. It is exposed
on chunk, typed extract, typed merge, typed normalize, and structured
completion requests as an optional pointer.
- An application-owned `ModelCandidate` carries the owned exact response bytes
and `CorrectionProtocol`. It is exposed on the corresponding producer
results as an optional pointer and is never serialized as part of a durable
artifact.
- `ValidationResult` gains optional `CorrectionGuidance`. Operator-facing
`Message` is not copied into this field implicitly.
- Each reason code is valid UTF-8, nonblank after trimming, and at most 128
bytes. Each validator-supplied correction guidance value is valid UTF-8,
nonblank after trimming, and at most 4 KiB.
- The exact defective assistant response may be at most 1 MiB. The aggregate
user correction message may be at most 64 KiB. The combined appended
content may therefore be at most 1,114,112 bytes. Reject invalid UTF-8,
empty/whitespace-only content, or over-limit content; never truncate it.
- Invalid producer-owned correction material is a framework contract error.
It is not a validator failure and cannot be converted to `reject_output` or
`warn_continue`.
- Aggregate correction entries are ordered by validator order, contain the
stable reason code plus guidance, and collapse only byte-identical duplicate
`(reason_code, correction_guidance)` pairs while retaining first occurrence.
A rejection without guidance receives bounded generic guidance based only on
its reason code.
- Terminal actions are closed enums: structural failure and semantic
rejection accept `fail_run` or `reject_output`; validator failure accepts
`warn_continue` or `fail_run`.
## Stage 1 ✅ — Adopt PromptKit v0.9.0
### Goal
Upgrade the dependency without changing Notarius stage behavior, prove the
existing integration remains compatible, and make the new upstream request
primitive available to later stages.
### Work
- Update `go.mod` and `go.sum` to PromptKit v0.9.0 with `go get` and
`go mod tidy`. Accept the catalog modules selected transitively by PromptKit;
do not import or register them directly.
- Review every Notarius `promptkit.RunRequest` literal and retain keyed form.
Verify production prompt roles are limited to `system` and `user`.
- Update the conservative built-in-profile checkpoint marker in
`internal/framework/llm/promptkit_profile_fingerprint.go` from v0.8.0 to
v0.9.0. Do not duplicate the separately versioned catalog module versions.
- Update `docs/integrations/pkg-promptkit.md` only to describe the newly
implemented dependency version and unchanged current integration boundary.
Include the strict role vocabulary and external catalog ownership, but do
not document Notarius correction behavior yet.
### Verification
- Run `go test ./internal/framework/llm ./internal/cli`.
- Run `go test ./...`, `go test -race ./internal/framework/llm`, `go vet ./...`,
and `go build ./cmd/notarius`.
### Exit Criteria
Notarius builds and passes its suite on PromptKit v0.9.0, its compatibility
document and checkpoint marker name v0.9.0, and no Notarius package directly
depends on either upstream catalog module. This stage is one Terra prompt.
## Stage 2 ✅ — Record The Architecture And Add Transport-Neutral Contracts
### Goal
Record the durable decision and introduce owned correction/candidate types
without changing runtime retry behavior.
### Work
- Add ADR-0014 under `docs/adr/` using the repository ADR format. Record the
three distinct retry budgets, complete validator aggregation, the fresh
two-message correction protocol, exact-response producer capability,
non-recursive validator failure handling, terminal-policy ownership and
defaults, checkpoint conservatism, and sensitive-data constraints.
- Add `CorrectionProtocol`, `SemanticCorrection`, and `ModelCandidate` to the
domain-neutral framework contracts, with constructors/clone helpers that
validate the settled bounds and defensively copy bytes.
- Add optional correction input to `ChunkRequest`,
`TypedExtractionRequest`, `TypedMergeRequest`, and
`TypedNormalizeRequest`.
- Add optional model-candidate output to `ChunkPlanResult`,
`TypedExtractionResult`, `TypedMergeResult`, and
`TypedNormalizeResult`.
- Add `CorrectionGuidance` to `ValidationResult` and centralize validation of
reason codes and corrective guidance at the framework boundary.
- Update type-erasure adapters, stored fakes, and clone paths so the new values
retain ownership and cannot alias caller buffers. Do not declare production
module capabilities or use the fields in the runner yet.
### Verification
- Add contract-level tests for accepted values, UTF-8 and size rejection,
defensive copying, nil behavior, and typed-erasure preservation.
- Run `go test ./internal/framework/contracts ./internal/framework/pipeline`.
- Run `go test ./...`.
### Exit Criteria
The generic contracts can safely carry correction input and exact candidate
material across every producer stage, no PromptKit type crosses the boundary,
and existing runtime behavior is unchanged. This stage is one Terra prompt.
## Stage 3 ✅ — Add Validation Policy Configuration And Resolution
### Goal
Implement strict configuration, inheritance, identity, and provenance for
terminal validation policy before the runner consumes it.
### Work
- Add optional `validation_policy` to a pipeline profile and to chunk,
extract, merge, and normalize producer bindings. Use pointer-backed override
fields so omission is distinguishable from an explicit value.
- Reject an explicitly null policy object, null policy fields, duplicate or
unknown fields, and values outside the settled enums. Keep the current file
configuration version.
- Reject `validation_policy` on input, output, and validator bindings. Reject
a binding-level explicit `producer_structural_failure` value on a
deterministic producer; a pipeline-level structural default remains valid
because a pipeline may contain LLM-backed producers.
- Resolve each policy field independently in binding, pipeline, application
default order. Store one detached concrete effective policy for chunk and
for each lane's extract, merge, and normalize stage; do not leave runtime
inheritance to the runner.
- Include both configured overrides and effective policies at their existing
appropriate configuration/provenance boundaries. Include the effective
values in resolved-pipeline digest and checkpoint identity. Preserve
deterministic serialization and redaction.
- Update `examples/dnd-minimal/config.yml` and
`examples/dnd-complete/config.yml` only if their current shape must change to
remain valid; do not add redundant explicit defaults merely to demonstrate
the feature.
### Verification
- Add focused parsing tests for omission versus null, unknown/duplicate
fields, invalid placements, and invalid enums.
- Add resolution tests for field-by-field inheritance and mixed overrides.
- Add digest, clone, JSON/YAML round-trip, and redacted-effective-config tests.
- Run `go test ./internal/core/config ./internal/framework/pipeline ./internal/cli`.
### Exit Criteria
Every producer stage receives one immutable effective policy, invalid
configuration fails before execution, and policy changes invalidate resolved
identity. The runner still follows its old behavior until later stages. This
stage is one Terra prompt.
## Stage 4 ✅ — Declare And Validate Producer Correction Capability
### Goal
Make correction support an explicit module contract and prepare the framework
to reject impossible workflows.
### Work
- Extend `pipeline.ModuleSpec` with `CorrectionProtocol`; normalize, clone,
validate, and include it in relevant registry/spec fingerprints.
- Permit `single_response_v1` only for LLM-backed chunk, extract, merge, or
normalize modules. Reject it for deterministic, input, output, or validator
specs. Empty remains the default unsupported value.
- Carry the selected protocol into resolved and prepared producer metadata and
checkpoint fingerprints.
- Reject positive `retries` on deterministic validator bindings during
resolution. LLM-backed validator retries remain valid and independent of
producer capability.
### Verification
- Add spec normalization/clone tests, invalid stage/class combinations,
validator retry-class tests, and preparation tests for supported and
unsupported producer workflows.
- Run `go test ./internal/framework/pipeline ./internal/core/config`.
### Exit Criteria
Capability is explicit, fingerprinted, and testable, while existing production
pipelines remain executable pending their migrations. No dormant feature gate
or unused preparation check is introduced. This stage is one Terra prompt.
## Stage 5 — Adapt Corrections Through PromptKit ✅
### Goal
Map the transport-neutral correction contract onto PromptKit v0.9.0 without
changing ordinary requests.
### Work
- Add optional `SemanticCorrection` to `StructuredCompletionRequest` and its
debug-safe request representation.
- In `PromptKitClient.CompleteStructured`, validate and defensively copy the
correction, then map it to exactly two `promptkit.RenderedMessage` values in
`RunRequest.AppendedMessages`, using `promptkit.RoleAssistant` followed by
`promptkit.RoleUser`.
- Leave `AppendedMessages` nil for an ordinary request. Do not allow callers to
choose other roles through the Notarius contract.
- Ensure request formatting and default debug summaries expose only safe
counts/digests. The existing explicitly requested detailed PromptKit trace
may contain complete effective messages and must retain its existing
sensitive-data treatment.
- Preserve the same prompt identity, inputs, variables, session, profile,
execution overrides, repair attempts, scheduler, and prepared-execution
path for corrected calls.
### Verification
- Add focused adapter tests proving an ordinary request is unchanged and a
corrected request has the ordinary rendered prefix plus exactly the two
supplied messages in order with exact content.
- Test invalid/oversized correction rejection before preparation and absence
of raw content from ordinary formatting/errors.
- Add one representative test showing PromptKit structural repair remains
available on a request with appended messages; do not reproduce upstream's
complete repair suite.
- Run `go test ./internal/framework/llm ./internal/framework/pipeline` and
`go test -race ./internal/framework/llm`.
### Exit Criteria
Notarius has one safe, tested adapter path for fresh corrected requests, and no
stage invokes it yet. This stage is one Terra prompt.
## Stage 6 — Migrate The Scene Chunker And Foundational Extractors ✅
### Goal
Make the scene chunker and the foundational direct D&D extractors satisfy
`single_response_v1`.
### Work
- Update `dnd/scenes`, `dnd/npc-registry`, `dnd/item-registry`,
`dnd/location-registry`, `dnd/scene-descriptions`, and `dnd/spells` producer
implementations to pass request correction to their structured completion
and return an owned copy of the successful response's exact validated raw
bytes as `ModelCandidate`.
- Declare `single_response_v1` in each corresponding module spec.
- Do not serialize normalized artifacts to fabricate candidate material. Keep
artifact parsing, deterministic identity attachment, evidence validation,
warnings, and durable schemas unchanged.
- Update shared D&D extraction helpers only where the behavior is genuinely
common; keep domain prompt ownership in each module.
### Verification
- Use representative package-level tests to prove exact-response propagation,
correction forwarding, and input/result ownership. Update spec tests for the
declared protocol without multiplying the same behavior test across all six
modules.
- Run the affected module tests and `go test ./internal/modules/dnd/...`.
### Exit Criteria
All named producers truthfully advertise and implement the exact-response
protocol, with unchanged ordinary extraction behavior. This stage is one
Terra prompt.
## Stage 7 — Migrate Downstream D&D Extractors ✅
### Goal
Complete `single_response_v1` support for the remaining direct D&D extraction
producers.
### Work
- Migrate `dnd/npc-occurrences`, `dnd/item-occurrences`,
`dnd/location-occurrences`, `dnd/combat-turns`, and `dnd/enemy-events` using
the same contract as Stage 6.
- Preserve their contextual-name and deterministic-ID rules. Correction
messages must never introduce opaque registry IDs or ask the model to copy
them.
- Preserve combat-scene gating, reference projections, and prompt-cache prefix
ordering. The correction pair is appended after the complete existing
request and never changes stable prompt assets.
### Verification
- Add or adapt focused tests for correction forwarding and exact raw-response
retention at the shared boundary, plus one registry-grounded extractor case
proving no opaque IDs enter correction material.
- Run the affected module tests, `go test ./internal/modules/dnd/...`, and the
assembled D&D CLI contract tests.
### Exit Criteria
Every direct production D&D LLM extractor implements the same correction
protocol without changing artifact semantics. This stage is one Terra prompt.
## Stage 8 — Migrate Semantic Reconciliation Normalizers ✅
### Goal
Support correction for the current single-proposal reconciliation path and
activate capability enforcement for all eligible production producers.
### Work
- Extend `semanticreconcile.Request` to accept `SemanticCorrection` and its
result to expose the exact validated proposal response as owned
`ModelCandidate` when an LLM call actually occurred.
- Forward correction through the shared engine's ordinary structured
completion request. Preserve request-local integer candidate handles,
proposal validation, typed application policies, and fallback behavior.
- Update the NPC-, item-, and location-registry normalizers to carry the exact
proposal material through their typed result and declare
`single_response_v1`.
- Mark deterministic skip/limit/fallback outcomes as having no model candidate.
If such a candidate is later rejected, it is not correctable and must not
consume a retry merely because its module execution class is LLM-backed.
- Audit all production LLM-backed producer specs. Add the preparation rule that
an LLM-backed producer with a non-empty validator chain and `retries > 0`
must declare `single_response_v1`. Any genuine multi-response producer
remains unsupported and a configured validator-backed semantic retry for it
must fail preparation.
### Verification
- Add engine tests for initial and corrected requests, exact proposal bytes,
deterministic no-call outcomes, invalid proposal outcomes, and ownership.
- Add representative normalizer and preparation tests for supported,
unsupported, and no-validator/no-retry configurations.
- Run `go test ./internal/framework/semanticreconcile ./internal/framework/pipeline ./internal/modules/dnd/... ./internal/cli`.
### Exit Criteria
All eligible production LLM producers implement the declared protocol,
unsupported workflows fail before source parsing, and ordinary operational
retries remain allowed when semantic correction cannot occur. This stage is
one Terra prompt.
## Stage 9 — Build Complete Validator-Chain Execution ✅
### Goal
Replace first-result validation with one reusable, deterministic executor that
aggregates the complete chain and isolates validator retries.
### Work
- Introduce an internal immutable `validationReport` model with one ordered
invocation record per validator: approved, rejected, failed, or skipped;
validator attempt count; safe reason/message metadata; warnings; and bounded
correction guidance.
- Implement one shared sequential executor around stage-specific invocation
closures. It must continue after rejection and isolated execution failure,
retry an LLM-backed failed validator against the same immutable candidate up
to its binding budget, and stop retrying after a contract-valid approval or
rejection.
- Reconstruct the same ordinary validator request for each validator retry.
Do not append feedback to validator prompts and do not regenerate the
producer candidate.
- Preserve only one final warning for an exhausted validator failure; retain
individual attempt details for debug. Keep warnings from completed validators
in configured order.
- Build aggregate correction guidance from all rejections using the settled
ordering, deduplication, generic fallback, and bounds. Validator failures,
skips, warnings, and operator messages must not enter it.
- Retain `skipped` as a framework-owned outcome for an otherwise selected
validator whose runtime prerequisites are unavailable. Module construction,
type incompatibility, candidate cloning failure, cancellation, and debug
persistence failure remain framework errors rather than skips.
### Verification
- Add table-driven behavior tests for all approved, multiple rejections,
rejection plus failure, failure only, skipped, retry success, retry
exhaustion, immutable candidate reuse, deterministic ordering, duplicate
guidance, and bounds.
- Prove deterministic validators cannot receive positive retry budgets and
LLM validator retry counts do not consume producer attempts.
- Run `go test ./internal/framework/pipeline` and
`go test -race ./internal/framework/pipeline`.
### Exit Criteria
The shared executor returns a complete ordered report without deciding whether
the producer candidate advances. Existing stage callers may still adapt the
report through their old disposition path until the next stages. This stage is
one Terra prompt.
## Stage 10 — Implement The Generic Producer Attempt State Machine ✅
### Goal
Replace the boolean retry helper with one explicit domain-neutral state
machine used by later stage integrations.
### Work
- Refactor `runWithRetry` into an attempt engine that accepts producer and
complete-validation closures and returns a terminal result plus ordered
attempt provenance. Keep stage-specific artifact handling outside it.
- Classify attempts as initial, operational-error retry, structural retry,
module-requested retry, or semantic correction. Preserve one total producer
budget of `retries + 1` attempts.
- On semantic rejection, correct only when another attempt exists and the
current candidate contains valid `single_response_v1` material. Construct a
fresh `SemanticCorrection` from that latest response and aggregate feedback.
- Apply outcome precedence and effective terminal policy exactly as the
feature roadmap specifies. Ordinary producer errors remain framework errors
after retries. Only `ErrInvalidStructuredOutput` uses the structural-failure
policy; `reject_output` records no usable artifact.
- A deterministic or no-model candidate applies semantic terminal policy
immediately without spending an ineffective retry.
- With rejection plus validator failure, use rejection guidance for a
correction while recording incomplete validation. With failure only, apply
validator-failure policy without regenerating the producer.
- Keep cancellation and debug-persistence failures immediately terminal.
Abandoned-attempt warnings must not be promoted.
### Verification
- Test the state machine through stable behavior with fake closures: budget
accounting, fresh correction history, latest-response replacement, all
terminal actions, failure precedence, deterministic candidates,
cancellation, and warning promotion.
- Do not assert private helper call choreography or exact correction prose.
- Run `go test ./internal/framework/pipeline` and its race-enabled suite.
### Exit Criteria
One tested state machine owns attempt budgets and disposition, but no stage is
partially migrated. This stage is one Terra prompt.
## Stage 11 — Integrate Chunk Validation And Correction ✅
### Goal
Move chunk generation, validation, cache reuse, and retry disposition onto the
new state machine.
### Work
- Adapt chunk candidate materialization and both chunk/serialized validator
targets to the complete-chain executor.
- Pass correction only to a generated chunker attempt and retain the exact raw
chunker response associated with the materialized plan.
- Preserve chunk-plan cache identity and mode semantics. A rejected automatic
cache hit is unusable for the invocation and falls through to generated
attempt one without pretending the cache hit has model material. Publish a
newly generated plan only after complete accepted validation.
- Apply chunk effective terminal policy. A terminal chunk rejection prevents
lane execution; a framework error still prevents output encoding.
- Preserve annotation materialization, deterministic plan canonicalization,
attempt debug paths, scheduler use, and checkpoint behavior.
### Verification
- Add focused tests for corrected plan acceptance, multi-validator feedback,
rejected cache hit regeneration, cache non-overwrite, incomplete-validation
non-publication, policy outcomes, cancellation, and debug failure.
- Run `go test ./internal/framework/pipeline ./internal/modules/dnd/chunk/... ./internal/cli` and the pipeline race tests.
### Exit Criteria
Chunk is the first complete production stage using feedback-aware retries, with
cache and terminal behavior matching the roadmap. This stage is one Terra
prompt.
## Stage 12 ✅ — Integrate Per-Chunk Extraction
### Goal
Apply the state machine independently to every concurrent extraction job.
### Work
- Adapt typed and serialized extract validation to the complete-chain executor
and use the extract binding's effective policy.
- Keep correction state local to one lane/chunk job. Rebuild the same initial
extraction request with correction attached only on a semantic retry.
- Preserve the run-wide worker pool, scheduled LLM client, chunk-first/lane-
second public ordering, stable error selection, cancellation, and lane
continuation rules.
- Ensure `reject_output` records the terminal chunk-scoped rejection without
advancing it to merge. `warn_continue` is available only for validator
failure with no semantic rejection and marks the accepted extract as
validation-incomplete.
### Verification
- Add representative extraction tests for correction success, exhausted
rejection, validator failure policy, independent concurrent jobs, stable
ordering, cancellation, and warning promotion.
- Add one assembled D&D pipeline test using fakes—not a live provider—to prove
a rejected direct extraction is corrected using the exact prior response.
- Run `go test ./internal/framework/pipeline ./internal/modules/dnd/... ./internal/cli` and relevant race tests.
### Exit Criteria
Every extraction job has an isolated bounded correction conversation and
continues to obey existing concurrency and ordering contracts. This stage is
one Terra prompt.
## Stage 13 ✅ — Integrate Merge And Normalize
### Goal
Complete stage coverage and reconcile semantic correction with the existing
normalizer fallback retry directive.
### Work
- Adapt typed and serialized merge and normalize validation to the shared
executor and their respective effective policies.
- Keep merge and normalize serial within a lane and reuse the exact same
accepted upstream artifacts, references, source input, profile, and session
on every attempt.
- Fold `NormalizeRetry` into the one producer attempt state machine. It consumes
the same remaining stage budget, retains its validated safe fallback when
exhausted, and never overrides a semantic rejection of that fallback or a
later candidate.
- Do not send correction to deterministic mergers/normalizers. A rejected
deterministic candidate applies terminal policy immediately.
- Preserve typed erasure checks, codec boundaries, generated handoffs, and
lane continuation/cancellation behavior.
### Verification
- Add one representative merge path and normalize paths for correction
success, deterministic rejection, `NormalizeRetry` success/exhaustion,
fallback rejection, validator failure, and terminal policies.
- Run `go test ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/dnd/... ./internal/cli` and relevant race tests.
### Exit Criteria
Chunk, extract, merge, and normalize all use the same attempt and validation
semantics, with no nested normalizer retry loop. This stage is one Terra
prompt.
## Stage 14 ✅ — Finalize Provenance, Checkpoints, Debug, And Durable Results
### Goal
Make the new state machine auditable and safe without leaking correction
content or allowing degraded output to become reusable state.
### Work
- Extend internal attempt/debug records with attempt kind, PromptKit repair
count and usage, ordered validator outcomes and attempt counts, aggregate
reason codes, validation completeness, effective policy, and terminal
decision.
- Keep raw assistant responses and complete correction messages only in the
existing explicitly requested detailed trace. Default summaries contain
counts, digests, identities, and bounded safe fields.
- Add one reusable durable validation summary with the JSON fields
`status`, `rejecting_validators`, `reason_codes`,
`incomplete_validators`, `producer_attempt_count`, and `terminal_action`.
`status` is one of `complete`, `rejected`, or `incomplete`; lists preserve
configured order and omit duplicates after their first occurrence. Embed or
project this summary at the manifest, rejection, and run-result boundaries
that already expose the affected stage outcome. Preserve the existing
singular rejection fields as the first configured rejection for downstream
continuity; do not put raw guidance or responses in the summary.
- Emit one genuine, bounded, deterministically ordered warning per validator
whose execution budget is exhausted under `warn_continue`; do not emit a
warning merely because a later correction succeeded.
- Checkpoint only accepted and completely validated results. Never write or
reuse rejected, structurally invalid, or validation-incomplete producer
output. Ensure correction protocol and effective policy participate in
fingerprints.
- Confirm cache, resume, generated-reference handoff, output encoding, and CLI
result publication cannot treat a rejected or incomplete checkpoint as
accepted.
### Verification
- Add focused manifest, receipt, warning, debug, redaction, checkpoint reuse,
generated-handoff, and sensitive-content tests at their canonical owners.
- Test that corrected success is auditable but quiet, warn-continue output is
never checkpointed, and raw content is absent from ordinary durable files.
- Run `go test ./internal/framework/pipeline ./internal/framework/checkpoint ./internal/framework/artifacts ./internal/cli` and relevant race tests.
### Exit Criteria
Every terminal path has accurate bounded provenance, no sensitive correction
content leaks by default, and only completely validated output is reusable.
This stage is one Terra prompt.
## Stage 15 ✅ — Update Canonical Documentation And Perform Final Verification
### Goal
Document the implemented contracts in their canonical homes and prove the
repository is ready for review.
### Work
- Update `docs/policy/architecture.md` with durable validator aggregation,
retry-budget separation, producer correction capability, terminal-policy,
checkpoint, and sensitive-data invariants. Link to ADR-0014 for rationale.
- Update `docs/config.md` with the exact `validation_policy` schema, enum
values, defaults, inheritance, placement restrictions, retry-budget
meanings, and invalid combinations.
- Update `docs/operations.md` with costs, terminal outcomes, warnings, debug
sensitivity, retry exhaustion, resume/cache consequences, and recovery.
- Update `docs/internal/pipeline.md`, `docs/internal/llm.md`, and
`docs/internal/modules.md` with implemented mechanics and focused test
routing. Keep public configuration definitions in `docs/config.md`.
- Complete `docs/integrations/pkg-promptkit.md` for v0.9.0 appended messages,
supported roles, application-owned bounds, and the external catalog
boundary. Update the run-result and affected output/subprocess integration
contracts for any durable fields added in Stage 14.
- Update maintained examples only to demonstrate implemented behavior. Use a
minimal policy override in the complete D&D example if it materially aids
operators; keep the minimal example minimal. Ensure all links point to
canonical owners and remove stale v0.8.0 claims outside historical release
notes.
- Do not create a release note until an actual release is prepared.
### Verification
- Run `gofmt` on all changed Go files.
- Run `go test ./...`, `go test -race ./...`, `go vet ./...`, and
`go build ./cmd/notarius`.
- Run the repository's example/config validation tests and documentation link
checks. If no standalone link checker exists, verify changed relative links
and record that manual check in the implementation report.
- Use targeted searches to confirm no current documentation still claims
PromptKit v0.8.0, no production correction-capable spec is missing its
protocol, no PromptKit type escaped `internal/framework/llm`, and no raw
correction content is serialized outside explicit detailed debug data.
- Review `git diff --check` and `git status --short`; do not include generated
binaries, temporary files, or unrelated work.
### Exit Criteria
All roadmap acceptance criteria are satisfied, canonical documentation matches
the code, maintained examples validate, the complete ordinary and race-enabled
test suites pass, and the worktree contains only intentional implementation
changes. This stage is one Terra prompt.
## Open Questions
None. PromptKit transport, producer representation, terminal-policy ownership,
bounds, retry budgets, outcome precedence, and checkpoint treatment are all
settled by the feature roadmap and this plan.

609
docs/roadmap/validation-retries.md
View File

@@ -1,609 +0,0 @@
# Feedback-Aware Stage Validation Retries
## Status
Proposed. This is the active feature roadmap for the next Notarius work set.
Its design decisions are settled. Current behavior remains authoritative until
this roadmap is implemented and the corresponding ADR and canonical
documentation are updated.
## Purpose
Make validation an effective corrective boundary around LLM-produced stage
candidates. When deterministic or LLM-backed validators reject a structurally
valid candidate, Notarius should give the producing model the complete,
ordered validation feedback and use the stage's existing retry budget to ask
for a corrected replacement. The feature must distinguish semantic rejection
from producer failure and validator execution failure, preserve the boundary
between PromptKit repair and Notarius stage retries, and remain safe under
concurrency, cancellation, caching, checkpoints, and sensitive input.
This work is domain-neutral. It establishes the framework behavior required by
future LLM-backed validators such as D&D combat-scene review, but it does not
add that validator.
## User Intent
- A stage candidate should be evaluated by every applicable configured
validator before Notarius decides whether to retry or terminate.
- A semantic retry should be materially more useful than repeating the same
request. The producing model should see its latest defective response and
all actionable semantic feedback.
- PromptKit's bounded structural repair and Notarius's stage retry loop are
separate. Each stage attempt receives its own complete PromptKit repair
budget; PromptKit repair never consumes or replenishes the stage budget.
- Deterministic rejection, semantic rejection, producer structural failure,
and validator execution failure are different outcomes and must not be
collapsed into one generic error path.
- The default posture is strict for known-invalid producer output and tolerant
but visible when a validator itself cannot make a decision.
- Corrective prompts must not expose opaque application identifiers, secrets,
or unbounded diagnostic content merely because those values exist in an
internal artifact or operator-facing error.
## Current State
The current runner already provides useful foundations:
- chunk, extract, merge, and normalize producer bindings have one `retries`
value interpreted as additional stage attempts;
- `runWithRetry` retries producer errors and semantic rejections within that
budget;
- PromptKit performs bounded structural repair inside each structured
completion;
- validator targets, execution classes, profile selection, repair policy,
attempts, debug scopes, checkpoint identity, and deterministic public
ordering are already explicit; and
- the structured-completion response retains the model's validated raw bytes
and PromptKit repair metadata.
The current behavior is not yet the desired corrective workflow:
- the runner repeats the ordinary producer request after rejection and does
not pass the previous model response or validator feedback;
- validation stops at the first rejection or execution failure, so later
applicable validators do not contribute findings;
- validator execution failure is immediately a framework error rather than a
configurable incomplete-validation outcome;
- `ValidationResult.Message` currently serves operator diagnostics and does
not define separately bounded model-facing guidance;
- typed stage results do not carry the exact model response needed for the
next correction attempt;
- validator-binding `retries` values participate in resolved configuration but
are not used to retry a failed validator against the same candidate; and
- Notarius still pins PromptKit v0.8, while PromptKit v0.9.0 now provides the
append-only request-message API needed for application-owned correction
attempts.
## Target End State
For chunk, extract, merge, and normalize stages, Notarius owns one explicit
candidate-attempt state machine:
1. The producer creates one candidate using the ordinary request. An
LLM-backed producer may use PromptKit structural repair internally.
2. The framework establishes one immutable validation candidate and runs every
applicable validator sequentially in configured order.
3. The framework aggregates approvals, warnings, semantic rejections,
execution failures, and skipped-validator diagnostics without allowing one
validator to mutate the candidate seen by another.
4. A candidate with one or more semantic rejections is never accepted. If the
LLM-backed producer has another stage attempt available, Notarius rebuilds
the complete original prompt and appends the latest defective assistant
response followed by one application-owned correction message containing
every actionable rejection. It then requests one complete replacement
candidate.
5. A producer error consumes the same stage attempt budget under the existing
retry rules, but semantic correction material is used only when a
structurally valid candidate was actually rejected.
6. A validator execution failure is retried, when configured, against the same
immutable candidate. It never regenerates the producer candidate by itself.
7. When budgets are exhausted, the configured terminal policies decide
whether the run fails, a rejected output is recorded, or a structurally
valid candidate advances with explicitly incomplete validation.
The first attempt remains byte-for-byte the ordinary prompt rendered from the
selected prompt definition. Every correction attempt starts from that same
ordinary prompt rather than from the prior correction conversation. It appends
exactly two messages:
- an `assistant` message containing the producer-supplied exact defective
response for the latest candidate; and
- a `user` message containing deterministic, bounded, application-owned
correction guidance and asking for one complete replacement response.
The session ID, prompt ID and version, selected profile, reasoning settings,
structured-output contract, repair budget, named inputs, variables, references,
and reusable prompt prefix remain unchanged across stage attempts.
## Architectural Ownership
### PromptKit
PromptKit continues to own prompt loading and rendering, profile resolution,
backend admission, provider generation, structural validation, and bounded
structural repair within one completion. A PromptKit repair conversation is
private to that completion and is not exposed as a Notarius stage attempt.
PromptKit v0.9.0 owns the mechanical operation of appending explicitly supplied
messages to a normally rendered prompt before creating the immutable prepared
execution. `RunRequest.AppendedMessages` preserves the original rendered
messages as an exact prefix, validates and defensively copies additions,
includes the complete sequence in prepared details and the rendered-prompt
hash, and runs it through the ordinary generation and structural-repair path.
PromptKit does not impose message-count, byte-size, token, or context-window
limits and permits empty content, so Notarius retains its stricter
application-owned correction validation and bounds.
### Notarius Framework
The framework owns stage budgets, immutable candidate preparation, complete
validator-chain execution, result aggregation, outcome precedence, correction
message construction, terminal policy, public ordering, checkpoint effects,
manifest summaries, warnings, and debug lifecycle.
The framework must remain domain-neutral. It may format stable reason codes and
validator-supplied corrective guidance, but it must not infer D&D or other
domain rules from artifact JSON.
### Producers And Artifact Families
The producing module owns prompt selection, prompt inputs, typed decoding, and
the model-facing representation that corresponds to its candidate. An
LLM-backed producer that supports feedback-aware correction must return the
exact response material that the model should see as its prior assistant turn.
It must not substitute a normalized artifact containing deterministically
attached UUIDs or other opaque application identity.
Artifact-family validators own semantic decisions and domain-specific
corrective guidance. Operator-facing explanation and model-facing correction
are separate contract fields even when their concise text happens to match.
## Validation Outcome Model
Each validator invocation produces one of four framework outcomes:
| Outcome | Meaning | Effect |
| --- | --- | --- |
| Approved | The validator completed and accepted the whole candidate. | Retain its warnings and continue the chain. |
| Rejected | The validator completed and found a semantic defect in the candidate. | Record the finding, continue the chain, and make the candidate ineligible for acceptance. |
| Failed | The validator could not return a usable decision because of an internal, transport, generation, structural-output, or result-invariant failure. | Retry that validator when eligible, then record incomplete validation and continue the chain unless cancellation or framework integrity prevents it. |
| Skipped | Runtime prerequisites for an otherwise selected validator cannot be satisfied. | Record a deterministic incomplete-validation diagnostic and continue; do not invent a semantic decision. |
Configured validator order controls invocation order and aggregate feedback
order. Execution remains sequential initially. The framework must continue
after a rejection and after an isolated validator failure when it can safely
prepare the remaining validator requests. Cancellation, inability to preserve
an immutable candidate, debug persistence failure, or another framework
integrity failure remains immediately terminal.
### Outcome Precedence
For one candidate, apply this precedence:
1. A producer structural failure means no acceptable candidate exists and
cannot be converted into validator approval.
2. Any completed semantic rejection makes the candidate rejected, even when
another validator failed or was skipped.
3. With no semantic rejection, a validator failure or skip makes validation
incomplete and invokes the validator-failure policy.
4. Only a structurally valid candidate with no rejection and either complete
validation or an explicit `warn_continue` decision may advance.
Do not turn a known rejection into acceptance through a permissive
validator-failure policy. Do not turn a structurally invalid response into a
rejected-but-usable artifact.
## Corrective Feedback Contract
`ValidationResult` should gain a separately bounded, optional model-facing
correction field. A rejecting production validator should provide:
- a stable reason code suitable for aggregation and provenance;
- an operator-facing message suitable for ordinary diagnostics; and
- concise corrective guidance that explains the violated rule without asking
the model to reproduce opaque identity or leaking unrelated source data.
The framework constructs one deterministic correction message from all
rejections in validator order. Each entry identifies the stable reason code
and corrective guidance. Duplicate identical entries may be collapsed while
preserving first occurrence; distinct findings must not be discarded merely
to shorten the message. If a validator rejects without model-facing guidance,
the framework uses a generic reason-code-based correction rather than copying
the operator message automatically.
Warnings, validator failures, skipped diagnostics, provider messages, stack
traces, debug paths, and sensitive values are not corrective guidance. They may
be recorded through their proper diagnostic channels but must not be presented
to the producer as candidate defects.
The framework must validate UTF-8, role, non-empty content, and
application-owned size limits before constructing the correction request. Oversized or
invalid correction material is a framework-owned inability to perform a
feedback retry; it must never be silently truncated into a misleading or
syntactically defective assistant response.
## Producer Correction Contracts
Introduce application-owned, defensively copied correction contracts at the
framework boundary:
- chunk, typed extraction, typed merge, and typed normalize results can carry
optional model-facing candidate material associated with their returned
value;
- the corresponding requests can carry an optional correction containing the
latest assistant material and aggregated guidance;
- `StructuredCompletionRequest` can carry the two bounded appended messages
without importing PromptKit types into module or pipeline contracts; and
- the PromptKit adapter translates those application-owned messages into
`RunRequest.AppendedMessages` using `promptkit.RoleAssistant` and
`promptkit.RoleUser` before preparation.
A semantic correction always supplies exactly two appended messages: the
latest defective response as `assistant`, followed by the aggregate correction
request as `user`. The framework does not expose the other PromptKit-supported
roles through this contract and does not accumulate messages from earlier
stage attempts. PromptKit preserves message content exactly, but Notarius must
reject empty content and enforce its own per-message and aggregate byte limits
before the adapter is called.
Correction material is attempt-local sensitive data. It is not part of the
artifact schema, checkpoint value, cache key, durable output bundle, ordinary
error, or configuration summary. The policy and capability that affect
execution do participate in resolved pipeline and checkpoint identity.
LLM-backed modules selected with both `retries > 0` and a non-empty validator
chain must declare whether they can produce and consume correction material.
Preparation must reject a pipeline that could request feedback-aware semantic
retries from an LLM-backed producer without that capability. An LLM-backed
producer with no validators may continue to use its retry budget for
operational failures without declaring semantic-correction capability.
A correction-capable producer must supply the exact single LLM response that
directly controlled the candidate being validated. Direct D&D chunk and
extraction producers expose their exact structured response. The shared
semantic-reconciliation path exposes its exact proposal response through its
typed normalizers without turning request-local batch handles into durable
identity. Deterministic transformations after that response are permitted only
when the validated candidate remains directly traceable to it.
A producer whose candidate combines multiple LLM responses is not
correction-capable under this initial protocol. It may continue to use ordinary
operational retries when no semantic correction can occur, but configuration
must reject a validator-backed retry workflow for it. Supporting compound
producers later requires a separately reviewed multi-response protocol; the
framework must not synthesize an assistant message by serializing the final
typed artifact.
Deterministic producers do not receive correction material. A deterministic
candidate rejected by validation immediately applies the terminal semantic
rejection policy without consuming retries that cannot change the result.
## Retry Budgets
### Producer Stage Budget
The existing producer binding `retries` field remains the sole outer stage
budget. `retries: N` means at most `N` additional complete producer attempts
after the initial attempt. Producer operational errors, producer structural
failures, module-requested normalize retries, and semantic corrections all
draw from this same budget. Do not add a separate semantic retry counter.
Every LLM-backed producer attempt receives the configured PromptKit
`structured_output_repair_attempts` value independently. Notarius does not
decrement that value across stage attempts.
### Validator Budget
Use the existing `retries` field on an LLM-backed validator binding for
additional attempts to obtain a usable decision about the same immutable
candidate. A completed approval or rejection is terminal for that validator
and does not consume another validator attempt. A validator retry reconstructs
the same ordinary validator prompt; it does not append semantic feedback about
the validator's prior failed judgment and does not create a recursive
Notarius correction loop.
Reject a positive validator `retries` value on a deterministic validator at
configuration resolution because repeating the same pure decision cannot
improve it. Validator retries do not consume the producer stage budget.
## PromptKit v0.9.0 Adoption
The target end state pins PromptKit v0.9.0 for correction requests. The
resolved dependency graph includes its independently versioned
OpenRouter and Rakestrawhome catalog modules through ordinary Go module
resolution; Notarius must not import or register those catalogs directly.
PromptKit continues to own their built-in backend and profile IDs, source
precedence, credentials, and capacity behavior.
Notarius's PromptKit compatibility documentation and built-in-profile
checkpoint marker identify v0.9.0 rather than v0.8.0. The PromptKit release
identity remains the conservative checkpoint identity for the exact catalog
versions selected by that release; Notarius should not duplicate upstream
catalog module versions in a second hand-maintained marker.
PromptKit v0.9.0 restricts text-chat roles to `developer`, `system`, `user`, and
`assistant`. Maintained Notarius prompt definitions already use only `system`
and `user`; correction requests add only `assistant` and `user`. PromptKit
`RunRequest` literals remain keyed. These compatibility conditions must remain
covered by the ordinary production-asset and adapter checks without adding a
brittle inventory test that merely counts prompt messages or literals.
## Terminal Policy Configuration
Add an optional `validation_policy` object at pipeline scope and on chunk,
extract, merge, and normalize producer bindings:
```yaml
validation_policy:
producer_structural_failure: fail_run
semantic_rejection: fail_run
validator_failure: warn_continue
```
The binding object overrides individual pipeline values; resolution is
field-by-field in binding, pipeline, application-default order. Omitted values
inherit rather than replacing the complete object. Explicit null, unknown
fields, and unknown enum values are invalid. The effective policy is resolved
and detached before execution, appears in redacted effective configuration and
run provenance, and participates in the resolved pipeline digest and checkpoint
identity.
The initial enum values and defaults are:
- `producer_structural_failure`: `fail_run` by default; `reject_output` may
retain a terminal rejection and final raw candidate for debug, but may not
advance or publish an invalid artifact;
- `semantic_rejection`: `fail_run` by default after stage attempts are
exhausted; `reject_output` records the aggregate rejection and allows
unrelated work to complete without advancing that candidate; and
- `validator_failure`: `warn_continue` by default, which advances a
structurally valid and otherwise unrejected candidate with explicit
incomplete-validation provenance and one genuine warning; `fail_run`
terminates the run.
Producer structural policy applies only to LLM-backed producers. Semantic and
validator-failure policies apply to any validated producer. Input and output
bindings do not accept `validation_policy`, and validator bindings do not own
terminal policy; they own only their decision and their own operational retry
budget. Candidate disposition belongs to the chunk, extract, merge, or
normalize producer binding after its complete validator chain has run.
Keep the current file-configuration version. The syntax is strictly
decodable without a version change. The project is pre-v1, but the behavior
and output changes should still be called out in the next release note and
downstream documentation.
## Stage-Specific Behavior
### Chunk
A generated chunk plan is structurally validated and materialized before the
validator chain runs. Semantic feedback applies to the exact raw chunker
response associated with that plan.
When an automatically reused chunk-plan record is rejected by the current
validator chain, treat the record as unusable for this invocation and enter
ordinary generation at attempt one. A cache hit is not a new model attempt and
does not supply model-facing assistant material. Do not overwrite the cached
record until a newly generated plan is accepted. Refresh and bypass modes
retain their existing publication rules.
### Extract
Each chunk-scoped extraction job owns its own attempt state and correction
conversation. One rejected chunk candidate does not cancel unrelated chunks or
lanes unless terminal policy converts it into a framework error. Deterministic
public ordering remains chunk-first and lane-second regardless of concurrent
completion.
### Merge And Normalize
Merge and normalize remain serial within a lane. A correction attempt receives
the same accepted upstream artifacts and references as the initial attempt.
The existing safe-fallback `NormalizeRetry` mechanism must be reconciled with
the shared attempt state rather than layered into a second retry loop: it uses
the same stage budget, retains its documented fallback behavior, and cannot
override a known validator rejection.
The initial feature supports one exact producer-supplied assistant response per
candidate attempt. Future multi-request normalization or batching must define
which response directly represents the candidate, or supply a new explicitly
reviewed correction protocol, before it can claim feedback-aware correction.
## LLM-Backed Validators
An LLM-backed validator uses the same scheduled PromptKit client, selected
profile, session, timeout, and structural-repair policy as other LLM-backed
modules. PromptKit may structurally repair its response inside one validator
attempt.
- A contract-valid validator response is its decision; Notarius does not ask a
second LLM to judge that judgment.
- A structurally invalid final validator response, transport failure, or
deterministic violation of the validator-result contract is a validator
execution failure.
- Validator execution retries reuse the immutable producer candidate and do
not regenerate it.
- Exhaustion invokes `validator_failure` policy and emits a genuine warning
under `warn_continue`.
This feature supplies the generic execution model only. It does not register a
production LLM-backed validator or change a D&D default validator chain.
## Provenance, Diagnostics, And Sensitive Data
Attempt debug output should make the state machine auditable. When debug is
enabled, record:
- producer attempt number and whether it was initial, error retry, module
retry, or semantic correction;
- PromptKit repair count and cumulative usage for every completion;
- each validator's configured-order outcome and validator attempt count;
- aggregate rejection codes and the bounded correction message;
- effective terminal policy and the decision it produced; and
- whether validation was complete, rejected, or incomplete.
Raw assistant responses and correction messages belong only in explicitly
requested detailed debug traces, following existing allowlisted content-file,
redaction, permission, and retention rules. Ordinary errors, CLI output,
warnings, manifests, checkpoints, caches, and run receipts contain identities,
counts, bounded safe summaries, and reason codes—not raw source or model
content.
The durable run manifest and rejection summaries should record enough
structured information to distinguish:
- the number and kinds of producer attempts;
- completed semantic rejection and all rejecting validator identities;
- incomplete validation and failed or skipped validator identities;
- the effective terminal policy and terminal result; and
- successful use of a correction attempt without treating it as a warning.
Warnings from abandoned producer attempts must not be promoted. Warnings from
the accepted attempt remain eligible. A warn-and-continue validator failure
produces one bounded, deterministically ordered warning per affected validator
after its retry budget is exhausted; detailed repeated failures stay in debug
provenance.
## Checkpoints, Caches, Concurrency, And Cancellation
- Effective validation policy, producer correction capability/protocol
version, validator chain, validator retry budgets, and prompt assets must all
affect checkpoint identity.
- Only accepted, completely validated stage outputs may be checkpointed or
reused. Rejected, structurally invalid, and validation-incomplete outputs
accepted under a permissive policy must not be written as reusable stage
checkpoints. This conservative rule avoids treating a transient validator
outage as durable validation success; a future checkpoint-status contract may
revisit it explicitly.
- Correction attempts use the same run-wide scheduler and worker bounds as
initial completions. No retry path may bypass provider admission.
- A scheduled permit covers the complete PromptKit operation, including its
internal structural repair, and is reacquired normally for a later Notarius
stage attempt.
- Parent cancellation dominates producer, validator, retry, debug, cache, and
checkpoint work. Cancellation never becomes a rejection, warning, or
incomplete-validation acceptance.
- Framework errors retain deterministic selection and cancellation behavior
across concurrently executing chunks and lanes.
## Architecture Record And Canonical Documentation
The target end state includes an accepted ADR that records:
- the separation between PromptKit structural repair, producer stage attempts,
and validator execution retries;
- the complete validator-chain aggregation rule and outcome precedence;
- the fresh reconstruction plus two-message correction protocol;
- module ownership of model-facing candidate material;
- deterministic producer and non-recursive validator behavior;
- default fail-closed and fail-open terminal policies; and
- provenance, cache, identity, and sensitive-data constraints.
The canonical owners describe the implemented behavior without duplicating
one another:
- `docs/policy/architecture.md` for durable validation and retry invariants;
- `docs/config.md` for fields, values, precedence, defaults, and validation;
- `docs/operations.md` for costs, failure behavior, warnings, debug handling,
and recovery;
- `docs/internal/pipeline.md` for the attempt state machine, aggregation,
checkpoint behavior, and concurrency;
- `docs/internal/llm.md` for appended correction messages and the distinction
from PromptKit repair;
- `docs/internal/modules.md` for producer and validator contracts;
- `docs/integrations/pkg-promptkit.md` for PromptKit v0.9.0,
`RunRequest.AppendedMessages`, supported message roles, application-owned
bounds, and the independently versioned upstream catalog boundary; and
- affected output and subprocess integration documents for durable validation
status and rejection summaries.
Until this target state is implemented, canonical current-state documentation
continues to describe the existing behavior.
## Testing Strategy
Tests should protect observable state-machine behavior rather than private
helper layout or exact prose. The target test suite includes:
- contract tests proving the first request is unchanged and a correction
request contains the same initial messages plus exactly one assistant and one
user message;
- behavioral runner tests for all-approved, multiple-rejection,
rejection-plus-failure, failure-only, skipped, retry-success, and each
terminal policy outcome;
- one representative path for chunk, extract, merge, and normalize, without
duplicating the complete state matrix at every stage;
- proof that all validators see immutable equivalent candidates and execute in
configured order after an earlier rejection or isolated failure;
- proof that validator retries reuse the candidate and do not consume producer
retries;
- proof that deterministic rejection does not repeat the producer;
- focused PromptKit-adapter tests proving that application-owned correction
messages map to the two intended PromptKit roles without content leakage;
- config parsing, precedence, invalid-placement, round-trip, redaction, and
digest tests for effective policy;
- checkpoint and chunk-cache tests for rejected, corrected, incomplete, and
accepted outcomes;
- warning, manifest, receipt, debug, and sensitive-content tests at their
canonical boundaries; and
- a small assembled D&D pipeline test proving a rejected direct extraction can
be corrected without a live provider.
Tests remain offline and deterministic. Do not reproduce PromptKit's internal
message-copying, rendered-hash, prepared-execution, capacity, or repair suite.
One representative adapter or assembled-run test should prove that PromptKit
structural repair remains usable after Notarius appends semantic-correction
messages. Do not snapshot full prompts or error prose, assert private
constants, or multiply equivalent tests across every D&D artifact family.
## Acceptance Criteria
- Every applicable validator runs in configured order and contributes one
explicit outcome before candidate disposition.
- Multiple semantic rejections produce one bounded, deterministic correction
request containing all actionable findings.
- Correction attempts reconstruct the exact ordinary prompt and append only
the latest defective assistant response and one correction message.
- Notarius pins PromptKit v0.9.0 and routes correction messages through
`RunRequest.AppendedMessages`; it does not maintain paired correction prompt
manifests or bypass PromptKit's normal execution path.
- Every correction-capable LLM producer exposes the exact single response that
directly controlled its candidate. Configuration rejects semantic retries
for compound producers that cannot satisfy that contract.
- The existing producer `retries` value is the only producer-stage budget;
PromptKit structural repair and validator execution retries remain separate.
- Deterministic producers are not repeated after semantic rejection.
- Validator execution failure is never described to the producer as a
candidate defect and never creates recursive semantic validation.
- Default terminal behavior is `fail_run` for structural failure and semantic
rejection, and `warn_continue` with explicit incomplete validation for
validator failure.
- Terminal policy resolves field by field from producer-binding override to
pipeline default to application default; individual validators do not own
candidate disposition.
- Permissive policy never advances known rejected or structurally invalid
output.
- Raw model responses and correction content are confined to model requests and
explicitly requested debug traces.
- Checkpoint, cache, manifest, warning, concurrency, cancellation, and
deterministic-ordering invariants remain intact.
- Canonical architecture, configuration, operations, internal, integration,
and ADR documentation accurately describe the implemented behavior.
- Focused, full, and race-enabled Go tests; vet; builds; example validation;
formatting; link checks; and repository hygiene checks pass.
## Non-Goals
- Adding the D&D combat-scene semantic validator.
- Redesigning the warning taxonomy beyond the warnings required for validator
failure and retry outcomes.
- Concurrent validator execution.
- Unbounded or accumulating conversational history.
- A second semantic retry counter.
- Recursive LLM judgment of LLM-validator decisions.
- Provider-specific retry policy or bypassing PromptKit.
- General workflow graphs or new pipeline stages.
- Large-collection reconciliation batching or a generic multi-response
correction protocol.