# Feedback-Aware Stage Validation Retries ## Status Proposed. This is the active feature roadmap for the next Notarius work set. Its design decisions are settled. Current behavior remains authoritative until this roadmap is implemented and the corresponding ADR and canonical documentation are updated. ## Purpose Make validation an effective corrective boundary around LLM-produced stage candidates. When deterministic or LLM-backed validators reject a structurally valid candidate, Notarius should give the producing model the complete, ordered validation feedback and use the stage's existing retry budget to ask for a corrected replacement. The feature must distinguish semantic rejection from producer failure and validator execution failure, preserve the boundary between PromptKit repair and Notarius stage retries, and remain safe under concurrency, cancellation, caching, checkpoints, and sensitive input. This work is domain-neutral. It establishes the framework behavior required by future LLM-backed validators such as D&D combat-scene review, but it does not add that validator. ## User Intent - A stage candidate should be evaluated by every applicable configured validator before Notarius decides whether to retry or terminate. - A semantic retry should be materially more useful than repeating the same request. The producing model should see its latest defective response and all actionable semantic feedback. - PromptKit's bounded structural repair and Notarius's stage retry loop are separate. Each stage attempt receives its own complete PromptKit repair budget; PromptKit repair never consumes or replenishes the stage budget. - Deterministic rejection, semantic rejection, producer structural failure, and validator execution failure are different outcomes and must not be collapsed into one generic error path. - The default posture is strict for known-invalid producer output and tolerant but visible when a validator itself cannot make a decision. - Corrective prompts must not expose opaque application identifiers, secrets, or unbounded diagnostic content merely because those values exist in an internal artifact or operator-facing error. ## Current State The current runner already provides useful foundations: - chunk, extract, merge, and normalize producer bindings have one `retries` value interpreted as additional stage attempts; - `runWithRetry` retries producer errors and semantic rejections within that budget; - PromptKit performs bounded structural repair inside each structured completion; - validator targets, execution classes, profile selection, repair policy, attempts, debug scopes, checkpoint identity, and deterministic public ordering are already explicit; and - the structured-completion response retains the model's validated raw bytes and PromptKit repair metadata. The current behavior is not yet the desired corrective workflow: - the runner repeats the ordinary producer request after rejection and does not pass the previous model response or validator feedback; - validation stops at the first rejection or execution failure, so later applicable validators do not contribute findings; - validator execution failure is immediately a framework error rather than a configurable incomplete-validation outcome; - `ValidationResult.Message` currently serves operator diagnostics and does not define separately bounded model-facing guidance; - typed stage results do not carry the exact model response needed for the next correction attempt; - validator-binding `retries` values participate in resolved configuration but are not used to retry a failed validator against the same candidate; and - Notarius still pins PromptKit v0.8, while PromptKit v0.9.0 now provides the append-only request-message API needed for application-owned correction attempts. ## Target End State For chunk, extract, merge, and normalize stages, Notarius owns one explicit candidate-attempt state machine: 1. The producer creates one candidate using the ordinary request. An LLM-backed producer may use PromptKit structural repair internally. 2. The framework establishes one immutable validation candidate and runs every applicable validator sequentially in configured order. 3. The framework aggregates approvals, warnings, semantic rejections, execution failures, and skipped-validator diagnostics without allowing one validator to mutate the candidate seen by another. 4. A candidate with one or more semantic rejections is never accepted. If the LLM-backed producer has another stage attempt available, Notarius rebuilds the complete original prompt and appends the latest defective assistant response followed by one application-owned correction message containing every actionable rejection. It then requests one complete replacement candidate. 5. A producer error consumes the same stage attempt budget under the existing retry rules, but semantic correction material is used only when a structurally valid candidate was actually rejected. 6. A validator execution failure is retried, when configured, against the same immutable candidate. It never regenerates the producer candidate by itself. 7. When budgets are exhausted, the configured terminal policies decide whether the run fails, a rejected output is recorded, or a structurally valid candidate advances with explicitly incomplete validation. The first attempt remains byte-for-byte the ordinary prompt rendered from the selected prompt definition. Every correction attempt starts from that same ordinary prompt rather than from the prior correction conversation. It appends exactly two messages: - an `assistant` message containing the producer-supplied exact defective response for the latest candidate; and - a `user` message containing deterministic, bounded, application-owned correction guidance and asking for one complete replacement response. The session ID, prompt ID and version, selected profile, reasoning settings, structured-output contract, repair budget, named inputs, variables, references, and reusable prompt prefix remain unchanged across stage attempts. ## Architectural Ownership ### PromptKit PromptKit continues to own prompt loading and rendering, profile resolution, backend admission, provider generation, structural validation, and bounded structural repair within one completion. A PromptKit repair conversation is private to that completion and is not exposed as a Notarius stage attempt. PromptKit v0.9.0 owns the mechanical operation of appending explicitly supplied messages to a normally rendered prompt before creating the immutable prepared execution. `RunRequest.AppendedMessages` preserves the original rendered messages as an exact prefix, validates and defensively copies additions, includes the complete sequence in prepared details and the rendered-prompt hash, and runs it through the ordinary generation and structural-repair path. PromptKit does not impose message-count, byte-size, token, or context-window limits and permits empty content, so Notarius retains its stricter application-owned correction validation and bounds. ### Notarius Framework The framework owns stage budgets, immutable candidate preparation, complete validator-chain execution, result aggregation, outcome precedence, correction message construction, terminal policy, public ordering, checkpoint effects, manifest summaries, warnings, and debug lifecycle. The framework must remain domain-neutral. It may format stable reason codes and validator-supplied corrective guidance, but it must not infer D&D or other domain rules from artifact JSON. ### Producers And Artifact Families The producing module owns prompt selection, prompt inputs, typed decoding, and the model-facing representation that corresponds to its candidate. An LLM-backed producer that supports feedback-aware correction must return the exact response material that the model should see as its prior assistant turn. It must not substitute a normalized artifact containing deterministically attached UUIDs or other opaque application identity. Artifact-family validators own semantic decisions and domain-specific corrective guidance. Operator-facing explanation and model-facing correction are separate contract fields even when their concise text happens to match. ## Validation Outcome Model Each validator invocation produces one of four framework outcomes: | Outcome | Meaning | Effect | | --- | --- | --- | | Approved | The validator completed and accepted the whole candidate. | Retain its warnings and continue the chain. | | Rejected | The validator completed and found a semantic defect in the candidate. | Record the finding, continue the chain, and make the candidate ineligible for acceptance. | | Failed | The validator could not return a usable decision because of an internal, transport, generation, structural-output, or result-invariant failure. | Retry that validator when eligible, then record incomplete validation and continue the chain unless cancellation or framework integrity prevents it. | | Skipped | Runtime prerequisites for an otherwise selected validator cannot be satisfied. | Record a deterministic incomplete-validation diagnostic and continue; do not invent a semantic decision. | Configured validator order controls invocation order and aggregate feedback order. Execution remains sequential initially. The framework must continue after a rejection and after an isolated validator failure when it can safely prepare the remaining validator requests. Cancellation, inability to preserve an immutable candidate, debug persistence failure, or another framework integrity failure remains immediately terminal. ### Outcome Precedence For one candidate, apply this precedence: 1. A producer structural failure means no acceptable candidate exists and cannot be converted into validator approval. 2. Any completed semantic rejection makes the candidate rejected, even when another validator failed or was skipped. 3. With no semantic rejection, a validator failure or skip makes validation incomplete and invokes the validator-failure policy. 4. Only a structurally valid candidate with no rejection and either complete validation or an explicit `warn_continue` decision may advance. Do not turn a known rejection into acceptance through a permissive validator-failure policy. Do not turn a structurally invalid response into a rejected-but-usable artifact. ## Corrective Feedback Contract `ValidationResult` should gain a separately bounded, optional model-facing correction field. A rejecting production validator should provide: - a stable reason code suitable for aggregation and provenance; - an operator-facing message suitable for ordinary diagnostics; and - concise corrective guidance that explains the violated rule without asking the model to reproduce opaque identity or leaking unrelated source data. The framework constructs one deterministic correction message from all rejections in validator order. Each entry identifies the stable reason code and corrective guidance. Duplicate identical entries may be collapsed while preserving first occurrence; distinct findings must not be discarded merely to shorten the message. If a validator rejects without model-facing guidance, the framework uses a generic reason-code-based correction rather than copying the operator message automatically. Warnings, validator failures, skipped diagnostics, provider messages, stack traces, debug paths, and sensitive values are not corrective guidance. They may be recorded through their proper diagnostic channels but must not be presented to the producer as candidate defects. The framework must validate UTF-8, role, non-empty content, and application-owned size limits before constructing the correction request. Oversized or invalid correction material is a framework-owned inability to perform a feedback retry; it must never be silently truncated into a misleading or syntactically defective assistant response. ## Producer Correction Contracts Introduce application-owned, defensively copied correction contracts at the framework boundary: - chunk, typed extraction, typed merge, and typed normalize results can carry optional model-facing candidate material associated with their returned value; - the corresponding requests can carry an optional correction containing the latest assistant material and aggregated guidance; - `StructuredCompletionRequest` can carry the two bounded appended messages without importing PromptKit types into module or pipeline contracts; and - the PromptKit adapter translates those application-owned messages into `RunRequest.AppendedMessages` using `promptkit.RoleAssistant` and `promptkit.RoleUser` before preparation. A semantic correction always supplies exactly two appended messages: the latest defective response as `assistant`, followed by the aggregate correction request as `user`. The framework does not expose the other PromptKit-supported roles through this contract and does not accumulate messages from earlier stage attempts. PromptKit preserves message content exactly, but Notarius must reject empty content and enforce its own per-message and aggregate byte limits before the adapter is called. Correction material is attempt-local sensitive data. It is not part of the artifact schema, checkpoint value, cache key, durable output bundle, ordinary error, or configuration summary. The policy and capability that affect execution do participate in resolved pipeline and checkpoint identity. LLM-backed modules selected with both `retries > 0` and a non-empty validator chain must declare whether they can produce and consume correction material. Preparation must reject a pipeline that could request feedback-aware semantic retries from an LLM-backed producer without that capability. An LLM-backed producer with no validators may continue to use its retry budget for operational failures without declaring semantic-correction capability. A correction-capable producer must supply the exact single LLM response that directly controlled the candidate being validated. Direct D&D chunk and extraction producers expose their exact structured response. The shared semantic-reconciliation path exposes its exact proposal response through its typed normalizers without turning request-local batch handles into durable identity. Deterministic transformations after that response are permitted only when the validated candidate remains directly traceable to it. A producer whose candidate combines multiple LLM responses is not correction-capable under this initial protocol. It may continue to use ordinary operational retries when no semantic correction can occur, but configuration must reject a validator-backed retry workflow for it. Supporting compound producers later requires a separately reviewed multi-response protocol; the framework must not synthesize an assistant message by serializing the final typed artifact. Deterministic producers do not receive correction material. A deterministic candidate rejected by validation immediately applies the terminal semantic rejection policy without consuming retries that cannot change the result. ## Retry Budgets ### Producer Stage Budget The existing producer binding `retries` field remains the sole outer stage budget. `retries: N` means at most `N` additional complete producer attempts after the initial attempt. Producer operational errors, producer structural failures, module-requested normalize retries, and semantic corrections all draw from this same budget. Do not add a separate semantic retry counter. Every LLM-backed producer attempt receives the configured PromptKit `structured_output_repair_attempts` value independently. Notarius does not decrement that value across stage attempts. ### Validator Budget Use the existing `retries` field on an LLM-backed validator binding for additional attempts to obtain a usable decision about the same immutable candidate. A completed approval or rejection is terminal for that validator and does not consume another validator attempt. A validator retry reconstructs the same ordinary validator prompt; it does not append semantic feedback about the validator's prior failed judgment and does not create a recursive Notarius correction loop. Reject a positive validator `retries` value on a deterministic validator at configuration resolution because repeating the same pure decision cannot improve it. Validator retries do not consume the producer stage budget. ## PromptKit v0.9.0 Adoption The target end state pins PromptKit v0.9.0 for correction requests. The resolved dependency graph includes its independently versioned OpenRouter and Rakestrawhome catalog modules through ordinary Go module resolution; Notarius must not import or register those catalogs directly. PromptKit continues to own their built-in backend and profile IDs, source precedence, credentials, and capacity behavior. Notarius's PromptKit compatibility documentation and built-in-profile checkpoint marker identify v0.9.0 rather than v0.8.0. The PromptKit release identity remains the conservative checkpoint identity for the exact catalog versions selected by that release; Notarius should not duplicate upstream catalog module versions in a second hand-maintained marker. PromptKit v0.9.0 restricts text-chat roles to `developer`, `system`, `user`, and `assistant`. Maintained Notarius prompt definitions already use only `system` and `user`; correction requests add only `assistant` and `user`. PromptKit `RunRequest` literals remain keyed. These compatibility conditions must remain covered by the ordinary production-asset and adapter checks without adding a brittle inventory test that merely counts prompt messages or literals. ## Terminal Policy Configuration Add an optional `validation_policy` object at pipeline scope and on chunk, extract, merge, and normalize producer bindings: ```yaml validation_policy: producer_structural_failure: fail_run semantic_rejection: fail_run validator_failure: warn_continue ``` The binding object overrides individual pipeline values; resolution is field-by-field in binding, pipeline, application-default order. Omitted values inherit rather than replacing the complete object. Explicit null, unknown fields, and unknown enum values are invalid. The effective policy is resolved and detached before execution, appears in redacted effective configuration and run provenance, and participates in the resolved pipeline digest and checkpoint identity. The initial enum values and defaults are: - `producer_structural_failure`: `fail_run` by default; `reject_output` may retain a terminal rejection and final raw candidate for debug, but may not advance or publish an invalid artifact; - `semantic_rejection`: `fail_run` by default after stage attempts are exhausted; `reject_output` records the aggregate rejection and allows unrelated work to complete without advancing that candidate; and - `validator_failure`: `warn_continue` by default, which advances a structurally valid and otherwise unrejected candidate with explicit incomplete-validation provenance and one genuine warning; `fail_run` terminates the run. Producer structural policy applies only to LLM-backed producers. Semantic and validator-failure policies apply to any validated producer. Input and output bindings do not accept `validation_policy`, and validator bindings do not own terminal policy; they own only their decision and their own operational retry budget. Candidate disposition belongs to the chunk, extract, merge, or normalize producer binding after its complete validator chain has run. Keep the current file-configuration version. The syntax is strictly decodable without a version change. The project is pre-v1, but the behavior and output changes should still be called out in the next release note and downstream documentation. ## Stage-Specific Behavior ### Chunk A generated chunk plan is structurally validated and materialized before the validator chain runs. Semantic feedback applies to the exact raw chunker response associated with that plan. When an automatically reused chunk-plan record is rejected by the current validator chain, treat the record as unusable for this invocation and enter ordinary generation at attempt one. A cache hit is not a new model attempt and does not supply model-facing assistant material. Do not overwrite the cached record until a newly generated plan is accepted. Refresh and bypass modes retain their existing publication rules. ### Extract Each chunk-scoped extraction job owns its own attempt state and correction conversation. One rejected chunk candidate does not cancel unrelated chunks or lanes unless terminal policy converts it into a framework error. Deterministic public ordering remains chunk-first and lane-second regardless of concurrent completion. ### Merge And Normalize Merge and normalize remain serial within a lane. A correction attempt receives the same accepted upstream artifacts and references as the initial attempt. The existing safe-fallback `NormalizeRetry` mechanism must be reconciled with the shared attempt state rather than layered into a second retry loop: it uses the same stage budget, retains its documented fallback behavior, and cannot override a known validator rejection. The initial feature supports one exact producer-supplied assistant response per candidate attempt. Future multi-request normalization or batching must define which response directly represents the candidate, or supply a new explicitly reviewed correction protocol, before it can claim feedback-aware correction. ## LLM-Backed Validators An LLM-backed validator uses the same scheduled PromptKit client, selected profile, session, timeout, and structural-repair policy as other LLM-backed modules. PromptKit may structurally repair its response inside one validator attempt. - A contract-valid validator response is its decision; Notarius does not ask a second LLM to judge that judgment. - A structurally invalid final validator response, transport failure, or deterministic violation of the validator-result contract is a validator execution failure. - Validator execution retries reuse the immutable producer candidate and do not regenerate it. - Exhaustion invokes `validator_failure` policy and emits a genuine warning under `warn_continue`. This feature supplies the generic execution model only. It does not register a production LLM-backed validator or change a D&D default validator chain. ## Provenance, Diagnostics, And Sensitive Data Attempt debug output should make the state machine auditable. When debug is enabled, record: - producer attempt number and whether it was initial, error retry, module retry, or semantic correction; - PromptKit repair count and cumulative usage for every completion; - each validator's configured-order outcome and validator attempt count; - aggregate rejection codes and the bounded correction message; - effective terminal policy and the decision it produced; and - whether validation was complete, rejected, or incomplete. Raw assistant responses and correction messages belong only in explicitly requested detailed debug traces, following existing allowlisted content-file, redaction, permission, and retention rules. Ordinary errors, CLI output, warnings, manifests, checkpoints, caches, and run receipts contain identities, counts, bounded safe summaries, and reason codes—not raw source or model content. The durable run manifest and rejection summaries should record enough structured information to distinguish: - the number and kinds of producer attempts; - completed semantic rejection and all rejecting validator identities; - incomplete validation and failed or skipped validator identities; - the effective terminal policy and terminal result; and - successful use of a correction attempt without treating it as a warning. Warnings from abandoned producer attempts must not be promoted. Warnings from the accepted attempt remain eligible. A warn-and-continue validator failure produces one bounded, deterministically ordered warning per affected validator after its retry budget is exhausted; detailed repeated failures stay in debug provenance. ## Checkpoints, Caches, Concurrency, And Cancellation - Effective validation policy, producer correction capability/protocol version, validator chain, validator retry budgets, and prompt assets must all affect checkpoint identity. - Only accepted, completely validated stage outputs may be checkpointed or reused. Rejected, structurally invalid, and validation-incomplete outputs accepted under a permissive policy must not be written as reusable stage checkpoints. This conservative rule avoids treating a transient validator outage as durable validation success; a future checkpoint-status contract may revisit it explicitly. - Correction attempts use the same run-wide scheduler and worker bounds as initial completions. No retry path may bypass provider admission. - A scheduled permit covers the complete PromptKit operation, including its internal structural repair, and is reacquired normally for a later Notarius stage attempt. - Parent cancellation dominates producer, validator, retry, debug, cache, and checkpoint work. Cancellation never becomes a rejection, warning, or incomplete-validation acceptance. - Framework errors retain deterministic selection and cancellation behavior across concurrently executing chunks and lanes. ## Architecture Record And Canonical Documentation The target end state includes an accepted ADR that records: - the separation between PromptKit structural repair, producer stage attempts, and validator execution retries; - the complete validator-chain aggregation rule and outcome precedence; - the fresh reconstruction plus two-message correction protocol; - module ownership of model-facing candidate material; - deterministic producer and non-recursive validator behavior; - default fail-closed and fail-open terminal policies; and - provenance, cache, identity, and sensitive-data constraints. The canonical owners describe the implemented behavior without duplicating one another: - `docs/policy/architecture.md` for durable validation and retry invariants; - `docs/config.md` for fields, values, precedence, defaults, and validation; - `docs/operations.md` for costs, failure behavior, warnings, debug handling, and recovery; - `docs/internal/pipeline.md` for the attempt state machine, aggregation, checkpoint behavior, and concurrency; - `docs/internal/llm.md` for appended correction messages and the distinction from PromptKit repair; - `docs/internal/modules.md` for producer and validator contracts; - `docs/integrations/pkg-promptkit.md` for PromptKit v0.9.0, `RunRequest.AppendedMessages`, supported message roles, application-owned bounds, and the independently versioned upstream catalog boundary; and - affected output and subprocess integration documents for durable validation status and rejection summaries. Until this target state is implemented, canonical current-state documentation continues to describe the existing behavior. ## Testing Strategy Tests should protect observable state-machine behavior rather than private helper layout or exact prose. The target test suite includes: - contract tests proving the first request is unchanged and a correction request contains the same initial messages plus exactly one assistant and one user message; - behavioral runner tests for all-approved, multiple-rejection, rejection-plus-failure, failure-only, skipped, retry-success, and each terminal policy outcome; - one representative path for chunk, extract, merge, and normalize, without duplicating the complete state matrix at every stage; - proof that all validators see immutable equivalent candidates and execute in configured order after an earlier rejection or isolated failure; - proof that validator retries reuse the candidate and do not consume producer retries; - proof that deterministic rejection does not repeat the producer; - focused PromptKit-adapter tests proving that application-owned correction messages map to the two intended PromptKit roles without content leakage; - config parsing, precedence, invalid-placement, round-trip, redaction, and digest tests for effective policy; - checkpoint and chunk-cache tests for rejected, corrected, incomplete, and accepted outcomes; - warning, manifest, receipt, debug, and sensitive-content tests at their canonical boundaries; and - a small assembled D&D pipeline test proving a rejected direct extraction can be corrected without a live provider. Tests remain offline and deterministic. Do not reproduce PromptKit's internal message-copying, rendered-hash, prepared-execution, capacity, or repair suite. One representative adapter or assembled-run test should prove that PromptKit structural repair remains usable after Notarius appends semantic-correction messages. Do not snapshot full prompts or error prose, assert private constants, or multiply equivalent tests across every D&D artifact family. ## Acceptance Criteria - Every applicable validator runs in configured order and contributes one explicit outcome before candidate disposition. - Multiple semantic rejections produce one bounded, deterministic correction request containing all actionable findings. - Correction attempts reconstruct the exact ordinary prompt and append only the latest defective assistant response and one correction message. - Notarius pins PromptKit v0.9.0 and routes correction messages through `RunRequest.AppendedMessages`; it does not maintain paired correction prompt manifests or bypass PromptKit's normal execution path. - Every correction-capable LLM producer exposes the exact single response that directly controlled its candidate. Configuration rejects semantic retries for compound producers that cannot satisfy that contract. - The existing producer `retries` value is the only producer-stage budget; PromptKit structural repair and validator execution retries remain separate. - Deterministic producers are not repeated after semantic rejection. - Validator execution failure is never described to the producer as a candidate defect and never creates recursive semantic validation. - Default terminal behavior is `fail_run` for structural failure and semantic rejection, and `warn_continue` with explicit incomplete validation for validator failure. - Terminal policy resolves field by field from producer-binding override to pipeline default to application default; individual validators do not own candidate disposition. - Permissive policy never advances known rejected or structurally invalid output. - Raw model responses and correction content are confined to model requests and explicitly requested debug traces. - Checkpoint, cache, manifest, warning, concurrency, cancellation, and deterministic-ordering invariants remain intact. - Canonical architecture, configuration, operations, internal, integration, and ADR documentation accurately describe the implemented behavior. - Focused, full, and race-enabled Go tests; vet; builds; example validation; formatting; link checks; and repository hygiene checks pass. ## Non-Goals - Adding the D&D combat-scene semantic validator. - Redesigning the warning taxonomy beyond the warnings required for validator failure and retry outcomes. - Concurrent validator execution. - Unbounded or accumulating conversational history. - A second semantic retry counter. - Recursive LLM judgment of LLM-validator decisions. - Provider-specific retry policy or bypassing PromptKit. - General workflow graphs or new pipeline stages. - Large-collection reconciliation batching or a generic multi-response correction protocol.