From b2e83bd6e7b392dce96f6e233829c4a0d64beedd Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 27 Aug 2026 01:35:23 +0000 Subject: [PATCH] Document feedback-aware validation retries --- docs/config.md | 34 ++++++++++++++----- docs/consumers/dnd-pipeline.md | 6 ++++ docs/consumers/subprocess.md | 7 ++++ docs/integrations/json-output.md | 17 ++++++++-- docs/integrations/pkg-promptkit.md | 20 ++++++++++- docs/integrations/run-result.md | 23 +++++++++++-- docs/internal/llm.md | 24 ++++++++++--- docs/internal/modules.md | 17 ++++++++++ docs/internal/pipeline.md | 25 ++++++++++++-- docs/operations.md | 54 +++++++++++++++++++++++++----- docs/policy/architecture.md | 38 ++++++++++++++++----- docs/roadmap/implementation.md | 2 +- 12 files changed, 229 insertions(+), 38 deletions(-) diff --git a/docs/config.md b/docs/config.md index 9fcc44c4..94231c8b 100644 --- a/docs/config.md +++ b/docs/config.md @@ -132,7 +132,7 @@ model: example-model Keep credentials out of the local-backend object. A PromptKit profile may name its credential environment variable through `api_key_env`; set that variable only in the run environment. PromptKit owns the -[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md), +[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md), including `base_profile` inheritance. Notarius passes profiles through without merging them. Filesystem profiles cannot express PromptKit's in-memory `APIKeyRequired` setting; an unset `api_key_env` is optional and may reach the @@ -252,12 +252,11 @@ explicit `null` and non-integer values are invalid. An explicit value on a deterministic binding or validator is invalid, while a pipeline value simply does not apply to deterministic selections. -`validation_policy` records terminal-policy metadata for a complete producer -attempt and validation chain. It may appear on a pipeline or a **chunk**, +`validation_policy` controls terminal disposition for one complete producer +attempt and validator chain. It may appear on a pipeline or a **chunk**, **extract**, **merge**, or **normalize** module binding; input, output, and validator bindings reject it. Every field is optional and resolves in binding, -pipeline, then application-default order. The current runner records the -resolved values but does not yet consume them for terminal disposition: +pipeline, then application-default order: | Field | Values | Default | | --- | --- | --- | @@ -270,6 +269,16 @@ rejected. A deterministic producer may not explicitly set **producer_structural_failure** on its binding, although a pipeline-level default remains valid for pipelines that include LLM-backed producers. +After the producer binding's retry budget is exhausted, an invalid structured +response uses **producer_structural_failure**. One or more semantic validator +rejections use **semantic_rejection**; rejection takes precedence over an +exhausted validator failure or skip. With no rejection, an exhausted validator +failure or skip uses **validator_failure**. `reject_output` records the +terminal rejection without advancing that candidate. `warn_continue` is valid +only for validator execution failure: it advances a structurally valid, +otherwise unrejected result with incomplete-validation provenance and without +making it reusable checkpoint state. + A lane has these fields: | Field | Type | Default | Rules | @@ -309,7 +318,7 @@ extract: | **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. | | **structured_output_repair_attempts** | integer | pipeline or prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for an LLM-backed binding. It overrides the pipeline value; explicit 0 disables structural repair. | | **validation_policy** | object | pipeline or application defaults | Optional field-by-field terminal-policy override for a chunk, extract, merge, or normalize binding. | -| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. | +| **retries** | integer | 0 | Non-negative additional complete producer attempts for chunk, extract, merge, and normalize bindings. This single budget covers operational errors, invalid structured output, module-requested normalization retry, and semantic correction. | | **options** | object | none | Must satisfy the selected module. | | **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. | | **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. | @@ -318,13 +327,22 @@ Omitting **validators** uses the registered chain. **validators: []** selects an empty chain; a non-empty list replaces the chain in the listed order. Validator bindings accept only **module**, **llm_profile**, **structured_output_repair_attempts**, **retries**, and **options**. Their -**retries** value is a non-negative validator-execution budget and is valid -only when the selected validator is LLM-backed. They reject +**retries** value is a non-negative additional validator-execution budget and +is valid only when the selected validator is LLM-backed. A validator retry +rechecks the same immutable candidate; it never regenerates the producer. +They reject **validation_policy**, **references**, and nested **validators**. Deterministic validators reject explicit **llm_profile** and **structured_output_repair_attempts**. Deterministic module bindings also reject those explicit fields. +An LLM-backed chunk, extract, merge, or normalize producer with both a +non-empty validator chain and positive **retries** must declare the supported +single-response correction capability. Preparation rejects a configuration +that could require semantic correction from a producer that cannot provide an +exact prior response. A deterministic producer, or an LLM attempt that did +not make a model call, cannot consume a semantic retry after rejection. + The **json** output module accepts optional **include_chunk_map** and **evidence_context** settings: diff --git a/docs/consumers/dnd-pipeline.md b/docs/consumers/dnd-pipeline.md index 727c2784..57b1173c 100644 --- a/docs/consumers/dnd-pipeline.md +++ b/docs/consumers/dnd-pipeline.md @@ -101,6 +101,12 @@ root. Do not scan the output root for its newest directory, guess a run ID, or construct a bundle path. Resolve `index_file` beneath `output_directory` and reject an absolute logical path or any result that escapes the bundle root. +The complete configuration uses the application validation defaults. A caller +that requires fully validated D&D artifacts must also require receipt +`validation_status: approved`; a successful `incomplete` result reflects the +configured validator-failure continuation policy and carries its bounded +validator provenance in `validation_summaries`. + Read `index.json` and locate each requested lane in `output_files` by its exact `lane_id`. Do not guess a lane filename. Before decoding a payload: diff --git a/docs/consumers/subprocess.md b/docs/consumers/subprocess.md index 3e55b868..e5dc94be 100644 --- a/docs/consumers/subprocess.md +++ b/docs/consumers/subprocess.md @@ -69,6 +69,13 @@ lanes. The caller decides which lane IDs are required for its own work and which are optional; it should make that decision explicitly rather than infer failure from the receipt counts alone. +When complete validation is required, also require receipt +`validation_status: approved` and inspect `validation_summaries`. A successful +run with `validation_status: incomplete` contains a structurally valid result +that advanced after validator execution could not complete under the configured +`warn_continue` policy. It is not reusable checkpoint state and should not be +silently treated as fully reviewed by the caller. + ## Preserve Provenance And Handle Data Carefully Keep the receipt with the published `manifest.json`, and retain diff --git a/docs/integrations/json-output.md b/docs/integrations/json-output.md index 8e2abfd4..9d83607a 100644 --- a/docs/integrations/json-output.md +++ b/docs/integrations/json-output.md @@ -92,7 +92,7 @@ group into the following externally observable summaries: | Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` | | Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` | | Source and references | `source_digests`, `references` | -| Published result summaries | `normalized_outputs`, `rejected_outputs` | +| Published result summaries | `normalized_outputs`, `rejected_outputs`, `validation_summaries` | | Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` | `references` records provenance such as the target, slot, origin, digest, @@ -102,6 +102,16 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is provenance for the plan used by this run; cache records, debug artifacts, and other operational state are not published as bundle files. +Each `validation_summaries` entry is a bounded outcome for one producer result. +It has required `status`, `producer_attempt_count`, and `terminal_action`; +the stage and affected step, lane, module, or chunk identity are present when +applicable. `status` is `complete`, `rejected`, or `incomplete`. +`rejecting_validators`, `reason_codes`, and `incomplete_validators` preserve +configured validator order and omit later duplicates. Entries contain no raw +candidate response, correction guidance, validator diagnostic message, or +artifact payload. The same shape may appear as `validation` on an affected +rejection entry. + When present, `metadata.session_id` is the effective non-secret routing correlation identifier used for the run. It can be visible to providers and is not a substitute for a cache or checkpoint identity. Its generation and @@ -127,7 +137,10 @@ distinct even when their profile, provider, and model are otherwise equal. `rejected.json` is always an object with a `rejected` array. Each entry has required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`, `chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and -`diagnostic_artifact_path` are present only when applicable. +`diagnostic_artifact_path` are present only when applicable. An entry may also +contain the bounded `validation` summary described above; the existing singular +validator and reason fields remain the first configured rejection for +compatibility. `warnings.json` is always an object with a `warnings` array. Each warning has `reason_code` and `message`; `scope` is optional. Both arrays are empty when diff --git a/docs/integrations/pkg-promptkit.md b/docs/integrations/pkg-promptkit.md index 30e5b17a..5edcc95f 100644 --- a/docs/integrations/pkg-promptkit.md +++ b/docs/integrations/pkg-promptkit.md @@ -15,7 +15,8 @@ Notarius relies on the root `promptkit` package to: - construct an `Engine` with filesystem-backed prompt, schema, and optional operator and application-fallback profile sources; - prepare one frozen execution from a `RunRequest` with named inline artifacts, - variables, a direct session ID, prompt identity, and profile selection, then + variables, a direct session ID, prompt identity, profile selection, and + optional appended rendered messages, then record credential-redacted details and run that exact execution; - return rendered debug material, validated structured output, selected profile, backend, effective model metadata, and token usage; @@ -94,6 +95,23 @@ PromptKit v0.9.0 accepts only the `developer`, `system`, `user`, and `assistant` text-chat roles after normalizing case and surrounding whitespace. Maintained Notarius prompt definitions use only `system` and `user`. +For application-owned semantic correction, Notarius uses PromptKit v0.9.0's +`RunRequest.AppendedMessages` after the ordinary rendered prompt. It supplies +exactly two messages in order: the latest validated producer response with +role `assistant`, then deterministic validation guidance with role `user`. +It never exposes a general caller-selected role API, accumulates earlier +correction turns, or changes the ordinary prompt prefix. Ordinary requests +leave appended messages unset. + +PromptKit preserves supplied content but does not own Notarius's correction +bounds. Notarius rejects invalid UTF-8, blank, or oversized assistant material +(at most 1 MiB), guidance (at most 64 KiB), and combined content (at most +1,114,112 bytes) before preparing the request. The transport-neutral +application contract owns defensive copying and these limits. Default request +and terminal summaries retain only safe counts, digests, identities, and usage; +complete appended messages remain limited to the explicitly requested detailed +debug trace. + PromptKit now obtains its maintained OpenRouter and Rakestrawhome backend and profile catalogs from independently versioned transitive modules. Notarius does not import or register either catalog; PromptKit retains catalog source, diff --git a/docs/integrations/run-result.md b/docs/integrations/run-result.md index 3310a14b..2d2c531b 100644 --- a/docs/integrations/run-result.md +++ b/docs/integrations/run-result.md @@ -22,11 +22,15 @@ The current schema version is `notarius.run-result.v1`. | `rejected_output_count` | Yes | Number of recorded rejected outputs. | | `warning_count` | Yes | Number of final run warnings. | | `validation_status` | Yes | The final run manifest validation status. | +| `validation_summaries` | No | Bounded per-producer validation outcomes; present when producer work ran. | | `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. | For the production `json` output module, `index_file` is present only when the completed run returned exactly one logical output file named `index.json`. For another output module, its absence does not indicate a failed run. +`validation_status` is `approved`, `rejected`, or `incomplete`; `incomplete` +means one or more otherwise accepted results advanced under validator-failure +`warn_continue`. ```json { @@ -38,7 +42,17 @@ For another output module, its absence does not indicate a failed run. "normalized_output_count": 6, "rejected_output_count": 2, "warning_count": 1, - "validation_status": "rejected" + "validation_status": "incomplete", + "validation_summaries": [ + { + "stage": "extract", + "lane_id": "spells", + "status": "incomplete", + "incomplete_validators": ["dnd/spells/source_refs"], + "producer_attempt_count": 1, + "terminal_action": "warn_continue" + } + ] } ``` @@ -48,8 +62,11 @@ For another output module, its absence does not indicate a failed run. paths. They identify the paths used by Notarius and do not resolve symlinks. `output_directory` is the run-specific bundle, not the configured output root. -The receipt is a summary and discovery document. It does not contain lane -descriptors, payloads, manifest data, rejections, warnings, or file contents. +The receipt is a summary and discovery document. Its optional validation +summaries contain only stable status, identity, validator names, reason codes, +attempt counts, and terminal actions. It does not contain lane descriptors, +payloads, manifest payloads, rejection messages, warnings, raw model responses, +correction guidance, or file contents. For the production JSON output, resolve `index_file` beneath `output_directory`, reject path escapes, and use the [Published JSON Output contract](json-output.md) to discover logical files and diff --git a/docs/internal/llm.md b/docs/internal/llm.md index 8021f938..95ac3630 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -50,6 +50,14 @@ correction do not add messages or introduce caller roles. Ordinary request summaries record correction byte counts and digests only; complete messages are available solely in an explicitly requested debug trace. +The adapter leaves the ordinary rendered message prefix, named inputs, +variables, session, profile, execution overrides, prepared-execution path, and +PromptKit repair policy unchanged for a corrected request. It never imports a +PromptKit message type into a module or pipeline contract. PromptKit reports +actual structural repair count and cumulative token usage per completion; the +pipeline's safe terminal debug record projects those values without copying +message content. + Client construction may also receive a run-wide reasoning-effort override from the CLI factory boundary. The adapter copies the caller-owned pointer and creates a fresh PromptKit execution override for each request: a nil pointer @@ -237,10 +245,11 @@ or corrective call remain provider-neutral operational errors with the same redaction boundary. Structural repair does not replace pipeline retry behavior: a binding's -configured retry count reruns its complete stage attempt after an error or -rejection. The pipeline owns attempt lifecycle, validation chains, and retry -diagnostics; see [Pipeline Internals](pipeline.md#validation-retries-and-output) -and the [binding reference](../config.md#module-bindings-and-validators). +configured retry count reruns its complete stage attempt after an operational +or structural error, module-requested retry, or actionable semantic rejection. +The pipeline owns attempt lifecycle, validation chains, and retry diagnostics; +see [Pipeline Internals](pipeline.md#validation-retries-and-output) and the +[binding reference](../config.md#module-bindings-and-validators). ## Timeout Ownership @@ -270,6 +279,13 @@ surfaced; when the completion already failed, its call error remains the result. Debug-bundle location, retention, and handling are operational concerns documented in [Operations](../operations.md#debug-bundles). +The attempt-terminal summary is a separate safe trace record: it contains +attempt kinds, validator outcome counts and reason codes, effective policy, +terminal action, and repair/usage references. It excludes raw assistant +responses and correction text. Those values can appear only in the explicitly +requested detailed prompt and response artifacts, which require sensitive-data +handling. + Run manifests receive selected profile summaries, including optional effective backend and reasoning provenance, and component identities—not prompt, schema, source, reference, or response content. The published field semantics belong diff --git a/docs/internal/modules.md b/docs/internal/modules.md index 815bf998..cb84cac7 100644 --- a/docs/internal/modules.md +++ b/docs/internal/modules.md @@ -22,6 +22,15 @@ to bindings whose declared execution class is `llm_backed` and rejects a binding-specific profile on a deterministic module. The user-facing precedence contract belongs in [Configuration](../config.md#pipelines). +An eligible LLM-backed chunk, extract, merge, or normalize producer may also +declare correction protocol `single_response_v1`. That declaration is a +promise that the implementation accepts one attempt-local semantic correction +and returns an owned copy of the exact one model response that directly +controlled the candidate. It must forward correction only to its structured +completion request; it must not manufacture prior-response material by +serializing a normalized artifact or expose opaque application IDs. Input, +output, validator, and deterministic specs cannot declare the protocol. + Implementations that accept options must provide both an option validator and a builder. The validator is used while resolving configuration; the builder decodes the same options and constructs the implementation from the prepared @@ -91,6 +100,12 @@ combined-material bound preserves the deterministic result under the family's fallback policy. Provider, transport, cancellation, and context-construction failures remain execution errors. +When the engine actually makes a proposal call, its typed result carries the +owned exact proposal response under the same correction contract as other +eligible producers. Deterministic skip, limit, and fallback outcomes carry no +model candidate, so a later rejection applies terminal policy without spending +an ineffective semantic retry. + The core supplies a conservative generic prompt and the single private response schema. A domain prompt may substitute its semantic instructions but mounts the core-owned protocol and candidate/transcript presentation assets. @@ -111,6 +126,8 @@ its domain prompt. 3. Implement strict option decoding, construction, and the typed stage interface. Preserve caller ownership: do not retain mutable request data and return defensive copies where an implementation exposes stored data. + If declaring correction capability, forward the request correction and + retain only the exact validated response that controlled the result. 4. Register the module through its typed registry helper and add it to the owning family registrar. Add a default validator chain only when that family owns the behavior; otherwise require an explicit compatible chain. diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 88dcd801..03b0119f 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -147,9 +147,10 @@ failure outcome for each validator. The runner applies the binding's retry policy around a stage operation and its complete validation chain. It preserves warnings only from the final accepted -or rejected attempt. Cancellation stops retries. Normalizer-specific retry -directives consume this same budget and validate any final safe fallback through -the normalizer chain. +or rejected attempt, plus one fixed warning per validator whose execution +budget was exhausted under `warn_continue`. Cancellation stops retries. +Normalizer-specific retry directives consume this same budget and validate any +final safe fallback through the normalizer chain. The artifact-neutral producer-attempt state machine owns that shared budget, attempt provenance, semantic-correction material, and terminal-policy @@ -181,6 +182,15 @@ The CLI publishes those files only after the runner returns without a framework error. Logical file names and schemas are defined by the [output integration contracts](../integrations/). +For every completed producer disposition, the runner projects one bounded +validation summary to the manifest, the affected rejection when present, and +the CLI result receipt. The summary records status, configured-order rejecting +validators and reason codes, incomplete validators, producer-attempt count, +and terminal action. It contains no operator message, correction guidance, or +model response. `complete`, `rejected`, and `incomplete` describe the final +candidate disposition; a run-level `incomplete` status indicates at least one +current-run output advanced under `warn_continue`. + ## Checkpoint And Debug Hooks The runner receives checkpoint and debug interfaces rather than roots. It @@ -190,6 +200,15 @@ handoff. Generated-reference dependencies participate in checkpoint decisions. Selective recomputation can require a canonical accepted normalized predecessor before a dependent lane starts. +The runner writes successful checkpoint artifacts only after complete accepted +validation. Chunk plans follow the same rule for publication. A rejection, +invalid structured response, or incomplete validation is never reusable state; +the current run may still hand off an otherwise valid `warn_continue` result +according to its terminal policy. Attempt debug records retain safe kind, +validator, repair-usage, policy, and terminal-decision provenance. Full +assistant and correction content remains confined to the requested detailed +LLM trace. + Debug recording is attempt-scoped and application-owned. A failure to persist required debug data is a framework error. State roots, persistence, reason-code meanings, resume, and cleanup are intentionally owned by diff --git a/docs/operations.md b/docs/operations.md index 0734a3a4..f29a5d86 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -109,6 +109,33 @@ On success, the command reports the output bundle path. A warning-bearing run still succeeds and reports its warning count on standard error. Errors and their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses). +## Validation Retries And Terminal Outcomes + +Each producer binding has one outer **retries** budget. It covers complete +producer attempts for operational failures, invalid structured output, +normalizer fallback retry, and semantic correction. It is independent from +PromptKit's structural-repair calls inside one completion and from an +LLM-backed validator's own retry budget. A semantic correction rebuilds the +ordinary producer request and supplies only the latest rejected model response +plus aggregated validator guidance; it is not a conversation replay. + +After the applicable budgets are exhausted, the resolved +[`validation_policy`](config.md#pipelines) determines the result. Structural +failure and semantic rejection normally fail the run; an explicit +`reject_output` records a rejection and allows unrelated work to finish. A +validator execution failure normally uses `warn_continue`, which keeps an +otherwise accepted result in the current run with `incomplete` validation +provenance. It emits one bounded warning for every validator whose execution +budget was exhausted. A corrected result that later passes validation does not +retain abandoned-attempt warnings. + +Treat a successful process exit as a completed run, not as proof that every +candidate was fully validated. Inspect the receipt's `validation_status`, +`validation_summaries`, rejection count, and warning count when an orchestrator +requires complete validation. The durable fields and their meanings are owned +by the [run-result receipt](integrations/run-result.md) and +[published JSON output contract](integrations/json-output.md). + ## Output Bundles Each successful run receives a generated safe run identifier and writes beneath: @@ -158,7 +185,9 @@ The configured cache mode controls one invocation: A reused plan is still materialized and validated against the current source. If a prior plan no longer gives acceptable results, use a refresh run rather than editing cache files. Deleting a plan is recoverable but can repeat costly -chunking work. +chunking work. A plan accepted only under incomplete validation is not +published, and a rejected cache hit falls through to ordinary generation rather +than becoming a correction candidate. ## Checkpoint Recording, Resume, And Recompute @@ -183,6 +212,12 @@ Reasoning-effort inheritance, replacement, and explicit clearing are distinct runtime identities, so checkpoints created under one state are not reused by either of the others. +Only accepted, completely validated chunk, extract, merge, and normalize +results are checkpointed for reuse. Rejected, structurally invalid, and +validation-incomplete producer results remain non-reusable, even when a +`warn_continue` result advanced during its original run. A resumed invocation +therefore reruns that producer rather than treating degraded state as accepted. + Checkpoint state is confined below an identity-specific path: ~~~ @@ -237,13 +272,16 @@ Only a [debug-enabled run](cli.md#run) creates a bundle: ~~~ The summary contains redacted invocation and resolution information plus run, -warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace -contains allowlisted application diagnostic records and can include source or -derived application data. Neither surface is a cache input. Do not treat a -debug bundle as safe to share merely because its configuration summary is -redacted. Invocation metadata omits reasoning effort when it is inherited, -records the replacement value when one is supplied, and records an empty value -when inherited reasoning was explicitly cleared. +warning, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt +terminal records contain bounded attempt kinds, validator outcomes, policy, +decision, PromptKit repair count, and usage; they do not contain assistant +responses or complete correction messages. The trace contains allowlisted +application diagnostic records and can include source, model, and correction +content. Neither surface is a cache input. Do not treat a debug bundle as safe +to share merely because its configuration summary is redacted. Invocation +metadata omits reasoning effort when it is inherited, records the replacement +value when one is supplied, and records an empty value when inherited reasoning +was explicitly cleared. Notarius never creates debug state without an explicit request and never automatically deletes a requested bundle. If allocation succeeds, the command diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index c8a389f3..1de3f86f 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -167,18 +167,22 @@ starting, waits for started work, and prevents output encoding. ## Validation Validation is a framework-managed boundary around outputs from chunk, extract, -merge, and normalize stages. Validators receive immutable stage output -and make an explicit whole-output decision: approve, approve with warnings, or -reject. +merge, and normalize stages. Validators receive immutable stage output and +make an explicit whole-output decision: approve, approve with warnings, +reject, fail, or skip when a runtime prerequisite is unavailable. Typed artifact validators receive the domain value directly. Chunk validators receive source-zone chunks, while serialized validators receive immutable representation bytes and declared schema metadata. A validator registered for one target or artifact kind cannot satisfy an incompatible selection. -Rejection is a recorded pipeline outcome, not a framework execution error. -Validator execution failures are framework errors. Rejected output does not -advance to the next stage. +The framework runs every applicable validator sequentially in configured order. +It aggregates rejections, exhausted validator failures, and skips before the +producer policy chooses a disposition. A completed rejection never advances. +With no rejection, an exhausted validator failure may fail the run or, under +the configured `warn_continue` policy, advance a structurally valid candidate +with explicit incomplete-validation provenance. Validators report findings; +they do not choose candidate disposition. Default validator chains are production composition policy and are registered centrally by stage and module. Configuration may replace a stage-local default, @@ -199,6 +203,15 @@ PromptKit owns bounded structural correction within one structured completion. Notarius owns outer stage attempts, semantic validation, and acceptance policy; the two budgets must remain separate. +An LLM-backed producer can participate in semantic correction only when it +declares `single_response_v1` and returns the exact one response that directly +controlled its candidate. On an actionable rejection, the framework rebuilds +the ordinary request and appends only the latest defective response as an +`assistant` message plus one aggregated `user` correction message. This is a +fresh replacement request, not a growing conversation. The retry budgets, +terminal policy, and sensitive-data rationale are recorded in +[ADR-0014](../adr/0014-feedback-aware-validation-retries.md). + When a model selects an application entity, callers must supply a contextual selection and deterministically attach the opaque application identity whenever the selection resolves exactly. Models do not receive or reproduce opaque @@ -233,7 +246,8 @@ invalid or incompatible. Run manifests record enough resolved pipeline, module, source, reference, and LLM provenance to make a run auditable after configuration changes. Manifests -record identities and summaries rather than secret or large payload content. +record identities and bounded validation summaries rather than secret, raw +model, correction, or large payload content. ## State, Output, And Safety @@ -253,12 +267,20 @@ an invocation that explicitly requests resume. Debug is never a cache input and is never created without an explicit request. Pipeline modules receive collaborator interfaces and never physical roots. +Only accepted, completely validated producer output is reusable checkpoint or +chunk-plan state. Rejected, structurally invalid, and validation-incomplete +results cannot become cache or checkpoint inputs, even when a +`warn_continue` result is allowed to advance in the current run. + Writes are atomic where practical. Paths for writes, moves, overwrites, and deletion must be narrow and explicit. Notarius never automatically deletes output or requested debug bundles; cache cleanup is explicit and recoverable. Secrets must not appear in errors, logs, output, cache, debug summaries, -traces, manifests, documentation, examples, or redacted configuration. Debug +traces, manifests, documentation, examples, or redacted configuration. Raw +assistant responses and complete correction messages are attempt-local and are +excluded from ordinary durable records and summaries; the requested detailed +debug trace is the sole diagnostic surface allowed to retain them. Debug collection is allowlisted to application-owned payloads and must not capture unrelated process environment values or filesystem content. Trace data may contain application data and therefore inherits its sensitivity; operators own diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index e4a687b4..b6f49736 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -671,7 +671,7 @@ Every terminal path has accurate bounded provenance, no sensitive correction content leaks by default, and only completely validated output is reusable. This stage is one Terra prompt. -## Stage 15 — Update Canonical Documentation And Perform Final Verification +## Stage 15 ✅ — Update Canonical Documentation And Perform Final Verification ### Goal