Document feedback-aware validation retries

This commit is contained in:
2026-08-27 01:35:23 +00:00
parent 7f01c3e79e
commit b2e83bd6e7
12 changed files with 229 additions and 38 deletions

View File

@@ -132,7 +132,7 @@ model: example-model
Keep credentials out of the local-backend object. A PromptKit profile may name Keep credentials out of the local-backend object. A PromptKit profile may name
its credential environment variable through `api_key_env`; set that variable its credential environment variable through `api_key_env`; set that variable
only in the run environment. PromptKit owns the only in the run environment. PromptKit owns the
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md), [pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md),
including `base_profile` inheritance. Notarius passes profiles through without including `base_profile` inheritance. Notarius passes profiles through without
merging them. Filesystem profiles cannot express PromptKit's in-memory merging them. Filesystem profiles cannot express PromptKit's in-memory
`APIKeyRequired` setting; an unset `api_key_env` is optional and may reach the `APIKeyRequired` setting; an unset `api_key_env` is optional and may reach the
@@ -252,12 +252,11 @@ explicit `null` and non-integer values are invalid. An explicit value on a
deterministic binding or validator is invalid, while a pipeline value simply deterministic binding or validator is invalid, while a pipeline value simply
does not apply to deterministic selections. does not apply to deterministic selections.
`validation_policy` records terminal-policy metadata for a complete producer `validation_policy` controls terminal disposition for one complete producer
attempt and validation chain. It may appear on a pipeline or a **chunk**, attempt and validator chain. It may appear on a pipeline or a **chunk**,
**extract**, **merge**, or **normalize** module binding; input, output, and **extract**, **merge**, or **normalize** module binding; input, output, and
validator bindings reject it. Every field is optional and resolves in binding, validator bindings reject it. Every field is optional and resolves in binding,
pipeline, then application-default order. The current runner records the pipeline, then application-default order:
resolved values but does not yet consume them for terminal disposition:
| Field | Values | Default | | Field | Values | Default |
| --- | --- | --- | | --- | --- | --- |
@@ -270,6 +269,16 @@ rejected. A deterministic producer may not explicitly set
**producer_structural_failure** on its binding, although a pipeline-level **producer_structural_failure** on its binding, although a pipeline-level
default remains valid for pipelines that include LLM-backed producers. default remains valid for pipelines that include LLM-backed producers.
After the producer binding's retry budget is exhausted, an invalid structured
response uses **producer_structural_failure**. One or more semantic validator
rejections use **semantic_rejection**; rejection takes precedence over an
exhausted validator failure or skip. With no rejection, an exhausted validator
failure or skip uses **validator_failure**. `reject_output` records the
terminal rejection without advancing that candidate. `warn_continue` is valid
only for validator execution failure: it advances a structurally valid,
otherwise unrejected result with incomplete-validation provenance and without
making it reusable checkpoint state.
A lane has these fields: A lane has these fields:
| Field | Type | Default | Rules | | Field | Type | Default | Rules |
@@ -309,7 +318,7 @@ extract:
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. | | **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
| **structured_output_repair_attempts** | integer | pipeline or prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for an LLM-backed binding. It overrides the pipeline value; explicit 0 disables structural repair. | | **structured_output_repair_attempts** | integer | pipeline or prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for an LLM-backed binding. It overrides the pipeline value; explicit 0 disables structural repair. |
| **validation_policy** | object | pipeline or application defaults | Optional field-by-field terminal-policy override for a chunk, extract, merge, or normalize binding. | | **validation_policy** | object | pipeline or application defaults | Optional field-by-field terminal-policy override for a chunk, extract, merge, or normalize binding. |
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. | | **retries** | integer | 0 | Non-negative additional complete producer attempts for chunk, extract, merge, and normalize bindings. This single budget covers operational errors, invalid structured output, module-requested normalization retry, and semantic correction. |
| **options** | object | none | Must satisfy the selected module. | | **options** | object | none | Must satisfy the selected module. |
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. | | **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
| **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. | | **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. |
@@ -318,13 +327,22 @@ Omitting **validators** uses the registered chain. **validators: []** selects
an empty chain; a non-empty list replaces the chain in the listed order. an empty chain; a non-empty list replaces the chain in the listed order.
Validator bindings accept only **module**, **llm_profile**, Validator bindings accept only **module**, **llm_profile**,
**structured_output_repair_attempts**, **retries**, and **options**. Their **structured_output_repair_attempts**, **retries**, and **options**. Their
**retries** value is a non-negative validator-execution budget and is valid **retries** value is a non-negative additional validator-execution budget and
only when the selected validator is LLM-backed. They reject is valid only when the selected validator is LLM-backed. A validator retry
rechecks the same immutable candidate; it never regenerates the producer.
They reject
**validation_policy**, **references**, and nested **validators**. Deterministic **validation_policy**, **references**, and nested **validators**. Deterministic
validators reject explicit **llm_profile** and validators reject explicit **llm_profile** and
**structured_output_repair_attempts**. **structured_output_repair_attempts**.
Deterministic module bindings also reject those explicit fields. Deterministic module bindings also reject those explicit fields.
An LLM-backed chunk, extract, merge, or normalize producer with both a
non-empty validator chain and positive **retries** must declare the supported
single-response correction capability. Preparation rejects a configuration
that could require semantic correction from a producer that cannot provide an
exact prior response. A deterministic producer, or an LLM attempt that did
not make a model call, cannot consume a semantic retry after rejection.
The **json** output module accepts optional **include_chunk_map** and The **json** output module accepts optional **include_chunk_map** and
**evidence_context** settings: **evidence_context** settings:

View File

@@ -101,6 +101,12 @@ root. Do not scan the output root for its newest directory, guess a run ID, or
construct a bundle path. Resolve `index_file` beneath `output_directory` and construct a bundle path. Resolve `index_file` beneath `output_directory` and
reject an absolute logical path or any result that escapes the bundle root. reject an absolute logical path or any result that escapes the bundle root.
The complete configuration uses the application validation defaults. A caller
that requires fully validated D&D artifacts must also require receipt
`validation_status: approved`; a successful `incomplete` result reflects the
configured validator-failure continuation policy and carries its bounded
validator provenance in `validation_summaries`.
Read `index.json` and locate each requested lane in `output_files` by its exact Read `index.json` and locate each requested lane in `output_files` by its exact
`lane_id`. Do not guess a lane filename. Before decoding a payload: `lane_id`. Do not guess a lane filename. Before decoding a payload:

View File

@@ -69,6 +69,13 @@ lanes. The caller decides which lane IDs are required for its own work and
which are optional; it should make that decision explicitly rather than infer which are optional; it should make that decision explicitly rather than infer
failure from the receipt counts alone. failure from the receipt counts alone.
When complete validation is required, also require receipt
`validation_status: approved` and inspect `validation_summaries`. A successful
run with `validation_status: incomplete` contains a structurally valid result
that advanced after validator execution could not complete under the configured
`warn_continue` policy. It is not reusable checkpoint state and should not be
silently treated as fully reviewed by the caller.
## Preserve Provenance And Handle Data Carefully ## Preserve Provenance And Handle Data Carefully
Keep the receipt with the published `manifest.json`, and retain Keep the receipt with the published `manifest.json`, and retain

View File

@@ -92,7 +92,7 @@ group into the following externally observable summaries:
| Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` | | Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
| Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` | | Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
| Source and references | `source_digests`, `references` | | Source and references | `source_digests`, `references` |
| Published result summaries | `normalized_outputs`, `rejected_outputs` | | Published result summaries | `normalized_outputs`, `rejected_outputs`, `validation_summaries` |
| Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` | | Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
`references` records provenance such as the target, slot, origin, digest, `references` records provenance such as the target, slot, origin, digest,
@@ -102,6 +102,16 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
provenance for the plan used by this run; cache records, debug artifacts, and provenance for the plan used by this run; cache records, debug artifacts, and
other operational state are not published as bundle files. other operational state are not published as bundle files.
Each `validation_summaries` entry is a bounded outcome for one producer result.
It has required `status`, `producer_attempt_count`, and `terminal_action`;
the stage and affected step, lane, module, or chunk identity are present when
applicable. `status` is `complete`, `rejected`, or `incomplete`.
`rejecting_validators`, `reason_codes`, and `incomplete_validators` preserve
configured validator order and omit later duplicates. Entries contain no raw
candidate response, correction guidance, validator diagnostic message, or
artifact payload. The same shape may appear as `validation` on an affected
rejection entry.
When present, `metadata.session_id` is the effective non-secret routing When present, `metadata.session_id` is the effective non-secret routing
correlation identifier used for the run. It can be visible to providers and is correlation identifier used for the run. It can be visible to providers and is
not a substitute for a cache or checkpoint identity. Its generation and not a substitute for a cache or checkpoint identity. Its generation and
@@ -127,7 +137,10 @@ distinct even when their profile, provider, and model are otherwise equal.
`rejected.json` is always an object with a `rejected` array. Each entry has `rejected.json` is always an object with a `rejected` array. Each entry has
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`, required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
`chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and `chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
`diagnostic_artifact_path` are present only when applicable. `diagnostic_artifact_path` are present only when applicable. An entry may also
contain the bounded `validation` summary described above; the existing singular
validator and reason fields remain the first configured rejection for
compatibility.
`warnings.json` is always an object with a `warnings` array. Each warning has `warnings.json` is always an object with a `warnings` array. Each warning has
`reason_code` and `message`; `scope` is optional. Both arrays are empty when `reason_code` and `message`; `scope` is optional. Both arrays are empty when

View File

@@ -15,7 +15,8 @@ Notarius relies on the root `promptkit` package to:
- construct an `Engine` with filesystem-backed prompt, schema, and optional - construct an `Engine` with filesystem-backed prompt, schema, and optional
operator and application-fallback profile sources; operator and application-fallback profile sources;
- prepare one frozen execution from a `RunRequest` with named inline artifacts, - prepare one frozen execution from a `RunRequest` with named inline artifacts,
variables, a direct session ID, prompt identity, and profile selection, then variables, a direct session ID, prompt identity, profile selection, and
optional appended rendered messages, then
record credential-redacted details and run that exact execution; record credential-redacted details and run that exact execution;
- return rendered debug material, validated structured output, selected - return rendered debug material, validated structured output, selected
profile, backend, effective model metadata, and token usage; profile, backend, effective model metadata, and token usage;
@@ -94,6 +95,23 @@ PromptKit v0.9.0 accepts only the `developer`, `system`, `user`, and
`assistant` text-chat roles after normalizing case and surrounding whitespace. `assistant` text-chat roles after normalizing case and surrounding whitespace.
Maintained Notarius prompt definitions use only `system` and `user`. Maintained Notarius prompt definitions use only `system` and `user`.
For application-owned semantic correction, Notarius uses PromptKit v0.9.0's
`RunRequest.AppendedMessages` after the ordinary rendered prompt. It supplies
exactly two messages in order: the latest validated producer response with
role `assistant`, then deterministic validation guidance with role `user`.
It never exposes a general caller-selected role API, accumulates earlier
correction turns, or changes the ordinary prompt prefix. Ordinary requests
leave appended messages unset.
PromptKit preserves supplied content but does not own Notarius's correction
bounds. Notarius rejects invalid UTF-8, blank, or oversized assistant material
(at most 1 MiB), guidance (at most 64 KiB), and combined content (at most
1,114,112 bytes) before preparing the request. The transport-neutral
application contract owns defensive copying and these limits. Default request
and terminal summaries retain only safe counts, digests, identities, and usage;
complete appended messages remain limited to the explicitly requested detailed
debug trace.
PromptKit now obtains its maintained OpenRouter and Rakestrawhome backend and PromptKit now obtains its maintained OpenRouter and Rakestrawhome backend and
profile catalogs from independently versioned transitive modules. Notarius profile catalogs from independently versioned transitive modules. Notarius
does not import or register either catalog; PromptKit retains catalog source, does not import or register either catalog; PromptKit retains catalog source,

View File

@@ -22,11 +22,15 @@ The current schema version is `notarius.run-result.v1`.
| `rejected_output_count` | Yes | Number of recorded rejected outputs. | | `rejected_output_count` | Yes | Number of recorded rejected outputs. |
| `warning_count` | Yes | Number of final run warnings. | | `warning_count` | Yes | Number of final run warnings. |
| `validation_status` | Yes | The final run manifest validation status. | | `validation_status` | Yes | The final run manifest validation status. |
| `validation_summaries` | No | Bounded per-producer validation outcomes; present when producer work ran. |
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. | | `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
For the production `json` output module, `index_file` is present only when the For the production `json` output module, `index_file` is present only when the
completed run returned exactly one logical output file named `index.json`. completed run returned exactly one logical output file named `index.json`.
For another output module, its absence does not indicate a failed run. For another output module, its absence does not indicate a failed run.
`validation_status` is `approved`, `rejected`, or `incomplete`; `incomplete`
means one or more otherwise accepted results advanced under validator-failure
`warn_continue`.
```json ```json
{ {
@@ -38,7 +42,17 @@ For another output module, its absence does not indicate a failed run.
"normalized_output_count": 6, "normalized_output_count": 6,
"rejected_output_count": 2, "rejected_output_count": 2,
"warning_count": 1, "warning_count": 1,
"validation_status": "rejected" "validation_status": "incomplete",
"validation_summaries": [
{
"stage": "extract",
"lane_id": "spells",
"status": "incomplete",
"incomplete_validators": ["dnd/spells/source_refs"],
"producer_attempt_count": 1,
"terminal_action": "warn_continue"
}
]
} }
``` ```
@@ -48,8 +62,11 @@ For another output module, its absence does not indicate a failed run.
paths. They identify the paths used by Notarius and do not resolve symlinks. paths. They identify the paths used by Notarius and do not resolve symlinks.
`output_directory` is the run-specific bundle, not the configured output root. `output_directory` is the run-specific bundle, not the configured output root.
The receipt is a summary and discovery document. It does not contain lane The receipt is a summary and discovery document. Its optional validation
descriptors, payloads, manifest data, rejections, warnings, or file contents. summaries contain only stable status, identity, validator names, reason codes,
attempt counts, and terminal actions. It does not contain lane descriptors,
payloads, manifest payloads, rejection messages, warnings, raw model responses,
correction guidance, or file contents.
For the production JSON output, resolve `index_file` beneath For the production JSON output, resolve `index_file` beneath
`output_directory`, reject path escapes, and use the `output_directory`, reject path escapes, and use the
[Published JSON Output contract](json-output.md) to discover logical files and [Published JSON Output contract](json-output.md) to discover logical files and

View File

@@ -50,6 +50,14 @@ correction do not add messages or introduce caller roles. Ordinary request
summaries record correction byte counts and digests only; complete messages are summaries record correction byte counts and digests only; complete messages are
available solely in an explicitly requested debug trace. available solely in an explicitly requested debug trace.
The adapter leaves the ordinary rendered message prefix, named inputs,
variables, session, profile, execution overrides, prepared-execution path, and
PromptKit repair policy unchanged for a corrected request. It never imports a
PromptKit message type into a module or pipeline contract. PromptKit reports
actual structural repair count and cumulative token usage per completion; the
pipeline's safe terminal debug record projects those values without copying
message content.
Client construction may also receive a run-wide reasoning-effort override from Client construction may also receive a run-wide reasoning-effort override from
the CLI factory boundary. The adapter copies the caller-owned pointer and the CLI factory boundary. The adapter copies the caller-owned pointer and
creates a fresh PromptKit execution override for each request: a nil pointer creates a fresh PromptKit execution override for each request: a nil pointer
@@ -237,10 +245,11 @@ or corrective call remain provider-neutral operational errors with the same
redaction boundary. redaction boundary.
Structural repair does not replace pipeline retry behavior: a binding's Structural repair does not replace pipeline retry behavior: a binding's
configured retry count reruns its complete stage attempt after an error or configured retry count reruns its complete stage attempt after an operational
rejection. The pipeline owns attempt lifecycle, validation chains, and retry or structural error, module-requested retry, or actionable semantic rejection.
diagnostics; see [Pipeline Internals](pipeline.md#validation-retries-and-output) The pipeline owns attempt lifecycle, validation chains, and retry diagnostics;
and the [binding reference](../config.md#module-bindings-and-validators). see [Pipeline Internals](pipeline.md#validation-retries-and-output) and the
[binding reference](../config.md#module-bindings-and-validators).
## Timeout Ownership ## Timeout Ownership
@@ -270,6 +279,13 @@ surfaced; when the completion already failed, its call error remains the
result. Debug-bundle location, retention, and handling are operational concerns result. Debug-bundle location, retention, and handling are operational concerns
documented in [Operations](../operations.md#debug-bundles). documented in [Operations](../operations.md#debug-bundles).
The attempt-terminal summary is a separate safe trace record: it contains
attempt kinds, validator outcome counts and reason codes, effective policy,
terminal action, and repair/usage references. It excludes raw assistant
responses and correction text. Those values can appear only in the explicitly
requested detailed prompt and response artifacts, which require sensitive-data
handling.
Run manifests receive selected profile summaries, including optional effective Run manifests receive selected profile summaries, including optional effective
backend and reasoning provenance, and component identities—not prompt, schema, backend and reasoning provenance, and component identities—not prompt, schema,
source, reference, or response content. The published field semantics belong source, reference, or response content. The published field semantics belong

View File

@@ -22,6 +22,15 @@ to bindings whose declared execution class is `llm_backed` and rejects a
binding-specific profile on a deterministic module. The user-facing precedence binding-specific profile on a deterministic module. The user-facing precedence
contract belongs in [Configuration](../config.md#pipelines). contract belongs in [Configuration](../config.md#pipelines).
An eligible LLM-backed chunk, extract, merge, or normalize producer may also
declare correction protocol `single_response_v1`. That declaration is a
promise that the implementation accepts one attempt-local semantic correction
and returns an owned copy of the exact one model response that directly
controlled the candidate. It must forward correction only to its structured
completion request; it must not manufacture prior-response material by
serializing a normalized artifact or expose opaque application IDs. Input,
output, validator, and deterministic specs cannot declare the protocol.
Implementations that accept options must provide both an option validator and Implementations that accept options must provide both an option validator and
a builder. The validator is used while resolving configuration; the builder a builder. The validator is used while resolving configuration; the builder
decodes the same options and constructs the implementation from the prepared decodes the same options and constructs the implementation from the prepared
@@ -91,6 +100,12 @@ combined-material bound preserves the deterministic result under the family's
fallback policy. Provider, transport, cancellation, and context-construction fallback policy. Provider, transport, cancellation, and context-construction
failures remain execution errors. failures remain execution errors.
When the engine actually makes a proposal call, its typed result carries the
owned exact proposal response under the same correction contract as other
eligible producers. Deterministic skip, limit, and fallback outcomes carry no
model candidate, so a later rejection applies terminal policy without spending
an ineffective semantic retry.
The core supplies a conservative generic prompt and the single private The core supplies a conservative generic prompt and the single private
response schema. A domain prompt may substitute its semantic instructions but response schema. A domain prompt may substitute its semantic instructions but
mounts the core-owned protocol and candidate/transcript presentation assets. mounts the core-owned protocol and candidate/transcript presentation assets.
@@ -111,6 +126,8 @@ its domain prompt.
3. Implement strict option decoding, construction, and the typed stage 3. Implement strict option decoding, construction, and the typed stage
interface. Preserve caller ownership: do not retain mutable request data interface. Preserve caller ownership: do not retain mutable request data
and return defensive copies where an implementation exposes stored data. and return defensive copies where an implementation exposes stored data.
If declaring correction capability, forward the request correction and
retain only the exact validated response that controlled the result.
4. Register the module through its typed registry helper and add it to the 4. Register the module through its typed registry helper and add it to the
owning family registrar. Add a default validator chain only when that owning family registrar. Add a default validator chain only when that
family owns the behavior; otherwise require an explicit compatible chain. family owns the behavior; otherwise require an explicit compatible chain.

View File

@@ -147,9 +147,10 @@ failure outcome for each validator.
The runner applies the binding's retry policy around a stage operation and its The runner applies the binding's retry policy around a stage operation and its
complete validation chain. It preserves warnings only from the final accepted complete validation chain. It preserves warnings only from the final accepted
or rejected attempt. Cancellation stops retries. Normalizer-specific retry or rejected attempt, plus one fixed warning per validator whose execution
directives consume this same budget and validate any final safe fallback through budget was exhausted under `warn_continue`. Cancellation stops retries.
the normalizer chain. Normalizer-specific retry directives consume this same budget and validate any
final safe fallback through the normalizer chain.
The artifact-neutral producer-attempt state machine owns that shared budget, The artifact-neutral producer-attempt state machine owns that shared budget,
attempt provenance, semantic-correction material, and terminal-policy attempt provenance, semantic-correction material, and terminal-policy
@@ -181,6 +182,15 @@ The CLI publishes those files only after the runner returns without a framework
error. Logical file names and schemas are defined by the [output integration error. Logical file names and schemas are defined by the [output integration
contracts](../integrations/). contracts](../integrations/).
For every completed producer disposition, the runner projects one bounded
validation summary to the manifest, the affected rejection when present, and
the CLI result receipt. The summary records status, configured-order rejecting
validators and reason codes, incomplete validators, producer-attempt count,
and terminal action. It contains no operator message, correction guidance, or
model response. `complete`, `rejected`, and `incomplete` describe the final
candidate disposition; a run-level `incomplete` status indicates at least one
current-run output advanced under `warn_continue`.
## Checkpoint And Debug Hooks ## Checkpoint And Debug Hooks
The runner receives checkpoint and debug interfaces rather than roots. It The runner receives checkpoint and debug interfaces rather than roots. It
@@ -190,6 +200,15 @@ handoff. Generated-reference dependencies participate in checkpoint decisions.
Selective recomputation can require a canonical accepted normalized predecessor Selective recomputation can require a canonical accepted normalized predecessor
before a dependent lane starts. before a dependent lane starts.
The runner writes successful checkpoint artifacts only after complete accepted
validation. Chunk plans follow the same rule for publication. A rejection,
invalid structured response, or incomplete validation is never reusable state;
the current run may still hand off an otherwise valid `warn_continue` result
according to its terminal policy. Attempt debug records retain safe kind,
validator, repair-usage, policy, and terminal-decision provenance. Full
assistant and correction content remains confined to the requested detailed
LLM trace.
Debug recording is attempt-scoped and application-owned. A failure to persist Debug recording is attempt-scoped and application-owned. A failure to persist
required debug data is a framework error. State roots, persistence, reason-code required debug data is a framework error. State roots, persistence, reason-code
meanings, resume, and cleanup are intentionally owned by meanings, resume, and cleanup are intentionally owned by

View File

@@ -109,6 +109,33 @@ On success, the command reports the output bundle path. A warning-bearing run
still succeeds and reports its warning count on standard error. Errors and still succeeds and reports its warning count on standard error. Errors and
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses). their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
## Validation Retries And Terminal Outcomes
Each producer binding has one outer **retries** budget. It covers complete
producer attempts for operational failures, invalid structured output,
normalizer fallback retry, and semantic correction. It is independent from
PromptKit's structural-repair calls inside one completion and from an
LLM-backed validator's own retry budget. A semantic correction rebuilds the
ordinary producer request and supplies only the latest rejected model response
plus aggregated validator guidance; it is not a conversation replay.
After the applicable budgets are exhausted, the resolved
[`validation_policy`](config.md#pipelines) determines the result. Structural
failure and semantic rejection normally fail the run; an explicit
`reject_output` records a rejection and allows unrelated work to finish. A
validator execution failure normally uses `warn_continue`, which keeps an
otherwise accepted result in the current run with `incomplete` validation
provenance. It emits one bounded warning for every validator whose execution
budget was exhausted. A corrected result that later passes validation does not
retain abandoned-attempt warnings.
Treat a successful process exit as a completed run, not as proof that every
candidate was fully validated. Inspect the receipt's `validation_status`,
`validation_summaries`, rejection count, and warning count when an orchestrator
requires complete validation. The durable fields and their meanings are owned
by the [run-result receipt](integrations/run-result.md) and
[published JSON output contract](integrations/json-output.md).
## Output Bundles ## Output Bundles
Each successful run receives a generated safe run identifier and writes beneath: Each successful run receives a generated safe run identifier and writes beneath:
@@ -158,7 +185,9 @@ The configured cache mode controls one invocation:
A reused plan is still materialized and validated against the current source. A reused plan is still materialized and validated against the current source.
If a prior plan no longer gives acceptable results, use a refresh run rather If a prior plan no longer gives acceptable results, use a refresh run rather
than editing cache files. Deleting a plan is recoverable but can repeat costly than editing cache files. Deleting a plan is recoverable but can repeat costly
chunking work. chunking work. A plan accepted only under incomplete validation is not
published, and a rejected cache hit falls through to ordinary generation rather
than becoming a correction candidate.
## Checkpoint Recording, Resume, And Recompute ## Checkpoint Recording, Resume, And Recompute
@@ -183,6 +212,12 @@ Reasoning-effort inheritance, replacement, and explicit clearing are distinct
runtime identities, so checkpoints created under one state are not reused by runtime identities, so checkpoints created under one state are not reused by
either of the others. either of the others.
Only accepted, completely validated chunk, extract, merge, and normalize
results are checkpointed for reuse. Rejected, structurally invalid, and
validation-incomplete producer results remain non-reusable, even when a
`warn_continue` result advanced during its original run. A resumed invocation
therefore reruns that producer rather than treating degraded state as accepted.
Checkpoint state is confined below an identity-specific path: Checkpoint state is confined below an identity-specific path:
~~~ ~~~
@@ -237,13 +272,16 @@ Only a [debug-enabled run](cli.md#run) creates a bundle:
~~~ ~~~
The summary contains redacted invocation and resolution information plus run, The summary contains redacted invocation and resolution information plus run,
warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace warning, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
contains allowlisted application diagnostic records and can include source or terminal records contain bounded attempt kinds, validator outcomes, policy,
derived application data. Neither surface is a cache input. Do not treat a decision, PromptKit repair count, and usage; they do not contain assistant
debug bundle as safe to share merely because its configuration summary is responses or complete correction messages. The trace contains allowlisted
redacted. Invocation metadata omits reasoning effort when it is inherited, application diagnostic records and can include source, model, and correction
records the replacement value when one is supplied, and records an empty value content. Neither surface is a cache input. Do not treat a debug bundle as safe
when inherited reasoning was explicitly cleared. to share merely because its configuration summary is redacted. Invocation
metadata omits reasoning effort when it is inherited, records the replacement
value when one is supplied, and records an empty value when inherited reasoning
was explicitly cleared.
Notarius never creates debug state without an explicit request and never Notarius never creates debug state without an explicit request and never
automatically deletes a requested bundle. If allocation succeeds, the command automatically deletes a requested bundle. If allocation succeeds, the command

View File

@@ -167,18 +167,22 @@ starting, waits for started work, and prevents output encoding.
## Validation ## Validation
Validation is a framework-managed boundary around outputs from chunk, extract, Validation is a framework-managed boundary around outputs from chunk, extract,
merge, and normalize stages. Validators receive immutable stage output merge, and normalize stages. Validators receive immutable stage output and
and make an explicit whole-output decision: approve, approve with warnings, or make an explicit whole-output decision: approve, approve with warnings,
reject. reject, fail, or skip when a runtime prerequisite is unavailable.
Typed artifact validators receive the domain value directly. Chunk validators Typed artifact validators receive the domain value directly. Chunk validators
receive source-zone chunks, while serialized validators receive immutable receive source-zone chunks, while serialized validators receive immutable
representation bytes and declared schema metadata. A validator registered for representation bytes and declared schema metadata. A validator registered for
one target or artifact kind cannot satisfy an incompatible selection. one target or artifact kind cannot satisfy an incompatible selection.
Rejection is a recorded pipeline outcome, not a framework execution error. The framework runs every applicable validator sequentially in configured order.
Validator execution failures are framework errors. Rejected output does not It aggregates rejections, exhausted validator failures, and skips before the
advance to the next stage. producer policy chooses a disposition. A completed rejection never advances.
With no rejection, an exhausted validator failure may fail the run or, under
the configured `warn_continue` policy, advance a structurally valid candidate
with explicit incomplete-validation provenance. Validators report findings;
they do not choose candidate disposition.
Default validator chains are production composition policy and are registered Default validator chains are production composition policy and are registered
centrally by stage and module. Configuration may replace a stage-local default, centrally by stage and module. Configuration may replace a stage-local default,
@@ -199,6 +203,15 @@ PromptKit owns bounded structural correction within one structured completion.
Notarius owns outer stage attempts, semantic validation, and acceptance policy; Notarius owns outer stage attempts, semantic validation, and acceptance policy;
the two budgets must remain separate. the two budgets must remain separate.
An LLM-backed producer can participate in semantic correction only when it
declares `single_response_v1` and returns the exact one response that directly
controlled its candidate. On an actionable rejection, the framework rebuilds
the ordinary request and appends only the latest defective response as an
`assistant` message plus one aggregated `user` correction message. This is a
fresh replacement request, not a growing conversation. The retry budgets,
terminal policy, and sensitive-data rationale are recorded in
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md).
When a model selects an application entity, callers must supply a contextual When a model selects an application entity, callers must supply a contextual
selection and deterministically attach the opaque application identity whenever selection and deterministically attach the opaque application identity whenever
the selection resolves exactly. Models do not receive or reproduce opaque the selection resolves exactly. Models do not receive or reproduce opaque
@@ -233,7 +246,8 @@ invalid or incompatible.
Run manifests record enough resolved pipeline, module, source, reference, and Run manifests record enough resolved pipeline, module, source, reference, and
LLM provenance to make a run auditable after configuration changes. Manifests LLM provenance to make a run auditable after configuration changes. Manifests
record identities and summaries rather than secret or large payload content. record identities and bounded validation summaries rather than secret, raw
model, correction, or large payload content.
## State, Output, And Safety ## State, Output, And Safety
@@ -253,12 +267,20 @@ an invocation that explicitly requests resume. Debug is never a cache input and
is never created without an explicit request. Pipeline modules receive is never created without an explicit request. Pipeline modules receive
collaborator interfaces and never physical roots. collaborator interfaces and never physical roots.
Only accepted, completely validated producer output is reusable checkpoint or
chunk-plan state. Rejected, structurally invalid, and validation-incomplete
results cannot become cache or checkpoint inputs, even when a
`warn_continue` result is allowed to advance in the current run.
Writes are atomic where practical. Paths for writes, moves, overwrites, and Writes are atomic where practical. Paths for writes, moves, overwrites, and
deletion must be narrow and explicit. Notarius never automatically deletes deletion must be narrow and explicit. Notarius never automatically deletes
output or requested debug bundles; cache cleanup is explicit and recoverable. output or requested debug bundles; cache cleanup is explicit and recoverable.
Secrets must not appear in errors, logs, output, cache, debug summaries, Secrets must not appear in errors, logs, output, cache, debug summaries,
traces, manifests, documentation, examples, or redacted configuration. Debug traces, manifests, documentation, examples, or redacted configuration. Raw
assistant responses and complete correction messages are attempt-local and are
excluded from ordinary durable records and summaries; the requested detailed
debug trace is the sole diagnostic surface allowed to retain them. Debug
collection is allowlisted to application-owned payloads and must not capture collection is allowlisted to application-owned payloads and must not capture
unrelated process environment values or filesystem content. Trace data may unrelated process environment values or filesystem content. Trace data may
contain application data and therefore inherits its sensitivity; operators own contain application data and therefore inherits its sensitivity; operators own

View File

@@ -671,7 +671,7 @@ Every terminal path has accurate bounded provenance, no sensitive correction
content leaks by default, and only completely validated output is reusable. content leaks by default, and only completely validated output is reusable.
This stage is one Terra prompt. This stage is one Terra prompt.
## Stage 15 — Update Canonical Documentation And Perform Final Verification ## Stage 15 — Update Canonical Documentation And Perform Final Verification
### Goal ### Goal