Document feedback-aware validation retries
This commit is contained in:
@@ -132,7 +132,7 @@ model: example-model
|
|||||||
Keep credentials out of the local-backend object. A PromptKit profile may name
|
Keep credentials out of the local-backend object. A PromptKit profile may name
|
||||||
its credential environment variable through `api_key_env`; set that variable
|
its credential environment variable through `api_key_env`; set that variable
|
||||||
only in the run environment. PromptKit owns the
|
only in the run environment. PromptKit owns the
|
||||||
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/formats.md),
|
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.9.0/docs/formats.md),
|
||||||
including `base_profile` inheritance. Notarius passes profiles through without
|
including `base_profile` inheritance. Notarius passes profiles through without
|
||||||
merging them. Filesystem profiles cannot express PromptKit's in-memory
|
merging them. Filesystem profiles cannot express PromptKit's in-memory
|
||||||
`APIKeyRequired` setting; an unset `api_key_env` is optional and may reach the
|
`APIKeyRequired` setting; an unset `api_key_env` is optional and may reach the
|
||||||
@@ -252,12 +252,11 @@ explicit `null` and non-integer values are invalid. An explicit value on a
|
|||||||
deterministic binding or validator is invalid, while a pipeline value simply
|
deterministic binding or validator is invalid, while a pipeline value simply
|
||||||
does not apply to deterministic selections.
|
does not apply to deterministic selections.
|
||||||
|
|
||||||
`validation_policy` records terminal-policy metadata for a complete producer
|
`validation_policy` controls terminal disposition for one complete producer
|
||||||
attempt and validation chain. It may appear on a pipeline or a **chunk**,
|
attempt and validator chain. It may appear on a pipeline or a **chunk**,
|
||||||
**extract**, **merge**, or **normalize** module binding; input, output, and
|
**extract**, **merge**, or **normalize** module binding; input, output, and
|
||||||
validator bindings reject it. Every field is optional and resolves in binding,
|
validator bindings reject it. Every field is optional and resolves in binding,
|
||||||
pipeline, then application-default order. The current runner records the
|
pipeline, then application-default order:
|
||||||
resolved values but does not yet consume them for terminal disposition:
|
|
||||||
|
|
||||||
| Field | Values | Default |
|
| Field | Values | Default |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
@@ -270,6 +269,16 @@ rejected. A deterministic producer may not explicitly set
|
|||||||
**producer_structural_failure** on its binding, although a pipeline-level
|
**producer_structural_failure** on its binding, although a pipeline-level
|
||||||
default remains valid for pipelines that include LLM-backed producers.
|
default remains valid for pipelines that include LLM-backed producers.
|
||||||
|
|
||||||
|
After the producer binding's retry budget is exhausted, an invalid structured
|
||||||
|
response uses **producer_structural_failure**. One or more semantic validator
|
||||||
|
rejections use **semantic_rejection**; rejection takes precedence over an
|
||||||
|
exhausted validator failure or skip. With no rejection, an exhausted validator
|
||||||
|
failure or skip uses **validator_failure**. `reject_output` records the
|
||||||
|
terminal rejection without advancing that candidate. `warn_continue` is valid
|
||||||
|
only for validator execution failure: it advances a structurally valid,
|
||||||
|
otherwise unrejected result with incomplete-validation provenance and without
|
||||||
|
making it reusable checkpoint state.
|
||||||
|
|
||||||
A lane has these fields:
|
A lane has these fields:
|
||||||
|
|
||||||
| Field | Type | Default | Rules |
|
| Field | Type | Default | Rules |
|
||||||
@@ -309,7 +318,7 @@ extract:
|
|||||||
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
|
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
|
||||||
| **structured_output_repair_attempts** | integer | pipeline or prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for an LLM-backed binding. It overrides the pipeline value; explicit 0 disables structural repair. |
|
| **structured_output_repair_attempts** | integer | pipeline or prompt-owned (1 in maintained production prompts) | Optional structural-repair limit from 0 through 3 for an LLM-backed binding. It overrides the pipeline value; explicit 0 disables structural repair. |
|
||||||
| **validation_policy** | object | pipeline or application defaults | Optional field-by-field terminal-policy override for a chunk, extract, merge, or normalize binding. |
|
| **validation_policy** | object | pipeline or application defaults | Optional field-by-field terminal-policy override for a chunk, extract, merge, or normalize binding. |
|
||||||
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
|
| **retries** | integer | 0 | Non-negative additional complete producer attempts for chunk, extract, merge, and normalize bindings. This single budget covers operational errors, invalid structured output, module-requested normalization retry, and semantic correction. |
|
||||||
| **options** | object | none | Must satisfy the selected module. |
|
| **options** | object | none | Must satisfy the selected module. |
|
||||||
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||||
| **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. |
|
| **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. |
|
||||||
@@ -318,13 +327,22 @@ Omitting **validators** uses the registered chain. **validators: []** selects
|
|||||||
an empty chain; a non-empty list replaces the chain in the listed order.
|
an empty chain; a non-empty list replaces the chain in the listed order.
|
||||||
Validator bindings accept only **module**, **llm_profile**,
|
Validator bindings accept only **module**, **llm_profile**,
|
||||||
**structured_output_repair_attempts**, **retries**, and **options**. Their
|
**structured_output_repair_attempts**, **retries**, and **options**. Their
|
||||||
**retries** value is a non-negative validator-execution budget and is valid
|
**retries** value is a non-negative additional validator-execution budget and
|
||||||
only when the selected validator is LLM-backed. They reject
|
is valid only when the selected validator is LLM-backed. A validator retry
|
||||||
|
rechecks the same immutable candidate; it never regenerates the producer.
|
||||||
|
They reject
|
||||||
**validation_policy**, **references**, and nested **validators**. Deterministic
|
**validation_policy**, **references**, and nested **validators**. Deterministic
|
||||||
validators reject explicit **llm_profile** and
|
validators reject explicit **llm_profile** and
|
||||||
**structured_output_repair_attempts**.
|
**structured_output_repair_attempts**.
|
||||||
Deterministic module bindings also reject those explicit fields.
|
Deterministic module bindings also reject those explicit fields.
|
||||||
|
|
||||||
|
An LLM-backed chunk, extract, merge, or normalize producer with both a
|
||||||
|
non-empty validator chain and positive **retries** must declare the supported
|
||||||
|
single-response correction capability. Preparation rejects a configuration
|
||||||
|
that could require semantic correction from a producer that cannot provide an
|
||||||
|
exact prior response. A deterministic producer, or an LLM attempt that did
|
||||||
|
not make a model call, cannot consume a semantic retry after rejection.
|
||||||
|
|
||||||
The **json** output module accepts optional **include_chunk_map** and
|
The **json** output module accepts optional **include_chunk_map** and
|
||||||
**evidence_context** settings:
|
**evidence_context** settings:
|
||||||
|
|
||||||
|
|||||||
@@ -101,6 +101,12 @@ root. Do not scan the output root for its newest directory, guess a run ID, or
|
|||||||
construct a bundle path. Resolve `index_file` beneath `output_directory` and
|
construct a bundle path. Resolve `index_file` beneath `output_directory` and
|
||||||
reject an absolute logical path or any result that escapes the bundle root.
|
reject an absolute logical path or any result that escapes the bundle root.
|
||||||
|
|
||||||
|
The complete configuration uses the application validation defaults. A caller
|
||||||
|
that requires fully validated D&D artifacts must also require receipt
|
||||||
|
`validation_status: approved`; a successful `incomplete` result reflects the
|
||||||
|
configured validator-failure continuation policy and carries its bounded
|
||||||
|
validator provenance in `validation_summaries`.
|
||||||
|
|
||||||
Read `index.json` and locate each requested lane in `output_files` by its exact
|
Read `index.json` and locate each requested lane in `output_files` by its exact
|
||||||
`lane_id`. Do not guess a lane filename. Before decoding a payload:
|
`lane_id`. Do not guess a lane filename. Before decoding a payload:
|
||||||
|
|
||||||
|
|||||||
@@ -69,6 +69,13 @@ lanes. The caller decides which lane IDs are required for its own work and
|
|||||||
which are optional; it should make that decision explicitly rather than infer
|
which are optional; it should make that decision explicitly rather than infer
|
||||||
failure from the receipt counts alone.
|
failure from the receipt counts alone.
|
||||||
|
|
||||||
|
When complete validation is required, also require receipt
|
||||||
|
`validation_status: approved` and inspect `validation_summaries`. A successful
|
||||||
|
run with `validation_status: incomplete` contains a structurally valid result
|
||||||
|
that advanced after validator execution could not complete under the configured
|
||||||
|
`warn_continue` policy. It is not reusable checkpoint state and should not be
|
||||||
|
silently treated as fully reviewed by the caller.
|
||||||
|
|
||||||
## Preserve Provenance And Handle Data Carefully
|
## Preserve Provenance And Handle Data Carefully
|
||||||
|
|
||||||
Keep the receipt with the published `manifest.json`, and retain
|
Keep the receipt with the published `manifest.json`, and retain
|
||||||
|
|||||||
@@ -92,7 +92,7 @@ group into the following externally observable summaries:
|
|||||||
| Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
|
| Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
|
||||||
| Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
|
| Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
|
||||||
| Source and references | `source_digests`, `references` |
|
| Source and references | `source_digests`, `references` |
|
||||||
| Published result summaries | `normalized_outputs`, `rejected_outputs` |
|
| Published result summaries | `normalized_outputs`, `rejected_outputs`, `validation_summaries` |
|
||||||
| Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
|
| Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
|
||||||
|
|
||||||
`references` records provenance such as the target, slot, origin, digest,
|
`references` records provenance such as the target, slot, origin, digest,
|
||||||
@@ -102,6 +102,16 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
|
|||||||
provenance for the plan used by this run; cache records, debug artifacts, and
|
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||||
other operational state are not published as bundle files.
|
other operational state are not published as bundle files.
|
||||||
|
|
||||||
|
Each `validation_summaries` entry is a bounded outcome for one producer result.
|
||||||
|
It has required `status`, `producer_attempt_count`, and `terminal_action`;
|
||||||
|
the stage and affected step, lane, module, or chunk identity are present when
|
||||||
|
applicable. `status` is `complete`, `rejected`, or `incomplete`.
|
||||||
|
`rejecting_validators`, `reason_codes`, and `incomplete_validators` preserve
|
||||||
|
configured validator order and omit later duplicates. Entries contain no raw
|
||||||
|
candidate response, correction guidance, validator diagnostic message, or
|
||||||
|
artifact payload. The same shape may appear as `validation` on an affected
|
||||||
|
rejection entry.
|
||||||
|
|
||||||
When present, `metadata.session_id` is the effective non-secret routing
|
When present, `metadata.session_id` is the effective non-secret routing
|
||||||
correlation identifier used for the run. It can be visible to providers and is
|
correlation identifier used for the run. It can be visible to providers and is
|
||||||
not a substitute for a cache or checkpoint identity. Its generation and
|
not a substitute for a cache or checkpoint identity. Its generation and
|
||||||
@@ -127,7 +137,10 @@ distinct even when their profile, provider, and model are otherwise equal.
|
|||||||
`rejected.json` is always an object with a `rejected` array. Each entry has
|
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||||
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
|
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
|
||||||
`chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
|
`chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
|
||||||
`diagnostic_artifact_path` are present only when applicable.
|
`diagnostic_artifact_path` are present only when applicable. An entry may also
|
||||||
|
contain the bounded `validation` summary described above; the existing singular
|
||||||
|
validator and reason fields remain the first configured rejection for
|
||||||
|
compatibility.
|
||||||
|
|
||||||
`warnings.json` is always an object with a `warnings` array. Each warning has
|
`warnings.json` is always an object with a `warnings` array. Each warning has
|
||||||
`reason_code` and `message`; `scope` is optional. Both arrays are empty when
|
`reason_code` and `message`; `scope` is optional. Both arrays are empty when
|
||||||
|
|||||||
@@ -15,7 +15,8 @@ Notarius relies on the root `promptkit` package to:
|
|||||||
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
- construct an `Engine` with filesystem-backed prompt, schema, and optional
|
||||||
operator and application-fallback profile sources;
|
operator and application-fallback profile sources;
|
||||||
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
|
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
|
||||||
variables, a direct session ID, prompt identity, and profile selection, then
|
variables, a direct session ID, prompt identity, profile selection, and
|
||||||
|
optional appended rendered messages, then
|
||||||
record credential-redacted details and run that exact execution;
|
record credential-redacted details and run that exact execution;
|
||||||
- return rendered debug material, validated structured output, selected
|
- return rendered debug material, validated structured output, selected
|
||||||
profile, backend, effective model metadata, and token usage;
|
profile, backend, effective model metadata, and token usage;
|
||||||
@@ -94,6 +95,23 @@ PromptKit v0.9.0 accepts only the `developer`, `system`, `user`, and
|
|||||||
`assistant` text-chat roles after normalizing case and surrounding whitespace.
|
`assistant` text-chat roles after normalizing case and surrounding whitespace.
|
||||||
Maintained Notarius prompt definitions use only `system` and `user`.
|
Maintained Notarius prompt definitions use only `system` and `user`.
|
||||||
|
|
||||||
|
For application-owned semantic correction, Notarius uses PromptKit v0.9.0's
|
||||||
|
`RunRequest.AppendedMessages` after the ordinary rendered prompt. It supplies
|
||||||
|
exactly two messages in order: the latest validated producer response with
|
||||||
|
role `assistant`, then deterministic validation guidance with role `user`.
|
||||||
|
It never exposes a general caller-selected role API, accumulates earlier
|
||||||
|
correction turns, or changes the ordinary prompt prefix. Ordinary requests
|
||||||
|
leave appended messages unset.
|
||||||
|
|
||||||
|
PromptKit preserves supplied content but does not own Notarius's correction
|
||||||
|
bounds. Notarius rejects invalid UTF-8, blank, or oversized assistant material
|
||||||
|
(at most 1 MiB), guidance (at most 64 KiB), and combined content (at most
|
||||||
|
1,114,112 bytes) before preparing the request. The transport-neutral
|
||||||
|
application contract owns defensive copying and these limits. Default request
|
||||||
|
and terminal summaries retain only safe counts, digests, identities, and usage;
|
||||||
|
complete appended messages remain limited to the explicitly requested detailed
|
||||||
|
debug trace.
|
||||||
|
|
||||||
PromptKit now obtains its maintained OpenRouter and Rakestrawhome backend and
|
PromptKit now obtains its maintained OpenRouter and Rakestrawhome backend and
|
||||||
profile catalogs from independently versioned transitive modules. Notarius
|
profile catalogs from independently versioned transitive modules. Notarius
|
||||||
does not import or register either catalog; PromptKit retains catalog source,
|
does not import or register either catalog; PromptKit retains catalog source,
|
||||||
|
|||||||
@@ -22,11 +22,15 @@ The current schema version is `notarius.run-result.v1`.
|
|||||||
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
|
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
|
||||||
| `warning_count` | Yes | Number of final run warnings. |
|
| `warning_count` | Yes | Number of final run warnings. |
|
||||||
| `validation_status` | Yes | The final run manifest validation status. |
|
| `validation_status` | Yes | The final run manifest validation status. |
|
||||||
|
| `validation_summaries` | No | Bounded per-producer validation outcomes; present when producer work ran. |
|
||||||
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
|
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
|
||||||
|
|
||||||
For the production `json` output module, `index_file` is present only when the
|
For the production `json` output module, `index_file` is present only when the
|
||||||
completed run returned exactly one logical output file named `index.json`.
|
completed run returned exactly one logical output file named `index.json`.
|
||||||
For another output module, its absence does not indicate a failed run.
|
For another output module, its absence does not indicate a failed run.
|
||||||
|
`validation_status` is `approved`, `rejected`, or `incomplete`; `incomplete`
|
||||||
|
means one or more otherwise accepted results advanced under validator-failure
|
||||||
|
`warn_continue`.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -38,7 +42,17 @@ For another output module, its absence does not indicate a failed run.
|
|||||||
"normalized_output_count": 6,
|
"normalized_output_count": 6,
|
||||||
"rejected_output_count": 2,
|
"rejected_output_count": 2,
|
||||||
"warning_count": 1,
|
"warning_count": 1,
|
||||||
"validation_status": "rejected"
|
"validation_status": "incomplete",
|
||||||
|
"validation_summaries": [
|
||||||
|
{
|
||||||
|
"stage": "extract",
|
||||||
|
"lane_id": "spells",
|
||||||
|
"status": "incomplete",
|
||||||
|
"incomplete_validators": ["dnd/spells/source_refs"],
|
||||||
|
"producer_attempt_count": 1,
|
||||||
|
"terminal_action": "warn_continue"
|
||||||
|
}
|
||||||
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -48,8 +62,11 @@ For another output module, its absence does not indicate a failed run.
|
|||||||
paths. They identify the paths used by Notarius and do not resolve symlinks.
|
paths. They identify the paths used by Notarius and do not resolve symlinks.
|
||||||
`output_directory` is the run-specific bundle, not the configured output root.
|
`output_directory` is the run-specific bundle, not the configured output root.
|
||||||
|
|
||||||
The receipt is a summary and discovery document. It does not contain lane
|
The receipt is a summary and discovery document. Its optional validation
|
||||||
descriptors, payloads, manifest data, rejections, warnings, or file contents.
|
summaries contain only stable status, identity, validator names, reason codes,
|
||||||
|
attempt counts, and terminal actions. It does not contain lane descriptors,
|
||||||
|
payloads, manifest payloads, rejection messages, warnings, raw model responses,
|
||||||
|
correction guidance, or file contents.
|
||||||
For the production JSON output, resolve `index_file` beneath
|
For the production JSON output, resolve `index_file` beneath
|
||||||
`output_directory`, reject path escapes, and use the
|
`output_directory`, reject path escapes, and use the
|
||||||
[Published JSON Output contract](json-output.md) to discover logical files and
|
[Published JSON Output contract](json-output.md) to discover logical files and
|
||||||
|
|||||||
@@ -50,6 +50,14 @@ correction do not add messages or introduce caller roles. Ordinary request
|
|||||||
summaries record correction byte counts and digests only; complete messages are
|
summaries record correction byte counts and digests only; complete messages are
|
||||||
available solely in an explicitly requested debug trace.
|
available solely in an explicitly requested debug trace.
|
||||||
|
|
||||||
|
The adapter leaves the ordinary rendered message prefix, named inputs,
|
||||||
|
variables, session, profile, execution overrides, prepared-execution path, and
|
||||||
|
PromptKit repair policy unchanged for a corrected request. It never imports a
|
||||||
|
PromptKit message type into a module or pipeline contract. PromptKit reports
|
||||||
|
actual structural repair count and cumulative token usage per completion; the
|
||||||
|
pipeline's safe terminal debug record projects those values without copying
|
||||||
|
message content.
|
||||||
|
|
||||||
Client construction may also receive a run-wide reasoning-effort override from
|
Client construction may also receive a run-wide reasoning-effort override from
|
||||||
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||||||
creates a fresh PromptKit execution override for each request: a nil pointer
|
creates a fresh PromptKit execution override for each request: a nil pointer
|
||||||
@@ -237,10 +245,11 @@ or corrective call remain provider-neutral operational errors with the same
|
|||||||
redaction boundary.
|
redaction boundary.
|
||||||
|
|
||||||
Structural repair does not replace pipeline retry behavior: a binding's
|
Structural repair does not replace pipeline retry behavior: a binding's
|
||||||
configured retry count reruns its complete stage attempt after an error or
|
configured retry count reruns its complete stage attempt after an operational
|
||||||
rejection. The pipeline owns attempt lifecycle, validation chains, and retry
|
or structural error, module-requested retry, or actionable semantic rejection.
|
||||||
diagnostics; see [Pipeline Internals](pipeline.md#validation-retries-and-output)
|
The pipeline owns attempt lifecycle, validation chains, and retry diagnostics;
|
||||||
and the [binding reference](../config.md#module-bindings-and-validators).
|
see [Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||||
|
[binding reference](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
## Timeout Ownership
|
## Timeout Ownership
|
||||||
|
|
||||||
@@ -270,6 +279,13 @@ surfaced; when the completion already failed, its call error remains the
|
|||||||
result. Debug-bundle location, retention, and handling are operational concerns
|
result. Debug-bundle location, retention, and handling are operational concerns
|
||||||
documented in [Operations](../operations.md#debug-bundles).
|
documented in [Operations](../operations.md#debug-bundles).
|
||||||
|
|
||||||
|
The attempt-terminal summary is a separate safe trace record: it contains
|
||||||
|
attempt kinds, validator outcome counts and reason codes, effective policy,
|
||||||
|
terminal action, and repair/usage references. It excludes raw assistant
|
||||||
|
responses and correction text. Those values can appear only in the explicitly
|
||||||
|
requested detailed prompt and response artifacts, which require sensitive-data
|
||||||
|
handling.
|
||||||
|
|
||||||
Run manifests receive selected profile summaries, including optional effective
|
Run manifests receive selected profile summaries, including optional effective
|
||||||
backend and reasoning provenance, and component identities—not prompt, schema,
|
backend and reasoning provenance, and component identities—not prompt, schema,
|
||||||
source, reference, or response content. The published field semantics belong
|
source, reference, or response content. The published field semantics belong
|
||||||
|
|||||||
@@ -22,6 +22,15 @@ to bindings whose declared execution class is `llm_backed` and rejects a
|
|||||||
binding-specific profile on a deterministic module. The user-facing precedence
|
binding-specific profile on a deterministic module. The user-facing precedence
|
||||||
contract belongs in [Configuration](../config.md#pipelines).
|
contract belongs in [Configuration](../config.md#pipelines).
|
||||||
|
|
||||||
|
An eligible LLM-backed chunk, extract, merge, or normalize producer may also
|
||||||
|
declare correction protocol `single_response_v1`. That declaration is a
|
||||||
|
promise that the implementation accepts one attempt-local semantic correction
|
||||||
|
and returns an owned copy of the exact one model response that directly
|
||||||
|
controlled the candidate. It must forward correction only to its structured
|
||||||
|
completion request; it must not manufacture prior-response material by
|
||||||
|
serializing a normalized artifact or expose opaque application IDs. Input,
|
||||||
|
output, validator, and deterministic specs cannot declare the protocol.
|
||||||
|
|
||||||
Implementations that accept options must provide both an option validator and
|
Implementations that accept options must provide both an option validator and
|
||||||
a builder. The validator is used while resolving configuration; the builder
|
a builder. The validator is used while resolving configuration; the builder
|
||||||
decodes the same options and constructs the implementation from the prepared
|
decodes the same options and constructs the implementation from the prepared
|
||||||
@@ -91,6 +100,12 @@ combined-material bound preserves the deterministic result under the family's
|
|||||||
fallback policy. Provider, transport, cancellation, and context-construction
|
fallback policy. Provider, transport, cancellation, and context-construction
|
||||||
failures remain execution errors.
|
failures remain execution errors.
|
||||||
|
|
||||||
|
When the engine actually makes a proposal call, its typed result carries the
|
||||||
|
owned exact proposal response under the same correction contract as other
|
||||||
|
eligible producers. Deterministic skip, limit, and fallback outcomes carry no
|
||||||
|
model candidate, so a later rejection applies terminal policy without spending
|
||||||
|
an ineffective semantic retry.
|
||||||
|
|
||||||
The core supplies a conservative generic prompt and the single private
|
The core supplies a conservative generic prompt and the single private
|
||||||
response schema. A domain prompt may substitute its semantic instructions but
|
response schema. A domain prompt may substitute its semantic instructions but
|
||||||
mounts the core-owned protocol and candidate/transcript presentation assets.
|
mounts the core-owned protocol and candidate/transcript presentation assets.
|
||||||
@@ -111,6 +126,8 @@ its domain prompt.
|
|||||||
3. Implement strict option decoding, construction, and the typed stage
|
3. Implement strict option decoding, construction, and the typed stage
|
||||||
interface. Preserve caller ownership: do not retain mutable request data
|
interface. Preserve caller ownership: do not retain mutable request data
|
||||||
and return defensive copies where an implementation exposes stored data.
|
and return defensive copies where an implementation exposes stored data.
|
||||||
|
If declaring correction capability, forward the request correction and
|
||||||
|
retain only the exact validated response that controlled the result.
|
||||||
4. Register the module through its typed registry helper and add it to the
|
4. Register the module through its typed registry helper and add it to the
|
||||||
owning family registrar. Add a default validator chain only when that
|
owning family registrar. Add a default validator chain only when that
|
||||||
family owns the behavior; otherwise require an explicit compatible chain.
|
family owns the behavior; otherwise require an explicit compatible chain.
|
||||||
|
|||||||
@@ -147,9 +147,10 @@ failure outcome for each validator.
|
|||||||
|
|
||||||
The runner applies the binding's retry policy around a stage operation and its
|
The runner applies the binding's retry policy around a stage operation and its
|
||||||
complete validation chain. It preserves warnings only from the final accepted
|
complete validation chain. It preserves warnings only from the final accepted
|
||||||
or rejected attempt. Cancellation stops retries. Normalizer-specific retry
|
or rejected attempt, plus one fixed warning per validator whose execution
|
||||||
directives consume this same budget and validate any final safe fallback through
|
budget was exhausted under `warn_continue`. Cancellation stops retries.
|
||||||
the normalizer chain.
|
Normalizer-specific retry directives consume this same budget and validate any
|
||||||
|
final safe fallback through the normalizer chain.
|
||||||
|
|
||||||
The artifact-neutral producer-attempt state machine owns that shared budget,
|
The artifact-neutral producer-attempt state machine owns that shared budget,
|
||||||
attempt provenance, semantic-correction material, and terminal-policy
|
attempt provenance, semantic-correction material, and terminal-policy
|
||||||
@@ -181,6 +182,15 @@ The CLI publishes those files only after the runner returns without a framework
|
|||||||
error. Logical file names and schemas are defined by the [output integration
|
error. Logical file names and schemas are defined by the [output integration
|
||||||
contracts](../integrations/).
|
contracts](../integrations/).
|
||||||
|
|
||||||
|
For every completed producer disposition, the runner projects one bounded
|
||||||
|
validation summary to the manifest, the affected rejection when present, and
|
||||||
|
the CLI result receipt. The summary records status, configured-order rejecting
|
||||||
|
validators and reason codes, incomplete validators, producer-attempt count,
|
||||||
|
and terminal action. It contains no operator message, correction guidance, or
|
||||||
|
model response. `complete`, `rejected`, and `incomplete` describe the final
|
||||||
|
candidate disposition; a run-level `incomplete` status indicates at least one
|
||||||
|
current-run output advanced under `warn_continue`.
|
||||||
|
|
||||||
## Checkpoint And Debug Hooks
|
## Checkpoint And Debug Hooks
|
||||||
|
|
||||||
The runner receives checkpoint and debug interfaces rather than roots. It
|
The runner receives checkpoint and debug interfaces rather than roots. It
|
||||||
@@ -190,6 +200,15 @@ handoff. Generated-reference dependencies participate in checkpoint decisions.
|
|||||||
Selective recomputation can require a canonical accepted normalized predecessor
|
Selective recomputation can require a canonical accepted normalized predecessor
|
||||||
before a dependent lane starts.
|
before a dependent lane starts.
|
||||||
|
|
||||||
|
The runner writes successful checkpoint artifacts only after complete accepted
|
||||||
|
validation. Chunk plans follow the same rule for publication. A rejection,
|
||||||
|
invalid structured response, or incomplete validation is never reusable state;
|
||||||
|
the current run may still hand off an otherwise valid `warn_continue` result
|
||||||
|
according to its terminal policy. Attempt debug records retain safe kind,
|
||||||
|
validator, repair-usage, policy, and terminal-decision provenance. Full
|
||||||
|
assistant and correction content remains confined to the requested detailed
|
||||||
|
LLM trace.
|
||||||
|
|
||||||
Debug recording is attempt-scoped and application-owned. A failure to persist
|
Debug recording is attempt-scoped and application-owned. A failure to persist
|
||||||
required debug data is a framework error. State roots, persistence, reason-code
|
required debug data is a framework error. State roots, persistence, reason-code
|
||||||
meanings, resume, and cleanup are intentionally owned by
|
meanings, resume, and cleanup are intentionally owned by
|
||||||
|
|||||||
@@ -109,6 +109,33 @@ On success, the command reports the output bundle path. A warning-bearing run
|
|||||||
still succeeds and reports its warning count on standard error. Errors and
|
still succeeds and reports its warning count on standard error. Errors and
|
||||||
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
|
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
|
||||||
|
|
||||||
|
## Validation Retries And Terminal Outcomes
|
||||||
|
|
||||||
|
Each producer binding has one outer **retries** budget. It covers complete
|
||||||
|
producer attempts for operational failures, invalid structured output,
|
||||||
|
normalizer fallback retry, and semantic correction. It is independent from
|
||||||
|
PromptKit's structural-repair calls inside one completion and from an
|
||||||
|
LLM-backed validator's own retry budget. A semantic correction rebuilds the
|
||||||
|
ordinary producer request and supplies only the latest rejected model response
|
||||||
|
plus aggregated validator guidance; it is not a conversation replay.
|
||||||
|
|
||||||
|
After the applicable budgets are exhausted, the resolved
|
||||||
|
[`validation_policy`](config.md#pipelines) determines the result. Structural
|
||||||
|
failure and semantic rejection normally fail the run; an explicit
|
||||||
|
`reject_output` records a rejection and allows unrelated work to finish. A
|
||||||
|
validator execution failure normally uses `warn_continue`, which keeps an
|
||||||
|
otherwise accepted result in the current run with `incomplete` validation
|
||||||
|
provenance. It emits one bounded warning for every validator whose execution
|
||||||
|
budget was exhausted. A corrected result that later passes validation does not
|
||||||
|
retain abandoned-attempt warnings.
|
||||||
|
|
||||||
|
Treat a successful process exit as a completed run, not as proof that every
|
||||||
|
candidate was fully validated. Inspect the receipt's `validation_status`,
|
||||||
|
`validation_summaries`, rejection count, and warning count when an orchestrator
|
||||||
|
requires complete validation. The durable fields and their meanings are owned
|
||||||
|
by the [run-result receipt](integrations/run-result.md) and
|
||||||
|
[published JSON output contract](integrations/json-output.md).
|
||||||
|
|
||||||
## Output Bundles
|
## Output Bundles
|
||||||
|
|
||||||
Each successful run receives a generated safe run identifier and writes beneath:
|
Each successful run receives a generated safe run identifier and writes beneath:
|
||||||
@@ -158,7 +185,9 @@ The configured cache mode controls one invocation:
|
|||||||
A reused plan is still materialized and validated against the current source.
|
A reused plan is still materialized and validated against the current source.
|
||||||
If a prior plan no longer gives acceptable results, use a refresh run rather
|
If a prior plan no longer gives acceptable results, use a refresh run rather
|
||||||
than editing cache files. Deleting a plan is recoverable but can repeat costly
|
than editing cache files. Deleting a plan is recoverable but can repeat costly
|
||||||
chunking work.
|
chunking work. A plan accepted only under incomplete validation is not
|
||||||
|
published, and a rejected cache hit falls through to ordinary generation rather
|
||||||
|
than becoming a correction candidate.
|
||||||
|
|
||||||
## Checkpoint Recording, Resume, And Recompute
|
## Checkpoint Recording, Resume, And Recompute
|
||||||
|
|
||||||
@@ -183,6 +212,12 @@ Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
|||||||
runtime identities, so checkpoints created under one state are not reused by
|
runtime identities, so checkpoints created under one state are not reused by
|
||||||
either of the others.
|
either of the others.
|
||||||
|
|
||||||
|
Only accepted, completely validated chunk, extract, merge, and normalize
|
||||||
|
results are checkpointed for reuse. Rejected, structurally invalid, and
|
||||||
|
validation-incomplete producer results remain non-reusable, even when a
|
||||||
|
`warn_continue` result advanced during its original run. A resumed invocation
|
||||||
|
therefore reruns that producer rather than treating degraded state as accepted.
|
||||||
|
|
||||||
Checkpoint state is confined below an identity-specific path:
|
Checkpoint state is confined below an identity-specific path:
|
||||||
|
|
||||||
~~~
|
~~~
|
||||||
@@ -237,13 +272,16 @@ Only a [debug-enabled run](cli.md#run) creates a bundle:
|
|||||||
~~~
|
~~~
|
||||||
|
|
||||||
The summary contains redacted invocation and resolution information plus run,
|
The summary contains redacted invocation and resolution information plus run,
|
||||||
warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
|
warning, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
|
||||||
contains allowlisted application diagnostic records and can include source or
|
terminal records contain bounded attempt kinds, validator outcomes, policy,
|
||||||
derived application data. Neither surface is a cache input. Do not treat a
|
decision, PromptKit repair count, and usage; they do not contain assistant
|
||||||
debug bundle as safe to share merely because its configuration summary is
|
responses or complete correction messages. The trace contains allowlisted
|
||||||
redacted. Invocation metadata omits reasoning effort when it is inherited,
|
application diagnostic records and can include source, model, and correction
|
||||||
records the replacement value when one is supplied, and records an empty value
|
content. Neither surface is a cache input. Do not treat a debug bundle as safe
|
||||||
when inherited reasoning was explicitly cleared.
|
to share merely because its configuration summary is redacted. Invocation
|
||||||
|
metadata omits reasoning effort when it is inherited, records the replacement
|
||||||
|
value when one is supplied, and records an empty value when inherited reasoning
|
||||||
|
was explicitly cleared.
|
||||||
|
|
||||||
Notarius never creates debug state without an explicit request and never
|
Notarius never creates debug state without an explicit request and never
|
||||||
automatically deletes a requested bundle. If allocation succeeds, the command
|
automatically deletes a requested bundle. If allocation succeeds, the command
|
||||||
|
|||||||
@@ -167,18 +167,22 @@ starting, waits for started work, and prevents output encoding.
|
|||||||
## Validation
|
## Validation
|
||||||
|
|
||||||
Validation is a framework-managed boundary around outputs from chunk, extract,
|
Validation is a framework-managed boundary around outputs from chunk, extract,
|
||||||
merge, and normalize stages. Validators receive immutable stage output
|
merge, and normalize stages. Validators receive immutable stage output and
|
||||||
and make an explicit whole-output decision: approve, approve with warnings, or
|
make an explicit whole-output decision: approve, approve with warnings,
|
||||||
reject.
|
reject, fail, or skip when a runtime prerequisite is unavailable.
|
||||||
|
|
||||||
Typed artifact validators receive the domain value directly. Chunk validators
|
Typed artifact validators receive the domain value directly. Chunk validators
|
||||||
receive source-zone chunks, while serialized validators receive immutable
|
receive source-zone chunks, while serialized validators receive immutable
|
||||||
representation bytes and declared schema metadata. A validator registered for
|
representation bytes and declared schema metadata. A validator registered for
|
||||||
one target or artifact kind cannot satisfy an incompatible selection.
|
one target or artifact kind cannot satisfy an incompatible selection.
|
||||||
|
|
||||||
Rejection is a recorded pipeline outcome, not a framework execution error.
|
The framework runs every applicable validator sequentially in configured order.
|
||||||
Validator execution failures are framework errors. Rejected output does not
|
It aggregates rejections, exhausted validator failures, and skips before the
|
||||||
advance to the next stage.
|
producer policy chooses a disposition. A completed rejection never advances.
|
||||||
|
With no rejection, an exhausted validator failure may fail the run or, under
|
||||||
|
the configured `warn_continue` policy, advance a structurally valid candidate
|
||||||
|
with explicit incomplete-validation provenance. Validators report findings;
|
||||||
|
they do not choose candidate disposition.
|
||||||
|
|
||||||
Default validator chains are production composition policy and are registered
|
Default validator chains are production composition policy and are registered
|
||||||
centrally by stage and module. Configuration may replace a stage-local default,
|
centrally by stage and module. Configuration may replace a stage-local default,
|
||||||
@@ -199,6 +203,15 @@ PromptKit owns bounded structural correction within one structured completion.
|
|||||||
Notarius owns outer stage attempts, semantic validation, and acceptance policy;
|
Notarius owns outer stage attempts, semantic validation, and acceptance policy;
|
||||||
the two budgets must remain separate.
|
the two budgets must remain separate.
|
||||||
|
|
||||||
|
An LLM-backed producer can participate in semantic correction only when it
|
||||||
|
declares `single_response_v1` and returns the exact one response that directly
|
||||||
|
controlled its candidate. On an actionable rejection, the framework rebuilds
|
||||||
|
the ordinary request and appends only the latest defective response as an
|
||||||
|
`assistant` message plus one aggregated `user` correction message. This is a
|
||||||
|
fresh replacement request, not a growing conversation. The retry budgets,
|
||||||
|
terminal policy, and sensitive-data rationale are recorded in
|
||||||
|
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md).
|
||||||
|
|
||||||
When a model selects an application entity, callers must supply a contextual
|
When a model selects an application entity, callers must supply a contextual
|
||||||
selection and deterministically attach the opaque application identity whenever
|
selection and deterministically attach the opaque application identity whenever
|
||||||
the selection resolves exactly. Models do not receive or reproduce opaque
|
the selection resolves exactly. Models do not receive or reproduce opaque
|
||||||
@@ -233,7 +246,8 @@ invalid or incompatible.
|
|||||||
|
|
||||||
Run manifests record enough resolved pipeline, module, source, reference, and
|
Run manifests record enough resolved pipeline, module, source, reference, and
|
||||||
LLM provenance to make a run auditable after configuration changes. Manifests
|
LLM provenance to make a run auditable after configuration changes. Manifests
|
||||||
record identities and summaries rather than secret or large payload content.
|
record identities and bounded validation summaries rather than secret, raw
|
||||||
|
model, correction, or large payload content.
|
||||||
|
|
||||||
## State, Output, And Safety
|
## State, Output, And Safety
|
||||||
|
|
||||||
@@ -253,12 +267,20 @@ an invocation that explicitly requests resume. Debug is never a cache input and
|
|||||||
is never created without an explicit request. Pipeline modules receive
|
is never created without an explicit request. Pipeline modules receive
|
||||||
collaborator interfaces and never physical roots.
|
collaborator interfaces and never physical roots.
|
||||||
|
|
||||||
|
Only accepted, completely validated producer output is reusable checkpoint or
|
||||||
|
chunk-plan state. Rejected, structurally invalid, and validation-incomplete
|
||||||
|
results cannot become cache or checkpoint inputs, even when a
|
||||||
|
`warn_continue` result is allowed to advance in the current run.
|
||||||
|
|
||||||
Writes are atomic where practical. Paths for writes, moves, overwrites, and
|
Writes are atomic where practical. Paths for writes, moves, overwrites, and
|
||||||
deletion must be narrow and explicit. Notarius never automatically deletes
|
deletion must be narrow and explicit. Notarius never automatically deletes
|
||||||
output or requested debug bundles; cache cleanup is explicit and recoverable.
|
output or requested debug bundles; cache cleanup is explicit and recoverable.
|
||||||
|
|
||||||
Secrets must not appear in errors, logs, output, cache, debug summaries,
|
Secrets must not appear in errors, logs, output, cache, debug summaries,
|
||||||
traces, manifests, documentation, examples, or redacted configuration. Debug
|
traces, manifests, documentation, examples, or redacted configuration. Raw
|
||||||
|
assistant responses and complete correction messages are attempt-local and are
|
||||||
|
excluded from ordinary durable records and summaries; the requested detailed
|
||||||
|
debug trace is the sole diagnostic surface allowed to retain them. Debug
|
||||||
collection is allowlisted to application-owned payloads and must not capture
|
collection is allowlisted to application-owned payloads and must not capture
|
||||||
unrelated process environment values or filesystem content. Trace data may
|
unrelated process environment values or filesystem content. Trace data may
|
||||||
contain application data and therefore inherits its sensitivity; operators own
|
contain application data and therefore inherits its sensitivity; operators own
|
||||||
|
|||||||
@@ -671,7 +671,7 @@ Every terminal path has accurate bounded provenance, no sensitive correction
|
|||||||
content leaks by default, and only completely validated output is reusable.
|
content leaks by default, and only completely validated output is reusable.
|
||||||
This stage is one Terra prompt.
|
This stage is one Terra prompt.
|
||||||
|
|
||||||
## Stage 15 — Update Canonical Documentation And Perform Final Verification
|
## Stage 15 ✅ — Update Canonical Documentation And Perform Final Verification
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user