eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document feedback-aware validation retries

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-27 01:35:23 +00:00
parent 7f01c3e79e
commit b2e83bd6e7
12 changed files with 229 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
docs/policy/architecture.md
View File

@@ -167,18 +167,22 @@ starting, waits for started work, and prevents output encoding.
## Validation
Validation is a framework-managed boundary around outputs from chunk, extract,
merge, and normalize stages. Validators receive immutable stage output
and make an explicit whole-output decision: approve, approve with warnings, or
reject.
merge, and normalize stages. Validators receive immutable stage output and
make an explicit whole-output decision: approve, approve with warnings,
reject, fail, or skip when a runtime prerequisite is unavailable.
Typed artifact validators receive the domain value directly. Chunk validators
receive source-zone chunks, while serialized validators receive immutable
representation bytes and declared schema metadata. A validator registered for
one target or artifact kind cannot satisfy an incompatible selection.
Rejection is a recorded pipeline outcome, not a framework execution error.
Validator execution failures are framework errors. Rejected output does not
advance to the next stage.
The framework runs every applicable validator sequentially in configured order.
It aggregates rejections, exhausted validator failures, and skips before the
producer policy chooses a disposition. A completed rejection never advances.
With no rejection, an exhausted validator failure may fail the run or, under
the configured `warn_continue` policy, advance a structurally valid candidate
with explicit incomplete-validation provenance. Validators report findings;
they do not choose candidate disposition.
Default validator chains are production composition policy and are registered
centrally by stage and module. Configuration may replace a stage-local default,
@@ -199,6 +203,15 @@ PromptKit owns bounded structural correction within one structured completion.
Notarius owns outer stage attempts, semantic validation, and acceptance policy;
the two budgets must remain separate.
An LLM-backed producer can participate in semantic correction only when it
declares `single_response_v1` and returns the exact one response that directly
controlled its candidate. On an actionable rejection, the framework rebuilds
the ordinary request and appends only the latest defective response as an
`assistant` message plus one aggregated `user` correction message. This is a
fresh replacement request, not a growing conversation. The retry budgets,
terminal policy, and sensitive-data rationale are recorded in
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md).
When a model selects an application entity, callers must supply a contextual
selection and deterministically attach the opaque application identity whenever
the selection resolves exactly. Models do not receive or reproduce opaque
@@ -233,7 +246,8 @@ invalid or incompatible.
Run manifests record enough resolved pipeline, module, source, reference, and
LLM provenance to make a run auditable after configuration changes. Manifests
record identities and summaries rather than secret or large payload content.
record identities and bounded validation summaries rather than secret, raw
model, correction, or large payload content.
## State, Output, And Safety
@@ -253,12 +267,20 @@ an invocation that explicitly requests resume. Debug is never a cache input and
is never created without an explicit request. Pipeline modules receive
collaborator interfaces and never physical roots.
Only accepted, completely validated producer output is reusable checkpoint or
chunk-plan state. Rejected, structurally invalid, and validation-incomplete
results cannot become cache or checkpoint inputs, even when a
`warn_continue` result is allowed to advance in the current run.
Writes are atomic where practical. Paths for writes, moves, overwrites, and
deletion must be narrow and explicit. Notarius never automatically deletes
output or requested debug bundles; cache cleanup is explicit and recoverable.
Secrets must not appear in errors, logs, output, cache, debug summaries,
traces, manifests, documentation, examples, or redacted configuration. Debug
traces, manifests, documentation, examples, or redacted configuration. Raw
assistant responses and complete correction messages are attempt-local and are
excluded from ordinary durable records and summaries; the requested detailed
debug trace is the sole diagnostic surface allowed to retain them. Debug
collection is allowlisted to application-owned payloads and must not capture
unrelated process environment values or filesystem content. Trace data may
contain application data and therefore inherits its sensitivity; operators own