Plan the PromptKit 0.8 upgrade
This commit is contained in:
@@ -15,34 +15,11 @@ is warranted.
|
||||
|
||||
### 1. Upgrade To PromptKit v0.8.0
|
||||
|
||||
- Upgrade the PromptKit dependency and follow the upstream v0.8.0 release
|
||||
guide, including its now-active `repair_attempts` behavior and compatibility
|
||||
checks for prompt definitions, prepared execution, validation results,
|
||||
capacity accounting, error adaptation, debug data, and cumulative token
|
||||
usage.
|
||||
- Use PromptKit's bounded repair loop for eligible structural output failures.
|
||||
A repair must resend the immutable original prompt followed by the latest
|
||||
defective assistant response and one bounded user correction message. It
|
||||
must not accumulate the history of every defective candidate.
|
||||
- Review every maintained LLM prompt's structural repair budget. Prefer a
|
||||
small positive budget where a corrected structured response can be useful,
|
||||
while retaining zero where a second call would be inappropriate or where
|
||||
observational evidence does not justify it.
|
||||
- Keep PromptKit repair attempts distinct from Notarius stage retries.
|
||||
PromptKit repairs a response that fails its deterministic `basic`, `json`, or
|
||||
`json_schema` contract within one stage attempt; Notarius may later retry a
|
||||
complete stage candidate after application validation. Document and test the
|
||||
multiplicative maximum provider-call cost created by both configured
|
||||
budgets, including PromptKit repair calls made by any LLM-backed validators.
|
||||
Attribute generation count, token usage, latency, and failure provenance to
|
||||
the producer, validator, PromptKit repair, and Notarius correction layers so
|
||||
operators can explain the actual cost of a run.
|
||||
- Treat exhaustion of PromptKit structural repair as the absence of a usable
|
||||
structured candidate. The default terminal policy is to fail the run after
|
||||
the applicable Notarius stage-attempt budget is also exhausted. A future
|
||||
configurable alternative may reject the affected stage or lane where the
|
||||
pipeline can represent that outcome, but it must never accept undecodable or
|
||||
structurally invalid output merely with a warning.
|
||||
This item has been promoted to the standalone
|
||||
[PromptKit v0.8.0 Upgrade](promptkit-v0.8.md) roadmap. That document owns the
|
||||
release-by-release compatibility review, adopted features, structured-repair
|
||||
policy, target integration boundary, acceptance criteria, and settled design
|
||||
decisions.
|
||||
|
||||
### 2. Feedback-Aware Stage Validation Retries
|
||||
|
||||
@@ -245,8 +222,7 @@ The implemented source-backed core and initial D&D registry adoption are
|
||||
described by [Module Internals](../internal/modules.md#semantic-reconciliation)
|
||||
and
|
||||
[D&D Module Internals](../internal/dnd.md#semantic-registry-reconciliation).
|
||||
The [Semantic Reconciliation Roadmap](semantic-reconciliation.md) retains the
|
||||
original feature scope; the sections below keep broader extensions deferred.
|
||||
The sections below keep broader extensions deferred.
|
||||
|
||||
### Large-Collection Semantic Reconciliation
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
520
docs/roadmap/promptkit-v0.8.md
Normal file
520
docs/roadmap/promptkit-v0.8.md
Normal file
@@ -0,0 +1,520 @@
|
||||
# PromptKit v0.8.0 Upgrade
|
||||
|
||||
## Status
|
||||
|
||||
Proposed.
|
||||
|
||||
## Purpose
|
||||
|
||||
Upgrade Notarius from PromptKit v0.5.0 to v0.8.0 and deliberately adopt the
|
||||
useful correctness, profile-composition, provider-diagnostic, backend, and
|
||||
structured-output-repair capabilities introduced in PromptKit v0.6.0, v0.7.0,
|
||||
and v0.8.0.
|
||||
|
||||
The upgrade should improve structured-output reliability without confusing
|
||||
PromptKit's bounded deterministic repair with Notarius's existing stage retry
|
||||
budget or the future feedback-aware semantic-validation loop. PromptKit types
|
||||
and provider behavior must remain behind Notarius's transport-neutral LLM
|
||||
boundary.
|
||||
|
||||
## Current State
|
||||
|
||||
Notarius currently pins PromptKit v0.5.0. Its production adapter prepares one
|
||||
frozen execution, records credential-redacted details, and runs that same
|
||||
prepared value. It maps PromptKit capacity failures to an application-owned
|
||||
error, maps failed structured validation to `ErrInvalidStructuredOutput`, and
|
||||
returns PromptKit's raw validated bytes and usage metadata.
|
||||
|
||||
Every maintained production prompt uses JSON Schema validation and currently
|
||||
declares `repair_attempts: 0`. Notarius stage bindings separately expose
|
||||
`retries`, which reruns a complete stage operation after an error or rejected
|
||||
candidate. The two mechanisms have different ownership and must remain
|
||||
independent.
|
||||
|
||||
Notarius also maintains:
|
||||
|
||||
- embedded prompt, schema, and fallback-profile filesystems;
|
||||
- operator profile-file and profile-directory sources;
|
||||
- one optional conventional `local` backend registration;
|
||||
- explicit profile preflight through PromptKit inspection;
|
||||
- one application-wide scheduled LLM client around the PromptKit adapter;
|
||||
- PromptKit profile-source fingerprints for checkpoint safety; and
|
||||
- redacted debug and manifest provenance at application-owned boundaries.
|
||||
|
||||
The upgrade must preserve those established responsibilities while revising
|
||||
the pinned integration contract and any behavior affected by the three
|
||||
intervening releases.
|
||||
|
||||
This roadmap is based on PromptKit's pinned release guides for
|
||||
[v0.6.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.6.0.md),
|
||||
[v0.7.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.7.0.md),
|
||||
and
|
||||
[v0.8.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.8.0/docs/releases/v0.8.0.md),
|
||||
plus the public API and format documentation at the v0.8.0 tag.
|
||||
|
||||
## Target End State
|
||||
|
||||
- `go.mod` and `go.sum` pin PromptKit v0.8.0 without a local replacement or
|
||||
vendored copy.
|
||||
- Every maintained PromptKit prompt and profile prepares successfully under
|
||||
v0.8.0's stricter validation and source-loading rules.
|
||||
- Eligible Notarius structured completions use one PromptKit corrective call by
|
||||
default after a structurally invalid response. Operators can explicitly set
|
||||
a value from zero through three for a configured pipeline, with a more local
|
||||
LLM-backed binding override where needed.
|
||||
- PromptKit repair remains an inner operation within one Notarius stage
|
||||
attempt. It never consumes or replenishes the binding's `retries` budget.
|
||||
- A successful repaired result exposes cumulative usage and the actual repair
|
||||
count to Notarius's application-owned response and debug models. A repaired
|
||||
success is not itself a warning.
|
||||
- Exhausted PromptKit validation remains an invalid structured-output result,
|
||||
preserving the final candidate and diagnostics for debug and for any
|
||||
applicable outer Notarius stage policy. Invalid structured output is never
|
||||
accepted merely because the repair budget was exhausted.
|
||||
- Profile inheritance, the built-in Rakestrawhome backend/profile, optional
|
||||
credential behavior, and structured generation errors work through the
|
||||
existing Notarius PromptKit boundary and are accurately documented.
|
||||
- Provider-specific PromptKit types do not escape `internal/framework/llm`.
|
||||
- Checkpoint identity, effective configuration, redacted summaries, and debug
|
||||
provenance reflect every execution-affecting repair or profile change.
|
||||
- Current documentation pins and describes v0.8.0; future Notarius semantic
|
||||
validation retries remain roadmap behavior rather than being conflated with
|
||||
this dependency upgrade.
|
||||
|
||||
## Release-by-Release Adoption
|
||||
|
||||
### PromptKit v0.6.0: Correctness, Safety, And Efficiency
|
||||
|
||||
PromptKit v0.6.0 adds no public declarations, but intentionally rejects several
|
||||
formerly permissive or ambiguous inputs. The upgrade must audit Notarius's
|
||||
embedded and operator-facing integration against these rules:
|
||||
|
||||
- YAML `id` and `version` metadata, rather than filenames, define prompt and
|
||||
profile identity.
|
||||
- Prompt `content_file` paths are exact, relative, contained paths; built-in
|
||||
file artifacts must resolve to regular files.
|
||||
- execution controls, output contracts, and repair budgets must be finite and
|
||||
within their documented ranges;
|
||||
- provider endpoints must be absolute HTTP or HTTPS URLs with a host and no
|
||||
user information, query, or fragment;
|
||||
- JSON documents and successful provider responses contain exactly one value;
|
||||
- successful provider responses are bounded to 16 MiB; and
|
||||
- JSON-compatible values are bounded for depth and expansion.
|
||||
|
||||
Notarius should rely on PromptKit for these rules rather than duplicate its
|
||||
parsers or internal limits. Existing Notarius validation may retain a narrower
|
||||
application rule where it has independent value, but overlapping validation
|
||||
must agree with PromptKit and must not accept a value PromptKit will reject
|
||||
later.
|
||||
|
||||
The upgrade automatically receives operation-local schema-plan reuse,
|
||||
artifact-text memoization, improved cancellation checks, and transport error
|
||||
identity preservation. Notarius should verify these changes through its real
|
||||
adapter boundary and avoid adding a second cache or response-body layer that
|
||||
would duplicate PromptKit's ownership.
|
||||
|
||||
### PromptKit v0.7.0: Profiles, Backend Access, And Generation Errors
|
||||
|
||||
#### Profile Inheritance
|
||||
|
||||
Operator profiles may use `base_profile` to alias or selectively refine a
|
||||
built-in, fallback, or higher-precedence operator profile. Notarius must pass
|
||||
profile sources through unchanged and let PromptKit own parent lookup, merge
|
||||
rules, source precedence, cycle detection, and fully resolved prepared targets.
|
||||
|
||||
Preflight inspection must resolve inherited profiles through the same source
|
||||
and backend composition used at execution. The selected leaf profile ID remains
|
||||
the public profile identity, while effective backend, endpoint, model, and
|
||||
reasoning provenance reflect the resolved chain. Notarius must not implement a
|
||||
second inheritance parser.
|
||||
|
||||
The existing complete `dnd-extraction` fallback remains a standalone profile:
|
||||
PromptKit v0.8.0 does not provide a built-in `openai/gpt-5.6-luna` profile that
|
||||
would be an appropriate parent. Documentation should nevertheless explain how
|
||||
operators can use inheritance for environment-specific workload profiles and
|
||||
should link to PromptKit's pinned format contract rather than duplicate its
|
||||
field-by-field merge algorithm.
|
||||
|
||||
Checkpoint safety must cover inherited behavior. Operator file/directory
|
||||
digests already cover changes to definitions in those sources, fallback asset
|
||||
digests cover application parents, and the PromptKit built-in catalog marker
|
||||
must change from its v0.5.0 identity to v0.8.0 so a changed built-in parent
|
||||
cannot reuse an incompatible checkpoint.
|
||||
|
||||
#### Rakestrawhome Backend And Profile
|
||||
|
||||
PromptKit's reserved `rakestrawhome` backend and
|
||||
`rakestrawhome-gemma-4-31b` profile become available without Notarius-specific
|
||||
registration. Notarius must not register or shadow the reserved backend ID.
|
||||
Profile preflight, backend-capacity reporting, scheduling, generation, and
|
||||
provenance should work for it through the same generic paths used by OpenRouter
|
||||
and `local`.
|
||||
|
||||
The D&D default remains `dnd-extraction`; this upgrade does not silently move a
|
||||
production workload to Rakestrawhome. Operator documentation should identify
|
||||
the built-in profile as an available selection and link to PromptKit for its
|
||||
endpoint, credential environment, model, and capacity defaults.
|
||||
|
||||
#### Optional Credentials
|
||||
|
||||
An absent or blank optional `APIKeyEnv` now causes PromptKit to omit the
|
||||
`Authorization` header and send the request. Notarius must not restore the old
|
||||
failure behavior by pre-reading provider credential environment variables or
|
||||
by adding provider-specific authentication logic.
|
||||
|
||||
Profile inspection may report an explicit `APIKeyRequired` policy without
|
||||
reading the credential, and execution remains the boundary at which that
|
||||
requirement is enforced. For optional profiles, an authentication-requiring
|
||||
provider may instead return a structured 401 or 403 generation failure. The
|
||||
configuration and operations documentation must explain this distinction.
|
||||
Notarius does not currently expose PromptKit's in-memory profile-registration
|
||||
API to operators, and PromptKit's filesystem profile format does not expose
|
||||
`APIKeyRequired`; therefore Notarius must not promise that an operator profile
|
||||
can force local credential preflight. Operators should provision the named
|
||||
environment variable, while Notarius should preserve the provider's structured
|
||||
authentication failure when it is absent.
|
||||
|
||||
Notarius must continue to document mechanisms and environment-variable names,
|
||||
never secret values.
|
||||
|
||||
#### Structured Generation Errors
|
||||
|
||||
The adapter should recognize `*promptkit.GenerationError` with `errors.As` and
|
||||
translate useful information into an immutable, provider-neutral Notarius
|
||||
error classification. At minimum, retain the HTTP status code so callers and
|
||||
future retry policy can distinguish transport success with provider rejection
|
||||
from other generation failures.
|
||||
|
||||
PromptKit's provider code, type, and message accessors are bounded but remain
|
||||
untrusted and potentially sensitive. They must never appear automatically in
|
||||
ordinary CLI output, warnings, manifests, checkpoint identity, or cache data.
|
||||
If retained for an explicitly requested debug trace, they must pass through
|
||||
Notarius's known-secret and bearer redaction and remain clearly identified as
|
||||
untrusted provider diagnostics. Default error formatting should continue to
|
||||
use a bounded, redacted application-owned message.
|
||||
|
||||
Capacity and cancellation retain their current more specific classifications
|
||||
and precedence. This upgrade does not add automatic provider-error retry
|
||||
classification; it only preserves safe structured data needed for diagnosis
|
||||
and later policy.
|
||||
|
||||
### PromptKit v0.8.0: Bounded Structured-Output Repair
|
||||
|
||||
#### Default Policy
|
||||
|
||||
Every maintained production prompt whose output is consumed as structured data
|
||||
should declare one repair attempt. All current production prompts use eligible
|
||||
JSON Schema validation, so no current prompt needs a zero default merely
|
||||
because of its output mode.
|
||||
|
||||
One repair means at most one corrective generation after the initial
|
||||
candidate. PromptKit reconstructs the immutable original conversation and
|
||||
appends only the latest invalid assistant candidate and latest deterministic
|
||||
validation diagnostics. It preserves the selected target, direct session ID,
|
||||
provider-native structured-output contract, and backend capacity policy. This
|
||||
shape preserves the original cacheable prompt prefix and avoids accumulating
|
||||
unbounded failed history.
|
||||
|
||||
The default is deliberately small. A single repair captures the common case in
|
||||
which a capable model can correct malformed JSON or a schema violation after
|
||||
receiving an exact diagnostic, while bounding the extra latency and cost of a
|
||||
single structured completion.
|
||||
|
||||
#### Configuration Contract
|
||||
|
||||
The public configuration is an optional, presence-aware
|
||||
`structured_output_repair_attempts` integer at pipeline scope and at each
|
||||
LLM-backed module or validator binding. Its effective precedence is:
|
||||
|
||||
1. the binding value, when present;
|
||||
2. the pipeline value, when present; and
|
||||
3. the selected prompt's declared `repair_attempts` value.
|
||||
|
||||
The value must be from zero through three. Explicit zero disables PromptKit
|
||||
repair at that scope. A deterministic binding must reject the field because it
|
||||
cannot perform structured LLM repair. Validator bindings may use it only when
|
||||
the selected validator is LLM-backed. Shorthand module bindings continue to
|
||||
inherit the pipeline or prompt default.
|
||||
|
||||
The long, provider-neutral name is intentional: it distinguishes PromptKit's
|
||||
inner structural repair from the existing binding `retries` field, which owns
|
||||
complete stage attempts, without exposing a dependency name in generic
|
||||
pipeline contracts.
|
||||
|
||||
The effective value must survive file parsing, cloning, redacted summaries,
|
||||
pipeline resolution, and pipeline digest construction without pointer aliasing
|
||||
or loss of presence. It must affect checkpoint identity because it can change
|
||||
the selected result, latency, token usage, and provider cost.
|
||||
|
||||
#### Adapter Contract
|
||||
|
||||
The transport-neutral structured-completion request should carry an optional
|
||||
application-owned structural-repair budget. No `promptkit.OutputContract` or
|
||||
other PromptKit type may cross the adapter boundary.
|
||||
|
||||
PromptKit v0.8.0 request validation replaces the complete prompt output
|
||||
contract rather than merging one field. When Notarius has a configured
|
||||
override, the adapter must therefore inspect the selected prompt, copy its
|
||||
normalized declared format, validation mode, and schema path, change only the
|
||||
repair count, and supply that complete contract on the prepared request. A nil
|
||||
override continues to use the prompt declaration directly. Inspection and
|
||||
preparation must use the same immutable engine sources; a small adapter-local
|
||||
cache keyed by normalized prompt ID and version is acceptable but not required
|
||||
without measured need.
|
||||
|
||||
This approach prevents configuration from accidentally dropping JSON Schema
|
||||
validation, avoids duplicating schema paths in pipeline YAML, and keeps prompt
|
||||
assets authoritative for every output-contract field other than the explicit
|
||||
operator override.
|
||||
|
||||
The transport-neutral structured-completion response should report the actual
|
||||
number of PromptKit repair calls. PromptKit's returned token usage is already
|
||||
cumulative and must be passed through without re-summing it. Debug records
|
||||
should distinguish the configured budget from the actual count. Ordinary run
|
||||
manifests need not gain raw prompt or response data merely to report repairs;
|
||||
any durable aggregate should be added only if it has a clear consumer contract.
|
||||
|
||||
#### Result And Failure Semantics
|
||||
|
||||
- A valid initial candidate returns normally with zero actual repairs.
|
||||
- A valid corrected candidate returns normally with cumulative usage and its
|
||||
positive actual repair count. It does not emit a warning solely because a
|
||||
repair occurred.
|
||||
- Exhausting the repair budget returns PromptKit's final candidate and failed
|
||||
validation result. The adapter maps this to
|
||||
`ErrInvalidStructuredOutput`, preserves the response and debug material, and
|
||||
does not decode or accept the candidate.
|
||||
- An explicitly empty or whitespace-only candidate participates in the
|
||||
declared structural validation and repair flow. Missing, `null`, or
|
||||
non-string provider content remains a malformed provider response.
|
||||
- A generation failure during a corrective call is an operational generation
|
||||
failure and uses the same safe structured-error adaptation as an initial
|
||||
generation failure.
|
||||
- Context cancellation remains authoritative throughout the initial and
|
||||
corrective calls.
|
||||
|
||||
PromptKit repair happens inside one scheduled `CompleteStructured` operation.
|
||||
The Notarius scheduler holds one permit for that logical operation while
|
||||
PromptKit performs its initial and serial corrective calls; PromptKit
|
||||
reacquires its own selected-backend capacity for each corrective generation.
|
||||
Because corrective calls are serial, this cannot expand actual concurrent
|
||||
provider work beyond the number of admitted Notarius operations, but
|
||||
documentation must stop describing the Notarius permit as a separate admission
|
||||
event for every internal repair call.
|
||||
|
||||
One `CompleteStructured` invocation with effective PromptKit repair budget `R`
|
||||
may make at most `R + 1` provider calls. If one stage attempt makes `C`
|
||||
structured-completion invocations, a binding with `retries: N` has an upper
|
||||
bound of `(N + 1) * C * (R + 1)` provider calls; `C` may itself be a bounded,
|
||||
data-dependent module property, as it is for batched semantic reconciliation.
|
||||
LLM-backed validators have their own corresponding invocation counts, budgets,
|
||||
and costs. These formulas are upper bounds, not promises that every failure is
|
||||
retryable or that every attempt reaches the provider.
|
||||
|
||||
## Profile And Prompt Source Compatibility
|
||||
|
||||
The upgrade must preserve Notarius's source precedence: an operator source,
|
||||
then registered application fallback profiles, then PromptKit built-ins. A
|
||||
selected malformed definition remains authoritative and fails rather than
|
||||
falling through. Parent resolution introduced by profile inheritance observes
|
||||
that same precedence.
|
||||
|
||||
All embedded prompt manifests, shared content fragments, response schemas, and
|
||||
fallback profiles must be prepared or inspected offline under v0.8.0. The
|
||||
review should specifically catch:
|
||||
|
||||
- IDs inferred accidentally from filenames;
|
||||
- stale or escaping `content_file` paths;
|
||||
- missing or non-regular embedded artifacts;
|
||||
- repair values outside zero through three or paired with ineligible
|
||||
validation;
|
||||
- schemas or examples that are not exact single JSON documents;
|
||||
- unsupported endpoint forms; and
|
||||
- JSON-compatible variables or profile extras that exceed upstream bounds.
|
||||
|
||||
No prompt prose, schema shape, durable D&D artifact contract, or default D&D
|
||||
model should change merely to exercise the dependency. Prompt manifests should
|
||||
change only as needed to enable the adopted repair default and satisfy v0.8.0
|
||||
contracts.
|
||||
|
||||
## Provenance, Debugging, And Security
|
||||
|
||||
- Update the opaque PromptKit built-in profile-catalog identity from v0.5.0 to
|
||||
v0.8.0. Do not hash or publish PromptKit's internal catalog bytes.
|
||||
- Ensure a prompt's repair default remains covered by its existing prompt asset
|
||||
fingerprint and a configured effective override remains covered by the
|
||||
resolved pipeline digest.
|
||||
- Preserve selected leaf profile identity while recording the inherited
|
||||
effective target already exposed by PromptKit inspection and prepared
|
||||
details.
|
||||
- Add actual structural-repair count and, when useful, the configured budget to
|
||||
application-owned debug material. Token totals remain PromptKit's cumulative
|
||||
values.
|
||||
- Do not generate a warning for a successful repair. Repair exhaustion is an
|
||||
invalid-output failure, while provider rejection is a generation failure.
|
||||
- Never expose raw provider diagnostic fields without explicit debug capture
|
||||
and application redaction. Do not place them in normal errors or durable
|
||||
summaries.
|
||||
- Preserve context and transport error identity sufficiently for
|
||||
`errors.Is`-based cancellation and deadline handling after adapting the
|
||||
external error.
|
||||
|
||||
## Documentation And Examples
|
||||
|
||||
Implementation should update current-state documentation only when the new
|
||||
behavior lands:
|
||||
|
||||
- `docs/integrations/pkg-promptkit.md` must pin v0.8.0 and define the revised
|
||||
prepared-execution, repair, profile-inheritance, backend, credential, and
|
||||
error-adaptation boundary.
|
||||
- `docs/config.md` must own the repair configuration fields, precedence,
|
||||
allowed range, explicit-zero behavior, profile inheritance availability, and
|
||||
optional credential semantics.
|
||||
- `docs/operations.md` must explain structural repair cost, timeout and
|
||||
concurrency effects, credential failures, and its distinction from stage
|
||||
retries.
|
||||
- `docs/internal/llm.md` must describe adapter contract replacement, actual
|
||||
repair metadata, error adaptation, source compatibility, and scheduling.
|
||||
- `docs/internal/pipeline.md` must describe how effective repair configuration
|
||||
is resolved and how inner repair differs from outer stage attempts.
|
||||
- `docs/policy/architecture.md` should receive only the durable ownership rule:
|
||||
PromptKit owns bounded deterministic structural repair within one completion,
|
||||
while Notarius owns stage attempts and semantic validation policy. Detailed
|
||||
fields and retry formulas belong in their canonical configuration and
|
||||
operations documents.
|
||||
|
||||
Update maintained configuration examples only if the public Notarius
|
||||
configuration contract changes. A short inheritance illustration may remain in
|
||||
the configuration reference; do not create a complete example solely to copy
|
||||
PromptKit's upstream profile catalog. All upstream links must point to the
|
||||
v0.8.0 tag. Historical release or archived roadmap references should remain
|
||||
historical.
|
||||
|
||||
No ADR is required solely to pin a newer dependency. The durable separation
|
||||
between PromptKit structural repair and Notarius semantic stage retries should
|
||||
be stated in architecture documentation now; the more extensive future
|
||||
validation state machine still warrants the separate ADR already identified in
|
||||
`future.md` when that work is promoted.
|
||||
|
||||
## Validation And Acceptance Criteria
|
||||
|
||||
The implementation is complete when:
|
||||
|
||||
- the repository builds and tests against PromptKit v0.8.0 with no replacement
|
||||
directive, workspace dependency, or vendored source;
|
||||
- every maintained prompt and profile prepares or inspects successfully under
|
||||
the v0.8.0 source, path, endpoint, output-contract, and JSON-value rules;
|
||||
- an invalid first JSON Schema candidate followed by a valid correction returns
|
||||
the valid raw output, cumulative usage, and actual repair count through the
|
||||
Notarius adapter;
|
||||
- repair exhaustion returns the final raw candidate and debug material with an
|
||||
error matching `ErrInvalidStructuredOutput`;
|
||||
- a corrective generation failure retains safe generation classification and
|
||||
provider status without leaking untrusted provider detail;
|
||||
- explicit empty content follows structural validation rather than being
|
||||
misclassified by Notarius;
|
||||
- repair configuration is presence-aware, range checked, rejected on
|
||||
deterministic bindings, resolved with documented precedence, and included in
|
||||
effective pipeline identity;
|
||||
- inherited profiles resolve consistently during preflight and execution, and
|
||||
changes to any relevant operator, fallback, or built-in parent invalidate
|
||||
checkpoint reuse;
|
||||
- the Rakestrawhome built-in profile reaches generic preflight, scheduling, and
|
||||
provenance paths without application-specific registration;
|
||||
- optional missing credentials and explicitly required credentials behave as
|
||||
documented without contacting real providers in tests;
|
||||
- cancellation, timeout, backend capacity, prepared-execution snapshot,
|
||||
session ID, raw-output, debug-redaction, and existing profile provenance
|
||||
behavior remain intact;
|
||||
- maintained examples validate successfully; and
|
||||
- canonical documentation contains no active v0.5.0 pin or claim that PromptKit
|
||||
is always single-pass.
|
||||
|
||||
Tests should follow `docs/policy/testing.md`: exercise observable Notarius
|
||||
contracts with offline fake clients or `httptest` boundaries, and do not copy
|
||||
PromptKit's entire internal repair test suite or assert its exact correction
|
||||
message prose. The dependency's internal wording is not a Notarius contract.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Implementing Notarius's future feedback-aware semantic stage-retry loop.
|
||||
- Adding the D&D combat-scene semantic validator.
|
||||
- Redesigning warning policy or treating successful structural repair as a
|
||||
warning.
|
||||
- Adding provider transport retries or deciding which HTTP statuses should
|
||||
consume a stage retry.
|
||||
- Exposing PromptKit request, response, profile, validation, capacity, or error
|
||||
types outside the LLM adapter.
|
||||
- Changing durable artifact schemas, D&D prompt semantics, the D&D default
|
||||
model, or the fixed pipeline shape.
|
||||
- Reimplementing PromptKit profile inheritance, schema validation, response
|
||||
bounds, repair conversations, backend admission, or provider parsing inside
|
||||
Notarius.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Default Structured-Output Repair Budget
|
||||
|
||||
**Decision: default to one repair attempt.** Set every maintained
|
||||
eligible production prompt to `repair_attempts: 1`. One corrective call is a
|
||||
strong fit for Notarius because every current production LLM response has a
|
||||
strict JSON Schema contract, smaller cost-effective models are a deliberate
|
||||
deployment target, and a precise structural diagnostic often makes one retry
|
||||
materially more successful. The budget is paid only after a structurally
|
||||
invalid candidate and remains tightly bounded.
|
||||
|
||||
**Alternative considered: retain zero by default.** This preserves single-pass
|
||||
cost and latency and requires operators to opt in. It is preferable for an
|
||||
environment where every additional request is expensive or where upstream
|
||||
provider-native schema enforcement already produces negligible invalid output.
|
||||
It is less suitable as the Notarius default because one malformed response can
|
||||
otherwise discard substantial completed pipeline work.
|
||||
|
||||
**Alternative considered: default to two.** This may improve recovery for
|
||||
weak models, but it doubles the worst-case corrective cost relative to the
|
||||
selected default and compounds with outer stage retries. It should be an
|
||||
operator choice supported by configuration, not the initial default, unless
|
||||
observational evidence shows that the second correction has a worthwhile
|
||||
marginal success rate.
|
||||
|
||||
### 2. Repair Override Scope
|
||||
|
||||
**Decision: support both pipeline and LLM-backed binding overrides.** Use
|
||||
the presence-aware `structured_output_repair_attempts` field and precedence
|
||||
defined above. A pipeline value provides the convenient one-line control the
|
||||
operator requested, while a binding value permits an expensive normalizer or
|
||||
future LLM-backed validator to use a deliberately different budget. This
|
||||
mirrors Notarius's established pipeline/binding profile inheritance and scales
|
||||
without editing embedded prompts.
|
||||
|
||||
**Alternative considered: support only a pipeline override.** This is smaller to
|
||||
implement and document and still permits global enablement or disablement for
|
||||
one pipeline. Its drawback is that one exceptional prompt cannot opt out or
|
||||
request a larger budget without changing an embedded asset for every pipeline.
|
||||
|
||||
**Alternative considered: expose one global value under the top-level
|
||||
`promptkit` configuration.** This makes client construction simple, but applies
|
||||
the same budget to unrelated pipelines and leaks an execution policy into the
|
||||
dependency configuration block. It is less compositional than pipeline-owned
|
||||
policy and therefore not recommended.
|
||||
|
||||
### 3. Retention Of Provider-Supplied Generation Details
|
||||
|
||||
**Decision: retain status in the application-owned error contract and
|
||||
retain redacted provider code, type, and message only in explicitly requested
|
||||
debug traces.** Status is useful for diagnosis and future retry policy without
|
||||
usually containing sensitive data. The other fields can materially explain a
|
||||
400 response but may echo request or schema content, so they belong only in the
|
||||
already-sensitive debug surface after Notarius redaction.
|
||||
|
||||
**Alternative considered: retain only HTTP status and discard all provider fields.**
|
||||
This is the safest and smallest policy and still improves typed failure
|
||||
handling. It sacrifices potentially decisive provider diagnostics, leaving an
|
||||
operator with less information when a provider returns a terse status and the
|
||||
problem cannot be reproduced easily.
|
||||
|
||||
**Alternative considered: include bounded provider code and type in normal
|
||||
errors while keeping message debug-only.** Codes and types are often stable and
|
||||
less sensitive than messages, but PromptKit explicitly classifies every
|
||||
provider field as untrusted. Promoting them to ordinary output creates a
|
||||
disclosure and compatibility burden that is not currently justified.
|
||||
Reference in New Issue
Block a user