783 lines
38 KiB
Markdown
783 lines
38 KiB
Markdown
# PromptKit v0.8.0 Upgrade Implementation Plan
|
|
|
|
## Purpose
|
|
|
|
Implement the target state defined by the
|
|
[PromptKit v0.8.0 Upgrade](promptkit-v0.8.md): adopt the useful PromptKit
|
|
v0.6.0, v0.7.0, and v0.8.0 changes; enable one bounded structural correction
|
|
by default; expose pipeline and binding overrides; preserve safe provider
|
|
diagnostics; and keep PromptKit behind Notarius's transport-neutral LLM
|
|
boundary.
|
|
|
|
This plan is ordered. Each numbered stage is one implementation prompt for a
|
|
gpt-5.6-terra coding agent. Complete and validate one stage before beginning
|
|
the next. Read `docs/development.md` and every policy under `docs/policy/` at
|
|
the start of each stage, inspect the current code and tests named by that
|
|
stage, preserve unrelated worktree changes, and update current-behavior
|
|
documentation in the same stage as the behavior it describes.
|
|
|
|
Do not retire this plan or `promptkit-v0.8.md` during implementation. Keep both
|
|
until the completed work has passed a separate review. Do not implement the
|
|
future Notarius semantic-validation retry loop, D&D combat-scene validator, or
|
|
warning redesign as part of this plan.
|
|
|
|
## Decisions Fixed For Implementation
|
|
|
|
- Pin `gitea.maximumdirect.net/eric/promptkit` v0.8.0 directly, with no
|
|
`replace`, workspace dependency, or vendored source.
|
|
- Every maintained eligible production prompt defaults to exactly one
|
|
PromptKit structural repair attempt.
|
|
- Add the exact configuration key
|
|
`structured_output_repair_attempts` at pipeline scope and on LLM-backed
|
|
module and validator bindings.
|
|
- Effective precedence is binding value, then pipeline value, then the prompt's
|
|
declared `repair_attempts` value. Omission inherits; explicit zero disables
|
|
structural repair at that scope.
|
|
- Accepted values are integers from zero through three. Explicit null and
|
|
non-integer values are invalid. An explicit binding value on a deterministic
|
|
module or validator is invalid. A pipeline value is applied only to selected
|
|
LLM-backed bindings and does not make deterministic bindings invalid.
|
|
- Keep file configuration version 4. This is an additive pre-release field and
|
|
does not require parallel versioned behavior.
|
|
- Use `StructuredOutputRepairAttempts *int` for presence-aware internal Go
|
|
fields. Clone pointers at every ownership boundary.
|
|
- A configured override never replaces schema identity, output format, or
|
|
validation mode. The PromptKit adapter calls `InspectPrompt`, copies the
|
|
complete normalized prompt-owned output contract, changes only
|
|
`RepairAttempts`, and supplies the complete replacement on `RunRequest`.
|
|
Do not add an inspection cache initially.
|
|
- PromptKit repair is internal to one `CompleteStructured` call and does not
|
|
consume or replenish a binding's existing `retries` budget.
|
|
- Add `RepairAttempts int` to Notarius's structured-completion response. It is
|
|
the actual corrective-call count reported by PromptKit; token usage remains
|
|
PromptKit's cumulative usage and must not be summed again.
|
|
- A valid repaired response is successful and produces no warning solely
|
|
because repair occurred. Exhausted structural validation maps to
|
|
`ErrInvalidStructuredOutput` with the final candidate and debug material
|
|
retained.
|
|
- Add an application-owned generation-error sentinel and typed status-bearing
|
|
error. PromptKit error types must not cross `internal/framework/llm`.
|
|
- HTTP status may appear in the application-owned generation error. Provider
|
|
code, type, and message are excluded from ordinary errors, warnings,
|
|
manifests, cache, and checkpoint identity; they may appear only in an
|
|
explicitly requested debug trace after Notarius redaction.
|
|
- Profile inheritance is owned entirely by PromptKit. Notarius passes sources
|
|
through, inspects and records the resolved target, and does not parse or merge
|
|
`base_profile` itself.
|
|
- PromptKit's built-in `rakestrawhome` backend and
|
|
`rakestrawhome-gemma-4-31b` profile are available generically. Notarius does
|
|
not register, shadow, or select them by default.
|
|
- Missing optional credential environment values are allowed to reach the
|
|
provider without `Authorization`; Notarius does not recreate v0.5.0's local
|
|
failure or add provider-specific authentication logic.
|
|
- No dependency-upgrade ADR is required. Update architecture only with the
|
|
durable ownership distinction between PromptKit structural repair and
|
|
Notarius stage/semantic validation policy.
|
|
|
|
## Stage 1: Upgrade The Dependency And Establish A Clean v0.8.0 Baseline ✅
|
|
|
|
### Goal
|
|
|
|
Move the repository to PromptKit v0.8.0, resolve source-compatibility issues,
|
|
and establish a passing baseline before adopting new behavior.
|
|
|
|
### Implementation
|
|
|
|
1. Re-read the upstream v0.6.0, v0.7.0, and v0.8.0 release guides and the
|
|
v0.8.0 package consumer and format documentation. Treat the pinned v0.8.0
|
|
tag, not the sibling checkout's moving branch, as authoritative.
|
|
2. Update `go.mod` and `go.sum` to PromptKit v0.8.0 and run `go mod tidy` with
|
|
`GOWORK=off`.
|
|
3. Compile before making compatibility edits. Correct only actual source or
|
|
behavior incompatibilities. In particular:
|
|
- convert any positional `promptkit.Profile` or
|
|
`promptkit.OpenAICompatibleProfileConfig` literals to keyed literals;
|
|
- confirm Notarius does not register the newly reserved `rakestrawhome`
|
|
backend ID; and
|
|
- preserve `PrepareExecution`/`Details`/`RunPrepared` snapshot ownership,
|
|
`Discard`, session forwarding, reasoning override, profile preflight,
|
|
and capacity adaptation.
|
|
4. Change `promptKitBuiltinProfileCatalogID` in
|
|
`internal/framework/llm/promptkit_profile_fingerprint.go` from the v0.5.0
|
|
catalog marker to an opaque v0.8.0 marker. Do not hash PromptKit internal
|
|
files or include catalog content in manifests.
|
|
5. Update `docs/integrations/pkg-promptkit.md` to pin and link v0.8.0 and to
|
|
state that this stage still leaves the production prompt-declared repair
|
|
budget at its current value. Do not document later configuration or default
|
|
behavior before it exists.
|
|
6. Update only those existing tests whose public PromptKit types or stable
|
|
v0.8.0 behavior genuinely changed. Do not rewrite tests merely to match
|
|
upstream diagnostic wording.
|
|
|
|
### Tests And Validation
|
|
|
|
```sh
|
|
GOWORK=off go mod tidy -diff
|
|
GOWORK=off go test ./internal/framework/llm ./internal/cli
|
|
GOWORK=off go test ./...
|
|
GOWORK=off go vet ./...
|
|
GOWORK=off go build ./cmd/notarius
|
|
git diff --check
|
|
```
|
|
|
|
### Acceptance Criteria
|
|
|
|
- `go list -m gitea.maximumdirect.net/eric/promptkit` reports v0.8.0.
|
|
- There is no PromptKit `replace`, active Go workspace dependency, or vendor
|
|
tree.
|
|
- The adapter still uses one frozen prepared execution and all existing LLM
|
|
tests pass.
|
|
- Checkpoint profile identity includes the v0.8.0 built-in catalog marker.
|
|
- Current integration documentation pins v0.8.0 without claiming that
|
|
later stages are already active.
|
|
- The full ordinary test suite, vet, and command build pass.
|
|
|
|
## Stage 2: Verify v0.6.0 Compatibility And Hardening ✅
|
|
|
|
### Goal
|
|
|
|
Audit Notarius's assets and boundary values against PromptKit v0.6.0's stricter
|
|
source, path, endpoint, JSON, and cancellation contracts, fixing only concrete
|
|
incompatibilities.
|
|
|
|
### Implementation
|
|
|
|
1. Inspect `internal/framework/llm/asset_registry.go`, prompt/profile source
|
|
composition, all registered asset roots, the conventional local backend,
|
|
and their focused tests.
|
|
2. Exercise every production asset registry through PromptKit engine
|
|
construction and the existing production composition tests. Confirm that:
|
|
- YAML IDs and versions, not filenames, select definitions;
|
|
- every `content_file` path is exact, relative, contained, and points to a
|
|
regular embedded file;
|
|
- every schema and JSON asset is one complete JSON value;
|
|
- every current output contract is valid under v0.8.0; and
|
|
- unrelated malformed definitions do not create a second Notarius identity
|
|
or fallback mechanism.
|
|
3. Review local endpoint parsing and validation. Retain a narrower Notarius
|
|
rule only if it has independent application value; otherwise rely on
|
|
PromptKit's absolute HTTP/HTTPS URL contract. Never accept a value that the
|
|
adapter will later reject.
|
|
4. Review conversion of Notarius variables, inputs, profile extras, and debug
|
|
values at the adapter boundary for PromptKit's bounded JSON-compatible-value
|
|
rules. Do not add a second generic JSON walker or duplicate upstream numeric
|
|
limits.
|
|
5. Verify cancellation and deadline identity through existing adapter tests.
|
|
Add or refine one focused regression only if Notarius currently destroys an
|
|
`errors.Is`-relevant context or transport error that the application owns.
|
|
6. Do not add a cross-operation schema cache, artifact cache, provider-body
|
|
reader, or duplicate JSON framing validation; v0.6.0 owns those mechanisms.
|
|
|
|
### Tests And Validation
|
|
|
|
Run the focused asset, profile-source, and adapter packages, then the ordinary
|
|
and race-enabled suites:
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/framework/llm ./internal/cli
|
|
GOWORK=off go test ./...
|
|
GOWORK=off go test -race ./...
|
|
git diff --check
|
|
```
|
|
|
|
Tests must remain offline and should validate Notarius's assembled boundary,
|
|
not reproduce PromptKit's internal path, JSON-depth, or response-size matrices.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Every maintained embedded prompt, schema, and fallback profile can be loaded
|
|
through the assembled v0.8.0 engine.
|
|
- Current local endpoint and JSON-compatible values either satisfy the stricter
|
|
upstream contract or fail during preparation with safe diagnostics.
|
|
- No duplicate PromptKit-owned cache, JSON, or response-bound mechanism is
|
|
introduced.
|
|
- Cancellation and deadline behavior remains discoverable at the Notarius
|
|
boundary.
|
|
- Ordinary and race-enabled tests pass.
|
|
|
|
## Stage 3: Adopt Profile Inheritance, Rakestrawhome, And Optional Credentials ✅
|
|
|
|
### Goal
|
|
|
|
Make the useful PromptKit v0.7.0 profile and backend behavior work through
|
|
Notarius's existing generic profile boundary without adding provider-specific
|
|
composition logic.
|
|
|
|
### Implementation
|
|
|
|
1. Inspect `promptkit_profiles.go`, `asset_registry.go`, profile fingerprinting,
|
|
CLI profile preflight, profile provenance recording, and their tests before
|
|
editing.
|
|
2. Add an offline integration test using a temporary operator profile source
|
|
whose leaf uses `base_profile`. Prove that:
|
|
- preflight reports the leaf ID;
|
|
- the effective backend, model, reasoning, and other inherited values match
|
|
the resolved PromptKit target;
|
|
- execution uses the same resolved target as inspection; and
|
|
- a missing parent or cycle fails before provider generation with a safe
|
|
profile-load diagnostic.
|
|
Do not duplicate PromptKit's entire field-by-field merge test matrix.
|
|
3. Add a checkpoint-safety test showing that changing a parent definition in
|
|
an operator profile directory changes Notarius's profile-source fingerprint
|
|
while profile content and paths remain absent from the fingerprint value.
|
|
Retain the v0.8.0 catalog marker as coverage for built-in-parent changes.
|
|
4. Verify `rakestrawhome-gemma-4-31b` through the ordinary profile inspector.
|
|
Assert its selected backend reaches Notarius's application-owned inspection
|
|
and provenance fields. Use a fake PromptKit client or transport if execution
|
|
coverage is needed; never contact the live service or require credentials.
|
|
5. Verify that Notarius registers no `rakestrawhome` override and that the
|
|
existing `local` registration remains independent.
|
|
6. Add one `httptest`-backed adapter integration test for a filesystem profile
|
|
with a missing optional `api_key_env`. The request must reach the test server
|
|
without an `Authorization` header. Add a focused in-memory PromptKit profile
|
|
test for `APIKeyRequired` only if needed to prove Notarius preserves upstream
|
|
preflight behavior; do not expose a new operator profile API.
|
|
7. Keep `assets/dnd/profiles/dnd-extraction.yaml` standalone and unchanged. No
|
|
matching v0.8.0 built-in profile owns its `openai/gpt-5.6-luna` target.
|
|
8. Update the current profile-source, deployment, and pinned-integration
|
|
sections in `docs/config.md`, `docs/operations.md`,
|
|
`docs/internal/llm.md`, and `docs/integrations/pkg-promptkit.md`. Link to the
|
|
pinned PromptKit format rules for inheritance. Explain that filesystem
|
|
profiles cannot express PromptKit's in-memory `APIKeyRequired` field and
|
|
that an optional missing credential may result in a provider 401/403.
|
|
|
|
### Tests And Validation
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/framework/llm ./internal/core/config ./internal/cli
|
|
GOWORK=off go test ./...
|
|
GOWORK=off go test -race ./internal/framework/llm ./internal/cli
|
|
git diff --check
|
|
```
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Inherited operator profiles resolve identically during preflight and
|
|
execution, with the leaf ID and effective target kept distinct.
|
|
- Parent changes invalidate checkpoint reuse without leaking profile content or
|
|
paths.
|
|
- Rakestrawhome is available through generic PromptKit profile handling and is
|
|
not selected by default or registered by Notarius.
|
|
- Missing optional credentials omit authorization and reach the controlled
|
|
test provider; explicitly required credentials retain upstream behavior.
|
|
- Current documentation accurately describes the implemented profile and
|
|
credential behavior without duplicating PromptKit's merge algorithm.
|
|
|
|
## Stage 4: Adapt Structured Generation Errors Safely
|
|
|
|
### Goal
|
|
|
|
Use PromptKit v0.7.0's structured generation errors for stable status
|
|
classification and debug-only provider diagnostics without leaking PromptKit
|
|
types or sensitive provider text.
|
|
|
|
### Implementation
|
|
|
|
1. In `internal/framework/contracts`, add:
|
|
- `ErrLLMGeneration` as the provider-neutral generation-failure sentinel;
|
|
- an application-owned `LLMGenerationError` with private status and safe
|
|
diagnostic fields, `Error`, `Unwrap`, and `StatusCode` methods; and
|
|
- a constructor that accepts a nonnegative status and an already-redacted
|
|
diagnostic. Status zero means no HTTP status was available.
|
|
Ordinary callers may inspect status with `errors.As` and category with
|
|
`errors.Is`, but cannot obtain provider code, type, or message from the
|
|
error.
|
|
2. Add an application-owned `LLMDebugProviderError` with `status_code`, `code`,
|
|
`type`, and `message` fields, referenced optionally from
|
|
`LLMDebugResponse`. This is debug material, not a manifest or durable public
|
|
artifact contract.
|
|
3. In `PromptKitClient.CompleteStructured`, preserve precedence in this order:
|
|
caller context cancellation/deadline, PromptKit capacity error, structured
|
|
PromptKit generation error, then other PromptKit generation failures.
|
|
Map every generation failure to `ErrLLMGeneration`; map
|
|
`*promptkit.GenerationError` to `LLMGenerationError` with its status.
|
|
Never wrap or return the PromptKit error value itself.
|
|
4. Keep the ordinary diagnostic limited to PromptKit's safe default error
|
|
formatting after bearer and known-credential redaction. Do not append
|
|
`ProviderCode`, `ProviderType`, or `ProviderMessage` to it.
|
|
5. For an explicitly requested debug path, preserve prepared prompt details and
|
|
attach the PromptKit provider code, type, and message after:
|
|
- reading only the selected prepared target's `APIKeyEnv`, if any, to obtain
|
|
the exact known credential solely for redaction;
|
|
- applying `RedactSecrets` and the existing bearer/key-pattern redaction;
|
|
- retaining PromptKit's already-normalized bounds; and
|
|
- discarding the credential value immediately rather than storing it.
|
|
Do not scan unrelated environment variables.
|
|
6. Return prompt/debug material alongside the error so the existing debug LLM
|
|
wrapper can persist it only when debug recording is enabled. Confirm that
|
|
provider fields do not appear in ordinary error text, warnings, manifests,
|
|
cache, checkpoint data, or a run without debug output.
|
|
7. Refactor error mapping into small helpers if needed to keep
|
|
`CompleteStructured` readable; do not create provider-specific policy in
|
|
modules or the pipeline runner.
|
|
8. Update the error and observability sections of `docs/internal/llm.md` and
|
|
`docs/integrations/pkg-promptkit.md`. Keep operator disclosure rules in
|
|
`docs/operations.md` concise and link to the internal boundary where useful.
|
|
|
|
### Tests And Validation
|
|
|
|
- Use `httptest.Server` to return representative structured 400 and 503
|
|
responses. Assert `errors.Is(ErrLLMGeneration)`, `errors.As` to the
|
|
application-owned type, and the exact status without asserting complete
|
|
human wording.
|
|
- Include a provider message containing the selected test credential and a
|
|
bearer-shaped value. Verify both are absent from the ordinary error and
|
|
debug artifact, while a non-sensitive marker appears only in the requested
|
|
debug trace.
|
|
- Retain existing capacity and context tests to prove their more specific
|
|
classifications still win.
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline ./internal/cli
|
|
GOWORK=off go test ./...
|
|
GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline
|
|
git diff --check
|
|
```
|
|
|
|
### Acceptance Criteria
|
|
|
|
- PromptKit generation errors never escape the adapter error chain.
|
|
- All generation failures match `ErrLLMGeneration`; structured non-success
|
|
responses expose only application-owned HTTP status to ordinary callers.
|
|
- Provider code, type, and message are available only in an explicitly
|
|
requested, redacted debug trace.
|
|
- Capacity and context classifications remain unchanged and more specific.
|
|
- Security tests prove selected credentials and bearer tokens are not leaked.
|
|
|
|
## Stage 5: Add Adapter-Level Structured Repair Support
|
|
|
|
### Goal
|
|
|
|
Teach the transport-neutral completion boundary and PromptKit adapter to apply
|
|
an optional repair override and report actual repair behavior, without yet
|
|
exposing the setting in pipeline configuration.
|
|
|
|
### Implementation
|
|
|
|
1. Add `StructuredOutputRepairAttempts *int` to
|
|
`contracts.StructuredCompletionRequest`. Copy the pointed-to value wherever
|
|
requests are cloned or retained.
|
|
2. Add `RepairAttempts int` to `contracts.StructuredCompletionResponse`. It is
|
|
the actual number of corrective generation calls, not the configured budget
|
|
and not the number of total candidates.
|
|
3. Validate a non-nil request value as zero through three at the adapter
|
|
boundary so programmatic callers cannot bypass later file/config validation.
|
|
4. When the request value is nil, leave `promptkit.RunRequest.Validation` nil
|
|
so the prompt's complete contract remains authoritative.
|
|
5. When the value is non-nil:
|
|
- call `Engine.InspectPrompt(ctx, promptID, promptVersion)`;
|
|
- copy `PromptInspection.OutputContract` by value;
|
|
- replace only `RepairAttempts`;
|
|
- pass the complete copied contract as `RunRequest.Validation`; and
|
|
- prepare and execute exactly as before.
|
|
Do not infer or hard-code schema paths, validation modes, or formats. Do not
|
|
cache inspection in this stage.
|
|
6. Map `result.Validation.RepairAttempts` to the response and leave
|
|
`result.Usage` cumulative values unchanged. The existing debug validation
|
|
object and prepared output contract should show actual and configured values
|
|
respectively.
|
|
7. Preserve result semantics:
|
|
- valid initial and repaired candidates decode normally;
|
|
- repair exhaustion returns the final raw candidate/debug material with an
|
|
error matching `ErrInvalidStructuredOutput`;
|
|
- explicit empty or whitespace-only content follows PromptKit validation;
|
|
- missing/null/non-string content remains a generation/provider failure;
|
|
- corrective-call generation errors use Stage 4's application-owned mapping;
|
|
and
|
|
- context cancellation wins at every error boundary.
|
|
8. Keep `CompleteStructured` and its helpers provider neutral outside this
|
|
adapter package. Do not expose PromptKit validation or inspection types.
|
|
9. Update only the adapter-owned repair behavior in `docs/internal/llm.md` and
|
|
`docs/integrations/pkg-promptkit.md`. State that public pipeline configuration
|
|
and the production default are added by later stages of this plan.
|
|
|
|
### Tests And Validation
|
|
|
|
Add adapter-level behavioral tests using a deterministic fake PromptKit LLM:
|
|
|
|
- nil override uses the prompt declaration;
|
|
- explicit zero overrides a positive prompt declaration without dropping its
|
|
JSON Schema contract;
|
|
- explicit one turns an invalid first candidate followed by a valid candidate
|
|
into one successful response with the final raw bytes, actual repair count
|
|
one, and cumulative usage;
|
|
- repair exhaustion returns the final candidate and validation diagnostics as
|
|
`ErrInvalidStructuredOutput`;
|
|
- explicit empty content is eligible for repair;
|
|
- a corrective generation failure maps through Stage 4; and
|
|
- invalid direct values below zero or above three fail before provider work.
|
|
|
|
Do not assert PromptKit's exact assistant/user correction prose or copy its
|
|
full internal repair matrix.
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/framework/contracts ./internal/framework/llm ./internal/framework/pipeline
|
|
GOWORK=off go test -race ./internal/framework/llm ./internal/framework/pipeline
|
|
GOWORK=off go test ./...
|
|
git diff --check
|
|
```
|
|
|
|
### Acceptance Criteria
|
|
|
|
- The adapter changes only repair count when applying a request override.
|
|
- Nil and explicit zero remain distinct.
|
|
- Repaired success returns final raw output, cumulative usage, and actual count
|
|
without a warning.
|
|
- Exhaustion, empty content, corrective generation failure, and cancellation
|
|
match the target semantics.
|
|
- No PromptKit type crosses the LLM package boundary.
|
|
|
|
## Stage 6: Propagate Repair Policy Through Framework Requests
|
|
|
|
### Goal
|
|
|
|
Carry an optional effective repair budget from each resolved stage or validator
|
|
binding to its module request without changing public file configuration yet.
|
|
|
|
### Implementation
|
|
|
|
1. Add `StructuredOutputRepairAttempts *int` alongside `LLMProfile` to every
|
|
stage request that can belong to an LLM-backed binding:
|
|
- `ParseRequest`;
|
|
- `ChunkRequest`;
|
|
- `TypedExtractionRequest`;
|
|
- `TypedMergeRequest`;
|
|
- `TypedNormalizeRequest`;
|
|
- `OutputRequest`;
|
|
- `TypedValidationRequest`;
|
|
- `ChunkValidationRequest`; and
|
|
- `SerializedValidationRequest`.
|
|
2. Add the same optional field to the erased/internal request carriers used by
|
|
registry builders, preparation, runner stage attempts, validator targets,
|
|
retry closures, and debug wrappers. Copy pointer values; never share a
|
|
mutable pointer owned by configuration.
|
|
3. At every runner stage invocation, obtain the value from the exact resolved
|
|
producer binding. At every validator invocation, obtain it from that exact
|
|
resolved validator binding. Do not use the producer's value for a validator
|
|
or vice versa.
|
|
4. Ensure all retry attempts for the same binding receive the same effective
|
|
structural-repair value. Do not decrement it in Notarius; PromptKit owns the
|
|
inner budget independently on each `CompleteStructured` call.
|
|
5. Extend `semanticreconcile.Request` with the optional field and carry it into
|
|
each generic reconciliation completion. A batched reconciliation may make
|
|
several completion calls; each call receives the same effective budget.
|
|
6. Update registry erasure/adaptation code for typed merge, normalize, and
|
|
validation requests so no field is lost. Preserve input/output support even
|
|
though current production input and output modules are deterministic.
|
|
7. Add focused framework tests for one chunk producer, one extraction
|
|
producer, one normalizer, and one LLM-backed validator. Verify exact pointer
|
|
value propagation and separation between producer and validator settings.
|
|
Do not add repetitive tests for every generic adapter.
|
|
|
|
### Tests And Validation
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/framework/contracts ./internal/framework/pipeline ./internal/framework/semanticreconcile
|
|
GOWORK=off go test -race ./internal/framework/pipeline ./internal/framework/semanticreconcile
|
|
GOWORK=off go test ./...
|
|
git diff --check
|
|
```
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Every stage and validator request can carry a detached optional repair value.
|
|
- The runner sources the value from the exact resolved binding.
|
|
- Producer and validator values cannot overwrite one another.
|
|
- Stage retries reuse but do not mutate or consume the inner repair budget.
|
|
- Semantic reconciliation forwards the budget to every one of its completion
|
|
calls.
|
|
- Existing behavior remains unchanged while all values are nil.
|
|
|
|
## Stage 7: Forward Repair Policy From Every LLM-Backed Module
|
|
|
|
### Goal
|
|
|
|
Complete the internal end-to-end path by having every production LLM-backed
|
|
module forward its stage request value to `CompleteStructured`.
|
|
|
|
### Implementation
|
|
|
|
1. Inventory every production `CompleteStructured` call with code search before
|
|
editing. The expected current owners include:
|
|
- `dnd/scenes` chunking;
|
|
- the combat-turn, enemy-event, item-occurrence, item-registry,
|
|
location-occurrence, location-registry, NPC-occurrence, NPC-registry,
|
|
scene-description, and spell extractors; and
|
|
- generic semantic reconciliation used by the item, location, and NPC
|
|
registry normalizers.
|
|
Reconcile this list with the actual repository; do not omit a newly added
|
|
production caller merely because it is not named here.
|
|
2. In each direct caller, set
|
|
`StructuredCompletionRequest.StructuredOutputRepairAttempts` from the
|
|
corresponding stage request. Clone the pointer or use a small shared helper
|
|
if that reduces repeated ownership mistakes without moving domain logic.
|
|
3. Ensure D&D registry normalizers pass their typed normalize request value into
|
|
`semanticreconcile.Request`, and that the generic engine forwards it as
|
|
established in Stage 6.
|
|
4. Update existing module prompt-mapping tests that already inspect a captured
|
|
structured-completion request to assert the new field. Do not create a new
|
|
one-test-per-module suite solely to memorialize field plumbing; rely on the
|
|
existing request-contract tests plus a final complete call-site audit.
|
|
5. Search again after editing for production `CompleteStructured` calls and
|
|
verify each either forwards the field or documents why it cannot receive a
|
|
pipeline binding. Test-only fakes need only preserve the field when their
|
|
contract test depends on it.
|
|
6. Do not set a module-specific fallback value. Nil must reach the adapter so
|
|
the prompt declaration remains authoritative.
|
|
|
|
### Tests And Validation
|
|
|
|
Run focused D&D and semantic-reconciliation packages, then the full suite:
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/modules/dnd/... ./internal/framework/semanticreconcile
|
|
GOWORK=off go test -race ./internal/modules/dnd/... ./internal/framework/semanticreconcile
|
|
GOWORK=off go test ./...
|
|
git diff --check
|
|
```
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Every production LLM-backed completion receives the exact stage or validator
|
|
repair value.
|
|
- No module invents a default or imports PromptKit.
|
|
- Registry normalizers preserve the value through semantic reconciliation.
|
|
- Existing request-contract tests remain concise and pass.
|
|
- A final call-site audit finds no silent production omission.
|
|
|
|
## Stage 8: Add The Public Repair Configuration Contract
|
|
|
|
### Goal
|
|
|
|
Add presence-aware pipeline and binding configuration for
|
|
`structured_output_repair_attempts` without yet changing runtime resolution.
|
|
|
|
### Implementation
|
|
|
|
1. Add `StructuredOutputRepairAttempts *int` to
|
|
`pipeline.PipelineProfile` and `pipeline.ModuleBinding`, using
|
|
`json:"structured_output_repair_attempts,omitempty"`.
|
|
2. Add presence-aware YAML support at pipeline and object-binding scope:
|
|
- exact key `structured_output_repair_attempts`;
|
|
- integer values zero through three;
|
|
- explicit null, non-integer, and out-of-range values rejected with scoped
|
|
diagnostics; and
|
|
- scalar shorthand bindings continue to omit the binding override.
|
|
Preserve file configuration version 4.
|
|
3. Update every configuration clone, conversion, redaction, summary, and JSON
|
|
round-trip carrier. Copy pointers by value into newly allocated storage so
|
|
parsed, configured, and redacted values do not alias.
|
|
4. Preserve omission versus explicit zero through YAML parsing, profile
|
|
inheritance, module-binding object form, and JSON round trips. Keep scalar
|
|
shorthand bindings equivalent to omission.
|
|
5. Do not add a top-level `promptkit.repair_attempts` setting or CLI override.
|
|
6. Add concise parser and ownership tests. Defer execution-class checks,
|
|
effective precedence, resolved digests, and runtime forwarding to Stage 9,
|
|
where module metadata is available.
|
|
|
|
### Tests And Validation
|
|
|
|
At the parser/config boundary, test omitted, explicit zero, positive bounds,
|
|
negative, above-three, null, non-integer, scalar shorthand, cloning, redaction,
|
|
and JSON round-trip behavior. Use relational boundary tests for the allowed
|
|
range and avoid duplicating the same cases at every layer.
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/core/config ./internal/cli
|
|
GOWORK=off go test -race ./internal/core/config
|
|
GOWORK=off go test ./...
|
|
go run ./cmd/notarius config validate \
|
|
--config examples/dnd-minimal.config.yml \
|
|
--pipeline dnd-session
|
|
go run ./cmd/notarius config validate \
|
|
--config examples/dnd-complete.config.yml \
|
|
--pipeline dnd-session
|
|
git diff --check
|
|
```
|
|
|
|
### Acceptance Criteria
|
|
|
|
- The exact public field parses at pipeline and object-binding scope with the
|
|
fixed range.
|
|
- Nil and explicit zero remain distinguishable through parsing, cloning,
|
|
inheritance, redaction, summaries, and round trips.
|
|
- Both maintained configurations remain valid without requiring the new field.
|
|
- No runtime or prompt default has changed prematurely.
|
|
|
|
## Stage 9: Resolve And Apply Repair Configuration
|
|
|
|
### Goal
|
|
|
|
Resolve the public field against module execution classes, incorporate the
|
|
effective value into pipeline identity, and connect it to the request plumbing
|
|
completed in Stages 6 and 7.
|
|
|
|
### Implementation
|
|
|
|
1. During resolution, compute the effective value for every selected binding:
|
|
- explicit binding value wins;
|
|
- otherwise an explicit pipeline value applies to an LLM-backed binding;
|
|
- otherwise leave nil for prompt-owned policy.
|
|
Apply the pipeline value to LLM-backed validators as well as producers.
|
|
2. Reject an explicit binding value on a deterministic module or deterministic
|
|
validator using the same execution-class knowledge used for `llm_profile`.
|
|
Do not reject a pipeline-level value merely because a selected pipeline also
|
|
contains deterministic bindings; simply do not apply it to those bindings.
|
|
3. Clone every resolved pointer so the parsed pipeline, resolved profile,
|
|
redacted summaries, and runner requests have distinct ownership.
|
|
4. Include the effective field in resolved pipeline JSON and digest input. A
|
|
change between nil, zero, and a positive value must change the resolved
|
|
digest when it changes an LLM-backed selected binding. Unselected lanes must
|
|
retain the repository's existing digest and selection semantics.
|
|
5. Pass the resolved value into the Stage 6 request field for every selected
|
|
input, chunk, extract, merge, normalize, output, and validator binding.
|
|
6. Update `docs/config.md` as the canonical field, range, and precedence
|
|
contract; `docs/internal/pipeline.md` as the resolution owner; and
|
|
`docs/operations.md` for the distinction from binding `retries`. The
|
|
prompt-owned production default remains unchanged until Stage 10.
|
|
7. Add focused resolution and runner tests. Cover representative execution
|
|
classes rather than repeating the same assertion for every module type.
|
|
|
|
### Tests And Validation
|
|
|
|
Test:
|
|
|
|
- binding over pipeline over nil precedence;
|
|
- inheritance into each selected LLM-backed stage and validator;
|
|
- no inheritance into deterministic bindings;
|
|
- explicit deterministic-binding rejection;
|
|
- detached pointers;
|
|
- runner forwarding for representative producer and validator bindings; and
|
|
- digest changes for execution-relevant nil, zero, and positive changes.
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/framework/pipeline ./internal/cli
|
|
GOWORK=off go test -race ./internal/framework/pipeline
|
|
GOWORK=off go test ./...
|
|
go run ./cmd/notarius config validate \
|
|
--config examples/dnd-minimal.config.yml \
|
|
--pipeline dnd-session
|
|
go run ./cmd/notarius config validate \
|
|
--config examples/dnd-complete.config.yml \
|
|
--pipeline dnd-session
|
|
git diff --check
|
|
```
|
|
|
|
### Acceptance Criteria
|
|
|
|
- The exact public field has the fixed binding-over-pipeline-over-prompt
|
|
precedence for every selected LLM-backed producer and validator.
|
|
- Deterministic binding misuse fails during resolution before execution, while
|
|
a pipeline value coexists with deterministic bindings.
|
|
- Nil and explicit zero remain distinguishable through resolution, runtime,
|
|
summaries, and digests.
|
|
- A policy change invalidates checkpoint identity when it changes an effective
|
|
selected binding.
|
|
- Current configuration, pipeline, and operations documentation matches the
|
|
implemented behavior.
|
|
|
|
## Stage 10: Enable The Default, Finish Documentation, And Verify The Feature
|
|
|
|
### Goal
|
|
|
|
Set the accepted production default of one repair, reconcile all canonical
|
|
documentation, and run the full repository verification pass.
|
|
|
|
### Implementation
|
|
|
|
1. Change `repair_attempts: 0` to `repair_attempts: 1` in every maintained
|
|
production prompt manifest that produces structured output, including the
|
|
generic semantic-reconciliation prompt and every D&D chunk, extraction, and
|
|
registry-normalization prompt. Do not mechanically change unrelated test
|
|
fixtures whose purpose is to exercise zero.
|
|
2. Inspect every production prompt output contract after the edit. Confirm that
|
|
each positive budget uses `basic`, `json`, or `json_schema`, remains no
|
|
greater than three, and retains its existing format and schema path.
|
|
3. Add or refine the smallest durable assembled-assets test that proves the
|
|
production engine can prepare the maintained prompts with the activated
|
|
contracts. Do not add a brittle test that asserts an exact prompt count,
|
|
file count, message prose, correction text, or asset length. The public
|
|
default may be tested at one canonical assembled boundary because its
|
|
literal value is an operational contract.
|
|
4. Confirm a successful repair does not create a warning and that exhausted
|
|
repair remains `ErrInvalidStructuredOutput`. Verify the debug prompt records
|
|
the configured contract, the debug response records actual repair count,
|
|
and cumulative usage is not double-counted.
|
|
5. Confirm scheduling behavior with one focused test or existing coverage: the
|
|
Notarius scheduled client admits one logical `CompleteStructured` operation
|
|
while PromptKit may make serial corrective provider calls inside it. Do not
|
|
attempt to reacquire a Notarius permit from inside PromptKit or add a second
|
|
scheduler.
|
|
6. Reconcile current-state documentation:
|
|
- `docs/integrations/pkg-promptkit.md` owns the pinned upstream boundary;
|
|
- `docs/config.md` owns field names, range, default, and precedence;
|
|
- `docs/operations.md` owns latency/cost, optional credentials, concurrency,
|
|
timeout, and the upper-bound formula;
|
|
- `docs/internal/llm.md` owns inspection-based contract replacement,
|
|
cumulative usage, actual repair count, generation errors, and debug data;
|
|
- `docs/internal/pipeline.md` owns effective policy propagation and the
|
|
separation from stage retries; and
|
|
- `docs/policy/architecture.md` adds only the durable rule that PromptKit
|
|
owns deterministic structural repair inside one completion while Notarius
|
|
owns stage attempts and semantic validation.
|
|
7. Remove current-behavior claims that PromptKit is v0.5.0, that every
|
|
production repair budget is zero, or that PromptKit is always single-pass.
|
|
Do not alter historical release notes or archived roadmaps.
|
|
8. Keep the maintained minimal and complete examples secret-free and valid.
|
|
They may omit the new field to demonstrate the default; do not add a
|
|
redundant complete profile or Rakestrawhome example merely to exercise an
|
|
upstream catalog entry.
|
|
9. Review `docs/roadmap/future.md` only for consistency. Leave the future
|
|
feedback-aware stage retry, combat-scene validator, and warning-reform work
|
|
unimplemented and clearly separate.
|
|
|
|
### Tests And Validation
|
|
|
|
Run focused tests first, then all repository checks:
|
|
|
|
```sh
|
|
GOWORK=off go test ./internal/framework/llm ./internal/framework/pipeline ./internal/framework/semanticreconcile ./internal/modules/dnd/...
|
|
GOWORK=off go test ./...
|
|
GOWORK=off go test -race ./...
|
|
GOWORK=off go vet ./...
|
|
GOWORK=off go build ./cmd/notarius
|
|
GOWORK=off go mod tidy -diff
|
|
go run ./cmd/notarius config validate \
|
|
--config examples/dnd-minimal.config.yml \
|
|
--pipeline dnd-session
|
|
go run ./cmd/notarius config validate \
|
|
--config examples/dnd-complete.config.yml \
|
|
--pipeline dnd-session
|
|
git diff --check
|
|
```
|
|
|
|
Also perform focused repository searches that exclude `docs/roadmap/archive/`
|
|
and historical release notes:
|
|
|
|
- no active v0.5.0 PromptKit pins or links remain;
|
|
- no maintained production prompt still declares `repair_attempts: 0`;
|
|
- every production `CompleteStructured` caller forwards the repair field; and
|
|
- no provider code, type, or message is added to ordinary errors, warnings,
|
|
manifests, cache, or checkpoint schemas.
|
|
|
|
If the repository's source-release checker is available and the ordinary
|
|
checks above pass, run `./scripts/check-release-source.sh v0.0.0` as the final
|
|
integrated validation. It must not create a tag, release note, or repository
|
|
artifact.
|
|
|
|
### Acceptance Criteria
|
|
|
|
- Every maintained structured prompt defaults to one corrective call and can
|
|
be overridden to zero through three at pipeline or binding scope.
|
|
- A real assembled Notarius completion follows the PromptKit v0.8.0 repair
|
|
contract without changing prompt schema identity or cacheable prefix.
|
|
- Actual repair count, cumulative usage, error classification, debug-only
|
|
provider diagnostics, scheduling, and checkpoint identity match the feature
|
|
roadmap.
|
|
- Profile inheritance, Rakestrawhome availability, optional credentials, and
|
|
v0.6.0 hardening remain covered and documented.
|
|
- All canonical documentation describes implemented v0.8.0 behavior in its
|
|
assigned home and leaves future semantic validation work in the roadmap.
|
|
- Maintained examples validate, all ordinary/race/vet/build/module checks pass,
|
|
and the worktree contains no generated or sensitive artifacts.
|