338 lines
18 KiB
Markdown
338 lines
18 KiB
Markdown
# ConfigKit And PromptKit Upgrades
|
|
|
|
## Purpose
|
|
|
|
Adopt ConfigKit v0.4.0 and PromptKit v0.11.0 in one coordinated work set:
|
|
|
|
- Consistently call ConfigKit's selected configuration layer an **overlay**,
|
|
distinct from PromptKit LLM profiles.
|
|
- Adopt configurable transient HTTP retries, provider-neutral failure
|
|
classifications, and execution observations from PromptKit.
|
|
|
|
Preserve application/framework boundaries and keep the two upgrades as distinct
|
|
implementation workstreams. Neither requires a redesign of pipeline execution.
|
|
|
|
This feature roadmap records the target state and scope. Its original
|
|
implementation plan was completed and later replaced by a new work plan.
|
|
Neither document authorized release publication.
|
|
|
|
## Current Baseline
|
|
|
|
The reviewed baseline is Notarius v0.7.1 (`53d9284`):
|
|
|
|
- `go.mod` pins ConfigKit v0.3.0 and PromptKit v0.10.0.
|
|
- `internal/core/config/load.go` passes `LoadOptions.Profile` into
|
|
`configkit.Options[Document].Profile`.
|
|
- `--config-profile` selects `profiles/<name>.yml` beneath the configuration
|
|
root for `run`, `config validate`, `config effective`, `config explain`, and
|
|
`pipelines list`.
|
|
- The maintained selected-layer example is
|
|
`examples/profiles/development.yml`; configuration and CLI tests create
|
|
equivalent directories and exercise selection and inspection.
|
|
- ConfigKit owns composition and inspection metadata. Notarius owns its
|
|
version 5 document, defaults, validation, source-relative path resolution,
|
|
runtime conversion, and CLI discovery.
|
|
|
|
A dependency-only trial against v0.4.0 fails to compile at `load.go:33` because
|
|
`configkit.Options.Profile` no longer exists. The review found no direct
|
|
Notarius references to `LayerProfile`, `CodeProfile*`, or `LoadSecretsDir`.
|
|
|
|
An isolated dependency-only trial against PromptKit v0.11.0 passes the complete
|
|
Notarius test suite and `go vet ./...`. That proves baseline compatibility, not
|
|
adoption of the new controls or observations. The current adapter does not
|
|
configure transient retries or attach an execution observer. It classifies
|
|
generation errors broadly and records successful-result usage and structural
|
|
repair counts. The outer scheduler holds a permit for the whole completion.
|
|
|
|
## Upstream Changes
|
|
|
|
### ConfigKit v0.4.0
|
|
|
|
The [ConfigKit v0.4.0 release guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.4.0/docs/releases/v0.4.0.md)
|
|
defines four breaking naming changes:
|
|
|
|
| Surface | Previous | Target |
|
|
| --- | --- | --- |
|
|
| Selection API | `Options.Profile` | `Options.Overlay` |
|
|
| Selected file | `profiles/<name>.yml` | `overlays/<name>.yml` |
|
|
| Provenance layer | `LayerProfile` / `profile` | `LayerOverlay` / `overlay` |
|
|
| Diagnostic identifiers | `CodeProfile*` / `profile.*` | `CodeOverlay*` / `overlay.*` |
|
|
|
|
There are no legacy aliases or directory fallback. Explicit imports from a
|
|
directory named `profiles` remain valid ordinary imports. Merge rules,
|
|
precedence, lifecycle, source-origin APIs, minimum Go version, and YAML parser
|
|
dependency are unchanged.
|
|
|
|
The intervening v0.3.1 release makes the opt-in `LoadSecretsDir` API ignore
|
|
dot-prefixed entries. Notarius does not use this API; adding secrets-directory
|
|
bootstrap is outside this work.
|
|
|
|
### PromptKit v0.11.0
|
|
|
|
The [PromptKit v0.11.0 release guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/releases/v0.11.0.md)
|
|
introduces additive runtime APIs and one changed default:
|
|
|
|
- Built-in generation defaults to two additional transient HTTP attempts for
|
|
statuses 429, 500, 502, 503, and 504. Engine and request controls can override
|
|
the count, including zero. Injected `LLMClient` implementations are not
|
|
automatically replayed; a custom HTTP client on the built-in path still is.
|
|
- Backoff is cancellable and honors valid provider `Retry-After` timing.
|
|
Generation and HTTP timeouts apply per attempt; the caller context bounds
|
|
the whole operation. Transport failures, malformed responses, other HTTP
|
|
statuses, and invalid generated content are not transient-retry candidates.
|
|
- `RunRequest.Observer` reports lifecycle metadata, attempts, waits, outcomes,
|
|
and presence-aware usage, including work before a failed execution. It works
|
|
with prepared execution; preparation itself emits no execution events.
|
|
- `ErrTransport`, `ErrMalformedResponse`, `ErrRetriesExhausted`, and
|
|
`RetryExhaustedError` support typed classification without parsing error text.
|
|
- The license changes to LGPL-3.0-or-later. Catalog selections and the minimum
|
|
Go version are unchanged from v0.10.0.
|
|
|
|
## Target End State
|
|
|
|
### Dependency And Ownership
|
|
|
|
Pin ConfigKit v0.4.0 and PromptKit v0.11.0 and tidy the dependency graph.
|
|
Continue using only ConfigKit's
|
|
public root API behind `internal/core/config`. No compatibility parser,
|
|
application-owned composition engine, or new configuration abstraction is
|
|
needed. PromptKit remains behind `internal/framework/llm`; its types must not
|
|
leak into domain modules or pipeline contracts. Retain the existing prepared
|
|
execution path, profile choices, and D&D model settings.
|
|
|
|
### Consistent Overlay Selection
|
|
|
|
- Replace the configuration-selection CLI flag with `--config-overlay` on all
|
|
commands that currently accept `--config-profile`.
|
|
- Rename application-owned fields, variables, helpers, help text, and test
|
|
descriptions that specifically represent this selection to overlay terms.
|
|
- Pass the selection through `configkit.Options.Overlay`.
|
|
- Select files only from `overlays/<name>.yml`, following ConfigKit's contract.
|
|
- Preserve empty and `default` selection as the composed base configuration
|
|
without a selected overlay.
|
|
- Use a clean cutover: no old CLI alias, directory fallback, automatic file
|
|
moves, or custom migration machinery. Normal unknown-flag handling is
|
|
sufficient for the retired flag; explain migration in documentation.
|
|
|
|
This rename is deliberately narrow. Preserve `llm_profile`, `--llm-profile`,
|
|
`promptkit.profile_dir`, `promptkit.profile_file`, `base_profile`, embedded
|
|
`dnd-extraction` profiles, and `examples/llm-profiles/`. Other application
|
|
concepts named `Profile`, including pipeline/runtime types, must not be renamed
|
|
merely because they share the word.
|
|
|
|
### Preserved Behavior
|
|
|
|
Retain configuration version 5, with the additive retry setting described below.
|
|
Preserve existing defaults other than the explicitly adopted transient retry policy,
|
|
imports, overlay merge behavior (including sequence replacement), explicit CLI
|
|
override precedence, strict decoding, domain validation, and secret redaction.
|
|
Do not add new restrictions or relax existing explicit contracts as part of the
|
|
rename.
|
|
|
|
Relative paths continue to resolve from the file supplying the surviving
|
|
declaration, including overlay files. Invocation-relative CLI paths and
|
|
main-file-relative built-in defaults retain their current behavior.
|
|
|
|
Effective configuration, explanations, and configuration-load diagnostics use
|
|
the new upstream overlay provenance and diagnostic vocabulary. Preserve source
|
|
file labels and actionable error context; do not translate the new metadata
|
|
back into legacy profile terminology.
|
|
|
|
Pipeline stage order, semantic acceptance, LLM selection, domain artifact
|
|
schemas, and subprocess result receipts remain unchanged. No blanket checkpoint
|
|
invalidation is added merely for the ConfigKit rename. Update the existing
|
|
PromptKit-version identity used by checkpoint fingerprinting with its dependency
|
|
pin, following the current adapter convention; no cache redesign is needed.
|
|
|
|
### Transient Retry Policy
|
|
|
|
Expose `promptkit.transient_retries` as a nonnegative integer setting with a
|
|
Notarius default of **2 additional attempts**. Explicit zero disables transient
|
|
retries. Apply the resolved setting to the shared PromptKit engine so modules,
|
|
normalizers, validators, and semantic correction calls receive the same policy.
|
|
Include the effective value in configuration inspection. Do not impose an
|
|
arbitrary application maximum or add per-binding controls in this work set.
|
|
|
|
Maintain three separate budgets and owners:
|
|
|
|
| Budget | Owner | Repeated work |
|
|
| --- | --- | --- |
|
|
| Transient retries | PromptKit | The same generation request after eligible HTTP failures |
|
|
| Structural repairs | PromptKit | A corrective generation for invalid structured output |
|
|
| Producer/validator retries | Notarius | The configured stage or validator operation, with semantic feedback where applicable |
|
|
|
|
Do not silently reduce, repurpose, or remove existing stage/validator budgets.
|
|
Each structural repair generation receives its own transient budget. For a
|
|
single-completion producer, two additional stage attempts, one structural
|
|
repair, and two transient retries permit up to 18 HTTP attempts, excluding any
|
|
separate validator calls. Document this multiplication and the possibility of
|
|
additional cost; retries do not imply exactly-once provider execution.
|
|
|
|
Preserve caller cancellation throughout generation and waiting. Do not describe
|
|
a profile's generation timeout as a total completion deadline: provider
|
|
`Retry-After` can extend waits beyond local backoff, and each HTTP attempt has
|
|
its own timeout. A new total-deadline configuration surface is outside this
|
|
upgrade; accurately document the current controls and limitations.
|
|
|
|
Retain Notarius's completion-scoped scheduler permit during upstream backoff.
|
|
PromptKit releases its active backend permit during that wait, but this does
|
|
not release the outer Notarius permit. This remains bounded and correct; do
|
|
not add an attempt-level scheduler redesign.
|
|
|
|
### Provider-Neutral Failure Diagnostics
|
|
|
|
Translate the new typed errors into application-owned classifications for
|
|
transport failures and malformed provider responses. Represent transient retry
|
|
exhaustion separately from its underlying failure/status, retaining the actual
|
|
attempt count where available. Preserve existing HTTP-status reporting,
|
|
capacity classification, cancellation, deadline handling, and redaction.
|
|
|
|
Do not turn a malformed provider envelope into a semantic rejection or confuse
|
|
it with a structurally invalid model response. Do not expose PromptKit error
|
|
types across the adapter boundary or parse prose to recover classifications.
|
|
Existing stage disposition policies continue to decide what happens after a
|
|
completion fails; this work does not redesign those policies.
|
|
|
|
### Execution Observations
|
|
|
|
Attach a request-local observer through the existing prepared-execution path.
|
|
Translate selected metadata into Notarius-owned completion/debug summaries:
|
|
|
|
- dispatched attempt, transient retry, and structural repair-call counts;
|
|
- capacity-wait, generation, retry-delay, and total execution durations;
|
|
- terminal outcome, failure classification, HTTP status, and exhaustion state;
|
|
- observed token counts with explicit known/complete indicators.
|
|
|
|
Retain available summaries on failed completions, including cancellation, rather
|
|
than discarding them with the absent `RunResult`. Preparation failures have no
|
|
execution observation; do not fabricate one. Keep existing successful-result
|
|
usage fields and their meanings unchanged. Observation totals are separate
|
|
presence-aware metadata, not a second set of tokens to add to existing usage.
|
|
Unknown counts are not known zero; partial totals are not a billing ledger.
|
|
|
|
Use lightweight per-request accumulation with no unbounded event history or
|
|
asynchronous queue. Callbacks are synchronous: avoid blocking I/O or acquisition
|
|
of scheduler resources inside them, and isolate concurrent requests. Persist
|
|
the projection through the existing opt-in debug path, including safe terminal
|
|
summaries where applicable. Do not serialize upstream event structs directly;
|
|
their JSON layout is not a public contract. Allowlist metadata and preserve
|
|
existing identifier/content handling and redaction boundaries.
|
|
|
|
Successful transient recovery is ordinary success, not a warning. This work
|
|
does not add routine console messages, change the warning policy, introduce a
|
|
new durable telemetry artifact, or add complete cost accounting.
|
|
|
|
## Required Changes
|
|
|
|
### Code And Examples
|
|
|
|
- Update `go.mod` and `go.sum` and the configuration loader/options boundary.
|
|
- Update CLI selection plumbing in `internal/cli/run.go` and
|
|
`configkit_commands.go`, including usage, flag parsing, argument classification,
|
|
explicit empty-value handling, and error paths.
|
|
- Move the maintained example to `examples/overlays/development.yml` and update
|
|
example discovery and related fixtures. Inspect relative paths when moving
|
|
files; do not assume all occurrences of `profiles` are configuration overlays.
|
|
- Update affected configuration and CLI tests, particularly document paths,
|
|
operational defaults, safe inspection, command selection, diagnostic source
|
|
labels, run contracts, and maintained-example coverage.
|
|
- Add the retry setting through the typed configuration document, defaults,
|
|
validation, runtime conversion, and CLI client-construction boundary. Update
|
|
the complete example to demonstrate it without adding unnecessary settings
|
|
to the minimal example.
|
|
- Update the PromptKit adapter's engine construction, request observation,
|
|
error conversion, and checkpoint version identity. Extend transport-neutral
|
|
response/debug contracts only with the application-owned metadata needed.
|
|
- Carry observations through the pipeline debug wrapper and safe terminal
|
|
reporting without double-counting usage or losing failed-call summaries.
|
|
|
|
### Documentation
|
|
|
|
Update current documentation with the implementation, keeping canonical
|
|
ownership under [Documentation Policy](../../policy/documentation.md):
|
|
|
|
- `docs/cli.md`: new flag on every affected command and matching help behavior.
|
|
- `docs/config.md`: overlay terminology, directory layout, precedence, and the
|
|
explicit distinction from LLM profiles; transient retry setting and default.
|
|
- `docs/integrations/pkg-configkit.md`: pinned release and supported upstream
|
|
naming/inspection contract.
|
|
- `docs/internal/configuration.md` and relevant CLI internals: new selection
|
|
terminology and unchanged boundaries.
|
|
- `docs/consumers/subprocess.md` and any other current callers/examples: updated
|
|
invocation syntax. Review operations and README references where applicable.
|
|
- `docs/integrations/pkg-promptkit.md`: updated pin, typed errors, observer
|
|
support, and retry contract. Refresh pinned upstream links, including its
|
|
relocated release-policy document.
|
|
- `docs/internal/llm.md` and relevant pipeline/debug internals: observation
|
|
projection, failure handling, scheduler interaction, and separate retry
|
|
budgets. Remove the obsolete claim that PromptKit has no provider retry loop.
|
|
- `docs/operations.md`: latency/cost implications, timeout limits, opt-in debug
|
|
evidence, and how to disable transient retries.
|
|
|
|
Preserve historical release notes and the accepted text of ADR-0018. Its
|
|
ConfigKit ownership decision remains valid; a terminology migration alone does
|
|
not require a new ADR or supersession. Current documentation must not portray
|
|
the old dependency pin as the current supported version.
|
|
|
|
Record PromptKit's LGPL-3.0-or-later change in the integration documentation and
|
|
eventual release note. Review its notices and distribution implications within
|
|
Notarius's source-only release model. Do not silently relicense Notarius, copy
|
|
upstream release policy into this repository, or introduce binary packaging.
|
|
Any future binary distribution needs its own license-compliance review.
|
|
|
|
## Operator Migration And Release Impact
|
|
|
|
This work belongs in the next **minor** Notarius release, not a patch, because
|
|
both the documented selected-file layout and CLI syntax change.
|
|
|
|
Operators move selected configuration files from `profiles/` to `overlays/`
|
|
and replace `--config-profile` with `--config-overlay` in scripts, service
|
|
definitions, and subprocess callers. Preserve unrelated PromptKit profile
|
|
directories and explicit imports. Review relative paths and explicit imports
|
|
that refer to moved files. No YAML body version migration is required.
|
|
|
|
The eventual release note must explain this cutover, the newly enabled transient
|
|
retries and zero-value opt-out, and the dependency license change. PromptKit's
|
|
catalogs are unchanged from v0.10.0; this upgrade adds no further profile-name
|
|
migration or change to the D&D model defaults. Release preparation,
|
|
tagging, and publication remain a separately requested task under
|
|
[Source Releases](../../release.md).
|
|
|
|
## Acceptance And Verification
|
|
|
|
- Notarius builds against ConfigKit v0.4.0 and PromptKit v0.11.0 without
|
|
compatibility shims.
|
|
- All five affected commands use the new flag; non-default overlay selection,
|
|
omitted selection, empty selection, and `default` retain their intended
|
|
behavior. The old flag is not accepted.
|
|
- Selected overlays load from the new directory. A legacy-only selected file
|
|
does not silently load; ordinary explicit imports remain supported.
|
|
- Representative tests demonstrate overlay precedence, CLI overrides,
|
|
declaration-relative paths, redacted inspection, and source-aware failures.
|
|
- Maintained examples validate, including the development overlay. LLM profile
|
|
loading and defaults remain unaffected.
|
|
- Tests prove the default transient retry policy, explicit zero, configured
|
|
counts, and negative-value rejection through the relevant configuration and
|
|
adapter boundaries. Use local HTTP test servers for eligible/noneligible
|
|
statuses, exhaustion, cancellation, and interaction with structural repair;
|
|
an injected LLM fake alone does not exercise built-in HTTP retries.
|
|
- Provider-neutral errors preserve underlying status and cancellation identity
|
|
without leaking credentials or upstream types.
|
|
- Observations are available for success and failure, preserve unknown versus
|
|
known-zero usage, avoid aggregate double-counting, and stay isolated under
|
|
concurrent completions. Successful retry recovery emits no warning.
|
|
- Representative scheduler/repair/retry checks preserve bounded execution and
|
|
cancellation; use existing coverage where it already proves the contract.
|
|
- Current documentation, help text, examples, and tests agree. Searches for old
|
|
vocabulary are reviewed semantically, excluding intentional historical and
|
|
LLM-profile references rather than enforcing a blanket text ban.
|
|
- Run focused `go test ./internal/core/config ./internal/cli ./internal/framework/llm ./internal/framework/pipeline`, followed by
|
|
`go test ./...`, `go vet ./...`, `go build ./...`, `go mod tidy -diff`, and
|
|
`git diff --check`, with `GOWORK=off`. Run race-enabled tests for the modified
|
|
adapter, scheduler, and observation integration. Validate both maintained configurations
|
|
and the complete configuration with `--config-overlay development`.
|
|
|
|
Use existing behavioral tests where possible, following
|
|
[Testing Policy](../../policy/testing.md). No live model calls or production API
|
|
credentials are needed to validate this migration.
|