Plan the PromptKit v0.8.0 upgrade
This commit is contained in:
571
docs/roadmap/implementation.md
Normal file
571
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,571 @@
|
||||
# PromptKit v0.8.0 Upgrade Implementation Plan
|
||||
|
||||
Status: Ready for implementation.
|
||||
|
||||
## Purpose And Authority
|
||||
|
||||
This plan translates the accepted
|
||||
[PromptKit v0.8.0 upgrade roadmap](promptkit-v0.8.0.md) into an ordered,
|
||||
decision-complete implementation procedure. The feature roadmap owns purpose,
|
||||
scope, policy, and the desired end state. This document owns implementation
|
||||
order, concrete work allocation, stage boundaries, and verification until the
|
||||
upgrade is complete.
|
||||
|
||||
The implementing agent must complete the stages in numerical order. Each stage
|
||||
is sized for one focused prompt handled by `gpt-5.6-terra` with high reasoning.
|
||||
Do not combine stages merely because adjacent work touches the same package.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
The following decisions are final for this implementation:
|
||||
|
||||
- upgrade directly from PromptKit `v0.5.0` to `v0.8.0`;
|
||||
- declare one corrective call in each embedded prompt through
|
||||
`repair_attempts: 1` rather than adding WeatherReporter repair logic;
|
||||
- keep the repair budget in the exact embedded prompt definition and add no
|
||||
global, per-report, CLI, profile, or operator configuration override;
|
||||
- advance all four exact prompt versions from `2.0.0` to `2.1.0`;
|
||||
- make `weather-light`, `weather-balanced`, and `weather-deep` minimal aliases
|
||||
of the corresponding PromptKit built-ins through `base_profile`;
|
||||
- retain the existing report-to-profile assignments and effective model
|
||||
ladder;
|
||||
- allow successfully inspected endpoint-only profiles to have an empty backend
|
||||
ID while continuing to require a nonblank model;
|
||||
- treat `APIKeyEnv` as an optional lookup source and reject only profiles that
|
||||
report `APIKeyRequired`, because WeatherReporter supplies no direct request
|
||||
credential;
|
||||
- support PromptKit's built-in `rakestrawhome-gemma-4-31b` profile without
|
||||
WeatherReporter-specific backend configuration;
|
||||
- expose provider HTTP status through a project-owned safe generation error,
|
||||
while writing provider code, type, and message only to explicit secure debug
|
||||
capture;
|
||||
- emit only `weatherreporter.comparison.v2`, with repair provenance, and do not
|
||||
preserve v1 guarded-replacement support; and
|
||||
- retain PromptKit dependency types inside the adapter and preserve all
|
||||
stateless execution, atomic publication, comparison independence, and
|
||||
disclosure invariants.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
For every stage:
|
||||
|
||||
- read `docs/development.md`, all files under `docs/policy/`, this plan, the
|
||||
feature roadmap, and the task-specific documents named by the stage;
|
||||
- inspect the current code and tests before editing; use the repository's code
|
||||
knowledge graph first for code discovery and fall back to text search for
|
||||
literals, assets, and documentation;
|
||||
- implement only the stage's scope and preserve unrelated user changes;
|
||||
- keep PromptKit/provider types, client construction, YAML parsing, repair
|
||||
mechanics, and provider transport inside the existing adapter boundary;
|
||||
- use deterministic, offline, credential-free tests and injected clients or
|
||||
synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local
|
||||
endpoint calls;
|
||||
- add tests at the narrowest stable owner identified by the testing policy and
|
||||
avoid copying PromptKit's internal test matrices;
|
||||
- update the canonical documentation owners listed for that stage in the same
|
||||
change as the implemented contract;
|
||||
- run `gofmt` on changed Go files, the stage's focused tests,
|
||||
`GOWORK=off go test -count=1 ./...`, and `git diff --check`; and
|
||||
- leave the repository passing before proceeding to the next stage.
|
||||
|
||||
Stages affecting concurrent comparison, cancellation, or secure debug
|
||||
filesystem work must also run the named focused packages with `-race`. Do not
|
||||
weaken an existing assertion solely to accommodate the new dependency. When a
|
||||
test encodes an intentionally changed contract, replace it with a behavioral
|
||||
assertion for the accepted policy.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
### Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline
|
||||
|
||||
Purpose: move to the tagged dependency and isolate compatibility changes before
|
||||
adopting new WeatherReporter behavior.
|
||||
|
||||
Work:
|
||||
|
||||
1. Update `go.mod` to require
|
||||
`gitea.maximumdirect.net/eric/promptkit v0.8.0` and refresh `go.sum` with
|
||||
`GOWORK=off go mod tidy`. Do not add `go.work`, `vendor`, or a `replace`
|
||||
directive and do not change WeatherReporter's Go version.
|
||||
2. Resolve any compile failures using PromptKit's public root package only.
|
||||
Keep all `Profile` and `OpenAICompatibleProfileConfig` literals keyed. Do not
|
||||
register the now-reserved `rakestrawhome` backend.
|
||||
3. Reconcile adapter tests that directly exercise PromptKit's changed optional
|
||||
credential behavior. A profile whose only credential metadata is
|
||||
`api_key_env` must reach an injected client when the environment value is
|
||||
absent; it must no longer expect PromptKit to return
|
||||
`ErrAPIKeyEnvMissing`. Do not change WeatherReporter's application preflight
|
||||
in this stage.
|
||||
4. Verify that every current embedded prompt, content file, schema, and fallback
|
||||
profile inspects under v0.8.0. Verify selected invalid local endpoints and
|
||||
malformed selected profile definitions still map to project-owned
|
||||
configuration or profile-load categories.
|
||||
5. Review the v0.6.0 compatibility corrections against supported
|
||||
WeatherReporter inputs: metadata-authoritative identities, exact contained
|
||||
`content_file` paths, regular embedded files, structurally valid endpoints,
|
||||
bounded JSON-compatible values, cancellation identity, and strict response
|
||||
framing. Add consumer tests only for a WeatherReporter boundary not already
|
||||
protected by PromptKit.
|
||||
|
||||
Do not enable profile inheritance or output repair yet. The expected result is
|
||||
the current WeatherReporter feature set running against PromptKit v0.8.0.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets
|
||||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
|
||||
```
|
||||
|
||||
### Stage 2: Adopt Profile Inheritance And Current Credential Routing
|
||||
|
||||
Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional
|
||||
credential semantics, and the Rakestrawhome built-in without changing the model
|
||||
ladder.
|
||||
|
||||
Work:
|
||||
|
||||
1. Replace the three embedded profile bodies with these exact leaf/base
|
||||
relationships and no duplicated execution settings:
|
||||
|
||||
| Leaf | Base |
|
||||
| --- | --- |
|
||||
| `weather-light` | `deepseek-4-flash` |
|
||||
| `weather-balanced` | `gemini-flash-latest` |
|
||||
| `weather-deep` | `claude-sonnet-latest` |
|
||||
|
||||
2. Update prompt-asset fixtures and tests to understand `base_profile`. Assert
|
||||
that all three leaf IDs remain selected identities and resolve to the same
|
||||
backend, model, timeout, service tier, and reasoning settings exposed by the
|
||||
current standalone definitions. Test relationships and effective behavior,
|
||||
not copied private constants beyond the intentional model-ladder contract.
|
||||
3. Preserve source precedence. Cover a standalone same-ID operator override, a
|
||||
derived operator override, a configured source that shadows a base ID, and
|
||||
selected missing-base, cyclic, malformed-base, and incomplete-target
|
||||
failures. Do not implement inheritance or merging in WeatherReporter; all
|
||||
resolution must remain PromptKit-owned.
|
||||
4. Change application profile preflight to accept a successful inspection with
|
||||
a nonblank model and an empty backend ID. Trust PromptKit inspection to have
|
||||
resolved either a backend or endpoint; do not add the endpoint to
|
||||
`promptexec.ProfileInspection` or ordinary provenance.
|
||||
5. Remove application-level environment lookup and rejection for a nonblank
|
||||
`APIKeyEnv`. Remove the now-unused `LookupEnv` fields and plumbing from
|
||||
prompt, batch, and comparison inspection requests. Continue rejecting
|
||||
`CredentialRequired`/`APIKeyRequired` before weather collection with the
|
||||
existing missing-credential category.
|
||||
6. Add an end-to-end offline regression proving the maintained endpoint-only
|
||||
`weather-light` example passes application inspection, retains an empty
|
||||
backend ID, and does not expose its endpoint.
|
||||
7. Prove `rakestrawhome-gemma-4-31b` can pass ordinary and comparison profile
|
||||
inspection through the existing adapter and reports the PromptKit
|
||||
`rakestrawhome` backend ID. Do not make a provider call or add
|
||||
Rakestrawhome-specific configuration.
|
||||
|
||||
Canonical documentation in this stage:
|
||||
|
||||
- update `docs/policy/architecture.md` so only direct-key-required profiles
|
||||
fail credential preflight and backend identity is optional for endpoint-only
|
||||
profiles;
|
||||
- update `docs/config.md` to distinguish same-ID source replacement from
|
||||
`base_profile` chain inheritance and to describe optional environment
|
||||
credentials;
|
||||
- update `docs/integrations/promptkit.md` for profile composition, parent
|
||||
lookup precedence, endpoint-only identity, optional credentials, and
|
||||
Rakestrawhome availability; and
|
||||
- update `docs/internal/promptkit-adapter.md` and focused app internals for the
|
||||
implemented inspection behavior.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app
|
||||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
|
||||
```
|
||||
|
||||
### Stage 3: Extend The Project-Owned Prompt Execution Contract
|
||||
|
||||
Purpose: establish dependency-neutral repair and structured-generation-error
|
||||
values before the adapter or application relies on them.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add `RepairAttempts int` to `promptexec.OutputContract`. It is the configured
|
||||
additional-call budget from the exact prompt contract.
|
||||
2. Add `RepairAttempts int` to `promptexec.Validation`. It is the number of
|
||||
corrective calls actually started for the completed result. Update
|
||||
`NewValidation` and every caller so construction is explicit; reject or
|
||||
normalize no values here because PromptKit owns output-contract validity.
|
||||
3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so
|
||||
repair values are retained without sharing mutable state.
|
||||
4. Add a project-owned immutable `promptexec.GenerationError` with unexported
|
||||
status and provider-detail fields plus safe accessors:
|
||||
|
||||
- `StatusCode() int`
|
||||
- `ProviderCode() string`
|
||||
- `ProviderType() string`
|
||||
- `ProviderMessage() string`
|
||||
- `Category() ErrorCategory`, always returning `Generation`
|
||||
- `Error()`, exposing only the WeatherReporter generation category/message
|
||||
and optional HTTP status
|
||||
- `GoString()`, returning the same safe representation
|
||||
- `Unwrap()`, preserving a project-owned `*promptexec.Error`
|
||||
|
||||
5. Provide one constructor used by adapters. Defensively normalize valid UTF-8
|
||||
and bound code/type to 256 Unicode code points and message to 4,096 Unicode
|
||||
code points, even though PromptKit already bounds its accessors. Do not
|
||||
expose fields through struct formatting, JSON tags, or exported mutable
|
||||
fields. Preserve the dependency cause only behind the project-owned error so
|
||||
`errors.Is`/`errors.As` identities remain available without entering error
|
||||
text.
|
||||
6. Add focused tests proving nil/zero safety, category and unwrap behavior,
|
||||
status-only ordinary formatting, `%#v` redaction, provider-detail bounds,
|
||||
and repair-value copying.
|
||||
|
||||
Do not import PromptKit from `internal/promptexec` and do not change CLI or
|
||||
artifact schemas in this stage.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/promptexec
|
||||
```
|
||||
|
||||
### Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter
|
||||
|
||||
Purpose: make the adapter faithfully translate v0.8.0 preparation, execution,
|
||||
validation, usage, and failure values into the Stage 3 contract.
|
||||
|
||||
Work:
|
||||
|
||||
1. Map `promptkit.OutputContract.RepairAttempts` in prompt inspection and
|
||||
prepared-execution details. Map
|
||||
`promptkit.ValidationResult.RepairAttempts` in completed execution.
|
||||
2. Preserve PromptKit's cumulative usage exactly as reported across the initial
|
||||
call and every completed correction. Continue returning only the final raw
|
||||
candidate and final validation result, subject to WeatherReporter's 64 KiB
|
||||
generated-output bound.
|
||||
3. In adapter error classification, retain cancellation, deadline, and capacity
|
||||
precedence. Before the generic `ErrLLMGenerate` branch, use `errors.As` for
|
||||
`*promptkit.GenerationError` and construct the project-owned
|
||||
`promptexec.GenerationError` with status, code, type, message, and hidden
|
||||
cause. Initial and corrective generation failures use the same mapping.
|
||||
4. Extend the injected adapter client used by tests so it can return an ordered
|
||||
sequence of responses or errors and record each request safely.
|
||||
5. Use a synthetic PromptKit prompt with JSON Schema validation and
|
||||
`repair_attempts: 1` to cover:
|
||||
|
||||
- first-pass valid output with zero corrections;
|
||||
- explicitly empty or invalid output followed by valid corrected output;
|
||||
- one-attempt exhaustion returning a final failed validation result rather
|
||||
than an operational error;
|
||||
- a non-2xx-style `GenerationError` during correction;
|
||||
- cumulative token usage and actual repair count; and
|
||||
- the same prepared prompt/profile identity across the corrective flow.
|
||||
|
||||
6. Keep these tests at the adapter boundary. Do not assert PromptKit's private
|
||||
corrective-message wording or reconstruct its internal repair algorithm.
|
||||
|
||||
Canonical documentation in this stage: update
|
||||
`docs/internal/promptkit-adapter.md` for the repair/result/error mappings. Do
|
||||
not yet claim that embedded WeatherReporter prompts enable repair.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec
|
||||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
|
||||
```
|
||||
|
||||
### Stage 5: Carry Repair Provenance Through Application Workflows
|
||||
|
||||
Purpose: make application orchestration understand configured and actual repair
|
||||
counts before changing the embedded prompt policy.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add an expected generated-text repair budget to `report.Definition` and set
|
||||
it explicitly to zero for all four current `2.0.0` definitions in this
|
||||
stage. Include it in report-definition validation and retained contract
|
||||
tests.
|
||||
2. Extend exact prompt preflight so format, validation mode, schema path, and
|
||||
repair budget must all match the resolved report definition. Extend
|
||||
preparation and completion provenance checks to require the same repair
|
||||
budget across inspection and the opaque prepared snapshot.
|
||||
3. Add `RepairAttempts *int` to application outcomes where execution may fail
|
||||
before validation exists. Set it to a fresh pointer immediately after a
|
||||
non-nil completed execution is returned, before WeatherReporter's secondary
|
||||
generated-text validation. A pointer is required so completed first-pass
|
||||
zero is distinguishable from unavailable provenance.
|
||||
4. Carry independent copies through `ReportResult`, `BatchReportResult`, batch
|
||||
conversion, comparison execution's internal outcome, and relevant test
|
||||
fakes. Do not expose the new value in CLI or comparison JSON yet.
|
||||
5. Preserve the actual count on PromptKit validation rejection and on later
|
||||
WeatherReporter generated-text or render failures. Leave it unavailable on
|
||||
preparation, capacity, cancellation, deadline, and generation errors that
|
||||
return no completed PromptKit result.
|
||||
6. Add focused tests for provenance mismatch, completed zero, completed
|
||||
positive, validation rejection, later local validation failure, early
|
||||
operational failure, batch copying, and independent concurrent profile
|
||||
outcomes.
|
||||
|
||||
Canonical documentation in this stage: update the focused prepared-report and
|
||||
app-orchestration internals to describe configured versus actual repair
|
||||
provenance. Current public documents should continue to report the embedded
|
||||
budget as zero until Stage 6.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli
|
||||
GOWORK=off go test -race -count=1 ./internal/app
|
||||
```
|
||||
|
||||
### Stage 6: Activate One Repair And Expose Ordinary Result Provenance
|
||||
|
||||
Purpose: switch the operational prompts to the accepted one-correction policy
|
||||
and make ordinary generate/run/batch output report what occurred.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add `repair_attempts: 1` to the output contract of all four embedded prompt
|
||||
definitions and change each exact prompt version from `2.0.0` to `2.1.0`.
|
||||
Do not change prompt text or generated-text schemas solely for this upgrade.
|
||||
2. Change all four report registry definitions to exact prompt version `2.1.0`
|
||||
and expected repair budget one. Update exact-version fixtures and assertions
|
||||
throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests
|
||||
that classify `repair_attempts` as a retired setting and replace them with
|
||||
an exact one-attempt contract assertion.
|
||||
3. Add `repairAttempts` to successful and failed generate and batch JSON result
|
||||
shapes through the Stage 5 pointers. Emit integer zero for a completed
|
||||
first-pass result, a positive integer for a completed repaired result, and
|
||||
omit the field when no completed validation made it available.
|
||||
4. Keep the existing `validationStatus` and failure categories authoritative.
|
||||
A repaired valid result proceeds normally. Repair exhaustion remains
|
||||
`validation_rejected`, publishes no report for that profile, and retains the
|
||||
actual attempt count.
|
||||
5. Add representative offline assembled tests proving first-pass success,
|
||||
repaired success, exhaustion, explicit empty initial content, and batch
|
||||
result propagation. Reuse the real PromptKit adapter with an injected
|
||||
sequence client for at least one end-to-end repaired execution; use the
|
||||
existing app fake at other boundaries where lower-level repair is already
|
||||
covered.
|
||||
6. Confirm no application loop, provider retry, profile fallback, or
|
||||
request-level `OutputContract` override was introduced.
|
||||
|
||||
Canonical documentation in this stage:
|
||||
|
||||
- update `docs/policy/architecture.md` with PromptKit-owned bounded repair and
|
||||
failed-exhaustion invariants;
|
||||
- update `docs/integrations/promptkit.md` with exact prompt version `2.1.0`, one
|
||||
configured repair, actual-count semantics, cumulative usage, explicit-empty
|
||||
handling, and the distinction from operational retries;
|
||||
- update `docs/cli.md` for generate and batch `repairAttempts` fields;
|
||||
- update `docs/internal/report-registry.md`, prepared-report internals, and app
|
||||
orchestration internals for exact version and repair flow; and
|
||||
- keep configuration documentation unchanged because no repair setting is
|
||||
added.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
|
||||
```
|
||||
|
||||
### Stage 7: Migrate Comparison Bundles To v2
|
||||
|
||||
Purpose: preserve repair activity in the profile-evaluation artifact and make
|
||||
the strict durable schema change explicit.
|
||||
|
||||
Work:
|
||||
|
||||
1. Change `comparison.SchemaVersion` to
|
||||
`weatherreporter.comparison.v2`. Emit and recognize v2 only; do not retain a
|
||||
v1 parser or guarded-replacement compatibility path.
|
||||
2. Add `RepairAttempts *int` to each application comparison profile result,
|
||||
CLI comparison profile summary, and durable `comparison.Result`. Propagate a
|
||||
fresh copy from Stage 5's execution outcome.
|
||||
3. Place `repairAttempts` immediately after `validationStatus` in the canonical
|
||||
result-object JSON field order. Encode zero for completed first-pass
|
||||
validation, a positive integer for completed correction, and omit it only
|
||||
when no completed validation exists.
|
||||
4. Tighten manifest invariants: every non-nil repair count is non-negative; a
|
||||
successful result must have `validationStatus: "passed"` and a non-nil
|
||||
repair count; a failed result with a completed validation status must also
|
||||
have a non-nil count; and an early operational failure may omit both.
|
||||
5. Update the strict token-level JSON recognizer to accept only the canonical
|
||||
`repairAttempts` field at its correct object level, reject duplicate,
|
||||
unknown, negative, fractional, string, overflow, and malformed values, and
|
||||
continue rejecting v1 as an unsupported current bundle.
|
||||
6. Update manifest construction, cloning, validation, exact serialization
|
||||
tests, guarded replacement tests, malicious bundle tests, partial-success
|
||||
tests, and CLI comparison summaries. Preserve flat layout, result ordering,
|
||||
hashes, atomic publication, cancellation safety, and no Distributor calls.
|
||||
7. Cover concurrent peers where one succeeds first-pass, one repairs, one
|
||||
exhausts, and one fails operationally. The counts must remain attached to
|
||||
the selected profile positions without races or cross-contamination.
|
||||
|
||||
Canonical documentation in this stage:
|
||||
|
||||
- replace the v1 contract in `docs/integrations/comparison-bundle.md` with v2,
|
||||
including exact field order, presence rules, and the lack of v1 replacement
|
||||
compatibility;
|
||||
- update `docs/cli.md` for comparison `repairAttempts`;
|
||||
- update `docs/operations.md` to tell operators to move or remove an existing
|
||||
v1 bundle before replacing at the same destination; and
|
||||
- update comparison execution/publication internals and architecture policy as
|
||||
needed for the current-only version invariant.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli
|
||||
GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app
|
||||
```
|
||||
|
||||
### Stage 8: Add Secure Provider-Failure Debug Capture
|
||||
|
||||
Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the
|
||||
existing explicit secure debug boundary while keeping ordinary errors safe.
|
||||
|
||||
Work:
|
||||
|
||||
1. Extend prompt preparation debug output with configured
|
||||
`repairAttempts` and advance its schema identifier from
|
||||
`weatherreporter.prompt_preparation_debug.v2` to
|
||||
`weatherreporter.prompt_preparation_debug.v3`.
|
||||
2. Extend execution validation debug output with actual `repairAttempts` and
|
||||
advance its schema identifier from
|
||||
`weatherreporter.prompt_execution_debug.v2` to
|
||||
`weatherreporter.prompt_execution_debug.v3`. Retain cumulative token usage.
|
||||
3. Add a dedicated `failure.json` artifact with schema identifier
|
||||
`weatherreporter.prompt_failure_debug.v1`. Its canonical fields are:
|
||||
|
||||
- top level: `schemaVersion`, `reportId`, `validDate`, `runId`, `failure`;
|
||||
- failure object: `category`, `statusCode`, `providerCode`, `providerType`,
|
||||
`providerMessage`;
|
||||
- omit absent provider fields and zero status; and
|
||||
- never include the raw provider body, headers, endpoint, credentials,
|
||||
request, schema, rendered prompt, or generated candidate.
|
||||
|
||||
4. Add `PromptDebugWriter.WriteFailure` using the existing handle-relative
|
||||
secure run directory, `0700` directory and `0600` file modes, canonical JSON
|
||||
encoding, and no-follow/atomic replacement behavior. Disabled writers must
|
||||
perform no filesystem work.
|
||||
5. When execution returns an error, use `errors.As` only against the
|
||||
project-owned `*promptexec.GenerationError`. If explicit debug capture is
|
||||
enabled, write `failure.json` using that profile's existing debug reference.
|
||||
This applies equally to initial and corrective provider failures and keeps
|
||||
comparison profile directories isolated.
|
||||
6. If failure-debug writing also fails, retain the generation failure as the
|
||||
primary categorized error and join the safe debug-write failure rather than
|
||||
replacing or hiding the provider failure. Never place provider code, type,
|
||||
or message in the joined error text.
|
||||
7. Ordinary generate, batch, and comparison errors should gain only the safe
|
||||
HTTP status already rendered by `promptexec.GenerationError.Error`; do not
|
||||
add provider detail fields to CLI summaries, comparison manifests, logs, or
|
||||
Distributor requests.
|
||||
8. Add adversarial tests for formatter redaction, malicious provider strings,
|
||||
JSON escaping, bounds, absent fields, file modes, symlink/path attacks,
|
||||
write failure, cancellation identity, initial versus corrective failures,
|
||||
and concurrent comparison captures.
|
||||
|
||||
Canonical documentation in this stage:
|
||||
|
||||
- update `docs/operations.md` with the three debug artifact versions,
|
||||
`failure.json`, sensitivity, permissions, and retention;
|
||||
- update `docs/integrations/promptkit.md` with ordinary status-only disclosure
|
||||
and debug-only provider detail;
|
||||
- update prompt-debug, PromptKit-adapter, and app-orchestration internals; and
|
||||
- ensure `docs/policy/architecture.md` explicitly prohibits provider-controlled
|
||||
diagnostics from ordinary outputs.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||
GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app
|
||||
```
|
||||
|
||||
### Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit
|
||||
|
||||
Purpose: verify the complete end state as one coherent WeatherReporter feature
|
||||
and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile,
|
||||
repair, or debug claims.
|
||||
|
||||
Work:
|
||||
|
||||
1. Re-read the feature roadmap, `docs/development.md`, every policy document,
|
||||
and every canonical document changed by Stages 1-8. Reconcile them against
|
||||
executable behavior and remove duplicated or stale definitions. Keep
|
||||
unimplemented future ideas in `docs/roadmap/future.md`, not current-state
|
||||
documents.
|
||||
2. Search code, embedded assets, examples, tests, and documentation for stale
|
||||
contractual literals and review every occurrence of:
|
||||
|
||||
- PromptKit `v0.5.0`, `v0.6.0`, and `v0.7.0` as an active dependency claim;
|
||||
- prompt version `2.0.0`;
|
||||
- `weatherreporter.comparison.v1`;
|
||||
- prompt debug schema v2 identifiers;
|
||||
- claims that profile fields never inherit;
|
||||
- claims that every `APIKeyEnv` must be populated;
|
||||
- claims that backend ID is always required;
|
||||
- claims that repair is disabled or `repair_attempts` is retired; and
|
||||
- provider detail in ordinary output.
|
||||
|
||||
Historical release documents may retain accurate historical literals.
|
||||
3. Verify canonical ownership:
|
||||
|
||||
- architecture owns invariants and boundaries;
|
||||
- config owns operator profile and credential behavior, but no repair field;
|
||||
- PromptKit integration owns logical prompt/profile/output contracts;
|
||||
- CLI owns result fields;
|
||||
- operations owns explicit debug handling and old comparison-bundle cleanup;
|
||||
- comparison integration owns the complete v2 manifest; and
|
||||
- internal documents own implementation flow without duplicating the public
|
||||
references.
|
||||
|
||||
4. Verify maintained examples remain valid, secret-free, and tested. The local
|
||||
`weather-light` example remains a standalone endpoint-only profile rather
|
||||
than inheriting an OpenRouter backend it cannot clear.
|
||||
5. Review the complete diff for architecture leakage. Production packages
|
||||
outside `internal/adapters/promptkit` must not import PromptKit; no
|
||||
application repair loop, provider client, raw provider diagnostic, profile
|
||||
YAML parser, or durable application state may have appeared.
|
||||
6. Review tests under the testing policy. Keep consumer contract and regression
|
||||
coverage, remove accidental duplication of upstream implementation tests,
|
||||
and ensure every default test is offline and repeatable.
|
||||
7. Run the complete validation set:
|
||||
|
||||
```sh
|
||||
gofmt -w <all changed Go files>
|
||||
GOWORK=off go test -count=1 ./...
|
||||
GOWORK=off go test -race -count=1 ./...
|
||||
GOWORK=off go vet ./...
|
||||
GOWORK=off go build ./...
|
||||
GOWORK=off go mod tidy -diff
|
||||
go run ./cmd/weatherreporter --help
|
||||
go run ./cmd/weatherreporter compare --help
|
||||
test -z "$(git ls-files go.work go.work.sum)"
|
||||
test ! -e vendor
|
||||
git diff --check
|
||||
```
|
||||
|
||||
8. Confirm `go.mod` has no `replace`, the resolved PromptKit module is exactly
|
||||
v0.8.0, and no live credential or provider call occurred during validation.
|
||||
9. After every check passes, update this plan's status to Completed and add a
|
||||
concise completion note listing the implemented stages. Do not delete either
|
||||
roadmap until the maintainer has reviewed the implementation. Do not create
|
||||
a release document or tag; release preparation remains a separate maintainer
|
||||
action once a version is selected.
|
||||
|
||||
## Completion Standard
|
||||
|
||||
The implementation is complete only when all nine stages pass their focused
|
||||
and repository-wide checks, all locked decisions are observable in code and
|
||||
canonical documentation, and the feature roadmap's completion criteria are
|
||||
satisfied. Passing compilation alone is insufficient. The final state must
|
||||
demonstrate repaired success, repair exhaustion, comparison provenance,
|
||||
endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome
|
||||
inspection, safe ordinary provider failures, secure debug-only detail, and
|
||||
unchanged publication and concurrency invariants.
|
||||
488
docs/roadmap/promptkit-v0.8.0.md
Normal file
488
docs/roadmap/promptkit-v0.8.0.md
Normal file
@@ -0,0 +1,488 @@
|
||||
# PromptKit v0.8.0 Upgrade Roadmap
|
||||
|
||||
Status: Accepted feature direction; implementation has not started.
|
||||
|
||||
## Purpose
|
||||
|
||||
WeatherReporter should upgrade its PromptKit dependency from `v0.5.0` to
|
||||
`v0.8.0` and deliberately adopt the useful consumer-facing capabilities added
|
||||
in `v0.6.0`, `v0.7.0`, and `v0.8.0`. The upgrade should improve output-contract
|
||||
reliability, profile composition, local and alternate endpoint support, and
|
||||
provider-failure diagnosis without moving PromptKit responsibilities into
|
||||
WeatherReporter or weakening the application's stateless and security
|
||||
boundaries.
|
||||
|
||||
This roadmap defines the intended scope, policy, and end state. The
|
||||
[implementation plan](implementation.md) owns the procedure for reaching that
|
||||
state.
|
||||
|
||||
## User Intent
|
||||
|
||||
The upgrade is intended to:
|
||||
|
||||
- use PromptKit's bounded output repair to recover from occasional malformed
|
||||
structured weather prose;
|
||||
- keep WeatherReporter's domain profile IDs stable while inheriting maintained
|
||||
PromptKit model definitions;
|
||||
- make PromptKit's additional built-in backend and profile available for
|
||||
explicit generation and profile comparisons;
|
||||
- support unauthenticated or optionally authenticated OpenAI-compatible
|
||||
endpoints without inventing a WeatherReporter transport layer;
|
||||
- make provider HTTP failures more actionable under an explicit
|
||||
WeatherReporter disclosure policy; and
|
||||
- receive PromptKit's intervening correctness, safety, cancellation, resource,
|
||||
and efficiency improvements as part of one tested dependency upgrade.
|
||||
|
||||
The model ladder and report assignments do not change as part of this work:
|
||||
Hourly continues to select `weather-light`; Daily, Today, and Tomorrow continue
|
||||
to select `weather-balanced`; and `weather-deep` remains available for explicit
|
||||
selection. This upgrade does not promote the new Rakestrawhome profile into
|
||||
that default ladder.
|
||||
|
||||
## Current State
|
||||
|
||||
WeatherReporter currently depends on
|
||||
`gitea.maximumdirect.net/eric/promptkit` at `v0.5.0`. The PromptKit adapter
|
||||
supplies embedded prompts, JSON Schemas, and
|
||||
application-fallback profiles, plus an optional configured profile source and
|
||||
the conventional local backend.
|
||||
|
||||
The four generated-text prompts are exact version `2.0.0` JSON Schema prompts.
|
||||
They omit `repair_attempts`, so execution is single-pass. The project-owned
|
||||
`promptexec.OutputContract` and validation result also omit repair budgets and
|
||||
actual repair counts.
|
||||
|
||||
The three embedded WeatherReporter profiles duplicate the effective fields of
|
||||
these PromptKit built-ins:
|
||||
|
||||
| WeatherReporter profile | PromptKit built-in with the same target |
|
||||
| --- | --- |
|
||||
| `weather-light` | `deepseek-4-flash` |
|
||||
| `weather-balanced` | `gemini-flash-latest` |
|
||||
| `weather-deep` | `claude-sonnet-latest` |
|
||||
|
||||
WeatherReporter preflights any nonblank `api_key_env` as a required credential,
|
||||
even though PromptKit v0.7.0 distinguishes an optional environment source from
|
||||
an explicit `APIKeyRequired` target. Provider generation failures are reduced
|
||||
to WeatherReporter's safe `generation` category; the PromptKit dependency error
|
||||
is retained as a hidden cause, but its structured HTTP status and provider
|
||||
diagnostics are not mapped into project-owned values.
|
||||
|
||||
The maintained `weather-light` local override is an endpoint-only profile, and
|
||||
the configuration contract says endpoint-only profiles are supported. PromptKit
|
||||
inspection correctly reports no backend ID for that form, but WeatherReporter
|
||||
application preflight currently requires both a nonblank backend and model.
|
||||
That mismatch prevents the documented example from reaching generation and
|
||||
should be corrected as part of adopting the current PromptKit target contract.
|
||||
|
||||
## Upstream Release Assessment
|
||||
|
||||
### PromptKit v0.6.0
|
||||
|
||||
`v0.6.0` adds no public declarations, but it is a material compatibility and
|
||||
safety release. It centralizes execution-setting, output-contract, endpoint,
|
||||
and JSON-compatible-value validation; makes YAML metadata authoritative for
|
||||
prompt and profile identity; hardens `content_file` containment and regular-file
|
||||
requirements; validates OpenAI-compatible endpoints structurally; bounds JSON
|
||||
trees and successful provider bodies; requires exactly one JSON value in
|
||||
provider responses; preserves cancellation and transport error identities; and
|
||||
reuses schema and rendered-artifact work within an operation.
|
||||
|
||||
WeatherReporter should receive these improvements directly from the dependency
|
||||
and audit its own supported assets and configuration paths against the stricter
|
||||
contracts. It should not duplicate PromptKit's internal validators or tests.
|
||||
The existing embedded prompt paths, inline data-package input, generated-output
|
||||
limit, and adapter boundary remain conceptually correct.
|
||||
|
||||
### PromptKit v0.7.0
|
||||
|
||||
`v0.7.0` adds four potentially useful consumer features:
|
||||
|
||||
- linear, cycle-safe profile inheritance through `base_profile` and
|
||||
`Profile.BaseProfileID`;
|
||||
- the built-in `rakestrawhome` backend and
|
||||
`rakestrawhome-gemma-4-31b` profile;
|
||||
- optional API-key environment sources, with `APIKeyRequired` reserved for an
|
||||
explicit local credential requirement; and
|
||||
- bounded structured `GenerationError` details for non-2xx responses from the
|
||||
built-in OpenAI-compatible client.
|
||||
|
||||
WeatherReporter has no manual `rakestrawhome` registration and uses keyed
|
||||
PromptKit profile literals, so the two source-compatibility hazards called out
|
||||
by the release do not require migration shims. The profile, credential, and
|
||||
error features do require deliberate application-policy choices described
|
||||
below.
|
||||
|
||||
### PromptKit v0.8.0
|
||||
|
||||
`v0.8.0` activates the existing output-contract repair budget. A positive
|
||||
`repair_attempts` value authorizes up to that many corrective model calls after
|
||||
eligible `basic`, `json`, or `json_schema` validation failures. The supported
|
||||
budget is zero through three. Repairs preserve the original rendered
|
||||
conversation, effective target, session, structured-output contract, and
|
||||
backend capacity policy. The final result reports cumulative token usage and
|
||||
the number of corrective calls actually made.
|
||||
|
||||
Repair exhaustion is a completed generation with failed validation, not an
|
||||
operational error. WeatherReporter's existing policy should continue to reject
|
||||
that result and publish no report for that profile. Explicit empty provider
|
||||
content now reaches output validation; for WeatherReporter's JSON Schema
|
||||
prompts it is therefore eligible for repair rather than being misclassified as
|
||||
a malformed provider envelope.
|
||||
|
||||
## Desired End State
|
||||
|
||||
WeatherReporter builds and tests against PromptKit `v0.8.0` with no workspace,
|
||||
vendor, or module replacement dependency. Its public behavior remains
|
||||
stateless, its PromptKit dependency types remain confined to the adapter, and
|
||||
its ordinary summaries and logs remain safe.
|
||||
|
||||
The completed integration:
|
||||
|
||||
- benefits from the v0.6.0 safety and efficiency corrections;
|
||||
- composes WeatherReporter domain profiles from PromptKit's maintained built-in
|
||||
profiles while preserving WeatherReporter-owned leaf IDs and operator
|
||||
override precedence;
|
||||
- accepts a successfully inspected endpoint-only profile with a nonblank model
|
||||
even though it has no logical backend ID;
|
||||
- permits explicit use of PromptKit's Rakestrawhome profile without custom
|
||||
backend wiring;
|
||||
- applies an accepted bounded-repair policy to every operational structured
|
||||
prompt;
|
||||
- validates repair configuration during preflight and records actual repair
|
||||
activity in project-owned result values;
|
||||
- retains PromptKit's cumulative usage accounting in explicit debug output;
|
||||
- distinguishes safe provider HTTP status from potentially sensitive provider
|
||||
diagnostics; and
|
||||
- documents the changed profile, credential, repair, comparison, debug, and
|
||||
failure contracts in their canonical owners.
|
||||
|
||||
## Dependency And Compatibility Policy
|
||||
|
||||
The module requirement should move directly from `v0.5.0` to `v0.8.0`, followed
|
||||
by a clean module tidy. WeatherReporter already requires Go 1.26 while PromptKit
|
||||
`v0.8.0` requires Go 1.25.5, so no Go version change is needed for this upgrade.
|
||||
|
||||
Consumer validation must cover the paths called out by PromptKit v0.6.0:
|
||||
|
||||
- every embedded prompt, content file, schema, and fallback profile inspects
|
||||
through PromptKit `v0.8.0`;
|
||||
- configured single-file and directory profile sources retain their lazy,
|
||||
metadata-authoritative identity and precedence behavior;
|
||||
- malformed selected profiles and invalid local endpoints retain actionable
|
||||
WeatherReporter categories;
|
||||
- the inline YAML data package and prepared-execution path remain within the
|
||||
new JSON and response bounds; and
|
||||
- cancellation, deadline, and backend-capacity identities still cross the
|
||||
adapter correctly.
|
||||
|
||||
PromptKit owns its 16 MiB successful transport-response bound and JSON framing.
|
||||
WeatherReporter retains its stricter 64 KiB generated-text acceptance bound.
|
||||
The consumer suite should protect that relationship without reproducing
|
||||
PromptKit's lower-level transport matrix.
|
||||
|
||||
## Domain Profile Composition
|
||||
|
||||
The embedded profiles should become application-owned aliases:
|
||||
|
||||
```yaml
|
||||
id: weather-light
|
||||
base_profile: deepseek-4-flash
|
||||
```
|
||||
|
||||
```yaml
|
||||
id: weather-balanced
|
||||
base_profile: gemini-flash-latest
|
||||
```
|
||||
|
||||
```yaml
|
||||
id: weather-deep
|
||||
base_profile: claude-sonnet-latest
|
||||
```
|
||||
|
||||
The effective backend, model, timeout, service tier, and reasoning settings
|
||||
must initially remain identical to the current WeatherReporter definitions.
|
||||
The selected leaf remains the durable logical profile identity even though its
|
||||
effective target is inherited.
|
||||
|
||||
Profile source precedence remains:
|
||||
|
||||
1. explicit in-memory profiles used by tests or embedding consumers;
|
||||
2. the configured `profile_file` or `profile_dir` source;
|
||||
3. WeatherReporter's embedded fallback catalog; and
|
||||
4. PromptKit's built-in catalog.
|
||||
|
||||
Sources still do not merge definitions of the same ID. Once a selected
|
||||
definition names `base_profile`, however, each parent ID is resolved through
|
||||
that same precedence order and the resulting linear chain is merged from root
|
||||
to leaf according to PromptKit's inheritance contract. Documentation must make
|
||||
that distinction explicit. A malformed leaf, missing or malformed base, cycle,
|
||||
overlong chain, or incomplete resolved target fails profile inspection before
|
||||
weather collection.
|
||||
|
||||
An operator may continue to replace `weather-light`, `weather-balanced`, or
|
||||
`weather-deep` with a standalone definition. An operator may also define a
|
||||
derived replacement. The maintained local endpoint example should remain
|
||||
standalone because PromptKit profile inheritance has no clearing syntax: using
|
||||
an OpenRouter base would retain its backend identity and capacity policy even
|
||||
when the child replaces the endpoint.
|
||||
|
||||
WeatherReporter should treat the adapter's successful profile inspection as
|
||||
authoritative that PromptKit resolved a usable route. A nonblank model remains
|
||||
required, but backend ID is optional for an endpoint-only profile and should be
|
||||
omitted from safe provenance where unavailable. WeatherReporter still must not
|
||||
surface the endpoint outside explicit debug capture. This aligns application
|
||||
preflight with PromptKit and with the existing CLI, comparison, and debug value
|
||||
shapes, all of which already permit an absent backend identity.
|
||||
|
||||
## Rakestrawhome Availability
|
||||
|
||||
The reserved `rakestrawhome` backend and built-in
|
||||
`rakestrawhome-gemma-4-31b` profile should be supported automatically through
|
||||
ordinary PromptKit selection. Operators may choose that profile with the
|
||||
existing global profile setting or as one entry in `compare`, and PromptKit's
|
||||
backend capacity policy remains authoritative.
|
||||
|
||||
WeatherReporter should not register, wrap, or duplicate the backend or profile,
|
||||
and should not add a Rakestrawhome-specific configuration field. Its canonical
|
||||
PromptKit integration documentation should link to PromptKit for the current
|
||||
built-in catalog and credential contract rather than copying volatile endpoint
|
||||
or capacity values. Offline inspection coverage should prove that the built-in
|
||||
profile crosses the WeatherReporter adapter with the expected logical backend
|
||||
identity.
|
||||
|
||||
## Bounded Structured-Output Repair
|
||||
|
||||
The accepted repair budget belongs to the exact PromptKit output contract, not
|
||||
to a WeatherReporter retry loop. PromptKit alone should construct corrective
|
||||
messages, perform additional calls, enforce the budget, aggregate usage, and
|
||||
coordinate backend capacity. WeatherReporter must not retry provider failures,
|
||||
switch profiles, or layer another repair mechanism around `RunPrepared`.
|
||||
|
||||
All four embedded prompt definitions should declare `repair_attempts: 1`.
|
||||
Because this changes prompt
|
||||
execution behavior, latency, cost, hash, and provenance, each definition and
|
||||
its report-registry binding should advance from exact version `2.0.0` to
|
||||
`2.1.0`. Prompt text and generated-text schemas do not need to change solely
|
||||
for this feature.
|
||||
|
||||
The embedded prompt definition is the per-report pipeline policy owner. This
|
||||
upgrade should not add a global or per-report operator configuration field for
|
||||
repair attempts and should not construct a request-level replacement output
|
||||
contract. A future pipeline may select another budget only through a deliberate
|
||||
prompt-definition and exact-version change.
|
||||
|
||||
The project-owned PromptKit boundary should retain:
|
||||
|
||||
- the configured repair budget in prompt inspection and preparation output
|
||||
contracts;
|
||||
- the number of corrective calls actually made in completed validation;
|
||||
- cumulative PromptKit token usage across initial and corrective calls; and
|
||||
- the final candidate and final validation result only, consistent with the
|
||||
PromptKit contract.
|
||||
|
||||
Prompt inspection and preparation provenance must require the repair budget to
|
||||
match the exact expected prompt definition just as they currently require the
|
||||
format, validation mode, and schema path to match. A zero-attempt successful
|
||||
result is normal when the first candidate passes. A repair-exhausted result
|
||||
continues through WeatherReporter's ordinary `validation_rejected` failure
|
||||
path, and an operational or generation failure during correction remains that
|
||||
profile's ordinary operational failure.
|
||||
|
||||
For concurrent comparison, every profile should use the same prompt repair
|
||||
budget. A corrective call remains part of that profile's one prepared
|
||||
execution and uses PromptKit's existing backend capacity pool. One profile's
|
||||
repair or failure must not cancel independent peers.
|
||||
|
||||
## Repair Observability And Comparison Contract
|
||||
|
||||
The actual repair count is safe operational provenance and should be visible
|
||||
where WeatherReporter already reports completed validation. Generation, batch,
|
||||
and comparison action summaries should expose it without exposing candidates,
|
||||
schemas, or diagnostics. Explicit execution debug output should add it to the
|
||||
validation object alongside PromptKit's already mapped cumulative usage.
|
||||
|
||||
Profile comparison needs this value in `comparison.json`: a successful result
|
||||
that required correction is materially different from a first-pass success
|
||||
when evaluating model reliability, latency, and cost. The manifest should
|
||||
therefore advance to `weatherreporter.comparison.v2` and add a non-negative
|
||||
`repairAttempts` field to each result. The field is zero when no corrective
|
||||
call began, including ordinary first-pass success. A failure carries the count
|
||||
when PromptKit returned a completed validation result; it is omitted only when
|
||||
execution failed before a completed validation result made the value known.
|
||||
|
||||
The v2 manifest should remain flat, strict, deterministic, and atomically
|
||||
published. WeatherReporter does not need to preserve v1 replacement
|
||||
compatibility: comparison bundles are operator-owned development outputs, and
|
||||
the current integration contract intentionally recognizes only its current
|
||||
schema. The release notes and comparison documentation must call out the
|
||||
version change so an operator can remove or relocate an older bundle before
|
||||
using guarded replacement at the same destination.
|
||||
|
||||
## Credential Semantics
|
||||
|
||||
PromptKit v0.7.0 treats `APIKeyEnv` as an optional lookup source. If the
|
||||
environment variable is absent or blank and no direct credential is supplied,
|
||||
the built-in client omits `Authorization` and lets the endpoint respond.
|
||||
`APIKeyRequired` is the distinct signal that a usable credential must be
|
||||
provided locally.
|
||||
|
||||
WeatherReporter cannot supply PromptKit's request-scoped direct API-key value,
|
||||
so a profile reporting `APIKeyRequired` remains unsupported and must fail
|
||||
before weather collection. WeatherReporter should not require a nonblank value
|
||||
for an optional `APIKeyEnv` during application preflight. The built-in client
|
||||
should omit `Authorization` when that source is unavailable and let the
|
||||
endpoint return any authentication failure through the ordinary structured
|
||||
generation-error path.
|
||||
|
||||
## Structured Generation Failures
|
||||
|
||||
PromptKit v0.7.0's `GenerationError` can report a provider HTTP status plus
|
||||
bounded provider code, type, and message. WeatherReporter should consume that
|
||||
type only inside the PromptKit adapter and map any adopted fields into a
|
||||
project-owned immutable error. PromptKit dependency types must not become app
|
||||
or CLI contracts.
|
||||
|
||||
HTTP status is safe enough for ordinary diagnostics. Provider code, type, and
|
||||
message remain untrusted and may contain request or schema fragments. They
|
||||
must never enter ordinary errors, action summaries, comparison manifests,
|
||||
logs, generated reports, or Distributor payloads. The full structured
|
||||
diagnostic belongs only in an explicitly requested secure `--llm-debug-dir`
|
||||
`failure.json` artifact. That artifact may contain PromptKit's normalized
|
||||
bounded fields but never the raw provider body, headers, endpoint, credentials,
|
||||
or reconstructed request.
|
||||
|
||||
Initial-call and corrective-call non-2xx responses should follow the same
|
||||
mapping. Cancellation and deadline categories continue to take precedence over
|
||||
provider classification where PromptKit preserves those identities.
|
||||
|
||||
## Testing Policy
|
||||
|
||||
The default suite must remain deterministic, offline, and credential-free.
|
||||
Use injected PromptKit clients and synthetic embedded or temporary assets for
|
||||
consumer behavior; do not call OpenRouter, Rakestrawhome, or a local endpoint.
|
||||
|
||||
Risk-based coverage should include:
|
||||
|
||||
- all embedded prompts and inherited domain profiles inspecting successfully
|
||||
under PromptKit `v0.8.0`;
|
||||
- unchanged effective targets and report-to-profile assignments after the
|
||||
alias refactor;
|
||||
- external standalone and derived profile precedence, plus selected missing,
|
||||
cyclic, and malformed-base failures at the WeatherReporter boundary;
|
||||
- end-to-end preflight and prepared execution through the maintained
|
||||
endpoint-only `weather-light` override without exposing its endpoint;
|
||||
- offline inspection of `rakestrawhome-gemma-4-31b`;
|
||||
- a first-pass valid result with zero repairs;
|
||||
- an invalid structured result repaired successfully within one corrective
|
||||
call;
|
||||
- one-attempt exhaustion returning failed validation and no published report;
|
||||
- a corrective generation failure retaining its safe category and provider
|
||||
status policy;
|
||||
- cumulative usage and actual repair-count mapping;
|
||||
- comparison peers remaining independent when one profile repairs, exhausts,
|
||||
or fails;
|
||||
- v2 comparison manifest validation and guarded replacement; and
|
||||
- absent or blank optional `APIKeyEnv` values reaching the provider without an
|
||||
`Authorization` header, while `APIKeyRequired` profiles fail preflight.
|
||||
|
||||
Do not reproduce PromptKit's internal matrices for path traversal, JSON tree
|
||||
bounds, response framing, inheritance depth, repair prompt construction, or
|
||||
provider-detail normalization. WeatherReporter tests should protect only its
|
||||
adapter mappings, application policy, provenance, publication, and public
|
||||
contracts. Run ordinary and race-enabled repository tests because the repaired
|
||||
execution path participates in concurrent comparisons.
|
||||
|
||||
## Documentation And Release Impact
|
||||
|
||||
Implementation must update each canonical owner whose contract changes:
|
||||
|
||||
- `docs/policy/architecture.md` for prompt-execution, credential, diagnostic,
|
||||
and comparison invariants;
|
||||
- `docs/config.md` and the maintained local profile example for profile-source,
|
||||
inheritance, and credential semantics;
|
||||
- `docs/integrations/promptkit.md` for exact prompt versions, repair policy,
|
||||
profile composition, Rakestrawhome availability, and safe errors;
|
||||
- `docs/integrations/comparison-bundle.md` for the v2 manifest and repair count;
|
||||
- `docs/cli.md` for repair-count fields in action summaries;
|
||||
- `docs/operations.md` for changed failure behavior and any explicit provider
|
||||
diagnostic capture;
|
||||
- focused internal PromptKit adapter, app orchestration, prompt-debug, and
|
||||
comparison documentation; and
|
||||
- release notes for the dependency jump, prompt version change, possible
|
||||
additional model call, credential behavior, profile inheritance, diagnostic
|
||||
behavior, and comparison schema change.
|
||||
|
||||
Current-state documentation must not describe this behavior until the
|
||||
implementation lands. PromptKit remains the canonical owner of its complete
|
||||
built-in catalogs, YAML merge rules, transport limits, corrective-message
|
||||
construction, and public Go API.
|
||||
|
||||
## Scope
|
||||
|
||||
The completed feature includes:
|
||||
|
||||
- the direct module upgrade and tidy dependency graph;
|
||||
- a v0.6.0 compatibility audit of WeatherReporter's supported PromptKit paths;
|
||||
- inherited WeatherReporter domain profile definitions with unchanged
|
||||
effective targets;
|
||||
- correction of application preflight so PromptKit endpoint-only profiles work
|
||||
as documented while retaining a required model identity;
|
||||
- ordinary access to the Rakestrawhome built-in profile;
|
||||
- PromptKit's optional-credential policy, while direct-key-required profiles
|
||||
remain unsupported;
|
||||
- `repair_attempts` on all operational prompts and exact prompt-version bumps;
|
||||
- project-owned repair budget, actual-attempt, usage, and provenance mappings;
|
||||
- repair observability in action summaries, explicit debug output, and a v2
|
||||
comparison manifest;
|
||||
- safe provider HTTP status in ordinary errors and bounded provider detail only
|
||||
in explicit secure debug capture;
|
||||
- focused offline and race-enabled regression coverage; and
|
||||
- canonical current-state and release documentation updated with the code.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This upgrade does not include:
|
||||
|
||||
- application-implemented repair prompts or provider transport;
|
||||
- retries for HTTP, network, timeout, capacity, or other operational failures;
|
||||
- automatic profile escalation, fallback, ranking, or resampling;
|
||||
- changing the weather profile ladder, default report assignments, or concrete
|
||||
model targets beyond inheriting their maintained PromptKit definitions;
|
||||
- making Rakestrawhome a default or adding provider-specific configuration;
|
||||
- live-provider tests or a permanent benchmark framework;
|
||||
- exposing raw provider responses or sensitive diagnostics routinely;
|
||||
- a general prompt-source or pipeline plugin system; or
|
||||
- compatibility shims for PromptKit versions older than `v0.8.0`.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The roadmap is complete when:
|
||||
|
||||
- `go.mod` and `go.sum` resolve PromptKit `v0.8.0` without a replacement,
|
||||
workspace, or vendor tree;
|
||||
- all PromptKit v0.6.0 compatibility points relevant to WeatherReporter have
|
||||
been checked and valid supported inputs retain project-owned error identity;
|
||||
- the three WeatherReporter profiles inherit the intended PromptKit built-ins,
|
||||
retain their logical IDs, and inspect to the intended effective targets;
|
||||
- external standalone and inherited overrides obey documented precedence and
|
||||
failure behavior;
|
||||
- the maintained endpoint-only local override passes application preflight,
|
||||
retains an empty backend ID, and keeps its endpoint out of ordinary values;
|
||||
- `rakestrawhome-gemma-4-31b` is selectable through ordinary generation and
|
||||
comparison paths without WeatherReporter backend registration;
|
||||
- every operational prompt has one bounded repair attempt at exact
|
||||
version `2.1.0`;
|
||||
- inspection, preparation, execution, debug, and comparison values accurately
|
||||
preserve configured and actual repair counts;
|
||||
- first-pass success, repaired success, repair exhaustion, repair generation
|
||||
failure, and explicit empty content follow the documented outcomes;
|
||||
- the v2 comparison bundle distinguishes first-pass and repaired results;
|
||||
- credential preflight accepts absent optional environment credentials while
|
||||
rejecting direct-key-required profiles, and provider-error disclosure does
|
||||
not leak sensitive values;
|
||||
- the default test suite is offline and deterministic, ordinary and race
|
||||
validation pass, and no redundant upstream implementation suite is copied;
|
||||
and
|
||||
- every implemented contract is documented by its canonical current-state
|
||||
owner and disclosed in the eventual release notes.
|
||||
Reference in New Issue
Block a user