diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..33614d2 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,571 @@ +# PromptKit v0.8.0 Upgrade Implementation Plan + +Status: Ready for implementation. + +## Purpose And Authority + +This plan translates the accepted +[PromptKit v0.8.0 upgrade roadmap](promptkit-v0.8.0.md) into an ordered, +decision-complete implementation procedure. The feature roadmap owns purpose, +scope, policy, and the desired end state. This document owns implementation +order, concrete work allocation, stage boundaries, and verification until the +upgrade is complete. + +The implementing agent must complete the stages in numerical order. Each stage +is sized for one focused prompt handled by `gpt-5.6-terra` with high reasoning. +Do not combine stages merely because adjacent work touches the same package. + +## Locked Decisions + +The following decisions are final for this implementation: + +- upgrade directly from PromptKit `v0.5.0` to `v0.8.0`; +- declare one corrective call in each embedded prompt through + `repair_attempts: 1` rather than adding WeatherReporter repair logic; +- keep the repair budget in the exact embedded prompt definition and add no + global, per-report, CLI, profile, or operator configuration override; +- advance all four exact prompt versions from `2.0.0` to `2.1.0`; +- make `weather-light`, `weather-balanced`, and `weather-deep` minimal aliases + of the corresponding PromptKit built-ins through `base_profile`; +- retain the existing report-to-profile assignments and effective model + ladder; +- allow successfully inspected endpoint-only profiles to have an empty backend + ID while continuing to require a nonblank model; +- treat `APIKeyEnv` as an optional lookup source and reject only profiles that + report `APIKeyRequired`, because WeatherReporter supplies no direct request + credential; +- support PromptKit's built-in `rakestrawhome-gemma-4-31b` profile without + WeatherReporter-specific backend configuration; +- expose provider HTTP status through a project-owned safe generation error, + while writing provider code, type, and message only to explicit secure debug + capture; +- emit only `weatherreporter.comparison.v2`, with repair provenance, and do not + preserve v1 guarded-replacement support; and +- retain PromptKit dependency types inside the adapter and preserve all + stateless execution, atomic publication, comparison independence, and + disclosure invariants. + +## Implementation Rules + +For every stage: + +- read `docs/development.md`, all files under `docs/policy/`, this plan, the + feature roadmap, and the task-specific documents named by the stage; +- inspect the current code and tests before editing; use the repository's code + knowledge graph first for code discovery and fall back to text search for + literals, assets, and documentation; +- implement only the stage's scope and preserve unrelated user changes; +- keep PromptKit/provider types, client construction, YAML parsing, repair + mechanics, and provider transport inside the existing adapter boundary; +- use deterministic, offline, credential-free tests and injected clients or + synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local + endpoint calls; +- add tests at the narrowest stable owner identified by the testing policy and + avoid copying PromptKit's internal test matrices; +- update the canonical documentation owners listed for that stage in the same + change as the implemented contract; +- run `gofmt` on changed Go files, the stage's focused tests, + `GOWORK=off go test -count=1 ./...`, and `git diff --check`; and +- leave the repository passing before proceeding to the next stage. + +Stages affecting concurrent comparison, cancellation, or secure debug +filesystem work must also run the named focused packages with `-race`. Do not +weaken an existing assertion solely to accommodate the new dependency. When a +test encodes an intentionally changed contract, replace it with a behavioral +assertion for the accepted policy. + +## Implementation Stages + +### Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline + +Purpose: move to the tagged dependency and isolate compatibility changes before +adopting new WeatherReporter behavior. + +Work: + +1. Update `go.mod` to require + `gitea.maximumdirect.net/eric/promptkit v0.8.0` and refresh `go.sum` with + `GOWORK=off go mod tidy`. Do not add `go.work`, `vendor`, or a `replace` + directive and do not change WeatherReporter's Go version. +2. Resolve any compile failures using PromptKit's public root package only. + Keep all `Profile` and `OpenAICompatibleProfileConfig` literals keyed. Do not + register the now-reserved `rakestrawhome` backend. +3. Reconcile adapter tests that directly exercise PromptKit's changed optional + credential behavior. A profile whose only credential metadata is + `api_key_env` must reach an injected client when the environment value is + absent; it must no longer expect PromptKit to return + `ErrAPIKeyEnvMissing`. Do not change WeatherReporter's application preflight + in this stage. +4. Verify that every current embedded prompt, content file, schema, and fallback + profile inspects under v0.8.0. Verify selected invalid local endpoints and + malformed selected profile definitions still map to project-owned + configuration or profile-load categories. +5. Review the v0.6.0 compatibility corrections against supported + WeatherReporter inputs: metadata-authoritative identities, exact contained + `content_file` paths, regular embedded files, structurally valid endpoints, + bounded JSON-compatible values, cancellation identity, and strict response + framing. Add consumer tests only for a WeatherReporter boundary not already + protected by PromptKit. + +Do not enable profile inheritance or output repair yet. The expected result is +the current WeatherReporter feature set running against PromptKit v0.8.0. + +Focused verification: + +```sh +GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets +GOWORK=off go test -race -count=1 ./internal/adapters/promptkit +``` + +### Stage 2: Adopt Profile Inheritance And Current Credential Routing + +Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional +credential semantics, and the Rakestrawhome built-in without changing the model +ladder. + +Work: + +1. Replace the three embedded profile bodies with these exact leaf/base + relationships and no duplicated execution settings: + + | Leaf | Base | + | --- | --- | + | `weather-light` | `deepseek-4-flash` | + | `weather-balanced` | `gemini-flash-latest` | + | `weather-deep` | `claude-sonnet-latest` | + +2. Update prompt-asset fixtures and tests to understand `base_profile`. Assert + that all three leaf IDs remain selected identities and resolve to the same + backend, model, timeout, service tier, and reasoning settings exposed by the + current standalone definitions. Test relationships and effective behavior, + not copied private constants beyond the intentional model-ladder contract. +3. Preserve source precedence. Cover a standalone same-ID operator override, a + derived operator override, a configured source that shadows a base ID, and + selected missing-base, cyclic, malformed-base, and incomplete-target + failures. Do not implement inheritance or merging in WeatherReporter; all + resolution must remain PromptKit-owned. +4. Change application profile preflight to accept a successful inspection with + a nonblank model and an empty backend ID. Trust PromptKit inspection to have + resolved either a backend or endpoint; do not add the endpoint to + `promptexec.ProfileInspection` or ordinary provenance. +5. Remove application-level environment lookup and rejection for a nonblank + `APIKeyEnv`. Remove the now-unused `LookupEnv` fields and plumbing from + prompt, batch, and comparison inspection requests. Continue rejecting + `CredentialRequired`/`APIKeyRequired` before weather collection with the + existing missing-credential category. +6. Add an end-to-end offline regression proving the maintained endpoint-only + `weather-light` example passes application inspection, retains an empty + backend ID, and does not expose its endpoint. +7. Prove `rakestrawhome-gemma-4-31b` can pass ordinary and comparison profile + inspection through the existing adapter and reports the PromptKit + `rakestrawhome` backend ID. Do not make a provider call or add + Rakestrawhome-specific configuration. + +Canonical documentation in this stage: + +- update `docs/policy/architecture.md` so only direct-key-required profiles + fail credential preflight and backend identity is optional for endpoint-only + profiles; +- update `docs/config.md` to distinguish same-ID source replacement from + `base_profile` chain inheritance and to describe optional environment + credentials; +- update `docs/integrations/promptkit.md` for profile composition, parent + lookup precedence, endpoint-only identity, optional credentials, and + Rakestrawhome availability; and +- update `docs/internal/promptkit-adapter.md` and focused app internals for the + implemented inspection behavior. + +Focused verification: + +```sh +GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app +GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app +``` + +### Stage 3: Extend The Project-Owned Prompt Execution Contract + +Purpose: establish dependency-neutral repair and structured-generation-error +values before the adapter or application relies on them. + +Work: + +1. Add `RepairAttempts int` to `promptexec.OutputContract`. It is the configured + additional-call budget from the exact prompt contract. +2. Add `RepairAttempts int` to `promptexec.Validation`. It is the number of + corrective calls actually started for the completed result. Update + `NewValidation` and every caller so construction is explicit; reject or + normalize no values here because PromptKit owns output-contract validity. +3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so + repair values are retained without sharing mutable state. +4. Add a project-owned immutable `promptexec.GenerationError` with unexported + status and provider-detail fields plus safe accessors: + + - `StatusCode() int` + - `ProviderCode() string` + - `ProviderType() string` + - `ProviderMessage() string` + - `Category() ErrorCategory`, always returning `Generation` + - `Error()`, exposing only the WeatherReporter generation category/message + and optional HTTP status + - `GoString()`, returning the same safe representation + - `Unwrap()`, preserving a project-owned `*promptexec.Error` + +5. Provide one constructor used by adapters. Defensively normalize valid UTF-8 + and bound code/type to 256 Unicode code points and message to 4,096 Unicode + code points, even though PromptKit already bounds its accessors. Do not + expose fields through struct formatting, JSON tags, or exported mutable + fields. Preserve the dependency cause only behind the project-owned error so + `errors.Is`/`errors.As` identities remain available without entering error + text. +6. Add focused tests proving nil/zero safety, category and unwrap behavior, + status-only ordinary formatting, `%#v` redaction, provider-detail bounds, + and repair-value copying. + +Do not import PromptKit from `internal/promptexec` and do not change CLI or +artifact schemas in this stage. + +Focused verification: + +```sh +GOWORK=off go test -count=1 ./internal/promptexec +``` + +### Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter + +Purpose: make the adapter faithfully translate v0.8.0 preparation, execution, +validation, usage, and failure values into the Stage 3 contract. + +Work: + +1. Map `promptkit.OutputContract.RepairAttempts` in prompt inspection and + prepared-execution details. Map + `promptkit.ValidationResult.RepairAttempts` in completed execution. +2. Preserve PromptKit's cumulative usage exactly as reported across the initial + call and every completed correction. Continue returning only the final raw + candidate and final validation result, subject to WeatherReporter's 64 KiB + generated-output bound. +3. In adapter error classification, retain cancellation, deadline, and capacity + precedence. Before the generic `ErrLLMGenerate` branch, use `errors.As` for + `*promptkit.GenerationError` and construct the project-owned + `promptexec.GenerationError` with status, code, type, message, and hidden + cause. Initial and corrective generation failures use the same mapping. +4. Extend the injected adapter client used by tests so it can return an ordered + sequence of responses or errors and record each request safely. +5. Use a synthetic PromptKit prompt with JSON Schema validation and + `repair_attempts: 1` to cover: + + - first-pass valid output with zero corrections; + - explicitly empty or invalid output followed by valid corrected output; + - one-attempt exhaustion returning a final failed validation result rather + than an operational error; + - a non-2xx-style `GenerationError` during correction; + - cumulative token usage and actual repair count; and + - the same prepared prompt/profile identity across the corrective flow. + +6. Keep these tests at the adapter boundary. Do not assert PromptKit's private + corrective-message wording or reconstruct its internal repair algorithm. + +Canonical documentation in this stage: update +`docs/internal/promptkit-adapter.md` for the repair/result/error mappings. Do +not yet claim that embedded WeatherReporter prompts enable repair. + +Focused verification: + +```sh +GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec +GOWORK=off go test -race -count=1 ./internal/adapters/promptkit +``` + +### Stage 5: Carry Repair Provenance Through Application Workflows + +Purpose: make application orchestration understand configured and actual repair +counts before changing the embedded prompt policy. + +Work: + +1. Add an expected generated-text repair budget to `report.Definition` and set + it explicitly to zero for all four current `2.0.0` definitions in this + stage. Include it in report-definition validation and retained contract + tests. +2. Extend exact prompt preflight so format, validation mode, schema path, and + repair budget must all match the resolved report definition. Extend + preparation and completion provenance checks to require the same repair + budget across inspection and the opaque prepared snapshot. +3. Add `RepairAttempts *int` to application outcomes where execution may fail + before validation exists. Set it to a fresh pointer immediately after a + non-nil completed execution is returned, before WeatherReporter's secondary + generated-text validation. A pointer is required so completed first-pass + zero is distinguishable from unavailable provenance. +4. Carry independent copies through `ReportResult`, `BatchReportResult`, batch + conversion, comparison execution's internal outcome, and relevant test + fakes. Do not expose the new value in CLI or comparison JSON yet. +5. Preserve the actual count on PromptKit validation rejection and on later + WeatherReporter generated-text or render failures. Leave it unavailable on + preparation, capacity, cancellation, deadline, and generation errors that + return no completed PromptKit result. +6. Add focused tests for provenance mismatch, completed zero, completed + positive, validation rejection, later local validation failure, early + operational failure, batch copying, and independent concurrent profile + outcomes. + +Canonical documentation in this stage: update the focused prepared-report and +app-orchestration internals to describe configured versus actual repair +provenance. Current public documents should continue to report the embedded +budget as zero until Stage 6. + +Focused verification: + +```sh +GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli +GOWORK=off go test -race -count=1 ./internal/app +``` + +### Stage 6: Activate One Repair And Expose Ordinary Result Provenance + +Purpose: switch the operational prompts to the accepted one-correction policy +and make ordinary generate/run/batch output report what occurred. + +Work: + +1. Add `repair_attempts: 1` to the output contract of all four embedded prompt + definitions and change each exact prompt version from `2.0.0` to `2.1.0`. + Do not change prompt text or generated-text schemas solely for this upgrade. +2. Change all four report registry definitions to exact prompt version `2.1.0` + and expected repair budget one. Update exact-version fixtures and assertions + throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests + that classify `repair_attempts` as a retired setting and replace them with + an exact one-attempt contract assertion. +3. Add `repairAttempts` to successful and failed generate and batch JSON result + shapes through the Stage 5 pointers. Emit integer zero for a completed + first-pass result, a positive integer for a completed repaired result, and + omit the field when no completed validation made it available. +4. Keep the existing `validationStatus` and failure categories authoritative. + A repaired valid result proceeds normally. Repair exhaustion remains + `validation_rejected`, publishes no report for that profile, and retains the + actual attempt count. +5. Add representative offline assembled tests proving first-pass success, + repaired success, exhaustion, explicit empty initial content, and batch + result propagation. Reuse the real PromptKit adapter with an injected + sequence client for at least one end-to-end repaired execution; use the + existing app fake at other boundaries where lower-level repair is already + covered. +6. Confirm no application loop, provider retry, profile fallback, or + request-level `OutputContract` override was introduced. + +Canonical documentation in this stage: + +- update `docs/policy/architecture.md` with PromptKit-owned bounded repair and + failed-exhaustion invariants; +- update `docs/integrations/promptkit.md` with exact prompt version `2.1.0`, one + configured repair, actual-count semantics, cumulative usage, explicit-empty + handling, and the distinction from operational retries; +- update `docs/cli.md` for generate and batch `repairAttempts` fields; +- update `docs/internal/report-registry.md`, prepared-report internals, and app + orchestration internals for exact version and repair flow; and +- keep configuration documentation unchanged because no repair setting is + added. + +Focused verification: + +```sh +GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli +GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app +``` + +### Stage 7: Migrate Comparison Bundles To v2 + +Purpose: preserve repair activity in the profile-evaluation artifact and make +the strict durable schema change explicit. + +Work: + +1. Change `comparison.SchemaVersion` to + `weatherreporter.comparison.v2`. Emit and recognize v2 only; do not retain a + v1 parser or guarded-replacement compatibility path. +2. Add `RepairAttempts *int` to each application comparison profile result, + CLI comparison profile summary, and durable `comparison.Result`. Propagate a + fresh copy from Stage 5's execution outcome. +3. Place `repairAttempts` immediately after `validationStatus` in the canonical + result-object JSON field order. Encode zero for completed first-pass + validation, a positive integer for completed correction, and omit it only + when no completed validation exists. +4. Tighten manifest invariants: every non-nil repair count is non-negative; a + successful result must have `validationStatus: "passed"` and a non-nil + repair count; a failed result with a completed validation status must also + have a non-nil count; and an early operational failure may omit both. +5. Update the strict token-level JSON recognizer to accept only the canonical + `repairAttempts` field at its correct object level, reject duplicate, + unknown, negative, fractional, string, overflow, and malformed values, and + continue rejecting v1 as an unsupported current bundle. +6. Update manifest construction, cloning, validation, exact serialization + tests, guarded replacement tests, malicious bundle tests, partial-success + tests, and CLI comparison summaries. Preserve flat layout, result ordering, + hashes, atomic publication, cancellation safety, and no Distributor calls. +7. Cover concurrent peers where one succeeds first-pass, one repairs, one + exhausts, and one fails operationally. The counts must remain attached to + the selected profile positions without races or cross-contamination. + +Canonical documentation in this stage: + +- replace the v1 contract in `docs/integrations/comparison-bundle.md` with v2, + including exact field order, presence rules, and the lack of v1 replacement + compatibility; +- update `docs/cli.md` for comparison `repairAttempts`; +- update `docs/operations.md` to tell operators to move or remove an existing + v1 bundle before replacing at the same destination; and +- update comparison execution/publication internals and architecture policy as + needed for the current-only version invariant. + +Focused verification: + +```sh +GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli +GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app +``` + +### Stage 8: Add Secure Provider-Failure Debug Capture + +Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the +existing explicit secure debug boundary while keeping ordinary errors safe. + +Work: + +1. Extend prompt preparation debug output with configured + `repairAttempts` and advance its schema identifier from + `weatherreporter.prompt_preparation_debug.v2` to + `weatherreporter.prompt_preparation_debug.v3`. +2. Extend execution validation debug output with actual `repairAttempts` and + advance its schema identifier from + `weatherreporter.prompt_execution_debug.v2` to + `weatherreporter.prompt_execution_debug.v3`. Retain cumulative token usage. +3. Add a dedicated `failure.json` artifact with schema identifier + `weatherreporter.prompt_failure_debug.v1`. Its canonical fields are: + + - top level: `schemaVersion`, `reportId`, `validDate`, `runId`, `failure`; + - failure object: `category`, `statusCode`, `providerCode`, `providerType`, + `providerMessage`; + - omit absent provider fields and zero status; and + - never include the raw provider body, headers, endpoint, credentials, + request, schema, rendered prompt, or generated candidate. + +4. Add `PromptDebugWriter.WriteFailure` using the existing handle-relative + secure run directory, `0700` directory and `0600` file modes, canonical JSON + encoding, and no-follow/atomic replacement behavior. Disabled writers must + perform no filesystem work. +5. When execution returns an error, use `errors.As` only against the + project-owned `*promptexec.GenerationError`. If explicit debug capture is + enabled, write `failure.json` using that profile's existing debug reference. + This applies equally to initial and corrective provider failures and keeps + comparison profile directories isolated. +6. If failure-debug writing also fails, retain the generation failure as the + primary categorized error and join the safe debug-write failure rather than + replacing or hiding the provider failure. Never place provider code, type, + or message in the joined error text. +7. Ordinary generate, batch, and comparison errors should gain only the safe + HTTP status already rendered by `promptexec.GenerationError.Error`; do not + add provider detail fields to CLI summaries, comparison manifests, logs, or + Distributor requests. +8. Add adversarial tests for formatter redaction, malicious provider strings, + JSON escaping, bounds, absent fields, file modes, symlink/path attacks, + write failure, cancellation identity, initial versus corrective failures, + and concurrent comparison captures. + +Canonical documentation in this stage: + +- update `docs/operations.md` with the three debug artifact versions, + `failure.json`, sensitivity, permissions, and retention; +- update `docs/integrations/promptkit.md` with ordinary status-only disclosure + and debug-only provider detail; +- update prompt-debug, PromptKit-adapter, and app-orchestration internals; and +- ensure `docs/policy/architecture.md` explicitly prohibits provider-controlled + diagnostics from ordinary outputs. + +Focused verification: + +```sh +GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli +GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app +``` + +### Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit + +Purpose: verify the complete end state as one coherent WeatherReporter feature +and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile, +repair, or debug claims. + +Work: + +1. Re-read the feature roadmap, `docs/development.md`, every policy document, + and every canonical document changed by Stages 1-8. Reconcile them against + executable behavior and remove duplicated or stale definitions. Keep + unimplemented future ideas in `docs/roadmap/future.md`, not current-state + documents. +2. Search code, embedded assets, examples, tests, and documentation for stale + contractual literals and review every occurrence of: + + - PromptKit `v0.5.0`, `v0.6.0`, and `v0.7.0` as an active dependency claim; + - prompt version `2.0.0`; + - `weatherreporter.comparison.v1`; + - prompt debug schema v2 identifiers; + - claims that profile fields never inherit; + - claims that every `APIKeyEnv` must be populated; + - claims that backend ID is always required; + - claims that repair is disabled or `repair_attempts` is retired; and + - provider detail in ordinary output. + + Historical release documents may retain accurate historical literals. +3. Verify canonical ownership: + + - architecture owns invariants and boundaries; + - config owns operator profile and credential behavior, but no repair field; + - PromptKit integration owns logical prompt/profile/output contracts; + - CLI owns result fields; + - operations owns explicit debug handling and old comparison-bundle cleanup; + - comparison integration owns the complete v2 manifest; and + - internal documents own implementation flow without duplicating the public + references. + +4. Verify maintained examples remain valid, secret-free, and tested. The local + `weather-light` example remains a standalone endpoint-only profile rather + than inheriting an OpenRouter backend it cannot clear. +5. Review the complete diff for architecture leakage. Production packages + outside `internal/adapters/promptkit` must not import PromptKit; no + application repair loop, provider client, raw provider diagnostic, profile + YAML parser, or durable application state may have appeared. +6. Review tests under the testing policy. Keep consumer contract and regression + coverage, remove accidental duplication of upstream implementation tests, + and ensure every default test is offline and repeatable. +7. Run the complete validation set: + +```sh +gofmt -w +GOWORK=off go test -count=1 ./... +GOWORK=off go test -race -count=1 ./... +GOWORK=off go vet ./... +GOWORK=off go build ./... +GOWORK=off go mod tidy -diff +go run ./cmd/weatherreporter --help +go run ./cmd/weatherreporter compare --help +test -z "$(git ls-files go.work go.work.sum)" +test ! -e vendor +git diff --check +``` + +8. Confirm `go.mod` has no `replace`, the resolved PromptKit module is exactly + v0.8.0, and no live credential or provider call occurred during validation. +9. After every check passes, update this plan's status to Completed and add a + concise completion note listing the implemented stages. Do not delete either + roadmap until the maintainer has reviewed the implementation. Do not create + a release document or tag; release preparation remains a separate maintainer + action once a version is selected. + +## Completion Standard + +The implementation is complete only when all nine stages pass their focused +and repository-wide checks, all locked decisions are observable in code and +canonical documentation, and the feature roadmap's completion criteria are +satisfied. Passing compilation alone is insufficient. The final state must +demonstrate repaired success, repair exhaustion, comparison provenance, +endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome +inspection, safe ordinary provider failures, secure debug-only detail, and +unchanged publication and concurrency invariants. diff --git a/docs/roadmap/promptkit-v0.8.0.md b/docs/roadmap/promptkit-v0.8.0.md new file mode 100644 index 0000000..32de678 --- /dev/null +++ b/docs/roadmap/promptkit-v0.8.0.md @@ -0,0 +1,488 @@ +# PromptKit v0.8.0 Upgrade Roadmap + +Status: Accepted feature direction; implementation has not started. + +## Purpose + +WeatherReporter should upgrade its PromptKit dependency from `v0.5.0` to +`v0.8.0` and deliberately adopt the useful consumer-facing capabilities added +in `v0.6.0`, `v0.7.0`, and `v0.8.0`. The upgrade should improve output-contract +reliability, profile composition, local and alternate endpoint support, and +provider-failure diagnosis without moving PromptKit responsibilities into +WeatherReporter or weakening the application's stateless and security +boundaries. + +This roadmap defines the intended scope, policy, and end state. The +[implementation plan](implementation.md) owns the procedure for reaching that +state. + +## User Intent + +The upgrade is intended to: + +- use PromptKit's bounded output repair to recover from occasional malformed + structured weather prose; +- keep WeatherReporter's domain profile IDs stable while inheriting maintained + PromptKit model definitions; +- make PromptKit's additional built-in backend and profile available for + explicit generation and profile comparisons; +- support unauthenticated or optionally authenticated OpenAI-compatible + endpoints without inventing a WeatherReporter transport layer; +- make provider HTTP failures more actionable under an explicit + WeatherReporter disclosure policy; and +- receive PromptKit's intervening correctness, safety, cancellation, resource, + and efficiency improvements as part of one tested dependency upgrade. + +The model ladder and report assignments do not change as part of this work: +Hourly continues to select `weather-light`; Daily, Today, and Tomorrow continue +to select `weather-balanced`; and `weather-deep` remains available for explicit +selection. This upgrade does not promote the new Rakestrawhome profile into +that default ladder. + +## Current State + +WeatherReporter currently depends on +`gitea.maximumdirect.net/eric/promptkit` at `v0.5.0`. The PromptKit adapter +supplies embedded prompts, JSON Schemas, and +application-fallback profiles, plus an optional configured profile source and +the conventional local backend. + +The four generated-text prompts are exact version `2.0.0` JSON Schema prompts. +They omit `repair_attempts`, so execution is single-pass. The project-owned +`promptexec.OutputContract` and validation result also omit repair budgets and +actual repair counts. + +The three embedded WeatherReporter profiles duplicate the effective fields of +these PromptKit built-ins: + +| WeatherReporter profile | PromptKit built-in with the same target | +| --- | --- | +| `weather-light` | `deepseek-4-flash` | +| `weather-balanced` | `gemini-flash-latest` | +| `weather-deep` | `claude-sonnet-latest` | + +WeatherReporter preflights any nonblank `api_key_env` as a required credential, +even though PromptKit v0.7.0 distinguishes an optional environment source from +an explicit `APIKeyRequired` target. Provider generation failures are reduced +to WeatherReporter's safe `generation` category; the PromptKit dependency error +is retained as a hidden cause, but its structured HTTP status and provider +diagnostics are not mapped into project-owned values. + +The maintained `weather-light` local override is an endpoint-only profile, and +the configuration contract says endpoint-only profiles are supported. PromptKit +inspection correctly reports no backend ID for that form, but WeatherReporter +application preflight currently requires both a nonblank backend and model. +That mismatch prevents the documented example from reaching generation and +should be corrected as part of adopting the current PromptKit target contract. + +## Upstream Release Assessment + +### PromptKit v0.6.0 + +`v0.6.0` adds no public declarations, but it is a material compatibility and +safety release. It centralizes execution-setting, output-contract, endpoint, +and JSON-compatible-value validation; makes YAML metadata authoritative for +prompt and profile identity; hardens `content_file` containment and regular-file +requirements; validates OpenAI-compatible endpoints structurally; bounds JSON +trees and successful provider bodies; requires exactly one JSON value in +provider responses; preserves cancellation and transport error identities; and +reuses schema and rendered-artifact work within an operation. + +WeatherReporter should receive these improvements directly from the dependency +and audit its own supported assets and configuration paths against the stricter +contracts. It should not duplicate PromptKit's internal validators or tests. +The existing embedded prompt paths, inline data-package input, generated-output +limit, and adapter boundary remain conceptually correct. + +### PromptKit v0.7.0 + +`v0.7.0` adds four potentially useful consumer features: + +- linear, cycle-safe profile inheritance through `base_profile` and + `Profile.BaseProfileID`; +- the built-in `rakestrawhome` backend and + `rakestrawhome-gemma-4-31b` profile; +- optional API-key environment sources, with `APIKeyRequired` reserved for an + explicit local credential requirement; and +- bounded structured `GenerationError` details for non-2xx responses from the + built-in OpenAI-compatible client. + +WeatherReporter has no manual `rakestrawhome` registration and uses keyed +PromptKit profile literals, so the two source-compatibility hazards called out +by the release do not require migration shims. The profile, credential, and +error features do require deliberate application-policy choices described +below. + +### PromptKit v0.8.0 + +`v0.8.0` activates the existing output-contract repair budget. A positive +`repair_attempts` value authorizes up to that many corrective model calls after +eligible `basic`, `json`, or `json_schema` validation failures. The supported +budget is zero through three. Repairs preserve the original rendered +conversation, effective target, session, structured-output contract, and +backend capacity policy. The final result reports cumulative token usage and +the number of corrective calls actually made. + +Repair exhaustion is a completed generation with failed validation, not an +operational error. WeatherReporter's existing policy should continue to reject +that result and publish no report for that profile. Explicit empty provider +content now reaches output validation; for WeatherReporter's JSON Schema +prompts it is therefore eligible for repair rather than being misclassified as +a malformed provider envelope. + +## Desired End State + +WeatherReporter builds and tests against PromptKit `v0.8.0` with no workspace, +vendor, or module replacement dependency. Its public behavior remains +stateless, its PromptKit dependency types remain confined to the adapter, and +its ordinary summaries and logs remain safe. + +The completed integration: + +- benefits from the v0.6.0 safety and efficiency corrections; +- composes WeatherReporter domain profiles from PromptKit's maintained built-in + profiles while preserving WeatherReporter-owned leaf IDs and operator + override precedence; +- accepts a successfully inspected endpoint-only profile with a nonblank model + even though it has no logical backend ID; +- permits explicit use of PromptKit's Rakestrawhome profile without custom + backend wiring; +- applies an accepted bounded-repair policy to every operational structured + prompt; +- validates repair configuration during preflight and records actual repair + activity in project-owned result values; +- retains PromptKit's cumulative usage accounting in explicit debug output; +- distinguishes safe provider HTTP status from potentially sensitive provider + diagnostics; and +- documents the changed profile, credential, repair, comparison, debug, and + failure contracts in their canonical owners. + +## Dependency And Compatibility Policy + +The module requirement should move directly from `v0.5.0` to `v0.8.0`, followed +by a clean module tidy. WeatherReporter already requires Go 1.26 while PromptKit +`v0.8.0` requires Go 1.25.5, so no Go version change is needed for this upgrade. + +Consumer validation must cover the paths called out by PromptKit v0.6.0: + +- every embedded prompt, content file, schema, and fallback profile inspects + through PromptKit `v0.8.0`; +- configured single-file and directory profile sources retain their lazy, + metadata-authoritative identity and precedence behavior; +- malformed selected profiles and invalid local endpoints retain actionable + WeatherReporter categories; +- the inline YAML data package and prepared-execution path remain within the + new JSON and response bounds; and +- cancellation, deadline, and backend-capacity identities still cross the + adapter correctly. + +PromptKit owns its 16 MiB successful transport-response bound and JSON framing. +WeatherReporter retains its stricter 64 KiB generated-text acceptance bound. +The consumer suite should protect that relationship without reproducing +PromptKit's lower-level transport matrix. + +## Domain Profile Composition + +The embedded profiles should become application-owned aliases: + +```yaml +id: weather-light +base_profile: deepseek-4-flash +``` + +```yaml +id: weather-balanced +base_profile: gemini-flash-latest +``` + +```yaml +id: weather-deep +base_profile: claude-sonnet-latest +``` + +The effective backend, model, timeout, service tier, and reasoning settings +must initially remain identical to the current WeatherReporter definitions. +The selected leaf remains the durable logical profile identity even though its +effective target is inherited. + +Profile source precedence remains: + +1. explicit in-memory profiles used by tests or embedding consumers; +2. the configured `profile_file` or `profile_dir` source; +3. WeatherReporter's embedded fallback catalog; and +4. PromptKit's built-in catalog. + +Sources still do not merge definitions of the same ID. Once a selected +definition names `base_profile`, however, each parent ID is resolved through +that same precedence order and the resulting linear chain is merged from root +to leaf according to PromptKit's inheritance contract. Documentation must make +that distinction explicit. A malformed leaf, missing or malformed base, cycle, +overlong chain, or incomplete resolved target fails profile inspection before +weather collection. + +An operator may continue to replace `weather-light`, `weather-balanced`, or +`weather-deep` with a standalone definition. An operator may also define a +derived replacement. The maintained local endpoint example should remain +standalone because PromptKit profile inheritance has no clearing syntax: using +an OpenRouter base would retain its backend identity and capacity policy even +when the child replaces the endpoint. + +WeatherReporter should treat the adapter's successful profile inspection as +authoritative that PromptKit resolved a usable route. A nonblank model remains +required, but backend ID is optional for an endpoint-only profile and should be +omitted from safe provenance where unavailable. WeatherReporter still must not +surface the endpoint outside explicit debug capture. This aligns application +preflight with PromptKit and with the existing CLI, comparison, and debug value +shapes, all of which already permit an absent backend identity. + +## Rakestrawhome Availability + +The reserved `rakestrawhome` backend and built-in +`rakestrawhome-gemma-4-31b` profile should be supported automatically through +ordinary PromptKit selection. Operators may choose that profile with the +existing global profile setting or as one entry in `compare`, and PromptKit's +backend capacity policy remains authoritative. + +WeatherReporter should not register, wrap, or duplicate the backend or profile, +and should not add a Rakestrawhome-specific configuration field. Its canonical +PromptKit integration documentation should link to PromptKit for the current +built-in catalog and credential contract rather than copying volatile endpoint +or capacity values. Offline inspection coverage should prove that the built-in +profile crosses the WeatherReporter adapter with the expected logical backend +identity. + +## Bounded Structured-Output Repair + +The accepted repair budget belongs to the exact PromptKit output contract, not +to a WeatherReporter retry loop. PromptKit alone should construct corrective +messages, perform additional calls, enforce the budget, aggregate usage, and +coordinate backend capacity. WeatherReporter must not retry provider failures, +switch profiles, or layer another repair mechanism around `RunPrepared`. + +All four embedded prompt definitions should declare `repair_attempts: 1`. +Because this changes prompt +execution behavior, latency, cost, hash, and provenance, each definition and +its report-registry binding should advance from exact version `2.0.0` to +`2.1.0`. Prompt text and generated-text schemas do not need to change solely +for this feature. + +The embedded prompt definition is the per-report pipeline policy owner. This +upgrade should not add a global or per-report operator configuration field for +repair attempts and should not construct a request-level replacement output +contract. A future pipeline may select another budget only through a deliberate +prompt-definition and exact-version change. + +The project-owned PromptKit boundary should retain: + +- the configured repair budget in prompt inspection and preparation output + contracts; +- the number of corrective calls actually made in completed validation; +- cumulative PromptKit token usage across initial and corrective calls; and +- the final candidate and final validation result only, consistent with the + PromptKit contract. + +Prompt inspection and preparation provenance must require the repair budget to +match the exact expected prompt definition just as they currently require the +format, validation mode, and schema path to match. A zero-attempt successful +result is normal when the first candidate passes. A repair-exhausted result +continues through WeatherReporter's ordinary `validation_rejected` failure +path, and an operational or generation failure during correction remains that +profile's ordinary operational failure. + +For concurrent comparison, every profile should use the same prompt repair +budget. A corrective call remains part of that profile's one prepared +execution and uses PromptKit's existing backend capacity pool. One profile's +repair or failure must not cancel independent peers. + +## Repair Observability And Comparison Contract + +The actual repair count is safe operational provenance and should be visible +where WeatherReporter already reports completed validation. Generation, batch, +and comparison action summaries should expose it without exposing candidates, +schemas, or diagnostics. Explicit execution debug output should add it to the +validation object alongside PromptKit's already mapped cumulative usage. + +Profile comparison needs this value in `comparison.json`: a successful result +that required correction is materially different from a first-pass success +when evaluating model reliability, latency, and cost. The manifest should +therefore advance to `weatherreporter.comparison.v2` and add a non-negative +`repairAttempts` field to each result. The field is zero when no corrective +call began, including ordinary first-pass success. A failure carries the count +when PromptKit returned a completed validation result; it is omitted only when +execution failed before a completed validation result made the value known. + +The v2 manifest should remain flat, strict, deterministic, and atomically +published. WeatherReporter does not need to preserve v1 replacement +compatibility: comparison bundles are operator-owned development outputs, and +the current integration contract intentionally recognizes only its current +schema. The release notes and comparison documentation must call out the +version change so an operator can remove or relocate an older bundle before +using guarded replacement at the same destination. + +## Credential Semantics + +PromptKit v0.7.0 treats `APIKeyEnv` as an optional lookup source. If the +environment variable is absent or blank and no direct credential is supplied, +the built-in client omits `Authorization` and lets the endpoint respond. +`APIKeyRequired` is the distinct signal that a usable credential must be +provided locally. + +WeatherReporter cannot supply PromptKit's request-scoped direct API-key value, +so a profile reporting `APIKeyRequired` remains unsupported and must fail +before weather collection. WeatherReporter should not require a nonblank value +for an optional `APIKeyEnv` during application preflight. The built-in client +should omit `Authorization` when that source is unavailable and let the +endpoint return any authentication failure through the ordinary structured +generation-error path. + +## Structured Generation Failures + +PromptKit v0.7.0's `GenerationError` can report a provider HTTP status plus +bounded provider code, type, and message. WeatherReporter should consume that +type only inside the PromptKit adapter and map any adopted fields into a +project-owned immutable error. PromptKit dependency types must not become app +or CLI contracts. + +HTTP status is safe enough for ordinary diagnostics. Provider code, type, and +message remain untrusted and may contain request or schema fragments. They +must never enter ordinary errors, action summaries, comparison manifests, +logs, generated reports, or Distributor payloads. The full structured +diagnostic belongs only in an explicitly requested secure `--llm-debug-dir` +`failure.json` artifact. That artifact may contain PromptKit's normalized +bounded fields but never the raw provider body, headers, endpoint, credentials, +or reconstructed request. + +Initial-call and corrective-call non-2xx responses should follow the same +mapping. Cancellation and deadline categories continue to take precedence over +provider classification where PromptKit preserves those identities. + +## Testing Policy + +The default suite must remain deterministic, offline, and credential-free. +Use injected PromptKit clients and synthetic embedded or temporary assets for +consumer behavior; do not call OpenRouter, Rakestrawhome, or a local endpoint. + +Risk-based coverage should include: + +- all embedded prompts and inherited domain profiles inspecting successfully + under PromptKit `v0.8.0`; +- unchanged effective targets and report-to-profile assignments after the + alias refactor; +- external standalone and derived profile precedence, plus selected missing, + cyclic, and malformed-base failures at the WeatherReporter boundary; +- end-to-end preflight and prepared execution through the maintained + endpoint-only `weather-light` override without exposing its endpoint; +- offline inspection of `rakestrawhome-gemma-4-31b`; +- a first-pass valid result with zero repairs; +- an invalid structured result repaired successfully within one corrective + call; +- one-attempt exhaustion returning failed validation and no published report; +- a corrective generation failure retaining its safe category and provider + status policy; +- cumulative usage and actual repair-count mapping; +- comparison peers remaining independent when one profile repairs, exhausts, + or fails; +- v2 comparison manifest validation and guarded replacement; and +- absent or blank optional `APIKeyEnv` values reaching the provider without an + `Authorization` header, while `APIKeyRequired` profiles fail preflight. + +Do not reproduce PromptKit's internal matrices for path traversal, JSON tree +bounds, response framing, inheritance depth, repair prompt construction, or +provider-detail normalization. WeatherReporter tests should protect only its +adapter mappings, application policy, provenance, publication, and public +contracts. Run ordinary and race-enabled repository tests because the repaired +execution path participates in concurrent comparisons. + +## Documentation And Release Impact + +Implementation must update each canonical owner whose contract changes: + +- `docs/policy/architecture.md` for prompt-execution, credential, diagnostic, + and comparison invariants; +- `docs/config.md` and the maintained local profile example for profile-source, + inheritance, and credential semantics; +- `docs/integrations/promptkit.md` for exact prompt versions, repair policy, + profile composition, Rakestrawhome availability, and safe errors; +- `docs/integrations/comparison-bundle.md` for the v2 manifest and repair count; +- `docs/cli.md` for repair-count fields in action summaries; +- `docs/operations.md` for changed failure behavior and any explicit provider + diagnostic capture; +- focused internal PromptKit adapter, app orchestration, prompt-debug, and + comparison documentation; and +- release notes for the dependency jump, prompt version change, possible + additional model call, credential behavior, profile inheritance, diagnostic + behavior, and comparison schema change. + +Current-state documentation must not describe this behavior until the +implementation lands. PromptKit remains the canonical owner of its complete +built-in catalogs, YAML merge rules, transport limits, corrective-message +construction, and public Go API. + +## Scope + +The completed feature includes: + +- the direct module upgrade and tidy dependency graph; +- a v0.6.0 compatibility audit of WeatherReporter's supported PromptKit paths; +- inherited WeatherReporter domain profile definitions with unchanged + effective targets; +- correction of application preflight so PromptKit endpoint-only profiles work + as documented while retaining a required model identity; +- ordinary access to the Rakestrawhome built-in profile; +- PromptKit's optional-credential policy, while direct-key-required profiles + remain unsupported; +- `repair_attempts` on all operational prompts and exact prompt-version bumps; +- project-owned repair budget, actual-attempt, usage, and provenance mappings; +- repair observability in action summaries, explicit debug output, and a v2 + comparison manifest; +- safe provider HTTP status in ordinary errors and bounded provider detail only + in explicit secure debug capture; +- focused offline and race-enabled regression coverage; and +- canonical current-state and release documentation updated with the code. + +## Non-Goals + +This upgrade does not include: + +- application-implemented repair prompts or provider transport; +- retries for HTTP, network, timeout, capacity, or other operational failures; +- automatic profile escalation, fallback, ranking, or resampling; +- changing the weather profile ladder, default report assignments, or concrete + model targets beyond inheriting their maintained PromptKit definitions; +- making Rakestrawhome a default or adding provider-specific configuration; +- live-provider tests or a permanent benchmark framework; +- exposing raw provider responses or sensitive diagnostics routinely; +- a general prompt-source or pipeline plugin system; or +- compatibility shims for PromptKit versions older than `v0.8.0`. + +## Completion Criteria + +The roadmap is complete when: + +- `go.mod` and `go.sum` resolve PromptKit `v0.8.0` without a replacement, + workspace, or vendor tree; +- all PromptKit v0.6.0 compatibility points relevant to WeatherReporter have + been checked and valid supported inputs retain project-owned error identity; +- the three WeatherReporter profiles inherit the intended PromptKit built-ins, + retain their logical IDs, and inspect to the intended effective targets; +- external standalone and inherited overrides obey documented precedence and + failure behavior; +- the maintained endpoint-only local override passes application preflight, + retains an empty backend ID, and keeps its endpoint out of ordinary values; +- `rakestrawhome-gemma-4-31b` is selectable through ordinary generation and + comparison paths without WeatherReporter backend registration; +- every operational prompt has one bounded repair attempt at exact + version `2.1.0`; +- inspection, preparation, execution, debug, and comparison values accurately + preserve configured and actual repair counts; +- first-pass success, repaired success, repair exhaustion, repair generation + failure, and explicit empty content follow the documented outcomes; +- the v2 comparison bundle distinguishes first-pass and repaired results; +- credential preflight accepts absent optional environment credentials while + rejecting direct-key-required profiles, and provider-error disclosure does + not leak sensitive values; +- the default test suite is offline and deterministic, ordinary and race + validation pass, and no redundant upstream implementation suite is copied; + and +- every implemented contract is documented by its canonical current-state + owner and disclosed in the eventual release notes.