From cb572e4b93fd8ce9138db1e73a3e10c2cdece5e0 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 29 Aug 2026 17:36:41 +0000 Subject: [PATCH] Remove completed PromptKit plans --- docs/roadmap/implementation.md | 595 ------------------------------- docs/roadmap/promptkit-v0.8.0.md | 488 ------------------------- 2 files changed, 1083 deletions(-) delete mode 100644 docs/roadmap/implementation.md delete mode 100644 docs/roadmap/promptkit-v0.8.0.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 538c1df..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,595 +0,0 @@ -# PromptKit v0.8.0 Upgrade Implementation Plan - -Status: Complete. - -Completion note: Stages 1–9 upgraded PromptKit, adopted inherited profiles and -current credential handling, added repair and provider-failure contracts, -enabled one corrective generation for v2.1.0 prompts, exposed repair -provenance in ordinary and comparison results, migrated comparison bundles to -v2, and added secure failure-debug capture. - -## Purpose And Authority - -This plan translates the accepted -[PromptKit v0.8.0 upgrade roadmap](promptkit-v0.8.0.md) into an ordered, -decision-complete implementation procedure. The feature roadmap owns purpose, -scope, policy, and the desired end state. This document owns implementation -order, concrete work allocation, stage boundaries, and verification until the -upgrade is complete. - -The implementing agent must complete the stages in numerical order. Each stage -is sized for one focused prompt handled by `gpt-5.6-terra` with high reasoning. -Do not combine stages merely because adjacent work touches the same package. - -## Locked Decisions - -The following decisions are final for this implementation: - -- upgrade directly from PromptKit `v0.5.0` to `v0.8.0`; -- declare one corrective call in each embedded prompt through - `repair_attempts: 1` rather than adding WeatherReporter repair logic; -- keep the repair budget in the exact embedded prompt definition and add no - global, per-report, CLI, profile, or operator configuration override; -- advance all four exact prompt versions from `2.0.0` to `2.1.0`; -- make `weather-light`, `weather-balanced`, and `weather-deep` minimal aliases - of the corresponding PromptKit built-ins through `base_profile`; -- retain the existing report-to-profile assignments and effective model - ladder; -- allow successfully inspected endpoint-only profiles to have an empty backend - ID while continuing to require a nonblank model; -- treat `APIKeyEnv` as an optional lookup source and reject only profiles that - report `APIKeyRequired`, because WeatherReporter supplies no direct request - credential; -- support PromptKit's built-in `rakestrawhome-gemma-4-31b` profile without - WeatherReporter-specific backend configuration; -- expose provider HTTP status through a project-owned safe generation error, - while writing provider code, type, and message only to explicit secure debug - capture; -- emit only `weatherreporter.comparison.v2`, with repair provenance, and do not - preserve v1 guarded-replacement support; and -- retain PromptKit dependency types inside the adapter and preserve all - stateless execution, atomic publication, comparison independence, and - disclosure invariants. - -## Implementation Rules - -For every stage: - -- read `docs/development.md`, all files under `docs/policy/`, this plan, the - feature roadmap, and the task-specific documents named by the stage; -- inspect the current code and tests before editing; use the repository's code - knowledge graph first for code discovery and fall back to text search for - literals, assets, and documentation; -- implement only the stage's scope and preserve unrelated user changes; -- keep PromptKit/provider types, client construction, YAML parsing, repair - mechanics, and provider transport inside the existing adapter boundary; -- use deterministic, offline, credential-free tests and injected clients or - synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local - endpoint calls; -- add tests at the narrowest stable owner identified by the testing policy and - avoid copying PromptKit's internal test matrices; -- update the canonical documentation owners listed for that stage in the same - change as the implemented contract; -- run `gofmt` on changed Go files, the stage's focused tests, - `GOWORK=off go test -count=1 ./...`, and `git diff --check`; and -- leave the repository passing before proceeding to the next stage. - -Stages affecting concurrent comparison, cancellation, or secure debug -filesystem work must also run the named focused packages with `-race`. Do not -weaken an existing assertion solely to accommodate the new dependency. When a -test encodes an intentionally changed contract, replace it with a behavioral -assertion for the accepted policy. - -## Implementation Stages - -### Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline - -Status: Complete. - -Purpose: move to the tagged dependency and isolate compatibility changes before -adopting new WeatherReporter behavior. - -Work: - -1. Update `go.mod` to require - `gitea.maximumdirect.net/eric/promptkit v0.8.0` and refresh `go.sum` with - `GOWORK=off go mod tidy`. Do not add `go.work`, `vendor`, or a `replace` - directive and do not change WeatherReporter's Go version. -2. Resolve any compile failures using PromptKit's public root package only. - Keep all `Profile` and `OpenAICompatibleProfileConfig` literals keyed. Do not - register the now-reserved `rakestrawhome` backend. -3. Reconcile adapter tests that directly exercise PromptKit's changed optional - credential behavior. A profile whose only credential metadata is - `api_key_env` must reach an injected client when the environment value is - absent; it must no longer expect PromptKit to return - `ErrAPIKeyEnvMissing`. Do not change WeatherReporter's application preflight - in this stage. -4. Verify that every current embedded prompt, content file, schema, and fallback - profile inspects under v0.8.0. Verify selected invalid local endpoints and - malformed selected profile definitions still map to project-owned - configuration or profile-load categories. -5. Review the v0.6.0 compatibility corrections against supported - WeatherReporter inputs: metadata-authoritative identities, exact contained - `content_file` paths, regular embedded files, structurally valid endpoints, - bounded JSON-compatible values, cancellation identity, and strict response - framing. Add consumer tests only for a WeatherReporter boundary not already - protected by PromptKit. - -Do not enable profile inheritance or output repair yet. The expected result is -the current WeatherReporter feature set running against PromptKit v0.8.0. - -Focused verification: - -```sh -GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets -GOWORK=off go test -race -count=1 ./internal/adapters/promptkit -``` - -### Stage 2: Adopt Profile Inheritance And Current Credential Routing - -Status: Complete. - -Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional -credential semantics, and the Rakestrawhome built-in without changing the model -ladder. - -Work: - -1. Replace the three embedded profile bodies with these exact leaf/base - relationships and no duplicated execution settings: - - | Leaf | Base | - | --- | --- | - | `weather-light` | `deepseek-4-flash` | - | `weather-balanced` | `gemini-flash-latest` | - | `weather-deep` | `claude-sonnet-latest` | - -2. Update prompt-asset fixtures and tests to understand `base_profile`. Assert - that all three leaf IDs remain selected identities and resolve to the same - backend, model, timeout, service tier, and reasoning settings exposed by the - current standalone definitions. Test relationships and effective behavior, - not copied private constants beyond the intentional model-ladder contract. -3. Preserve source precedence. Cover a standalone same-ID operator override, a - derived operator override, a configured source that shadows a base ID, and - selected missing-base, cyclic, malformed-base, and incomplete-target - failures. Do not implement inheritance or merging in WeatherReporter; all - resolution must remain PromptKit-owned. -4. Change application profile preflight to accept a successful inspection with - a nonblank model and an empty backend ID. Trust PromptKit inspection to have - resolved either a backend or endpoint; do not add the endpoint to - `promptexec.ProfileInspection` or ordinary provenance. -5. Remove application-level environment lookup and rejection for a nonblank - `APIKeyEnv`. Remove the now-unused `LookupEnv` fields and plumbing from - prompt, batch, and comparison inspection requests. Continue rejecting - `CredentialRequired`/`APIKeyRequired` before weather collection with the - existing missing-credential category. -6. Add an end-to-end offline regression proving the maintained endpoint-only - `weather-light` example passes application inspection, retains an empty - backend ID, and does not expose its endpoint. -7. Prove `rakestrawhome-gemma-4-31b` can pass ordinary and comparison profile - inspection through the existing adapter and reports the PromptKit - `rakestrawhome` backend ID. Do not make a provider call or add - Rakestrawhome-specific configuration. - -Canonical documentation in this stage: - -- update `docs/policy/architecture.md` so only direct-key-required profiles - fail credential preflight and backend identity is optional for endpoint-only - profiles; -- update `docs/config.md` to distinguish same-ID source replacement from - `base_profile` chain inheritance and to describe optional environment - credentials; -- update `docs/integrations/promptkit.md` for profile composition, parent - lookup precedence, endpoint-only identity, optional credentials, and - Rakestrawhome availability; and -- update `docs/internal/promptkit-adapter.md` and focused app internals for the - implemented inspection behavior. - -Focused verification: - -```sh -GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app -GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app -``` - -### Stage 3: Extend The Project-Owned Prompt Execution Contract - -Status: Complete. - -Purpose: establish dependency-neutral repair and structured-generation-error -values before the adapter or application relies on them. - -Work: - -1. Add `RepairAttempts int` to `promptexec.OutputContract`. It is the configured - additional-call budget from the exact prompt contract. -2. Add `RepairAttempts int` to `promptexec.Validation`. It is the number of - corrective calls actually started for the completed result. Update - `NewValidation` and every caller so construction is explicit; reject or - normalize no values here because PromptKit owns output-contract validity. -3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so - repair values are retained without sharing mutable state. -4. Add a project-owned immutable `promptexec.GenerationError` with unexported - status and provider-detail fields plus safe accessors: - - - `StatusCode() int` - - `ProviderCode() string` - - `ProviderType() string` - - `ProviderMessage() string` - - `Category() ErrorCategory`, always returning `Generation` - - `Error()`, exposing only the WeatherReporter generation category/message - and optional HTTP status - - `GoString()`, returning the same safe representation - - `Unwrap()`, preserving a project-owned `*promptexec.Error` - -5. Provide one constructor used by adapters. Defensively normalize valid UTF-8 - and bound code/type to 256 Unicode code points and message to 4,096 Unicode - code points, even though PromptKit already bounds its accessors. Do not - expose fields through struct formatting, JSON tags, or exported mutable - fields. Preserve the dependency cause only behind the project-owned error so - `errors.Is`/`errors.As` identities remain available without entering error - text. -6. Add focused tests proving nil/zero safety, category and unwrap behavior, - status-only ordinary formatting, `%#v` redaction, provider-detail bounds, - and repair-value copying. - -Do not import PromptKit from `internal/promptexec` and do not change CLI or -artifact schemas in this stage. - -Focused verification: - -```sh -GOWORK=off go test -count=1 ./internal/promptexec -``` - -### Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter - -Status: Complete. - -Purpose: make the adapter faithfully translate v0.8.0 preparation, execution, -validation, usage, and failure values into the Stage 3 contract. - -Work: - -1. Map `promptkit.OutputContract.RepairAttempts` in prompt inspection and - prepared-execution details. Map - `promptkit.ValidationResult.RepairAttempts` in completed execution. -2. Preserve PromptKit's cumulative usage exactly as reported across the initial - call and every completed correction. Continue returning only the final raw - candidate and final validation result, subject to WeatherReporter's 64 KiB - generated-output bound. -3. In adapter error classification, retain cancellation, deadline, and capacity - precedence. Before the generic `ErrLLMGenerate` branch, use `errors.As` for - `*promptkit.GenerationError` and construct the project-owned - `promptexec.GenerationError` with status, code, type, message, and hidden - cause. Initial and corrective generation failures use the same mapping. -4. Extend the injected adapter client used by tests so it can return an ordered - sequence of responses or errors and record each request safely. -5. Use a synthetic PromptKit prompt with JSON Schema validation and - `repair_attempts: 1` to cover: - - - first-pass valid output with zero corrections; - - explicitly empty or invalid output followed by valid corrected output; - - one-attempt exhaustion returning a final failed validation result rather - than an operational error; - - a non-2xx-style `GenerationError` during correction; - - cumulative token usage and actual repair count; and - - the same prepared prompt/profile identity across the corrective flow. - -6. Keep these tests at the adapter boundary. Do not assert PromptKit's private - corrective-message wording or reconstruct its internal repair algorithm. - -Canonical documentation in this stage: update -`docs/internal/promptkit-adapter.md` for the repair/result/error mappings. Do -not yet claim that embedded WeatherReporter prompts enable repair. - -Focused verification: - -```sh -GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec -GOWORK=off go test -race -count=1 ./internal/adapters/promptkit -``` - -### Stage 5: Carry Repair Provenance Through Application Workflows - -Status: Complete. - -Purpose: make application orchestration understand configured and actual repair -counts before changing the embedded prompt policy. - -Work: - -1. Add an expected generated-text repair budget to `report.Definition` and set - it explicitly to zero for all four current `2.0.0` definitions in this - stage. Include it in report-definition validation and retained contract - tests. -2. Extend exact prompt preflight so format, validation mode, schema path, and - repair budget must all match the resolved report definition. Extend - preparation and completion provenance checks to require the same repair - budget across inspection and the opaque prepared snapshot. -3. Add `RepairAttempts *int` to application outcomes where execution may fail - before validation exists. Set it to a fresh pointer immediately after a - non-nil completed execution is returned, before WeatherReporter's secondary - generated-text validation. A pointer is required so completed first-pass - zero is distinguishable from unavailable provenance. -4. Carry independent copies through `ReportResult`, `BatchReportResult`, batch - conversion, comparison execution's internal outcome, and relevant test - fakes. Do not expose the new value in CLI or comparison JSON yet. -5. Preserve the actual count on PromptKit validation rejection and on later - WeatherReporter generated-text or render failures. Leave it unavailable on - preparation, capacity, cancellation, deadline, and generation errors that - return no completed PromptKit result. -6. Add focused tests for provenance mismatch, completed zero, completed - positive, validation rejection, later local validation failure, early - operational failure, batch copying, and independent concurrent profile - outcomes. - -Canonical documentation in this stage: update the focused prepared-report and -app-orchestration internals to describe configured versus actual repair -provenance. Current public documents should continue to report the embedded -budget as zero until Stage 6. - -Focused verification: - -```sh -GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli -GOWORK=off go test -race -count=1 ./internal/app -``` - -### Stage 6: Activate One Repair And Expose Ordinary Result Provenance - -Status: Complete. - -Purpose: switch the operational prompts to the accepted one-correction policy -and make ordinary generate/run/batch output report what occurred. - -Work: - -1. Add `repair_attempts: 1` to the output contract of all four embedded prompt - definitions and change each exact prompt version from `2.0.0` to `2.1.0`. - Do not change prompt text or generated-text schemas solely for this upgrade. -2. Change all four report registry definitions to exact prompt version `2.1.0` - and expected repair budget one. Update exact-version fixtures and assertions - throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests - that classify `repair_attempts` as a retired setting and replace them with - an exact one-attempt contract assertion. -3. Add `repairAttempts` to successful and failed generate and batch JSON result - shapes through the Stage 5 pointers. Emit integer zero for a completed - first-pass result, a positive integer for a completed repaired result, and - omit the field when no completed validation made it available. -4. Keep the existing `validationStatus` and failure categories authoritative. - A repaired valid result proceeds normally. Repair exhaustion remains - `validation_rejected`, publishes no report for that profile, and retains the - actual attempt count. -5. Add representative offline assembled tests proving first-pass success, - repaired success, exhaustion, explicit empty initial content, and batch - result propagation. Reuse the real PromptKit adapter with an injected - sequence client for at least one end-to-end repaired execution; use the - existing app fake at other boundaries where lower-level repair is already - covered. -6. Confirm no application loop, provider retry, profile fallback, or - request-level `OutputContract` override was introduced. - -Canonical documentation in this stage: - -- update `docs/policy/architecture.md` with PromptKit-owned bounded repair and - failed-exhaustion invariants; -- update `docs/integrations/promptkit.md` with exact prompt version `2.1.0`, one - configured repair, actual-count semantics, cumulative usage, explicit-empty - handling, and the distinction from operational retries; -- update `docs/cli.md` for generate and batch `repairAttempts` fields; -- update `docs/internal/report-registry.md`, prepared-report internals, and app - orchestration internals for exact version and repair flow; and -- keep configuration documentation unchanged because no repair setting is - added. - -Focused verification: - -```sh -GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli -GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app -``` - -### Stage 7: Migrate Comparison Bundles To v2 - -Status: Complete. - -Purpose: preserve repair activity in the profile-evaluation artifact and make -the strict durable schema change explicit. - -Work: - -1. Change `comparison.SchemaVersion` to - `weatherreporter.comparison.v2`. Emit and recognize v2 only; do not retain a - v1 parser or guarded-replacement compatibility path. -2. Add `RepairAttempts *int` to each application comparison profile result, - CLI comparison profile summary, and durable `comparison.Result`. Propagate a - fresh copy from Stage 5's execution outcome. -3. Place `repairAttempts` immediately after `validationStatus` in the canonical - result-object JSON field order. Encode zero for completed first-pass - validation, a positive integer for completed correction, and omit it only - when no completed validation exists. -4. Tighten manifest invariants: every non-nil repair count is non-negative; a - successful result must have `validationStatus: "passed"` and a non-nil - repair count; a failed result with a completed validation status must also - have a non-nil count; and an early operational failure may omit both. -5. Update the strict token-level JSON recognizer to accept only the canonical - `repairAttempts` field at its correct object level, reject duplicate, - unknown, negative, fractional, string, overflow, and malformed values, and - continue rejecting v1 as an unsupported current bundle. -6. Update manifest construction, cloning, validation, exact serialization - tests, guarded replacement tests, malicious bundle tests, partial-success - tests, and CLI comparison summaries. Preserve flat layout, result ordering, - hashes, atomic publication, cancellation safety, and no Distributor calls. -7. Cover concurrent peers where one succeeds first-pass, one repairs, one - exhausts, and one fails operationally. The counts must remain attached to - the selected profile positions without races or cross-contamination. - -Canonical documentation in this stage: - -- replace the v1 contract in `docs/integrations/comparison-bundle.md` with v2, - including exact field order, presence rules, and the lack of v1 replacement - compatibility; -- update `docs/cli.md` for comparison `repairAttempts`; -- update `docs/operations.md` to tell operators to move or remove an existing - v1 bundle before replacing at the same destination; and -- update comparison execution/publication internals and architecture policy as - needed for the current-only version invariant. - -Focused verification: - -```sh -GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli -GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app -``` - -### Stage 8: Add Secure Provider-Failure Debug Capture - -Status: Complete. - -Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the -existing explicit secure debug boundary while keeping ordinary errors safe. - -Work: - -1. Extend prompt preparation debug output with configured - `repairAttempts` and advance its schema identifier from - `weatherreporter.prompt_preparation_debug.v2` to - `weatherreporter.prompt_preparation_debug.v3`. -2. Extend execution validation debug output with actual `repairAttempts` and - advance its schema identifier from - `weatherreporter.prompt_execution_debug.v2` to - `weatherreporter.prompt_execution_debug.v3`. Retain cumulative token usage. -3. Add a dedicated `failure.json` artifact with schema identifier - `weatherreporter.prompt_failure_debug.v1`. Its canonical fields are: - - - top level: `schemaVersion`, `reportId`, `validDate`, `runId`, `failure`; - - failure object: `category`, `statusCode`, `providerCode`, `providerType`, - `providerMessage`; - - omit absent provider fields and zero status; and - - never include the raw provider body, headers, endpoint, credentials, - request, schema, rendered prompt, or generated candidate. - -4. Add `PromptDebugWriter.WriteFailure` using the existing handle-relative - secure run directory, `0700` directory and `0600` file modes, canonical JSON - encoding, and no-follow/atomic replacement behavior. Disabled writers must - perform no filesystem work. -5. When execution returns an error, use `errors.As` only against the - project-owned `*promptexec.GenerationError`. If explicit debug capture is - enabled, write `failure.json` using that profile's existing debug reference. - This applies equally to initial and corrective provider failures and keeps - comparison profile directories isolated. -6. If failure-debug writing also fails, retain the generation failure as the - primary categorized error and join the safe debug-write failure rather than - replacing or hiding the provider failure. Never place provider code, type, - or message in the joined error text. -7. Ordinary generate, batch, and comparison errors should gain only the safe - HTTP status already rendered by `promptexec.GenerationError.Error`; do not - add provider detail fields to CLI summaries, comparison manifests, logs, or - Distributor requests. -8. Add adversarial tests for formatter redaction, malicious provider strings, - JSON escaping, bounds, absent fields, file modes, symlink/path attacks, - write failure, cancellation identity, initial versus corrective failures, - and concurrent comparison captures. - -Canonical documentation in this stage: - -- update `docs/operations.md` with the three debug artifact versions, - `failure.json`, sensitivity, permissions, and retention; -- update `docs/integrations/promptkit.md` with ordinary status-only disclosure - and debug-only provider detail; -- update prompt-debug, PromptKit-adapter, and app-orchestration internals; and -- ensure `docs/policy/architecture.md` explicitly prohibits provider-controlled - diagnostics from ordinary outputs. - -Focused verification: - -```sh -GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli -GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app -``` - -### Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit - -Status: Complete. - -Purpose: verify the complete end state as one coherent WeatherReporter feature -and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile, -repair, or debug claims. - -Work: - -1. Re-read the feature roadmap, `docs/development.md`, every policy document, - and every canonical document changed by Stages 1-8. Reconcile them against - executable behavior and remove duplicated or stale definitions. Keep - unimplemented future ideas in `docs/roadmap/future.md`, not current-state - documents. -2. Search code, embedded assets, examples, tests, and documentation for stale - contractual literals and review every occurrence of: - - - PromptKit `v0.5.0`, `v0.6.0`, and `v0.7.0` as an active dependency claim; - - prompt version `2.0.0`; - - `weatherreporter.comparison.v1`; - - prompt debug schema v2 identifiers; - - claims that profile fields never inherit; - - claims that every `APIKeyEnv` must be populated; - - claims that backend ID is always required; - - claims that repair is disabled or `repair_attempts` is retired; and - - provider detail in ordinary output. - - Historical release documents may retain accurate historical literals. -3. Verify canonical ownership: - - - architecture owns invariants and boundaries; - - config owns operator profile and credential behavior, but no repair field; - - PromptKit integration owns logical prompt/profile/output contracts; - - CLI owns result fields; - - operations owns explicit debug handling and old comparison-bundle cleanup; - - comparison integration owns the complete v2 manifest; and - - internal documents own implementation flow without duplicating the public - references. - -4. Verify maintained examples remain valid, secret-free, and tested. The local - `weather-light` example remains a standalone endpoint-only profile rather - than inheriting an OpenRouter backend it cannot clear. -5. Review the complete diff for architecture leakage. Production packages - outside `internal/adapters/promptkit` must not import PromptKit; no - application repair loop, provider client, raw provider diagnostic, profile - YAML parser, or durable application state may have appeared. -6. Review tests under the testing policy. Keep consumer contract and regression - coverage, remove accidental duplication of upstream implementation tests, - and ensure every default test is offline and repeatable. -7. Run the complete validation set: - -```sh -gofmt -w -GOWORK=off go test -count=1 ./... -GOWORK=off go test -race -count=1 ./... -GOWORK=off go vet ./... -GOWORK=off go build ./... -GOWORK=off go mod tidy -diff -go run ./cmd/weatherreporter --help -go run ./cmd/weatherreporter compare --help -test -z "$(git ls-files go.work go.work.sum)" -test ! -e vendor -git diff --check -``` - -8. Confirm `go.mod` has no `replace`, the resolved PromptKit module is exactly - v0.8.0, and no live credential or provider call occurred during validation. -9. After every check passes, update this plan's status to Completed and add a - concise completion note listing the implemented stages. Do not delete either - roadmap until the maintainer has reviewed the implementation. Do not create - a release document or tag; release preparation remains a separate maintainer - action once a version is selected. - -## Completion Standard - -The implementation is complete only when all nine stages pass their focused -and repository-wide checks, all locked decisions are observable in code and -canonical documentation, and the feature roadmap's completion criteria are -satisfied. Passing compilation alone is insufficient. The final state must -demonstrate repaired success, repair exhaustion, comparison provenance, -endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome -inspection, safe ordinary provider failures, secure debug-only detail, and -unchanged publication and concurrency invariants. diff --git a/docs/roadmap/promptkit-v0.8.0.md b/docs/roadmap/promptkit-v0.8.0.md deleted file mode 100644 index a13178c..0000000 --- a/docs/roadmap/promptkit-v0.8.0.md +++ /dev/null @@ -1,488 +0,0 @@ -# PromptKit v0.8.0 Upgrade Roadmap - -Status: Implemented. - -## Purpose - -WeatherReporter should upgrade its PromptKit dependency from `v0.5.0` to -`v0.8.0` and deliberately adopt the useful consumer-facing capabilities added -in `v0.6.0`, `v0.7.0`, and `v0.8.0`. The upgrade should improve output-contract -reliability, profile composition, local and alternate endpoint support, and -provider-failure diagnosis without moving PromptKit responsibilities into -WeatherReporter or weakening the application's stateless and security -boundaries. - -This roadmap defines the intended scope, policy, and end state. The -[implementation plan](implementation.md) owns the procedure for reaching that -state. - -## User Intent - -The upgrade is intended to: - -- use PromptKit's bounded output repair to recover from occasional malformed - structured weather prose; -- keep WeatherReporter's domain profile IDs stable while inheriting maintained - PromptKit model definitions; -- make PromptKit's additional built-in backend and profile available for - explicit generation and profile comparisons; -- support unauthenticated or optionally authenticated OpenAI-compatible - endpoints without inventing a WeatherReporter transport layer; -- make provider HTTP failures more actionable under an explicit - WeatherReporter disclosure policy; and -- receive PromptKit's intervening correctness, safety, cancellation, resource, - and efficiency improvements as part of one tested dependency upgrade. - -The model ladder and report assignments do not change as part of this work: -Hourly continues to select `weather-light`; Daily, Today, and Tomorrow continue -to select `weather-balanced`; and `weather-deep` remains available for explicit -selection. This upgrade does not promote the new Rakestrawhome profile into -that default ladder. - -## Current State - -WeatherReporter currently depends on -`gitea.maximumdirect.net/eric/promptkit` at `v0.5.0`. The PromptKit adapter -supplies embedded prompts, JSON Schemas, and -application-fallback profiles, plus an optional configured profile source and -the conventional local backend. - -The four generated-text prompts are exact version `2.0.0` JSON Schema prompts. -They omit `repair_attempts`, so execution is single-pass. The project-owned -`promptexec.OutputContract` and validation result also omit repair budgets and -actual repair counts. - -The three embedded WeatherReporter profiles duplicate the effective fields of -these PromptKit built-ins: - -| WeatherReporter profile | PromptKit built-in with the same target | -| --- | --- | -| `weather-light` | `deepseek-4-flash` | -| `weather-balanced` | `gemini-flash-latest` | -| `weather-deep` | `claude-sonnet-latest` | - -WeatherReporter preflights any nonblank `api_key_env` as a required credential, -even though PromptKit v0.7.0 distinguishes an optional environment source from -an explicit `APIKeyRequired` target. Provider generation failures are reduced -to WeatherReporter's safe `generation` category; the PromptKit dependency error -is retained as a hidden cause, but its structured HTTP status and provider -diagnostics are not mapped into project-owned values. - -The maintained `weather-light` local override is an endpoint-only profile, and -the configuration contract says endpoint-only profiles are supported. PromptKit -inspection correctly reports no backend ID for that form, but WeatherReporter -application preflight currently requires both a nonblank backend and model. -That mismatch prevents the documented example from reaching generation and -should be corrected as part of adopting the current PromptKit target contract. - -## Upstream Release Assessment - -### PromptKit v0.6.0 - -`v0.6.0` adds no public declarations, but it is a material compatibility and -safety release. It centralizes execution-setting, output-contract, endpoint, -and JSON-compatible-value validation; makes YAML metadata authoritative for -prompt and profile identity; hardens `content_file` containment and regular-file -requirements; validates OpenAI-compatible endpoints structurally; bounds JSON -trees and successful provider bodies; requires exactly one JSON value in -provider responses; preserves cancellation and transport error identities; and -reuses schema and rendered-artifact work within an operation. - -WeatherReporter should receive these improvements directly from the dependency -and audit its own supported assets and configuration paths against the stricter -contracts. It should not duplicate PromptKit's internal validators or tests. -The existing embedded prompt paths, inline data-package input, generated-output -limit, and adapter boundary remain conceptually correct. - -### PromptKit v0.7.0 - -`v0.7.0` adds four potentially useful consumer features: - -- linear, cycle-safe profile inheritance through `base_profile` and - `Profile.BaseProfileID`; -- the built-in `rakestrawhome` backend and - `rakestrawhome-gemma-4-31b` profile; -- optional API-key environment sources, with `APIKeyRequired` reserved for an - explicit local credential requirement; and -- bounded structured `GenerationError` details for non-2xx responses from the - built-in OpenAI-compatible client. - -WeatherReporter has no manual `rakestrawhome` registration and uses keyed -PromptKit profile literals, so the two source-compatibility hazards called out -by the release do not require migration shims. The profile, credential, and -error features do require deliberate application-policy choices described -below. - -### PromptKit v0.8.0 - -`v0.8.0` activates the existing output-contract repair budget. A positive -`repair_attempts` value authorizes up to that many corrective model calls after -eligible `basic`, `json`, or `json_schema` validation failures. The supported -budget is zero through three. Repairs preserve the original rendered -conversation, effective target, session, structured-output contract, and -backend capacity policy. The final result reports cumulative token usage and -the number of corrective calls actually made. - -Repair exhaustion is a completed generation with failed validation, not an -operational error. WeatherReporter's existing policy should continue to reject -that result and publish no report for that profile. Explicit empty provider -content now reaches output validation; for WeatherReporter's JSON Schema -prompts it is therefore eligible for repair rather than being misclassified as -a malformed provider envelope. - -## Desired End State - -WeatherReporter builds and tests against PromptKit `v0.8.0` with no workspace, -vendor, or module replacement dependency. Its public behavior remains -stateless, its PromptKit dependency types remain confined to the adapter, and -its ordinary summaries and logs remain safe. - -The completed integration: - -- benefits from the v0.6.0 safety and efficiency corrections; -- composes WeatherReporter domain profiles from PromptKit's maintained built-in - profiles while preserving WeatherReporter-owned leaf IDs and operator - override precedence; -- accepts a successfully inspected endpoint-only profile with a nonblank model - even though it has no logical backend ID; -- permits explicit use of PromptKit's Rakestrawhome profile without custom - backend wiring; -- applies an accepted bounded-repair policy to every operational structured - prompt; -- validates repair configuration during preflight and records actual repair - activity in project-owned result values; -- retains PromptKit's cumulative usage accounting in explicit debug output; -- distinguishes safe provider HTTP status from potentially sensitive provider - diagnostics; and -- documents the changed profile, credential, repair, comparison, debug, and - failure contracts in their canonical owners. - -## Dependency And Compatibility Policy - -The module requirement should move directly from `v0.5.0` to `v0.8.0`, followed -by a clean module tidy. WeatherReporter already requires Go 1.26 while PromptKit -`v0.8.0` requires Go 1.25.5, so no Go version change is needed for this upgrade. - -Consumer validation must cover the paths called out by PromptKit v0.6.0: - -- every embedded prompt, content file, schema, and fallback profile inspects - through PromptKit `v0.8.0`; -- configured single-file and directory profile sources retain their lazy, - metadata-authoritative identity and precedence behavior; -- malformed selected profiles and invalid local endpoints retain actionable - WeatherReporter categories; -- the inline YAML data package and prepared-execution path remain within the - new JSON and response bounds; and -- cancellation, deadline, and backend-capacity identities still cross the - adapter correctly. - -PromptKit owns its 16 MiB successful transport-response bound and JSON framing. -WeatherReporter retains its stricter 64 KiB generated-text acceptance bound. -The consumer suite should protect that relationship without reproducing -PromptKit's lower-level transport matrix. - -## Domain Profile Composition - -The embedded profiles should become application-owned aliases: - -```yaml -id: weather-light -base_profile: deepseek-4-flash -``` - -```yaml -id: weather-balanced -base_profile: gemini-flash-latest -``` - -```yaml -id: weather-deep -base_profile: claude-sonnet-latest -``` - -The effective backend, model, timeout, service tier, and reasoning settings -must initially remain identical to the current WeatherReporter definitions. -The selected leaf remains the durable logical profile identity even though its -effective target is inherited. - -Profile source precedence remains: - -1. explicit in-memory profiles used by tests or embedding consumers; -2. the configured `profile_file` or `profile_dir` source; -3. WeatherReporter's embedded fallback catalog; and -4. PromptKit's built-in catalog. - -Sources still do not merge definitions of the same ID. Once a selected -definition names `base_profile`, however, each parent ID is resolved through -that same precedence order and the resulting linear chain is merged from root -to leaf according to PromptKit's inheritance contract. Documentation must make -that distinction explicit. A malformed leaf, missing or malformed base, cycle, -overlong chain, or incomplete resolved target fails profile inspection before -weather collection. - -An operator may continue to replace `weather-light`, `weather-balanced`, or -`weather-deep` with a standalone definition. An operator may also define a -derived replacement. The maintained local endpoint example should remain -standalone because PromptKit profile inheritance has no clearing syntax: using -an OpenRouter base would retain its backend identity and capacity policy even -when the child replaces the endpoint. - -WeatherReporter should treat the adapter's successful profile inspection as -authoritative that PromptKit resolved a usable route. A nonblank model remains -required, but backend ID is optional for an endpoint-only profile and should be -omitted from safe provenance where unavailable. WeatherReporter still must not -surface the endpoint outside explicit debug capture. This aligns application -preflight with PromptKit and with the existing CLI, comparison, and debug value -shapes, all of which already permit an absent backend identity. - -## Rakestrawhome Availability - -The reserved `rakestrawhome` backend and built-in -`rakestrawhome-gemma-4-31b` profile should be supported automatically through -ordinary PromptKit selection. Operators may choose that profile with the -existing global profile setting or as one entry in `compare`, and PromptKit's -backend capacity policy remains authoritative. - -WeatherReporter should not register, wrap, or duplicate the backend or profile, -and should not add a Rakestrawhome-specific configuration field. Its canonical -PromptKit integration documentation should link to PromptKit for the current -built-in catalog and credential contract rather than copying volatile endpoint -or capacity values. Offline inspection coverage should prove that the built-in -profile crosses the WeatherReporter adapter with the expected logical backend -identity. - -## Bounded Structured-Output Repair - -The accepted repair budget belongs to the exact PromptKit output contract, not -to a WeatherReporter retry loop. PromptKit alone should construct corrective -messages, perform additional calls, enforce the budget, aggregate usage, and -coordinate backend capacity. WeatherReporter must not retry provider failures, -switch profiles, or layer another repair mechanism around `RunPrepared`. - -All four embedded prompt definitions should declare `repair_attempts: 1`. -Because this changes prompt -execution behavior, latency, cost, hash, and provenance, each definition and -its report-registry binding should advance from exact version `2.0.0` to -`2.1.0`. Prompt text and generated-text schemas do not need to change solely -for this feature. - -The embedded prompt definition is the per-report pipeline policy owner. This -upgrade should not add a global or per-report operator configuration field for -repair attempts and should not construct a request-level replacement output -contract. A future pipeline may select another budget only through a deliberate -prompt-definition and exact-version change. - -The project-owned PromptKit boundary should retain: - -- the configured repair budget in prompt inspection and preparation output - contracts; -- the number of corrective calls actually made in completed validation; -- cumulative PromptKit token usage across initial and corrective calls; and -- the final candidate and final validation result only, consistent with the - PromptKit contract. - -Prompt inspection and preparation provenance must require the repair budget to -match the exact expected prompt definition just as they currently require the -format, validation mode, and schema path to match. A zero-attempt successful -result is normal when the first candidate passes. A repair-exhausted result -continues through WeatherReporter's ordinary `validation_rejected` failure -path, and an operational or generation failure during correction remains that -profile's ordinary operational failure. - -For concurrent comparison, every profile should use the same prompt repair -budget. A corrective call remains part of that profile's one prepared -execution and uses PromptKit's existing backend capacity pool. One profile's -repair or failure must not cancel independent peers. - -## Repair Observability And Comparison Contract - -The actual repair count is safe operational provenance and should be visible -where WeatherReporter already reports completed validation. Generation, batch, -and comparison action summaries should expose it without exposing candidates, -schemas, or diagnostics. Explicit execution debug output should add it to the -validation object alongside PromptKit's already mapped cumulative usage. - -Profile comparison needs this value in `comparison.json`: a successful result -that required correction is materially different from a first-pass success -when evaluating model reliability, latency, and cost. The manifest should -therefore advance to `weatherreporter.comparison.v2` and add a non-negative -`repairAttempts` field to each result. The field is zero when no corrective -call began, including ordinary first-pass success. A failure carries the count -when PromptKit returned a completed validation result; it is omitted only when -execution failed before a completed validation result made the value known. - -The v2 manifest should remain flat, strict, deterministic, and atomically -published. WeatherReporter does not need to preserve v1 replacement -compatibility: comparison bundles are operator-owned development outputs, and -the current integration contract intentionally recognizes only its current -schema. The release notes and comparison documentation must call out the -version change so an operator can remove or relocate an older bundle before -using guarded replacement at the same destination. - -## Credential Semantics - -PromptKit v0.7.0 treats `APIKeyEnv` as an optional lookup source. If the -environment variable is absent or blank and no direct credential is supplied, -the built-in client omits `Authorization` and lets the endpoint respond. -`APIKeyRequired` is the distinct signal that a usable credential must be -provided locally. - -WeatherReporter cannot supply PromptKit's request-scoped direct API-key value, -so a profile reporting `APIKeyRequired` remains unsupported and must fail -before weather collection. WeatherReporter should not require a nonblank value -for an optional `APIKeyEnv` during application preflight. The built-in client -should omit `Authorization` when that source is unavailable and let the -endpoint return any authentication failure through the ordinary structured -generation-error path. - -## Structured Generation Failures - -PromptKit v0.7.0's `GenerationError` can report a provider HTTP status plus -bounded provider code, type, and message. WeatherReporter should consume that -type only inside the PromptKit adapter and map any adopted fields into a -project-owned immutable error. PromptKit dependency types must not become app -or CLI contracts. - -HTTP status is safe enough for ordinary diagnostics. Provider code, type, and -message remain untrusted and may contain request or schema fragments. They -must never enter ordinary errors, action summaries, comparison manifests, -logs, generated reports, or Distributor payloads. The full structured -diagnostic belongs only in an explicitly requested secure `--llm-debug-dir` -`failure.json` artifact. That artifact may contain PromptKit's normalized -bounded fields but never the raw provider body, headers, endpoint, credentials, -or reconstructed request. - -Initial-call and corrective-call non-2xx responses should follow the same -mapping. Cancellation and deadline categories continue to take precedence over -provider classification where PromptKit preserves those identities. - -## Testing Policy - -The default suite must remain deterministic, offline, and credential-free. -Use injected PromptKit clients and synthetic embedded or temporary assets for -consumer behavior; do not call OpenRouter, Rakestrawhome, or a local endpoint. - -Risk-based coverage should include: - -- all embedded prompts and inherited domain profiles inspecting successfully - under PromptKit `v0.8.0`; -- unchanged effective targets and report-to-profile assignments after the - alias refactor; -- external standalone and derived profile precedence, plus selected missing, - cyclic, and malformed-base failures at the WeatherReporter boundary; -- end-to-end preflight and prepared execution through the maintained - endpoint-only `weather-light` override without exposing its endpoint; -- offline inspection of `rakestrawhome-gemma-4-31b`; -- a first-pass valid result with zero repairs; -- an invalid structured result repaired successfully within one corrective - call; -- one-attempt exhaustion returning failed validation and no published report; -- a corrective generation failure retaining its safe category and provider - status policy; -- cumulative usage and actual repair-count mapping; -- comparison peers remaining independent when one profile repairs, exhausts, - or fails; -- v2 comparison manifest validation and guarded replacement; and -- absent or blank optional `APIKeyEnv` values reaching the provider without an - `Authorization` header, while `APIKeyRequired` profiles fail preflight. - -Do not reproduce PromptKit's internal matrices for path traversal, JSON tree -bounds, response framing, inheritance depth, repair prompt construction, or -provider-detail normalization. WeatherReporter tests should protect only its -adapter mappings, application policy, provenance, publication, and public -contracts. Run ordinary and race-enabled repository tests because the repaired -execution path participates in concurrent comparisons. - -## Documentation And Release Impact - -Implementation must update each canonical owner whose contract changes: - -- `docs/policy/architecture.md` for prompt-execution, credential, diagnostic, - and comparison invariants; -- `docs/config.md` and the maintained local profile example for profile-source, - inheritance, and credential semantics; -- `docs/integrations/promptkit.md` for exact prompt versions, repair policy, - profile composition, Rakestrawhome availability, and safe errors; -- `docs/integrations/comparison-bundle.md` for the v2 manifest and repair count; -- `docs/cli.md` for repair-count fields in action summaries; -- `docs/operations.md` for changed failure behavior and any explicit provider - diagnostic capture; -- focused internal PromptKit adapter, app orchestration, prompt-debug, and - comparison documentation; and -- release notes for the dependency jump, prompt version change, possible - additional model call, credential behavior, profile inheritance, diagnostic - behavior, and comparison schema change. - -Current-state documentation must not describe this behavior until the -implementation lands. PromptKit remains the canonical owner of its complete -built-in catalogs, YAML merge rules, transport limits, corrective-message -construction, and public Go API. - -## Scope - -The completed feature includes: - -- the direct module upgrade and tidy dependency graph; -- a v0.6.0 compatibility audit of WeatherReporter's supported PromptKit paths; -- inherited WeatherReporter domain profile definitions with unchanged - effective targets; -- correction of application preflight so PromptKit endpoint-only profiles work - as documented while retaining a required model identity; -- ordinary access to the Rakestrawhome built-in profile; -- PromptKit's optional-credential policy, while direct-key-required profiles - remain unsupported; -- `repair_attempts` on all operational prompts and exact prompt-version bumps; -- project-owned repair budget, actual-attempt, usage, and provenance mappings; -- repair observability in action summaries, explicit debug output, and a v2 - comparison manifest; -- safe provider HTTP status in ordinary errors and bounded provider detail only - in explicit secure debug capture; -- focused offline and race-enabled regression coverage; and -- canonical current-state and release documentation updated with the code. - -## Non-Goals - -This upgrade does not include: - -- application-implemented repair prompts or provider transport; -- retries for HTTP, network, timeout, capacity, or other operational failures; -- automatic profile escalation, fallback, ranking, or resampling; -- changing the weather profile ladder, default report assignments, or concrete - model targets beyond inheriting their maintained PromptKit definitions; -- making Rakestrawhome a default or adding provider-specific configuration; -- live-provider tests or a permanent benchmark framework; -- exposing raw provider responses or sensitive diagnostics routinely; -- a general prompt-source or pipeline plugin system; or -- compatibility shims for PromptKit versions older than `v0.8.0`. - -## Completion Criteria - -The roadmap is complete when: - -- `go.mod` and `go.sum` resolve PromptKit `v0.8.0` without a replacement, - workspace, or vendor tree; -- all PromptKit v0.6.0 compatibility points relevant to WeatherReporter have - been checked and valid supported inputs retain project-owned error identity; -- the three WeatherReporter profiles inherit the intended PromptKit built-ins, - retain their logical IDs, and inspect to the intended effective targets; -- external standalone and inherited overrides obey documented precedence and - failure behavior; -- the maintained endpoint-only local override passes application preflight, - retains an empty backend ID, and keeps its endpoint out of ordinary values; -- `rakestrawhome-gemma-4-31b` is selectable through ordinary generation and - comparison paths without WeatherReporter backend registration; -- every operational prompt has one bounded repair attempt at exact - version `2.1.0`; -- inspection, preparation, execution, debug, and comparison values accurately - preserve configured and actual repair counts; -- first-pass success, repaired success, repair exhaustion, repair generation - failure, and explicit empty content follow the documented outcomes; -- the v2 comparison bundle distinguishes first-pass and repaired results; -- credential preflight accepts absent optional environment credentials while - rejecting direct-key-required profiles, and provider-error disclosure does - not leak sensitive values; -- the default test suite is offline and deterministic, ordinary and race - validation pass, and no redundant upstream implementation suite is copied; - and -- every implemented contract is documented by its canonical current-state - owner and disclosed in the eventual release notes.