weatherreporter/docs/roadmap/implementation.md

PromptKit v0.8.0 Upgrade Implementation Plan

Status: Ready for implementation.

Purpose And Authority

This plan translates the accepted PromptKit v0.8.0 upgrade roadmap into an ordered, decision-complete implementation procedure. The feature roadmap owns purpose, scope, policy, and the desired end state. This document owns implementation order, concrete work allocation, stage boundaries, and verification until the upgrade is complete.

The implementing agent must complete the stages in numerical order. Each stage is sized for one focused prompt handled by gpt-5.6-terra with high reasoning. Do not combine stages merely because adjacent work touches the same package.

Locked Decisions

The following decisions are final for this implementation:

• upgrade directly from PromptKit v0.5.0 to v0.8.0;
• declare one corrective call in each embedded prompt through repair_attempts: 1 rather than adding WeatherReporter repair logic;
• keep the repair budget in the exact embedded prompt definition and add no global, per-report, CLI, profile, or operator configuration override;
• advance all four exact prompt versions from 2.0.0 to 2.1.0;
• make weather-light, weather-balanced, and weather-deep minimal aliases of the corresponding PromptKit built-ins through base_profile;
• retain the existing report-to-profile assignments and effective model ladder;
• allow successfully inspected endpoint-only profiles to have an empty backend ID while continuing to require a nonblank model;
• treat APIKeyEnv as an optional lookup source and reject only profiles that report APIKeyRequired, because WeatherReporter supplies no direct request credential;
• support PromptKit's built-in rakestrawhome-gemma-4-31b profile without WeatherReporter-specific backend configuration;
• expose provider HTTP status through a project-owned safe generation error, while writing provider code, type, and message only to explicit secure debug capture;
• emit only weatherreporter.comparison.v2, with repair provenance, and do not preserve v1 guarded-replacement support; and
• retain PromptKit dependency types inside the adapter and preserve all stateless execution, atomic publication, comparison independence, and disclosure invariants.

Implementation Rules

For every stage:

• read docs/development.md, all files under docs/policy/, this plan, the feature roadmap, and the task-specific documents named by the stage;
• inspect the current code and tests before editing; use the repository's code knowledge graph first for code discovery and fall back to text search for literals, assets, and documentation;
• implement only the stage's scope and preserve unrelated user changes;
• keep PromptKit/provider types, client construction, YAML parsing, repair mechanics, and provider transport inside the existing adapter boundary;
• use deterministic, offline, credential-free tests and injected clients or synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local endpoint calls;
• add tests at the narrowest stable owner identified by the testing policy and avoid copying PromptKit's internal test matrices;
• update the canonical documentation owners listed for that stage in the same change as the implemented contract;
• run gofmt on changed Go files, the stage's focused tests, GOWORK=off go test -count=1 ./..., and git diff --check; and
• leave the repository passing before proceeding to the next stage.

Stages affecting concurrent comparison, cancellation, or secure debug filesystem work must also run the named focused packages with -race. Do not weaken an existing assertion solely to accommodate the new dependency. When a test encodes an intentionally changed contract, replace it with a behavioral assertion for the accepted policy.

Implementation Stages

Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline

Status: Complete.

Purpose: move to the tagged dependency and isolate compatibility changes before adopting new WeatherReporter behavior.

Work:

1. Update go.mod to require gitea.maximumdirect.net/eric/promptkit v0.8.0 and refresh go.sum with GOWORK=off go mod tidy. Do not add go.work, vendor, or a replace directive and do not change WeatherReporter's Go version.
2. Resolve any compile failures using PromptKit's public root package only. Keep all Profile and OpenAICompatibleProfileConfig literals keyed. Do not register the now-reserved rakestrawhome backend.
3. Reconcile adapter tests that directly exercise PromptKit's changed optional credential behavior. A profile whose only credential metadata is api_key_env must reach an injected client when the environment value is absent; it must no longer expect PromptKit to return ErrAPIKeyEnvMissing. Do not change WeatherReporter's application preflight in this stage.
4. Verify that every current embedded prompt, content file, schema, and fallback profile inspects under v0.8.0. Verify selected invalid local endpoints and malformed selected profile definitions still map to project-owned configuration or profile-load categories.
5. Review the v0.6.0 compatibility corrections against supported WeatherReporter inputs: metadata-authoritative identities, exact contained content_file paths, regular embedded files, structurally valid endpoints, bounded JSON-compatible values, cancellation identity, and strict response framing. Add consumer tests only for a WeatherReporter boundary not already protected by PromptKit.

Do not enable profile inheritance or output repair yet. The expected result is the current WeatherReporter feature set running against PromptKit v0.8.0.

Focused verification:

GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit

Stage 2: Adopt Profile Inheritance And Current Credential Routing

Status: Complete.

Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional credential semantics, and the Rakestrawhome built-in without changing the model ladder.

Work:

1. Replace the three embedded profile bodies with these exact leaf/base relationships and no duplicated execution settings:

   Leaf	Base
   weather-light	deepseek-4-flash
   weather-balanced	gemini-flash-latest
   weather-deep	claude-sonnet-latest

2. Update prompt-asset fixtures and tests to understand base_profile. Assert that all three leaf IDs remain selected identities and resolve to the same backend, model, timeout, service tier, and reasoning settings exposed by the current standalone definitions. Test relationships and effective behavior, not copied private constants beyond the intentional model-ladder contract.

3. Preserve source precedence. Cover a standalone same-ID operator override, a derived operator override, a configured source that shadows a base ID, and selected missing-base, cyclic, malformed-base, and incomplete-target failures. Do not implement inheritance or merging in WeatherReporter; all resolution must remain PromptKit-owned.

4. Change application profile preflight to accept a successful inspection with a nonblank model and an empty backend ID. Trust PromptKit inspection to have resolved either a backend or endpoint; do not add the endpoint to promptexec.ProfileInspection or ordinary provenance.

5. Remove application-level environment lookup and rejection for a nonblank APIKeyEnv. Remove the now-unused LookupEnv fields and plumbing from prompt, batch, and comparison inspection requests. Continue rejecting CredentialRequired/APIKeyRequired before weather collection with the existing missing-credential category.

6. Add an end-to-end offline regression proving the maintained endpoint-only weather-light example passes application inspection, retains an empty backend ID, and does not expose its endpoint.

7. Prove rakestrawhome-gemma-4-31b can pass ordinary and comparison profile inspection through the existing adapter and reports the PromptKit rakestrawhome backend ID. Do not make a provider call or add Rakestrawhome-specific configuration.

Canonical documentation in this stage:

• update docs/policy/architecture.md so only direct-key-required profiles fail credential preflight and backend identity is optional for endpoint-only profiles;
• update docs/config.md to distinguish same-ID source replacement from base_profile chain inheritance and to describe optional environment credentials;
• update docs/integrations/promptkit.md for profile composition, parent lookup precedence, endpoint-only identity, optional credentials, and Rakestrawhome availability; and
• update docs/internal/promptkit-adapter.md and focused app internals for the implemented inspection behavior.

Focused verification:

GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app

Stage 3: Extend The Project-Owned Prompt Execution Contract

Status: Complete.

Purpose: establish dependency-neutral repair and structured-generation-error values before the adapter or application relies on them.

Work:

1. Add RepairAttempts int to promptexec.OutputContract. It is the configured additional-call budget from the exact prompt contract.

2. Add RepairAttempts int to promptexec.Validation. It is the number of corrective calls actually started for the completed result. Update NewValidation and every caller so construction is explicit; reject or normalize no values here because PromptKit owns output-contract validity.

3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so repair values are retained without sharing mutable state.

4. Add a project-owned immutable promptexec.GenerationError with unexported status and provider-detail fields plus safe accessors:

   • StatusCode() int
   • ProviderCode() string
   • ProviderType() string
   • ProviderMessage() string
   • Category() ErrorCategory, always returning Generation
   • Error(), exposing only the WeatherReporter generation category/message and optional HTTP status
   • GoString(), returning the same safe representation
   • Unwrap(), preserving a project-owned *promptexec.Error

5. Provide one constructor used by adapters. Defensively normalize valid UTF-8 and bound code/type to 256 Unicode code points and message to 4,096 Unicode code points, even though PromptKit already bounds its accessors. Do not expose fields through struct formatting, JSON tags, or exported mutable fields. Preserve the dependency cause only behind the project-owned error so errors.Is/errors.As identities remain available without entering error text.

6. Add focused tests proving nil/zero safety, category and unwrap behavior, status-only ordinary formatting, %#v redaction, provider-detail bounds, and repair-value copying.

Do not import PromptKit from internal/promptexec and do not change CLI or artifact schemas in this stage.

Focused verification:

GOWORK=off go test -count=1 ./internal/promptexec

Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter

Purpose: make the adapter faithfully translate v0.8.0 preparation, execution, validation, usage, and failure values into the Stage 3 contract.

Work:

1. Map promptkit.OutputContract.RepairAttempts in prompt inspection and prepared-execution details. Map promptkit.ValidationResult.RepairAttempts in completed execution.

2. Preserve PromptKit's cumulative usage exactly as reported across the initial call and every completed correction. Continue returning only the final raw candidate and final validation result, subject to WeatherReporter's 64 KiB generated-output bound.

3. In adapter error classification, retain cancellation, deadline, and capacity precedence. Before the generic ErrLLMGenerate branch, use errors.As for *promptkit.GenerationError and construct the project-owned promptexec.GenerationError with status, code, type, message, and hidden cause. Initial and corrective generation failures use the same mapping.

4. Extend the injected adapter client used by tests so it can return an ordered sequence of responses or errors and record each request safely.

5. Use a synthetic PromptKit prompt with JSON Schema validation and repair_attempts: 1 to cover:

   • first-pass valid output with zero corrections;
   • explicitly empty or invalid output followed by valid corrected output;
   • one-attempt exhaustion returning a final failed validation result rather than an operational error;
   • a non-2xx-style GenerationError during correction;
   • cumulative token usage and actual repair count; and
   • the same prepared prompt/profile identity across the corrective flow.

6. Keep these tests at the adapter boundary. Do not assert PromptKit's private corrective-message wording or reconstruct its internal repair algorithm.

Canonical documentation in this stage: update docs/internal/promptkit-adapter.md for the repair/result/error mappings. Do not yet claim that embedded WeatherReporter prompts enable repair.

Focused verification:

GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit

Stage 5: Carry Repair Provenance Through Application Workflows

Purpose: make application orchestration understand configured and actual repair counts before changing the embedded prompt policy.

Work:

1. Add an expected generated-text repair budget to report.Definition and set it explicitly to zero for all four current 2.0.0 definitions in this stage. Include it in report-definition validation and retained contract tests.
2. Extend exact prompt preflight so format, validation mode, schema path, and repair budget must all match the resolved report definition. Extend preparation and completion provenance checks to require the same repair budget across inspection and the opaque prepared snapshot.
3. Add RepairAttempts *int to application outcomes where execution may fail before validation exists. Set it to a fresh pointer immediately after a non-nil completed execution is returned, before WeatherReporter's secondary generated-text validation. A pointer is required so completed first-pass zero is distinguishable from unavailable provenance.
4. Carry independent copies through ReportResult, BatchReportResult, batch conversion, comparison execution's internal outcome, and relevant test fakes. Do not expose the new value in CLI or comparison JSON yet.
5. Preserve the actual count on PromptKit validation rejection and on later WeatherReporter generated-text or render failures. Leave it unavailable on preparation, capacity, cancellation, deadline, and generation errors that return no completed PromptKit result.
6. Add focused tests for provenance mismatch, completed zero, completed positive, validation rejection, later local validation failure, early operational failure, batch copying, and independent concurrent profile outcomes.

Canonical documentation in this stage: update the focused prepared-report and app-orchestration internals to describe configured versus actual repair provenance. Current public documents should continue to report the embedded budget as zero until Stage 6.

Focused verification:

GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/app

Stage 6: Activate One Repair And Expose Ordinary Result Provenance

Purpose: switch the operational prompts to the accepted one-correction policy and make ordinary generate/run/batch output report what occurred.

Work:

1. Add repair_attempts: 1 to the output contract of all four embedded prompt definitions and change each exact prompt version from 2.0.0 to 2.1.0. Do not change prompt text or generated-text schemas solely for this upgrade.
2. Change all four report registry definitions to exact prompt version 2.1.0 and expected repair budget one. Update exact-version fixtures and assertions throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests that classify repair_attempts as a retired setting and replace them with an exact one-attempt contract assertion.
3. Add repairAttempts to successful and failed generate and batch JSON result shapes through the Stage 5 pointers. Emit integer zero for a completed first-pass result, a positive integer for a completed repaired result, and omit the field when no completed validation made it available.
4. Keep the existing validationStatus and failure categories authoritative. A repaired valid result proceeds normally. Repair exhaustion remains validation_rejected, publishes no report for that profile, and retains the actual attempt count.
5. Add representative offline assembled tests proving first-pass success, repaired success, exhaustion, explicit empty initial content, and batch result propagation. Reuse the real PromptKit adapter with an injected sequence client for at least one end-to-end repaired execution; use the existing app fake at other boundaries where lower-level repair is already covered.
6. Confirm no application loop, provider retry, profile fallback, or request-level OutputContract override was introduced.

Canonical documentation in this stage:

• update docs/policy/architecture.md with PromptKit-owned bounded repair and failed-exhaustion invariants;
• update docs/integrations/promptkit.md with exact prompt version 2.1.0, one configured repair, actual-count semantics, cumulative usage, explicit-empty handling, and the distinction from operational retries;
• update docs/cli.md for generate and batch repairAttempts fields;
• update docs/internal/report-registry.md, prepared-report internals, and app orchestration internals for exact version and repair flow; and
• keep configuration documentation unchanged because no repair setting is added.

Focused verification:

GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app

Stage 7: Migrate Comparison Bundles To v2

Purpose: preserve repair activity in the profile-evaluation artifact and make the strict durable schema change explicit.

Work:

1. Change comparison.SchemaVersion to weatherreporter.comparison.v2. Emit and recognize v2 only; do not retain a v1 parser or guarded-replacement compatibility path.
2. Add RepairAttempts *int to each application comparison profile result, CLI comparison profile summary, and durable comparison.Result. Propagate a fresh copy from Stage 5's execution outcome.
3. Place repairAttempts immediately after validationStatus in the canonical result-object JSON field order. Encode zero for completed first-pass validation, a positive integer for completed correction, and omit it only when no completed validation exists.
4. Tighten manifest invariants: every non-nil repair count is non-negative; a successful result must have validationStatus: "passed" and a non-nil repair count; a failed result with a completed validation status must also have a non-nil count; and an early operational failure may omit both.
5. Update the strict token-level JSON recognizer to accept only the canonical repairAttempts field at its correct object level, reject duplicate, unknown, negative, fractional, string, overflow, and malformed values, and continue rejecting v1 as an unsupported current bundle.
6. Update manifest construction, cloning, validation, exact serialization tests, guarded replacement tests, malicious bundle tests, partial-success tests, and CLI comparison summaries. Preserve flat layout, result ordering, hashes, atomic publication, cancellation safety, and no Distributor calls.
7. Cover concurrent peers where one succeeds first-pass, one repairs, one exhausts, and one fails operationally. The counts must remain attached to the selected profile positions without races or cross-contamination.

Canonical documentation in this stage:

• replace the v1 contract in docs/integrations/comparison-bundle.md with v2, including exact field order, presence rules, and the lack of v1 replacement compatibility;
• update docs/cli.md for comparison repairAttempts;
• update docs/operations.md to tell operators to move or remove an existing v1 bundle before replacing at the same destination; and
• update comparison execution/publication internals and architecture policy as needed for the current-only version invariant.

Focused verification:

GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app

Stage 8: Add Secure Provider-Failure Debug Capture

Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the existing explicit secure debug boundary while keeping ordinary errors safe.

Work:

1. Extend prompt preparation debug output with configured repairAttempts and advance its schema identifier from weatherreporter.prompt_preparation_debug.v2 to weatherreporter.prompt_preparation_debug.v3.

2. Extend execution validation debug output with actual repairAttempts and advance its schema identifier from weatherreporter.prompt_execution_debug.v2 to weatherreporter.prompt_execution_debug.v3. Retain cumulative token usage.

3. Add a dedicated failure.json artifact with schema identifier weatherreporter.prompt_failure_debug.v1. Its canonical fields are:

   • top level: schemaVersion, reportId, validDate, runId, failure;
   • failure object: category, statusCode, providerCode, providerType, providerMessage;
   • omit absent provider fields and zero status; and
   • never include the raw provider body, headers, endpoint, credentials, request, schema, rendered prompt, or generated candidate.

4. Add PromptDebugWriter.WriteFailure using the existing handle-relative secure run directory, 0700 directory and 0600 file modes, canonical JSON encoding, and no-follow/atomic replacement behavior. Disabled writers must perform no filesystem work.

5. When execution returns an error, use errors.As only against the project-owned *promptexec.GenerationError. If explicit debug capture is enabled, write failure.json using that profile's existing debug reference. This applies equally to initial and corrective provider failures and keeps comparison profile directories isolated.

6. If failure-debug writing also fails, retain the generation failure as the primary categorized error and join the safe debug-write failure rather than replacing or hiding the provider failure. Never place provider code, type, or message in the joined error text.

7. Ordinary generate, batch, and comparison errors should gain only the safe HTTP status already rendered by promptexec.GenerationError.Error; do not add provider detail fields to CLI summaries, comparison manifests, logs, or Distributor requests.

8. Add adversarial tests for formatter redaction, malicious provider strings, JSON escaping, bounds, absent fields, file modes, symlink/path attacks, write failure, cancellation identity, initial versus corrective failures, and concurrent comparison captures.

Canonical documentation in this stage:

• update docs/operations.md with the three debug artifact versions, failure.json, sensitivity, permissions, and retention;
• update docs/integrations/promptkit.md with ordinary status-only disclosure and debug-only provider detail;
• update prompt-debug, PromptKit-adapter, and app-orchestration internals; and
• ensure docs/policy/architecture.md explicitly prohibits provider-controlled diagnostics from ordinary outputs.

Focused verification:

GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app

Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit

Purpose: verify the complete end state as one coherent WeatherReporter feature and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile, repair, or debug claims.

Work:

1. Re-read the feature roadmap, docs/development.md, every policy document, and every canonical document changed by Stages 1-8. Reconcile them against executable behavior and remove duplicated or stale definitions. Keep unimplemented future ideas in docs/roadmap/future.md, not current-state documents.

2. Search code, embedded assets, examples, tests, and documentation for stale contractual literals and review every occurrence of:

   • PromptKit v0.5.0, v0.6.0, and v0.7.0 as an active dependency claim;
   • prompt version 2.0.0;
   • weatherreporter.comparison.v1;
   • prompt debug schema v2 identifiers;
   • claims that profile fields never inherit;
   • claims that every APIKeyEnv must be populated;
   • claims that backend ID is always required;
   • claims that repair is disabled or repair_attempts is retired; and
   • provider detail in ordinary output.

   Historical release documents may retain accurate historical literals.

3. Verify canonical ownership:

   • architecture owns invariants and boundaries;
   • config owns operator profile and credential behavior, but no repair field;
   • PromptKit integration owns logical prompt/profile/output contracts;
   • CLI owns result fields;
   • operations owns explicit debug handling and old comparison-bundle cleanup;
   • comparison integration owns the complete v2 manifest; and
   • internal documents own implementation flow without duplicating the public references.

4. Verify maintained examples remain valid, secret-free, and tested. The local weather-light example remains a standalone endpoint-only profile rather than inheriting an OpenRouter backend it cannot clear.

5. Review the complete diff for architecture leakage. Production packages outside internal/adapters/promptkit must not import PromptKit; no application repair loop, provider client, raw provider diagnostic, profile YAML parser, or durable application state may have appeared.

6. Review tests under the testing policy. Keep consumer contract and regression coverage, remove accidental duplication of upstream implementation tests, and ensure every default test is offline and repeatable.

7. Run the complete validation set:

gofmt -w <all changed Go files>
GOWORK=off go test -count=1 ./...
GOWORK=off go test -race -count=1 ./...
GOWORK=off go vet ./...
GOWORK=off go build ./...
GOWORK=off go mod tidy -diff
go run ./cmd/weatherreporter --help
go run ./cmd/weatherreporter compare --help
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
git diff --check

8. Confirm go.mod has no replace, the resolved PromptKit module is exactly v0.8.0, and no live credential or provider call occurred during validation.
9. After every check passes, update this plan's status to Completed and add a concise completion note listing the implemented stages. Do not delete either roadmap until the maintainer has reviewed the implementation. Do not create a release document or tag; release preparation remains a separate maintainer action once a version is selected.

Completion Standard

The implementation is complete only when all nine stages pass their focused and repository-wide checks, all locked decisions are observable in code and canonical documentation, and the feature roadmap's completion criteria are satisfied. Passing compilation alone is insufficient. The final state must demonstrate repaired success, repair exhaustion, comparison provenance, endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome inspection, safe ordinary provider failures, secure debug-only detail, and unchanged publication and concurrency invariants.