Files
weatherreporter/docs/roadmap/implementation.md

572 lines
28 KiB
Markdown

# PromptKit v0.8.0 Upgrade Implementation Plan
Status: Ready for implementation.
## Purpose And Authority
This plan translates the accepted
[PromptKit v0.8.0 upgrade roadmap](promptkit-v0.8.0.md) into an ordered,
decision-complete implementation procedure. The feature roadmap owns purpose,
scope, policy, and the desired end state. This document owns implementation
order, concrete work allocation, stage boundaries, and verification until the
upgrade is complete.
The implementing agent must complete the stages in numerical order. Each stage
is sized for one focused prompt handled by `gpt-5.6-terra` with high reasoning.
Do not combine stages merely because adjacent work touches the same package.
## Locked Decisions
The following decisions are final for this implementation:
- upgrade directly from PromptKit `v0.5.0` to `v0.8.0`;
- declare one corrective call in each embedded prompt through
`repair_attempts: 1` rather than adding WeatherReporter repair logic;
- keep the repair budget in the exact embedded prompt definition and add no
global, per-report, CLI, profile, or operator configuration override;
- advance all four exact prompt versions from `2.0.0` to `2.1.0`;
- make `weather-light`, `weather-balanced`, and `weather-deep` minimal aliases
of the corresponding PromptKit built-ins through `base_profile`;
- retain the existing report-to-profile assignments and effective model
ladder;
- allow successfully inspected endpoint-only profiles to have an empty backend
ID while continuing to require a nonblank model;
- treat `APIKeyEnv` as an optional lookup source and reject only profiles that
report `APIKeyRequired`, because WeatherReporter supplies no direct request
credential;
- support PromptKit's built-in `rakestrawhome-gemma-4-31b` profile without
WeatherReporter-specific backend configuration;
- expose provider HTTP status through a project-owned safe generation error,
while writing provider code, type, and message only to explicit secure debug
capture;
- emit only `weatherreporter.comparison.v2`, with repair provenance, and do not
preserve v1 guarded-replacement support; and
- retain PromptKit dependency types inside the adapter and preserve all
stateless execution, atomic publication, comparison independence, and
disclosure invariants.
## Implementation Rules
For every stage:
- read `docs/development.md`, all files under `docs/policy/`, this plan, the
feature roadmap, and the task-specific documents named by the stage;
- inspect the current code and tests before editing; use the repository's code
knowledge graph first for code discovery and fall back to text search for
literals, assets, and documentation;
- implement only the stage's scope and preserve unrelated user changes;
- keep PromptKit/provider types, client construction, YAML parsing, repair
mechanics, and provider transport inside the existing adapter boundary;
- use deterministic, offline, credential-free tests and injected clients or
synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local
endpoint calls;
- add tests at the narrowest stable owner identified by the testing policy and
avoid copying PromptKit's internal test matrices;
- update the canonical documentation owners listed for that stage in the same
change as the implemented contract;
- run `gofmt` on changed Go files, the stage's focused tests,
`GOWORK=off go test -count=1 ./...`, and `git diff --check`; and
- leave the repository passing before proceeding to the next stage.
Stages affecting concurrent comparison, cancellation, or secure debug
filesystem work must also run the named focused packages with `-race`. Do not
weaken an existing assertion solely to accommodate the new dependency. When a
test encodes an intentionally changed contract, replace it with a behavioral
assertion for the accepted policy.
## Implementation Stages
### Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline
Purpose: move to the tagged dependency and isolate compatibility changes before
adopting new WeatherReporter behavior.
Work:
1. Update `go.mod` to require
`gitea.maximumdirect.net/eric/promptkit v0.8.0` and refresh `go.sum` with
`GOWORK=off go mod tidy`. Do not add `go.work`, `vendor`, or a `replace`
directive and do not change WeatherReporter's Go version.
2. Resolve any compile failures using PromptKit's public root package only.
Keep all `Profile` and `OpenAICompatibleProfileConfig` literals keyed. Do not
register the now-reserved `rakestrawhome` backend.
3. Reconcile adapter tests that directly exercise PromptKit's changed optional
credential behavior. A profile whose only credential metadata is
`api_key_env` must reach an injected client when the environment value is
absent; it must no longer expect PromptKit to return
`ErrAPIKeyEnvMissing`. Do not change WeatherReporter's application preflight
in this stage.
4. Verify that every current embedded prompt, content file, schema, and fallback
profile inspects under v0.8.0. Verify selected invalid local endpoints and
malformed selected profile definitions still map to project-owned
configuration or profile-load categories.
5. Review the v0.6.0 compatibility corrections against supported
WeatherReporter inputs: metadata-authoritative identities, exact contained
`content_file` paths, regular embedded files, structurally valid endpoints,
bounded JSON-compatible values, cancellation identity, and strict response
framing. Add consumer tests only for a WeatherReporter boundary not already
protected by PromptKit.
Do not enable profile inheritance or output repair yet. The expected result is
the current WeatherReporter feature set running against PromptKit v0.8.0.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
```
### Stage 2: Adopt Profile Inheritance And Current Credential Routing
Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional
credential semantics, and the Rakestrawhome built-in without changing the model
ladder.
Work:
1. Replace the three embedded profile bodies with these exact leaf/base
relationships and no duplicated execution settings:
| Leaf | Base |
| --- | --- |
| `weather-light` | `deepseek-4-flash` |
| `weather-balanced` | `gemini-flash-latest` |
| `weather-deep` | `claude-sonnet-latest` |
2. Update prompt-asset fixtures and tests to understand `base_profile`. Assert
that all three leaf IDs remain selected identities and resolve to the same
backend, model, timeout, service tier, and reasoning settings exposed by the
current standalone definitions. Test relationships and effective behavior,
not copied private constants beyond the intentional model-ladder contract.
3. Preserve source precedence. Cover a standalone same-ID operator override, a
derived operator override, a configured source that shadows a base ID, and
selected missing-base, cyclic, malformed-base, and incomplete-target
failures. Do not implement inheritance or merging in WeatherReporter; all
resolution must remain PromptKit-owned.
4. Change application profile preflight to accept a successful inspection with
a nonblank model and an empty backend ID. Trust PromptKit inspection to have
resolved either a backend or endpoint; do not add the endpoint to
`promptexec.ProfileInspection` or ordinary provenance.
5. Remove application-level environment lookup and rejection for a nonblank
`APIKeyEnv`. Remove the now-unused `LookupEnv` fields and plumbing from
prompt, batch, and comparison inspection requests. Continue rejecting
`CredentialRequired`/`APIKeyRequired` before weather collection with the
existing missing-credential category.
6. Add an end-to-end offline regression proving the maintained endpoint-only
`weather-light` example passes application inspection, retains an empty
backend ID, and does not expose its endpoint.
7. Prove `rakestrawhome-gemma-4-31b` can pass ordinary and comparison profile
inspection through the existing adapter and reports the PromptKit
`rakestrawhome` backend ID. Do not make a provider call or add
Rakestrawhome-specific configuration.
Canonical documentation in this stage:
- update `docs/policy/architecture.md` so only direct-key-required profiles
fail credential preflight and backend identity is optional for endpoint-only
profiles;
- update `docs/config.md` to distinguish same-ID source replacement from
`base_profile` chain inheritance and to describe optional environment
credentials;
- update `docs/integrations/promptkit.md` for profile composition, parent
lookup precedence, endpoint-only identity, optional credentials, and
Rakestrawhome availability; and
- update `docs/internal/promptkit-adapter.md` and focused app internals for the
implemented inspection behavior.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
```
### Stage 3: Extend The Project-Owned Prompt Execution Contract
Purpose: establish dependency-neutral repair and structured-generation-error
values before the adapter or application relies on them.
Work:
1. Add `RepairAttempts int` to `promptexec.OutputContract`. It is the configured
additional-call budget from the exact prompt contract.
2. Add `RepairAttempts int` to `promptexec.Validation`. It is the number of
corrective calls actually started for the completed result. Update
`NewValidation` and every caller so construction is explicit; reject or
normalize no values here because PromptKit owns output-contract validity.
3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so
repair values are retained without sharing mutable state.
4. Add a project-owned immutable `promptexec.GenerationError` with unexported
status and provider-detail fields plus safe accessors:
- `StatusCode() int`
- `ProviderCode() string`
- `ProviderType() string`
- `ProviderMessage() string`
- `Category() ErrorCategory`, always returning `Generation`
- `Error()`, exposing only the WeatherReporter generation category/message
and optional HTTP status
- `GoString()`, returning the same safe representation
- `Unwrap()`, preserving a project-owned `*promptexec.Error`
5. Provide one constructor used by adapters. Defensively normalize valid UTF-8
and bound code/type to 256 Unicode code points and message to 4,096 Unicode
code points, even though PromptKit already bounds its accessors. Do not
expose fields through struct formatting, JSON tags, or exported mutable
fields. Preserve the dependency cause only behind the project-owned error so
`errors.Is`/`errors.As` identities remain available without entering error
text.
6. Add focused tests proving nil/zero safety, category and unwrap behavior,
status-only ordinary formatting, `%#v` redaction, provider-detail bounds,
and repair-value copying.
Do not import PromptKit from `internal/promptexec` and do not change CLI or
artifact schemas in this stage.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/promptexec
```
### Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter
Purpose: make the adapter faithfully translate v0.8.0 preparation, execution,
validation, usage, and failure values into the Stage 3 contract.
Work:
1. Map `promptkit.OutputContract.RepairAttempts` in prompt inspection and
prepared-execution details. Map
`promptkit.ValidationResult.RepairAttempts` in completed execution.
2. Preserve PromptKit's cumulative usage exactly as reported across the initial
call and every completed correction. Continue returning only the final raw
candidate and final validation result, subject to WeatherReporter's 64 KiB
generated-output bound.
3. In adapter error classification, retain cancellation, deadline, and capacity
precedence. Before the generic `ErrLLMGenerate` branch, use `errors.As` for
`*promptkit.GenerationError` and construct the project-owned
`promptexec.GenerationError` with status, code, type, message, and hidden
cause. Initial and corrective generation failures use the same mapping.
4. Extend the injected adapter client used by tests so it can return an ordered
sequence of responses or errors and record each request safely.
5. Use a synthetic PromptKit prompt with JSON Schema validation and
`repair_attempts: 1` to cover:
- first-pass valid output with zero corrections;
- explicitly empty or invalid output followed by valid corrected output;
- one-attempt exhaustion returning a final failed validation result rather
than an operational error;
- a non-2xx-style `GenerationError` during correction;
- cumulative token usage and actual repair count; and
- the same prepared prompt/profile identity across the corrective flow.
6. Keep these tests at the adapter boundary. Do not assert PromptKit's private
corrective-message wording or reconstruct its internal repair algorithm.
Canonical documentation in this stage: update
`docs/internal/promptkit-adapter.md` for the repair/result/error mappings. Do
not yet claim that embedded WeatherReporter prompts enable repair.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
```
### Stage 5: Carry Repair Provenance Through Application Workflows
Purpose: make application orchestration understand configured and actual repair
counts before changing the embedded prompt policy.
Work:
1. Add an expected generated-text repair budget to `report.Definition` and set
it explicitly to zero for all four current `2.0.0` definitions in this
stage. Include it in report-definition validation and retained contract
tests.
2. Extend exact prompt preflight so format, validation mode, schema path, and
repair budget must all match the resolved report definition. Extend
preparation and completion provenance checks to require the same repair
budget across inspection and the opaque prepared snapshot.
3. Add `RepairAttempts *int` to application outcomes where execution may fail
before validation exists. Set it to a fresh pointer immediately after a
non-nil completed execution is returned, before WeatherReporter's secondary
generated-text validation. A pointer is required so completed first-pass
zero is distinguishable from unavailable provenance.
4. Carry independent copies through `ReportResult`, `BatchReportResult`, batch
conversion, comparison execution's internal outcome, and relevant test
fakes. Do not expose the new value in CLI or comparison JSON yet.
5. Preserve the actual count on PromptKit validation rejection and on later
WeatherReporter generated-text or render failures. Leave it unavailable on
preparation, capacity, cancellation, deadline, and generation errors that
return no completed PromptKit result.
6. Add focused tests for provenance mismatch, completed zero, completed
positive, validation rejection, later local validation failure, early
operational failure, batch copying, and independent concurrent profile
outcomes.
Canonical documentation in this stage: update the focused prepared-report and
app-orchestration internals to describe configured versus actual repair
provenance. Current public documents should continue to report the embedded
budget as zero until Stage 6.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/app
```
### Stage 6: Activate One Repair And Expose Ordinary Result Provenance
Purpose: switch the operational prompts to the accepted one-correction policy
and make ordinary generate/run/batch output report what occurred.
Work:
1. Add `repair_attempts: 1` to the output contract of all four embedded prompt
definitions and change each exact prompt version from `2.0.0` to `2.1.0`.
Do not change prompt text or generated-text schemas solely for this upgrade.
2. Change all four report registry definitions to exact prompt version `2.1.0`
and expected repair budget one. Update exact-version fixtures and assertions
throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests
that classify `repair_attempts` as a retired setting and replace them with
an exact one-attempt contract assertion.
3. Add `repairAttempts` to successful and failed generate and batch JSON result
shapes through the Stage 5 pointers. Emit integer zero for a completed
first-pass result, a positive integer for a completed repaired result, and
omit the field when no completed validation made it available.
4. Keep the existing `validationStatus` and failure categories authoritative.
A repaired valid result proceeds normally. Repair exhaustion remains
`validation_rejected`, publishes no report for that profile, and retains the
actual attempt count.
5. Add representative offline assembled tests proving first-pass success,
repaired success, exhaustion, explicit empty initial content, and batch
result propagation. Reuse the real PromptKit adapter with an injected
sequence client for at least one end-to-end repaired execution; use the
existing app fake at other boundaries where lower-level repair is already
covered.
6. Confirm no application loop, provider retry, profile fallback, or
request-level `OutputContract` override was introduced.
Canonical documentation in this stage:
- update `docs/policy/architecture.md` with PromptKit-owned bounded repair and
failed-exhaustion invariants;
- update `docs/integrations/promptkit.md` with exact prompt version `2.1.0`, one
configured repair, actual-count semantics, cumulative usage, explicit-empty
handling, and the distinction from operational retries;
- update `docs/cli.md` for generate and batch `repairAttempts` fields;
- update `docs/internal/report-registry.md`, prepared-report internals, and app
orchestration internals for exact version and repair flow; and
- keep configuration documentation unchanged because no repair setting is
added.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
```
### Stage 7: Migrate Comparison Bundles To v2
Purpose: preserve repair activity in the profile-evaluation artifact and make
the strict durable schema change explicit.
Work:
1. Change `comparison.SchemaVersion` to
`weatherreporter.comparison.v2`. Emit and recognize v2 only; do not retain a
v1 parser or guarded-replacement compatibility path.
2. Add `RepairAttempts *int` to each application comparison profile result,
CLI comparison profile summary, and durable `comparison.Result`. Propagate a
fresh copy from Stage 5's execution outcome.
3. Place `repairAttempts` immediately after `validationStatus` in the canonical
result-object JSON field order. Encode zero for completed first-pass
validation, a positive integer for completed correction, and omit it only
when no completed validation exists.
4. Tighten manifest invariants: every non-nil repair count is non-negative; a
successful result must have `validationStatus: "passed"` and a non-nil
repair count; a failed result with a completed validation status must also
have a non-nil count; and an early operational failure may omit both.
5. Update the strict token-level JSON recognizer to accept only the canonical
`repairAttempts` field at its correct object level, reject duplicate,
unknown, negative, fractional, string, overflow, and malformed values, and
continue rejecting v1 as an unsupported current bundle.
6. Update manifest construction, cloning, validation, exact serialization
tests, guarded replacement tests, malicious bundle tests, partial-success
tests, and CLI comparison summaries. Preserve flat layout, result ordering,
hashes, atomic publication, cancellation safety, and no Distributor calls.
7. Cover concurrent peers where one succeeds first-pass, one repairs, one
exhausts, and one fails operationally. The counts must remain attached to
the selected profile positions without races or cross-contamination.
Canonical documentation in this stage:
- replace the v1 contract in `docs/integrations/comparison-bundle.md` with v2,
including exact field order, presence rules, and the lack of v1 replacement
compatibility;
- update `docs/cli.md` for comparison `repairAttempts`;
- update `docs/operations.md` to tell operators to move or remove an existing
v1 bundle before replacing at the same destination; and
- update comparison execution/publication internals and architecture policy as
needed for the current-only version invariant.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app
```
### Stage 8: Add Secure Provider-Failure Debug Capture
Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the
existing explicit secure debug boundary while keeping ordinary errors safe.
Work:
1. Extend prompt preparation debug output with configured
`repairAttempts` and advance its schema identifier from
`weatherreporter.prompt_preparation_debug.v2` to
`weatherreporter.prompt_preparation_debug.v3`.
2. Extend execution validation debug output with actual `repairAttempts` and
advance its schema identifier from
`weatherreporter.prompt_execution_debug.v2` to
`weatherreporter.prompt_execution_debug.v3`. Retain cumulative token usage.
3. Add a dedicated `failure.json` artifact with schema identifier
`weatherreporter.prompt_failure_debug.v1`. Its canonical fields are:
- top level: `schemaVersion`, `reportId`, `validDate`, `runId`, `failure`;
- failure object: `category`, `statusCode`, `providerCode`, `providerType`,
`providerMessage`;
- omit absent provider fields and zero status; and
- never include the raw provider body, headers, endpoint, credentials,
request, schema, rendered prompt, or generated candidate.
4. Add `PromptDebugWriter.WriteFailure` using the existing handle-relative
secure run directory, `0700` directory and `0600` file modes, canonical JSON
encoding, and no-follow/atomic replacement behavior. Disabled writers must
perform no filesystem work.
5. When execution returns an error, use `errors.As` only against the
project-owned `*promptexec.GenerationError`. If explicit debug capture is
enabled, write `failure.json` using that profile's existing debug reference.
This applies equally to initial and corrective provider failures and keeps
comparison profile directories isolated.
6. If failure-debug writing also fails, retain the generation failure as the
primary categorized error and join the safe debug-write failure rather than
replacing or hiding the provider failure. Never place provider code, type,
or message in the joined error text.
7. Ordinary generate, batch, and comparison errors should gain only the safe
HTTP status already rendered by `promptexec.GenerationError.Error`; do not
add provider detail fields to CLI summaries, comparison manifests, logs, or
Distributor requests.
8. Add adversarial tests for formatter redaction, malicious provider strings,
JSON escaping, bounds, absent fields, file modes, symlink/path attacks,
write failure, cancellation identity, initial versus corrective failures,
and concurrent comparison captures.
Canonical documentation in this stage:
- update `docs/operations.md` with the three debug artifact versions,
`failure.json`, sensitivity, permissions, and retention;
- update `docs/integrations/promptkit.md` with ordinary status-only disclosure
and debug-only provider detail;
- update prompt-debug, PromptKit-adapter, and app-orchestration internals; and
- ensure `docs/policy/architecture.md` explicitly prohibits provider-controlled
diagnostics from ordinary outputs.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app
```
### Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit
Purpose: verify the complete end state as one coherent WeatherReporter feature
and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile,
repair, or debug claims.
Work:
1. Re-read the feature roadmap, `docs/development.md`, every policy document,
and every canonical document changed by Stages 1-8. Reconcile them against
executable behavior and remove duplicated or stale definitions. Keep
unimplemented future ideas in `docs/roadmap/future.md`, not current-state
documents.
2. Search code, embedded assets, examples, tests, and documentation for stale
contractual literals and review every occurrence of:
- PromptKit `v0.5.0`, `v0.6.0`, and `v0.7.0` as an active dependency claim;
- prompt version `2.0.0`;
- `weatherreporter.comparison.v1`;
- prompt debug schema v2 identifiers;
- claims that profile fields never inherit;
- claims that every `APIKeyEnv` must be populated;
- claims that backend ID is always required;
- claims that repair is disabled or `repair_attempts` is retired; and
- provider detail in ordinary output.
Historical release documents may retain accurate historical literals.
3. Verify canonical ownership:
- architecture owns invariants and boundaries;
- config owns operator profile and credential behavior, but no repair field;
- PromptKit integration owns logical prompt/profile/output contracts;
- CLI owns result fields;
- operations owns explicit debug handling and old comparison-bundle cleanup;
- comparison integration owns the complete v2 manifest; and
- internal documents own implementation flow without duplicating the public
references.
4. Verify maintained examples remain valid, secret-free, and tested. The local
`weather-light` example remains a standalone endpoint-only profile rather
than inheriting an OpenRouter backend it cannot clear.
5. Review the complete diff for architecture leakage. Production packages
outside `internal/adapters/promptkit` must not import PromptKit; no
application repair loop, provider client, raw provider diagnostic, profile
YAML parser, or durable application state may have appeared.
6. Review tests under the testing policy. Keep consumer contract and regression
coverage, remove accidental duplication of upstream implementation tests,
and ensure every default test is offline and repeatable.
7. Run the complete validation set:
```sh
gofmt -w <all changed Go files>
GOWORK=off go test -count=1 ./...
GOWORK=off go test -race -count=1 ./...
GOWORK=off go vet ./...
GOWORK=off go build ./...
GOWORK=off go mod tidy -diff
go run ./cmd/weatherreporter --help
go run ./cmd/weatherreporter compare --help
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
git diff --check
```
8. Confirm `go.mod` has no `replace`, the resolved PromptKit module is exactly
v0.8.0, and no live credential or provider call occurred during validation.
9. After every check passes, update this plan's status to Completed and add a
concise completion note listing the implemented stages. Do not delete either
roadmap until the maintainer has reviewed the implementation. Do not create
a release document or tag; release preparation remains a separate maintainer
action once a version is selected.
## Completion Standard
The implementation is complete only when all nine stages pass their focused
and repository-wide checks, all locked decisions are observable in code and
canonical documentation, and the feature roadmap's completion criteria are
satisfied. Passing compilation alone is insufficient. The final state must
demonstrate repaired success, repair exhaustion, comparison provenance,
endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome
inspection, safe ordinary provider failures, secure debug-only detail, and
unchanged publication and concurrency invariants.