596 lines
28 KiB
Markdown
596 lines
28 KiB
Markdown
# PromptKit v0.8.0 Upgrade Implementation Plan
|
||
|
||
Status: Complete.
|
||
|
||
Completion note: Stages 1–9 upgraded PromptKit, adopted inherited profiles and
|
||
current credential handling, added repair and provider-failure contracts,
|
||
enabled one corrective generation for v2.1.0 prompts, exposed repair
|
||
provenance in ordinary and comparison results, migrated comparison bundles to
|
||
v2, and added secure failure-debug capture.
|
||
|
||
## Purpose And Authority
|
||
|
||
This plan translates the accepted
|
||
[PromptKit v0.8.0 upgrade roadmap](promptkit-v0.8.0.md) into an ordered,
|
||
decision-complete implementation procedure. The feature roadmap owns purpose,
|
||
scope, policy, and the desired end state. This document owns implementation
|
||
order, concrete work allocation, stage boundaries, and verification until the
|
||
upgrade is complete.
|
||
|
||
The implementing agent must complete the stages in numerical order. Each stage
|
||
is sized for one focused prompt handled by `gpt-5.6-terra` with high reasoning.
|
||
Do not combine stages merely because adjacent work touches the same package.
|
||
|
||
## Locked Decisions
|
||
|
||
The following decisions are final for this implementation:
|
||
|
||
- upgrade directly from PromptKit `v0.5.0` to `v0.8.0`;
|
||
- declare one corrective call in each embedded prompt through
|
||
`repair_attempts: 1` rather than adding WeatherReporter repair logic;
|
||
- keep the repair budget in the exact embedded prompt definition and add no
|
||
global, per-report, CLI, profile, or operator configuration override;
|
||
- advance all four exact prompt versions from `2.0.0` to `2.1.0`;
|
||
- make `weather-light`, `weather-balanced`, and `weather-deep` minimal aliases
|
||
of the corresponding PromptKit built-ins through `base_profile`;
|
||
- retain the existing report-to-profile assignments and effective model
|
||
ladder;
|
||
- allow successfully inspected endpoint-only profiles to have an empty backend
|
||
ID while continuing to require a nonblank model;
|
||
- treat `APIKeyEnv` as an optional lookup source and reject only profiles that
|
||
report `APIKeyRequired`, because WeatherReporter supplies no direct request
|
||
credential;
|
||
- support PromptKit's built-in `rakestrawhome-gemma-4-31b` profile without
|
||
WeatherReporter-specific backend configuration;
|
||
- expose provider HTTP status through a project-owned safe generation error,
|
||
while writing provider code, type, and message only to explicit secure debug
|
||
capture;
|
||
- emit only `weatherreporter.comparison.v2`, with repair provenance, and do not
|
||
preserve v1 guarded-replacement support; and
|
||
- retain PromptKit dependency types inside the adapter and preserve all
|
||
stateless execution, atomic publication, comparison independence, and
|
||
disclosure invariants.
|
||
|
||
## Implementation Rules
|
||
|
||
For every stage:
|
||
|
||
- read `docs/development.md`, all files under `docs/policy/`, this plan, the
|
||
feature roadmap, and the task-specific documents named by the stage;
|
||
- inspect the current code and tests before editing; use the repository's code
|
||
knowledge graph first for code discovery and fall back to text search for
|
||
literals, assets, and documentation;
|
||
- implement only the stage's scope and preserve unrelated user changes;
|
||
- keep PromptKit/provider types, client construction, YAML parsing, repair
|
||
mechanics, and provider transport inside the existing adapter boundary;
|
||
- use deterministic, offline, credential-free tests and injected clients or
|
||
synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local
|
||
endpoint calls;
|
||
- add tests at the narrowest stable owner identified by the testing policy and
|
||
avoid copying PromptKit's internal test matrices;
|
||
- update the canonical documentation owners listed for that stage in the same
|
||
change as the implemented contract;
|
||
- run `gofmt` on changed Go files, the stage's focused tests,
|
||
`GOWORK=off go test -count=1 ./...`, and `git diff --check`; and
|
||
- leave the repository passing before proceeding to the next stage.
|
||
|
||
Stages affecting concurrent comparison, cancellation, or secure debug
|
||
filesystem work must also run the named focused packages with `-race`. Do not
|
||
weaken an existing assertion solely to accommodate the new dependency. When a
|
||
test encodes an intentionally changed contract, replace it with a behavioral
|
||
assertion for the accepted policy.
|
||
|
||
## Implementation Stages
|
||
|
||
### Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: move to the tagged dependency and isolate compatibility changes before
|
||
adopting new WeatherReporter behavior.
|
||
|
||
Work:
|
||
|
||
1. Update `go.mod` to require
|
||
`gitea.maximumdirect.net/eric/promptkit v0.8.0` and refresh `go.sum` with
|
||
`GOWORK=off go mod tidy`. Do not add `go.work`, `vendor`, or a `replace`
|
||
directive and do not change WeatherReporter's Go version.
|
||
2. Resolve any compile failures using PromptKit's public root package only.
|
||
Keep all `Profile` and `OpenAICompatibleProfileConfig` literals keyed. Do not
|
||
register the now-reserved `rakestrawhome` backend.
|
||
3. Reconcile adapter tests that directly exercise PromptKit's changed optional
|
||
credential behavior. A profile whose only credential metadata is
|
||
`api_key_env` must reach an injected client when the environment value is
|
||
absent; it must no longer expect PromptKit to return
|
||
`ErrAPIKeyEnvMissing`. Do not change WeatherReporter's application preflight
|
||
in this stage.
|
||
4. Verify that every current embedded prompt, content file, schema, and fallback
|
||
profile inspects under v0.8.0. Verify selected invalid local endpoints and
|
||
malformed selected profile definitions still map to project-owned
|
||
configuration or profile-load categories.
|
||
5. Review the v0.6.0 compatibility corrections against supported
|
||
WeatherReporter inputs: metadata-authoritative identities, exact contained
|
||
`content_file` paths, regular embedded files, structurally valid endpoints,
|
||
bounded JSON-compatible values, cancellation identity, and strict response
|
||
framing. Add consumer tests only for a WeatherReporter boundary not already
|
||
protected by PromptKit.
|
||
|
||
Do not enable profile inheritance or output repair yet. The expected result is
|
||
the current WeatherReporter feature set running against PromptKit v0.8.0.
|
||
|
||
Focused verification:
|
||
|
||
```sh
|
||
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets
|
||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
|
||
```
|
||
|
||
### Stage 2: Adopt Profile Inheritance And Current Credential Routing
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional
|
||
credential semantics, and the Rakestrawhome built-in without changing the model
|
||
ladder.
|
||
|
||
Work:
|
||
|
||
1. Replace the three embedded profile bodies with these exact leaf/base
|
||
relationships and no duplicated execution settings:
|
||
|
||
| Leaf | Base |
|
||
| --- | --- |
|
||
| `weather-light` | `deepseek-4-flash` |
|
||
| `weather-balanced` | `gemini-flash-latest` |
|
||
| `weather-deep` | `claude-sonnet-latest` |
|
||
|
||
2. Update prompt-asset fixtures and tests to understand `base_profile`. Assert
|
||
that all three leaf IDs remain selected identities and resolve to the same
|
||
backend, model, timeout, service tier, and reasoning settings exposed by the
|
||
current standalone definitions. Test relationships and effective behavior,
|
||
not copied private constants beyond the intentional model-ladder contract.
|
||
3. Preserve source precedence. Cover a standalone same-ID operator override, a
|
||
derived operator override, a configured source that shadows a base ID, and
|
||
selected missing-base, cyclic, malformed-base, and incomplete-target
|
||
failures. Do not implement inheritance or merging in WeatherReporter; all
|
||
resolution must remain PromptKit-owned.
|
||
4. Change application profile preflight to accept a successful inspection with
|
||
a nonblank model and an empty backend ID. Trust PromptKit inspection to have
|
||
resolved either a backend or endpoint; do not add the endpoint to
|
||
`promptexec.ProfileInspection` or ordinary provenance.
|
||
5. Remove application-level environment lookup and rejection for a nonblank
|
||
`APIKeyEnv`. Remove the now-unused `LookupEnv` fields and plumbing from
|
||
prompt, batch, and comparison inspection requests. Continue rejecting
|
||
`CredentialRequired`/`APIKeyRequired` before weather collection with the
|
||
existing missing-credential category.
|
||
6. Add an end-to-end offline regression proving the maintained endpoint-only
|
||
`weather-light` example passes application inspection, retains an empty
|
||
backend ID, and does not expose its endpoint.
|
||
7. Prove `rakestrawhome-gemma-4-31b` can pass ordinary and comparison profile
|
||
inspection through the existing adapter and reports the PromptKit
|
||
`rakestrawhome` backend ID. Do not make a provider call or add
|
||
Rakestrawhome-specific configuration.
|
||
|
||
Canonical documentation in this stage:
|
||
|
||
- update `docs/policy/architecture.md` so only direct-key-required profiles
|
||
fail credential preflight and backend identity is optional for endpoint-only
|
||
profiles;
|
||
- update `docs/config.md` to distinguish same-ID source replacement from
|
||
`base_profile` chain inheritance and to describe optional environment
|
||
credentials;
|
||
- update `docs/integrations/promptkit.md` for profile composition, parent
|
||
lookup precedence, endpoint-only identity, optional credentials, and
|
||
Rakestrawhome availability; and
|
||
- update `docs/internal/promptkit-adapter.md` and focused app internals for the
|
||
implemented inspection behavior.
|
||
|
||
Focused verification:
|
||
|
||
```sh
|
||
GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app
|
||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
|
||
```
|
||
|
||
### Stage 3: Extend The Project-Owned Prompt Execution Contract
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: establish dependency-neutral repair and structured-generation-error
|
||
values before the adapter or application relies on them.
|
||
|
||
Work:
|
||
|
||
1. Add `RepairAttempts int` to `promptexec.OutputContract`. It is the configured
|
||
additional-call budget from the exact prompt contract.
|
||
2. Add `RepairAttempts int` to `promptexec.Validation`. It is the number of
|
||
corrective calls actually started for the completed result. Update
|
||
`NewValidation` and every caller so construction is explicit; reject or
|
||
normalize no values here because PromptKit owns output-contract validity.
|
||
3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so
|
||
repair values are retained without sharing mutable state.
|
||
4. Add a project-owned immutable `promptexec.GenerationError` with unexported
|
||
status and provider-detail fields plus safe accessors:
|
||
|
||
- `StatusCode() int`
|
||
- `ProviderCode() string`
|
||
- `ProviderType() string`
|
||
- `ProviderMessage() string`
|
||
- `Category() ErrorCategory`, always returning `Generation`
|
||
- `Error()`, exposing only the WeatherReporter generation category/message
|
||
and optional HTTP status
|
||
- `GoString()`, returning the same safe representation
|
||
- `Unwrap()`, preserving a project-owned `*promptexec.Error`
|
||
|
||
5. Provide one constructor used by adapters. Defensively normalize valid UTF-8
|
||
and bound code/type to 256 Unicode code points and message to 4,096 Unicode
|
||
code points, even though PromptKit already bounds its accessors. Do not
|
||
expose fields through struct formatting, JSON tags, or exported mutable
|
||
fields. Preserve the dependency cause only behind the project-owned error so
|
||
`errors.Is`/`errors.As` identities remain available without entering error
|
||
text.
|
||
6. Add focused tests proving nil/zero safety, category and unwrap behavior,
|
||
status-only ordinary formatting, `%#v` redaction, provider-detail bounds,
|
||
and repair-value copying.
|
||
|
||
Do not import PromptKit from `internal/promptexec` and do not change CLI or
|
||
artifact schemas in this stage.
|
||
|
||
Focused verification:
|
||
|
||
```sh
|
||
GOWORK=off go test -count=1 ./internal/promptexec
|
||
```
|
||
|
||
### Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: make the adapter faithfully translate v0.8.0 preparation, execution,
|
||
validation, usage, and failure values into the Stage 3 contract.
|
||
|
||
Work:
|
||
|
||
1. Map `promptkit.OutputContract.RepairAttempts` in prompt inspection and
|
||
prepared-execution details. Map
|
||
`promptkit.ValidationResult.RepairAttempts` in completed execution.
|
||
2. Preserve PromptKit's cumulative usage exactly as reported across the initial
|
||
call and every completed correction. Continue returning only the final raw
|
||
candidate and final validation result, subject to WeatherReporter's 64 KiB
|
||
generated-output bound.
|
||
3. In adapter error classification, retain cancellation, deadline, and capacity
|
||
precedence. Before the generic `ErrLLMGenerate` branch, use `errors.As` for
|
||
`*promptkit.GenerationError` and construct the project-owned
|
||
`promptexec.GenerationError` with status, code, type, message, and hidden
|
||
cause. Initial and corrective generation failures use the same mapping.
|
||
4. Extend the injected adapter client used by tests so it can return an ordered
|
||
sequence of responses or errors and record each request safely.
|
||
5. Use a synthetic PromptKit prompt with JSON Schema validation and
|
||
`repair_attempts: 1` to cover:
|
||
|
||
- first-pass valid output with zero corrections;
|
||
- explicitly empty or invalid output followed by valid corrected output;
|
||
- one-attempt exhaustion returning a final failed validation result rather
|
||
than an operational error;
|
||
- a non-2xx-style `GenerationError` during correction;
|
||
- cumulative token usage and actual repair count; and
|
||
- the same prepared prompt/profile identity across the corrective flow.
|
||
|
||
6. Keep these tests at the adapter boundary. Do not assert PromptKit's private
|
||
corrective-message wording or reconstruct its internal repair algorithm.
|
||
|
||
Canonical documentation in this stage: update
|
||
`docs/internal/promptkit-adapter.md` for the repair/result/error mappings. Do
|
||
not yet claim that embedded WeatherReporter prompts enable repair.
|
||
|
||
Focused verification:
|
||
|
||
```sh
|
||
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec
|
||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
|
||
```
|
||
|
||
### Stage 5: Carry Repair Provenance Through Application Workflows
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: make application orchestration understand configured and actual repair
|
||
counts before changing the embedded prompt policy.
|
||
|
||
Work:
|
||
|
||
1. Add an expected generated-text repair budget to `report.Definition` and set
|
||
it explicitly to zero for all four current `2.0.0` definitions in this
|
||
stage. Include it in report-definition validation and retained contract
|
||
tests.
|
||
2. Extend exact prompt preflight so format, validation mode, schema path, and
|
||
repair budget must all match the resolved report definition. Extend
|
||
preparation and completion provenance checks to require the same repair
|
||
budget across inspection and the opaque prepared snapshot.
|
||
3. Add `RepairAttempts *int` to application outcomes where execution may fail
|
||
before validation exists. Set it to a fresh pointer immediately after a
|
||
non-nil completed execution is returned, before WeatherReporter's secondary
|
||
generated-text validation. A pointer is required so completed first-pass
|
||
zero is distinguishable from unavailable provenance.
|
||
4. Carry independent copies through `ReportResult`, `BatchReportResult`, batch
|
||
conversion, comparison execution's internal outcome, and relevant test
|
||
fakes. Do not expose the new value in CLI or comparison JSON yet.
|
||
5. Preserve the actual count on PromptKit validation rejection and on later
|
||
WeatherReporter generated-text or render failures. Leave it unavailable on
|
||
preparation, capacity, cancellation, deadline, and generation errors that
|
||
return no completed PromptKit result.
|
||
6. Add focused tests for provenance mismatch, completed zero, completed
|
||
positive, validation rejection, later local validation failure, early
|
||
operational failure, batch copying, and independent concurrent profile
|
||
outcomes.
|
||
|
||
Canonical documentation in this stage: update the focused prepared-report and
|
||
app-orchestration internals to describe configured versus actual repair
|
||
provenance. Current public documents should continue to report the embedded
|
||
budget as zero until Stage 6.
|
||
|
||
Focused verification:
|
||
|
||
```sh
|
||
GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli
|
||
GOWORK=off go test -race -count=1 ./internal/app
|
||
```
|
||
|
||
### Stage 6: Activate One Repair And Expose Ordinary Result Provenance
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: switch the operational prompts to the accepted one-correction policy
|
||
and make ordinary generate/run/batch output report what occurred.
|
||
|
||
Work:
|
||
|
||
1. Add `repair_attempts: 1` to the output contract of all four embedded prompt
|
||
definitions and change each exact prompt version from `2.0.0` to `2.1.0`.
|
||
Do not change prompt text or generated-text schemas solely for this upgrade.
|
||
2. Change all four report registry definitions to exact prompt version `2.1.0`
|
||
and expected repair budget one. Update exact-version fixtures and assertions
|
||
throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests
|
||
that classify `repair_attempts` as a retired setting and replace them with
|
||
an exact one-attempt contract assertion.
|
||
3. Add `repairAttempts` to successful and failed generate and batch JSON result
|
||
shapes through the Stage 5 pointers. Emit integer zero for a completed
|
||
first-pass result, a positive integer for a completed repaired result, and
|
||
omit the field when no completed validation made it available.
|
||
4. Keep the existing `validationStatus` and failure categories authoritative.
|
||
A repaired valid result proceeds normally. Repair exhaustion remains
|
||
`validation_rejected`, publishes no report for that profile, and retains the
|
||
actual attempt count.
|
||
5. Add representative offline assembled tests proving first-pass success,
|
||
repaired success, exhaustion, explicit empty initial content, and batch
|
||
result propagation. Reuse the real PromptKit adapter with an injected
|
||
sequence client for at least one end-to-end repaired execution; use the
|
||
existing app fake at other boundaries where lower-level repair is already
|
||
covered.
|
||
6. Confirm no application loop, provider retry, profile fallback, or
|
||
request-level `OutputContract` override was introduced.
|
||
|
||
Canonical documentation in this stage:
|
||
|
||
- update `docs/policy/architecture.md` with PromptKit-owned bounded repair and
|
||
failed-exhaustion invariants;
|
||
- update `docs/integrations/promptkit.md` with exact prompt version `2.1.0`, one
|
||
configured repair, actual-count semantics, cumulative usage, explicit-empty
|
||
handling, and the distinction from operational retries;
|
||
- update `docs/cli.md` for generate and batch `repairAttempts` fields;
|
||
- update `docs/internal/report-registry.md`, prepared-report internals, and app
|
||
orchestration internals for exact version and repair flow; and
|
||
- keep configuration documentation unchanged because no repair setting is
|
||
added.
|
||
|
||
Focused verification:
|
||
|
||
```sh
|
||
GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
|
||
```
|
||
|
||
### Stage 7: Migrate Comparison Bundles To v2
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: preserve repair activity in the profile-evaluation artifact and make
|
||
the strict durable schema change explicit.
|
||
|
||
Work:
|
||
|
||
1. Change `comparison.SchemaVersion` to
|
||
`weatherreporter.comparison.v2`. Emit and recognize v2 only; do not retain a
|
||
v1 parser or guarded-replacement compatibility path.
|
||
2. Add `RepairAttempts *int` to each application comparison profile result,
|
||
CLI comparison profile summary, and durable `comparison.Result`. Propagate a
|
||
fresh copy from Stage 5's execution outcome.
|
||
3. Place `repairAttempts` immediately after `validationStatus` in the canonical
|
||
result-object JSON field order. Encode zero for completed first-pass
|
||
validation, a positive integer for completed correction, and omit it only
|
||
when no completed validation exists.
|
||
4. Tighten manifest invariants: every non-nil repair count is non-negative; a
|
||
successful result must have `validationStatus: "passed"` and a non-nil
|
||
repair count; a failed result with a completed validation status must also
|
||
have a non-nil count; and an early operational failure may omit both.
|
||
5. Update the strict token-level JSON recognizer to accept only the canonical
|
||
`repairAttempts` field at its correct object level, reject duplicate,
|
||
unknown, negative, fractional, string, overflow, and malformed values, and
|
||
continue rejecting v1 as an unsupported current bundle.
|
||
6. Update manifest construction, cloning, validation, exact serialization
|
||
tests, guarded replacement tests, malicious bundle tests, partial-success
|
||
tests, and CLI comparison summaries. Preserve flat layout, result ordering,
|
||
hashes, atomic publication, cancellation safety, and no Distributor calls.
|
||
7. Cover concurrent peers where one succeeds first-pass, one repairs, one
|
||
exhausts, and one fails operationally. The counts must remain attached to
|
||
the selected profile positions without races or cross-contamination.
|
||
|
||
Canonical documentation in this stage:
|
||
|
||
- replace the v1 contract in `docs/integrations/comparison-bundle.md` with v2,
|
||
including exact field order, presence rules, and the lack of v1 replacement
|
||
compatibility;
|
||
- update `docs/cli.md` for comparison `repairAttempts`;
|
||
- update `docs/operations.md` to tell operators to move or remove an existing
|
||
v1 bundle before replacing at the same destination; and
|
||
- update comparison execution/publication internals and architecture policy as
|
||
needed for the current-only version invariant.
|
||
|
||
Focused verification:
|
||
|
||
```sh
|
||
GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli
|
||
GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app
|
||
```
|
||
|
||
### Stage 8: Add Secure Provider-Failure Debug Capture
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the
|
||
existing explicit secure debug boundary while keeping ordinary errors safe.
|
||
|
||
Work:
|
||
|
||
1. Extend prompt preparation debug output with configured
|
||
`repairAttempts` and advance its schema identifier from
|
||
`weatherreporter.prompt_preparation_debug.v2` to
|
||
`weatherreporter.prompt_preparation_debug.v3`.
|
||
2. Extend execution validation debug output with actual `repairAttempts` and
|
||
advance its schema identifier from
|
||
`weatherreporter.prompt_execution_debug.v2` to
|
||
`weatherreporter.prompt_execution_debug.v3`. Retain cumulative token usage.
|
||
3. Add a dedicated `failure.json` artifact with schema identifier
|
||
`weatherreporter.prompt_failure_debug.v1`. Its canonical fields are:
|
||
|
||
- top level: `schemaVersion`, `reportId`, `validDate`, `runId`, `failure`;
|
||
- failure object: `category`, `statusCode`, `providerCode`, `providerType`,
|
||
`providerMessage`;
|
||
- omit absent provider fields and zero status; and
|
||
- never include the raw provider body, headers, endpoint, credentials,
|
||
request, schema, rendered prompt, or generated candidate.
|
||
|
||
4. Add `PromptDebugWriter.WriteFailure` using the existing handle-relative
|
||
secure run directory, `0700` directory and `0600` file modes, canonical JSON
|
||
encoding, and no-follow/atomic replacement behavior. Disabled writers must
|
||
perform no filesystem work.
|
||
5. When execution returns an error, use `errors.As` only against the
|
||
project-owned `*promptexec.GenerationError`. If explicit debug capture is
|
||
enabled, write `failure.json` using that profile's existing debug reference.
|
||
This applies equally to initial and corrective provider failures and keeps
|
||
comparison profile directories isolated.
|
||
6. If failure-debug writing also fails, retain the generation failure as the
|
||
primary categorized error and join the safe debug-write failure rather than
|
||
replacing or hiding the provider failure. Never place provider code, type,
|
||
or message in the joined error text.
|
||
7. Ordinary generate, batch, and comparison errors should gain only the safe
|
||
HTTP status already rendered by `promptexec.GenerationError.Error`; do not
|
||
add provider detail fields to CLI summaries, comparison manifests, logs, or
|
||
Distributor requests.
|
||
8. Add adversarial tests for formatter redaction, malicious provider strings,
|
||
JSON escaping, bounds, absent fields, file modes, symlink/path attacks,
|
||
write failure, cancellation identity, initial versus corrective failures,
|
||
and concurrent comparison captures.
|
||
|
||
Canonical documentation in this stage:
|
||
|
||
- update `docs/operations.md` with the three debug artifact versions,
|
||
`failure.json`, sensitivity, permissions, and retention;
|
||
- update `docs/integrations/promptkit.md` with ordinary status-only disclosure
|
||
and debug-only provider detail;
|
||
- update prompt-debug, PromptKit-adapter, and app-orchestration internals; and
|
||
- ensure `docs/policy/architecture.md` explicitly prohibits provider-controlled
|
||
diagnostics from ordinary outputs.
|
||
|
||
Focused verification:
|
||
|
||
```sh
|
||
GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||
GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app
|
||
```
|
||
|
||
### Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit
|
||
|
||
Status: Complete.
|
||
|
||
Purpose: verify the complete end state as one coherent WeatherReporter feature
|
||
and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile,
|
||
repair, or debug claims.
|
||
|
||
Work:
|
||
|
||
1. Re-read the feature roadmap, `docs/development.md`, every policy document,
|
||
and every canonical document changed by Stages 1-8. Reconcile them against
|
||
executable behavior and remove duplicated or stale definitions. Keep
|
||
unimplemented future ideas in `docs/roadmap/future.md`, not current-state
|
||
documents.
|
||
2. Search code, embedded assets, examples, tests, and documentation for stale
|
||
contractual literals and review every occurrence of:
|
||
|
||
- PromptKit `v0.5.0`, `v0.6.0`, and `v0.7.0` as an active dependency claim;
|
||
- prompt version `2.0.0`;
|
||
- `weatherreporter.comparison.v1`;
|
||
- prompt debug schema v2 identifiers;
|
||
- claims that profile fields never inherit;
|
||
- claims that every `APIKeyEnv` must be populated;
|
||
- claims that backend ID is always required;
|
||
- claims that repair is disabled or `repair_attempts` is retired; and
|
||
- provider detail in ordinary output.
|
||
|
||
Historical release documents may retain accurate historical literals.
|
||
3. Verify canonical ownership:
|
||
|
||
- architecture owns invariants and boundaries;
|
||
- config owns operator profile and credential behavior, but no repair field;
|
||
- PromptKit integration owns logical prompt/profile/output contracts;
|
||
- CLI owns result fields;
|
||
- operations owns explicit debug handling and old comparison-bundle cleanup;
|
||
- comparison integration owns the complete v2 manifest; and
|
||
- internal documents own implementation flow without duplicating the public
|
||
references.
|
||
|
||
4. Verify maintained examples remain valid, secret-free, and tested. The local
|
||
`weather-light` example remains a standalone endpoint-only profile rather
|
||
than inheriting an OpenRouter backend it cannot clear.
|
||
5. Review the complete diff for architecture leakage. Production packages
|
||
outside `internal/adapters/promptkit` must not import PromptKit; no
|
||
application repair loop, provider client, raw provider diagnostic, profile
|
||
YAML parser, or durable application state may have appeared.
|
||
6. Review tests under the testing policy. Keep consumer contract and regression
|
||
coverage, remove accidental duplication of upstream implementation tests,
|
||
and ensure every default test is offline and repeatable.
|
||
7. Run the complete validation set:
|
||
|
||
```sh
|
||
gofmt -w <all changed Go files>
|
||
GOWORK=off go test -count=1 ./...
|
||
GOWORK=off go test -race -count=1 ./...
|
||
GOWORK=off go vet ./...
|
||
GOWORK=off go build ./...
|
||
GOWORK=off go mod tidy -diff
|
||
go run ./cmd/weatherreporter --help
|
||
go run ./cmd/weatherreporter compare --help
|
||
test -z "$(git ls-files go.work go.work.sum)"
|
||
test ! -e vendor
|
||
git diff --check
|
||
```
|
||
|
||
8. Confirm `go.mod` has no `replace`, the resolved PromptKit module is exactly
|
||
v0.8.0, and no live credential or provider call occurred during validation.
|
||
9. After every check passes, update this plan's status to Completed and add a
|
||
concise completion note listing the implemented stages. Do not delete either
|
||
roadmap until the maintainer has reviewed the implementation. Do not create
|
||
a release document or tag; release preparation remains a separate maintainer
|
||
action once a version is selected.
|
||
|
||
## Completion Standard
|
||
|
||
The implementation is complete only when all nine stages pass their focused
|
||
and repository-wide checks, all locked decisions are observable in code and
|
||
canonical documentation, and the feature roadmap's completion criteria are
|
||
satisfied. Passing compilation alone is insufficient. The final state must
|
||
demonstrate repaired success, repair exhaustion, comparison provenance,
|
||
endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome
|
||
inspection, safe ordinary provider failures, secure debug-only detail, and
|
||
unchanged publication and concurrency invariants.
|