Plan the PromptKit v0.8.0 upgrade
This commit is contained in:
571
docs/roadmap/implementation.md
Normal file
571
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,571 @@
|
||||
# PromptKit v0.8.0 Upgrade Implementation Plan
|
||||
|
||||
Status: Ready for implementation.
|
||||
|
||||
## Purpose And Authority
|
||||
|
||||
This plan translates the accepted
|
||||
[PromptKit v0.8.0 upgrade roadmap](promptkit-v0.8.0.md) into an ordered,
|
||||
decision-complete implementation procedure. The feature roadmap owns purpose,
|
||||
scope, policy, and the desired end state. This document owns implementation
|
||||
order, concrete work allocation, stage boundaries, and verification until the
|
||||
upgrade is complete.
|
||||
|
||||
The implementing agent must complete the stages in numerical order. Each stage
|
||||
is sized for one focused prompt handled by `gpt-5.6-terra` with high reasoning.
|
||||
Do not combine stages merely because adjacent work touches the same package.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
The following decisions are final for this implementation:
|
||||
|
||||
- upgrade directly from PromptKit `v0.5.0` to `v0.8.0`;
|
||||
- declare one corrective call in each embedded prompt through
|
||||
`repair_attempts: 1` rather than adding WeatherReporter repair logic;
|
||||
- keep the repair budget in the exact embedded prompt definition and add no
|
||||
global, per-report, CLI, profile, or operator configuration override;
|
||||
- advance all four exact prompt versions from `2.0.0` to `2.1.0`;
|
||||
- make `weather-light`, `weather-balanced`, and `weather-deep` minimal aliases
|
||||
of the corresponding PromptKit built-ins through `base_profile`;
|
||||
- retain the existing report-to-profile assignments and effective model
|
||||
ladder;
|
||||
- allow successfully inspected endpoint-only profiles to have an empty backend
|
||||
ID while continuing to require a nonblank model;
|
||||
- treat `APIKeyEnv` as an optional lookup source and reject only profiles that
|
||||
report `APIKeyRequired`, because WeatherReporter supplies no direct request
|
||||
credential;
|
||||
- support PromptKit's built-in `rakestrawhome-gemma-4-31b` profile without
|
||||
WeatherReporter-specific backend configuration;
|
||||
- expose provider HTTP status through a project-owned safe generation error,
|
||||
while writing provider code, type, and message only to explicit secure debug
|
||||
capture;
|
||||
- emit only `weatherreporter.comparison.v2`, with repair provenance, and do not
|
||||
preserve v1 guarded-replacement support; and
|
||||
- retain PromptKit dependency types inside the adapter and preserve all
|
||||
stateless execution, atomic publication, comparison independence, and
|
||||
disclosure invariants.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
For every stage:
|
||||
|
||||
- read `docs/development.md`, all files under `docs/policy/`, this plan, the
|
||||
feature roadmap, and the task-specific documents named by the stage;
|
||||
- inspect the current code and tests before editing; use the repository's code
|
||||
knowledge graph first for code discovery and fall back to text search for
|
||||
literals, assets, and documentation;
|
||||
- implement only the stage's scope and preserve unrelated user changes;
|
||||
- keep PromptKit/provider types, client construction, YAML parsing, repair
|
||||
mechanics, and provider transport inside the existing adapter boundary;
|
||||
- use deterministic, offline, credential-free tests and injected clients or
|
||||
synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local
|
||||
endpoint calls;
|
||||
- add tests at the narrowest stable owner identified by the testing policy and
|
||||
avoid copying PromptKit's internal test matrices;
|
||||
- update the canonical documentation owners listed for that stage in the same
|
||||
change as the implemented contract;
|
||||
- run `gofmt` on changed Go files, the stage's focused tests,
|
||||
`GOWORK=off go test -count=1 ./...`, and `git diff --check`; and
|
||||
- leave the repository passing before proceeding to the next stage.
|
||||
|
||||
Stages affecting concurrent comparison, cancellation, or secure debug
|
||||
filesystem work must also run the named focused packages with `-race`. Do not
|
||||
weaken an existing assertion solely to accommodate the new dependency. When a
|
||||
test encodes an intentionally changed contract, replace it with a behavioral
|
||||
assertion for the accepted policy.
|
||||
|
||||
## Implementation Stages
|
||||
|
||||
### Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline
|
||||
|
||||
Purpose: move to the tagged dependency and isolate compatibility changes before
|
||||
adopting new WeatherReporter behavior.
|
||||
|
||||
Work:
|
||||
|
||||
1. Update `go.mod` to require
|
||||
`gitea.maximumdirect.net/eric/promptkit v0.8.0` and refresh `go.sum` with
|
||||
`GOWORK=off go mod tidy`. Do not add `go.work`, `vendor`, or a `replace`
|
||||
directive and do not change WeatherReporter's Go version.
|
||||
2. Resolve any compile failures using PromptKit's public root package only.
|
||||
Keep all `Profile` and `OpenAICompatibleProfileConfig` literals keyed. Do not
|
||||
register the now-reserved `rakestrawhome` backend.
|
||||
3. Reconcile adapter tests that directly exercise PromptKit's changed optional
|
||||
credential behavior. A profile whose only credential metadata is
|
||||
`api_key_env` must reach an injected client when the environment value is
|
||||
absent; it must no longer expect PromptKit to return
|
||||
`ErrAPIKeyEnvMissing`. Do not change WeatherReporter's application preflight
|
||||
in this stage.
|
||||
4. Verify that every current embedded prompt, content file, schema, and fallback
|
||||
profile inspects under v0.8.0. Verify selected invalid local endpoints and
|
||||
malformed selected profile definitions still map to project-owned
|
||||
configuration or profile-load categories.
|
||||
5. Review the v0.6.0 compatibility corrections against supported
|
||||
WeatherReporter inputs: metadata-authoritative identities, exact contained
|
||||
`content_file` paths, regular embedded files, structurally valid endpoints,
|
||||
bounded JSON-compatible values, cancellation identity, and strict response
|
||||
framing. Add consumer tests only for a WeatherReporter boundary not already
|
||||
protected by PromptKit.
|
||||
|
||||
Do not enable profile inheritance or output repair yet. The expected result is
|
||||
the current WeatherReporter feature set running against PromptKit v0.8.0.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets
|
||||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
|
||||
```
|
||||
|
||||
### Stage 2: Adopt Profile Inheritance And Current Credential Routing
|
||||
|
||||
Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional
|
||||
credential semantics, and the Rakestrawhome built-in without changing the model
|
||||
ladder.
|
||||
|
||||
Work:
|
||||
|
||||
1. Replace the three embedded profile bodies with these exact leaf/base
|
||||
relationships and no duplicated execution settings:
|
||||
|
||||
| Leaf | Base |
|
||||
| --- | --- |
|
||||
| `weather-light` | `deepseek-4-flash` |
|
||||
| `weather-balanced` | `gemini-flash-latest` |
|
||||
| `weather-deep` | `claude-sonnet-latest` |
|
||||
|
||||
2. Update prompt-asset fixtures and tests to understand `base_profile`. Assert
|
||||
that all three leaf IDs remain selected identities and resolve to the same
|
||||
backend, model, timeout, service tier, and reasoning settings exposed by the
|
||||
current standalone definitions. Test relationships and effective behavior,
|
||||
not copied private constants beyond the intentional model-ladder contract.
|
||||
3. Preserve source precedence. Cover a standalone same-ID operator override, a
|
||||
derived operator override, a configured source that shadows a base ID, and
|
||||
selected missing-base, cyclic, malformed-base, and incomplete-target
|
||||
failures. Do not implement inheritance or merging in WeatherReporter; all
|
||||
resolution must remain PromptKit-owned.
|
||||
4. Change application profile preflight to accept a successful inspection with
|
||||
a nonblank model and an empty backend ID. Trust PromptKit inspection to have
|
||||
resolved either a backend or endpoint; do not add the endpoint to
|
||||
`promptexec.ProfileInspection` or ordinary provenance.
|
||||
5. Remove application-level environment lookup and rejection for a nonblank
|
||||
`APIKeyEnv`. Remove the now-unused `LookupEnv` fields and plumbing from
|
||||
prompt, batch, and comparison inspection requests. Continue rejecting
|
||||
`CredentialRequired`/`APIKeyRequired` before weather collection with the
|
||||
existing missing-credential category.
|
||||
6. Add an end-to-end offline regression proving the maintained endpoint-only
|
||||
`weather-light` example passes application inspection, retains an empty
|
||||
backend ID, and does not expose its endpoint.
|
||||
7. Prove `rakestrawhome-gemma-4-31b` can pass ordinary and comparison profile
|
||||
inspection through the existing adapter and reports the PromptKit
|
||||
`rakestrawhome` backend ID. Do not make a provider call or add
|
||||
Rakestrawhome-specific configuration.
|
||||
|
||||
Canonical documentation in this stage:
|
||||
|
||||
- update `docs/policy/architecture.md` so only direct-key-required profiles
|
||||
fail credential preflight and backend identity is optional for endpoint-only
|
||||
profiles;
|
||||
- update `docs/config.md` to distinguish same-ID source replacement from
|
||||
`base_profile` chain inheritance and to describe optional environment
|
||||
credentials;
|
||||
- update `docs/integrations/promptkit.md` for profile composition, parent
|
||||
lookup precedence, endpoint-only identity, optional credentials, and
|
||||
Rakestrawhome availability; and
|
||||
- update `docs/internal/promptkit-adapter.md` and focused app internals for the
|
||||
implemented inspection behavior.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app
|
||||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
|
||||
```
|
||||
|
||||
### Stage 3: Extend The Project-Owned Prompt Execution Contract
|
||||
|
||||
Purpose: establish dependency-neutral repair and structured-generation-error
|
||||
values before the adapter or application relies on them.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add `RepairAttempts int` to `promptexec.OutputContract`. It is the configured
|
||||
additional-call budget from the exact prompt contract.
|
||||
2. Add `RepairAttempts int` to `promptexec.Validation`. It is the number of
|
||||
corrective calls actually started for the completed result. Update
|
||||
`NewValidation` and every caller so construction is explicit; reject or
|
||||
normalize no values here because PromptKit owns output-contract validity.
|
||||
3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so
|
||||
repair values are retained without sharing mutable state.
|
||||
4. Add a project-owned immutable `promptexec.GenerationError` with unexported
|
||||
status and provider-detail fields plus safe accessors:
|
||||
|
||||
- `StatusCode() int`
|
||||
- `ProviderCode() string`
|
||||
- `ProviderType() string`
|
||||
- `ProviderMessage() string`
|
||||
- `Category() ErrorCategory`, always returning `Generation`
|
||||
- `Error()`, exposing only the WeatherReporter generation category/message
|
||||
and optional HTTP status
|
||||
- `GoString()`, returning the same safe representation
|
||||
- `Unwrap()`, preserving a project-owned `*promptexec.Error`
|
||||
|
||||
5. Provide one constructor used by adapters. Defensively normalize valid UTF-8
|
||||
and bound code/type to 256 Unicode code points and message to 4,096 Unicode
|
||||
code points, even though PromptKit already bounds its accessors. Do not
|
||||
expose fields through struct formatting, JSON tags, or exported mutable
|
||||
fields. Preserve the dependency cause only behind the project-owned error so
|
||||
`errors.Is`/`errors.As` identities remain available without entering error
|
||||
text.
|
||||
6. Add focused tests proving nil/zero safety, category and unwrap behavior,
|
||||
status-only ordinary formatting, `%#v` redaction, provider-detail bounds,
|
||||
and repair-value copying.
|
||||
|
||||
Do not import PromptKit from `internal/promptexec` and do not change CLI or
|
||||
artifact schemas in this stage.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/promptexec
|
||||
```
|
||||
|
||||
### Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter
|
||||
|
||||
Purpose: make the adapter faithfully translate v0.8.0 preparation, execution,
|
||||
validation, usage, and failure values into the Stage 3 contract.
|
||||
|
||||
Work:
|
||||
|
||||
1. Map `promptkit.OutputContract.RepairAttempts` in prompt inspection and
|
||||
prepared-execution details. Map
|
||||
`promptkit.ValidationResult.RepairAttempts` in completed execution.
|
||||
2. Preserve PromptKit's cumulative usage exactly as reported across the initial
|
||||
call and every completed correction. Continue returning only the final raw
|
||||
candidate and final validation result, subject to WeatherReporter's 64 KiB
|
||||
generated-output bound.
|
||||
3. In adapter error classification, retain cancellation, deadline, and capacity
|
||||
precedence. Before the generic `ErrLLMGenerate` branch, use `errors.As` for
|
||||
`*promptkit.GenerationError` and construct the project-owned
|
||||
`promptexec.GenerationError` with status, code, type, message, and hidden
|
||||
cause. Initial and corrective generation failures use the same mapping.
|
||||
4. Extend the injected adapter client used by tests so it can return an ordered
|
||||
sequence of responses or errors and record each request safely.
|
||||
5. Use a synthetic PromptKit prompt with JSON Schema validation and
|
||||
`repair_attempts: 1` to cover:
|
||||
|
||||
- first-pass valid output with zero corrections;
|
||||
- explicitly empty or invalid output followed by valid corrected output;
|
||||
- one-attempt exhaustion returning a final failed validation result rather
|
||||
than an operational error;
|
||||
- a non-2xx-style `GenerationError` during correction;
|
||||
- cumulative token usage and actual repair count; and
|
||||
- the same prepared prompt/profile identity across the corrective flow.
|
||||
|
||||
6. Keep these tests at the adapter boundary. Do not assert PromptKit's private
|
||||
corrective-message wording or reconstruct its internal repair algorithm.
|
||||
|
||||
Canonical documentation in this stage: update
|
||||
`docs/internal/promptkit-adapter.md` for the repair/result/error mappings. Do
|
||||
not yet claim that embedded WeatherReporter prompts enable repair.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec
|
||||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
|
||||
```
|
||||
|
||||
### Stage 5: Carry Repair Provenance Through Application Workflows
|
||||
|
||||
Purpose: make application orchestration understand configured and actual repair
|
||||
counts before changing the embedded prompt policy.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add an expected generated-text repair budget to `report.Definition` and set
|
||||
it explicitly to zero for all four current `2.0.0` definitions in this
|
||||
stage. Include it in report-definition validation and retained contract
|
||||
tests.
|
||||
2. Extend exact prompt preflight so format, validation mode, schema path, and
|
||||
repair budget must all match the resolved report definition. Extend
|
||||
preparation and completion provenance checks to require the same repair
|
||||
budget across inspection and the opaque prepared snapshot.
|
||||
3. Add `RepairAttempts *int` to application outcomes where execution may fail
|
||||
before validation exists. Set it to a fresh pointer immediately after a
|
||||
non-nil completed execution is returned, before WeatherReporter's secondary
|
||||
generated-text validation. A pointer is required so completed first-pass
|
||||
zero is distinguishable from unavailable provenance.
|
||||
4. Carry independent copies through `ReportResult`, `BatchReportResult`, batch
|
||||
conversion, comparison execution's internal outcome, and relevant test
|
||||
fakes. Do not expose the new value in CLI or comparison JSON yet.
|
||||
5. Preserve the actual count on PromptKit validation rejection and on later
|
||||
WeatherReporter generated-text or render failures. Leave it unavailable on
|
||||
preparation, capacity, cancellation, deadline, and generation errors that
|
||||
return no completed PromptKit result.
|
||||
6. Add focused tests for provenance mismatch, completed zero, completed
|
||||
positive, validation rejection, later local validation failure, early
|
||||
operational failure, batch copying, and independent concurrent profile
|
||||
outcomes.
|
||||
|
||||
Canonical documentation in this stage: update the focused prepared-report and
|
||||
app-orchestration internals to describe configured versus actual repair
|
||||
provenance. Current public documents should continue to report the embedded
|
||||
budget as zero until Stage 6.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli
|
||||
GOWORK=off go test -race -count=1 ./internal/app
|
||||
```
|
||||
|
||||
### Stage 6: Activate One Repair And Expose Ordinary Result Provenance
|
||||
|
||||
Purpose: switch the operational prompts to the accepted one-correction policy
|
||||
and make ordinary generate/run/batch output report what occurred.
|
||||
|
||||
Work:
|
||||
|
||||
1. Add `repair_attempts: 1` to the output contract of all four embedded prompt
|
||||
definitions and change each exact prompt version from `2.0.0` to `2.1.0`.
|
||||
Do not change prompt text or generated-text schemas solely for this upgrade.
|
||||
2. Change all four report registry definitions to exact prompt version `2.1.0`
|
||||
and expected repair budget one. Update exact-version fixtures and assertions
|
||||
throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests
|
||||
that classify `repair_attempts` as a retired setting and replace them with
|
||||
an exact one-attempt contract assertion.
|
||||
3. Add `repairAttempts` to successful and failed generate and batch JSON result
|
||||
shapes through the Stage 5 pointers. Emit integer zero for a completed
|
||||
first-pass result, a positive integer for a completed repaired result, and
|
||||
omit the field when no completed validation made it available.
|
||||
4. Keep the existing `validationStatus` and failure categories authoritative.
|
||||
A repaired valid result proceeds normally. Repair exhaustion remains
|
||||
`validation_rejected`, publishes no report for that profile, and retains the
|
||||
actual attempt count.
|
||||
5. Add representative offline assembled tests proving first-pass success,
|
||||
repaired success, exhaustion, explicit empty initial content, and batch
|
||||
result propagation. Reuse the real PromptKit adapter with an injected
|
||||
sequence client for at least one end-to-end repaired execution; use the
|
||||
existing app fake at other boundaries where lower-level repair is already
|
||||
covered.
|
||||
6. Confirm no application loop, provider retry, profile fallback, or
|
||||
request-level `OutputContract` override was introduced.
|
||||
|
||||
Canonical documentation in this stage:
|
||||
|
||||
- update `docs/policy/architecture.md` with PromptKit-owned bounded repair and
|
||||
failed-exhaustion invariants;
|
||||
- update `docs/integrations/promptkit.md` with exact prompt version `2.1.0`, one
|
||||
configured repair, actual-count semantics, cumulative usage, explicit-empty
|
||||
handling, and the distinction from operational retries;
|
||||
- update `docs/cli.md` for generate and batch `repairAttempts` fields;
|
||||
- update `docs/internal/report-registry.md`, prepared-report internals, and app
|
||||
orchestration internals for exact version and repair flow; and
|
||||
- keep configuration documentation unchanged because no repair setting is
|
||||
added.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
|
||||
```
|
||||
|
||||
### Stage 7: Migrate Comparison Bundles To v2
|
||||
|
||||
Purpose: preserve repair activity in the profile-evaluation artifact and make
|
||||
the strict durable schema change explicit.
|
||||
|
||||
Work:
|
||||
|
||||
1. Change `comparison.SchemaVersion` to
|
||||
`weatherreporter.comparison.v2`. Emit and recognize v2 only; do not retain a
|
||||
v1 parser or guarded-replacement compatibility path.
|
||||
2. Add `RepairAttempts *int` to each application comparison profile result,
|
||||
CLI comparison profile summary, and durable `comparison.Result`. Propagate a
|
||||
fresh copy from Stage 5's execution outcome.
|
||||
3. Place `repairAttempts` immediately after `validationStatus` in the canonical
|
||||
result-object JSON field order. Encode zero for completed first-pass
|
||||
validation, a positive integer for completed correction, and omit it only
|
||||
when no completed validation exists.
|
||||
4. Tighten manifest invariants: every non-nil repair count is non-negative; a
|
||||
successful result must have `validationStatus: "passed"` and a non-nil
|
||||
repair count; a failed result with a completed validation status must also
|
||||
have a non-nil count; and an early operational failure may omit both.
|
||||
5. Update the strict token-level JSON recognizer to accept only the canonical
|
||||
`repairAttempts` field at its correct object level, reject duplicate,
|
||||
unknown, negative, fractional, string, overflow, and malformed values, and
|
||||
continue rejecting v1 as an unsupported current bundle.
|
||||
6. Update manifest construction, cloning, validation, exact serialization
|
||||
tests, guarded replacement tests, malicious bundle tests, partial-success
|
||||
tests, and CLI comparison summaries. Preserve flat layout, result ordering,
|
||||
hashes, atomic publication, cancellation safety, and no Distributor calls.
|
||||
7. Cover concurrent peers where one succeeds first-pass, one repairs, one
|
||||
exhausts, and one fails operationally. The counts must remain attached to
|
||||
the selected profile positions without races or cross-contamination.
|
||||
|
||||
Canonical documentation in this stage:
|
||||
|
||||
- replace the v1 contract in `docs/integrations/comparison-bundle.md` with v2,
|
||||
including exact field order, presence rules, and the lack of v1 replacement
|
||||
compatibility;
|
||||
- update `docs/cli.md` for comparison `repairAttempts`;
|
||||
- update `docs/operations.md` to tell operators to move or remove an existing
|
||||
v1 bundle before replacing at the same destination; and
|
||||
- update comparison execution/publication internals and architecture policy as
|
||||
needed for the current-only version invariant.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli
|
||||
GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app
|
||||
```
|
||||
|
||||
### Stage 8: Add Secure Provider-Failure Debug Capture
|
||||
|
||||
Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the
|
||||
existing explicit secure debug boundary while keeping ordinary errors safe.
|
||||
|
||||
Work:
|
||||
|
||||
1. Extend prompt preparation debug output with configured
|
||||
`repairAttempts` and advance its schema identifier from
|
||||
`weatherreporter.prompt_preparation_debug.v2` to
|
||||
`weatherreporter.prompt_preparation_debug.v3`.
|
||||
2. Extend execution validation debug output with actual `repairAttempts` and
|
||||
advance its schema identifier from
|
||||
`weatherreporter.prompt_execution_debug.v2` to
|
||||
`weatherreporter.prompt_execution_debug.v3`. Retain cumulative token usage.
|
||||
3. Add a dedicated `failure.json` artifact with schema identifier
|
||||
`weatherreporter.prompt_failure_debug.v1`. Its canonical fields are:
|
||||
|
||||
- top level: `schemaVersion`, `reportId`, `validDate`, `runId`, `failure`;
|
||||
- failure object: `category`, `statusCode`, `providerCode`, `providerType`,
|
||||
`providerMessage`;
|
||||
- omit absent provider fields and zero status; and
|
||||
- never include the raw provider body, headers, endpoint, credentials,
|
||||
request, schema, rendered prompt, or generated candidate.
|
||||
|
||||
4. Add `PromptDebugWriter.WriteFailure` using the existing handle-relative
|
||||
secure run directory, `0700` directory and `0600` file modes, canonical JSON
|
||||
encoding, and no-follow/atomic replacement behavior. Disabled writers must
|
||||
perform no filesystem work.
|
||||
5. When execution returns an error, use `errors.As` only against the
|
||||
project-owned `*promptexec.GenerationError`. If explicit debug capture is
|
||||
enabled, write `failure.json` using that profile's existing debug reference.
|
||||
This applies equally to initial and corrective provider failures and keeps
|
||||
comparison profile directories isolated.
|
||||
6. If failure-debug writing also fails, retain the generation failure as the
|
||||
primary categorized error and join the safe debug-write failure rather than
|
||||
replacing or hiding the provider failure. Never place provider code, type,
|
||||
or message in the joined error text.
|
||||
7. Ordinary generate, batch, and comparison errors should gain only the safe
|
||||
HTTP status already rendered by `promptexec.GenerationError.Error`; do not
|
||||
add provider detail fields to CLI summaries, comparison manifests, logs, or
|
||||
Distributor requests.
|
||||
8. Add adversarial tests for formatter redaction, malicious provider strings,
|
||||
JSON escaping, bounds, absent fields, file modes, symlink/path attacks,
|
||||
write failure, cancellation identity, initial versus corrective failures,
|
||||
and concurrent comparison captures.
|
||||
|
||||
Canonical documentation in this stage:
|
||||
|
||||
- update `docs/operations.md` with the three debug artifact versions,
|
||||
`failure.json`, sensitivity, permissions, and retention;
|
||||
- update `docs/integrations/promptkit.md` with ordinary status-only disclosure
|
||||
and debug-only provider detail;
|
||||
- update prompt-debug, PromptKit-adapter, and app-orchestration internals; and
|
||||
- ensure `docs/policy/architecture.md` explicitly prohibits provider-controlled
|
||||
diagnostics from ordinary outputs.
|
||||
|
||||
Focused verification:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||
GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app
|
||||
```
|
||||
|
||||
### Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit
|
||||
|
||||
Purpose: verify the complete end state as one coherent WeatherReporter feature
|
||||
and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile,
|
||||
repair, or debug claims.
|
||||
|
||||
Work:
|
||||
|
||||
1. Re-read the feature roadmap, `docs/development.md`, every policy document,
|
||||
and every canonical document changed by Stages 1-8. Reconcile them against
|
||||
executable behavior and remove duplicated or stale definitions. Keep
|
||||
unimplemented future ideas in `docs/roadmap/future.md`, not current-state
|
||||
documents.
|
||||
2. Search code, embedded assets, examples, tests, and documentation for stale
|
||||
contractual literals and review every occurrence of:
|
||||
|
||||
- PromptKit `v0.5.0`, `v0.6.0`, and `v0.7.0` as an active dependency claim;
|
||||
- prompt version `2.0.0`;
|
||||
- `weatherreporter.comparison.v1`;
|
||||
- prompt debug schema v2 identifiers;
|
||||
- claims that profile fields never inherit;
|
||||
- claims that every `APIKeyEnv` must be populated;
|
||||
- claims that backend ID is always required;
|
||||
- claims that repair is disabled or `repair_attempts` is retired; and
|
||||
- provider detail in ordinary output.
|
||||
|
||||
Historical release documents may retain accurate historical literals.
|
||||
3. Verify canonical ownership:
|
||||
|
||||
- architecture owns invariants and boundaries;
|
||||
- config owns operator profile and credential behavior, but no repair field;
|
||||
- PromptKit integration owns logical prompt/profile/output contracts;
|
||||
- CLI owns result fields;
|
||||
- operations owns explicit debug handling and old comparison-bundle cleanup;
|
||||
- comparison integration owns the complete v2 manifest; and
|
||||
- internal documents own implementation flow without duplicating the public
|
||||
references.
|
||||
|
||||
4. Verify maintained examples remain valid, secret-free, and tested. The local
|
||||
`weather-light` example remains a standalone endpoint-only profile rather
|
||||
than inheriting an OpenRouter backend it cannot clear.
|
||||
5. Review the complete diff for architecture leakage. Production packages
|
||||
outside `internal/adapters/promptkit` must not import PromptKit; no
|
||||
application repair loop, provider client, raw provider diagnostic, profile
|
||||
YAML parser, or durable application state may have appeared.
|
||||
6. Review tests under the testing policy. Keep consumer contract and regression
|
||||
coverage, remove accidental duplication of upstream implementation tests,
|
||||
and ensure every default test is offline and repeatable.
|
||||
7. Run the complete validation set:
|
||||
|
||||
```sh
|
||||
gofmt -w <all changed Go files>
|
||||
GOWORK=off go test -count=1 ./...
|
||||
GOWORK=off go test -race -count=1 ./...
|
||||
GOWORK=off go vet ./...
|
||||
GOWORK=off go build ./...
|
||||
GOWORK=off go mod tidy -diff
|
||||
go run ./cmd/weatherreporter --help
|
||||
go run ./cmd/weatherreporter compare --help
|
||||
test -z "$(git ls-files go.work go.work.sum)"
|
||||
test ! -e vendor
|
||||
git diff --check
|
||||
```
|
||||
|
||||
8. Confirm `go.mod` has no `replace`, the resolved PromptKit module is exactly
|
||||
v0.8.0, and no live credential or provider call occurred during validation.
|
||||
9. After every check passes, update this plan's status to Completed and add a
|
||||
concise completion note listing the implemented stages. Do not delete either
|
||||
roadmap until the maintainer has reviewed the implementation. Do not create
|
||||
a release document or tag; release preparation remains a separate maintainer
|
||||
action once a version is selected.
|
||||
|
||||
## Completion Standard
|
||||
|
||||
The implementation is complete only when all nine stages pass their focused
|
||||
and repository-wide checks, all locked decisions are observable in code and
|
||||
canonical documentation, and the feature roadmap's completion criteria are
|
||||
satisfied. Passing compilation alone is insufficient. The final state must
|
||||
demonstrate repaired success, repair exhaustion, comparison provenance,
|
||||
endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome
|
||||
inspection, safe ordinary provider failures, secure debug-only detail, and
|
||||
unchanged publication and concurrency invariants.
|
||||
Reference in New Issue
Block a user