Prepare reports for Promptkit migration
This commit is contained in:
944
docs/roadmap/implementation.md
Normal file
944
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,944 @@
|
||||
# Promptkit Migration Implementation Plan
|
||||
|
||||
Status: Decision-complete implementation plan; unimplemented.
|
||||
|
||||
## Purpose And Authority
|
||||
|
||||
This document defines the ordered implementation procedure for the
|
||||
[Promptkit migration roadmap](promptkit.md). The feature roadmap is
|
||||
authoritative for scope, user intent, policy choices, and the desired end
|
||||
state. This plan is authoritative for sequencing, concrete package ownership,
|
||||
compatibility work, tests, documentation updates, and completion gates.
|
||||
|
||||
Implement the stages in order. Do not reinterpret a roadmap decision merely
|
||||
because the current Scriptorium implementation makes another path shorter.
|
||||
When implementation reveals a conflict with the roadmap, stop and update the
|
||||
roadmap and this plan deliberately rather than introducing an implicit policy
|
||||
change.
|
||||
|
||||
This plan follows the repository's
|
||||
[architecture](../policy/architecture.md),
|
||||
[documentation](../policy/documentation.md), and
|
||||
[testing](../policy/testing.md) policies. All stages are parts of one
|
||||
migration change. Intermediate stages may temporarily retain code needed by a
|
||||
later cutover, but no intermediate state should be released or documented as
|
||||
the final current behavior.
|
||||
|
||||
## Cross-Stage Rules
|
||||
|
||||
- Pin `gitea.maximumdirect.net/eric/promptkit` at exactly `v0.4.0`. Do not
|
||||
commit a `go.work`, local `replace`, pseudo-version, or unpublished commit.
|
||||
- Keep Promptkit types inside `internal/adapters/promptkit`, that package's
|
||||
tests, and the external contract test that verifies `internal/promptassets`
|
||||
with the real Promptkit inspector. App, CLI, report, state, and domain
|
||||
packages use project-owned types.
|
||||
- Keep Promptkit's opaque prepared-execution handle inside its adapter. Do not
|
||||
expose it through an app interface, serialize it, or make it restartable.
|
||||
- Construct one Promptkit engine per `generate` or `run` CLI invocation.
|
||||
Every report in a batch shares that engine. Inspection commands that only
|
||||
read persisted state do not construct an engine.
|
||||
- Keep batches sequential. Do not add retries, output repair, direct Markdown
|
||||
generation, arbitrary backend registration, or live-provider tests.
|
||||
- Preserve the exact persisted `data_package` bytes as the only model input.
|
||||
Promptkit receives those bytes through an inline artifact with the managed
|
||||
YAML path as provenance; it never receives an unrestricted file reference.
|
||||
- Persist successful preparation before `RunPrepared`. If preparation-record
|
||||
persistence or enabled debug persistence fails, discard the handle and do
|
||||
not call the provider.
|
||||
- Keep normal artifacts, errors, logs, and CLI summaries free of credentials,
|
||||
rendered messages, schema bodies, data-package bodies, provider response
|
||||
bodies, and full effective parameter maps.
|
||||
- Use atomic writes for durable state and debug files. Preserve every
|
||||
non-secret artifact reached before a later failure when practical.
|
||||
- Use deterministic Promptkit model-client fakes at the adapter boundary and
|
||||
project-owned executor fakes at the app boundary. The default test suite
|
||||
remains offline and credential-free.
|
||||
- At each stage, update or delete existing tests according to the testing
|
||||
policy. Protect contracts and failure behavior; do not preserve tests that
|
||||
exist only to assert the retired subprocess implementation.
|
||||
- Run the focused checks named by the stage while iterating. Before completing
|
||||
every stage, run `git diff --check`. Run the full repository gate in the
|
||||
final stage.
|
||||
|
||||
## Fixed Package And Contract Decisions
|
||||
|
||||
Use these ownership boundaries so later stages do not need to choose an
|
||||
architecture while editing:
|
||||
|
||||
| Area | Owner | Decision |
|
||||
| --- | --- | --- |
|
||||
| Embedded prompt definitions, referenced prompt content, and provider-facing schemas | `internal/promptassets` | Embed one centralized asset tree and expose read-only prompt and schema `fs.FS` sources plus schema lookup by report schema ID. This package does not import Promptkit. |
|
||||
| Project-owned prompt execution contract | `internal/promptexec` | Own prompt inspection, profile inspection, preparation, execution, validation, usage, debug, and neutral error values. This package does not import Promptkit, app, CLI, report, or state. |
|
||||
| Promptkit integration | `internal/adapters/promptkit` | Construct and own the Promptkit engine, translate project requests and results, classify public Promptkit errors, and enforce prepared-handle lifecycle. |
|
||||
| Workflow ordering | `internal/app` | Resolve reports, inspect prompt/profile selections, collect weather, build and save inputs, record preparation, invoke execution, persist results, perform final generated-text validation, render Markdown, and notify. |
|
||||
| Durable artifacts | `internal/state` | Own v2 paths, preparation and execution schemas, metadata compatibility, atomic writes, and inspection reads. |
|
||||
| Engine composition and flags | `internal/cli` | Construct one executor per action invocation, pass it into app requests, parse `--llm-debug-dir`, and expose project-owned summary paths. |
|
||||
|
||||
`internal/promptexec.Executor` must provide three operations:
|
||||
|
||||
1. inspect an exact prompt ID and version;
|
||||
2. inspect one explicit profile ID; and
|
||||
3. prepare and execute one request.
|
||||
|
||||
The execution operation accepts a preparation callback. The adapter calls that
|
||||
callback exactly once after successful `PrepareExecution` and before
|
||||
`RunPrepared`. The callback receives a safe project-owned preparation value
|
||||
and, only when requested, a project-owned sensitive debug value. A callback
|
||||
error aborts execution. A preparation failure returns a classified
|
||||
project-owned attempt error without calling the callback; app orchestration
|
||||
uses that error to persist a failure receipt.
|
||||
|
||||
The adapter execution request contains the exact prompt ID and version, the
|
||||
optional configured profile override, the exact YAML bytes, the managed
|
||||
data-package path used only as inline provenance, and whether sensitive debug
|
||||
capture is enabled. It does not contain a provider output path.
|
||||
|
||||
Project-owned errors use stable categories for invalid configuration, invalid
|
||||
request, prompt not found/load, profile not found/load, missing credential,
|
||||
artifact load, prompt render, capacity, generation, operational validation,
|
||||
completed validation rejection, cancellation, and deadline. A capacity error
|
||||
also carries the non-secret backend ID. Its `Error` text is
|
||||
Weatherreporter-owned and safe for CLI output; an unexported or non-serialized
|
||||
cause may remain available to `errors.Is` and `errors.As`. Never persist
|
||||
arbitrary dependency error prose.
|
||||
|
||||
## Stage 1: Correct The Implemented Report Surface
|
||||
|
||||
### Goal
|
||||
|
||||
Remove the unfinished three-day, weekend, and storm products before changing
|
||||
the LLM integration. The repository should expose only Daily, Today, Tomorrow,
|
||||
and Hourly while continuing to use Scriptorium temporarily for those four
|
||||
reports.
|
||||
|
||||
### Work
|
||||
|
||||
1. Remove the three report definitions and their files under `internal/report`.
|
||||
Remove their IDs, command names, aliases, period inputs, generation modes,
|
||||
batch flags, registry order, and direct-Markdown-only declarations.
|
||||
2. Add `PromptVersion string` to `report.Definition` and set it to `1.0.0` on
|
||||
all four retained definitions. Since every retained report uses generated
|
||||
text plus a repository template, remove the now-redundant
|
||||
`GenerationMode` and `Generated` fields and their branches.
|
||||
3. Remove three-day, weekend, and storm command parsing, help text, app request
|
||||
fields/constants, resolution branches, Distributor path variables, and
|
||||
configuration report aliases.
|
||||
4. Remove report-specific code whose only caller was one of the retired
|
||||
products. Use graph traces before deletion. This includes the dedicated
|
||||
three-day/weekend Recent Changes comparators, storm-window fact derivation,
|
||||
multi-day report derivation used only by these definitions, storm time
|
||||
parsing, and their focused tests when no retained behavior depends on them.
|
||||
Do not remove general meteorological behavior merely because a fixture or
|
||||
prose string contains words such as “storm” or “weekend.”
|
||||
5. Keep morning and evening planning behavior exactly as implemented:
|
||||
morning plans Today and Tomorrow, evening plans Tomorrow, and both may add
|
||||
eligible future Daily reports from complete hourly coverage.
|
||||
6. Update the current CLI, configuration, report-registry, facts, modules,
|
||||
changes, operations, troubleshooting, and other affected documentation to
|
||||
describe only the four implemented reports. Update `docs/roadmap/future.md`
|
||||
so it no longer claims that a manual Storm Report exists; future versions
|
||||
of these products remain explicitly unimplemented roadmap work.
|
||||
|
||||
### Tests
|
||||
|
||||
- Rewrite report registry and CLI parser tests around exactly four commands.
|
||||
- Remove fake successful-generation tests for nonexistent prompts.
|
||||
- Preserve Daily/Today/Tomorrow/Hourly period, module, batch-planning,
|
||||
Distributor path, and output-name coverage.
|
||||
- Add negative CLI/config tests showing the retired command names and aliases
|
||||
are rejected.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/report ./internal/config ./internal/facts \
|
||||
./internal/changes ./internal/app ./internal/cli
|
||||
go test ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
No implemented registry, CLI, configuration, app, current-state document, or
|
||||
test claims that three-day, weekend, or storm generation exists. The four
|
||||
retained reports still pass through the existing generated-text/template
|
||||
workflow, and each carries exact prompt version `1.0.0`.
|
||||
|
||||
## Stage 2: Promote And Reconcile Runtime Prompt Assets
|
||||
|
||||
### Goal
|
||||
|
||||
Create the single embedded prompt/schema corpus used by Promptkit and by
|
||||
Weatherreporter's final generated-text boundary.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add Promptkit `v0.4.0` to `go.mod` and `go.sum`.
|
||||
2. Create `internal/promptassets` with an embedded tree rooted at:
|
||||
|
||||
```text
|
||||
assets/
|
||||
prompts/
|
||||
common/
|
||||
daily/
|
||||
today/
|
||||
tomorrow/
|
||||
hourly/
|
||||
schemas/
|
||||
```
|
||||
|
||||
Expose copied/read-only prompt and schema filesystem views and a
|
||||
`Schema(id)` lookup for the four report schema IDs. Keep path constants
|
||||
private except where a stable project contract is necessary.
|
||||
3. Promote only the four `*_generated_text` prompt definitions from
|
||||
`docs/roadmap/scriptorium`. Do not promote `weather.daily_report`.
|
||||
4. Use the Scriptorium corpus's common system, data-package guidance, and
|
||||
report-specific files as the behavioral base. Reconcile rather than
|
||||
editorially rewrite:
|
||||
|
||||
- preserve its source weighting, hazard, precipitation, and style rules;
|
||||
- add the optional `confidence` field instruction to Daily, Today, and
|
||||
Hourly so all prompt text agrees with the canonical domain types and
|
||||
schemas;
|
||||
- retain Tomorrow's existing confidence instruction;
|
||||
- prefer the corpus's correct Tomorrow daily framing over the stale
|
||||
hourly-style file currently under `internal/reporttemplate/prompts`;
|
||||
- make no unrelated prompt-tone or product-policy changes.
|
||||
|
||||
5. Every prompt definition must use:
|
||||
|
||||
- its existing prompt ID;
|
||||
- version `1.0.0`;
|
||||
- `default_profile: gemini-flash-latest`;
|
||||
- one required `data_package` input with
|
||||
`content_type: application/yaml`;
|
||||
- JSON output with `validation_mode: json_schema`;
|
||||
- a path inside the embedded schema root; and
|
||||
- zero or omitted `repair_attempts`.
|
||||
|
||||
6. Move the four canonical schemas from `internal/reporttemplate/schemas` into
|
||||
`internal/promptassets/assets/schemas`. Preserve their current application
|
||||
contracts, including optional `confidence`, required summary/discussion,
|
||||
array discussion for day-style reports, string discussion for Hourly,
|
||||
`additionalProperties: false`, and the corrected Daily `$id` and title.
|
||||
7. Change `internal/generatedtext` to load these canonical schemas from
|
||||
`promptassets`. Remove schema and prompt-fragment ownership from
|
||||
`internal/reporttemplate`; it should embed and render Markdown templates and
|
||||
partials only.
|
||||
8. Retain `docs/roadmap/scriptorium` as migration source evidence until the
|
||||
final cleanup stage.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add table-driven asset tests for exactly four prompt IDs and versions,
|
||||
referenced-file resolution, default profiles, YAML input declarations,
|
||||
output contracts, and schema lookup.
|
||||
- Construct a Promptkit engine over the embedded sources in an external test
|
||||
package and call `InspectPrompt` for every report. Do not use a provider or
|
||||
credentials.
|
||||
- Retain generated-text schema and typed-validation tests, now reading the
|
||||
relocated canonical schemas.
|
||||
- Assert that no active embedded prompt uses `local-heavy`,
|
||||
`pipeline-weather/`, JSON input metadata, positive repair attempts, or
|
||||
`weather.daily_report`.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptassets ./internal/generatedtext \
|
||||
./internal/reporttemplate
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
One embedded source contains exactly four valid Promptkit prompt definitions
|
||||
and four canonical schemas. Prompt inspection succeeds offline for every
|
||||
report, and no duplicate runtime provider-facing schema remains.
|
||||
|
||||
## Stage 3: Define The Neutral Prompt Execution Contract
|
||||
|
||||
### Goal
|
||||
|
||||
Define the complete project-owned boundary between Weatherreporter and an LLM
|
||||
execution adapter. This stage introduces no Promptkit imports and makes no
|
||||
production orchestration changes.
|
||||
|
||||
### Work
|
||||
|
||||
1. Create `internal/promptexec` with the neutral contract fixed above. Include:
|
||||
|
||||
- exact prompt and profile inspection values;
|
||||
- safe preparation provenance;
|
||||
- execution result, validation, token usage, and timing;
|
||||
- optional sensitive preparation/execution debug values;
|
||||
- stable error categories and a typed capacity error; and
|
||||
- small helpers for safe diagnostic bounding and copying.
|
||||
|
||||
2. The safe preparation value includes prompt ID/version/hash, rendered prompt
|
||||
hash, input hashes, selected profile/backend, effective model name, output
|
||||
format/validation/schema path, preparation start/end/duration, and the
|
||||
managed data-package path. It excludes endpoints, API-key environment
|
||||
names, full parameters, rendered messages, schema bodies, and input bodies.
|
||||
3. The safe execution value includes Promptkit run ID, repeated prompt and
|
||||
input provenance, selected profile/backend/model, generated-content hash,
|
||||
usage, execution start/end/duration, completed validation status, bounded
|
||||
validation diagnostics, and raw output bytes. It excludes endpoint and full
|
||||
effective parameters.
|
||||
4. Bound routine validation diagnostics to at most 10 entries and at most
|
||||
1,024 bytes per entry, truncating on a valid UTF-8 boundary. Bound a
|
||||
persisted safe error message to 2,048 bytes. Prefer stable categories and
|
||||
fields over diagnostic prose.
|
||||
5. Specify the execution lifecycle in interface comments and tests:
|
||||
|
||||
- inspection is side-effect-free;
|
||||
- preparation invokes the callback exactly once only after successful
|
||||
preparation;
|
||||
- callback failure prevents provider execution;
|
||||
- completed validation rejection is a result, not an operational error;
|
||||
- operational failure does not fabricate a completed result; and
|
||||
- sensitive debug values are populated only when explicitly requested.
|
||||
6. Keep copying and truncation helpers internal to `promptexec`. Callers must
|
||||
not be able to mutate byte slices, maps, or diagnostics retained inside
|
||||
contract values.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add compile-time test fakes that implement the complete interface without
|
||||
importing Promptkit.
|
||||
- Add table-driven tests for every error category, capacity fields, diagnostic
|
||||
count and byte limits, UTF-8-safe truncation, error-message bounds, and
|
||||
defensive copying.
|
||||
- Assert through representative fixtures that safe contract values contain no
|
||||
endpoint, credential, rendered message, schema body, input body, response
|
||||
body, or full parameter map.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptexec
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
`internal/promptexec` provides a stable, documented, offline-tested contract
|
||||
that can represent every inspection, preparation, execution, validation,
|
||||
usage, debug, and failure outcome required by the roadmap. It has no Promptkit,
|
||||
app, CLI, report, or state dependency.
|
||||
|
||||
## Stage 4: Implement The Promptkit Adapter
|
||||
|
||||
### Goal
|
||||
|
||||
Implement and thoroughly test Promptkit behind the Stage 3 contract without
|
||||
modifying application or CLI orchestration.
|
||||
|
||||
### Work
|
||||
|
||||
1. Create `internal/adapters/promptkit`. Its constructor:
|
||||
|
||||
- uses `promptassets` through `WithPromptFS` and `WithSchemaFS`;
|
||||
- selects one external profile directory through Promptkit config or one
|
||||
profile file through `WithProfileFile`;
|
||||
- optionally registers
|
||||
`promptkit.LocalBackend(endpoint, concurrencyLimit)`;
|
||||
- sets the transport timeout;
|
||||
- accepts an injected Promptkit `LLMClient` only through an adapter-local
|
||||
test constructor or option; and
|
||||
- returns project-owned configuration errors.
|
||||
|
||||
2. Implement prompt and profile inspection with exact versions and explicit
|
||||
field-by-field translation into `promptexec` values. Do not return or embed
|
||||
Promptkit values.
|
||||
3. Implement execution:
|
||||
|
||||
- call `PrepareExecution` with the exact requested version and optional
|
||||
profile override;
|
||||
- pass the exact YAML as
|
||||
`InlineWithURI(dataPackagePath, string(bytes))`;
|
||||
- immediately `defer handle.Discard()` after a successful prepare;
|
||||
- map safe preparation details and invoke the preparation callback;
|
||||
- call `RunPrepared` only after the callback succeeds;
|
||||
- treat `ValidationFailed` as a completed result rather than an operational
|
||||
error; and
|
||||
- return no invented execution result for operational errors.
|
||||
|
||||
4. Translate Promptkit's public error sentinels with `errors.Is`, and
|
||||
`CapacityError` with `errors.As`. Preserve caller cancellation and deadline
|
||||
identities. Do not parse error strings.
|
||||
5. Capture sensitive debug values only when requested. Use project-owned debug
|
||||
structs and explicit field mapping rather than serializing whole Promptkit
|
||||
values, so a future upstream field cannot silently enter an artifact.
|
||||
6. Keep the prepared handle entirely within the adapter call. Discard it after
|
||||
every success or failure path and never make it serializable or reusable.
|
||||
|
||||
### Tests
|
||||
|
||||
- Use an injected deterministic Promptkit model client. Never use a live
|
||||
endpoint or credential.
|
||||
- Cover prompt/profile inspection, profile override precedence, built-in and
|
||||
external profiles, endpoint-only profiles, the conventional local backend,
|
||||
local concurrency/capacity rejection, missing credentials, and constructor
|
||||
validation.
|
||||
- Cover preparation callback ordering, callback failure preventing generation,
|
||||
exact inline bytes and URI, exact prompt version, successful execution,
|
||||
completed schema rejection with raw output, operational generation and
|
||||
validation failures with no partial result, cancellation, timeout, and
|
||||
handle discard.
|
||||
- Assert that safe values and errors do not contain rendered messages,
|
||||
endpoints, schema bodies, input bodies, response bodies, credentials, or
|
||||
full parameter maps.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptexec ./internal/adapters/promptkit
|
||||
go test -race ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
The Promptkit adapter is a complete offline-tested implementation of the
|
||||
neutral contract. No Promptkit type is referenced outside the adapter, its
|
||||
tests, or the external asset contract test, and no production path uses the
|
||||
adapter yet.
|
||||
|
||||
## Stage 5: Introduce Durable State V2
|
||||
|
||||
### Goal
|
||||
|
||||
Add the final normal-artifact model and backward-compatible metadata reading
|
||||
before app orchestration starts writing the new records. Sensitive debug
|
||||
storage remains out of scope until Stage 6.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add the new-run path members `Preparation` and `Execution`:
|
||||
|
||||
- preparation:
|
||||
`preflight/<group>/<date>/prompt_preparation.<runID>.json`;
|
||||
- execution:
|
||||
`snapshots/<group>/<date>/prompt_execution.<runID>.json`.
|
||||
|
||||
Keep `workspace.preflight_dir` and its default physical directory unchanged.
|
||||
Retain `Preflight` and `GeneratedTextResult` only as temporary legacy
|
||||
write-path members required by the uncut Scriptorium workflows; mark them
|
||||
for deletion in Stage 10. New Promptkit code must use only the new members.
|
||||
2. Add `PromptPreparationArtifact` with schema
|
||||
`weatherreporter.prompt_preparation.v1`. It represents either successful
|
||||
preparation or a failed preparation receipt and includes schema version,
|
||||
status, report/RunID, prompt identity, safe preparation provenance when
|
||||
available, timing, data-package path, and a bounded classified error when
|
||||
failed.
|
||||
3. Add `PromptExecutionArtifact` with schema
|
||||
`weatherreporter.prompt_execution.v1`. It represents success, completed
|
||||
validation rejection, or operational failure and includes safe execution
|
||||
provenance, validation, usage, timing, reached artifact paths, and a bounded
|
||||
classified error when failed. It never embeds generated content.
|
||||
4. Advance newly written metadata to `weatherreporter.metadata.v2`. Replace
|
||||
`preflightPath` and `generatedTextResultPath` with `preparationPath` and
|
||||
`executionPath`. Populate paths only after the corresponding artifact has
|
||||
actually been saved.
|
||||
5. Implement explicit v1/v2 metadata decoding:
|
||||
|
||||
- accept only the known v1 and v2 schema versions;
|
||||
- normalize v1 `preflightPath` and `generatedTextResultPath` internally for
|
||||
inspection;
|
||||
- preserve v1 field names when a loaded v1 record is marshaled by
|
||||
`inspect metadata`;
|
||||
- write only v2 through the new Promptkit state APIs; the temporary
|
||||
Scriptorium path may continue writing v1 until its Stage 10 removal; and
|
||||
- never dual-write legacy aliases.
|
||||
|
||||
Direct inspection of v1 metadata, modules, data packages, sources, and
|
||||
referenced artifacts remains available for any historical report ID.
|
||||
Prior-snapshot reconstruction is required only for the four retained report
|
||||
IDs; do not restore retired definitions solely for legacy comparison.
|
||||
6. Add typed `SavePromptPreparation`, `SavePromptExecution`, and corresponding
|
||||
typed load methods where inspection needs them. Temporarily retain the
|
||||
Scriptorium write methods so the current production paths compile through
|
||||
Stage 9; delete those methods during Stage 10. Keep long-term legacy support
|
||||
read-only.
|
||||
7. Update app/CLI result structs and JSON field names to
|
||||
`preparationPath`, `executionPath`, and optional `llmDebugPath`; temporarily
|
||||
adapt old orchestration so the tree compiles until the cutover stages.
|
||||
Do not write v2 records through the Scriptorium path.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add exact path, schema-defaulting, round-trip, required-field, atomic-write,
|
||||
and unknown-version tests.
|
||||
- Add v1 fixtures covering both generated-text and legacy preflight references;
|
||||
verify list, metadata, modules, data-package, and source inspection.
|
||||
- Verify that re-marshaled v1 inspection uses v1 field names and that v2 output
|
||||
contains no deprecated aliases.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/state ./internal/app ./internal/cli
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
State can read historical v1 runs and write the complete normal v2 artifact
|
||||
contract through the new APIs. The existing production generation path still
|
||||
compiles and remains unchanged until cutover; no new Promptkit artifact uses a
|
||||
Scriptorium-specific filename, schema, or JSON field.
|
||||
|
||||
## Stage 6: Add Secure LLM Debug Persistence
|
||||
|
||||
### Goal
|
||||
|
||||
Implement the explicitly enabled sensitive-debug store as a small, independently
|
||||
auditable boundary before any CLI command can request debug capture.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add a focused debug writer, owned by state or a narrow state-adjacent
|
||||
package, for an explicit operator root outside normal artifact derivation.
|
||||
It must:
|
||||
|
||||
- validate or create the root before collection or provider work;
|
||||
- use `<root>/<report-id>/<valid-date>/<run-id>/`;
|
||||
- write `preparation.json` and `execution.json` atomically;
|
||||
- create directories with `0700` and files with `0600`;
|
||||
- reject symlinks, path escape, non-directory roots, and unsafe
|
||||
run/report/date segments; and
|
||||
- return the per-run debug directory as the project-owned summary path.
|
||||
|
||||
2. Define explicit project-owned debug wire structs. Preparation debug may
|
||||
contain rendered messages, structured-output schema, and effective
|
||||
endpoint/parameters. Execution debug may additionally contain raw generated
|
||||
output and completed validation details.
|
||||
3. Exclude direct credentials, resolved environment credential values, and
|
||||
arbitrary dependency objects. Use field-by-field mapping so future
|
||||
`promptexec` or Promptkit fields are not serialized implicitly.
|
||||
4. Make disabled debug behavior a no-op that performs no filesystem access.
|
||||
A requested debug initialization or write error is returned to the caller;
|
||||
it is never silently downgraded.
|
||||
5. Keep this writer independent of normal state path derivation. Normal state
|
||||
inspection must not discover or serve sensitive debug artifacts.
|
||||
|
||||
### Tests
|
||||
|
||||
- Verify exact grouping, atomic replacement, disabled behavior, and returned
|
||||
per-run paths.
|
||||
- Verify directory `0700` and file `0600` permissions on supported platforms.
|
||||
- Cover traversal, absolute-segment, symlink-root, symlink-component,
|
||||
non-directory, and invalid report/date/run segment rejection.
|
||||
- Marshal representative debug fixtures and verify credentials and resolved
|
||||
secret values are absent while the explicitly allowed diagnostic fields are
|
||||
retained.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/state
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Sensitive debug persistence is secure, explicitly rooted, atomic, isolated
|
||||
from normal state, and fully tested. No production CLI path enables it yet.
|
||||
|
||||
## Stage 7: Add Promptkit Configuration, Composition, And Inspection
|
||||
|
||||
### Goal
|
||||
|
||||
Add the configuration and project-owned composition seams needed for cutover,
|
||||
and centralize pre-collection prompt/profile validation. Retain Scriptorium as
|
||||
the production generator until Stage 8 so this stage does not create a
|
||||
half-cut-over runtime.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add `config.PromptkitConfig` and nested local config with exactly:
|
||||
|
||||
- `profile`;
|
||||
- `profile_file`;
|
||||
- `profile_dir`;
|
||||
- `timeout`, default `2m`;
|
||||
- `local.endpoint`; and
|
||||
- `local.concurrency_limit`, default `1`.
|
||||
|
||||
Scriptorium config remains temporarily because production generation has
|
||||
not yet cut over. Promptkit validation rejects simultaneous profile
|
||||
sources, non-positive transport timeout, an invalid nonblank local
|
||||
endpoint, and negative concurrency. A blank local endpoint leaves `local`
|
||||
unregistered; concurrency zero means unlimited.
|
||||
2. Add an executor factory seam to `cli.Runner` using only project-owned types.
|
||||
Production construction delegates to `internal/adapters/promptkit`; CLI
|
||||
tests inject a fake factory. The factory creates one executor for an action,
|
||||
not one per report.
|
||||
3. Add a project-owned app inspection helper that:
|
||||
|
||||
- inspects the exact prompt ID and `report.Definition.PromptVersion`;
|
||||
- verifies exactly one required `data_package` input with
|
||||
`application/yaml`;
|
||||
- verifies the expected JSON Schema output contract and declared default
|
||||
profile;
|
||||
- selects `promptkit.profile` when nonblank, otherwise the prompt default;
|
||||
- inspects that explicit profile;
|
||||
- rejects a profile requiring a direct API key, because Weatherreporter has
|
||||
no direct-key configuration; and
|
||||
- requires a nonblank environment value for any reported `APIKeyEnv`.
|
||||
|
||||
4. Return only safe, project-owned inspection values and classified errors.
|
||||
The helper performs no collection, provider call, or durable write.
|
||||
5. Unit-test production factory argument mapping, but do not yet wire it into
|
||||
`generate` or `run`. Stage 8 performs that atomic production cutover.
|
||||
|
||||
### Tests
|
||||
|
||||
- Cover exact defaults, mutual profile-source exclusion, local endpoint
|
||||
validation, zero/unlimited and negative concurrency, and timeout validation.
|
||||
- Cover prompt/profile default and override precedence, prompt contract
|
||||
mismatch, unsupported direct-key profiles, missing environment credentials,
|
||||
and safe errors.
|
||||
- Verify the CLI factory maps embedded assets, external profile source, local
|
||||
backend, concurrency, and timeout exactly once without exposing Promptkit
|
||||
types.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/config ./internal/app ./internal/cli \
|
||||
./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Configuration, engine construction, and pre-collection inspection are
|
||||
decision-complete and offline-tested behind project-owned seams. Production
|
||||
generation still follows the old path, so there is no dual execution mode.
|
||||
|
||||
## Stage 8: Cut Over Single-Report Execution And Failure Persistence
|
||||
|
||||
### Goal
|
||||
|
||||
Move all four `generate` commands to Promptkit prepared execution while
|
||||
preserving report output behavior and inspectable partial failure. Debug CLI
|
||||
enablement and final summary presentation are deferred to Stage 9.
|
||||
|
||||
### Work
|
||||
|
||||
1. Construct exactly one executor after configuration is loaded for a
|
||||
`generate` action and pass it through `app.GenerateRequest`.
|
||||
2. Isolate the current shared generation function before changing it:
|
||||
|
||||
- move the Scriptorium implementation behind a temporary, unexported legacy
|
||||
batch helper;
|
||||
- make `GenerateDetailed` call the new Promptkit implementation described
|
||||
below;
|
||||
- leave `RunBatchDetailed` calling only the legacy helper until Stage 10;
|
||||
and
|
||||
- do not add a runtime switch, fallback, or dual invocation for the same
|
||||
report.
|
||||
|
||||
This temporary split is solely a staging seam. Stage 10 moves batches to
|
||||
the Promptkit implementation and deletes the helper.
|
||||
3. Reorder `GenerateDetailed`:
|
||||
|
||||
1. resolve the report and RunID;
|
||||
2. require the injected executor;
|
||||
3. run the Stage 7 exact prompt/profile/credential inspection; and only then
|
||||
4. collect weather.
|
||||
|
||||
Inspection failures occur before managed run artifacts exist.
|
||||
4. Implement the new project-owned report-generation core with the injected
|
||||
executor and the Stage 5 artifact contract:
|
||||
|
||||
- build, serialize once, and save the data package;
|
||||
- use those exact serialized bytes for execution;
|
||||
- persist a failed preparation receipt and v2 metadata when preparation
|
||||
fails;
|
||||
- use the preparation callback to save successful preparation provenance
|
||||
and metadata before provider work;
|
||||
- after execution success or completed validation rejection, save exact raw
|
||||
output, then the execution artifact, then metadata;
|
||||
- after operational execution failure, save a failure execution receipt and
|
||||
metadata without inventing raw output;
|
||||
- on Promptkit validation rejection, return a classified report failure
|
||||
after preserving raw output and bounded diagnostics;
|
||||
- on Promptkit validation success, run the existing typed
|
||||
`internal/generatedtext` validation, normalize/save generated text, build
|
||||
and save render context, and render managed Markdown as before.
|
||||
|
||||
5. Make partial failures return a non-nil `ReportResult` whenever a RunID and
|
||||
inspectable paths exist. Preserve preparation/execution/raw/normalized/
|
||||
context/report paths reached before preparation, capacity, generation,
|
||||
Promptkit validation, domain validation, template, copy, or notification
|
||||
failure.
|
||||
6. Remove Scriptorium result values from the single-report app requests,
|
||||
results, and test fakes used by the generate path. App tests depend only on
|
||||
`promptexec.Executor`.
|
||||
Retain the old adapter and configuration only for the still-uncut batch
|
||||
path.
|
||||
7. Do not expose `--llm-debug-dir` yet. Pass `CaptureDebug: false` during this
|
||||
stage; Stage 9 adds debug orchestration without changing provider execution
|
||||
semantics.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add representative offline app workflows for each of the four
|
||||
reports, using real state, generated-text validation, contexts, and
|
||||
templates with a fake executor.
|
||||
- Verify inspection and credential checks occur before collection.
|
||||
- Verify exact profile override/default precedence and exact prompt versions.
|
||||
- Verify persistence ordering by observing that execution is not called until
|
||||
the successful preparation artifact and metadata exist.
|
||||
- Cover preparation failure, callback/state failure, capacity rejection,
|
||||
credential disappearance at execution, cancellation, deadline, generation
|
||||
failure, operational validation failure, completed schema rejection,
|
||||
generated-text domain failure, template failure, output-copy failure, and
|
||||
notification failure.
|
||||
- Verify partial results retain every reached normal-artifact path without
|
||||
prompt or response content.
|
||||
- Verify single-report commands invoke only Promptkit and batch commands invoke
|
||||
only the temporary legacy helper during this intermediate stage.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/app ./internal/cli
|
||||
go test -race ./internal/app ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Every single-report command uses one Promptkit engine, performs inspection
|
||||
before collection, persists preparation before provider work, and produces the
|
||||
same managed Markdown/template and notification outcomes through project-owned
|
||||
contracts. Sensitive debug capture remains disabled.
|
||||
|
||||
## Stage 9: Add Single-Report Debug And CLI Summary Behavior
|
||||
|
||||
### Goal
|
||||
|
||||
Expose the opt-in debug workflow and finalize single-report CLI success and
|
||||
partial-failure summaries without expanding the normal artifact boundary.
|
||||
|
||||
### Work
|
||||
|
||||
1. Parse `--llm-debug-dir PATH` for all four generate commands. Do not add a
|
||||
YAML debug switch.
|
||||
2. Validate or create the debug root before prompt inspection or weather
|
||||
collection. A requested initialization failure is terminal and occurs
|
||||
before managed run artifacts exist.
|
||||
3. Pass debug intent through project-owned CLI and app request types. When
|
||||
enabled, request sensitive debug values from the executor:
|
||||
|
||||
- write preparation debug inside the preparation callback, after the normal
|
||||
preparation artifact is saved and before metadata is finalized and
|
||||
provider work begins;
|
||||
- if the debug write fails, return the callback error so `RunPrepared` is
|
||||
not called;
|
||||
- write execution debug immediately after receiving a completed execution
|
||||
result and before continuing with downstream validation/rendering; and
|
||||
- treat every requested debug write failure as terminal while preserving
|
||||
already reached normal artifacts.
|
||||
|
||||
4. Set `llmDebugPath` only after the per-run debug directory contains at least
|
||||
one successfully written debug artifact. Never copy sensitive debug content
|
||||
into a normal artifact, log, error, or summary.
|
||||
5. Finalize human and JSON CLI summaries around project-owned
|
||||
`preparationPath`, `executionPath`, and optional `llmDebugPath`. Include
|
||||
reached paths on partial failure; omit absent paths instead of inventing
|
||||
them.
|
||||
6. Preserve existing output behavior: quiet mode suppresses successful human
|
||||
output, JSON output remains machine-readable, and failures return a
|
||||
non-zero status with safe classified text.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add CLI parser and help tests for `--llm-debug-dir` on all four generate
|
||||
commands and its absence from YAML configuration.
|
||||
- Verify debug initialization precedes inspection and collection.
|
||||
- Verify preparation debug persistence precedes provider execution and that a
|
||||
write failure prevents the model-client call.
|
||||
- Cover execution debug success/failure, partial normal-artifact retention,
|
||||
`llmDebugPath` population rules, quiet mode, JSON summaries, and safe
|
||||
failure text.
|
||||
- Assert that summaries and routine logs contain no rendered prompt, schema
|
||||
body, input body, generated body, endpoint, full parameters, or credential.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/app ./internal/cli ./internal/state
|
||||
go test -race ./internal/app ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All single-report commands provide the complete Promptkit, v2 persistence,
|
||||
opt-in debug, and summary behavior required by the roadmap. Debug failure
|
||||
ordering is enforced without exposing sensitive content.
|
||||
|
||||
## Stage 10: Cut Over Batches And Remove Scriptorium
|
||||
|
||||
### Goal
|
||||
|
||||
Complete the production cutover, share one engine through each sequential
|
||||
batch, and delete the retired dependency boundary.
|
||||
|
||||
### Work
|
||||
|
||||
1. Construct one executor in the CLI for each `run morning` or `run evening`
|
||||
invocation and pass it through `BatchRequest` to every report.
|
||||
2. Parse and initialize `--llm-debug-dir` for run commands using the same
|
||||
policy as generate commands.
|
||||
3. Before collection, inspect the complete candidate set:
|
||||
|
||||
- morning: Today, Tomorrow, and Daily;
|
||||
- evening: Tomorrow and Daily.
|
||||
|
||||
Inspect exact prompt versions, validate declared input/output contracts,
|
||||
resolve unique effective profiles, and enforce credential availability.
|
||||
Daily inspection occurs before its collection-dependent future dates are
|
||||
known because every eligible Daily run uses the same exact prompt contract.
|
||||
4. Collect once, plan the batch as before, and execute every planned report
|
||||
sequentially through the shared executor, Stage 8 generation core, and
|
||||
store. Delete the temporary legacy batch helper after this call site moves.
|
||||
Preserve continuation after independent report failures and existing
|
||||
batch-notification gating.
|
||||
5. Include preparation, execution, and optional debug paths in each batch
|
||||
item, including failed items when those paths were reached.
|
||||
6. Replace Scriptorium configuration entirely with Promptkit configuration.
|
||||
Update defaults, validation, examples, config tests, and all construction
|
||||
sites. Explicitly reject a top-level `scriptorium:` key with an actionable
|
||||
migration error even though the general YAML loader is currently
|
||||
permissive; silently ignoring a former execution configuration is unsafe.
|
||||
Do not translate it or add a dual-run mode. The maintained examples contain
|
||||
only `promptkit:`.
|
||||
7. Delete:
|
||||
|
||||
- `internal/adapters/scriptorium`;
|
||||
- the Scriptorium renderer interface and all result/request types;
|
||||
- the legacy state path members and Scriptorium write methods retained in
|
||||
Stage 5;
|
||||
- subprocess fakes and subprocess-specific tests;
|
||||
- direct-Markdown branches and remaining Scriptorium defaults; and
|
||||
- dead helpers used only by the old adapter.
|
||||
|
||||
8. Run `go mod tidy` and verify the module graph contains Promptkit `v0.4.0`
|
||||
and no Scriptorium module or local replacement.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add batch tests proving one factory call/engine per CLI invocation, one
|
||||
executor shared across all reports, one collection, sequential ordering,
|
||||
later-report continuation, per-report path retention, and notification only
|
||||
after complete success.
|
||||
- Cover a capacity failure for one report followed by a later independent
|
||||
report and confirm no retry occurs.
|
||||
- Cover debug grouping for multiple reports and multiple Daily dates.
|
||||
- Update config tests for exact defaults, mutual exclusivity, local endpoint,
|
||||
zero/unlimited and negative concurrency, and maintained examples.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/config ./internal/app ./internal/cli
|
||||
go test -race ./internal/app ./internal/adapters/promptkit
|
||||
go test ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All production generation paths use Promptkit. A batch owns one engine and
|
||||
continues sequentially under existing failure policy. No Scriptorium code,
|
||||
configuration field, subprocess path, or dependency remains.
|
||||
|
||||
## Stage 11: Update Canonical Documentation And Complete Verification
|
||||
|
||||
### Goal
|
||||
|
||||
Make current-state documentation match the completed implementation, remove
|
||||
migration-only source material, and perform repository-wide verification.
|
||||
|
||||
### Work
|
||||
|
||||
1. Update canonical current-state owners in the same migration change:
|
||||
|
||||
- `docs/development.md` for repository orientation, package map, task
|
||||
routing, and validation language;
|
||||
- `docs/policy/architecture.md` for the Promptkit adapter boundary,
|
||||
generated-text-only flow, prepared execution, and four-report product;
|
||||
- `docs/policy/testing.md` for Promptkit/provider fakes instead of
|
||||
subprocess fakes and the offline external-boundary rule;
|
||||
- `docs/cli.md` for four commands, `--llm-debug-dir`, and renamed summary
|
||||
fields;
|
||||
- `docs/config.md` and `examples/` for the exact Promptkit contract;
|
||||
- `docs/operations.md` for v2 paths, sensitive debug retention and
|
||||
permissions, preparation/execution lifecycle, and v1 inspection;
|
||||
- `docs/troubleshooting.md` for inspection, profile/credential,
|
||||
preparation, capacity, execution, validation, and debug failures;
|
||||
- relevant `docs/internal/` files for app, CLI, report, prompt input,
|
||||
generated text, templates, state, collection, briefing, facts, changes,
|
||||
and package boundaries; and
|
||||
- `docs/templates.md` for Promptkit-generated prose and the relocated schema
|
||||
owner.
|
||||
|
||||
2. Replace `docs/integrations/scriptorium.md` and
|
||||
`docs/internal/scriptorium-adapter.md` with canonical Promptkit integration
|
||||
and adapter documents. The integration document owns the logical
|
||||
prompt/profile/schema and durable compatibility contract; the internal
|
||||
document owns construction, mapping, lifecycle, and tests. Link rather than
|
||||
duplicate the full CLI/config/operations references.
|
||||
3. Search all non-roadmap current-state documentation, examples, Go code,
|
||||
tests, help output, and module metadata for stale `scriptorium`,
|
||||
`local-heavy`, retired report commands, old artifact field names, old
|
||||
filenames, and old metadata versions. Retain old names only in explicit v1
|
||||
compatibility code/tests and historical roadmap discussion.
|
||||
4. Remove `docs/roadmap/scriptorium` after confirming every promoted runtime
|
||||
asset is represented under `internal/promptassets` and covered by asset
|
||||
tests. The unused legacy Daily Markdown prompt is deleted with this source
|
||||
directory.
|
||||
5. Keep the feature roadmap and this plan as temporary migration documents
|
||||
while implementation is under review. Once the implementation is accepted,
|
||||
mark both complete and move every still-useful contract to its canonical
|
||||
current-state owner before later archival or removal.
|
||||
6. Review all added links, fenced examples, flags, field names, defaults,
|
||||
schema IDs, file paths, and version strings against executable sources.
|
||||
Confirm examples contain no credentials or private infrastructure values.
|
||||
|
||||
### Final Verification
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w <all changed Go files>
|
||||
go mod tidy
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Then verify explicitly:
|
||||
|
||||
- `go list -m gitea.maximumdirect.net/eric/promptkit` reports `v0.4.0`;
|
||||
- no committed `go.work`, `replace`, secret fixture, or live-provider test
|
||||
exists;
|
||||
- the maintained examples load through config tests;
|
||||
- all four embedded prompts inspect at exact version `1.0.0`;
|
||||
- no runtime prompt requests repair attempts;
|
||||
- no ordinary artifact or CLI summary includes rendered prompts, schema
|
||||
bodies, input bodies, generated bodies, provider endpoints, or credentials;
|
||||
- v1 metadata fixtures remain inspectable and new runs write only v2;
|
||||
- help exposes only Daily, Today, Tomorrow, Hourly, morning, and evening; and
|
||||
- managed Markdown remains the only Distributor upload source.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Every completion criterion in the feature roadmap is demonstrably satisfied.
|
||||
Current-state documentation describes the implementation rather than the
|
||||
migration, and the repository passes all final verification commands.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and this implementation plan contain all product,
|
||||
architecture, configuration, compatibility, security, sequencing, and
|
||||
verification decisions required to begin implementation.
|
||||
Reference in New Issue
Block a user