diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md new file mode 100644 index 0000000..62ca427 --- /dev/null +++ b/docs/roadmap/promptkit.md @@ -0,0 +1,304 @@ +# Promptkit Migration Roadmap + +## Purpose + +This roadmap defines the scope and desired end state for replacing the +external Scriptorium CLI integration with the Promptkit Go library. The +migration is not yet implemented. Current Scriptorium behavior remains +documented outside `docs/roadmap/` until the replacement is complete. + +A separate staged implementation plan will describe how to move from the +current code to this target state. That plan should reference this roadmap +rather than redefine its architectural decisions or scope. + +## Desired End State + +Weatherreporter uses a pinned released version of +`gitea.maximumdirect.net/eric/promptkit` as its in-process prompt preparation +and LLM execution engine. The `scriptorium` executable, subprocess adapter, +configuration, runtime dependency, and integration documentation have been +removed. + +The migration does not change weatherreporter's fundamental product behavior. +Weather selection, forecast derivation, report periods, module construction, +Recent Changes, generated-text interpretation, Markdown templates, durable +state, inspection, output copies, and distributor notification remain owned by +weatherreporter. + +All report prompts and private response schemas are versioned application +assets. Operators may configure Promptkit execution profiles without replacing +the report-owned prompt and schema corpus. One Promptkit engine is constructed +per CLI invocation and shared by every report in that invocation, including +all reports in a morning or evening batch. + +Promptkit is isolated behind a weatherreporter-owned prompt execution contract. +Promptkit request, result, validation, error, profile, backend, and provider +types do not leak into application orchestration, report definitions, domain +packages, CLI summaries, state contracts, or distributor behavior. + +## Goals + +- Remove the runtime dependency on the `scriptorium` executable. +- Replace shell-free subprocess orchestration with typed in-process Promptkit + preparation and execution. +- Preserve the seven report definitions and their existing prompt IDs. +- Preserve both direct-Markdown and generated-text-template report workflows. +- Preserve deterministic module snapshots and structured Recent Changes. +- Preserve context cancellation, actionable errors, secret redaction, and + inspectable failures. +- Improve durable prompt provenance with prompt, input, profile, model, + validation, usage, and timing metadata. +- Keep content-rich prompt and response diagnostics separate from routine + metadata and CLI output. +- Keep tests offline and deterministic through injected Promptkit model + clients and fixtures. + +## Non-Goals + +The migration will not: + +- move meteorological selection, derivation, thresholds, or comparison logic + into prompts or Promptkit; +- send raw unbounded Weather API responses to the model; +- replace weatherreporter's generated-text domain validation or Markdown + template rendering; +- add a general workflow engine, provider plugin system, or arbitrary backend + registry to weatherreporter; +- add automatic provider, validation, or capacity retries; +- add concurrent report generation to the existing sequential batch workflow; +- expose Promptkit types as a weatherreporter component contract; +- keep a production-selectable Scriptorium/Promptkit dual-run mode; or +- use an unpublished Promptkit commit, committed Go workspace, or committed + local module replacement. + +## Locked Decisions + +### Dependency And Versioning + +- The initial integration will pin Promptkit `v0.3.0`. +- Coordinated local development may temporarily use the sibling Promptkit + checkout, but committed module metadata must reference the tagged release. +- A future Promptkit upgrade requires an explicit review of the public engine, + prompt/profile/schema formats, error identities, validation behavior, and + outbound provider contract used by weatherreporter. + +### Application Boundary + +- Promptkit remains an adapter boundary even though it runs in process. +- A weatherreporter-owned contract will represent preparation, execution, + output formats, validation, usage, provenance, and neutral error categories. +- The Promptkit adapter will map public Promptkit values into that contract at + the boundary. +- App orchestration and test fakes will depend on the weatherreporter contract, + not on Promptkit. +- Existing Scriptorium-specific generation mode names will be replaced with + provider-neutral names. + +### Prompt And Schema Ownership + +- Weatherreporter will embed all report prompt definitions, prompt content, + and private response schemas. +- Prompt assets will remain separate files rather than inline Go strings. +- The current Scriptorium prompt corpus will be retrieved before the + implementation stage that establishes the embedded Promptkit assets. +- The retrieved corpus will be reviewed and converted to the pinned Promptkit + format without changing report intent or prompt IDs. +- The four existing generated-text prompt fragments and schemas under + `internal/reporttemplate` will be reconciled with that corpus rather than + duplicated. +- Direct-Markdown prompt assets for the three-day, weekend, and storm reports + will become weatherreporter-owned assets. +- Weatherreporter needs one centralized embedded prompt/schema source; it does + not need Notarius's multi-module asset-flattening registry. + +### Profiles, Backends, And Credentials + +- Execution profiles remain operator-configurable rather than embedded report + policy. +- Configuration will support at most one external profile source: a profile + directory or a single profile file. +- Prompt definitions may provide their normal default profile, while + weatherreporter may support an explicit configured profile selection. +- Credential values remain in environment variables or file-backed + environment secrets. Configuration contains only credential source names. +- Provider credentials must not appear in logs, errors, CLI output, durable + metadata, preparation artifacts, execution artifacts, or debug summaries. +- Weatherreporter will not expose Promptkit's general backend registry as + arbitrary application configuration. + +### Engine Lifetime + +- One Promptkit engine will be constructed per CLI invocation at the + application composition boundary. +- Single-report generation will use that engine for preparation and execution. +- Morning and evening batches will share the same engine across every planned + report. +- Per-report orchestration will not construct its own default Promptkit engine. +- Promptkit backend capacity state and HTTP transport will therefore be shared + consistently for the invocation. + +### Prompt Input + +- Promptkit will continue to receive the curated `data_package` produced by + `internal/promptinput`. +- Weatherreporter will serialize the data package once, atomically persist + those exact bytes, and supply the same bytes as a Promptkit inline artifact. +- The managed data-package path may be supplied as non-secret artifact + provenance. +- Weatherreporter will not delegate unrestricted path loading to Promptkit's + default file artifact reader. +- The same immutable Promptkit request will be used for preparation and + execution so the preflight and run inputs cannot diverge. + +### Preparation And Execution + +- Promptkit `Prepare` replaces the current Scriptorium render preflight. +- Promptkit `Run` performs both Markdown and structured generated-text + execution. +- Promptkit basic validation will be used where appropriate for direct + Markdown output. +- Promptkit JSON Schema validation provides the provider-facing and first + structured-output check for generated-text reports. +- Weatherreporter's `internal/generatedtext` validation remains the final + report-specific domain boundary. +- Weatherreporter's `internal/reporttemplate` remains responsible for + generated-text Markdown rendering. +- Weatherreporter will atomically persist Promptkit output rather than asking + the dependency to write managed report files. +- The migration will not rely on Promptkit output repair. Promptkit v0.3.0's + public engine validates in a single pass even when a prompt declares repair + attempts. + +## Durable Artifacts And Observability + +Routine durable artifacts should retain useful non-secret provenance without +persisting full rendered prompts by default. + +The preparation record should contain: + +- prompt ID and version; +- prompt definition hash; +- rendered prompt hash; +- input hashes; +- selected profile and backend identity; +- effective model identity; +- output contract summary; and +- preparation timing. + +The execution record and run metadata should contain, when available: + +- Promptkit run ID; +- prompt ID, version, and hashes; +- input hashes; +- selected profile, backend, and model identity; +- generated-content hash; +- token usage; +- start, end, and duration; +- validation status and bounded diagnostics; and +- the path of any separately persisted raw generated output. + +Provider endpoints, full effective model parameter maps, rendered messages, +schema bodies, data-package contents, and generated content do not belong in +routine metadata or CLI summaries. + +Rendered messages and other content-rich preparation or response diagnostics +will be available only through an explicitly enabled debug mechanism. Debug +artifacts must be documented as potentially sensitive, must not contain +credentials, and must have a clear operator-owned retention policy. + +## Failure Contract + +Promptkit returns a completed `RunResult` for output-validation failure but no +partial result for operational preparation or execution errors. Weatherreporter +will preserve that distinction. + +- A preparation failure produces a redacted weatherreporter-owned failure + receipt with report, RunID, prompt, stage, timing, and classified error + context. It does not fabricate a Promptkit preparation result. +- An operational execution failure retains the successful preparation record + and adds a redacted execution failure receipt. No partial Promptkit result or + model output is invented. +- A Promptkit validation failure retains the returned result, raw generated + output, validation details, and safe provenance before the report fails. +- A later weatherreporter generated-text decode, domain-validation, or template + failure retains every raw and validated artifact reached before that stage. +- Context cancellation takes precedence when the caller context is canceled. +- Promptkit capacity rejection maps to a weatherreporter-owned error category. + It is an operational report failure, not invalid model output. +- Single-report commands return the classified failure with available + inspectable paths. +- Batch runs continue independent later reports under the existing batch + failure policy. +- The migration adds no automatic retries. Any future retry policy belongs to + app orchestration, not the Promptkit adapter. + +## Compatibility Requirements + +- Report IDs, prompt IDs, report selection, valid periods, artifact grouping, + output names, and distributor bundle behavior remain stable. +- Module snapshot and Recent Changes behavior remains deterministic. +- Promptkit receives only the existing curated prompt-input boundary. +- Generated reports continue to use the managed Markdown path as the + distributor upload source. +- RunID lookup and inspection remain available for successful and failed runs. +- Existing managed state paths remain stable where their meaning is unchanged. + Scriptorium-specific artifact names or schemas may change when retaining + them would misrepresent the new contract. +- Any artifact or metadata schema change is explicit, documented, and covered + by state and inspection tests. +- Prompt or generated content is not added to routine logs or CLI summaries. +- Tests do not require live Promptkit providers or credentials. + +## Verification And Completion Criteria + +The migration is complete when: + +- all seven reports prepare and execute through Promptkit using embedded + report-owned assets; +- direct-Markdown and generated-text-template paths have deterministic offline + adapter and app-level coverage; +- preparation, provider failure, capacity rejection, cancellation, timeout, + Promptkit validation failure, generated-text validation failure, template + failure, and successful generation preserve their specified artifacts; +- morning and evening batches construct one shared engine and preserve current + collection, planning, ordering, continuation, output, and notification + behavior; +- configuration examples load and contain no Scriptorium fields; +- CLI summaries and inspection commands expose the new artifact contract + without Promptkit dependency types; +- Scriptorium code, configuration, tests, and runtime documentation have been + removed; +- non-roadmap documentation describes only the implemented Promptkit + integration; +- `go test ./...`, CLI help validation, and `git diff --check` pass; and +- no committed `go.work`, local `replace`, live-provider test, or secret-bearing + fixture remains. + +Fixture-based comparison with the current Scriptorium behavior is sufficient +for migration verification. A production-selectable dual-run period is not +required because model calls are nondeterministic, costly, and difficult to +compare meaningfully. + +## External Prerequisite + +Before implementing the embedded asset stage, the current Scriptorium prompt +corpus must be made available in this repository. It should include the seven +prompt definitions, referenced content files, private response schemas, +relevant default-profile declarations, and any shared prompt fragments needed +to reproduce current report behavior. + +## Open Questions + +- What exact `promptkit.*` configuration fields should replace the current + Scriptorium fields, including the name and precedence of an optional explicit + profile override? +- Should weatherreporter expose Promptkit's conventional `local` backend + registration as a narrow configuration feature, or rely initially on + built-in and endpoint-only profiles? +- Should report definitions store an explicit Promptkit prompt version, or + should each embedded prompt ID be required to have exactly one version? +- What CLI or configuration control enables sensitive prompt/response debug + artifacts, and where should those artifacts live? +- What final names and schema versions should replace the + Scriptorium-specific preflight and run-result artifacts while balancing + semantic clarity with existing state-path compatibility?