weatherreporter/docs/roadmap/promptkit.md

Promptkit Migration Roadmap

Purpose

This roadmap defines the scope and desired end state for replacing the external Scriptorium CLI integration with the Promptkit Go library. The migration is not yet implemented. Current Scriptorium behavior remains documented outside docs/roadmap/ until the replacement is complete.

A separate staged implementation plan will describe how to move from the current code to this target state. That plan should reference this roadmap rather than redefine its architectural decisions or scope.

Desired End State

Weatherreporter uses a pinned released version of gitea.maximumdirect.net/eric/promptkit as its in-process prompt preparation and LLM execution engine. The scriptorium executable, subprocess adapter, configuration, runtime dependency, and integration documentation have been removed.

The migration does not change weatherreporter's fundamental product behavior. Weather selection, forecast derivation, report periods, module construction, Recent Changes, generated-text interpretation, Markdown templates, durable state, inspection, output copies, and distributor notification remain owned by weatherreporter.

All report prompts and private response schemas are versioned application assets. Operators may configure Promptkit execution profiles without replacing the report-owned prompt and schema corpus. One Promptkit engine is constructed per CLI invocation and shared by every report in that invocation, including all reports in a morning or evening batch.

Promptkit is isolated behind a weatherreporter-owned prompt execution contract. Promptkit request, result, validation, error, profile, backend, and provider types do not leak into application orchestration, report definitions, domain packages, CLI summaries, state contracts, or distributor behavior.

Goals

• Remove the runtime dependency on the scriptorium executable.
• Replace shell-free subprocess orchestration with typed in-process Promptkit preparation and execution.
• Preserve the seven report definitions and their existing prompt IDs.
• Preserve both direct-Markdown and generated-text-template report workflows.
• Preserve deterministic module snapshots and structured Recent Changes.
• Preserve context cancellation, actionable errors, secret redaction, and inspectable failures.
• Improve durable prompt provenance with prompt, input, profile, model, validation, usage, and timing metadata.
• Keep content-rich prompt and response diagnostics separate from routine metadata and CLI output.
• Keep tests offline and deterministic through injected Promptkit model clients and fixtures.

Non-Goals

The migration will not:

• move meteorological selection, derivation, thresholds, or comparison logic into prompts or Promptkit;
• send raw unbounded Weather API responses to the model;
• replace weatherreporter's generated-text domain validation or Markdown template rendering;
• add a general workflow engine, provider plugin system, or arbitrary backend registry to weatherreporter;
• add automatic provider, validation, or capacity retries;
• add concurrent report generation to the existing sequential batch workflow;
• expose Promptkit types as a weatherreporter component contract;
• keep a production-selectable Scriptorium/Promptkit dual-run mode; or
• use an unpublished Promptkit commit, committed Go workspace, or committed local module replacement.

Locked Decisions

Dependency And Versioning

• The initial integration will pin Promptkit v0.3.0.
• Coordinated local development may temporarily use the sibling Promptkit checkout, but committed module metadata must reference the tagged release.
• A future Promptkit upgrade requires an explicit review of the public engine, prompt/profile/schema formats, error identities, validation behavior, and outbound provider contract used by weatherreporter.

Application Boundary

• Promptkit remains an adapter boundary even though it runs in process.
• A weatherreporter-owned contract will represent preparation, execution, output formats, validation, usage, provenance, and neutral error categories.
• The Promptkit adapter will map public Promptkit values into that contract at the boundary.
• App orchestration and test fakes will depend on the weatherreporter contract, not on Promptkit.
• Existing Scriptorium-specific generation mode names will be replaced with provider-neutral names.

Prompt And Schema Ownership

• Weatherreporter will embed all report prompt definitions, prompt content, and private response schemas.
• Prompt assets will remain separate files rather than inline Go strings.
• The current Scriptorium prompt corpus will be retrieved before the implementation stage that establishes the embedded Promptkit assets.
• The retrieved corpus will be reviewed and converted to the pinned Promptkit format without changing report intent or prompt IDs.
• The four existing generated-text prompt fragments and schemas under internal/reporttemplate will be reconciled with that corpus rather than duplicated.
• Direct-Markdown prompt assets for the three-day, weekend, and storm reports will become weatherreporter-owned assets.
• Weatherreporter needs one centralized embedded prompt/schema source; it does not need Notarius's multi-module asset-flattening registry.

Profiles, Backends, And Credentials

• Execution profiles remain operator-configurable rather than embedded report policy.
• Configuration will support at most one external profile source: a profile directory or a single profile file.
• Prompt definitions may provide their normal default profile, while weatherreporter may support an explicit configured profile selection.
• Credential values remain in environment variables or file-backed environment secrets. Configuration contains only credential source names.
• Provider credentials must not appear in logs, errors, CLI output, durable metadata, preparation artifacts, execution artifacts, or debug summaries.
• Weatherreporter will not expose Promptkit's general backend registry as arbitrary application configuration.

Engine Lifetime

• One Promptkit engine will be constructed per CLI invocation at the application composition boundary.
• Single-report generation will use that engine for preparation and execution.
• Morning and evening batches will share the same engine across every planned report.
• Per-report orchestration will not construct its own default Promptkit engine.
• Promptkit backend capacity state and HTTP transport will therefore be shared consistently for the invocation.

Prompt Input

• Promptkit will continue to receive the curated data_package produced by internal/promptinput.
• Weatherreporter will serialize the data package once, atomically persist those exact bytes, and supply the same bytes as a Promptkit inline artifact.
• The managed data-package path may be supplied as non-secret artifact provenance.
• Weatherreporter will not delegate unrestricted path loading to Promptkit's default file artifact reader.
• The same immutable Promptkit request will be used for preparation and execution so the preflight and run inputs cannot diverge.

Preparation And Execution

• Promptkit Prepare replaces the current Scriptorium render preflight.
• Promptkit Run performs both Markdown and structured generated-text execution.
• Promptkit basic validation will be used where appropriate for direct Markdown output.
• Promptkit JSON Schema validation provides the provider-facing and first structured-output check for generated-text reports.
• Weatherreporter's internal/generatedtext validation remains the final report-specific domain boundary.
• Weatherreporter's internal/reporttemplate remains responsible for generated-text Markdown rendering.
• Weatherreporter will atomically persist Promptkit output rather than asking the dependency to write managed report files.
• The migration will not rely on Promptkit output repair. Promptkit v0.3.0's public engine validates in a single pass even when a prompt declares repair attempts.

Durable Artifacts And Observability

Routine durable artifacts should retain useful non-secret provenance without persisting full rendered prompts by default.

The preparation record should contain:

• prompt ID and version;
• prompt definition hash;
• rendered prompt hash;
• input hashes;
• selected profile and backend identity;
• effective model identity;
• output contract summary; and
• preparation timing.

The execution record and run metadata should contain, when available:

• Promptkit run ID;
• prompt ID, version, and hashes;
• input hashes;
• selected profile, backend, and model identity;
• generated-content hash;
• token usage;
• start, end, and duration;
• validation status and bounded diagnostics; and
• the path of any separately persisted raw generated output.

Provider endpoints, full effective model parameter maps, rendered messages, schema bodies, data-package contents, and generated content do not belong in routine metadata or CLI summaries.

Rendered messages and other content-rich preparation or response diagnostics will be available only through an explicitly enabled debug mechanism. Debug artifacts must be documented as potentially sensitive, must not contain credentials, and must have a clear operator-owned retention policy.

Failure Contract

Promptkit returns a completed RunResult for output-validation failure but no partial result for operational preparation or execution errors. Weatherreporter will preserve that distinction.

• A preparation failure produces a redacted weatherreporter-owned failure receipt with report, RunID, prompt, stage, timing, and classified error context. It does not fabricate a Promptkit preparation result.
• An operational execution failure retains the successful preparation record and adds a redacted execution failure receipt. No partial Promptkit result or model output is invented.
• A Promptkit validation failure retains the returned result, raw generated output, validation details, and safe provenance before the report fails.
• A later weatherreporter generated-text decode, domain-validation, or template failure retains every raw and validated artifact reached before that stage.
• Context cancellation takes precedence when the caller context is canceled.
• Promptkit capacity rejection maps to a weatherreporter-owned error category. It is an operational report failure, not invalid model output.
• Single-report commands return the classified failure with available inspectable paths.
• Batch runs continue independent later reports under the existing batch failure policy.
• The migration adds no automatic retries. Any future retry policy belongs to app orchestration, not the Promptkit adapter.

Compatibility Requirements

• Report IDs, prompt IDs, report selection, valid periods, artifact grouping, output names, and distributor bundle behavior remain stable.
• Module snapshot and Recent Changes behavior remains deterministic.
• Promptkit receives only the existing curated prompt-input boundary.
• Generated reports continue to use the managed Markdown path as the distributor upload source.
• RunID lookup and inspection remain available for successful and failed runs.
• Existing managed state paths remain stable where their meaning is unchanged. Scriptorium-specific artifact names or schemas may change when retaining them would misrepresent the new contract.
• Any artifact or metadata schema change is explicit, documented, and covered by state and inspection tests.
• Prompt or generated content is not added to routine logs or CLI summaries.
• Tests do not require live Promptkit providers or credentials.

Verification And Completion Criteria

The migration is complete when:

• all seven reports prepare and execute through Promptkit using embedded report-owned assets;
• direct-Markdown and generated-text-template paths have deterministic offline adapter and app-level coverage;
• preparation, provider failure, capacity rejection, cancellation, timeout, Promptkit validation failure, generated-text validation failure, template failure, and successful generation preserve their specified artifacts;
• morning and evening batches construct one shared engine and preserve current collection, planning, ordering, continuation, output, and notification behavior;
• configuration examples load and contain no Scriptorium fields;
• CLI summaries and inspection commands expose the new artifact contract without Promptkit dependency types;
• Scriptorium code, configuration, tests, and runtime documentation have been removed;
• non-roadmap documentation describes only the implemented Promptkit integration;
• go test ./..., CLI help validation, and git diff --check pass; and
• no committed go.work, local replace, live-provider test, or secret-bearing fixture remains.

Fixture-based comparison with the current Scriptorium behavior is sufficient for migration verification. A production-selectable dual-run period is not required because model calls are nondeterministic, costly, and difficult to compare meaningfully.

External Prerequisite

Before implementing the embedded asset stage, the current Scriptorium prompt corpus must be made available in this repository. It should include the seven prompt definitions, referenced content files, private response schemas, relevant default-profile declarations, and any shared prompt fragments needed to reproduce current report behavior.

Open Questions

• What exact promptkit.* configuration fields should replace the current Scriptorium fields, including the name and precedence of an optional explicit profile override?
• Should weatherreporter expose Promptkit's conventional local backend registration as a narrow configuration feature, or rely initially on built-in and endpoint-only profiles?
• Should report definitions store an explicit Promptkit prompt version, or should each embedded prompt ID be required to have exactly one version?
• What CLI or configuration control enables sensitive prompt/response debug artifacts, and where should those artifacts live?
• What final names and schema versions should replace the Scriptorium-specific preflight and run-result artifacts while balancing semantic clarity with existing state-path compatibility?