weatherreporter/docs/roadmap/promptkit.md

# Promptkit Migration Roadmap

## Purpose

This roadmap defines the scope and desired end state for replacing the
external Scriptorium CLI integration with the Promptkit Go library. The
migration is not yet implemented. Current Scriptorium behavior remains
documented outside `docs/roadmap/` until the replacement is complete.

A separate staged implementation plan will describe how to move from the
current code to this target state. That plan should reference this roadmap
rather than redefine its architectural decisions or scope.

## Desired End State

Weatherreporter uses a pinned released version of
`gitea.maximumdirect.net/eric/promptkit` as its in-process prompt preparation
and LLM execution engine. The `scriptorium` executable, subprocess adapter,
configuration, runtime dependency, and integration documentation have been
removed.

The migration does not change weatherreporter's fundamental product behavior.
Weather selection, forecast derivation, report periods, module construction,
Recent Changes, generated-text interpretation, Markdown templates, durable
state, inspection, output copies, and distributor notification remain owned by
weatherreporter.

All report prompts and private response schemas are versioned application
assets. Operators may configure Promptkit execution profiles without replacing
the report-owned prompt and schema corpus. One Promptkit engine is constructed
per CLI invocation and shared by every report in that invocation, including
all reports in a morning or evening batch.

Promptkit is isolated behind a weatherreporter-owned prompt execution contract.
Promptkit request, result, validation, error, profile, backend, and provider
types do not leak into application orchestration, report definitions, domain
packages, CLI summaries, state contracts, or distributor behavior.

## Goals

- Remove the runtime dependency on the `scriptorium` executable.
- Replace shell-free subprocess orchestration with typed in-process Promptkit
  preparation and execution.
- Preserve the seven report definitions and their existing prompt IDs.
- Preserve both direct-Markdown and generated-text-template report workflows.
- Preserve deterministic module snapshots and structured Recent Changes.
- Preserve context cancellation, actionable errors, secret redaction, and
  inspectable failures.
- Improve durable prompt provenance with prompt, input, profile, model,
  validation, usage, and timing metadata.
- Keep content-rich prompt and response diagnostics separate from routine
  metadata and CLI output.
- Keep tests offline and deterministic through injected Promptkit model
  clients and fixtures.

## Non-Goals

The migration will not:

- move meteorological selection, derivation, thresholds, or comparison logic
  into prompts or Promptkit;
- send raw unbounded Weather API responses to the model;
- replace weatherreporter's generated-text domain validation or Markdown
  template rendering;
- add a general workflow engine, provider plugin system, or arbitrary backend
  registry to weatherreporter;
- add automatic provider, validation, or capacity retries;
- add concurrent report generation to the existing sequential batch workflow;
- expose Promptkit types as a weatherreporter component contract;
- keep a production-selectable Scriptorium/Promptkit dual-run mode; or
- use an unpublished Promptkit commit, committed Go workspace, or committed
  local module replacement.

## Locked Decisions

### Dependency And Versioning

- The initial integration will pin Promptkit `v0.3.0`.
- Coordinated local development may temporarily use the sibling Promptkit
  checkout, but committed module metadata must reference the tagged release.
- A future Promptkit upgrade requires an explicit review of the public engine,
  prompt/profile/schema formats, error identities, validation behavior, and
  outbound provider contract used by weatherreporter.

### Application Boundary

- Promptkit remains an adapter boundary even though it runs in process.
- A weatherreporter-owned contract will represent preparation, execution,
  output formats, validation, usage, provenance, and neutral error categories.
- The Promptkit adapter will map public Promptkit values into that contract at
  the boundary.
- App orchestration and test fakes will depend on the weatherreporter contract,
  not on Promptkit.
- Existing Scriptorium-specific generation mode names will be replaced with
  provider-neutral names.

### Prompt And Schema Ownership

- Weatherreporter will embed all report prompt definitions, prompt content,
  and private response schemas.
- Prompt assets will remain separate files rather than inline Go strings.
- The current Scriptorium prompt corpus will be retrieved before the
  implementation stage that establishes the embedded Promptkit assets.
- The retrieved corpus will be reviewed and converted to the pinned Promptkit
  format without changing report intent or prompt IDs.
- The four existing generated-text prompt fragments and schemas under
  `internal/reporttemplate` will be reconciled with that corpus rather than
  duplicated.
- Direct-Markdown prompt assets for the three-day, weekend, and storm reports
  will become weatherreporter-owned assets.
- Weatherreporter needs one centralized embedded prompt/schema source; it does
  not need Notarius's multi-module asset-flattening registry.

### Profiles, Backends, And Credentials

- Execution profiles remain operator-configurable rather than embedded report
  policy.
- Configuration will support at most one external profile source: a profile
  directory or a single profile file.
- Prompt definitions may provide their normal default profile, while
  weatherreporter may support an explicit configured profile selection.
- Credential values remain in environment variables or file-backed
  environment secrets. Configuration contains only credential source names.
- Provider credentials must not appear in logs, errors, CLI output, durable
  metadata, preparation artifacts, execution artifacts, or debug summaries.
- Weatherreporter will not expose Promptkit's general backend registry as
  arbitrary application configuration.

### Engine Lifetime

- One Promptkit engine will be constructed per CLI invocation at the
  application composition boundary.
- Single-report generation will use that engine for preparation and execution.
- Morning and evening batches will share the same engine across every planned
  report.
- Per-report orchestration will not construct its own default Promptkit engine.
- Promptkit backend capacity state and HTTP transport will therefore be shared
  consistently for the invocation.

### Prompt Input

- Promptkit will continue to receive the curated `data_package` produced by
  `internal/promptinput`.
- Weatherreporter will serialize the data package once, atomically persist
  those exact bytes, and supply the same bytes as a Promptkit inline artifact.
- The managed data-package path may be supplied as non-secret artifact
  provenance.
- Weatherreporter will not delegate unrestricted path loading to Promptkit's
  default file artifact reader.
- The same immutable Promptkit request will be used for preparation and
  execution so the preflight and run inputs cannot diverge.

### Preparation And Execution

- Promptkit `Prepare` replaces the current Scriptorium render preflight.
- Promptkit `Run` performs both Markdown and structured generated-text
  execution.
- Promptkit basic validation will be used where appropriate for direct
  Markdown output.
- Promptkit JSON Schema validation provides the provider-facing and first
  structured-output check for generated-text reports.
- Weatherreporter's `internal/generatedtext` validation remains the final
  report-specific domain boundary.
- Weatherreporter's `internal/reporttemplate` remains responsible for
  generated-text Markdown rendering.
- Weatherreporter will atomically persist Promptkit output rather than asking
  the dependency to write managed report files.
- The migration will not rely on Promptkit output repair. Promptkit v0.3.0's
  public engine validates in a single pass even when a prompt declares repair
  attempts.

## Durable Artifacts And Observability

Routine durable artifacts should retain useful non-secret provenance without
persisting full rendered prompts by default.

The preparation record should contain:

- prompt ID and version;
- prompt definition hash;
- rendered prompt hash;
- input hashes;
- selected profile and backend identity;
- effective model identity;
- output contract summary; and
- preparation timing.

The execution record and run metadata should contain, when available:

- Promptkit run ID;
- prompt ID, version, and hashes;
- input hashes;
- selected profile, backend, and model identity;
- generated-content hash;
- token usage;
- start, end, and duration;
- validation status and bounded diagnostics; and
- the path of any separately persisted raw generated output.

Provider endpoints, full effective model parameter maps, rendered messages,
schema bodies, data-package contents, and generated content do not belong in
routine metadata or CLI summaries.

Rendered messages and other content-rich preparation or response diagnostics
will be available only through an explicitly enabled debug mechanism. Debug
artifacts must be documented as potentially sensitive, must not contain
credentials, and must have a clear operator-owned retention policy.

## Failure Contract

Promptkit returns a completed `RunResult` for output-validation failure but no
partial result for operational preparation or execution errors. Weatherreporter
will preserve that distinction.

- A preparation failure produces a redacted weatherreporter-owned failure
  receipt with report, RunID, prompt, stage, timing, and classified error
  context. It does not fabricate a Promptkit preparation result.
- An operational execution failure retains the successful preparation record
  and adds a redacted execution failure receipt. No partial Promptkit result or
  model output is invented.
- A Promptkit validation failure retains the returned result, raw generated
  output, validation details, and safe provenance before the report fails.
- A later weatherreporter generated-text decode, domain-validation, or template
  failure retains every raw and validated artifact reached before that stage.
- Context cancellation takes precedence when the caller context is canceled.
- Promptkit capacity rejection maps to a weatherreporter-owned error category.
  It is an operational report failure, not invalid model output.
- Single-report commands return the classified failure with available
  inspectable paths.
- Batch runs continue independent later reports under the existing batch
  failure policy.
- The migration adds no automatic retries. Any future retry policy belongs to
  app orchestration, not the Promptkit adapter.

## Compatibility Requirements

- Report IDs, prompt IDs, report selection, valid periods, artifact grouping,
  output names, and distributor bundle behavior remain stable.
- Module snapshot and Recent Changes behavior remains deterministic.
- Promptkit receives only the existing curated prompt-input boundary.
- Generated reports continue to use the managed Markdown path as the
  distributor upload source.
- RunID lookup and inspection remain available for successful and failed runs.
- Existing managed state paths remain stable where their meaning is unchanged.
  Scriptorium-specific artifact names or schemas may change when retaining
  them would misrepresent the new contract.
- Any artifact or metadata schema change is explicit, documented, and covered
  by state and inspection tests.
- Prompt or generated content is not added to routine logs or CLI summaries.
- Tests do not require live Promptkit providers or credentials.

## Verification And Completion Criteria

The migration is complete when:

- all seven reports prepare and execute through Promptkit using embedded
  report-owned assets;
- direct-Markdown and generated-text-template paths have deterministic offline
  adapter and app-level coverage;
- preparation, provider failure, capacity rejection, cancellation, timeout,
  Promptkit validation failure, generated-text validation failure, template
  failure, and successful generation preserve their specified artifacts;
- morning and evening batches construct one shared engine and preserve current
  collection, planning, ordering, continuation, output, and notification
  behavior;
- configuration examples load and contain no Scriptorium fields;
- CLI summaries and inspection commands expose the new artifact contract
  without Promptkit dependency types;
- Scriptorium code, configuration, tests, and runtime documentation have been
  removed;
- non-roadmap documentation describes only the implemented Promptkit
  integration;
- `go test ./...`, CLI help validation, and `git diff --check` pass; and
- no committed `go.work`, local `replace`, live-provider test, or secret-bearing
  fixture remains.

Fixture-based comparison with the current Scriptorium behavior is sufficient
for migration verification. A production-selectable dual-run period is not
required because model calls are nondeterministic, costly, and difficult to
compare meaningfully.

## External Prerequisite

Before implementing the embedded asset stage, the current Scriptorium prompt
corpus must be made available in this repository. It should include the seven
prompt definitions, referenced content files, private response schemas,
relevant default-profile declarations, and any shared prompt fragments needed
to reproduce current report behavior.

## Open Questions

- What exact `promptkit.*` configuration fields should replace the current
  Scriptorium fields, including the name and precedence of an optional explicit
  profile override?
- Should weatherreporter expose Promptkit's conventional `local` backend
  registration as a narrow configuration feature, or rely initially on
  built-in and endpoint-only profiles?
- Should report definitions store an explicit Promptkit prompt version, or
  should each embedded prompt ID be required to have exactly one version?
- What CLI or configuration control enables sensitive prompt/response debug
  artifacts, and where should those artifacts live?
- What final names and schema versions should replace the
  Scriptorium-specific preflight and run-result artifacts while balancing
  semantic clarity with existing state-path compatibility?