weatherreporter/docs/roadmap/promptkit.md

# Promptkit Migration Roadmap

Status: Accepted feature roadmap; the target state is unimplemented.

## Purpose

This roadmap defines the scope and desired end state for replacing the
external Scriptorium CLI integration with Promptkit. Current implemented
behavior remains documented outside `docs/roadmap/` until the replacement is
complete.

## Current Baseline

Status: Current migration input.

Weatherreporter currently exposes seven report definitions, but only four have
complete prompt-backed report implementations:

- Daily Report: `weather.daily_generated_text`
- Today Report: `weather.today_generated_text`
- Tomorrow Report: `weather.tomorrow_generated_text`
- Hourly Report: `weather.hourly_generated_text`

The three-day, weekend, and storm commands and registry definitions have no
corresponding Scriptorium prompt or schema and never formed complete
operational report products. The `weather.daily_report` Scriptorium prompt is
legacy source material and is not selected by the current registry.

The available Scriptorium source corpus is retained temporarily under
`docs/roadmap/scriptorium/`. It contains the four operational generated-text
prompt definitions, their referenced content, their private response schemas,
shared instructions, and the unused legacy Daily Markdown prompt.

## Desired End State

Status: Accepted target state; unimplemented.

Weatherreporter pins
`gitea.maximumdirect.net/eric/promptkit` at `v0.4.0` and uses it as the
in-process engine for prompt inspection, prepared execution, provider calls,
and first-pass output validation.

The `scriptorium` executable, subprocess adapter, configuration, runtime
dependency, direct-Markdown execution path, and integration documentation have
been removed. The four operational reports continue to use structured
generated text followed by weatherreporter-owned validation and Markdown
templates.

The unfinished three-day, weekend, and storm reports are not implemented as
part of this migration. Their incomplete CLI, registry, documentation, and
generation declarations are removed from the implemented surface before the
migration is considered complete. Any future implementation of those products
requires separate roadmap scope, prompt and schema design, tests, and
documentation.

Weather selection, forecast derivation, valid periods, module construction,
Recent Changes, generated-text interpretation, Markdown templates, durable
state, inspection, output copies, and Distributor notification remain owned by
weatherreporter.

The four report prompts and private response schemas are versioned embedded
application assets. Operators configure Promptkit profiles without replacing
the report-owned corpus. One Promptkit engine is constructed per CLI
invocation and shared by every report in that invocation, including all
reports in a morning or evening batch.

Promptkit is isolated behind a weatherreporter-owned execution contract.
Promptkit request, result, validation, error, profile, backend, and provider
types do not leak into application orchestration, report definitions, domain
packages, CLI summaries, durable state contracts, or Distributor behavior.

## Goals

Status: Accepted migration scope; unimplemented.

- Remove the Scriptorium runtime dependency and subprocess boundary.
- Migrate the four operational report prompts to Promptkit `v0.4.0`.
- Use prepared execution to persist preparation provenance before provider work
  while executing the exact frozen snapshot.
- Validate report prompt and profile selections before weather collection when
  the required information is available.
- Preserve deterministic module snapshots and structured Recent Changes.
- Preserve generated-text domain validation and repository-owned Markdown
  rendering.
- Preserve context cancellation, actionable errors, secret redaction, and
  inspectable failures.
- Improve durable prompt provenance with prompt, input, profile, model,
  validation, usage, and timing metadata.
- Keep content-rich prompt and response diagnostics separate from routine
  metadata and CLI output.
- Keep tests offline and deterministic through injected Promptkit model
  clients and fixtures.
- Remove incomplete report declarations from the implemented product surface
  rather than creating new report products during an integration migration.

## Non-Goals

Status: Accepted migration scope; unimplemented.

The migration will not:

- create prompts, schemas, templates, or completed products for three-day,
  weekend, or storm reports;
- preserve the unused `weather.daily_report` legacy Markdown prompt as an
  active runtime asset;
- preserve a direct-Markdown LLM generation mode;
- move meteorological selection, derivation, thresholds, or comparison logic
  into prompts or Promptkit;
- send raw unbounded Weather API responses to the model;
- replace weatherreporter's generated-text domain validation or Markdown
  template rendering;
- add a general workflow engine, provider plugin system, or arbitrary backend
  registry;
- add automatic provider, validation, repair, or capacity retries;
- add concurrent report generation to the sequential batch workflow;
- expose Promptkit types as weatherreporter contracts;
- keep a production-selectable Scriptorium/Promptkit dual-run mode;
- require Promptkit eager source validation, structured generation errors, or
  semantic execution-target fingerprints; or
- use an unpublished Promptkit commit, committed Go workspace, or committed
  local module replacement.

## Locked Decisions

Status: Accepted decisions for the unimplemented migration.

### Dependency And Upgrade Boundary

- The migration pins the tagged Promptkit `v0.4.0` release.
- Coordinated local development may temporarily use the sibling Promptkit
  checkout, but committed module metadata must reference the tagged release.
- The adapter relies on the public root Promptkit package only.
- A future Promptkit upgrade requires explicit review of prepared-execution
  lifecycle, prompt and profile inspection, prompt/profile/schema formats,
  error identities, validation behavior, capacity behavior, and the outbound
  provider contract.
- Promptkit's deferred eager source validation, structured generation errors,
  and semantic execution-target fingerprints do not block this migration.

### Operational Report Scope

- The migration preserves these prompt IDs:
  `weather.daily_generated_text`, `weather.today_generated_text`,
  `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`.
- Each operational report definition selects the exact embedded prompt version
  `1.0.0`; execution does not rely on ambiguous single-version lookup.
- Morning and evening batch membership remains based on Today, Tomorrow, and
  eligible future Daily reports.
- Three-day, weekend, and storm are removed from current CLI help, parsing,
  report registry membership, tests that claim implemented generation, and
  non-roadmap documentation.
- The future product concepts may remain under `docs/roadmap/`, but migration
  verification does not invent outputs or compare nonexistent prompts.

### Application Boundary

- Promptkit remains an adapter boundary even though it runs in process.
- A weatherreporter-owned contract represents prompt identity, preparation,
  execution, output, validation, usage, provenance, and neutral error
  categories.
- The Promptkit adapter maps public Promptkit values into that contract.
- App orchestration and test fakes depend on the project-owned contract, not
  Promptkit.
- Scriptorium-specific request, result, error, and generation-mode types are
  removed rather than renamed and retained.

### Prompt And Schema Ownership

- Weatherreporter embeds the four operational prompt definitions, referenced
  prompt content, shared prompt content, and private response schemas.
- Assets remain separate files rather than inline Go strings.
- The temporary corpus under `docs/roadmap/scriptorium/` is migration source
  material, not the final runtime location.
- Weatherreporter's existing generated-text domain types, schemas, and
  templates remain the canonical application contract. Imported Scriptorium
  assets are reconciled with that contract rather than copied blindly or kept
  as duplicate runtime schemas.
- The imported Daily schema's incorrect Today `$id` and title are corrected.
- `confidence` is handled consistently across each prompt, provider-facing
  schema, generated-text domain type, and template. The existing optional
  weatherreporter field remains supported unless a separate domain decision
  removes it.
- Prompt input metadata identifies the serialized data package as YAML rather
  than JSON.
- Imported `pipeline-weather/...` schema paths are replaced with paths valid
  inside the embedded Promptkit schema source.
- Imported `repair_attempts: 2` values are removed or set to zero. The
  migration does not rely on Promptkit's internal-only repair capability.
- The unused `weather.daily_report` prompt is not promoted into runtime assets.
- One centralized embedded prompt/schema source is sufficient; Weatherreporter
  does not need Notarius's multi-module asset-flattening registry.

### Profiles, Backends, And Credentials

- Execution profiles remain operator-configurable rather than embedded report
  policy.
- Each embedded operational prompt declares Promptkit's built-in
  `gemini-flash-latest` profile as its default.
- `gemini-flash-latest` is intentionally a moving model alias. The execution
  record captures the effective model identity, but operators who require a
  pinned model must select an explicit external profile.
- Configuration supports at most one external profile source:
  `promptkit.profile_file` or `promptkit.profile_dir`. The two fields are
  mutually exclusive.
- A nonblank `promptkit.profile` is the explicit request profile for every
  report in the invocation and takes precedence over each prompt's
  `default_profile`. A blank value uses the prompt default.
- Promptkit's normal profile-source precedence remains intact: an external
  matching profile takes precedence over an embedded built-in profile, and an
  invalid matching external profile is an error rather than a reason to fall
  back.
- Weatherreporter exposes Promptkit's conventional `local` backend through the
  narrow `promptkit.local.endpoint` and
  `promptkit.local.concurrency_limit` configuration fields. It does not expose
  arbitrary backend registration.
- A configured local endpoint registers the engine-scoped `local` backend. An
  operator-supplied external profile selects it with `backend: local` and owns
  the model-specific settings; Weatherreporter does not invent a local model
  profile.
- Local concurrency defaults to one. A value of zero means unlimited, matching
  Promptkit, and a negative value is invalid. Queue capacity and general
  backend parameters are not exposed.
- Credential values remain in environment variables or file-backed
  environment secrets. Configuration contains only credential source names.
- Provider credentials never appear in logs, errors, CLI output, durable
  metadata, preparation records, execution records, or debug summaries.
- Promptkit `InspectProfile` reports structural target and credential
  requirements; Weatherreporter owns policy for checking configured
  environment availability.
- Promptkit revalidates environment credentials at `RunPrepared`; a successful
  preparation does not promise that execution-time credentials remain
  available.

### Configuration Contract

The replacement configuration surface is:

```yaml
promptkit:
  profile: ""
  profile_file: ""
  profile_dir: ""
  timeout: 2m

  local:
    endpoint: ""
    concurrency_limit: 1
```

- `timeout` remains the transport-wide provider-call safety cap.
- A blank local endpoint leaves the conventional local backend unregistered.
- Scriptorium's `binary`, `config_path`, and `extra_args` settings have no
  Promptkit equivalents and are removed.
- Configuration validation rejects simultaneous `profile_file` and
  `profile_dir` values, invalid local endpoints, negative concurrency, and
  selected profiles that cannot resolve their backend.

### Engine Construction And Inspection

- One Promptkit engine is constructed per CLI invocation at the application
  composition boundary.
- Single-report generation and every report in a batch use that same engine.
- Per-report orchestration does not construct a default engine.
- Promptkit backend capacity state and HTTP transport are shared consistently
  for the invocation.
- Before collection, `InspectPrompt` checks every selected report's exact ID
  and version, declared `data_package` input, default-profile metadata, prompt
  hash availability, and declared output contract.
- `InspectPrompt` is a point-in-time structural check. It does not load a JSON
  Schema, resolve a profile, or freeze later execution.
- Explicit profile overrides and relevant prompt defaults are checked with
  `InspectProfile` before collection when application policy requires them.
- `InspectProfile` is also point-in-time and does not check credential values.
- Successful `PrepareExecution`, not inspection, is the per-run authority for
  loaded schema, rendered content, frozen inputs, effective settings, and
  durable execution provenance.

### Prompt Input

- Promptkit receives only the curated `data_package` produced by
  `internal/promptinput`.
- Weatherreporter serializes the package once, atomically persists those exact
  bytes, and supplies the same bytes with a Promptkit inline artifact.
- The managed data-package path may be supplied as non-secret provenance
  through the inline artifact URI.
- Weatherreporter does not delegate unrestricted path loading to Promptkit's
  default file artifact reader.
- Prompt inspection and adapter tests verify that `data_package` is required
  and declared with the chosen YAML media type.

### Prepared Execution

- `Engine.PrepareExecution` replaces Scriptorium render preflight.
- Weatherreporter obtains `PreparedExecution.Details`, maps a safe subset into
  its own preparation record, and persists that record before calling
  `Engine.RunPrepared`.
- `RunPrepared` executes the frozen prompt, profile, schema, inputs, rendered
  messages, target, and validation resources retained by the handle.
- Every acquired handle is followed immediately by `defer handle.Discard()`.
  Discard is safe after execution and releases unused private execution state.
- Handles remain adapter-local, engine-bound, one-shot, in-process values.
  They are never serialized, persisted, copied into app contracts, or treated
  as restartable jobs.
- Preparation and execution use independent contexts. Execution receives the
  active report workflow context.
- Capacity is not reserved during preparation. Capacity rejection can
  therefore occur after a preparation record has been persisted.
- `RunPrepared` consumes the handle on success and every operational failure.
- Preparation details remain available from the adapter after execution or
  discard, but rendered messages are not copied into routine durable state.
- Promptkit execution timing excludes preparation and consumer-held delay.
  Weatherreporter records preparation timing and execution timing separately.

### Execution And Validation

- All four operational reports use Promptkit JSON Schema output validation.
- A completed Promptkit validation rejection returns a `RunResult`; the
  adapter retains raw output and bounded validation details before failing the
  report.
- An operational generation or validation error returns no partial
  `RunResult`.
- Weatherreporter's `internal/generatedtext` validation remains the final
  report-specific decode and domain boundary.
- Weatherreporter's `internal/reporttemplate` remains responsible for managed
  Markdown rendering.
- Weatherreporter atomically persists Promptkit raw output and later artifacts
  rather than asking Promptkit to choose managed filesystem paths.
- No Promptkit output-repair behavior is assumed or requested.

## Durable Artifacts And Observability

Status: Accepted design constraints; unimplemented.

Routine durable state retains useful non-secret provenance without persisting
full rendered prompts.

The preparation record contains:

- prompt ID and exact version;
- prompt definition hash;
- rendered prompt hash;
- input hashes;
- selected profile and backend identity;
- effective model identity;
- output contract summary;
- preparation start, end, and duration; and
- the path of the exact persisted data package.

The execution record and run metadata contain, when available:

- Promptkit run ID;
- prompt ID, version, and hashes;
- input hashes;
- selected profile, backend, and model identity;
- generated-content hash;
- token usage;
- execution start, end, and duration;
- validation status and bounded diagnostics; and
- paths of separately persisted raw output, normalized generated text, render
  context, managed Markdown, and other artifacts reached by the workflow.

Provider endpoints, full effective model parameter maps, rendered messages,
schema bodies, data-package contents, and generated content do not belong in
routine metadata or CLI summaries.

Rendered messages and other content-rich preparation or response diagnostics
are available only when the operator supplies
`--llm-debug-dir <path>` to a single-report or batch command.

- There is initially no persistent YAML setting for debug capture.
- The debug root is validated or created before weather collection or provider
  work. A requested destination that cannot be secured or written is an error.
- Artifacts are grouped beneath
  `<path>/<report-id>/<valid-date>/<run-id>/`.
- Directories and files use owner-only permissions and atomic writes.
- Debug artifacts may contain rendered messages and content-rich preparation
  or response diagnostics, but never credentials.
- The debug path appears in command output only when debug capture is enabled;
  it is not added to routine durable metadata.
- Debug artifacts are not cache or comparison inputs. Their retention is owned
  by the operator who selected the directory.

### Artifact Identities And Versions

Weatherreporter replaces Scriptorium-specific artifact identities rather than
reusing names whose meanings have changed:

- `PromptPreparationArtifact` uses schema version
  `weatherreporter.prompt_preparation.v1`, is written as
  `prompt_preparation.<runID>.json`, and is referenced by
  `preparationPath`.
- `PromptExecutionArtifact` uses schema version
  `weatherreporter.prompt_execution.v1`, is written as
  `prompt_execution.<runID>.json`, and is referenced by `executionPath`.
- Run metadata advances to `weatherreporter.metadata.v2` and uses those new
  path fields.

Preparation files remain beneath the existing configurable `preflight/`
directory, and execution files remain beneath the existing `snapshots/` tree.
The stable physical grouping limits deployment disruption without preserving
misleading Scriptorium-era filenames or field names. Raw generated output,
normalized generated text, render context, managed Markdown, and other
artifacts whose meanings have not changed retain their existing names and
locations.

Run inspection remains able to read `weatherreporter.metadata.v1` and its
legacy `preflightPath` and `generatedTextResultPath` references. New runs write
only the v2 metadata and new artifact names; Weatherreporter does not
dual-write deprecated aliases. CLI summary fields adopt `preparationPath` and
`executionPath` as an explicit, documented contract change.

## Failure Contract

Status: Accepted design constraints; unimplemented.

- A preparation failure produces a redacted weatherreporter-owned failure
  receipt with report, RunID, prompt, stage, timing, and classified error
  context. It does not fabricate Promptkit preparation details.
- An operational execution failure retains the successful preparation record
  and adds a redacted execution failure receipt. No partial Promptkit result or
  model output is invented.
- A Promptkit validation rejection retains the returned result, raw generated
  output, validation details, and safe provenance before the report fails.
- A later generated-text decode, domain-validation, or template failure
  retains every raw and validated artifact reached before that stage.
- Caller cancellation takes precedence when the active workflow context is
  canceled.
- `promptkit.CapacityError` is recognized with `errors.As`; its backend ID is
  copied into a weatherreporter-owned capacity error while
  `ErrCapacityExceeded` remains the classification.
- Capacity rejection is an operational report failure, not invalid model
  output, and does not trigger an automatic retry.
- Other Promptkit public error identities are translated into the narrow
  weatherreporter error categories needed by CLI, metadata, and batch
  behavior. Diagnostic prose is not parsed as a contract.
- Single-report commands return the classified failure with available
  inspectable paths.
- Batch runs continue independent later reports under the existing batch
  failure policy.
- Any future retry policy belongs to app orchestration, not the adapter.

## Compatibility Requirements

Status: Accepted design constraints; unimplemented.

- Daily, Today, Tomorrow, and Hourly report IDs, prompt IDs, valid periods,
  artifact grouping, output names, and Distributor bundle behavior remain
  stable.
- Morning and evening batch collection, planning, ordering, and continuation
  behavior remains stable.
- Module snapshot and Recent Changes behavior remains deterministic.
- Promptkit receives only the existing curated prompt-input boundary.
- Managed Markdown remains the Distributor upload source.
- RunID lookup and inspection remain available for successful and failed runs.
- Existing managed paths remain stable where their meaning is unchanged.
  Scriptorium-specific artifact names or schemas change when retaining them
  would misrepresent the Promptkit contract.
- Existing v1 run metadata and referenced artifacts remain inspectable after
  the migration. New runs use the v2 metadata and Promptkit-era artifact
  identities without dual-writing deprecated aliases.
- Artifact or metadata schema changes are explicit, documented, and covered by
  state and inspection tests.
- Prompt or generated content is not added to routine logs or CLI summaries.
- Tests do not require live providers or credentials.
- Removing incomplete three-day, weekend, and storm surfaces is documented as
  correction of an unfinished product boundary, not as successful Promptkit
  migration of those reports.

## Verification And Completion Criteria

Status: Accepted completion criteria for the unimplemented migration.

The migration is complete when:

- the four operational reports inspect, prepare, and execute through Promptkit
  `v0.4.0` using embedded report-owned assets;
- every report uses exact prompt version `1.0.0`, requires the YAML
  `data_package`, and declares the expected JSON Schema output contract;
- prepared execution persists a safe preparation record before provider work
  and executes the same frozen snapshot;
- deterministic offline adapter and app tests cover success, preparation
  failure, credential revalidation, capacity rejection, cancellation, timeout,
  generation failure, Promptkit validation rejection, generated-text domain
  failure, template failure, and handle discard;
- morning and evening batches construct one engine and preserve current
  collection, planning, ordering, continuation, output, and notification
  behavior;
- the temporary corpus has been reconciled into one runtime prompt/schema
  source without duplicate provider-facing schemas;
- configuration examples load and contain no Scriptorium fields;
- CLI summaries and inspection commands expose the new project-owned artifact
  contract without Promptkit types;
- Scriptorium code, configuration, tests, and runtime documentation have been
  removed;
- incomplete three-day, weekend, and storm commands, registry entries, tests,
  and current-behavior documentation have been removed or moved to roadmap
  scope;
- non-roadmap documentation describes only the implemented Promptkit
  integration;
- `go test ./...`, required focused or race-enabled checks, CLI help
  validation, and `git diff --check` pass; and
- no committed `go.work`, local `replace`, live-provider test, or
  secret-bearing fixture remains.

Fixture-based comparison with prior Scriptorium behavior is sufficient.
Production dual-run is not required because model calls are nondeterministic,
costly, and difficult to compare meaningfully.

## Decision Status

Status: Decision-complete.

The roadmap has no remaining open product or architecture questions. Any later
question that changes migration scope, policy, or the desired end state must be
resolved here rather than implicitly in code.