Add feature roadmap for adoption of the promptkit LLM adapter library

docs/roadmap/promptkit.md

@@ -0,0 +1,304 @@
+# Promptkit Migration Roadmap
+
+## Purpose
+
+This roadmap defines the scope and desired end state for replacing the
+external Scriptorium CLI integration with the Promptkit Go library. The
+migration is not yet implemented. Current Scriptorium behavior remains
+documented outside `docs/roadmap/` until the replacement is complete.
+
+A separate staged implementation plan will describe how to move from the
+current code to this target state. That plan should reference this roadmap
+rather than redefine its architectural decisions or scope.
+
+## Desired End State
+
+Weatherreporter uses a pinned released version of
+`gitea.maximumdirect.net/eric/promptkit` as its in-process prompt preparation
+and LLM execution engine. The `scriptorium` executable, subprocess adapter,
+configuration, runtime dependency, and integration documentation have been
+removed.
+
+The migration does not change weatherreporter's fundamental product behavior.
+Weather selection, forecast derivation, report periods, module construction,
+Recent Changes, generated-text interpretation, Markdown templates, durable
+state, inspection, output copies, and distributor notification remain owned by
+weatherreporter.
+
+All report prompts and private response schemas are versioned application
+assets. Operators may configure Promptkit execution profiles without replacing
+the report-owned prompt and schema corpus. One Promptkit engine is constructed
+per CLI invocation and shared by every report in that invocation, including
+all reports in a morning or evening batch.
+
+Promptkit is isolated behind a weatherreporter-owned prompt execution contract.
+Promptkit request, result, validation, error, profile, backend, and provider
+types do not leak into application orchestration, report definitions, domain
+packages, CLI summaries, state contracts, or distributor behavior.
+
+## Goals
+
+- Remove the runtime dependency on the `scriptorium` executable.
+- Replace shell-free subprocess orchestration with typed in-process Promptkit
+  preparation and execution.
+- Preserve the seven report definitions and their existing prompt IDs.
+- Preserve both direct-Markdown and generated-text-template report workflows.
+- Preserve deterministic module snapshots and structured Recent Changes.
+- Preserve context cancellation, actionable errors, secret redaction, and
+  inspectable failures.
+- Improve durable prompt provenance with prompt, input, profile, model,
+  validation, usage, and timing metadata.
+- Keep content-rich prompt and response diagnostics separate from routine
+  metadata and CLI output.
+- Keep tests offline and deterministic through injected Promptkit model
+  clients and fixtures.
+
+## Non-Goals
+
+The migration will not:
+
+- move meteorological selection, derivation, thresholds, or comparison logic
+  into prompts or Promptkit;
+- send raw unbounded Weather API responses to the model;
+- replace weatherreporter's generated-text domain validation or Markdown
+  template rendering;
+- add a general workflow engine, provider plugin system, or arbitrary backend
+  registry to weatherreporter;
+- add automatic provider, validation, or capacity retries;
+- add concurrent report generation to the existing sequential batch workflow;
+- expose Promptkit types as a weatherreporter component contract;
+- keep a production-selectable Scriptorium/Promptkit dual-run mode; or
+- use an unpublished Promptkit commit, committed Go workspace, or committed
+  local module replacement.
+
+## Locked Decisions
+
+### Dependency And Versioning
+
+- The initial integration will pin Promptkit `v0.3.0`.
+- Coordinated local development may temporarily use the sibling Promptkit
+  checkout, but committed module metadata must reference the tagged release.
+- A future Promptkit upgrade requires an explicit review of the public engine,
+  prompt/profile/schema formats, error identities, validation behavior, and
+  outbound provider contract used by weatherreporter.
+
+### Application Boundary
+
+- Promptkit remains an adapter boundary even though it runs in process.
+- A weatherreporter-owned contract will represent preparation, execution,
+  output formats, validation, usage, provenance, and neutral error categories.
+- The Promptkit adapter will map public Promptkit values into that contract at
+  the boundary.
+- App orchestration and test fakes will depend on the weatherreporter contract,
+  not on Promptkit.
+- Existing Scriptorium-specific generation mode names will be replaced with
+  provider-neutral names.
+
+### Prompt And Schema Ownership
+
+- Weatherreporter will embed all report prompt definitions, prompt content,
+  and private response schemas.
+- Prompt assets will remain separate files rather than inline Go strings.
+- The current Scriptorium prompt corpus will be retrieved before the
+  implementation stage that establishes the embedded Promptkit assets.
+- The retrieved corpus will be reviewed and converted to the pinned Promptkit
+  format without changing report intent or prompt IDs.
+- The four existing generated-text prompt fragments and schemas under
+  `internal/reporttemplate` will be reconciled with that corpus rather than
+  duplicated.
+- Direct-Markdown prompt assets for the three-day, weekend, and storm reports
+  will become weatherreporter-owned assets.
+- Weatherreporter needs one centralized embedded prompt/schema source; it does
+  not need Notarius's multi-module asset-flattening registry.
+
+### Profiles, Backends, And Credentials
+
+- Execution profiles remain operator-configurable rather than embedded report
+  policy.
+- Configuration will support at most one external profile source: a profile
+  directory or a single profile file.
+- Prompt definitions may provide their normal default profile, while
+  weatherreporter may support an explicit configured profile selection.
+- Credential values remain in environment variables or file-backed
+  environment secrets. Configuration contains only credential source names.
+- Provider credentials must not appear in logs, errors, CLI output, durable
+  metadata, preparation artifacts, execution artifacts, or debug summaries.
+- Weatherreporter will not expose Promptkit's general backend registry as
+  arbitrary application configuration.
+
+### Engine Lifetime
+
+- One Promptkit engine will be constructed per CLI invocation at the
+  application composition boundary.
+- Single-report generation will use that engine for preparation and execution.
+- Morning and evening batches will share the same engine across every planned
+  report.
+- Per-report orchestration will not construct its own default Promptkit engine.
+- Promptkit backend capacity state and HTTP transport will therefore be shared
+  consistently for the invocation.
+
+### Prompt Input
+
+- Promptkit will continue to receive the curated `data_package` produced by
+  `internal/promptinput`.
+- Weatherreporter will serialize the data package once, atomically persist
+  those exact bytes, and supply the same bytes as a Promptkit inline artifact.
+- The managed data-package path may be supplied as non-secret artifact
+  provenance.
+- Weatherreporter will not delegate unrestricted path loading to Promptkit's
+  default file artifact reader.
+- The same immutable Promptkit request will be used for preparation and
+  execution so the preflight and run inputs cannot diverge.
+
+### Preparation And Execution
+
+- Promptkit `Prepare` replaces the current Scriptorium render preflight.
+- Promptkit `Run` performs both Markdown and structured generated-text
+  execution.
+- Promptkit basic validation will be used where appropriate for direct
+  Markdown output.
+- Promptkit JSON Schema validation provides the provider-facing and first
+  structured-output check for generated-text reports.
+- Weatherreporter's `internal/generatedtext` validation remains the final
+  report-specific domain boundary.
+- Weatherreporter's `internal/reporttemplate` remains responsible for
+  generated-text Markdown rendering.
+- Weatherreporter will atomically persist Promptkit output rather than asking
+  the dependency to write managed report files.
+- The migration will not rely on Promptkit output repair. Promptkit v0.3.0's
+  public engine validates in a single pass even when a prompt declares repair
+  attempts.
+
+## Durable Artifacts And Observability
+
+Routine durable artifacts should retain useful non-secret provenance without
+persisting full rendered prompts by default.
+
+The preparation record should contain:
+
+- prompt ID and version;
+- prompt definition hash;
+- rendered prompt hash;
+- input hashes;
+- selected profile and backend identity;
+- effective model identity;
+- output contract summary; and
+- preparation timing.
+
+The execution record and run metadata should contain, when available:
+
+- Promptkit run ID;
+- prompt ID, version, and hashes;
+- input hashes;
+- selected profile, backend, and model identity;
+- generated-content hash;
+- token usage;
+- start, end, and duration;
+- validation status and bounded diagnostics; and
+- the path of any separately persisted raw generated output.
+
+Provider endpoints, full effective model parameter maps, rendered messages,
+schema bodies, data-package contents, and generated content do not belong in
+routine metadata or CLI summaries.
+
+Rendered messages and other content-rich preparation or response diagnostics
+will be available only through an explicitly enabled debug mechanism. Debug
+artifacts must be documented as potentially sensitive, must not contain
+credentials, and must have a clear operator-owned retention policy.
+
+## Failure Contract
+
+Promptkit returns a completed `RunResult` for output-validation failure but no
+partial result for operational preparation or execution errors. Weatherreporter
+will preserve that distinction.
+
+- A preparation failure produces a redacted weatherreporter-owned failure
+  receipt with report, RunID, prompt, stage, timing, and classified error
+  context. It does not fabricate a Promptkit preparation result.
+- An operational execution failure retains the successful preparation record
+  and adds a redacted execution failure receipt. No partial Promptkit result or
+  model output is invented.
+- A Promptkit validation failure retains the returned result, raw generated
+  output, validation details, and safe provenance before the report fails.
+- A later weatherreporter generated-text decode, domain-validation, or template
+  failure retains every raw and validated artifact reached before that stage.
+- Context cancellation takes precedence when the caller context is canceled.
+- Promptkit capacity rejection maps to a weatherreporter-owned error category.
+  It is an operational report failure, not invalid model output.
+- Single-report commands return the classified failure with available
+  inspectable paths.
+- Batch runs continue independent later reports under the existing batch
+  failure policy.
+- The migration adds no automatic retries. Any future retry policy belongs to
+  app orchestration, not the Promptkit adapter.
+
+## Compatibility Requirements
+
+- Report IDs, prompt IDs, report selection, valid periods, artifact grouping,
+  output names, and distributor bundle behavior remain stable.
+- Module snapshot and Recent Changes behavior remains deterministic.
+- Promptkit receives only the existing curated prompt-input boundary.
+- Generated reports continue to use the managed Markdown path as the
+  distributor upload source.
+- RunID lookup and inspection remain available for successful and failed runs.
+- Existing managed state paths remain stable where their meaning is unchanged.
+  Scriptorium-specific artifact names or schemas may change when retaining
+  them would misrepresent the new contract.
+- Any artifact or metadata schema change is explicit, documented, and covered
+  by state and inspection tests.
+- Prompt or generated content is not added to routine logs or CLI summaries.
+- Tests do not require live Promptkit providers or credentials.
+
+## Verification And Completion Criteria
+
+The migration is complete when:
+
+- all seven reports prepare and execute through Promptkit using embedded
+  report-owned assets;
+- direct-Markdown and generated-text-template paths have deterministic offline
+  adapter and app-level coverage;
+- preparation, provider failure, capacity rejection, cancellation, timeout,
+  Promptkit validation failure, generated-text validation failure, template
+  failure, and successful generation preserve their specified artifacts;
+- morning and evening batches construct one shared engine and preserve current
+  collection, planning, ordering, continuation, output, and notification
+  behavior;
+- configuration examples load and contain no Scriptorium fields;
+- CLI summaries and inspection commands expose the new artifact contract
+  without Promptkit dependency types;
+- Scriptorium code, configuration, tests, and runtime documentation have been
+  removed;
+- non-roadmap documentation describes only the implemented Promptkit
+  integration;
+- `go test ./...`, CLI help validation, and `git diff --check` pass; and
+- no committed `go.work`, local `replace`, live-provider test, or secret-bearing
+  fixture remains.
+
+Fixture-based comparison with the current Scriptorium behavior is sufficient
+for migration verification. A production-selectable dual-run period is not
+required because model calls are nondeterministic, costly, and difficult to
+compare meaningfully.
+
+## External Prerequisite
+
+Before implementing the embedded asset stage, the current Scriptorium prompt
+corpus must be made available in this repository. It should include the seven
+prompt definitions, referenced content files, private response schemas,
+relevant default-profile declarations, and any shared prompt fragments needed
+to reproduce current report behavior.
+
+## Open Questions
+
+- What exact `promptkit.*` configuration fields should replace the current
+  Scriptorium fields, including the name and precedence of an optional explicit
+  profile override?
+- Should weatherreporter expose Promptkit's conventional `local` backend
+  registration as a narrow configuration feature, or rely initially on
+  built-in and endpoint-only profiles?
+- Should report definitions store an explicit Promptkit prompt version, or
+  should each embedded prompt ID be required to have exactly one version?
+- What CLI or configuration control enables sensitive prompt/response debug
+  artifacts, and where should those artifacts live?
+- What final names and schema versions should replace the
+  Scriptorium-specific preflight and run-result artifacts while balancing
+  semantic clarity with existing state-path compatibility?