weatherreporter/docs/roadmap/promptkit.md

Promptkit Migration Roadmap

Status: Completed roadmap record.

Purpose

This roadmap records the scope, decisions, and completed outcome of replacing the external Scriptorium CLI integration with Promptkit. Canonical documentation outside docs/roadmap/ owns the implemented behavior.

Pre-Migration Baseline

Status: Historical migration input.

Before the migration, Weatherreporter exposed seven report definitions, but only four had complete prompt-backed report implementations:

• Daily Report: weather.daily_generated_text
• Today Report: weather.today_generated_text
• Tomorrow Report: weather.tomorrow_generated_text
• Hourly Report: weather.hourly_generated_text

The three-day, weekend, and storm commands and registry definitions had no corresponding Scriptorium prompt or schema and never formed complete operational report products. The weather.daily_report Scriptorium prompt was legacy source material and was not selected by the registry.

The Scriptorium source corpus was retained temporarily under docs/roadmap/scriptorium/ as migration input. It contained the four operational generated-text prompt definitions, their referenced content, private response schemas, shared instructions, and the unused legacy Daily Markdown prompt. The temporary corpus was removed after the runtime assets were reconciled and embedded.

Implemented End State

Status: Completed.

Weatherreporter pins gitea.maximumdirect.net/eric/promptkit at v0.4.0 and uses it as the in-process engine for prompt inspection, prepared execution, provider calls, and first-pass output validation.

The scriptorium executable, subprocess adapter, configuration, runtime dependency, direct-Markdown execution path, and integration documentation have been removed. The four operational reports continue to use structured generated text followed by weatherreporter-owned validation and Markdown templates.

The unfinished three-day, weekend, and storm reports are not implemented as part of this migration. Their incomplete CLI, registry, documentation, and generation declarations are removed from the implemented surface before the migration is considered complete. Any future implementation of those products requires separate roadmap scope, prompt and schema design, tests, and documentation.

Weather selection, forecast derivation, valid periods, module construction, Recent Changes, generated-text interpretation, Markdown templates, durable state, inspection, output copies, and Distributor notification remain owned by weatherreporter.

The four report prompts and private response schemas are versioned embedded application assets. Operators configure Promptkit profiles without replacing the report-owned corpus. One Promptkit engine is constructed per CLI invocation and shared by every report in that invocation, including all reports in a morning or evening batch.

Promptkit is isolated behind a weatherreporter-owned execution contract. Promptkit request, result, validation, error, profile, backend, and provider types do not leak into application orchestration, report definitions, domain packages, CLI summaries, durable state contracts, or Distributor behavior.

Goals

Status: Completed migration outcomes.

• Removed the Scriptorium runtime dependency and subprocess boundary.
• Migrated the four operational report prompts to Promptkit v0.4.0.
• Used prepared execution to persist preparation provenance before provider work while executing the exact frozen snapshot.
• Validated report prompt and profile selections before weather collection when the required information is available.
• Preserved deterministic module snapshots and structured Recent Changes.
• Preserved generated-text domain validation and repository-owned Markdown rendering.
• Preserved context cancellation, actionable errors, secret redaction, and inspectable failures.
• Improved durable prompt provenance with prompt, input, profile, model, validation, usage, and timing metadata.
• Kept content-rich prompt and response diagnostics separate from routine metadata and CLI output.
• Kept tests offline and deterministic through injected Promptkit model clients and fixtures.
• Removed incomplete report declarations from the implemented product surface rather than creating new report products during an integration migration.

Non-Goals

Status: Completed migration constraints.

The completed migration did not:

• create prompts, schemas, templates, or completed products for three-day, weekend, or storm reports;
• preserve the unused weather.daily_report legacy Markdown prompt as an active runtime asset;
• preserve a direct-Markdown LLM generation mode;
• move meteorological selection, derivation, thresholds, or comparison logic into prompts or Promptkit;
• send raw unbounded Weather API responses to the model;
• replace weatherreporter's generated-text domain validation or Markdown template rendering;
• add a general workflow engine, provider plugin system, or arbitrary backend registry;
• add automatic provider, validation, repair, or capacity retries;
• add concurrent report generation to the sequential batch workflow;
• expose Promptkit types as weatherreporter contracts;
• keep a production-selectable Scriptorium/Promptkit dual-run mode;
• require Promptkit eager source validation, structured generation errors, or semantic execution-target fingerprints; or
• use an unpublished Promptkit commit, committed Go workspace, or committed local module replacement.

Locked Decisions

Status: Implemented migration decisions.

Dependency And Upgrade Boundary

• The migration pins the tagged Promptkit v0.4.0 release.
• Coordinated local development may temporarily use the sibling Promptkit checkout, but committed module metadata must reference the tagged release.
• The adapter relies on the public root Promptkit package only.
• A future Promptkit upgrade requires explicit review of prepared-execution lifecycle, prompt and profile inspection, prompt/profile/schema formats, error identities, validation behavior, capacity behavior, and the outbound provider contract.
• Promptkit's deferred eager source validation, structured generation errors, and semantic execution-target fingerprints do not block this migration.

Operational Report Scope

• The migration preserves these prompt IDs: weather.daily_generated_text, weather.today_generated_text, weather.tomorrow_generated_text, and weather.hourly_generated_text.
• Each operational report definition selects the exact embedded prompt version 1.0.0; execution does not rely on ambiguous single-version lookup.
• Morning and evening batch membership remains based on Today, Tomorrow, and eligible future Daily reports.
• Three-day, weekend, and storm are removed from current CLI help, parsing, report registry membership, tests that claim implemented generation, and non-roadmap documentation.
• The future product concepts may remain under docs/roadmap/, but migration verification does not invent outputs or compare nonexistent prompts.

Application Boundary

• Promptkit remains an adapter boundary even though it runs in process.
• A weatherreporter-owned contract represents prompt identity, preparation, execution, output, validation, usage, provenance, and neutral error categories.
• The Promptkit adapter maps public Promptkit values into that contract.
• App orchestration and test fakes depend on the project-owned contract, not Promptkit.
• Scriptorium-specific request, result, error, and generation-mode types are removed rather than renamed and retained.

Prompt And Schema Ownership

• Weatherreporter embeds the four operational prompt definitions, referenced prompt content, shared prompt content, and private response schemas.
• Assets remain separate files rather than inline Go strings.
• The temporary corpus under docs/roadmap/scriptorium/ is migration source material, not the final runtime location.
• Weatherreporter's existing generated-text domain types, schemas, and templates remain the canonical application contract. Imported Scriptorium assets are reconciled with that contract rather than copied blindly or kept as duplicate runtime schemas.
• The imported Daily schema's incorrect Today $id and title are corrected.
• confidence is handled consistently across each prompt, provider-facing schema, generated-text domain type, and template. The existing optional weatherreporter field remains supported unless a separate domain decision removes it.
• Prompt input metadata identifies the serialized data package as YAML rather than JSON.
• Imported pipeline-weather/... schema paths are replaced with paths valid inside the embedded Promptkit schema source.
• Imported repair_attempts: 2 values are removed or set to zero. The migration does not rely on Promptkit's internal-only repair capability.
• The unused weather.daily_report prompt is not promoted into runtime assets.
• One centralized embedded prompt/schema source is sufficient; Weatherreporter does not need Notarius's multi-module asset-flattening registry.

Profiles, Backends, And Credentials

• Execution profiles remain operator-configurable rather than embedded report policy.
• Each embedded operational prompt declares Promptkit's built-in gemini-flash-latest profile as its default.
• gemini-flash-latest is intentionally a moving model alias. The execution record captures the effective model identity, but operators who require a pinned model must select an explicit external profile.
• Configuration supports at most one external profile source: promptkit.profile_file or promptkit.profile_dir. The two fields are mutually exclusive.
• A nonblank promptkit.profile is the explicit request profile for every report in the invocation and takes precedence over each prompt's default_profile. A blank value uses the prompt default.
• Promptkit's normal profile-source precedence remains intact: an external matching profile takes precedence over an embedded built-in profile, and an invalid matching external profile is an error rather than a reason to fall back.
• Weatherreporter exposes Promptkit's conventional local backend through the narrow promptkit.local.endpoint and promptkit.local.concurrency_limit configuration fields. It does not expose arbitrary backend registration.
• A configured local endpoint registers the engine-scoped local backend. An operator-supplied external profile selects it with backend: local and owns the model-specific settings; Weatherreporter does not invent a local model profile.
• Local concurrency defaults to one. A value of zero means unlimited, matching Promptkit, and a negative value is invalid. Queue capacity and general backend parameters are not exposed.
• Credential values remain in environment variables or file-backed environment secrets. Configuration contains only credential source names.
• Provider credentials never appear in logs, errors, CLI output, durable metadata, preparation records, execution records, or debug summaries.
• Promptkit InspectProfile reports structural target and credential requirements; Weatherreporter owns policy for checking configured environment availability.
• Promptkit revalidates environment credentials at RunPrepared; a successful preparation does not promise that execution-time credentials remain available.

Configuration Contract

The replacement configuration surface is:

promptkit:
  profile: ""
  profile_file: ""
  profile_dir: ""
  timeout: 2m

  local:
    endpoint: ""
    concurrency_limit: 1

• timeout remains the transport-wide provider-call safety cap.
• A blank local endpoint leaves the conventional local backend unregistered.
• Scriptorium's binary, config_path, and extra_args settings have no Promptkit equivalents and are removed.
• Configuration validation rejects simultaneous profile_file and profile_dir values, invalid local endpoints, negative concurrency, and selected profiles that cannot resolve their backend.

Engine Construction And Inspection

• One Promptkit engine is constructed per CLI invocation at the application composition boundary.
• Single-report generation and every report in a batch use that same engine.
• Per-report orchestration does not construct a default engine.
• Promptkit backend capacity state and HTTP transport are shared consistently for the invocation.
• Before collection, InspectPrompt checks every selected report's exact ID and version, declared data_package input, default-profile metadata, prompt hash availability, and declared output contract.
• InspectPrompt is a point-in-time structural check. It does not load a JSON Schema, resolve a profile, or freeze later execution.
• Explicit profile overrides and relevant prompt defaults are checked with InspectProfile before collection when application policy requires them.
• InspectProfile is also point-in-time and does not check credential values.
• Successful PrepareExecution, not inspection, is the per-run authority for loaded schema, rendered content, frozen inputs, effective settings, and durable execution provenance.

Prompt Input

• Promptkit receives only the curated data_package produced by internal/promptinput.
• Weatherreporter serializes the package once, atomically persists those exact bytes, and supplies the same bytes with a Promptkit inline artifact.
• The managed data-package path may be supplied as non-secret provenance through the inline artifact URI.
• Weatherreporter does not delegate unrestricted path loading to Promptkit's default file artifact reader.
• Prompt inspection and adapter tests verify that data_package is required and declared with the chosen YAML media type.

Prepared Execution

• Engine.PrepareExecution replaces Scriptorium render preflight.
• Weatherreporter obtains PreparedExecution.Details, maps a safe subset into its own preparation record, and persists that record before calling Engine.RunPrepared.
• RunPrepared executes the frozen prompt, profile, schema, inputs, rendered messages, target, and validation resources retained by the handle.
• Every acquired handle is followed immediately by defer handle.Discard(). Discard is safe after execution and releases unused private execution state.
• Handles remain adapter-local, engine-bound, one-shot, in-process values. They are never serialized, persisted, copied into app contracts, or treated as restartable jobs.
• Preparation and execution use independent contexts. Execution receives the active report workflow context.
• Capacity is not reserved during preparation. Capacity rejection can therefore occur after a preparation record has been persisted.
• RunPrepared consumes the handle on success and every operational failure.
• Preparation details remain available from the adapter after execution or discard, but rendered messages are not copied into routine durable state.
• Promptkit execution timing excludes preparation and consumer-held delay. Weatherreporter records preparation timing and execution timing separately.

Execution And Validation

• All four operational reports use Promptkit JSON Schema output validation.
• A completed Promptkit validation rejection returns a RunResult; the adapter retains raw output and bounded validation details before failing the report.
• An operational generation or validation error returns no partial RunResult.
• Weatherreporter's internal/generatedtext validation remains the final report-specific decode and domain boundary.
• Weatherreporter's internal/reporttemplate remains responsible for managed Markdown rendering.
• Weatherreporter atomically persists Promptkit raw output and later artifacts rather than asking Promptkit to choose managed filesystem paths.
• No Promptkit output-repair behavior is assumed or requested.

Durable Artifacts And Observability

Status: Implemented design constraints.

Routine durable state retains useful non-secret provenance without persisting full rendered prompts.

The preparation record contains:

• prompt ID and exact version;
• prompt definition hash;
• rendered prompt hash;
• input hashes;
• selected profile and backend identity;
• effective model identity;
• output contract summary;
• preparation start, end, and duration; and
• the path of the exact persisted data package.

The execution record and run metadata contain, when available:

• Promptkit run ID;
• prompt ID, version, and hashes;
• input hashes;
• selected profile, backend, and model identity;
• generated-content hash;
• token usage;
• execution start, end, and duration;
• validation status and bounded diagnostics; and
• paths of separately persisted raw output, normalized generated text, render context, managed Markdown, and other artifacts reached by the workflow.

Provider endpoints, full effective model parameter maps, rendered messages, schema bodies, data-package contents, and generated content do not belong in routine metadata or CLI summaries.

Rendered messages and other content-rich preparation or response diagnostics are available only when the operator supplies --llm-debug-dir <path> to a single-report or batch command.

• There is no persistent YAML setting for debug capture.
• The debug root is validated or created before weather collection or provider work. A requested destination that cannot be secured or written is an error.
• Artifacts are grouped beneath <path>/<report-id>/<valid-date>/<run-id>/.
• Directories and files use owner-only permissions and atomic writes.
• Debug artifacts may contain rendered messages and content-rich preparation or response diagnostics, but never credentials.
• The debug path appears in command output only when debug capture is enabled; it is not added to routine durable metadata.
• Debug artifacts are not cache or comparison inputs. Their retention is owned by the operator who selected the directory.

Artifact Identities And Versions

Weatherreporter replaces Scriptorium-specific artifact identities rather than reusing names whose meanings have changed:

• PromptPreparationArtifact uses schema version weatherreporter.prompt_preparation.v1, is written as prompt_preparation.<runID>.json, and is referenced by preparationPath.
• PromptExecutionArtifact uses schema version weatherreporter.prompt_execution.v1, is written as prompt_execution.<runID>.json, and is referenced by executionPath.
• Run metadata advances to weatherreporter.metadata.v2 and uses those new path fields.

Preparation files remain beneath the existing configurable preflight/ directory, and execution files remain beneath the existing snapshots/ tree. The stable physical grouping limits deployment disruption without preserving misleading Scriptorium-era filenames or field names. Raw generated output, normalized generated text, render context, managed Markdown, and other artifacts whose meanings have not changed retain their existing names and locations.

Run inspection remains able to read weatherreporter.metadata.v1 and its legacy preflightPath and generatedTextResultPath references. New runs write only the v2 metadata and new artifact names; Weatherreporter does not dual-write deprecated aliases. CLI summary fields adopt preparationPath and executionPath as an explicit, documented contract change.

Failure Contract

Status: Implemented design constraints.

• A preparation failure produces a redacted weatherreporter-owned failure receipt with report, RunID, prompt, stage, timing, and classified error context. It does not fabricate Promptkit preparation details.
• An operational execution failure retains the successful preparation record and adds a redacted execution failure receipt. No partial Promptkit result or model output is invented.
• A Promptkit validation rejection retains the returned result, raw generated output, validation details, and safe provenance before the report fails.
• A later generated-text decode, domain-validation, or template failure retains every raw and validated artifact reached before that stage.
• Caller cancellation takes precedence when the active workflow context is canceled.
• promptkit.CapacityError is recognized with errors.As; its backend ID is copied into a weatherreporter-owned capacity error while ErrCapacityExceeded remains the classification.
• Capacity rejection is an operational report failure, not invalid model output, and does not trigger an automatic retry.
• Other Promptkit public error identities are translated into the narrow weatherreporter error categories needed by CLI, metadata, and batch behavior. Diagnostic prose is not parsed as a contract.
• Single-report commands return the classified failure with available inspectable paths.
• Batch runs continue independent later reports under the existing batch failure policy.
• Any future retry policy belongs to app orchestration, not the adapter.

Compatibility Requirements

Status: Implemented design constraints.

• Daily, Today, Tomorrow, and Hourly report IDs, prompt IDs, valid periods, artifact grouping, output names, and Distributor bundle behavior remain stable.
• Morning and evening batch collection, planning, ordering, and continuation behavior remains stable.
• Module snapshot and Recent Changes behavior remains deterministic.
• Promptkit receives only the existing curated prompt-input boundary.
• Managed Markdown remains the Distributor upload source.
• RunID lookup and inspection remain available for successful and failed runs.
• Existing managed paths remain stable where their meaning is unchanged. Scriptorium-specific artifact names or schemas change when retaining them would misrepresent the Promptkit contract.
• Existing v1 run metadata and referenced artifacts remain inspectable after the migration. New runs use the v2 metadata and Promptkit-era artifact identities without dual-writing deprecated aliases.
• Artifact or metadata schema changes are explicit, documented, and covered by state and inspection tests.
• Prompt or generated content is not added to routine logs or CLI summaries.
• Tests do not require live providers or credentials.
• Removing incomplete three-day, weekend, and storm surfaces is documented as correction of an unfinished product boundary, not as successful Promptkit migration of those reports.

Verification And Completion Criteria

Status: Completed and verified.

Completion was verified by the following outcomes:

• the four operational reports inspect, prepare, and execute through Promptkit v0.4.0 using embedded report-owned assets;
• every report uses exact prompt version 1.0.0, requires the YAML data_package, and declares the expected JSON Schema output contract;
• prepared execution persists a safe preparation record before provider work and executes the same frozen snapshot;
• deterministic offline adapter and app tests cover success, preparation failure, credential revalidation, capacity rejection, cancellation, timeout, generation failure, Promptkit validation rejection, generated-text domain failure, template failure, and handle discard;
• morning and evening batches construct one engine and preserve current collection, planning, ordering, continuation, output, and notification behavior;
• the temporary corpus has been reconciled into one runtime prompt/schema source without duplicate provider-facing schemas;
• configuration examples load and contain no Scriptorium fields;
• CLI summaries and inspection commands expose the new project-owned artifact contract without Promptkit types;
• Scriptorium code, configuration, tests, and runtime documentation have been removed;
• incomplete three-day, weekend, and storm commands, registry entries, tests, and current-behavior documentation have been removed or moved to roadmap scope;
• non-roadmap documentation describes only the implemented Promptkit integration;
• go test ./..., required focused or race-enabled checks, CLI help validation, and git diff --check pass; and
• no committed go.work, local replace, live-provider test, or secret-bearing fixture remains.

Fixture-based comparison with prior Scriptorium behavior is sufficient. Production dual-run is not required because model calls are nondeterministic, costly, and difficult to compare meaningfully.

Decision Status

Status: Completed.

The roadmap has no remaining open product or architecture questions. Later changes to this completed scope require new roadmap or decision-record scope rather than implicit changes to this historical record.