Prepare reports for Promptkit migration
This commit is contained in:
@@ -1,56 +1,87 @@
|
||||
# Promptkit Migration Roadmap
|
||||
|
||||
Status: Accepted migration policy; the migration itself is unimplemented.
|
||||
Status: Accepted feature roadmap; the target state is unimplemented.
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the scope and desired end state for replacing the
|
||||
external Scriptorium CLI integration with the Promptkit Go library. The
|
||||
migration is not yet implemented. Current Scriptorium behavior remains
|
||||
documented in the [Scriptorium integration guide](../integrations/scriptorium.md)
|
||||
until the replacement is complete.
|
||||
external Scriptorium CLI integration with Promptkit. Current implemented
|
||||
behavior remains documented outside `docs/roadmap/` until the replacement is
|
||||
complete.
|
||||
|
||||
A separate staged implementation plan will describe how to move from the
|
||||
current code to this target state. That plan should reference this roadmap
|
||||
rather than redefine its architectural decisions or scope.
|
||||
## Current Baseline
|
||||
|
||||
Status: Current migration input.
|
||||
|
||||
Weatherreporter currently exposes seven report definitions, but only four have
|
||||
complete prompt-backed report implementations:
|
||||
|
||||
- Daily Report: `weather.daily_generated_text`
|
||||
- Today Report: `weather.today_generated_text`
|
||||
- Tomorrow Report: `weather.tomorrow_generated_text`
|
||||
- Hourly Report: `weather.hourly_generated_text`
|
||||
|
||||
The three-day, weekend, and storm commands and registry definitions have no
|
||||
corresponding Scriptorium prompt or schema and never formed complete
|
||||
operational report products. The `weather.daily_report` Scriptorium prompt is
|
||||
legacy source material and is not selected by the current registry.
|
||||
|
||||
The available Scriptorium source corpus is retained temporarily under
|
||||
`docs/roadmap/scriptorium/`. It contains the four operational generated-text
|
||||
prompt definitions, their referenced content, their private response schemas,
|
||||
shared instructions, and the unused legacy Daily Markdown prompt.
|
||||
|
||||
## Desired End State
|
||||
|
||||
Status: Accepted target state; unimplemented.
|
||||
|
||||
Weatherreporter uses a pinned released version of
|
||||
`gitea.maximumdirect.net/eric/promptkit` as its in-process prompt preparation
|
||||
and LLM execution engine. The `scriptorium` executable, subprocess adapter,
|
||||
configuration, runtime dependency, and integration documentation have been
|
||||
removed.
|
||||
Weatherreporter pins
|
||||
`gitea.maximumdirect.net/eric/promptkit` at `v0.4.0` and uses it as the
|
||||
in-process engine for prompt inspection, prepared execution, provider calls,
|
||||
and first-pass output validation.
|
||||
|
||||
The migration does not change weatherreporter's fundamental product behavior.
|
||||
Weather selection, forecast derivation, report periods, module construction,
|
||||
The `scriptorium` executable, subprocess adapter, configuration, runtime
|
||||
dependency, direct-Markdown execution path, and integration documentation have
|
||||
been removed. The four operational reports continue to use structured
|
||||
generated text followed by weatherreporter-owned validation and Markdown
|
||||
templates.
|
||||
|
||||
The unfinished three-day, weekend, and storm reports are not implemented as
|
||||
part of this migration. Their incomplete CLI, registry, documentation, and
|
||||
generation declarations are removed from the implemented surface before the
|
||||
migration is considered complete. Any future implementation of those products
|
||||
requires separate roadmap scope, prompt and schema design, tests, and
|
||||
documentation.
|
||||
|
||||
Weather selection, forecast derivation, valid periods, module construction,
|
||||
Recent Changes, generated-text interpretation, Markdown templates, durable
|
||||
state, inspection, output copies, and distributor notification remain owned by
|
||||
state, inspection, output copies, and Distributor notification remain owned by
|
||||
weatherreporter.
|
||||
|
||||
All report prompts and private response schemas are versioned application
|
||||
assets. Operators may configure Promptkit execution profiles without replacing
|
||||
the report-owned prompt and schema corpus. One Promptkit engine is constructed
|
||||
per CLI invocation and shared by every report in that invocation, including
|
||||
all reports in a morning or evening batch.
|
||||
The four report prompts and private response schemas are versioned embedded
|
||||
application assets. Operators configure Promptkit profiles without replacing
|
||||
the report-owned corpus. One Promptkit engine is constructed per CLI
|
||||
invocation and shared by every report in that invocation, including all
|
||||
reports in a morning or evening batch.
|
||||
|
||||
Promptkit is isolated behind a weatherreporter-owned prompt execution contract.
|
||||
Promptkit is isolated behind a weatherreporter-owned execution contract.
|
||||
Promptkit request, result, validation, error, profile, backend, and provider
|
||||
types do not leak into application orchestration, report definitions, domain
|
||||
packages, CLI summaries, state contracts, or distributor behavior.
|
||||
packages, CLI summaries, durable state contracts, or Distributor behavior.
|
||||
|
||||
## Goals
|
||||
|
||||
Status: Accepted migration scope; unimplemented.
|
||||
|
||||
- Remove the runtime dependency on the `scriptorium` executable.
|
||||
- Replace shell-free subprocess orchestration with typed in-process Promptkit
|
||||
preparation and execution.
|
||||
- Preserve the seven report definitions and their existing prompt IDs.
|
||||
- Preserve both direct-Markdown and generated-text-template report workflows.
|
||||
- Remove the Scriptorium runtime dependency and subprocess boundary.
|
||||
- Migrate the four operational report prompts to Promptkit `v0.4.0`.
|
||||
- Use prepared execution to persist preparation provenance before provider work
|
||||
while executing the exact frozen snapshot.
|
||||
- Validate report prompt and profile selections before weather collection when
|
||||
the required information is available.
|
||||
- Preserve deterministic module snapshots and structured Recent Changes.
|
||||
- Preserve generated-text domain validation and repository-owned Markdown
|
||||
rendering.
|
||||
- Preserve context cancellation, actionable errors, secret redaction, and
|
||||
inspectable failures.
|
||||
- Improve durable prompt provenance with prompt, input, profile, model,
|
||||
@@ -59,6 +90,8 @@ Status: Accepted migration scope; unimplemented.
|
||||
metadata and CLI output.
|
||||
- Keep tests offline and deterministic through injected Promptkit model
|
||||
clients and fixtures.
|
||||
- Remove incomplete report declarations from the implemented product surface
|
||||
rather than creating new report products during an integration migration.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
@@ -66,17 +99,24 @@ Status: Accepted migration scope; unimplemented.
|
||||
|
||||
The migration will not:
|
||||
|
||||
- create prompts, schemas, templates, or completed products for three-day,
|
||||
weekend, or storm reports;
|
||||
- preserve the unused `weather.daily_report` legacy Markdown prompt as an
|
||||
active runtime asset;
|
||||
- preserve a direct-Markdown LLM generation mode;
|
||||
- move meteorological selection, derivation, thresholds, or comparison logic
|
||||
into prompts or Promptkit;
|
||||
- send raw unbounded Weather API responses to the model;
|
||||
- replace weatherreporter's generated-text domain validation or Markdown
|
||||
template rendering;
|
||||
- add a general workflow engine, provider plugin system, or arbitrary backend
|
||||
registry to weatherreporter;
|
||||
- add automatic provider, validation, or capacity retries;
|
||||
- add concurrent report generation to the existing sequential batch workflow;
|
||||
- expose Promptkit types as a weatherreporter component contract;
|
||||
- keep a production-selectable Scriptorium/Promptkit dual-run mode; or
|
||||
registry;
|
||||
- add automatic provider, validation, repair, or capacity retries;
|
||||
- add concurrent report generation to the sequential batch workflow;
|
||||
- expose Promptkit types as weatherreporter contracts;
|
||||
- keep a production-selectable Scriptorium/Promptkit dual-run mode;
|
||||
- require Promptkit eager source validation, structured generation errors, or
|
||||
semantic execution-target fingerprints; or
|
||||
- use an unpublished Promptkit commit, committed Go workspace, or committed
|
||||
local module replacement.
|
||||
|
||||
@@ -84,121 +124,229 @@ The migration will not:
|
||||
|
||||
Status: Accepted decisions for the unimplemented migration.
|
||||
|
||||
### Dependency And Versioning
|
||||
### Dependency And Upgrade Boundary
|
||||
|
||||
- The initial integration will pin Promptkit `v0.3.0`.
|
||||
- The migration pins the tagged Promptkit `v0.4.0` release.
|
||||
- Coordinated local development may temporarily use the sibling Promptkit
|
||||
checkout, but committed module metadata must reference the tagged release.
|
||||
- A future Promptkit upgrade requires an explicit review of the public engine,
|
||||
prompt/profile/schema formats, error identities, validation behavior, and
|
||||
outbound provider contract used by weatherreporter.
|
||||
- The adapter relies on the public root Promptkit package only.
|
||||
- A future Promptkit upgrade requires explicit review of prepared-execution
|
||||
lifecycle, prompt and profile inspection, prompt/profile/schema formats,
|
||||
error identities, validation behavior, capacity behavior, and the outbound
|
||||
provider contract.
|
||||
- Promptkit's deferred eager source validation, structured generation errors,
|
||||
and semantic execution-target fingerprints do not block this migration.
|
||||
|
||||
### Operational Report Scope
|
||||
|
||||
- The migration preserves these prompt IDs:
|
||||
`weather.daily_generated_text`, `weather.today_generated_text`,
|
||||
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`.
|
||||
- Each operational report definition selects the exact embedded prompt version
|
||||
`1.0.0`; execution does not rely on ambiguous single-version lookup.
|
||||
- Morning and evening batch membership remains based on Today, Tomorrow, and
|
||||
eligible future Daily reports.
|
||||
- Three-day, weekend, and storm are removed from current CLI help, parsing,
|
||||
report registry membership, tests that claim implemented generation, and
|
||||
non-roadmap documentation.
|
||||
- The future product concepts may remain under `docs/roadmap/`, but migration
|
||||
verification does not invent outputs or compare nonexistent prompts.
|
||||
|
||||
### Application Boundary
|
||||
|
||||
- Promptkit remains an adapter boundary even though it runs in process.
|
||||
- A weatherreporter-owned contract will represent preparation, execution,
|
||||
output formats, validation, usage, provenance, and neutral error categories.
|
||||
- The Promptkit adapter will map public Promptkit values into that contract at
|
||||
the boundary.
|
||||
- App orchestration and test fakes will depend on the weatherreporter contract,
|
||||
not on Promptkit.
|
||||
- Existing Scriptorium-specific generation mode names will be replaced with
|
||||
provider-neutral names.
|
||||
- A weatherreporter-owned contract represents prompt identity, preparation,
|
||||
execution, output, validation, usage, provenance, and neutral error
|
||||
categories.
|
||||
- The Promptkit adapter maps public Promptkit values into that contract.
|
||||
- App orchestration and test fakes depend on the project-owned contract, not
|
||||
Promptkit.
|
||||
- Scriptorium-specific request, result, error, and generation-mode types are
|
||||
removed rather than renamed and retained.
|
||||
|
||||
### Prompt And Schema Ownership
|
||||
|
||||
- Weatherreporter will embed all report prompt definitions, prompt content,
|
||||
and private response schemas.
|
||||
- Prompt assets will remain separate files rather than inline Go strings.
|
||||
- The current Scriptorium prompt corpus will be retrieved before the
|
||||
implementation stage that establishes the embedded Promptkit assets.
|
||||
- The retrieved corpus will be reviewed and converted to the pinned Promptkit
|
||||
format without changing report intent or prompt IDs.
|
||||
- The four existing generated-text prompt fragments and schemas under
|
||||
`internal/reporttemplate` will be reconciled with that corpus rather than
|
||||
duplicated.
|
||||
- Direct-Markdown prompt assets for the three-day, weekend, and storm reports
|
||||
will become weatherreporter-owned assets.
|
||||
- Weatherreporter needs one centralized embedded prompt/schema source; it does
|
||||
not need Notarius's multi-module asset-flattening registry.
|
||||
- Weatherreporter embeds the four operational prompt definitions, referenced
|
||||
prompt content, shared prompt content, and private response schemas.
|
||||
- Assets remain separate files rather than inline Go strings.
|
||||
- The temporary corpus under `docs/roadmap/scriptorium/` is migration source
|
||||
material, not the final runtime location.
|
||||
- Weatherreporter's existing generated-text domain types, schemas, and
|
||||
templates remain the canonical application contract. Imported Scriptorium
|
||||
assets are reconciled with that contract rather than copied blindly or kept
|
||||
as duplicate runtime schemas.
|
||||
- The imported Daily schema's incorrect Today `$id` and title are corrected.
|
||||
- `confidence` is handled consistently across each prompt, provider-facing
|
||||
schema, generated-text domain type, and template. The existing optional
|
||||
weatherreporter field remains supported unless a separate domain decision
|
||||
removes it.
|
||||
- Prompt input metadata identifies the serialized data package as YAML rather
|
||||
than JSON.
|
||||
- Imported `pipeline-weather/...` schema paths are replaced with paths valid
|
||||
inside the embedded Promptkit schema source.
|
||||
- Imported `repair_attempts: 2` values are removed or set to zero. The
|
||||
migration does not rely on Promptkit's internal-only repair capability.
|
||||
- The unused `weather.daily_report` prompt is not promoted into runtime assets.
|
||||
- One centralized embedded prompt/schema source is sufficient; Weatherreporter
|
||||
does not need Notarius's multi-module asset-flattening registry.
|
||||
|
||||
### Profiles, Backends, And Credentials
|
||||
|
||||
- Execution profiles remain operator-configurable rather than embedded report
|
||||
policy.
|
||||
- Configuration will support at most one external profile source: a profile
|
||||
directory or a single profile file.
|
||||
- Prompt definitions may provide their normal default profile, while
|
||||
weatherreporter may support an explicit configured profile selection.
|
||||
- Each embedded operational prompt declares Promptkit's built-in
|
||||
`gemini-flash-latest` profile as its default.
|
||||
- `gemini-flash-latest` is intentionally a moving model alias. The execution
|
||||
record captures the effective model identity, but operators who require a
|
||||
pinned model must select an explicit external profile.
|
||||
- Configuration supports at most one external profile source:
|
||||
`promptkit.profile_file` or `promptkit.profile_dir`. The two fields are
|
||||
mutually exclusive.
|
||||
- A nonblank `promptkit.profile` is the explicit request profile for every
|
||||
report in the invocation and takes precedence over each prompt's
|
||||
`default_profile`. A blank value uses the prompt default.
|
||||
- Promptkit's normal profile-source precedence remains intact: an external
|
||||
matching profile takes precedence over an embedded built-in profile, and an
|
||||
invalid matching external profile is an error rather than a reason to fall
|
||||
back.
|
||||
- Weatherreporter exposes Promptkit's conventional `local` backend through the
|
||||
narrow `promptkit.local.endpoint` and
|
||||
`promptkit.local.concurrency_limit` configuration fields. It does not expose
|
||||
arbitrary backend registration.
|
||||
- A configured local endpoint registers the engine-scoped `local` backend. An
|
||||
operator-supplied external profile selects it with `backend: local` and owns
|
||||
the model-specific settings; Weatherreporter does not invent a local model
|
||||
profile.
|
||||
- Local concurrency defaults to one. A value of zero means unlimited, matching
|
||||
Promptkit, and a negative value is invalid. Queue capacity and general
|
||||
backend parameters are not exposed.
|
||||
- Credential values remain in environment variables or file-backed
|
||||
environment secrets. Configuration contains only credential source names.
|
||||
- Provider credentials must not appear in logs, errors, CLI output, durable
|
||||
metadata, preparation artifacts, execution artifacts, or debug summaries.
|
||||
- Weatherreporter will not expose Promptkit's general backend registry as
|
||||
arbitrary application configuration.
|
||||
- Provider credentials never appear in logs, errors, CLI output, durable
|
||||
metadata, preparation records, execution records, or debug summaries.
|
||||
- Promptkit `InspectProfile` reports structural target and credential
|
||||
requirements; Weatherreporter owns policy for checking configured
|
||||
environment availability.
|
||||
- Promptkit revalidates environment credentials at `RunPrepared`; a successful
|
||||
preparation does not promise that execution-time credentials remain
|
||||
available.
|
||||
|
||||
### Engine Lifetime
|
||||
### Configuration Contract
|
||||
|
||||
- One Promptkit engine will be constructed per CLI invocation at the
|
||||
application composition boundary.
|
||||
- Single-report generation will use that engine for preparation and execution.
|
||||
- Morning and evening batches will share the same engine across every planned
|
||||
report.
|
||||
- Per-report orchestration will not construct its own default Promptkit engine.
|
||||
- Promptkit backend capacity state and HTTP transport will therefore be shared
|
||||
consistently for the invocation.
|
||||
The replacement configuration surface is:
|
||||
|
||||
```yaml
|
||||
promptkit:
|
||||
profile: ""
|
||||
profile_file: ""
|
||||
profile_dir: ""
|
||||
timeout: 2m
|
||||
|
||||
local:
|
||||
endpoint: ""
|
||||
concurrency_limit: 1
|
||||
```
|
||||
|
||||
- `timeout` remains the transport-wide provider-call safety cap.
|
||||
- A blank local endpoint leaves the conventional local backend unregistered.
|
||||
- Scriptorium's `binary`, `config_path`, and `extra_args` settings have no
|
||||
Promptkit equivalents and are removed.
|
||||
- Configuration validation rejects simultaneous `profile_file` and
|
||||
`profile_dir` values, invalid local endpoints, negative concurrency, and
|
||||
selected profiles that cannot resolve their backend.
|
||||
|
||||
### Engine Construction And Inspection
|
||||
|
||||
- One Promptkit engine is constructed per CLI invocation at the application
|
||||
composition boundary.
|
||||
- Single-report generation and every report in a batch use that same engine.
|
||||
- Per-report orchestration does not construct a default engine.
|
||||
- Promptkit backend capacity state and HTTP transport are shared consistently
|
||||
for the invocation.
|
||||
- Before collection, `InspectPrompt` checks every selected report's exact ID
|
||||
and version, declared `data_package` input, default-profile metadata, prompt
|
||||
hash availability, and declared output contract.
|
||||
- `InspectPrompt` is a point-in-time structural check. It does not load a JSON
|
||||
Schema, resolve a profile, or freeze later execution.
|
||||
- Explicit profile overrides and relevant prompt defaults are checked with
|
||||
`InspectProfile` before collection when application policy requires them.
|
||||
- `InspectProfile` is also point-in-time and does not check credential values.
|
||||
- Successful `PrepareExecution`, not inspection, is the per-run authority for
|
||||
loaded schema, rendered content, frozen inputs, effective settings, and
|
||||
durable execution provenance.
|
||||
|
||||
### Prompt Input
|
||||
|
||||
- Promptkit will continue to receive the curated `data_package` produced by
|
||||
- Promptkit receives only the curated `data_package` produced by
|
||||
`internal/promptinput`.
|
||||
- Weatherreporter will serialize the data package once, atomically persist
|
||||
those exact bytes, and supply the same bytes as a Promptkit inline artifact.
|
||||
- The managed data-package path may be supplied as non-secret artifact
|
||||
provenance.
|
||||
- Weatherreporter will not delegate unrestricted path loading to Promptkit's
|
||||
- Weatherreporter serializes the package once, atomically persists those exact
|
||||
bytes, and supplies the same bytes with a Promptkit inline artifact.
|
||||
- The managed data-package path may be supplied as non-secret provenance
|
||||
through the inline artifact URI.
|
||||
- Weatherreporter does not delegate unrestricted path loading to Promptkit's
|
||||
default file artifact reader.
|
||||
- The same immutable Promptkit request will be used for preparation and
|
||||
execution so the preflight and run inputs cannot diverge.
|
||||
- Prompt inspection and adapter tests verify that `data_package` is required
|
||||
and declared with the chosen YAML media type.
|
||||
|
||||
### Preparation And Execution
|
||||
### Prepared Execution
|
||||
|
||||
- Promptkit `Prepare` replaces the current Scriptorium render preflight.
|
||||
- Promptkit `Run` performs both Markdown and structured generated-text
|
||||
execution.
|
||||
- Promptkit basic validation will be used where appropriate for direct
|
||||
Markdown output.
|
||||
- Promptkit JSON Schema validation provides the provider-facing and first
|
||||
structured-output check for generated-text reports.
|
||||
- `Engine.PrepareExecution` replaces Scriptorium render preflight.
|
||||
- Weatherreporter obtains `PreparedExecution.Details`, maps a safe subset into
|
||||
its own preparation record, and persists that record before calling
|
||||
`Engine.RunPrepared`.
|
||||
- `RunPrepared` executes the frozen prompt, profile, schema, inputs, rendered
|
||||
messages, target, and validation resources retained by the handle.
|
||||
- Every acquired handle is followed immediately by `defer handle.Discard()`.
|
||||
Discard is safe after execution and releases unused private execution state.
|
||||
- Handles remain adapter-local, engine-bound, one-shot, in-process values.
|
||||
They are never serialized, persisted, copied into app contracts, or treated
|
||||
as restartable jobs.
|
||||
- Preparation and execution use independent contexts. Execution receives the
|
||||
active report workflow context.
|
||||
- Capacity is not reserved during preparation. Capacity rejection can
|
||||
therefore occur after a preparation record has been persisted.
|
||||
- `RunPrepared` consumes the handle on success and every operational failure.
|
||||
- Preparation details remain available from the adapter after execution or
|
||||
discard, but rendered messages are not copied into routine durable state.
|
||||
- Promptkit execution timing excludes preparation and consumer-held delay.
|
||||
Weatherreporter records preparation timing and execution timing separately.
|
||||
|
||||
### Execution And Validation
|
||||
|
||||
- All four operational reports use Promptkit JSON Schema output validation.
|
||||
- A completed Promptkit validation rejection returns a `RunResult`; the
|
||||
adapter retains raw output and bounded validation details before failing the
|
||||
report.
|
||||
- An operational generation or validation error returns no partial
|
||||
`RunResult`.
|
||||
- Weatherreporter's `internal/generatedtext` validation remains the final
|
||||
report-specific domain boundary.
|
||||
- Weatherreporter's `internal/reporttemplate` remains responsible for
|
||||
generated-text Markdown rendering.
|
||||
- Weatherreporter will atomically persist Promptkit output rather than asking
|
||||
the dependency to write managed report files.
|
||||
- The migration will not rely on Promptkit output repair. Promptkit v0.3.0's
|
||||
public engine validates in a single pass even when a prompt declares repair
|
||||
attempts.
|
||||
report-specific decode and domain boundary.
|
||||
- Weatherreporter's `internal/reporttemplate` remains responsible for managed
|
||||
Markdown rendering.
|
||||
- Weatherreporter atomically persists Promptkit raw output and later artifacts
|
||||
rather than asking Promptkit to choose managed filesystem paths.
|
||||
- No Promptkit output-repair behavior is assumed or requested.
|
||||
|
||||
## Durable Artifacts And Observability
|
||||
|
||||
Status: Accepted design constraints; unimplemented.
|
||||
|
||||
Routine durable artifacts should retain useful non-secret provenance without
|
||||
persisting full rendered prompts by default.
|
||||
Routine durable state retains useful non-secret provenance without persisting
|
||||
full rendered prompts.
|
||||
|
||||
The preparation record should contain:
|
||||
The preparation record contains:
|
||||
|
||||
- prompt ID and version;
|
||||
- prompt ID and exact version;
|
||||
- prompt definition hash;
|
||||
- rendered prompt hash;
|
||||
- input hashes;
|
||||
- selected profile and backend identity;
|
||||
- effective model identity;
|
||||
- output contract summary; and
|
||||
- preparation timing.
|
||||
- output contract summary;
|
||||
- preparation start, end, and duration; and
|
||||
- the path of the exact persisted data package.
|
||||
|
||||
The execution record and run metadata should contain, when available:
|
||||
The execution record and run metadata contain, when available:
|
||||
|
||||
- Promptkit run ID;
|
||||
- prompt ID, version, and hashes;
|
||||
@@ -206,122 +354,162 @@ The execution record and run metadata should contain, when available:
|
||||
- selected profile, backend, and model identity;
|
||||
- generated-content hash;
|
||||
- token usage;
|
||||
- start, end, and duration;
|
||||
- execution start, end, and duration;
|
||||
- validation status and bounded diagnostics; and
|
||||
- the path of any separately persisted raw generated output.
|
||||
- paths of separately persisted raw output, normalized generated text, render
|
||||
context, managed Markdown, and other artifacts reached by the workflow.
|
||||
|
||||
Provider endpoints, full effective model parameter maps, rendered messages,
|
||||
schema bodies, data-package contents, and generated content do not belong in
|
||||
routine metadata or CLI summaries.
|
||||
|
||||
Rendered messages and other content-rich preparation or response diagnostics
|
||||
will be available only through an explicitly enabled debug mechanism. Debug
|
||||
artifacts must be documented as potentially sensitive, must not contain
|
||||
credentials, and must have a clear operator-owned retention policy.
|
||||
are available only when the operator supplies
|
||||
`--llm-debug-dir <path>` to a single-report or batch command.
|
||||
|
||||
- There is initially no persistent YAML setting for debug capture.
|
||||
- The debug root is validated or created before weather collection or provider
|
||||
work. A requested destination that cannot be secured or written is an error.
|
||||
- Artifacts are grouped beneath
|
||||
`<path>/<report-id>/<valid-date>/<run-id>/`.
|
||||
- Directories and files use owner-only permissions and atomic writes.
|
||||
- Debug artifacts may contain rendered messages and content-rich preparation
|
||||
or response diagnostics, but never credentials.
|
||||
- The debug path appears in command output only when debug capture is enabled;
|
||||
it is not added to routine durable metadata.
|
||||
- Debug artifacts are not cache or comparison inputs. Their retention is owned
|
||||
by the operator who selected the directory.
|
||||
|
||||
### Artifact Identities And Versions
|
||||
|
||||
Weatherreporter replaces Scriptorium-specific artifact identities rather than
|
||||
reusing names whose meanings have changed:
|
||||
|
||||
- `PromptPreparationArtifact` uses schema version
|
||||
`weatherreporter.prompt_preparation.v1`, is written as
|
||||
`prompt_preparation.<runID>.json`, and is referenced by
|
||||
`preparationPath`.
|
||||
- `PromptExecutionArtifact` uses schema version
|
||||
`weatherreporter.prompt_execution.v1`, is written as
|
||||
`prompt_execution.<runID>.json`, and is referenced by `executionPath`.
|
||||
- Run metadata advances to `weatherreporter.metadata.v2` and uses those new
|
||||
path fields.
|
||||
|
||||
Preparation files remain beneath the existing configurable `preflight/`
|
||||
directory, and execution files remain beneath the existing `snapshots/` tree.
|
||||
The stable physical grouping limits deployment disruption without preserving
|
||||
misleading Scriptorium-era filenames or field names. Raw generated output,
|
||||
normalized generated text, render context, managed Markdown, and other
|
||||
artifacts whose meanings have not changed retain their existing names and
|
||||
locations.
|
||||
|
||||
Run inspection remains able to read `weatherreporter.metadata.v1` and its
|
||||
legacy `preflightPath` and `generatedTextResultPath` references. New runs write
|
||||
only the v2 metadata and new artifact names; Weatherreporter does not
|
||||
dual-write deprecated aliases. CLI summary fields adopt `preparationPath` and
|
||||
`executionPath` as an explicit, documented contract change.
|
||||
|
||||
## Failure Contract
|
||||
|
||||
Status: Accepted design constraints; unimplemented.
|
||||
|
||||
Promptkit returns a completed `RunResult` for output-validation failure but no
|
||||
partial result for operational preparation or execution errors. Weatherreporter
|
||||
will preserve that distinction.
|
||||
|
||||
- A preparation failure produces a redacted weatherreporter-owned failure
|
||||
receipt with report, RunID, prompt, stage, timing, and classified error
|
||||
context. It does not fabricate a Promptkit preparation result.
|
||||
context. It does not fabricate Promptkit preparation details.
|
||||
- An operational execution failure retains the successful preparation record
|
||||
and adds a redacted execution failure receipt. No partial Promptkit result or
|
||||
model output is invented.
|
||||
- A Promptkit validation failure retains the returned result, raw generated
|
||||
- A Promptkit validation rejection retains the returned result, raw generated
|
||||
output, validation details, and safe provenance before the report fails.
|
||||
- A later weatherreporter generated-text decode, domain-validation, or template
|
||||
failure retains every raw and validated artifact reached before that stage.
|
||||
- Context cancellation takes precedence when the caller context is canceled.
|
||||
- Promptkit capacity rejection maps to a weatherreporter-owned error category.
|
||||
It is an operational report failure, not invalid model output.
|
||||
- A later generated-text decode, domain-validation, or template failure
|
||||
retains every raw and validated artifact reached before that stage.
|
||||
- Caller cancellation takes precedence when the active workflow context is
|
||||
canceled.
|
||||
- `promptkit.CapacityError` is recognized with `errors.As`; its backend ID is
|
||||
copied into a weatherreporter-owned capacity error while
|
||||
`ErrCapacityExceeded` remains the classification.
|
||||
- Capacity rejection is an operational report failure, not invalid model
|
||||
output, and does not trigger an automatic retry.
|
||||
- Other Promptkit public error identities are translated into the narrow
|
||||
weatherreporter error categories needed by CLI, metadata, and batch
|
||||
behavior. Diagnostic prose is not parsed as a contract.
|
||||
- Single-report commands return the classified failure with available
|
||||
inspectable paths.
|
||||
- Batch runs continue independent later reports under the existing batch
|
||||
failure policy.
|
||||
- The migration adds no automatic retries. Any future retry policy belongs to
|
||||
app orchestration, not the Promptkit adapter.
|
||||
- Any future retry policy belongs to app orchestration, not the adapter.
|
||||
|
||||
## Compatibility Requirements
|
||||
|
||||
Status: Accepted design constraints; unimplemented.
|
||||
|
||||
- Report IDs, prompt IDs, report selection, valid periods, artifact grouping,
|
||||
output names, and distributor bundle behavior remain stable.
|
||||
- Daily, Today, Tomorrow, and Hourly report IDs, prompt IDs, valid periods,
|
||||
artifact grouping, output names, and Distributor bundle behavior remain
|
||||
stable.
|
||||
- Morning and evening batch collection, planning, ordering, and continuation
|
||||
behavior remains stable.
|
||||
- Module snapshot and Recent Changes behavior remains deterministic.
|
||||
- Promptkit receives only the existing curated prompt-input boundary.
|
||||
- Generated reports continue to use the managed Markdown path as the
|
||||
distributor upload source.
|
||||
- Managed Markdown remains the Distributor upload source.
|
||||
- RunID lookup and inspection remain available for successful and failed runs.
|
||||
- Existing managed state paths remain stable where their meaning is unchanged.
|
||||
Scriptorium-specific artifact names or schemas may change when retaining
|
||||
them would misrepresent the new contract.
|
||||
- Any artifact or metadata schema change is explicit, documented, and covered
|
||||
by state and inspection tests.
|
||||
- Existing managed paths remain stable where their meaning is unchanged.
|
||||
Scriptorium-specific artifact names or schemas change when retaining them
|
||||
would misrepresent the Promptkit contract.
|
||||
- Existing v1 run metadata and referenced artifacts remain inspectable after
|
||||
the migration. New runs use the v2 metadata and Promptkit-era artifact
|
||||
identities without dual-writing deprecated aliases.
|
||||
- Artifact or metadata schema changes are explicit, documented, and covered by
|
||||
state and inspection tests.
|
||||
- Prompt or generated content is not added to routine logs or CLI summaries.
|
||||
- Tests do not require live Promptkit providers or credentials.
|
||||
- Tests do not require live providers or credentials.
|
||||
- Removing incomplete three-day, weekend, and storm surfaces is documented as
|
||||
correction of an unfinished product boundary, not as successful Promptkit
|
||||
migration of those reports.
|
||||
|
||||
## Verification And Completion Criteria
|
||||
|
||||
Status: Proposed completion criteria for the unimplemented migration.
|
||||
Status: Accepted completion criteria for the unimplemented migration.
|
||||
|
||||
The migration is complete when:
|
||||
|
||||
- all seven reports prepare and execute through Promptkit using embedded
|
||||
report-owned assets;
|
||||
- direct-Markdown and generated-text-template paths have deterministic offline
|
||||
adapter and app-level coverage;
|
||||
- preparation, provider failure, capacity rejection, cancellation, timeout,
|
||||
Promptkit validation failure, generated-text validation failure, template
|
||||
failure, and successful generation preserve their specified artifacts;
|
||||
- morning and evening batches construct one shared engine and preserve current
|
||||
- the four operational reports inspect, prepare, and execute through Promptkit
|
||||
`v0.4.0` using embedded report-owned assets;
|
||||
- every report uses exact prompt version `1.0.0`, requires the YAML
|
||||
`data_package`, and declares the expected JSON Schema output contract;
|
||||
- prepared execution persists a safe preparation record before provider work
|
||||
and executes the same frozen snapshot;
|
||||
- deterministic offline adapter and app tests cover success, preparation
|
||||
failure, credential revalidation, capacity rejection, cancellation, timeout,
|
||||
generation failure, Promptkit validation rejection, generated-text domain
|
||||
failure, template failure, and handle discard;
|
||||
- morning and evening batches construct one engine and preserve current
|
||||
collection, planning, ordering, continuation, output, and notification
|
||||
behavior;
|
||||
- the temporary corpus has been reconciled into one runtime prompt/schema
|
||||
source without duplicate provider-facing schemas;
|
||||
- configuration examples load and contain no Scriptorium fields;
|
||||
- CLI summaries and inspection commands expose the new artifact contract
|
||||
without Promptkit dependency types;
|
||||
- CLI summaries and inspection commands expose the new project-owned artifact
|
||||
contract without Promptkit types;
|
||||
- Scriptorium code, configuration, tests, and runtime documentation have been
|
||||
removed;
|
||||
- incomplete three-day, weekend, and storm commands, registry entries, tests,
|
||||
and current-behavior documentation have been removed or moved to roadmap
|
||||
scope;
|
||||
- non-roadmap documentation describes only the implemented Promptkit
|
||||
integration;
|
||||
- `go test ./...`, CLI help validation, and `git diff --check` pass; and
|
||||
- no committed `go.work`, local `replace`, live-provider test, or secret-bearing
|
||||
fixture remains.
|
||||
- `go test ./...`, required focused or race-enabled checks, CLI help
|
||||
validation, and `git diff --check` pass; and
|
||||
- no committed `go.work`, local `replace`, live-provider test, or
|
||||
secret-bearing fixture remains.
|
||||
|
||||
Fixture-based comparison with the current Scriptorium behavior is sufficient
|
||||
for migration verification. A production-selectable dual-run period is not
|
||||
required because model calls are nondeterministic, costly, and difficult to
|
||||
compare meaningfully.
|
||||
Fixture-based comparison with prior Scriptorium behavior is sufficient.
|
||||
Production dual-run is not required because model calls are nondeterministic,
|
||||
costly, and difficult to compare meaningfully.
|
||||
|
||||
## External Prerequisite
|
||||
## Decision Status
|
||||
|
||||
Status: Required and unimplemented.
|
||||
Status: Decision-complete.
|
||||
|
||||
Before implementing the embedded asset stage, the current Scriptorium prompt
|
||||
corpus must be made available in this repository. It should include the seven
|
||||
prompt definitions, referenced content files, private response schemas,
|
||||
relevant default-profile declarations, and any shared prompt fragments needed
|
||||
to reproduce current report behavior.
|
||||
|
||||
## Open Questions
|
||||
|
||||
Status: Open; these require decisions before implementation.
|
||||
|
||||
- What exact `promptkit.*` configuration fields should replace the current
|
||||
Scriptorium fields, including the name and precedence of an optional explicit
|
||||
profile override?
|
||||
- Should weatherreporter expose Promptkit's conventional `local` backend
|
||||
registration as a narrow configuration feature, or rely initially on
|
||||
built-in and endpoint-only profiles?
|
||||
- Should report definitions store an explicit Promptkit prompt version, or
|
||||
should each embedded prompt ID be required to have exactly one version?
|
||||
- What CLI or configuration control enables sensitive prompt/response debug
|
||||
artifacts, and where should those artifacts live?
|
||||
- What final names and schema versions should replace the
|
||||
Scriptorium-specific preflight and run-result artifacts while balancing
|
||||
semantic clarity with existing state-path compatibility?
|
||||
The roadmap has no remaining open product or architecture questions. Any later
|
||||
question that changes migration scope, policy, or the desired end state must be
|
||||
resolved here rather than implicitly in code.
|
||||
|
||||
Reference in New Issue
Block a user