Clean up roadmap and troubleshooting documentation
This commit is contained in:
@@ -20,6 +20,5 @@ Configure a Weather API endpoint first; see the
|
|||||||
- [CLI reference](docs/cli.md)
|
- [CLI reference](docs/cli.md)
|
||||||
- [Configuration reference](docs/config.md)
|
- [Configuration reference](docs/config.md)
|
||||||
- [Operations guide](docs/operations.md)
|
- [Operations guide](docs/operations.md)
|
||||||
- [Troubleshooting](docs/troubleshooting.md)
|
|
||||||
- [Development guide](docs/development.md)
|
- [Development guide](docs/development.md)
|
||||||
- [Architecture policy](docs/policy/architecture.md)
|
- [Architecture policy](docs/policy/architecture.md)
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ boundaries and invariants.
|
|||||||
| Recent Changes comparison | [Changes internals](internal/changes.md) and [operations guide](operations.md) | The internal guide owns structured comparison; operations owns user-visible artifact behavior. |
|
| Recent Changes comparison | [Changes internals](internal/changes.md) and [operations guide](operations.md) | The internal guide owns structured comparison; operations owns user-visible artifact behavior. |
|
||||||
| Prompt execution, profiles, prompt inputs, or result handling | `internal/promptexec`, the Promptkit adapter, and [prompt-input internals](internal/prompt-input.md) | These separate the executor contract and input construction. |
|
| Prompt execution, profiles, prompt inputs, or result handling | `internal/promptexec`, the Promptkit adapter, and [prompt-input internals](internal/prompt-input.md) | These separate the executor contract and input construction. |
|
||||||
| Generated-text schemas, validation, render contexts, templates, or Markdown rendering | [Generated-text internals](internal/generatedtext.md), [report-template internals](internal/reporttemplate.md), and [report template guide](templates.md) | These own structured text, renderer implementation, and the maintainer-facing template surface. |
|
| Generated-text schemas, validation, render contexts, templates, or Markdown rendering | [Generated-text internals](internal/generatedtext.md), [report-template internals](internal/reporttemplate.md), and [report template guide](templates.md) | These own structured text, renderer implementation, and the maintainer-facing template surface. |
|
||||||
| Workspace paths, metadata, atomic persistence, lookup, inspection, or recovery | [State internals](internal/state.md), [operations guide](operations.md), and [troubleshooting guide](troubleshooting.md) | These separate implementation, operator workflows, and symptom-based recovery. |
|
| Workspace paths, metadata, atomic persistence, lookup, inspection, diagnosis, or recovery | [State internals](internal/state.md) and [operations guide](operations.md) | These separate implementation behavior from operator workflows, diagnosis, and recovery. |
|
||||||
| Distributor bundles, uploads, notification artifacts, or failures | [Distributor adapter internals](internal/distributor-adapter.md), [Distributor integration contracts](integrations/distributor/), and [operations guide](operations.md) | These separate adapter behavior, external contracts, and operational lifecycle. |
|
| Distributor bundles, uploads, notification artifacts, or failures | [Distributor adapter internals](internal/distributor-adapter.md), [Distributor integration contracts](integrations/distributor/), and [operations guide](operations.md) | These separate adapter behavior, external contracts, and operational lifecycle. |
|
||||||
| Maintained example configuration | [Configuration reference](config.md) and files under `examples/` | The reference owns field meaning; examples own complete copyable files. |
|
| Maintained example configuration | [Configuration reference](config.md) and files under `examples/` | The reference owns field meaning; examples own complete copyable files. |
|
||||||
| Release preparation, tagging, publication, or verification | [Release procedure](release.md) | It owns version selection, release-note preparation, candidate validation, tag publication, CI behavior, and post-publication checks. |
|
| Release preparation, tagging, publication, or verification | [Release procedure](release.md) | It owns version selection, release-note preparation, candidate validation, tag publication, CI behavior, and post-publication checks. |
|
||||||
|
|||||||
@@ -3,8 +3,7 @@
|
|||||||
This guide covers normal operation, managed workspace state, inspection,
|
This guide covers normal operation, managed workspace state, inspection,
|
||||||
recovery, and operational caveats. See the [CLI reference](cli.md) for complete
|
recovery, and operational caveats. See the [CLI reference](cli.md) for complete
|
||||||
command syntax and the [configuration reference](config.md) for fields,
|
command syntax and the [configuration reference](config.md) for fields,
|
||||||
defaults, and notification templates. For symptom-based diagnosis, see
|
defaults, and notification templates.
|
||||||
[Troubleshooting](troubleshooting.md).
|
|
||||||
|
|
||||||
## Normal Operation
|
## Normal Operation
|
||||||
|
|
||||||
@@ -19,8 +18,8 @@ persists the module snapshot and prompt data package, records Promptkit
|
|||||||
preparation provenance before provider execution, then persists raw output and
|
preparation provenance before provider execution, then persists raw output and
|
||||||
execution provenance, validates the structured generated text, and renders the
|
execution provenance, validates the structured generated text, and renders the
|
||||||
managed Markdown report from the validated text and deterministic values.
|
managed Markdown report from the validated text and deterministic values.
|
||||||
The current receipts are transitional workspace state, not a cross-version
|
These receipts describe the current workspace implementation; they are not a
|
||||||
profile-provenance contract.
|
cross-version profile-provenance contract.
|
||||||
|
|
||||||
The managed report and its final metadata are saved before single-report
|
The managed report and its final metadata are saved before single-report
|
||||||
Distributor notification is attempted. `--out` writes an extra operator copy;
|
Distributor notification is attempted. `--out` writes an extra operator copy;
|
||||||
@@ -48,8 +47,9 @@ Prompt inspection occurs before weather collection. A malformed profile file,
|
|||||||
missing required credential, or unsupported selected backend stops the command
|
missing required credential, or unsupported selected backend stops the command
|
||||||
before collection. A reachable profile can still fail later if its local model
|
before collection. A reachable profile can still fail later if its local model
|
||||||
endpoint is unavailable; Weatherreporter does not switch to a remote profile.
|
endpoint is unavailable; Weatherreporter does not switch to a remote profile.
|
||||||
See the [configuration reference](config.md) for field definitions and the
|
See the [configuration reference](config.md) for field definitions. For a
|
||||||
[troubleshooting guide](troubleshooting.md) for recovery.
|
provider failure, start with the command error and enable secure prompt debug
|
||||||
|
capture when content-rich diagnostics are required.
|
||||||
|
|
||||||
## Optional Prompt Debug Capture
|
## Optional Prompt Debug Capture
|
||||||
|
|
||||||
@@ -113,9 +113,9 @@ The generated-text and render-context artifacts are written for every completed
|
|||||||
single-report generation.
|
single-report generation.
|
||||||
A report's metadata links the module snapshot, data package, preparation and
|
A report's metadata links the module snapshot, data package, preparation and
|
||||||
execution receipts, managed report, generated-text artifacts, and any available single-report
|
execution receipts, managed report, generated-text artifacts, and any available single-report
|
||||||
notification artifact. These current-version receipts remain transitional; use
|
notification artifact. These current-version receipts are implementation
|
||||||
the active command's classified error and explicit secure debug capture for
|
details rather than durable interfaces. Use the active command's error and
|
||||||
prompt diagnosis rather than relying on them as a durable interface. Batch
|
explicit secure debug capture for prompt diagnosis. Batch
|
||||||
notification artifacts are separate batch-level records under
|
notification artifacts are separate batch-level records under
|
||||||
`notifications/batches`.
|
`notifications/batches`.
|
||||||
|
|
||||||
@@ -191,7 +191,7 @@ remain available where they can be safely persisted.
|
|||||||
- A batch notification failure preserves each report's artifacts and adds the
|
- A batch notification failure preserves each report's artifacts and adds the
|
||||||
top-level batch notification artifact.
|
top-level batch notification artifact.
|
||||||
|
|
||||||
Use the action summary and its classified error first. For prompt or provider
|
Use the action summary and its error first. For prompt or provider
|
||||||
diagnosis, prefer an explicitly enabled secure debug capture; current-version
|
diagnosis, prefer an explicitly enabled secure debug capture; current-version
|
||||||
receipt paths may provide supplemental context when available. For a batch
|
receipt paths may provide supplemental context when available. For a batch
|
||||||
failure, inspect the summary first, then inspect the affected report RunIDs or
|
failure, inspect the summary first, then inspect the affected report RunIDs or
|
||||||
|
|||||||
@@ -86,8 +86,7 @@ mechanisms, not secret values.
|
|||||||
| Release notes | `docs/releases/` | One versioned, changelog-style summary for each release, including compatibility and operator action. The file at the tagged commit supplies the corresponding Gitea release body. | Current CLI, configuration, operations, integration, architecture, and internal contracts; release procedure; implementation plans. |
|
| Release notes | `docs/releases/` | One versioned, changelog-style summary for each release, including compatibility and operator action. The file at the tagged commit supplies the corresponding Gitea release body. | Current CLI, configuration, operations, integration, architecture, and internal contracts; release procedure; implementation plans. |
|
||||||
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, stdout and stderr behavior, summaries, and exit behavior. | Configuration field definitions, complete operating procedures, runtime filesystem layout, and command implementation. |
|
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, stdout and stderr behavior, summaries, and exit behavior. | Configuration field definitions, complete operating procedures, runtime filesystem layout, and command implementation. |
|
||||||
| Configuration contract | `docs/config.md` | Discovery and precedence, fields, defaults, secrets, validation rules, and user-selectable values. | Complete example files, CLI syntax, runtime state lifecycle, and loading implementation. |
|
| Configuration contract | `docs/config.md` | Discovery and precedence, fields, defaults, secrets, validation rules, and user-selectable values. | Complete example files, CLI syntax, runtime state lifecycle, and loading implementation. |
|
||||||
| Operations | `docs/operations.md` | Normal workflows, physical workspace layout, artifacts and metadata, inspection, notification behavior, recovery, cleanup, permissions, and operational caveats. | Complete CLI syntax, configuration field definitions, logical external contracts, and implementation mechanics. |
|
| Operations | `docs/operations.md` | Normal workflows, physical workspace layout, artifacts and metadata, inspection, notification behavior, diagnosis, recovery, cleanup, permissions, and operational caveats. | Complete CLI syntax, configuration field definitions, logical external contracts, and implementation mechanics. |
|
||||||
| Troubleshooting | `docs/troubleshooting.md` | Recurring symptoms, likely causes, diagnostic steps, safe fixes, and links to normal-operation references. | Complete command and configuration references, routine operating procedures, and implementation detail. |
|
|
||||||
| Report template surface | `docs/templates.md` | Implemented template files and partials, render-context fields, editing rules, and maintainer-facing template examples. | Weather derivation, module implementation, generated-text validation internals, and operator procedures. |
|
| Report template surface | `docs/templates.md` | Implemented template files and partials, render-context fields, editing rules, and maintainer-facing template examples. | Weather derivation, module implementation, generated-text validation internals, and operator procedures. |
|
||||||
| External and durable integration contracts | `docs/integrations/` | Weather API, Promptkit, Distributor, external formats and protocols, durable logical paths and schemas, compatibility behavior, and upstream or downstream responsibilities. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, and configuration defaults. |
|
| External and durable integration contracts | `docs/integrations/` | Weather API, Promptkit, Distributor, external formats and protocols, durable logical paths and schemas, compatibility behavior, and upstream or downstream responsibilities. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, and configuration defaults. |
|
||||||
| Internal subsystem behavior | `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, user-facing contracts, external schemas, operator procedures, and future package plans. |
|
| Internal subsystem behavior | `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, user-facing contracts, external schemas, operator procedures, and future package plans. |
|
||||||
@@ -110,13 +109,12 @@ structure and invariants. Focused internal documents own implementation
|
|||||||
behavior. These documents may link to one another but must not maintain
|
behavior. These documents may link to one another but must not maintain
|
||||||
parallel package or behavior references.
|
parallel package or behavior references.
|
||||||
|
|
||||||
### Commands, Configuration, Operations, And Troubleshooting
|
### Commands, Configuration, And Operations
|
||||||
|
|
||||||
CLI documentation answers how to invoke Weatherreporter and what its command
|
CLI documentation answers how to invoke Weatherreporter and what its command
|
||||||
interface does. Configuration documentation answers what settings mean.
|
interface does. Configuration documentation answers what settings mean.
|
||||||
Operations answers what happens to runtime state and how to operate or recover
|
Operations answers what happens to runtime state and how to operate or recover
|
||||||
the application. Troubleshooting starts from a symptom and leads to diagnosis
|
the application, including diagnosis and safe responses to runtime failures.
|
||||||
and a safe fix.
|
|
||||||
|
|
||||||
When a workflow crosses these topics, place the complete procedure with the
|
When a workflow crosses these topics, place the complete procedure with the
|
||||||
document that owns the task and link to the other contracts. Do not duplicate
|
document that owns the task and link to the other contracts. Do not duplicate
|
||||||
|
|||||||
@@ -1,283 +0,0 @@
|
|||||||
# Domain-Specific Prompt Profiles Roadmap
|
|
||||||
|
|
||||||
Status: Implemented.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Weatherreporter should provide stable, domain-specific Promptkit profile IDs
|
|
||||||
that express the relative resource and analysis needs of its report products.
|
|
||||||
These logical profiles should give each report an appropriate default while
|
|
||||||
allowing operators to replace any definition through the existing configured
|
|
||||||
profile source.
|
|
||||||
|
|
||||||
This roadmap records the scope, policy, and implemented end state. The
|
|
||||||
companion [implementation plan](implementation.md) records the ordered work
|
|
||||||
and verification used to reach it.
|
|
||||||
|
|
||||||
## User Intent
|
|
||||||
|
|
||||||
The feature is intended to provide three related benefits:
|
|
||||||
|
|
||||||
- frequent reports can use a cost-effective model by default;
|
|
||||||
- reports needing broader synthesis can select a stronger default without
|
|
||||||
forcing the same cost on every invocation; and
|
|
||||||
- an installation can map a stable Weatherreporter profile ID to a model on a
|
|
||||||
local network endpoint without modifying embedded prompts or application
|
|
||||||
code.
|
|
||||||
|
|
||||||
`weather-light` describes the profile's intended resource tier, not a latency
|
|
||||||
guarantee. A locally hosted lightweight model may still generate slowly on the
|
|
||||||
available hardware.
|
|
||||||
|
|
||||||
## Pre-Implementation Baseline
|
|
||||||
|
|
||||||
Before implementation, Daily, Today, Tomorrow, and Hourly each declared
|
|
||||||
Promptkit's `gemini-flash-latest` profile as their prompt default. The optional
|
|
||||||
`promptkit.profile` setting overrode that default for every selected report in
|
|
||||||
an invocation.
|
|
||||||
|
|
||||||
Weatherreporter accepted either `promptkit.profile_file` or
|
|
||||||
`promptkit.profile_dir` and passed that source to Promptkit. A matching external
|
|
||||||
profile could override a Promptkit built-in profile, and the configured local
|
|
||||||
backend could support profiles that select `backend: local`. Endpoint-only
|
|
||||||
OpenAI-compatible profiles could also provide their own endpoint.
|
|
||||||
|
|
||||||
Weatherreporter did not own or embed execution profiles. Promptkit v0.5.0
|
|
||||||
provided the fallback-profile layer used to add them without changing the
|
|
||||||
existing operator-source precedence.
|
|
||||||
|
|
||||||
## Prerequisite
|
|
||||||
|
|
||||||
Promptkit v0.5.0 provides the application fallback profile capability defined
|
|
||||||
in the companion
|
|
||||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
|
|
||||||
Weatherreporter now depends on that tagged release. The dependency upgrade has
|
|
||||||
passed the repository test suite and an operator smoke test. Weatherreporter
|
|
||||||
must continue to use only Promptkit's public API rather than depending on its
|
|
||||||
internal packages or reproducing its profile repository behavior.
|
|
||||||
|
|
||||||
## Implemented End State
|
|
||||||
|
|
||||||
Weatherreporter embeds usable definitions for these exact logical profile IDs:
|
|
||||||
|
|
||||||
- `weather-light`
|
|
||||||
- `weather-balanced`
|
|
||||||
- `weather-deep`
|
|
||||||
|
|
||||||
The profiles are Weatherreporter-owned assets and remain behind the existing
|
|
||||||
Promptkit adapter boundary. Prompt definitions select the logical IDs, while
|
|
||||||
Promptkit resolves the effective backend, endpoint, model, and generation
|
|
||||||
settings.
|
|
||||||
|
|
||||||
An operator can place a profile with the same ID in `profile_file` or
|
|
||||||
`profile_dir`. The operator definition completely replaces the embedded
|
|
||||||
Weatherreporter definition for that ID. If the external source does not contain
|
|
||||||
the selected ID, lookup falls through to Weatherreporter's embedded profile and
|
|
||||||
then to Promptkit's built-in catalog.
|
|
||||||
|
|
||||||
The existing global `promptkit.profile` setting remains available as an
|
|
||||||
explicit all-report override. No new configuration field is required for the
|
|
||||||
initial feature.
|
|
||||||
|
|
||||||
## Profile Catalog And Report Assignment
|
|
||||||
|
|
||||||
| Profile | Meaning | Initial default reports |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly |
|
|
||||||
| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow |
|
|
||||||
| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially |
|
|
||||||
|
|
||||||
The initial assignment recognizes that Weatherreporter's deterministic modules
|
|
||||||
already perform most weather selection and calculation. A higher-capability
|
|
||||||
model should not become a default merely because it is available. Moving an
|
|
||||||
existing report to `weather-deep` requires evidence that the stronger tier
|
|
||||||
materially improves supported reasoning or output quality.
|
|
||||||
|
|
||||||
The three profile IDs are capability policies, not permanent aliases for one
|
|
||||||
provider or model family. Their embedded definitions may change in a future
|
|
||||||
Weatherreporter release, with the change disclosed through normal release and
|
|
||||||
compatibility documentation.
|
|
||||||
|
|
||||||
## Selection And Definition Precedence
|
|
||||||
|
|
||||||
Profile ID selection and profile definition lookup are separate decisions.
|
|
||||||
|
|
||||||
Weatherreporter selects the profile ID in this order:
|
|
||||||
|
|
||||||
1. nonblank `promptkit.profile`; or
|
|
||||||
2. the exact prompt version's `default_profile`.
|
|
||||||
|
|
||||||
Promptkit then resolves the selected profile definition in this order:
|
|
||||||
|
|
||||||
1. explicit in-memory profiles, when used by an embedding consumer or test;
|
|
||||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
|
||||||
3. Weatherreporter's embedded fallback profiles; and
|
|
||||||
4. Promptkit's embedded built-in profiles.
|
|
||||||
|
|
||||||
A higher-precedence source falls through only when the selected ID is absent.
|
|
||||||
A matching but malformed operator profile fails before weather collection and
|
|
||||||
must not silently use the embedded definition.
|
|
||||||
|
|
||||||
## Local Endpoint Experience
|
|
||||||
|
|
||||||
An operator should be able to override `weather-light` with an endpoint-only
|
|
||||||
profile whose model name is understood by the local OpenAI-compatible server.
|
|
||||||
This path does not require a separate Weatherreporter local-backend setting.
|
|
||||||
|
|
||||||
Alternatively, an override may select `backend: local`; in that case the
|
|
||||||
existing `promptkit.local.endpoint` and concurrency settings continue to own
|
|
||||||
the shared local backend definition.
|
|
||||||
|
|
||||||
The selected local profile is deterministic configuration, not a preference
|
|
||||||
hint. Weatherreporter will not probe for a local model and will not
|
|
||||||
automatically fall back to a remote or paid profile when the endpoint is
|
|
||||||
unavailable. The failure remains visible and attributable to the selected
|
|
||||||
profile.
|
|
||||||
|
|
||||||
## Embedded Profile Policy
|
|
||||||
|
|
||||||
Each embedded profile must be a complete, valid Promptkit profile and must be
|
|
||||||
usable in a default installation with the documented credential mechanism. The
|
|
||||||
initial embedded profiles are expected to use Promptkit's `openrouter` backend,
|
|
||||||
allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment
|
|
||||||
variable without embedding credentials.
|
|
||||||
|
|
||||||
Embedded definitions should include only settings that are intentional for the
|
|
||||||
selected model and supported by its backend. Avoid incidental generation
|
|
||||||
parameters that reduce portability or trigger provider-specific request
|
|
||||||
failures without a demonstrated quality benefit.
|
|
||||||
|
|
||||||
The initial profile definitions are:
|
|
||||||
|
|
||||||
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
|
|
||||||
| --- | --- | --- | --- | --- |
|
|
||||||
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
|
|
||||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
|
||||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
|
||||||
|
|
||||||
These settings deliberately match the corresponding Promptkit v0.5.0
|
|
||||||
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
|
|
||||||
part of each OpenRouter rolling-alias identifier. The profiles do not set
|
|
||||||
temperature, `top_p`, or output-token limits; omission preserves provider
|
|
||||||
defaults and avoids unsupported incidental parameters.
|
|
||||||
|
|
||||||
## Prompt And Active Execution Contract
|
|
||||||
|
|
||||||
Changing a prompt's `default_profile` is a material prompt-definition change.
|
|
||||||
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
|
|
||||||
defaults are introduced. Prompt content and generated-text schemas need not
|
|
||||||
change solely for this feature.
|
|
||||||
|
|
||||||
Prompt inspection must continue to occur before weather collection. It should
|
|
||||||
report the selected logical profile ID and the resolved backend and model
|
|
||||||
without exposing endpoints or credentials.
|
|
||||||
|
|
||||||
The active execution contract should retain both the selected logical profile
|
|
||||||
identity and the resolved backend and model through inspection, preparation,
|
|
||||||
execution, errors, and command results where those values are already exposed.
|
|
||||||
This feature must not add a new durable-provenance or cross-version artifact
|
|
||||||
contract.
|
|
||||||
|
|
||||||
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
|
|
||||||
prompt provenance a non-goal. Existing workspace persistence may remain while
|
|
||||||
this feature lands, but it is transitional behavior and must not be expanded or
|
|
||||||
treated as part of the profile feature's desired end state. Prompt preparation
|
|
||||||
and execution artifacts written at `1.0.1` are not required to remain readable
|
|
||||||
after the prompt definitions advance to `1.1.0`.
|
|
||||||
|
|
||||||
## Evaluation Policy
|
|
||||||
|
|
||||||
Concrete model assignments should be evaluated with representative,
|
|
||||||
secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should
|
|
||||||
consider:
|
|
||||||
|
|
||||||
- strict-schema success rate;
|
|
||||||
- unsupported or invented weather claims;
|
|
||||||
- precipitation-timing accuracy and empty-string behavior;
|
|
||||||
- correct use of deterministic hazards, periods, and uncertainty;
|
|
||||||
- summary and forecast-discussion usefulness;
|
|
||||||
- generation latency;
|
|
||||||
- token use and provider cost; and
|
|
||||||
- behavior through a representative local OpenAI-compatible endpoint.
|
|
||||||
|
|
||||||
The purpose is to choose an appropriate default for each tier, not to add a
|
|
||||||
permanent benchmark framework or live-provider requirement to the ordinary
|
|
||||||
test suite. Repository tests remain offline and deterministic.
|
|
||||||
|
|
||||||
## Implemented Scope
|
|
||||||
|
|
||||||
The completed feature includes:
|
|
||||||
|
|
||||||
- Weatherreporter-owned embedded profile assets for all three logical IDs;
|
|
||||||
- Promptkit adapter wiring that supplies those assets as the application
|
|
||||||
fallback profile source;
|
|
||||||
- per-prompt default-profile assignments matching the catalog above;
|
|
||||||
- an exact prompt-version update for the changed definitions;
|
|
||||||
- preservation of the global profile override;
|
|
||||||
- same-ID override behavior through both supported external profile-source
|
|
||||||
forms;
|
|
||||||
- local-backend and endpoint-only override coverage;
|
|
||||||
- fail-fast inspection of missing, malformed, or unusable selected profiles;
|
|
||||||
- offline tests for selection, source precedence, effective model inspection,
|
|
||||||
batch reuse, and active execution behavior;
|
|
||||||
- maintained operator examples for overriding `weather-light` locally; and
|
|
||||||
- updates to the canonical configuration, Promptkit integration, report
|
|
||||||
registry, operations, troubleshooting, internal adapter, and release
|
|
||||||
documentation as applicable when implementation lands.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
The feature does not include:
|
|
||||||
|
|
||||||
- automatic discovery, health checking, or benchmarking of local endpoints;
|
|
||||||
- implicit failover between local and remote profiles;
|
|
||||||
- retries with a more expensive tier after provider or validation failure;
|
|
||||||
- per-report profile configuration fields outside prompt defaults;
|
|
||||||
- profile inheritance, aliases, or field-level merging;
|
|
||||||
- runtime model selection based on weather severity, token count, or report
|
|
||||||
content;
|
|
||||||
- moving Weatherreporter profile policy into Promptkit's built-in catalog;
|
|
||||||
- exposing Promptkit types outside the adapter boundary; or
|
|
||||||
- making live provider calls part of the default repository test suite.
|
|
||||||
|
|
||||||
## Compatibility And Operational Policy
|
|
||||||
|
|
||||||
Existing configurations with a nonblank `promptkit.profile` retain their
|
|
||||||
all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and
|
|
||||||
credential configuration fields retain their meanings.
|
|
||||||
|
|
||||||
Configurations that rely on the omitted profile setting will intentionally
|
|
||||||
observe new per-report defaults. This is a user-visible model-selection and
|
|
||||||
cost change and must be called out in release notes. Operators who require the
|
|
||||||
old all-report model can preserve it by setting an explicit global profile.
|
|
||||||
|
|
||||||
The prompt-version transition does not provide backward compatibility for
|
|
||||||
historical prompt preparation or execution artifacts. This is consistent with
|
|
||||||
the accepted ephemeral-state direction; the profile feature does not otherwise
|
|
||||||
redesign or remove the current workspace layout.
|
|
||||||
|
|
||||||
An external same-ID override is an operator-owned compatibility commitment.
|
|
||||||
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
|
|
||||||
silently merge an operator file.
|
|
||||||
|
|
||||||
## Completion Record
|
|
||||||
|
|
||||||
The following conditions are satisfied:
|
|
||||||
|
|
||||||
- a tagged Promptkit dependency supports the required fallback layer;
|
|
||||||
- every operational prompt selects its assigned logical profile at exact
|
|
||||||
version `1.1.0`;
|
|
||||||
- all three embedded profiles inspect successfully without an external profile
|
|
||||||
source;
|
|
||||||
- configured same-ID definitions override embedded definitions through both
|
|
||||||
`profile_file` and `profile_dir`;
|
|
||||||
- an invalid matching external definition fails without fallback;
|
|
||||||
- `weather-light` can resolve through an endpoint-only or configured-local
|
|
||||||
override without requiring code or prompt changes;
|
|
||||||
- global `promptkit.profile` still overrides every report in an invocation;
|
|
||||||
- active inspection and execution preserve the selected logical profile and
|
|
||||||
effective model through the project-owned execution contract;
|
|
||||||
- morning and evening batch preflight deduplicates inspection of shared
|
|
||||||
effective profile IDs as it does today;
|
|
||||||
- the default test suite remains offline and deterministic; and
|
|
||||||
- implemented behavior is documented by its canonical current-state owners.
|
|
||||||
@@ -247,7 +247,7 @@ The completed feature includes:
|
|||||||
of unbounded state growth;
|
of unbounded state growth;
|
||||||
- an Accepted ADR documenting the architectural decision; and
|
- an Accepted ADR documenting the architectural decision; and
|
||||||
- updates to canonical architecture, CLI, operations, configuration,
|
- updates to canonical architecture, CLI, operations, configuration,
|
||||||
troubleshooting, integration, internal, testing, and release documentation
|
integration, internal, testing, and release documentation
|
||||||
where their contracts change.
|
where their contracts change.
|
||||||
|
|
||||||
## Non-Goals
|
## Non-Goals
|
||||||
@@ -296,16 +296,11 @@ shares mutable filesystem state.
|
|||||||
|
|
||||||
## Relationship To Domain-Specific Profiles
|
## Relationship To Domain-Specific Profiles
|
||||||
|
|
||||||
The domain-specific profile feature can be implemented before this refactor,
|
The implemented domain-specific profiles do not add historical compatibility
|
||||||
but it should not add new historical compatibility or durable-provenance
|
or durable-provenance commitments. Profile inspection, selection, override
|
||||||
commitments. Profile inspection, selection, override precedence, and effective
|
precedence, and effective model resolution remain active-workflow behavior and
|
||||||
model resolution remain active-workflow behavior and survive the state change.
|
survive the state change. Existing prompt artifacts need not remain readable
|
||||||
|
after this refactor and must not constrain the target architecture.
|
||||||
The domain-profile roadmap and implementation plan should acknowledge that
|
|
||||||
prompt artifacts from version `1.0.1` need not remain readable after prompts
|
|
||||||
advance to `1.1.0`. Existing state persistence may remain temporarily while
|
|
||||||
the profile feature lands, but it should not be expanded or treated as the
|
|
||||||
target architecture.
|
|
||||||
|
|
||||||
## Completion Criteria
|
## Completion Criteria
|
||||||
|
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ Possible direction:
|
|||||||
|
|
||||||
1. Detect candidate storm events from alerts, forecast discussion, weather
|
1. Detect candidate storm events from alerts, forecast discussion, weather
|
||||||
story context, hourly thresholds, and material forecast changes.
|
story context, hourly thresholds, and material forecast changes.
|
||||||
2. Evaluate candidates through Scriptorium or another narrow evaluator adapter.
|
2. Evaluate candidates through Promptkit or another narrow evaluator adapter.
|
||||||
3. Persist storm lifecycle state.
|
3. Persist storm lifecycle state.
|
||||||
4. Generate or update a storm report only when a meaningful event is present.
|
4. Generate or update a storm report only when a meaningful event is present.
|
||||||
5. Suppress ordinary low-impact thunder or rain chances.
|
5. Suppress ordinary low-impact thunder or rain chances.
|
||||||
|
|||||||
@@ -1,599 +0,0 @@
|
|||||||
# Domain-Specific Prompt Profiles Implementation Plan
|
|
||||||
|
|
||||||
Status: Stages 1–7 completed; remediation Stage 8 ready.
|
|
||||||
|
|
||||||
## Purpose And Authority
|
|
||||||
|
|
||||||
This document records the implementation and post-implementation remediation
|
|
||||||
of the
|
|
||||||
[domain-specific prompt profiles roadmap](domain-profiles.md). The roadmap is
|
|
||||||
authoritative for scope, user intent, policy choices, and the intended end
|
|
||||||
state. This plan records implementation sequence, verification, audit findings,
|
|
||||||
and exit gates.
|
|
||||||
|
|
||||||
This plan follows the repository's
|
|
||||||
[architecture](../policy/architecture.md),
|
|
||||||
[documentation](../policy/documentation.md), and
|
|
||||||
[testing](../policy/testing.md) policies.
|
|
||||||
|
|
||||||
## Completed Prerequisite
|
|
||||||
|
|
||||||
Weatherreporter is already pinned to Promptkit v0.5.0. That release provides
|
|
||||||
the public `WithFallbackProfileFS` option and the required precedence across
|
|
||||||
inspection, preparation, and execution. The dependency upgrade passed
|
|
||||||
Weatherreporter's full offline test suite, race-enabled suite, CLI help check,
|
|
||||||
and an operator `generate hourly` smoke test. Do not repeat or replace the
|
|
||||||
dependency upgrade as part of these stages.
|
|
||||||
|
|
||||||
## Locked Product Decisions
|
|
||||||
|
|
||||||
Implement these exact Weatherreporter-owned profiles:
|
|
||||||
|
|
||||||
| Profile ID | Backend | Model | Reasoning effort | Timeout | Service tier |
|
|
||||||
| --- | --- | --- | --- | --- | --- |
|
|
||||||
| `weather-light` | `openrouter` | `deepseek/deepseek-v4-flash` | Omitted | 180 seconds | `flex` |
|
|
||||||
| `weather-balanced` | `openrouter` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
|
||||||
| `weather-deep` | `openrouter` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
|
||||||
|
|
||||||
Assign Hourly to `weather-light`; assign Daily, Today, and Tomorrow to
|
|
||||||
`weather-balanced`; assign no report to `weather-deep` initially. Advance all
|
|
||||||
four prompt definitions and matching report-registry entries from `1.0.1` to
|
|
||||||
`1.1.0` when their defaults change.
|
|
||||||
|
|
||||||
The leading `~` in the Gemini and Claude model IDs is required and denotes an
|
|
||||||
OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a
|
|
||||||
dated model version. Do not add temperature, `top_p`, maximum-token, endpoint,
|
|
||||||
or credential fields to the embedded definitions.
|
|
||||||
|
|
||||||
Prompt preparation and execution artifacts written at `1.0.1` are not required
|
|
||||||
to remain readable after the transition to `1.1.0`. Do not add a migration,
|
|
||||||
compatibility shim, or weaker historical-artifact validation for this feature.
|
|
||||||
|
|
||||||
Definition lookup must remain:
|
|
||||||
|
|
||||||
1. explicit Promptkit in-memory profiles used by tests or an embedding
|
|
||||||
consumer;
|
|
||||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
|
||||||
3. Weatherreporter's embedded fallback profiles; and
|
|
||||||
4. Promptkit's built-in catalog.
|
|
||||||
|
|
||||||
Selection remains a separate concern: a nonblank global `promptkit.profile`
|
|
||||||
selects the profile for every report in the invocation; otherwise the exact
|
|
||||||
prompt definition's `default_profile` selects it. A malformed matching
|
|
||||||
higher-precedence profile is an error and never falls through.
|
|
||||||
|
|
||||||
## Continuing Invariants
|
|
||||||
|
|
||||||
- Keep all Promptkit types and mechanics inside
|
|
||||||
`internal/adapters/promptkit`, its focused tests, and asset contract tests.
|
|
||||||
- Keep prompt inspection before weather collection and provider work.
|
|
||||||
- Keep one Promptkit engine per command action and one shared engine across a
|
|
||||||
sequential batch.
|
|
||||||
- Preserve logical profile ID and effective backend/model information through
|
|
||||||
active inspection and execution where the project-owned contract already
|
|
||||||
exposes it. Do not add new durable-provenance fields or compatibility
|
|
||||||
guarantees.
|
|
||||||
- Leave existing workspace persistence behavior otherwise unchanged. The
|
|
||||||
accepted [ephemeral-state roadmap](ephemeral-state.md) owns its future
|
|
||||||
removal and must not be partially implemented here.
|
|
||||||
- Do not expose endpoints, credentials, rendered messages, schemas, request
|
|
||||||
bodies, response bodies, or complete parameter maps through ordinary errors,
|
|
||||||
logs, summaries, or state.
|
|
||||||
- Keep the default suite deterministic, offline, and credential-free.
|
|
||||||
- Do not add endpoint discovery, health probing, provider failover, retries at
|
|
||||||
a more expensive tier, profile merging, per-report configuration fields, or
|
|
||||||
severity-driven model selection.
|
|
||||||
- Update canonical current-state documentation only in the stage where the
|
|
||||||
corresponding behavior becomes implemented.
|
|
||||||
- Run `git diff --check` before completing every stage.
|
|
||||||
|
|
||||||
## Stage 1: Add The Embedded Weather Profile Catalog
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Create one repository-owned, embedded profile source containing exactly the
|
|
||||||
three locked logical profiles.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Add strict YAML profile assets beneath `internal/promptassets` using the
|
|
||||||
exact IDs and definitions in this plan.
|
|
||||||
2. Extend `internal/promptassets` with a narrowly named accessor that returns
|
|
||||||
the embedded profile `fs.FS`. Follow the existing prompt and schema asset
|
|
||||||
pattern without exposing Promptkit types from the package.
|
|
||||||
3. Keep profile filenames and embed layout simple and deterministic. Do not
|
|
||||||
duplicate Promptkit's built-in directory taxonomy unless the application
|
|
||||||
assets require it.
|
|
||||||
4. Validate the assets through Promptkit's public engine/profile inspection
|
|
||||||
surface rather than adding a second YAML parser or a Weatherreporter-owned
|
|
||||||
profile representation.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Extend the asset contract tests to assert exactly the three logical IDs,
|
|
||||||
their exact effective model IDs, and the intentional parameters.
|
|
||||||
- Prove all three profiles inspect successfully offline when supplied as a
|
|
||||||
fallback source and no operator source is present.
|
|
||||||
- Assert that the catalog contains no endpoints, credentials, temperature,
|
|
||||||
`top_p`, or maximum-token settings.
|
|
||||||
- Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./internal/promptassets
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
The embedded catalog is complete, strictly valid, safe, and independently
|
|
||||||
inspectable through Promptkit v0.5.0's public API.
|
|
||||||
|
|
||||||
## Stage 2: Wire Fallback Resolution And Protect Precedence
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Supply the embedded catalog through Promptkit's application fallback layer
|
|
||||||
without changing existing operator configuration or application boundaries.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Add `promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".")` to
|
|
||||||
normal adapter engine construction.
|
|
||||||
2. Preserve existing `profile_file`, `profile_dir`, configured local backend,
|
|
||||||
timeout, prompt filesystem, schema filesystem, and test-option behavior.
|
|
||||||
3. Ensure ordinary production construction and the adapter's test
|
|
||||||
construction path exercise the same fallback wiring. Test-only explicit
|
|
||||||
profiles may retain Promptkit's documented highest precedence.
|
|
||||||
4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays,
|
|
||||||
existence checks, YAML parsing, or merge behavior to Weatherreporter.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- At the adapter boundary, prove fallback-only inspection of all three
|
|
||||||
Weatherreporter profiles.
|
|
||||||
- Prove same-ID overrides through both configured `profile_file` and
|
|
||||||
`profile_dir`, including resolution of the override's effective backend and
|
|
||||||
model.
|
|
||||||
- Prove an absent operator match falls through, while a malformed matching
|
|
||||||
operator definition fails without using the embedded profile.
|
|
||||||
- Prove a selected Promptkit built-in that is absent from both higher layers
|
|
||||||
still resolves.
|
|
||||||
- Prove an explicit in-memory test profile retains highest precedence.
|
|
||||||
- Cover both local override forms required by the roadmap: an endpoint-only
|
|
||||||
OpenAI-compatible `weather-light` profile and a `backend: local` profile
|
|
||||||
using the configured local endpoint. No test may contact either endpoint.
|
|
||||||
- Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./internal/adapters/promptkit
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Inspection and prepared execution use Promptkit's exact four-layer precedence,
|
|
||||||
operator errors remain visible, and local overrides require no prompt or code
|
|
||||||
changes.
|
|
||||||
|
|
||||||
## Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Move operational prompts from provider-oriented defaults to the three-tier
|
|
||||||
Weatherreporter policy with an exact, synchronized version transition.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Change Hourly's `default_profile` to `weather-light`.
|
|
||||||
2. Change Daily, Today, and Tomorrow to `weather-balanced`.
|
|
||||||
3. Advance the exact version in all four prompt YAML assets from `1.0.1` to
|
|
||||||
`1.1.0` without changing prompt text or generated-text schemas solely for
|
|
||||||
this feature.
|
|
||||||
4. Advance the four matching report-registry prompt versions to `1.1.0` in the
|
|
||||||
same change. Keep prompt IDs, report IDs, modules, periods, templates, and
|
|
||||||
output contracts unchanged.
|
|
||||||
5. Update fixtures and expectations that intentionally assert the current
|
|
||||||
prompt contract. Do not rewrite historical fixture versions or weaken tests
|
|
||||||
that protect actual compatibility.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Update asset and report-registry contract tests to require exact version
|
|
||||||
`1.1.0` and the report-to-profile assignments locked in this plan.
|
|
||||||
- Inspect every exact prompt version through the real embedded prompt, schema,
|
|
||||||
and fallback-profile filesystems.
|
|
||||||
- Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve
|
|
||||||
Gemini Flash Latest, and `weather-deep` remains inspectable but unassigned.
|
|
||||||
- Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Every operational prompt and registry definition agrees on exact version
|
|
||||||
`1.1.0`, selects its intended logical tier, and resolves its expected effective
|
|
||||||
model offline.
|
|
||||||
|
|
||||||
## Stage 4: Verify Application Selection And Batch Reuse
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Protect the assembled application behavior created by the new defaults and
|
|
||||||
confirm that logical identity is not lost during active effective-model
|
|
||||||
resolution.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Preserve the current pre-collection inspection order and fail-fast behavior
|
|
||||||
for missing credentials, unknown profiles, malformed profiles, and unusable
|
|
||||||
backends.
|
|
||||||
2. Preserve the global `promptkit.profile` all-report override. Do not add a
|
|
||||||
second override mechanism or report-specific configuration fields.
|
|
||||||
3. Preserve batch preflight deduplication by selected effective profile ID:
|
|
||||||
Today and Tomorrow in the same batch should inspect their shared
|
|
||||||
`weather-balanced` selection once.
|
|
||||||
4. Preserve the selected logical profile ID and resolved backend/model through
|
|
||||||
active inspection, preparation, and execution using the existing
|
|
||||||
project-owned contract. Do not add state fields, expand persisted parameter
|
|
||||||
detail, or create a new historical compatibility guarantee.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Add or update representative app tests for default Hourly and day-scale
|
|
||||||
selection, a global-profile override, and a morning/evening batch sharing
|
|
||||||
`weather-balanced`.
|
|
||||||
- Assert inspection completes before weather collection and provider
|
|
||||||
generation, including malformed same-ID operator overrides.
|
|
||||||
- Assert active inspection and execution expose the logical profile ID and
|
|
||||||
effective model for both embedded and overridden profiles.
|
|
||||||
- Assert endpoints and credentials remain absent from errors, summaries,
|
|
||||||
normal logs, and ordinary state.
|
|
||||||
- Use project-owned executor fakes or Promptkit provider fakes; do not make live
|
|
||||||
provider calls.
|
|
||||||
- Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./internal/app ./internal/cli
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Single-report and batch workflows select the intended tier, retain existing
|
|
||||||
override and preflight behavior, deduplicate shared batch inspection, and
|
|
||||||
preserve safe logical and effective model information during active execution
|
|
||||||
without adding a durable-provenance contract.
|
|
||||||
|
|
||||||
## Stage 5: Publish Canonical Operator And Maintainer Documentation
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Document the implemented feature once in each canonical owner and provide one
|
|
||||||
maintained, copyable local override example.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Update `docs/config.md` to explain global profile selection versus
|
|
||||||
`profile_file`/`profile_dir` definition lookup and link to the maintained
|
|
||||||
example. Keep the field reference in this canonical document.
|
|
||||||
2. Update the Promptkit integration document with the logical profile catalog,
|
|
||||||
source precedence, exact prompt-version relationship, and safe active
|
|
||||||
inspection and execution contract. Avoid restating complete configuration
|
|
||||||
syntax or presenting transitional persistence as the target architecture.
|
|
||||||
3. Update the report-registry, Promptkit adapter, app-orchestration, and state
|
|
||||||
internal documents only where their implemented contracts changed.
|
|
||||||
4. Update `docs/operations.md` with the normal local-override workflow and
|
|
||||||
`docs/troubleshooting.md` with malformed override, unavailable local
|
|
||||||
endpoint, missing credential, and unexpected effective-model diagnostics.
|
|
||||||
5. Add or update one secret-free file under `examples/` showing a
|
|
||||||
`weather-light` override for a local OpenAI-compatible endpoint. Choose one
|
|
||||||
supported form as the complete example and mention the other form only in
|
|
||||||
its canonical reference.
|
|
||||||
6. Update the architecture policy only if implementation changed a normative
|
|
||||||
boundary or invariant. Do not add future behavior to current-state docs.
|
|
||||||
7. Keep the feature roadmap and this plan in their pre-implementation statuses
|
|
||||||
until the final repository gate passes. Do not create release notes before
|
|
||||||
a release version is chosen.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Verify every changed repository-relative link and every profile/model ID.
|
|
||||||
- Validate maintained YAML examples through the same strict configuration or
|
|
||||||
Promptkit profile path used by production where practical.
|
|
||||||
- Run the focused tests that own any executable examples, followed by:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Users, operators, and maintainers can discover the tier defaults, precedence,
|
|
||||||
global override, local override, and failure behavior without duplicated or
|
|
||||||
future-state documentation.
|
|
||||||
|
|
||||||
## Stage 6: Complete Repository Verification And Roadmap Handoff
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Demonstrate that the complete feature is coherent, offline-testable, and ready
|
|
||||||
for review and a later release decision.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Review the complete diff against the roadmap, this plan, and all three
|
|
||||||
policy documents. Remove stale identifiers, temporary helpers, redundant
|
|
||||||
tests, and documentation duplication.
|
|
||||||
2. Confirm `go.mod` and `go.sum` retain tagged Promptkit v0.5.0 without a local
|
|
||||||
replacement or dependency drift.
|
|
||||||
3. Confirm only the four supported report products exist and no retired report
|
|
||||||
surfaces were reintroduced.
|
|
||||||
4. Confirm the roadmap's completion criteria one by one. Change its status to
|
|
||||||
implemented and this plan's status to completed only after every criterion
|
|
||||||
and command below passes.
|
|
||||||
5. Do not require a live provider for completion. If credentials and network
|
|
||||||
access are deliberately supplied by an operator, record live smoke results
|
|
||||||
separately as release-candidate evidence rather than adding them to the
|
|
||||||
default suite.
|
|
||||||
|
|
||||||
### Verification
|
|
||||||
|
|
||||||
Run `gofmt -w` on every changed Go file, then run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./...
|
|
||||||
go test -race ./...
|
|
||||||
go run ./cmd/weatherreporter --help
|
|
||||||
git diff --check
|
|
||||||
git status --short
|
|
||||||
```
|
|
||||||
|
|
||||||
Also inspect all three logical profiles through the application's normal
|
|
||||||
preflight path using offline provider doubles, including one same-ID local
|
|
||||||
override and one explicit global override.
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
All roadmap completion criteria are satisfied, all verification commands pass,
|
|
||||||
the working tree contains only intentional changes, and the canonical
|
|
||||||
documentation describes the implemented state. The feature is ready for code
|
|
||||||
review and release preparation.
|
|
||||||
|
|
||||||
## Post-Implementation Review
|
|
||||||
|
|
||||||
Stages 1–6 implemented the intended production behavior and passed their
|
|
||||||
offline verification gates. A subsequent review found no high-severity runtime
|
|
||||||
defect, but identified three test-quality issues and one remaining validation
|
|
||||||
obligation:
|
|
||||||
|
|
||||||
- one app test asserted durable preparation and execution artifact provenance,
|
|
||||||
contrary to the active-execution boundary and accepted ephemeral-state
|
|
||||||
direction;
|
|
||||||
- an adapter-package test depended upward on app orchestration and duplicated
|
|
||||||
test ownership;
|
|
||||||
- embedded fallback profiles were inspected but not exercised through one
|
|
||||||
prepared execution with a provider fake; and
|
|
||||||
- the roadmap's representative model-evaluation policy had no recorded
|
|
||||||
evidence.
|
|
||||||
|
|
||||||
Stages 7 and 8 address those findings without changing the profile catalog,
|
|
||||||
selection precedence, report assignments, prompt content, generated-text
|
|
||||||
schemas, or default offline test contract.
|
|
||||||
|
|
||||||
## Stage 7: Correct Test Ownership And Fallback Execution Coverage
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Remove accidental durable-state and cross-layer test commitments while adding
|
|
||||||
one focused offline execution test for the embedded fallback path.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Rewrite `TestGenerateDetailedPreservesSelectedProfileThroughExecution` so
|
|
||||||
it protects active workflow behavior only:
|
|
||||||
|
|
||||||
- retain the Hourly default, day-scale default, and global-override cases;
|
|
||||||
- assert the profile ID sent in `promptexec.ExecuteRequest`;
|
|
||||||
- have the executor fake record the preparation and execution values it
|
|
||||||
emits, then assert their logical profile ID and effective backend/model;
|
|
||||||
- do not load preparation, execution, or metadata files to establish a
|
|
||||||
durable profile-provenance contract; and
|
|
||||||
- remove artifact-content scans whose fake inputs cannot contain an endpoint
|
|
||||||
or credential.
|
|
||||||
|
|
||||||
2. Preserve meaningful safety coverage at the boundary that can expose the
|
|
||||||
sensitive value:
|
|
||||||
|
|
||||||
- retain adapter mapping coverage proving an endpoint from a real Promptkit
|
|
||||||
profile does not enter `promptexec.ProfileInspection`;
|
|
||||||
- retain app error coverage proving dependency errors containing an endpoint
|
|
||||||
or credential are replaced by a bounded classified error; and
|
|
||||||
- do not add profile endpoints or credentials to project-owned execution
|
|
||||||
types merely to make a leakage test possible.
|
|
||||||
|
|
||||||
3. Remove `internal/app`, app configuration, and report-registry dependencies
|
|
||||||
from `internal/adapters/promptkit/adapter_test.go`. Move the assembled
|
|
||||||
application-preflight test to a new app-owned external integration test,
|
|
||||||
such as `internal/app/prompt_profile_integration_test.go` with package
|
|
||||||
`app_test`:
|
|
||||||
|
|
||||||
- construct the real Promptkit adapter through its public `New` function;
|
|
||||||
- call the public app prompt-inspection operation;
|
|
||||||
- supply a deterministic credential lookup rather than reading the process
|
|
||||||
environment; and
|
|
||||||
- cover Hourly, one representative day-scale default, the explicit
|
|
||||||
`weather-deep` global override, and a same-ID endpoint-only
|
|
||||||
`weather-light` override. The asset contract tests already own the exact
|
|
||||||
mapping for all four prompts, so the integration test need not repeat all
|
|
||||||
four.
|
|
||||||
|
|
||||||
4. Add one adapter-owned, offline fake-client execution test using the real
|
|
||||||
embedded Hourly prompt at `1.1.0` and selected profile `weather-light`.
|
|
||||||
Execute through the normal prepared adapter path and assert:
|
|
||||||
|
|
||||||
- the preparation callback runs before the fake provider;
|
|
||||||
- preparation and execution report logical profile `weather-light`, backend
|
|
||||||
`openrouter`, and model `deepseek/deepseek-v4-flash`;
|
|
||||||
- the fake provider request targets `deepseek/deepseek-v4-flash`; and
|
|
||||||
- schema validation completes without contacting a live service.
|
|
||||||
|
|
||||||
One execution case is sufficient because Promptkit owns uniform source
|
|
||||||
precedence and the adapter's inspection tests already cover fallback,
|
|
||||||
operator file, operator directory, built-in, and explicit in-memory layers.
|
|
||||||
|
|
||||||
5. Reconcile the profile-related current-state documentation:
|
|
||||||
|
|
||||||
- it may accurately describe fields present in current preparation and
|
|
||||||
execution receipts;
|
|
||||||
- it must not promise cross-version readability or characterize those
|
|
||||||
receipts as the profile feature's durable target architecture; and
|
|
||||||
- troubleshooting should prefer active command errors and explicit secure
|
|
||||||
debug capture, mentioning current-version receipts only as transitional
|
|
||||||
state if they remain useful before the ephemeral-state refactor.
|
|
||||||
|
|
||||||
6. Do not change production profile resolution, prompt definitions, state
|
|
||||||
schemas, artifact validators, or the ephemeral-state roadmap in this stage.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli
|
|
||||||
go test -count=1 -race ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli
|
|
||||||
go test -count=1 ./...
|
|
||||||
go vet ./...
|
|
||||||
go run ./cmd/weatherreporter --help
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
Review the changed tests against the testing policy and confirm that adapter
|
|
||||||
tests own adapter behavior, app tests own orchestration, and state tests remain
|
|
||||||
the sole owner of durable artifact format and validation details.
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Active profile selection and effective-model propagation remain protected
|
|
||||||
without adding a durable-provenance commitment; the adapter test package no
|
|
||||||
longer imports the app layer; one embedded fallback profile completes prepared
|
|
||||||
execution through a provider fake; and every required check passes offline.
|
|
||||||
|
|
||||||
## Stage 8: Evaluate The Initial Model Ladder
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Produce explicit release-candidate evidence that the selected models are
|
|
||||||
acceptable for their intended report tiers and that a representative local
|
|
||||||
override provides the promised operator experience.
|
|
||||||
|
|
||||||
This is an opt-in evaluation stage, not an ordinary automated-test stage. It
|
|
||||||
requires operator-approved provider credentials, network access, and a local
|
|
||||||
OpenAI-compatible endpoint. Do not mark it complete when those prerequisites
|
|
||||||
are unavailable; report the missing prerequisite instead.
|
|
||||||
|
|
||||||
### Corpus
|
|
||||||
|
|
||||||
Use four representative, secret-free YAML data packages: one each for Daily,
|
|
||||||
Today, Tomorrow, and Hourly. The set must include at least one package with
|
|
||||||
precipitation windows and at least one with none. Remove precise private
|
|
||||||
location identifiers or other operationally sensitive values without changing
|
|
||||||
the meteorological relationships being evaluated.
|
|
||||||
|
|
||||||
Record a SHA-256 hash and a short, non-sensitive description for each package.
|
|
||||||
Do not commit full packages or generated prose unless the user separately
|
|
||||||
approves them as repository fixtures.
|
|
||||||
|
|
||||||
### Execution Matrix
|
|
||||||
|
|
||||||
Run these six evaluations from the exact package bytes:
|
|
||||||
|
|
||||||
| Case | Package | Profile |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| Hourly default | Hourly | `weather-light` |
|
|
||||||
| Daily default | Daily | `weather-balanced` |
|
|
||||||
| Today default | Today | `weather-balanced` |
|
|
||||||
| Tomorrow default | Tomorrow | `weather-balanced` |
|
|
||||||
| Deep comparison | The same Daily package used above | `weather-deep` |
|
|
||||||
| Local override | The same Hourly package used above | operator-defined `weather-light` endpoint profile |
|
|
||||||
|
|
||||||
After the successful local-override case, stop or deliberately address an
|
|
||||||
unavailable test endpoint and repeat it as a negative control. Confirm that the
|
|
||||||
request fails visibly and does not call or select an embedded remote profile.
|
|
||||||
This negative control is not an additional quality-evaluation case.
|
|
||||||
|
|
||||||
Use a temporary, untracked evaluation harness beneath the module when exact
|
|
||||||
package replay is needed. It should call the existing Promptkit adapter and
|
|
||||||
project-owned execution contract rather than duplicate prompt loading,
|
|
||||||
rendering, or schema validation. Remove the harness and all unapproved raw
|
|
||||||
outputs before completing the stage. Never print or record credentials.
|
|
||||||
|
|
||||||
### Evaluation Record
|
|
||||||
|
|
||||||
Add a concise `## Evaluation Record` section to
|
|
||||||
`docs/roadmap/domain-profiles.md`. For every case, record:
|
|
||||||
|
|
||||||
- evaluation date, logical profile, effective backend, and exact model
|
|
||||||
reported by execution;
|
|
||||||
- corpus hash, validation outcome, latency, prompt/completion/total token use,
|
|
||||||
and provider-reported or contemporaneously calculated cost;
|
|
||||||
- whether every generated claim is supported by the deterministic package;
|
|
||||||
- whether hazards, periods, uncertainty, and precipitation timing are used
|
|
||||||
correctly;
|
|
||||||
- whether `precipitation_timing` is exactly an empty string for the no-window
|
|
||||||
case;
|
|
||||||
- a short usefulness assessment for summary and forecast discussion; and
|
|
||||||
- any provider, alias, or local-endpoint caveat observed.
|
|
||||||
|
|
||||||
Do not include credentials, endpoints, complete effective parameter maps,
|
|
||||||
full data packages, rendered prompts, or full generated responses in the
|
|
||||||
record. The secure debug directory may be used temporarily for operator review
|
|
||||||
and remains operator-managed.
|
|
||||||
|
|
||||||
### Acceptance Rules
|
|
||||||
|
|
||||||
- Every case must complete strict JSON Schema validation without repair.
|
|
||||||
- Generated prose must contain no material unsupported weather claim or
|
|
||||||
contradiction of deterministic hazards, periods, or uncertainty.
|
|
||||||
- Precipitation timing must agree with the deterministic windows and use the
|
|
||||||
required empty-string representation when no window exists.
|
|
||||||
- The local override must select the operator model without modifying a prompt
|
|
||||||
or application code and must not fall back to a remote profile when the local
|
|
||||||
endpoint is unavailable.
|
|
||||||
- Latency, tokens, and cost must be recorded, but this initial evaluation does
|
|
||||||
not impose an invented numeric threshold. The operator decides whether the
|
|
||||||
observed tradeoff remains acceptable for the named tier.
|
|
||||||
- If a default case fails schema or factual acceptance, do not weaken the
|
|
||||||
schema or prompt to accommodate the model. Reopen the concrete model or
|
|
||||||
profile-setting decision in the feature roadmap and leave this stage
|
|
||||||
incomplete.
|
|
||||||
|
|
||||||
### Verification
|
|
||||||
|
|
||||||
After removing temporary evaluation material, run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test -count=1 ./...
|
|
||||||
git diff --check
|
|
||||||
git status --short
|
|
||||||
```
|
|
||||||
|
|
||||||
Confirm that the only intended repository change from this stage is the
|
|
||||||
concise evaluation record and any roadmap status correction required by its
|
|
||||||
result. Do not add live credentials, provider-dependent tests, a permanent
|
|
||||||
benchmark framework, or release notes before a release version is selected.
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
All six cases satisfy the acceptance rules, the roadmap contains concise and
|
|
||||||
safe evaluation evidence, no temporary corpus or response material remains in
|
|
||||||
the repository, and the default suite remains offline. Set this plan back to
|
|
||||||
`Status: Completed` only after both Stages 7 and 8 have passed.
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
None. The model identifiers, profile settings, report assignments, version
|
|
||||||
transition, precedence, compatibility behavior, test boundaries, and
|
|
||||||
documentation ownership are decision-complete.
|
|
||||||
@@ -1,169 +0,0 @@
|
|||||||
# Promptkit Feature Request: Application Fallback Profiles
|
|
||||||
|
|
||||||
Status: Implemented upstream in Promptkit v0.5.0.
|
|
||||||
|
|
||||||
Promptkit v0.5.0 resolved this request with the public
|
|
||||||
`WithFallbackProfileFS` engine option and the precedence and error semantics
|
|
||||||
specified below. This document is retained as the downstream rationale for
|
|
||||||
the capability.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Promptkit should allow a consuming application to supply an embedded fallback
|
|
||||||
profile source that sits below operator-configured profiles and above
|
|
||||||
Promptkit's own built-in profile catalog.
|
|
||||||
|
|
||||||
This capability would let an application publish stable, domain-specific
|
|
||||||
profile IDs with useful defaults while preserving Promptkit's existing
|
|
||||||
operator-override behavior. The capability must remain application-neutral;
|
|
||||||
Promptkit should provide the source layer but should not own downstream profile
|
|
||||||
names, model assignments, or configuration policy.
|
|
||||||
|
|
||||||
## Downstream Use Case
|
|
||||||
|
|
||||||
Weatherreporter wants to embed profiles such as `weather-light`,
|
|
||||||
`weather-balanced`, and `weather-deep`. Report prompts would select those
|
|
||||||
logical profiles instead of naming provider- or model-specific Promptkit
|
|
||||||
profiles directly.
|
|
||||||
|
|
||||||
An installation could then place a profile with the same ID in its configured
|
|
||||||
profile directory. For example, a local `weather-light` definition could point
|
|
||||||
to an OpenAI-compatible endpoint on the deployment network. When no operator
|
|
||||||
definition exists, Weatherreporter's embedded definition would keep the
|
|
||||||
application usable without additional profile files.
|
|
||||||
|
|
||||||
This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want
|
|
||||||
application-owned execution tiers or workload-specific defaults without
|
|
||||||
adding domain-specific profiles to Promptkit's general built-in catalog.
|
|
||||||
|
|
||||||
## Current Constraint
|
|
||||||
|
|
||||||
Promptkit currently resolves matching profile IDs in this order:
|
|
||||||
|
|
||||||
1. in-memory profiles supplied through `WithProfiles`;
|
|
||||||
2. one configured profile file, `fs.FS`, or directory source; and
|
|
||||||
3. Promptkit's embedded built-in profiles.
|
|
||||||
|
|
||||||
These layers do not express the desired application-default relationship:
|
|
||||||
|
|
||||||
- `WithProfiles` has higher precedence than the configured source, so it would
|
|
||||||
prevent an operator file from overriding an application profile with the
|
|
||||||
same ID.
|
|
||||||
- `WithProfileFS` can hold embedded application assets, but it occupies the
|
|
||||||
configured-source layer and therefore replaces rather than sits beneath a
|
|
||||||
configured profile directory or file.
|
|
||||||
- adding downstream profile IDs to Promptkit's built-in catalog would make the
|
|
||||||
library own application-specific policy.
|
|
||||||
|
|
||||||
A downstream application could build its own filesystem overlay, but that
|
|
||||||
would duplicate Promptkit's profile discovery, error, and precedence behavior
|
|
||||||
at the consumer boundary.
|
|
||||||
|
|
||||||
## Requested Capability
|
|
||||||
|
|
||||||
Add one optional application fallback profile source to engine construction.
|
|
||||||
When present, matching profile IDs should resolve in this order:
|
|
||||||
|
|
||||||
1. in-memory profiles supplied through `WithProfiles`;
|
|
||||||
2. the ordinary configured profile source selected through a profile option or
|
|
||||||
`Config.ProfileDir`;
|
|
||||||
3. the application fallback profile source; and
|
|
||||||
4. Promptkit's embedded built-in profiles.
|
|
||||||
|
|
||||||
When no application fallback is configured, existing source precedence and
|
|
||||||
behavior must remain unchanged.
|
|
||||||
|
|
||||||
The minimum useful public surface is an `fs.FS`-backed option because consumers
|
|
||||||
can embed YAML profile assets. A possible API shape is:
|
|
||||||
|
|
||||||
```go
|
|
||||||
promptkit.WithFallbackProfileFS(profileFS, ".")
|
|
||||||
```
|
|
||||||
|
|
||||||
The name is illustrative rather than prescriptive. A companion option for
|
|
||||||
validated `Profile` values could be added if Promptkit maintainers find it
|
|
||||||
generally useful, but it is not required for the Weatherreporter use case.
|
|
||||||
|
|
||||||
## Required Semantics
|
|
||||||
|
|
||||||
- A higher-precedence source falls through only when the requested profile ID
|
|
||||||
is absent.
|
|
||||||
- A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching
|
|
||||||
profile is an error and must not silently fall through.
|
|
||||||
- The fallback source uses the existing strict profile YAML format and profile
|
|
||||||
validation rules.
|
|
||||||
- Profile values are selected as a whole. This feature does not merge,
|
|
||||||
inherit, or partially overlay profile definitions.
|
|
||||||
- `InspectProfile`, `Prepare`, prepared execution, and ordinary execution use
|
|
||||||
the same profile-source precedence.
|
|
||||||
- An explicit request profile continues to take precedence over a prompt's
|
|
||||||
`default_profile`; this request concerns definition lookup after the profile
|
|
||||||
ID has been selected.
|
|
||||||
- Repeated fallback-source options should follow Promptkit's documented
|
|
||||||
same-category option convention, normally with the last value replacing the
|
|
||||||
earlier value.
|
|
||||||
- A canceled lookup, invalid fallback asset, or unknown resolved backend should
|
|
||||||
continue to cross the public facade through Promptkit's existing public error
|
|
||||||
identities.
|
|
||||||
- Exact profile inspection must remain side-effect free and must not contact a
|
|
||||||
model provider.
|
|
||||||
|
|
||||||
## Application And Library Boundaries
|
|
||||||
|
|
||||||
Promptkit should own:
|
|
||||||
|
|
||||||
- the additional repository layer;
|
|
||||||
- deterministic lookup and fallthrough behavior;
|
|
||||||
- validation of the supplied source through the existing profile contract;
|
|
||||||
- consistent use of the layer across inspection and execution; and
|
|
||||||
- public documentation and tests for the added precedence rule.
|
|
||||||
|
|
||||||
The consuming application should continue to own:
|
|
||||||
|
|
||||||
- whether it supplies fallback profiles;
|
|
||||||
- the profile IDs and their domain meaning;
|
|
||||||
- embedded profile contents and model choices;
|
|
||||||
- application configuration and override policy;
|
|
||||||
- report- or workload-to-profile assignment; and
|
|
||||||
- credential checks and operator-facing errors beyond Promptkit's public
|
|
||||||
contract.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
This request does not ask Promptkit to add:
|
|
||||||
|
|
||||||
- Weatherreporter-specific profile IDs to its built-in catalog;
|
|
||||||
- profile inheritance, aliases, or field-level merging;
|
|
||||||
- automatic endpoint discovery or availability probing;
|
|
||||||
- provider failover or fallback from a failed selected profile;
|
|
||||||
- per-request model benchmarking or tier selection;
|
|
||||||
- application configuration discovery; or
|
|
||||||
- eager validation of every profile in every source.
|
|
||||||
|
|
||||||
## Compatibility
|
|
||||||
|
|
||||||
The feature can be additive. Engines that do not configure an application
|
|
||||||
fallback source should retain their current public behavior and precedence.
|
|
||||||
Existing uses of `WithProfiles`, `WithProfileFile`, `WithProfileFS`, and
|
|
||||||
`Config.ProfileDir` should not change meaning.
|
|
||||||
|
|
||||||
The application fallback is deliberately lower precedence than every existing
|
|
||||||
consumer-configured source. This preserves the established expectation that a
|
|
||||||
custom profile definition can override a packaged default with the same ID.
|
|
||||||
|
|
||||||
## Acceptance Criteria
|
|
||||||
|
|
||||||
The capability is sufficient for downstream adoption when Promptkit can
|
|
||||||
demonstrate that:
|
|
||||||
|
|
||||||
- a fallback-only profile can be inspected and used for preparation and
|
|
||||||
execution;
|
|
||||||
- a configured directory, file, or `fs.FS` profile with the same ID overrides
|
|
||||||
the fallback profile;
|
|
||||||
- an absent configured profile falls through to the application fallback;
|
|
||||||
- an invalid configured match fails instead of falling through;
|
|
||||||
- an absent application fallback profile continues to resolve from Promptkit's
|
|
||||||
built-in catalog;
|
|
||||||
- `WithProfiles` retains highest precedence;
|
|
||||||
- behavior is identical across inspection, preparation, and execution; and
|
|
||||||
- omitting the new option preserves existing tests and public contracts.
|
|
||||||
@@ -1,516 +0,0 @@
|
|||||||
# Promptkit Migration Roadmap
|
|
||||||
|
|
||||||
Status: Completed roadmap record.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
This roadmap records the scope, decisions, and completed outcome of replacing
|
|
||||||
the external Scriptorium CLI integration with Promptkit. Canonical
|
|
||||||
documentation outside `docs/roadmap/` owns the implemented behavior.
|
|
||||||
|
|
||||||
## Pre-Migration Baseline
|
|
||||||
|
|
||||||
Status: Historical migration input.
|
|
||||||
|
|
||||||
Before the migration, Weatherreporter exposed seven report definitions, but
|
|
||||||
only four had complete prompt-backed report implementations:
|
|
||||||
|
|
||||||
- Daily Report: `weather.daily_generated_text`
|
|
||||||
- Today Report: `weather.today_generated_text`
|
|
||||||
- Tomorrow Report: `weather.tomorrow_generated_text`
|
|
||||||
- Hourly Report: `weather.hourly_generated_text`
|
|
||||||
|
|
||||||
The three-day, weekend, and storm commands and registry definitions had no
|
|
||||||
corresponding Scriptorium prompt or schema and never formed complete
|
|
||||||
operational report products. The `weather.daily_report` Scriptorium prompt was
|
|
||||||
legacy source material and was not selected by the registry.
|
|
||||||
|
|
||||||
The Scriptorium source corpus was retained temporarily under
|
|
||||||
`docs/roadmap/scriptorium/` as migration input. It contained the four
|
|
||||||
operational generated-text prompt definitions, their referenced content,
|
|
||||||
private response schemas, shared instructions, and the unused legacy Daily
|
|
||||||
Markdown prompt. The temporary corpus was removed after the runtime assets
|
|
||||||
were reconciled and embedded.
|
|
||||||
|
|
||||||
## Implemented End State
|
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
Weatherreporter pins
|
|
||||||
`gitea.maximumdirect.net/eric/promptkit` at `v0.4.0` and uses it as the
|
|
||||||
in-process engine for prompt inspection, prepared execution, provider calls,
|
|
||||||
and first-pass output validation.
|
|
||||||
|
|
||||||
The `scriptorium` executable, subprocess adapter, configuration, runtime
|
|
||||||
dependency, direct-Markdown execution path, and integration documentation have
|
|
||||||
been removed. The four operational reports continue to use structured
|
|
||||||
generated text followed by weatherreporter-owned validation and Markdown
|
|
||||||
templates.
|
|
||||||
|
|
||||||
The unfinished three-day, weekend, and storm reports are not implemented as
|
|
||||||
part of this migration. Their incomplete CLI, registry, documentation, and
|
|
||||||
generation declarations are removed from the implemented surface before the
|
|
||||||
migration is considered complete. Any future implementation of those products
|
|
||||||
requires separate roadmap scope, prompt and schema design, tests, and
|
|
||||||
documentation.
|
|
||||||
|
|
||||||
Weather selection, forecast derivation, valid periods, module construction,
|
|
||||||
Recent Changes, generated-text interpretation, Markdown templates, durable
|
|
||||||
state, inspection, output copies, and Distributor notification remain owned by
|
|
||||||
weatherreporter.
|
|
||||||
|
|
||||||
The four report prompts and private response schemas are versioned embedded
|
|
||||||
application assets. Operators configure Promptkit profiles without replacing
|
|
||||||
the report-owned corpus. One Promptkit engine is constructed per CLI
|
|
||||||
invocation and shared by every report in that invocation, including all
|
|
||||||
reports in a morning or evening batch.
|
|
||||||
|
|
||||||
Promptkit is isolated behind a weatherreporter-owned execution contract.
|
|
||||||
Promptkit request, result, validation, error, profile, backend, and provider
|
|
||||||
types do not leak into application orchestration, report definitions, domain
|
|
||||||
packages, CLI summaries, durable state contracts, or Distributor behavior.
|
|
||||||
|
|
||||||
## Goals
|
|
||||||
|
|
||||||
Status: Completed migration outcomes.
|
|
||||||
|
|
||||||
- Removed the Scriptorium runtime dependency and subprocess boundary.
|
|
||||||
- Migrated the four operational report prompts to Promptkit `v0.4.0`.
|
|
||||||
- Used prepared execution to persist preparation provenance before provider work
|
|
||||||
while executing the exact frozen snapshot.
|
|
||||||
- Validated report prompt and profile selections before weather collection when
|
|
||||||
the required information is available.
|
|
||||||
- Preserved deterministic module snapshots and structured Recent Changes.
|
|
||||||
- Preserved generated-text domain validation and repository-owned Markdown
|
|
||||||
rendering.
|
|
||||||
- Preserved context cancellation, actionable errors, secret redaction, and
|
|
||||||
inspectable failures.
|
|
||||||
- Improved durable prompt provenance with prompt, input, profile, model,
|
|
||||||
validation, usage, and timing metadata.
|
|
||||||
- Kept content-rich prompt and response diagnostics separate from routine
|
|
||||||
metadata and CLI output.
|
|
||||||
- Kept tests offline and deterministic through injected Promptkit model
|
|
||||||
clients and fixtures.
|
|
||||||
- Removed incomplete report declarations from the implemented product surface
|
|
||||||
rather than creating new report products during an integration migration.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
Status: Completed migration constraints.
|
|
||||||
|
|
||||||
The completed migration did not:
|
|
||||||
|
|
||||||
- create prompts, schemas, templates, or completed products for three-day,
|
|
||||||
weekend, or storm reports;
|
|
||||||
- preserve the unused `weather.daily_report` legacy Markdown prompt as an
|
|
||||||
active runtime asset;
|
|
||||||
- preserve a direct-Markdown LLM generation mode;
|
|
||||||
- move meteorological selection, derivation, thresholds, or comparison logic
|
|
||||||
into prompts or Promptkit;
|
|
||||||
- send raw unbounded Weather API responses to the model;
|
|
||||||
- replace weatherreporter's generated-text domain validation or Markdown
|
|
||||||
template rendering;
|
|
||||||
- add a general workflow engine, provider plugin system, or arbitrary backend
|
|
||||||
registry;
|
|
||||||
- add automatic provider, validation, repair, or capacity retries;
|
|
||||||
- add concurrent report generation to the sequential batch workflow;
|
|
||||||
- expose Promptkit types as weatherreporter contracts;
|
|
||||||
- keep a production-selectable Scriptorium/Promptkit dual-run mode;
|
|
||||||
- require Promptkit eager source validation, structured generation errors, or
|
|
||||||
semantic execution-target fingerprints; or
|
|
||||||
- use an unpublished Promptkit commit, committed Go workspace, or committed
|
|
||||||
local module replacement.
|
|
||||||
|
|
||||||
## Locked Decisions
|
|
||||||
|
|
||||||
Status: Implemented migration decisions.
|
|
||||||
|
|
||||||
### Dependency And Upgrade Boundary
|
|
||||||
|
|
||||||
- The migration pins the tagged Promptkit `v0.4.0` release.
|
|
||||||
- Coordinated local development may temporarily use the sibling Promptkit
|
|
||||||
checkout, but committed module metadata must reference the tagged release.
|
|
||||||
- The adapter relies on the public root Promptkit package only.
|
|
||||||
- A future Promptkit upgrade requires explicit review of prepared-execution
|
|
||||||
lifecycle, prompt and profile inspection, prompt/profile/schema formats,
|
|
||||||
error identities, validation behavior, capacity behavior, and the outbound
|
|
||||||
provider contract.
|
|
||||||
- Promptkit's deferred eager source validation, structured generation errors,
|
|
||||||
and semantic execution-target fingerprints do not block this migration.
|
|
||||||
|
|
||||||
### Operational Report Scope
|
|
||||||
|
|
||||||
- The migration preserves these prompt IDs:
|
|
||||||
`weather.daily_generated_text`, `weather.today_generated_text`,
|
|
||||||
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`.
|
|
||||||
- Each operational report definition selects the exact embedded prompt version
|
|
||||||
`1.0.0`; execution does not rely on ambiguous single-version lookup.
|
|
||||||
- Morning and evening batch membership remains based on Today, Tomorrow, and
|
|
||||||
eligible future Daily reports.
|
|
||||||
- Three-day, weekend, and storm are removed from current CLI help, parsing,
|
|
||||||
report registry membership, tests that claim implemented generation, and
|
|
||||||
non-roadmap documentation.
|
|
||||||
- The future product concepts may remain under `docs/roadmap/`, but migration
|
|
||||||
verification does not invent outputs or compare nonexistent prompts.
|
|
||||||
|
|
||||||
### Application Boundary
|
|
||||||
|
|
||||||
- Promptkit remains an adapter boundary even though it runs in process.
|
|
||||||
- A weatherreporter-owned contract represents prompt identity, preparation,
|
|
||||||
execution, output, validation, usage, provenance, and neutral error
|
|
||||||
categories.
|
|
||||||
- The Promptkit adapter maps public Promptkit values into that contract.
|
|
||||||
- App orchestration and test fakes depend on the project-owned contract, not
|
|
||||||
Promptkit.
|
|
||||||
- Scriptorium-specific request, result, error, and generation-mode types are
|
|
||||||
removed rather than renamed and retained.
|
|
||||||
|
|
||||||
### Prompt And Schema Ownership
|
|
||||||
|
|
||||||
- Weatherreporter embeds the four operational prompt definitions, referenced
|
|
||||||
prompt content, shared prompt content, and private response schemas.
|
|
||||||
- Assets remain separate files rather than inline Go strings.
|
|
||||||
- The temporary corpus under `docs/roadmap/scriptorium/` is migration source
|
|
||||||
material, not the final runtime location.
|
|
||||||
- Weatherreporter's existing generated-text domain types, schemas, and
|
|
||||||
templates remain the canonical application contract. Imported Scriptorium
|
|
||||||
assets are reconciled with that contract rather than copied blindly or kept
|
|
||||||
as duplicate runtime schemas.
|
|
||||||
- The imported Daily schema's incorrect Today `$id` and title are corrected.
|
|
||||||
- `confidence` is handled consistently across each prompt, provider-facing
|
|
||||||
schema, generated-text domain type, and template. The existing optional
|
|
||||||
weatherreporter field remains supported unless a separate domain decision
|
|
||||||
removes it.
|
|
||||||
- Prompt input metadata identifies the serialized data package as YAML rather
|
|
||||||
than JSON.
|
|
||||||
- Imported `pipeline-weather/...` schema paths are replaced with paths valid
|
|
||||||
inside the embedded Promptkit schema source.
|
|
||||||
- Imported `repair_attempts: 2` values are removed or set to zero. The
|
|
||||||
migration does not rely on Promptkit's internal-only repair capability.
|
|
||||||
- The unused `weather.daily_report` prompt is not promoted into runtime assets.
|
|
||||||
- One centralized embedded prompt/schema source is sufficient; Weatherreporter
|
|
||||||
does not need Notarius's multi-module asset-flattening registry.
|
|
||||||
|
|
||||||
### Profiles, Backends, And Credentials
|
|
||||||
|
|
||||||
- Execution profiles remain operator-configurable rather than embedded report
|
|
||||||
policy.
|
|
||||||
- Each embedded operational prompt declares Promptkit's built-in
|
|
||||||
`gemini-flash-latest` profile as its default.
|
|
||||||
- `gemini-flash-latest` is intentionally a moving model alias. The execution
|
|
||||||
record captures the effective model identity, but operators who require a
|
|
||||||
pinned model must select an explicit external profile.
|
|
||||||
- Configuration supports at most one external profile source:
|
|
||||||
`promptkit.profile_file` or `promptkit.profile_dir`. The two fields are
|
|
||||||
mutually exclusive.
|
|
||||||
- A nonblank `promptkit.profile` is the explicit request profile for every
|
|
||||||
report in the invocation and takes precedence over each prompt's
|
|
||||||
`default_profile`. A blank value uses the prompt default.
|
|
||||||
- Promptkit's normal profile-source precedence remains intact: an external
|
|
||||||
matching profile takes precedence over an embedded built-in profile, and an
|
|
||||||
invalid matching external profile is an error rather than a reason to fall
|
|
||||||
back.
|
|
||||||
- Weatherreporter exposes Promptkit's conventional `local` backend through the
|
|
||||||
narrow `promptkit.local.endpoint` and
|
|
||||||
`promptkit.local.concurrency_limit` configuration fields. It does not expose
|
|
||||||
arbitrary backend registration.
|
|
||||||
- A configured local endpoint registers the engine-scoped `local` backend. An
|
|
||||||
operator-supplied external profile selects it with `backend: local` and owns
|
|
||||||
the model-specific settings; Weatherreporter does not invent a local model
|
|
||||||
profile.
|
|
||||||
- Local concurrency defaults to one. A value of zero means unlimited, matching
|
|
||||||
Promptkit, and a negative value is invalid. Queue capacity and general
|
|
||||||
backend parameters are not exposed.
|
|
||||||
- Credential values remain in environment variables or file-backed
|
|
||||||
environment secrets. Configuration contains only credential source names.
|
|
||||||
- Provider credentials never appear in logs, errors, CLI output, durable
|
|
||||||
metadata, preparation records, execution records, or debug summaries.
|
|
||||||
- Promptkit `InspectProfile` reports structural target and credential
|
|
||||||
requirements; Weatherreporter owns policy for checking configured
|
|
||||||
environment availability.
|
|
||||||
- Promptkit revalidates environment credentials at `RunPrepared`; a successful
|
|
||||||
preparation does not promise that execution-time credentials remain
|
|
||||||
available.
|
|
||||||
|
|
||||||
### Configuration Contract
|
|
||||||
|
|
||||||
The replacement configuration surface is:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
promptkit:
|
|
||||||
profile: ""
|
|
||||||
profile_file: ""
|
|
||||||
profile_dir: ""
|
|
||||||
timeout: 2m
|
|
||||||
|
|
||||||
local:
|
|
||||||
endpoint: ""
|
|
||||||
concurrency_limit: 1
|
|
||||||
```
|
|
||||||
|
|
||||||
- `timeout` remains the transport-wide provider-call safety cap.
|
|
||||||
- A blank local endpoint leaves the conventional local backend unregistered.
|
|
||||||
- Scriptorium's `binary`, `config_path`, and `extra_args` settings have no
|
|
||||||
Promptkit equivalents and are removed.
|
|
||||||
- Configuration validation rejects simultaneous `profile_file` and
|
|
||||||
`profile_dir` values, invalid local endpoints, negative concurrency, and
|
|
||||||
selected profiles that cannot resolve their backend.
|
|
||||||
|
|
||||||
### Engine Construction And Inspection
|
|
||||||
|
|
||||||
- One Promptkit engine is constructed per CLI invocation at the application
|
|
||||||
composition boundary.
|
|
||||||
- Single-report generation and every report in a batch use that same engine.
|
|
||||||
- Per-report orchestration does not construct a default engine.
|
|
||||||
- Promptkit backend capacity state and HTTP transport are shared consistently
|
|
||||||
for the invocation.
|
|
||||||
- Before collection, `InspectPrompt` checks every selected report's exact ID
|
|
||||||
and version, declared `data_package` input, default-profile metadata, prompt
|
|
||||||
hash availability, and declared output contract.
|
|
||||||
- `InspectPrompt` is a point-in-time structural check. It does not load a JSON
|
|
||||||
Schema, resolve a profile, or freeze later execution.
|
|
||||||
- Explicit profile overrides and relevant prompt defaults are checked with
|
|
||||||
`InspectProfile` before collection when application policy requires them.
|
|
||||||
- `InspectProfile` is also point-in-time and does not check credential values.
|
|
||||||
- Successful `PrepareExecution`, not inspection, is the per-run authority for
|
|
||||||
loaded schema, rendered content, frozen inputs, effective settings, and
|
|
||||||
durable execution provenance.
|
|
||||||
|
|
||||||
### Prompt Input
|
|
||||||
|
|
||||||
- Promptkit receives only the curated `data_package` produced by
|
|
||||||
`internal/promptinput`.
|
|
||||||
- Weatherreporter serializes the package once, atomically persists those exact
|
|
||||||
bytes, and supplies the same bytes with a Promptkit inline artifact.
|
|
||||||
- The managed data-package path may be supplied as non-secret provenance
|
|
||||||
through the inline artifact URI.
|
|
||||||
- Weatherreporter does not delegate unrestricted path loading to Promptkit's
|
|
||||||
default file artifact reader.
|
|
||||||
- Prompt inspection and adapter tests verify that `data_package` is required
|
|
||||||
and declared with the chosen YAML media type.
|
|
||||||
|
|
||||||
### Prepared Execution
|
|
||||||
|
|
||||||
- `Engine.PrepareExecution` replaces Scriptorium render preflight.
|
|
||||||
- Weatherreporter obtains `PreparedExecution.Details`, maps a safe subset into
|
|
||||||
its own preparation record, and persists that record before calling
|
|
||||||
`Engine.RunPrepared`.
|
|
||||||
- `RunPrepared` executes the frozen prompt, profile, schema, inputs, rendered
|
|
||||||
messages, target, and validation resources retained by the handle.
|
|
||||||
- Every acquired handle is followed immediately by `defer handle.Discard()`.
|
|
||||||
Discard is safe after execution and releases unused private execution state.
|
|
||||||
- Handles remain adapter-local, engine-bound, one-shot, in-process values.
|
|
||||||
They are never serialized, persisted, copied into app contracts, or treated
|
|
||||||
as restartable jobs.
|
|
||||||
- Preparation and execution use independent contexts. Execution receives the
|
|
||||||
active report workflow context.
|
|
||||||
- Capacity is not reserved during preparation. Capacity rejection can
|
|
||||||
therefore occur after a preparation record has been persisted.
|
|
||||||
- `RunPrepared` consumes the handle on success and every operational failure.
|
|
||||||
- Preparation details remain available from the adapter after execution or
|
|
||||||
discard, but rendered messages are not copied into routine durable state.
|
|
||||||
- Promptkit execution timing excludes preparation and consumer-held delay.
|
|
||||||
Weatherreporter records preparation timing and execution timing separately.
|
|
||||||
|
|
||||||
### Execution And Validation
|
|
||||||
|
|
||||||
- All four operational reports use Promptkit JSON Schema output validation.
|
|
||||||
- A completed Promptkit validation rejection returns a `RunResult`; the
|
|
||||||
adapter retains raw output and bounded validation details before failing the
|
|
||||||
report.
|
|
||||||
- An operational generation or validation error returns no partial
|
|
||||||
`RunResult`.
|
|
||||||
- Weatherreporter's `internal/generatedtext` validation remains the final
|
|
||||||
report-specific decode and domain boundary.
|
|
||||||
- Weatherreporter's `internal/reporttemplate` remains responsible for managed
|
|
||||||
Markdown rendering.
|
|
||||||
- Weatherreporter atomically persists Promptkit raw output and later artifacts
|
|
||||||
rather than asking Promptkit to choose managed filesystem paths.
|
|
||||||
- No Promptkit output-repair behavior is assumed or requested.
|
|
||||||
|
|
||||||
## Durable Artifacts And Observability
|
|
||||||
|
|
||||||
Status: Implemented design constraints.
|
|
||||||
|
|
||||||
Routine durable state retains useful non-secret provenance without persisting
|
|
||||||
full rendered prompts.
|
|
||||||
|
|
||||||
The preparation record contains:
|
|
||||||
|
|
||||||
- prompt ID and exact version;
|
|
||||||
- prompt definition hash;
|
|
||||||
- rendered prompt hash;
|
|
||||||
- input hashes;
|
|
||||||
- selected profile and backend identity;
|
|
||||||
- effective model identity;
|
|
||||||
- output contract summary;
|
|
||||||
- preparation start, end, and duration; and
|
|
||||||
- the path of the exact persisted data package.
|
|
||||||
|
|
||||||
The execution record and run metadata contain, when available:
|
|
||||||
|
|
||||||
- Promptkit run ID;
|
|
||||||
- prompt ID, version, and hashes;
|
|
||||||
- input hashes;
|
|
||||||
- selected profile, backend, and model identity;
|
|
||||||
- generated-content hash;
|
|
||||||
- token usage;
|
|
||||||
- execution start, end, and duration;
|
|
||||||
- validation status and bounded diagnostics; and
|
|
||||||
- paths of separately persisted raw output, normalized generated text, render
|
|
||||||
context, managed Markdown, and other artifacts reached by the workflow.
|
|
||||||
|
|
||||||
Provider endpoints, full effective model parameter maps, rendered messages,
|
|
||||||
schema bodies, data-package contents, and generated content do not belong in
|
|
||||||
routine metadata or CLI summaries.
|
|
||||||
|
|
||||||
Rendered messages and other content-rich preparation or response diagnostics
|
|
||||||
are available only when the operator supplies
|
|
||||||
`--llm-debug-dir <path>` to a single-report or batch command.
|
|
||||||
|
|
||||||
- There is no persistent YAML setting for debug capture.
|
|
||||||
- The debug root is validated or created before weather collection or provider
|
|
||||||
work. A requested destination that cannot be secured or written is an error.
|
|
||||||
- Artifacts are grouped beneath
|
|
||||||
`<path>/<report-id>/<valid-date>/<run-id>/`.
|
|
||||||
- Directories and files use owner-only permissions and atomic writes.
|
|
||||||
- Debug artifacts may contain rendered messages and content-rich preparation
|
|
||||||
or response diagnostics, but never credentials.
|
|
||||||
- The debug path appears in command output only when debug capture is enabled;
|
|
||||||
it is not added to routine durable metadata.
|
|
||||||
- Debug artifacts are not cache or comparison inputs. Their retention is owned
|
|
||||||
by the operator who selected the directory.
|
|
||||||
|
|
||||||
### Artifact Identities And Versions
|
|
||||||
|
|
||||||
Weatherreporter replaces Scriptorium-specific artifact identities rather than
|
|
||||||
reusing names whose meanings have changed:
|
|
||||||
|
|
||||||
- `PromptPreparationArtifact` uses schema version
|
|
||||||
`weatherreporter.prompt_preparation.v1`, is written as
|
|
||||||
`prompt_preparation.<runID>.json`, and is referenced by
|
|
||||||
`preparationPath`.
|
|
||||||
- `PromptExecutionArtifact` uses schema version
|
|
||||||
`weatherreporter.prompt_execution.v1`, is written as
|
|
||||||
`prompt_execution.<runID>.json`, and is referenced by `executionPath`.
|
|
||||||
- Run metadata advances to `weatherreporter.metadata.v2` and uses those new
|
|
||||||
path fields.
|
|
||||||
|
|
||||||
Preparation files remain beneath the existing configurable `preflight/`
|
|
||||||
directory, and execution files remain beneath the existing `snapshots/` tree.
|
|
||||||
The stable physical grouping limits deployment disruption without preserving
|
|
||||||
misleading Scriptorium-era filenames or field names. Raw generated output,
|
|
||||||
normalized generated text, render context, managed Markdown, and other
|
|
||||||
artifacts whose meanings have not changed retain their existing names and
|
|
||||||
locations.
|
|
||||||
|
|
||||||
Run inspection remains able to read `weatherreporter.metadata.v1` and its
|
|
||||||
legacy `preflightPath` and `generatedTextResultPath` references. New runs write
|
|
||||||
only the v2 metadata and new artifact names; Weatherreporter does not
|
|
||||||
dual-write deprecated aliases. CLI summary fields adopt `preparationPath` and
|
|
||||||
`executionPath` as an explicit, documented contract change.
|
|
||||||
|
|
||||||
## Failure Contract
|
|
||||||
|
|
||||||
Status: Implemented design constraints.
|
|
||||||
|
|
||||||
- A preparation failure produces a redacted weatherreporter-owned failure
|
|
||||||
receipt with report, RunID, prompt, stage, timing, and classified error
|
|
||||||
context. It does not fabricate Promptkit preparation details.
|
|
||||||
- An operational execution failure retains the successful preparation record
|
|
||||||
and adds a redacted execution failure receipt. No partial Promptkit result or
|
|
||||||
model output is invented.
|
|
||||||
- A Promptkit validation rejection retains the returned result, raw generated
|
|
||||||
output, validation details, and safe provenance before the report fails.
|
|
||||||
- A later generated-text decode, domain-validation, or template failure
|
|
||||||
retains every raw and validated artifact reached before that stage.
|
|
||||||
- Caller cancellation takes precedence when the active workflow context is
|
|
||||||
canceled.
|
|
||||||
- `promptkit.CapacityError` is recognized with `errors.As`; its backend ID is
|
|
||||||
copied into a weatherreporter-owned capacity error while
|
|
||||||
`ErrCapacityExceeded` remains the classification.
|
|
||||||
- Capacity rejection is an operational report failure, not invalid model
|
|
||||||
output, and does not trigger an automatic retry.
|
|
||||||
- Other Promptkit public error identities are translated into the narrow
|
|
||||||
weatherreporter error categories needed by CLI, metadata, and batch
|
|
||||||
behavior. Diagnostic prose is not parsed as a contract.
|
|
||||||
- Single-report commands return the classified failure with available
|
|
||||||
inspectable paths.
|
|
||||||
- Batch runs continue independent later reports under the existing batch
|
|
||||||
failure policy.
|
|
||||||
- Any future retry policy belongs to app orchestration, not the adapter.
|
|
||||||
|
|
||||||
## Compatibility Requirements
|
|
||||||
|
|
||||||
Status: Implemented design constraints.
|
|
||||||
|
|
||||||
- Daily, Today, Tomorrow, and Hourly report IDs, prompt IDs, valid periods,
|
|
||||||
artifact grouping, output names, and Distributor bundle behavior remain
|
|
||||||
stable.
|
|
||||||
- Morning and evening batch collection, planning, ordering, and continuation
|
|
||||||
behavior remains stable.
|
|
||||||
- Module snapshot and Recent Changes behavior remains deterministic.
|
|
||||||
- Promptkit receives only the existing curated prompt-input boundary.
|
|
||||||
- Managed Markdown remains the Distributor upload source.
|
|
||||||
- RunID lookup and inspection remain available for successful and failed runs.
|
|
||||||
- Existing managed paths remain stable where their meaning is unchanged.
|
|
||||||
Scriptorium-specific artifact names or schemas change when retaining them
|
|
||||||
would misrepresent the Promptkit contract.
|
|
||||||
- Existing v1 run metadata and referenced artifacts remain inspectable after
|
|
||||||
the migration. New runs use the v2 metadata and Promptkit-era artifact
|
|
||||||
identities without dual-writing deprecated aliases.
|
|
||||||
- Artifact or metadata schema changes are explicit, documented, and covered by
|
|
||||||
state and inspection tests.
|
|
||||||
- Prompt or generated content is not added to routine logs or CLI summaries.
|
|
||||||
- Tests do not require live providers or credentials.
|
|
||||||
- Removing incomplete three-day, weekend, and storm surfaces is documented as
|
|
||||||
correction of an unfinished product boundary, not as successful Promptkit
|
|
||||||
migration of those reports.
|
|
||||||
|
|
||||||
## Verification And Completion Criteria
|
|
||||||
|
|
||||||
Status: Completed and verified.
|
|
||||||
|
|
||||||
Completion was verified by the following outcomes:
|
|
||||||
|
|
||||||
- the four operational reports inspect, prepare, and execute through Promptkit
|
|
||||||
`v0.4.0` using embedded report-owned assets;
|
|
||||||
- every report uses exact prompt version `1.0.0`, requires the YAML
|
|
||||||
`data_package`, and declares the expected JSON Schema output contract;
|
|
||||||
- prepared execution persists a safe preparation record before provider work
|
|
||||||
and executes the same frozen snapshot;
|
|
||||||
- deterministic offline adapter and app tests cover success, preparation
|
|
||||||
failure, credential revalidation, capacity rejection, cancellation, timeout,
|
|
||||||
generation failure, Promptkit validation rejection, generated-text domain
|
|
||||||
failure, template failure, and handle discard;
|
|
||||||
- morning and evening batches construct one engine and preserve current
|
|
||||||
collection, planning, ordering, continuation, output, and notification
|
|
||||||
behavior;
|
|
||||||
- the temporary corpus has been reconciled into one runtime prompt/schema
|
|
||||||
source without duplicate provider-facing schemas;
|
|
||||||
- configuration examples load and contain no Scriptorium fields;
|
|
||||||
- CLI summaries and inspection commands expose the new project-owned artifact
|
|
||||||
contract without Promptkit types;
|
|
||||||
- Scriptorium code, configuration, tests, and runtime documentation have been
|
|
||||||
removed;
|
|
||||||
- incomplete three-day, weekend, and storm commands, registry entries, tests,
|
|
||||||
and current-behavior documentation have been removed or moved to roadmap
|
|
||||||
scope;
|
|
||||||
- non-roadmap documentation describes only the implemented Promptkit
|
|
||||||
integration;
|
|
||||||
- `go test ./...`, required focused or race-enabled checks, CLI help
|
|
||||||
validation, and `git diff --check` pass; and
|
|
||||||
- no committed `go.work`, local `replace`, live-provider test, or
|
|
||||||
secret-bearing fixture remains.
|
|
||||||
|
|
||||||
Fixture-based comparison with prior Scriptorium behavior is sufficient.
|
|
||||||
Production dual-run is not required because model calls are nondeterministic,
|
|
||||||
costly, and difficult to compare meaningfully.
|
|
||||||
|
|
||||||
## Decision Status
|
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
The roadmap has no remaining open product or architecture questions. Later
|
|
||||||
changes to this completed scope require new roadmap or decision-record scope
|
|
||||||
rather than implicit changes to this historical record.
|
|
||||||
@@ -1,75 +0,0 @@
|
|||||||
# Troubleshooting
|
|
||||||
|
|
||||||
Start with the command's classified error. When content-rich prompt diagnostics
|
|
||||||
are needed, enable a new run with `--llm-debug-dir` and handle the resulting
|
|
||||||
secure capture as sensitive. Current-version workspace receipts can provide
|
|
||||||
additional context when present, but are transitional state rather than a
|
|
||||||
long-term troubleshooting interface.
|
|
||||||
|
|
||||||
## Prompt inspection or credentials fail before collection
|
|
||||||
|
|
||||||
A prompt/version, contract, selected profile, unsupported direct-key profile,
|
|
||||||
or required environment credential can fail before weather collection. Correct
|
|
||||||
the configured `promptkit` profile or profile source, confirm the exact
|
|
||||||
Promptkit asset is available, and supply any reported environment credential.
|
|
||||||
Do not add provider keys to YAML. See [configuration](config.md).
|
|
||||||
|
|
||||||
## Local profile override is malformed or selects an unexpected model
|
|
||||||
|
|
||||||
`promptkit.profile_file` and `promptkit.profile_dir` supply complete profile
|
|
||||||
definitions. A same-ID definition replaces the embedded profile, and a malformed
|
|
||||||
matching definition fails before collection instead of falling back. Validate
|
|
||||||
the selected profile's YAML, ID, backend or endpoint, and model. If the model
|
|
||||||
is unexpected, first check the global `promptkit.profile` selection and then
|
|
||||||
look for a same-ID definition in the configured file or directory.
|
|
||||||
|
|
||||||
Current-version preparation and execution receipts may retain the selected
|
|
||||||
profile ID and effective backend/model, but not an endpoint or credential.
|
|
||||||
Use them only as supplemental context after the active command error or an
|
|
||||||
explicit secure debug capture. See the maintained
|
|
||||||
[local `weather-light` profile example](../examples/weather-light-local-profile.yml).
|
|
||||||
|
|
||||||
## Local model endpoint is unavailable
|
|
||||||
|
|
||||||
An endpoint-only `weather-light` override can pass preflight and still fail
|
|
||||||
during provider preparation or execution when the local server is unavailable
|
|
||||||
or does not accept the configured model. Start the local server, correct the
|
|
||||||
endpoint or model in the profile, and run the command again. Weatherreporter
|
|
||||||
does not probe endpoints or automatically use a remote profile instead.
|
|
||||||
|
|
||||||
## Preparation, capacity, or execution fails
|
|
||||||
|
|
||||||
A preparation failure occurs before provider work; an execution failure occurs
|
|
||||||
after preparation. A capacity error for one batch report does not retry that
|
|
||||||
report or prevent later independent reports. Correct the profile or backend
|
|
||||||
condition identified by the bounded command error, then create a new run.
|
|
||||||
Use explicit secure debug capture only when additional content-rich diagnostics
|
|
||||||
are necessary. See [operations](operations.md).
|
|
||||||
|
|
||||||
## Generated text fails validation
|
|
||||||
|
|
||||||
Raw generated output may be saved but Markdown is not rendered when the JSON
|
|
||||||
does not match the report schema. Correct the Promptkit prompt/profile behavior
|
|
||||||
or the matching schema and validator in source control; do not edit raw output
|
|
||||||
to treat it as validated. See [templates](templates.md).
|
|
||||||
|
|
||||||
## Debug capture fails
|
|
||||||
|
|
||||||
`--llm-debug-dir` must be an absolute secure directory outside workspace state.
|
|
||||||
A debug-write failure stops the affected report to avoid continuing without the
|
|
||||||
requested diagnostic. Repair the named path's ownership or permissions, then
|
|
||||||
rerun. Treat capture files as sensitive. See [operations](operations.md).
|
|
||||||
|
|
||||||
## Weather, state, output, or notification fails
|
|
||||||
|
|
||||||
Collection errors precede planning. Later filesystem, output-copy, template,
|
|
||||||
or Distributor errors retain the reached safe paths in the summary. Repair only
|
|
||||||
the reported endpoint or path, leave successful managed reports intact, and
|
|
||||||
rerun the affected report or batch. A batch notification is intentionally
|
|
||||||
skipped when any report item fails.
|
|
||||||
|
|
||||||
## Secrets cannot be loaded
|
|
||||||
|
|
||||||
Secret files must be regular non-symlink files directly beneath
|
|
||||||
`secrets.directory` with valid environment-variable basenames. Correct the
|
|
||||||
reported file or directory without placing secret values in YAML.
|
|
||||||
Reference in New Issue
Block a user