Add embedded weather profile catalog
This commit is contained in:
@@ -1,373 +0,0 @@
|
|||||||
# Domain-Specific Prompt Profiles Implementation Plan
|
|
||||||
|
|
||||||
Status: Ready for implementation.
|
|
||||||
|
|
||||||
## Purpose And Authority
|
|
||||||
|
|
||||||
This document gives a coding agent the ordered work needed to implement the
|
|
||||||
[domain-specific prompt profiles roadmap](domain-profiles.md). The roadmap is
|
|
||||||
authoritative for scope, user intent, policy choices, and the desired end
|
|
||||||
state. This plan is authoritative for implementation sequence, verification,
|
|
||||||
and stage exit gates.
|
|
||||||
|
|
||||||
The target implementer is a `gpt-5.6-terra` coding agent using high reasoning.
|
|
||||||
Complete the stages in order. Each stage is scoped for one implementation
|
|
||||||
prompt and must leave the repository coherent, tested, and reviewable before
|
|
||||||
the next stage begins.
|
|
||||||
|
|
||||||
This plan follows the repository's
|
|
||||||
[architecture](../policy/architecture.md),
|
|
||||||
[documentation](../policy/documentation.md), and
|
|
||||||
[testing](../policy/testing.md) policies.
|
|
||||||
|
|
||||||
## Completed Prerequisite
|
|
||||||
|
|
||||||
Weatherreporter is already pinned to Promptkit v0.5.0. That release provides
|
|
||||||
the public `WithFallbackProfileFS` option and the required precedence across
|
|
||||||
inspection, preparation, and execution. The dependency upgrade passed
|
|
||||||
Weatherreporter's full offline test suite, race-enabled suite, CLI help check,
|
|
||||||
and an operator `generate hourly` smoke test. Do not repeat or replace the
|
|
||||||
dependency upgrade as part of these stages.
|
|
||||||
|
|
||||||
## Locked Product Decisions
|
|
||||||
|
|
||||||
Implement these exact Weatherreporter-owned profiles:
|
|
||||||
|
|
||||||
| Profile ID | Backend | Model | Reasoning effort | Timeout | Service tier |
|
|
||||||
| --- | --- | --- | --- | --- | --- |
|
|
||||||
| `weather-light` | `openrouter` | `deepseek/deepseek-v4-flash` | Omitted | 180 seconds | `flex` |
|
|
||||||
| `weather-balanced` | `openrouter` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
|
||||||
| `weather-deep` | `openrouter` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
|
||||||
|
|
||||||
Assign Hourly to `weather-light`; assign Daily, Today, and Tomorrow to
|
|
||||||
`weather-balanced`; assign no report to `weather-deep` initially. Advance all
|
|
||||||
four prompt definitions and matching report-registry entries from `1.0.1` to
|
|
||||||
`1.1.0` when their defaults change.
|
|
||||||
|
|
||||||
The leading `~` in the Gemini and Claude model IDs is required and denotes an
|
|
||||||
OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a
|
|
||||||
dated model version. Do not add temperature, `top_p`, maximum-token, endpoint,
|
|
||||||
or credential fields to the embedded definitions.
|
|
||||||
|
|
||||||
Prompt preparation and execution artifacts written at `1.0.1` are not required
|
|
||||||
to remain readable after the transition to `1.1.0`. Do not add a migration,
|
|
||||||
compatibility shim, or weaker historical-artifact validation for this feature.
|
|
||||||
|
|
||||||
Definition lookup must remain:
|
|
||||||
|
|
||||||
1. explicit Promptkit in-memory profiles used by tests or an embedding
|
|
||||||
consumer;
|
|
||||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
|
||||||
3. Weatherreporter's embedded fallback profiles; and
|
|
||||||
4. Promptkit's built-in catalog.
|
|
||||||
|
|
||||||
Selection remains a separate concern: a nonblank global `promptkit.profile`
|
|
||||||
selects the profile for every report in the invocation; otherwise the exact
|
|
||||||
prompt definition's `default_profile` selects it. A malformed matching
|
|
||||||
higher-precedence profile is an error and never falls through.
|
|
||||||
|
|
||||||
## Continuing Invariants
|
|
||||||
|
|
||||||
- Keep all Promptkit types and mechanics inside
|
|
||||||
`internal/adapters/promptkit`, its focused tests, and asset contract tests.
|
|
||||||
- Keep prompt inspection before weather collection and provider work.
|
|
||||||
- Keep one Promptkit engine per command action and one shared engine across a
|
|
||||||
sequential batch.
|
|
||||||
- Preserve logical profile ID and effective backend/model information through
|
|
||||||
active inspection and execution where the project-owned contract already
|
|
||||||
exposes it. Do not add new durable-provenance fields or compatibility
|
|
||||||
guarantees.
|
|
||||||
- Leave existing workspace persistence behavior otherwise unchanged. The
|
|
||||||
accepted [ephemeral-state roadmap](ephemeral-state.md) owns its future
|
|
||||||
removal and must not be partially implemented here.
|
|
||||||
- Do not expose endpoints, credentials, rendered messages, schemas, request
|
|
||||||
bodies, response bodies, or complete parameter maps through ordinary errors,
|
|
||||||
logs, summaries, or state.
|
|
||||||
- Keep the default suite deterministic, offline, and credential-free.
|
|
||||||
- Do not add endpoint discovery, health probing, provider failover, retries at
|
|
||||||
a more expensive tier, profile merging, per-report configuration fields, or
|
|
||||||
severity-driven model selection.
|
|
||||||
- Update canonical current-state documentation only in the stage where the
|
|
||||||
corresponding behavior becomes implemented.
|
|
||||||
- Run `git diff --check` before completing every stage.
|
|
||||||
|
|
||||||
## Stage 1: Add The Embedded Weather Profile Catalog
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Create one repository-owned, embedded profile source containing exactly the
|
|
||||||
three locked logical profiles.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Add strict YAML profile assets beneath `internal/promptassets` using the
|
|
||||||
exact IDs and definitions in this plan.
|
|
||||||
2. Extend `internal/promptassets` with a narrowly named accessor that returns
|
|
||||||
the embedded profile `fs.FS`. Follow the existing prompt and schema asset
|
|
||||||
pattern without exposing Promptkit types from the package.
|
|
||||||
3. Keep profile filenames and embed layout simple and deterministic. Do not
|
|
||||||
duplicate Promptkit's built-in directory taxonomy unless the application
|
|
||||||
assets require it.
|
|
||||||
4. Validate the assets through Promptkit's public engine/profile inspection
|
|
||||||
surface rather than adding a second YAML parser or a Weatherreporter-owned
|
|
||||||
profile representation.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Extend the asset contract tests to assert exactly the three logical IDs,
|
|
||||||
their exact effective model IDs, and the intentional parameters.
|
|
||||||
- Prove all three profiles inspect successfully offline when supplied as a
|
|
||||||
fallback source and no operator source is present.
|
|
||||||
- Assert that the catalog contains no endpoints, credentials, temperature,
|
|
||||||
`top_p`, or maximum-token settings.
|
|
||||||
- Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./internal/promptassets
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
The embedded catalog is complete, strictly valid, safe, and independently
|
|
||||||
inspectable through Promptkit v0.5.0's public API.
|
|
||||||
|
|
||||||
## Stage 2: Wire Fallback Resolution And Protect Precedence
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Supply the embedded catalog through Promptkit's application fallback layer
|
|
||||||
without changing existing operator configuration or application boundaries.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Add `promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".")` to
|
|
||||||
normal adapter engine construction.
|
|
||||||
2. Preserve existing `profile_file`, `profile_dir`, configured local backend,
|
|
||||||
timeout, prompt filesystem, schema filesystem, and test-option behavior.
|
|
||||||
3. Ensure ordinary production construction and the adapter's test
|
|
||||||
construction path exercise the same fallback wiring. Test-only explicit
|
|
||||||
profiles may retain Promptkit's documented highest precedence.
|
|
||||||
4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays,
|
|
||||||
existence checks, YAML parsing, or merge behavior to Weatherreporter.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- At the adapter boundary, prove fallback-only inspection of all three
|
|
||||||
Weatherreporter profiles.
|
|
||||||
- Prove same-ID overrides through both configured `profile_file` and
|
|
||||||
`profile_dir`, including resolution of the override's effective backend and
|
|
||||||
model.
|
|
||||||
- Prove an absent operator match falls through, while a malformed matching
|
|
||||||
operator definition fails without using the embedded profile.
|
|
||||||
- Prove a selected Promptkit built-in that is absent from both higher layers
|
|
||||||
still resolves.
|
|
||||||
- Prove an explicit in-memory test profile retains highest precedence.
|
|
||||||
- Cover both local override forms required by the roadmap: an endpoint-only
|
|
||||||
OpenAI-compatible `weather-light` profile and a `backend: local` profile
|
|
||||||
using the configured local endpoint. No test may contact either endpoint.
|
|
||||||
- Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./internal/adapters/promptkit
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Inspection and prepared execution use Promptkit's exact four-layer precedence,
|
|
||||||
operator errors remain visible, and local overrides require no prompt or code
|
|
||||||
changes.
|
|
||||||
|
|
||||||
## Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Move operational prompts from provider-oriented defaults to the three-tier
|
|
||||||
Weatherreporter policy with an exact, synchronized version transition.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Change Hourly's `default_profile` to `weather-light`.
|
|
||||||
2. Change Daily, Today, and Tomorrow to `weather-balanced`.
|
|
||||||
3. Advance the exact version in all four prompt YAML assets from `1.0.1` to
|
|
||||||
`1.1.0` without changing prompt text or generated-text schemas solely for
|
|
||||||
this feature.
|
|
||||||
4. Advance the four matching report-registry prompt versions to `1.1.0` in the
|
|
||||||
same change. Keep prompt IDs, report IDs, modules, periods, templates, and
|
|
||||||
output contracts unchanged.
|
|
||||||
5. Update fixtures and expectations that intentionally assert the current
|
|
||||||
prompt contract. Do not rewrite historical fixture versions or weaken tests
|
|
||||||
that protect actual compatibility.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Update asset and report-registry contract tests to require exact version
|
|
||||||
`1.1.0` and the report-to-profile assignments locked in this plan.
|
|
||||||
- Inspect every exact prompt version through the real embedded prompt, schema,
|
|
||||||
and fallback-profile filesystems.
|
|
||||||
- Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve
|
|
||||||
Gemini Flash Latest, and `weather-deep` remains inspectable but unassigned.
|
|
||||||
- Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Every operational prompt and registry definition agrees on exact version
|
|
||||||
`1.1.0`, selects its intended logical tier, and resolves its expected effective
|
|
||||||
model offline.
|
|
||||||
|
|
||||||
## Stage 4: Verify Application Selection And Batch Reuse
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Protect the assembled application behavior created by the new defaults and
|
|
||||||
confirm that logical identity is not lost during active effective-model
|
|
||||||
resolution.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Preserve the current pre-collection inspection order and fail-fast behavior
|
|
||||||
for missing credentials, unknown profiles, malformed profiles, and unusable
|
|
||||||
backends.
|
|
||||||
2. Preserve the global `promptkit.profile` all-report override. Do not add a
|
|
||||||
second override mechanism or report-specific configuration fields.
|
|
||||||
3. Preserve batch preflight deduplication by selected effective profile ID:
|
|
||||||
Today and Tomorrow in the same batch should inspect their shared
|
|
||||||
`weather-balanced` selection once.
|
|
||||||
4. Preserve the selected logical profile ID and resolved backend/model through
|
|
||||||
active inspection, preparation, and execution using the existing
|
|
||||||
project-owned contract. Do not add state fields, expand persisted parameter
|
|
||||||
detail, or create a new historical compatibility guarantee.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Add or update representative app tests for default Hourly and day-scale
|
|
||||||
selection, a global-profile override, and a morning/evening batch sharing
|
|
||||||
`weather-balanced`.
|
|
||||||
- Assert inspection completes before weather collection and provider
|
|
||||||
generation, including malformed same-ID operator overrides.
|
|
||||||
- Assert active inspection and execution expose the logical profile ID and
|
|
||||||
effective model for both embedded and overridden profiles.
|
|
||||||
- Assert endpoints and credentials remain absent from errors, summaries,
|
|
||||||
normal logs, and ordinary state.
|
|
||||||
- Use project-owned executor fakes or Promptkit provider fakes; do not make live
|
|
||||||
provider calls.
|
|
||||||
- Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./internal/app ./internal/cli
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Single-report and batch workflows select the intended tier, retain existing
|
|
||||||
override and preflight behavior, deduplicate shared batch inspection, and
|
|
||||||
preserve safe logical and effective model information during active execution
|
|
||||||
without adding a durable-provenance contract.
|
|
||||||
|
|
||||||
## Stage 5: Publish Canonical Operator And Maintainer Documentation
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Document the implemented feature once in each canonical owner and provide one
|
|
||||||
maintained, copyable local override example.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Update `docs/config.md` to explain global profile selection versus
|
|
||||||
`profile_file`/`profile_dir` definition lookup and link to the maintained
|
|
||||||
example. Keep the field reference in this canonical document.
|
|
||||||
2. Update the Promptkit integration document with the logical profile catalog,
|
|
||||||
source precedence, exact prompt-version relationship, and safe active
|
|
||||||
inspection and execution contract. Avoid restating complete configuration
|
|
||||||
syntax or presenting transitional persistence as the target architecture.
|
|
||||||
3. Update the report-registry, Promptkit adapter, app-orchestration, and state
|
|
||||||
internal documents only where their implemented contracts changed.
|
|
||||||
4. Update `docs/operations.md` with the normal local-override workflow and
|
|
||||||
`docs/troubleshooting.md` with malformed override, unavailable local
|
|
||||||
endpoint, missing credential, and unexpected effective-model diagnostics.
|
|
||||||
5. Add or update one secret-free file under `examples/` showing a
|
|
||||||
`weather-light` override for a local OpenAI-compatible endpoint. Choose one
|
|
||||||
supported form as the complete example and mention the other form only in
|
|
||||||
its canonical reference.
|
|
||||||
6. Update the architecture policy only if implementation changed a normative
|
|
||||||
boundary or invariant. Do not add future behavior to current-state docs.
|
|
||||||
7. Keep the feature roadmap and this plan in their pre-implementation statuses
|
|
||||||
until the final repository gate passes. Do not create release notes before
|
|
||||||
a release version is chosen.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Verify every changed repository-relative link and every profile/model ID.
|
|
||||||
- Validate maintained YAML examples through the same strict configuration or
|
|
||||||
Promptkit profile path used by production where practical.
|
|
||||||
- Run the focused tests that own any executable examples, followed by:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
Users, operators, and maintainers can discover the tier defaults, precedence,
|
|
||||||
global override, local override, and failure behavior without duplicated or
|
|
||||||
future-state documentation.
|
|
||||||
|
|
||||||
## Stage 6: Complete Repository Verification And Roadmap Handoff
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Demonstrate that the complete feature is coherent, offline-testable, and ready
|
|
||||||
for review and a later release decision.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Review the complete diff against the roadmap, this plan, and all three
|
|
||||||
policy documents. Remove stale identifiers, temporary helpers, redundant
|
|
||||||
tests, and documentation duplication.
|
|
||||||
2. Confirm `go.mod` and `go.sum` retain tagged Promptkit v0.5.0 without a local
|
|
||||||
replacement or dependency drift.
|
|
||||||
3. Confirm only the four supported report products exist and no retired report
|
|
||||||
surfaces were reintroduced.
|
|
||||||
4. Confirm the roadmap's completion criteria one by one. Change its status to
|
|
||||||
implemented and this plan's status to completed only after every criterion
|
|
||||||
and command below passes.
|
|
||||||
5. Do not require a live provider for completion. If credentials and network
|
|
||||||
access are deliberately supplied by an operator, record live smoke results
|
|
||||||
separately as release-candidate evidence rather than adding them to the
|
|
||||||
default suite.
|
|
||||||
|
|
||||||
### Verification
|
|
||||||
|
|
||||||
Run `gofmt -w` on every changed Go file, then run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go test ./...
|
|
||||||
go test -race ./...
|
|
||||||
go run ./cmd/weatherreporter --help
|
|
||||||
git diff --check
|
|
||||||
git status --short
|
|
||||||
```
|
|
||||||
|
|
||||||
Also inspect all three logical profiles through the application's normal
|
|
||||||
preflight path using offline provider doubles, including one same-ID local
|
|
||||||
override and one explicit global override.
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
All roadmap completion criteria are satisfied, all verification commands pass,
|
|
||||||
the working tree contains only intentional changes, and the canonical
|
|
||||||
documentation describes the implemented state. The feature is ready for code
|
|
||||||
review and release preparation.
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
None. The model identifiers, profile settings, report assignments, version
|
|
||||||
transition, precedence, compatibility behavior, test boundaries, and
|
|
||||||
documentation ownership are decision-complete.
|
|
||||||
@@ -1,555 +1,373 @@
|
|||||||
# Promptkit Migration Implementation Plan
|
# Domain-Specific Prompt Profiles Implementation Plan
|
||||||
|
|
||||||
Status: Completed; Stages 1–19 passed their exit gates.
|
Status: Ready for implementation.
|
||||||
|
|
||||||
## Purpose And Authority
|
## Purpose And Authority
|
||||||
|
|
||||||
This document records the completed implementation of the
|
This document gives a coding agent the ordered work needed to implement the
|
||||||
[Promptkit migration roadmap](promptkit.md) and its post-implementation audit
|
[domain-specific prompt profiles roadmap](domain-profiles.md). The roadmap is
|
||||||
remediation. The feature roadmap records scope, user intent, policy choices,
|
authoritative for scope, user intent, policy choices, and the desired end
|
||||||
and the implemented end state. This plan records implementation sequence,
|
state. This plan is authoritative for implementation sequence, verification,
|
||||||
tests, and completion gates.
|
and stage exit gates.
|
||||||
|
|
||||||
Stages 12–19 were completed in order. They fixed additional defects exposed by
|
The target implementer is a `gpt-5.6-terra` coding agent using high reasoning.
|
||||||
their required tests only when those defects were within the same stated
|
Complete the stages in order. Each stage is scoped for one implementation
|
||||||
contract; they did not add new product behavior or reinterpret roadmap
|
prompt and must leave the repository coherent, tested, and reviewable before
|
||||||
decisions.
|
the next stage begins.
|
||||||
|
|
||||||
This plan follows the repository's
|
This plan follows the repository's
|
||||||
[architecture](../policy/architecture.md),
|
[architecture](../policy/architecture.md),
|
||||||
[documentation](../policy/documentation.md), and
|
[documentation](../policy/documentation.md), and
|
||||||
[testing](../policy/testing.md) policies.
|
[testing](../policy/testing.md) policies.
|
||||||
|
|
||||||
|
## Completed Prerequisite
|
||||||
|
|
||||||
|
Weatherreporter is already pinned to Promptkit v0.5.0. That release provides
|
||||||
|
the public `WithFallbackProfileFS` option and the required precedence across
|
||||||
|
inspection, preparation, and execution. The dependency upgrade passed
|
||||||
|
Weatherreporter's full offline test suite, race-enabled suite, CLI help check,
|
||||||
|
and an operator `generate hourly` smoke test. Do not repeat or replace the
|
||||||
|
dependency upgrade as part of these stages.
|
||||||
|
|
||||||
|
## Locked Product Decisions
|
||||||
|
|
||||||
|
Implement these exact Weatherreporter-owned profiles:
|
||||||
|
|
||||||
|
| Profile ID | Backend | Model | Reasoning effort | Timeout | Service tier |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| `weather-light` | `openrouter` | `deepseek/deepseek-v4-flash` | Omitted | 180 seconds | `flex` |
|
||||||
|
| `weather-balanced` | `openrouter` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||||
|
| `weather-deep` | `openrouter` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||||
|
|
||||||
|
Assign Hourly to `weather-light`; assign Daily, Today, and Tomorrow to
|
||||||
|
`weather-balanced`; assign no report to `weather-deep` initially. Advance all
|
||||||
|
four prompt definitions and matching report-registry entries from `1.0.1` to
|
||||||
|
`1.1.0` when their defaults change.
|
||||||
|
|
||||||
|
The leading `~` in the Gemini and Claude model IDs is required and denotes an
|
||||||
|
OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a
|
||||||
|
dated model version. Do not add temperature, `top_p`, maximum-token, endpoint,
|
||||||
|
or credential fields to the embedded definitions.
|
||||||
|
|
||||||
|
Prompt preparation and execution artifacts written at `1.0.1` are not required
|
||||||
|
to remain readable after the transition to `1.1.0`. Do not add a migration,
|
||||||
|
compatibility shim, or weaker historical-artifact validation for this feature.
|
||||||
|
|
||||||
|
Definition lookup must remain:
|
||||||
|
|
||||||
|
1. explicit Promptkit in-memory profiles used by tests or an embedding
|
||||||
|
consumer;
|
||||||
|
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
||||||
|
3. Weatherreporter's embedded fallback profiles; and
|
||||||
|
4. Promptkit's built-in catalog.
|
||||||
|
|
||||||
|
Selection remains a separate concern: a nonblank global `promptkit.profile`
|
||||||
|
selects the profile for every report in the invocation; otherwise the exact
|
||||||
|
prompt definition's `default_profile` selects it. A malformed matching
|
||||||
|
higher-precedence profile is an error and never falls through.
|
||||||
|
|
||||||
## Continuing Invariants
|
## Continuing Invariants
|
||||||
|
|
||||||
- Keep `gitea.maximumdirect.net/eric/promptkit` pinned at exactly `v0.4.0`.
|
- Keep all Promptkit types and mechanics inside
|
||||||
- Keep Promptkit types inside `internal/adapters/promptkit`, its tests, and the
|
`internal/adapters/promptkit`, its focused tests, and asset contract tests.
|
||||||
external prompt-asset contract test.
|
- Keep prompt inspection before weather collection and provider work.
|
||||||
- Preserve one Promptkit engine per `generate` or `run` invocation and one
|
- Keep one Promptkit engine per command action and one shared engine across a
|
||||||
shared engine for every sequential report in a batch.
|
sequential batch.
|
||||||
- Preserve exact prompt version `1.0.0`, the exact persisted YAML data-package
|
- Preserve logical profile ID and effective backend/model information through
|
||||||
bytes, prepared execution, and preparation persistence before provider work.
|
active inspection and execution where the project-owned contract already
|
||||||
- Do not add retries, repair attempts, concurrent batch generation, direct
|
exposes it. Do not add new durable-provenance fields or compatibility
|
||||||
Markdown generation, arbitrary backend registration, or live-provider
|
guarantees.
|
||||||
tests.
|
- Leave existing workspace persistence behavior otherwise unchanged. The
|
||||||
- Keep ordinary artifacts, errors, logs, and summaries free of credentials,
|
accepted [ephemeral-state roadmap](ephemeral-state.md) owns its future
|
||||||
rendered messages, schemas, input bodies, generated bodies, endpoints, and
|
removal and must not be partially implemented here.
|
||||||
full effective parameter maps.
|
- Do not expose endpoints, credentials, rendered messages, schemas, request
|
||||||
- Keep sensitive debug artifacts opt-in, outside normal state, owner-only,
|
bodies, response bodies, or complete parameter maps through ordinary errors,
|
||||||
atomic, and free of credentials.
|
logs, summaries, or state.
|
||||||
- Treat an artifact path as reached only after the corresponding write or copy
|
- Keep the default suite deterministic, offline, and credential-free.
|
||||||
succeeds. Never persist or summarize a merely derivable future path.
|
- Do not add endpoint discovery, health probing, provider failover, retries at
|
||||||
- Preserve every safe reached path in partial app and CLI results even when a
|
a more expensive tier, profile merging, per-report configuration fields, or
|
||||||
later persistence, validation, rendering, copy, or notification step fails.
|
severity-driven model selection.
|
||||||
- Keep v1 metadata read compatibility and write only v2 metadata for new runs.
|
- Update canonical current-state documentation only in the stage where the
|
||||||
- Keep the default test suite deterministic, offline, and credential-free.
|
corresponding behavior becomes implemented.
|
||||||
- Run `git diff --check` before completing every stage. Run the full repository
|
- Run `git diff --check` before completing every stage.
|
||||||
gate in Stage 19.
|
|
||||||
|
|
||||||
## Completed Migration Summary
|
## Stage 1: Add The Embedded Weather Profile Catalog
|
||||||
|
|
||||||
Stages 1–11 are implemented and committed. They remain summarized here to
|
|
||||||
preserve the history and dependencies of the follow-up work.
|
|
||||||
|
|
||||||
| Stage | Completed outcome |
|
|
||||||
| --- | --- |
|
|
||||||
| 1 | Removed the unfinished three-day, weekend, and storm product surfaces and retained Daily, Today, Tomorrow, and Hourly with exact prompt version `1.0.0`. |
|
|
||||||
| 2 | Promoted the four operational prompts and canonical schemas into the embedded `internal/promptassets` source used by Promptkit and generated-text validation. |
|
|
||||||
| 3 | Added the project-owned `internal/promptexec` inspection, preparation, execution, validation, debug, and error contract. |
|
|
||||||
| 4 | Added the Promptkit v0.4.0 adapter with prepared execution, explicit value mapping, safe error classification, and offline model-client tests. |
|
|
||||||
| 5 | Added Promptkit-era preparation and execution artifacts, metadata v2, new paths, and v1 decoding support. |
|
|
||||||
| 6 | Added explicitly rooted, permission-restricted, atomic LLM debug persistence. |
|
|
||||||
| 7 | Added Promptkit configuration, executor composition, and pre-collection prompt/profile/credential inspection. |
|
|
||||||
| 8 | Cut single-report generation over to prepared Promptkit execution and v2 persistence. |
|
|
||||||
| 9 | Added `--llm-debug-dir` and Promptkit-era single-report summary fields. |
|
|
||||||
| 10 | Cut morning and evening batches over to one shared Promptkit executor and removed Scriptorium code, configuration, and dependency metadata. |
|
|
||||||
| 11 | Updated canonical Promptkit documentation, removed the temporary Scriptorium corpus, and ran the available repository checks. |
|
|
||||||
|
|
||||||
The post-implementation audit confirmed the principal dependency and package
|
|
||||||
boundaries, but found incorrect reached-path bookkeeping, incomplete execution
|
|
||||||
artifact updates, insufficient artifact validation, extensive loss of
|
|
||||||
behavioral tests during the final cutover, and roadmap lifecycle text that was
|
|
||||||
not finalized. The completed remediation addressed those findings without
|
|
||||||
changing the intended feature scope.
|
|
||||||
|
|
||||||
| Stage | Completed outcome |
|
|
||||||
| --- | --- |
|
|
||||||
| 12 | Corrected reached-path bookkeeping across metadata, app results, batch items, and CLI summaries. |
|
|
||||||
| 13 | Hardened Promptkit-era durable state validation and restored v1/v2 state coverage. |
|
|
||||||
| 14 | Recorded every downstream path reached after completed prompt execution. |
|
|
||||||
| 15 | Restored assembled single-report behavioral and failure coverage. |
|
|
||||||
| 16 | Simplified prompt-generation orchestration while preserving behavior. |
|
|
||||||
| 17 | Restored assembled batch, planning, artifact, and notification coverage. |
|
|
||||||
| 18 | Restored supported CLI, summary, safety, and historical inspection coverage. |
|
|
||||||
| 19 | Reconciled canonical documentation and passed the complete repository verification gate. |
|
|
||||||
|
|
||||||
## Stage 12: Correct Reached-Artifact Bookkeeping
|
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Make metadata, app results, batch items, and CLI summaries truthful at every
|
Create one repository-owned, embedded profile source containing exactly the
|
||||||
failure boundary: a nonblank path means that artifact was successfully
|
three locked logical profiles.
|
||||||
created.
|
|
||||||
|
|
||||||
### Work
|
### Work
|
||||||
|
|
||||||
1. Change `state.BuildPromptMetadataFromBriefingMetadata` so it initializes
|
1. Add strict YAML profile assets beneath `internal/promptassets` using the
|
||||||
identity, schema, metadata destination, and only artifacts already saved at
|
exact IDs and definitions in this plan.
|
||||||
the call site. It must not prepopulate raw-output, normalized-text,
|
2. Extend `internal/promptassets` with a narrowly named accessor that returns
|
||||||
render-context, managed-report, preparation, execution, notification, or
|
the embedded profile `fs.FS`. Follow the existing prompt and schema asset
|
||||||
output-copy paths.
|
pattern without exposing Promptkit types from the package.
|
||||||
2. In `generatePromptReport`, assign each metadata and `ReportResult` path
|
3. Keep profile filenames and embed layout simple and deterministic. Do not
|
||||||
immediately after that artifact write succeeds and before attempting the
|
duplicate Promptkit's built-in directory taxonomy unless the application
|
||||||
next write. In particular:
|
assets require it.
|
||||||
|
4. Validate the assets through Promptkit's public engine/profile inspection
|
||||||
- do not initialize `ReportResult.ReportPath` from `Store.Paths`;
|
surface rather than adding a second YAML parser or a Weatherreporter-owned
|
||||||
- record a saved failed-preparation receipt in the result before saving
|
profile representation.
|
||||||
metadata;
|
|
||||||
- record a saved failed or completed execution receipt before saving
|
|
||||||
metadata;
|
|
||||||
- retain raw, normalized, context, report, copy, and notification paths
|
|
||||||
when a later step fails; and
|
|
||||||
- keep `MetadataPath` unchanged when a metadata rewrite fails, because the
|
|
||||||
prior successfully written metadata record remains the reached version.
|
|
||||||
|
|
||||||
3. Remove batch-item prepopulation from derived `Store.Paths` values.
|
|
||||||
`BatchReportResult` receives paths only from the returned `ReportResult` or
|
|
||||||
from a write that the batch itself successfully completed.
|
|
||||||
4. Preserve current CLI field names and omission behavior. Human and JSON
|
|
||||||
summaries must omit every unreached path.
|
|
||||||
5. Do not change artifact locations, filenames, schemas, report output, or
|
|
||||||
notification policy in this stage.
|
|
||||||
|
|
||||||
### Tests
|
### Tests
|
||||||
|
|
||||||
- Add focused app tests for one representative report using real temporary
|
- Extend the asset contract tests to assert exactly the three logical IDs,
|
||||||
state plus a narrow failure-injecting store wrapper.
|
their exact effective model IDs, and the intentional parameters.
|
||||||
- Fail the next persistence step immediately after a successful preparation
|
- Prove all three profiles inspect successfully offline when supplied as a
|
||||||
receipt, execution receipt, raw output, normalized output, render context,
|
fallback source and no operator source is present.
|
||||||
managed report, output copy, and notification artifact; assert that the
|
- Assert that the catalog contains no endpoints, credentials, temperature,
|
||||||
returned result contains every reached path and no future path.
|
`top_p`, or maximum-token settings.
|
||||||
- Include one preparation failure, one operational execution failure, and one
|
|
||||||
completed validation rejection to cover the three execution outcome shapes.
|
|
||||||
- Add batch and CLI summary assertions proving unreached paths are omitted.
|
|
||||||
- Run:
|
- Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go test ./internal/state ./internal/app ./internal/cli
|
go test ./internal/promptassets
|
||||||
git diff --check
|
git diff --check
|
||||||
```
|
```
|
||||||
|
|
||||||
### Exit Gate
|
### Exit Gate
|
||||||
|
|
||||||
Every nonblank path in newly written metadata, app results, batch items, and
|
The embedded catalog is complete, strictly valid, safe, and independently
|
||||||
CLI summaries names an artifact that exists. Every safe artifact successfully
|
inspectable through Promptkit v0.5.0's public API.
|
||||||
written before a later failure remains discoverable from the returned partial
|
|
||||||
result.
|
|
||||||
|
|
||||||
## Stage 13: Harden Durable State Contracts And Restore State Coverage
|
## Stage 2: Wire Fallback Resolution And Protect Precedence
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Make the v1/v2 wire boundary and Promptkit-era artifact validation explicit,
|
Supply the embedded catalog through Promptkit's application fallback layer
|
||||||
strict, and durably tested.
|
without changing existing operator configuration or application boundaries.
|
||||||
|
|
||||||
### Work
|
### Work
|
||||||
|
|
||||||
1. Strengthen `PromptPreparationArtifact.Validate`:
|
1. Add `promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".")` to
|
||||||
|
normal adapter engine construction.
|
||||||
- require report ID, Weatherreporter RunID, prompt ID, exact prompt version,
|
2. Preserve existing `profile_file`, `profile_dir`, configured local backend,
|
||||||
data-package path, nonzero start/end times, nonnegative duration, and an
|
timeout, prompt filesystem, schema filesystem, and test-option behavior.
|
||||||
end not earlier than the start;
|
3. Ensure ordinary production construction and the adapter's test
|
||||||
- for success, require preparation provenance, prohibit an error, and
|
construction path exercise the same fallback wiring. Test-only explicit
|
||||||
require its prompt ID/version and data-package path to match the top-level
|
profiles may retain Promptkit's documented highest precedence.
|
||||||
artifact;
|
4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays,
|
||||||
- for failure, require a classified bounded error and prohibit fabricated
|
existence checks, YAML parsing, or merge behavior to Weatherreporter.
|
||||||
preparation provenance.
|
|
||||||
|
|
||||||
2. Strengthen `PromptExecutionArtifact.Validate`:
|
|
||||||
|
|
||||||
- require report ID, Weatherreporter RunID, prompt ID, exact prompt version,
|
|
||||||
nonzero start/end times, nonnegative duration, and an end not earlier than
|
|
||||||
the start;
|
|
||||||
- for success and validation rejection, require provenance and completed
|
|
||||||
validation, prohibit an operational error, and require the provenance
|
|
||||||
prompt ID/version to match the artifact;
|
|
||||||
- do not compare the provenance RunID with the Weatherreporter RunID because
|
|
||||||
the provenance value is Promptkit's run identity;
|
|
||||||
- for operational failure, require a classified bounded error and prohibit
|
|
||||||
invented provenance or completed validation.
|
|
||||||
|
|
||||||
3. Validate required provenance fields for completed executions, including
|
|
||||||
Promptkit RunID, prompt and rendered hashes, selected profile/backend/model,
|
|
||||||
and data-package path. Permit usage counters and generated hash to be zero
|
|
||||||
when the provider legitimately reports no value.
|
|
||||||
4. Restore focused filesystem and metadata tests for:
|
|
||||||
|
|
||||||
- exact v2 paths and filenames;
|
|
||||||
- preparation/execution round trips and required fields;
|
|
||||||
- metadata v2 round trips without legacy aliases;
|
|
||||||
- v1 decoding, normalized internal aliases, and v1-preserving re-marshaling;
|
|
||||||
- unknown schema rejection;
|
|
||||||
- report listing, RunID lookup, source/module/data-package inspection, and
|
|
||||||
retained v1 behavior for historical report IDs;
|
|
||||||
- atomic writes and unsafe workspace/path rejection; and
|
|
||||||
- prior-snapshot behavior for the four supported report IDs.
|
|
||||||
|
|
||||||
5. Adapt useful tests from the deleted filesystem suite rather than recreating
|
|
||||||
redundant low-value cases. Do not restore Scriptorium writes or retired
|
|
||||||
report behavior.
|
|
||||||
|
|
||||||
### Tests
|
### Tests
|
||||||
|
|
||||||
Run:
|
- At the adapter boundary, prove fallback-only inspection of all three
|
||||||
|
Weatherreporter profiles.
|
||||||
```sh
|
- Prove same-ID overrides through both configured `profile_file` and
|
||||||
go test ./internal/state ./internal/app
|
`profile_dir`, including resolution of the override's effective backend and
|
||||||
git diff --check
|
model.
|
||||||
```
|
- Prove an absent operator match falls through, while a malformed matching
|
||||||
|
operator definition fails without using the embedded profile.
|
||||||
### Exit Gate
|
- Prove a selected Promptkit built-in that is absent from both higher layers
|
||||||
|
still resolves.
|
||||||
The state package rejects incomplete or contradictory Promptkit-era artifacts,
|
- Prove an explicit in-memory test profile retains highest precedence.
|
||||||
reads historical v1 records, writes only valid v2 records, and has focused
|
- Cover both local override forms required by the roadmap: an endpoint-only
|
||||||
offline coverage for its durable compatibility and filesystem contracts.
|
OpenAI-compatible `weather-light` profile and a `backend: local` profile
|
||||||
|
using the configured local endpoint. No test may contact either endpoint.
|
||||||
## Stage 14: Complete Execution-Artifact Path Tracking
|
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Make `PromptExecutionArtifact.Paths` accurately record every downstream
|
|
||||||
artifact reached after a completed Promptkit run.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Treat the execution artifact as an atomically updated durable record of the
|
|
||||||
completed Promptkit execution and subsequent artifact destinations. Its
|
|
||||||
status, provenance, validation, usage, and timing remain the provider-run
|
|
||||||
outcome; later application failures do not change a successful Promptkit
|
|
||||||
status into an execution failure.
|
|
||||||
2. Save the initial execution artifact after raw output is persisted, with
|
|
||||||
`RawOutputPath` populated.
|
|
||||||
3. After each later successful write, update and atomically resave the same
|
|
||||||
execution artifact with the corresponding reached path:
|
|
||||||
|
|
||||||
- normalized generated text;
|
|
||||||
- render context;
|
|
||||||
- managed Markdown report;
|
|
||||||
- an explicitly requested extra output copy, only after the copy succeeds;
|
|
||||||
and
|
|
||||||
- a Distributor notification artifact, including a persisted failure or
|
|
||||||
status artifact when notification produced one.
|
|
||||||
|
|
||||||
4. Keep metadata and execution-artifact path values consistent after every
|
|
||||||
successful checkpoint. Save the execution artifact before metadata so a
|
|
||||||
metadata failure does not erase knowledge of a reached downstream artifact.
|
|
||||||
Failure to update the execution artifact is terminal and returns a partial
|
|
||||||
result containing the downstream artifact that was already written.
|
|
||||||
5. Refactor finalization return values only as needed to tell the orchestration
|
|
||||||
layer which copy and notification paths were actually written. Distributor
|
|
||||||
must continue uploading only the managed Markdown report.
|
|
||||||
6. A validation-rejected execution ends after raw output and therefore records
|
|
||||||
only the raw-output path. An operational execution failure has no completed
|
|
||||||
provenance and records only safe paths reached before that failure.
|
|
||||||
|
|
||||||
### Tests
|
|
||||||
|
|
||||||
- Add table-driven execution-artifact lifecycle tests for success and every
|
|
||||||
downstream failure point.
|
|
||||||
- Load the persisted execution artifact after normalized-text, context,
|
|
||||||
template, copy, metadata, and notification failures and assert its status and
|
|
||||||
exact reached paths.
|
|
||||||
- Assert that execution artifacts never contain generated bodies, rendered
|
|
||||||
prompts, schemas, endpoints, parameters, or credentials.
|
|
||||||
- Run:
|
- Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go test ./internal/state ./internal/app
|
go test ./internal/adapters/promptkit
|
||||||
git diff --check
|
git diff --check
|
||||||
```
|
```
|
||||||
|
|
||||||
### Exit Gate
|
### Exit Gate
|
||||||
|
|
||||||
For every completed Promptkit run, its execution artifact contains exactly the
|
Inspection and prepared execution use Promptkit's exact four-layer precedence,
|
||||||
safe downstream paths reached by the workflow and remains semantically correct
|
operator errors remain visible, and local overrides require no prompt or code
|
||||||
when a later application stage fails.
|
changes.
|
||||||
|
|
||||||
## Stage 15: Restore Single-Report Behavioral Coverage
|
## Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Restore the risk-based application coverage removed during final cutover and
|
Move operational prompts from provider-oriented defaults to the three-tier
|
||||||
prove the complete single-report Promptkit workflow through project-owned
|
Weatherreporter policy with an exact, synchronized version transition.
|
||||||
boundaries.
|
|
||||||
|
|
||||||
### Work
|
### Work
|
||||||
|
|
||||||
1. Reintroduce a focused app test harness using real state, prompt-input,
|
1. Change Hourly's `default_profile` to `weather-light`.
|
||||||
generated-text validation, render contexts, and templates with deterministic
|
2. Change Daily, Today, and Tomorrow to `weather-balanced`.
|
||||||
collector, executor, notifier, clock, and filesystem boundaries.
|
3. Advance the exact version in all four prompt YAML assets from `1.0.1` to
|
||||||
2. Add representative successful workflows for Daily, Today, Tomorrow, and
|
`1.1.0` without changing prompt text or generated-text schemas solely for
|
||||||
Hourly. Verify report identity, exact prompt version, one collection, exact
|
this feature.
|
||||||
persisted YAML bytes passed to the executor, expected template output,
|
4. Advance the four matching report-registry prompt versions to `1.1.0` in the
|
||||||
optional copy behavior, and managed-report notification source.
|
same change. Keep prompt IDs, report IDs, modules, periods, templates, and
|
||||||
3. Cover the required failure matrix:
|
output contracts unchanged.
|
||||||
|
5. Update fixtures and expectations that intentionally assert the current
|
||||||
- inspection and missing credentials before collection;
|
prompt contract. Do not rewrite historical fixture versions or weaken tests
|
||||||
- preparation failure and callback persistence failure before provider work;
|
that protect actual compatibility.
|
||||||
- execution-time credential disappearance;
|
|
||||||
- capacity rejection without retry;
|
|
||||||
- cancellation and deadline;
|
|
||||||
- generation and operational-validation failure;
|
|
||||||
- completed Promptkit schema rejection with retained raw output;
|
|
||||||
- generated-text decode/domain rejection;
|
|
||||||
- render-context and template failure;
|
|
||||||
- output-copy failure; and
|
|
||||||
- notification failure.
|
|
||||||
|
|
||||||
4. Verify preparation persistence precedes provider execution, debug-write
|
|
||||||
failure prevents provider execution, and execution-debug failure preserves
|
|
||||||
previously reached normal and debug artifacts.
|
|
||||||
5. Verify Recent Changes, prior-snapshot selection, output naming, and
|
|
||||||
Distributor template values for all four retained reports.
|
|
||||||
6. Adapt useful tests from the deleted app suite. Omit Scriptorium mechanics,
|
|
||||||
subprocess interaction assertions, and retired report products.
|
|
||||||
7. Fix defects exposed by these tests only when the expected behavior is
|
|
||||||
already decided by the roadmap or canonical policy. Record any new product
|
|
||||||
question instead of silently choosing it.
|
|
||||||
|
|
||||||
### Tests
|
### Tests
|
||||||
|
|
||||||
Run:
|
- Update asset and report-registry contract tests to require exact version
|
||||||
|
`1.1.0` and the report-to-profile assignments locked in this plan.
|
||||||
|
- Inspect every exact prompt version through the real embedded prompt, schema,
|
||||||
|
and fallback-profile filesystems.
|
||||||
|
- Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve
|
||||||
|
Gemini Flash Latest, and `weather-deep` remains inspectable but unassigned.
|
||||||
|
- Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go test ./internal/app
|
go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit
|
||||||
go test -race ./internal/app ./internal/adapters/promptkit
|
git diff --check
|
||||||
git diff --check
|
```
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
### Exit Gate
|
||||||
|
|
||||||
The single-report workflow has deterministic behavioral coverage for all four
|
Every operational prompt and registry definition agrees on exact version
|
||||||
reports, all consequential failure stages, artifact ordering, partial results,
|
`1.1.0`, selects its intended logical tier, and resolves its expected effective
|
||||||
debug isolation, output copying, and notification behavior.
|
model offline.
|
||||||
|
|
||||||
## Stage 16: Refactor Prompt Generation Orchestration
|
## Stage 4: Verify Application Selection And Batch Reuse
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Reduce the complexity and duplicated persistence logic in
|
Protect the assembled application behavior created by the new defaults and
|
||||||
`generatePromptReport` without changing observable behavior.
|
confirm that logical identity is not lost during active effective-model
|
||||||
|
resolution.
|
||||||
|
|
||||||
### Work
|
### Work
|
||||||
|
|
||||||
1. Use the Stage 12–15 tests as the refactoring safety boundary. Do not weaken
|
1. Preserve the current pre-collection inspection order and fail-fast behavior
|
||||||
assertions to accommodate structural changes.
|
for missing credentials, unknown profiles, malformed profiles, and unusable
|
||||||
2. Split the current orchestration into small app-owned operations with clear
|
backends.
|
||||||
inputs and outcomes for:
|
2. Preserve the global `promptkit.profile` all-report override. Do not add a
|
||||||
|
second override mechanism or report-specific configuration fields.
|
||||||
- deterministic input and initial state construction;
|
3. Preserve batch preflight deduplication by selected effective profile ID:
|
||||||
- preparation callback persistence;
|
Today and Tomorrow in the same batch should inspect their shared
|
||||||
- preparation-failure persistence;
|
`weather-balanced` selection once.
|
||||||
- operational-execution-failure persistence;
|
4. Preserve the selected logical profile ID and resolved backend/model through
|
||||||
- completed execution and raw-output persistence;
|
active inspection, preparation, and execution using the existing
|
||||||
- normalized text and render-context persistence;
|
project-owned contract. Do not add state fields, expand persisted parameter
|
||||||
- managed report, optional copy, metadata, and notification finalization;
|
detail, or create a new historical compatibility guarantee.
|
||||||
and
|
|
||||||
- reached-path updates shared by success and failure paths.
|
|
||||||
|
|
||||||
3. Keep workflow order visible in one coordinator. Do not introduce a generic
|
|
||||||
workflow engine, hidden retry loop, provider-specific app type, or mutable
|
|
||||||
global state.
|
|
||||||
4. Centralize the repeated rule that a successful artifact write updates the
|
|
||||||
result before any following write can fail.
|
|
||||||
5. Preserve error identities, safe text, atomic writes, exact bytes, debug
|
|
||||||
ordering, partial results, and notification behavior.
|
|
||||||
|
|
||||||
### Tests
|
### Tests
|
||||||
|
|
||||||
Run:
|
- Add or update representative app tests for default Hourly and day-scale
|
||||||
|
selection, a global-profile override, and a morning/evening batch sharing
|
||||||
|
`weather-balanced`.
|
||||||
|
- Assert inspection completes before weather collection and provider
|
||||||
|
generation, including malformed same-ID operator overrides.
|
||||||
|
- Assert active inspection and execution expose the logical profile ID and
|
||||||
|
effective model for both embedded and overridden profiles.
|
||||||
|
- Assert endpoints and credentials remain absent from errors, summaries,
|
||||||
|
normal logs, and ordinary state.
|
||||||
|
- Use project-owned executor fakes or Promptkit provider fakes; do not make live
|
||||||
|
provider calls.
|
||||||
|
- Run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
gofmt -w internal/app/*.go
|
go test ./internal/app ./internal/cli
|
||||||
go test ./internal/app ./internal/state ./internal/cli
|
git diff --check
|
||||||
go test -race ./internal/app
|
```
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
### Exit Gate
|
||||||
|
|
||||||
The top-level coordinator communicates the workflow order without containing
|
Single-report and batch workflows select the intended tier, retain existing
|
||||||
the full persistence implementation, duplicate failure branches are reduced,
|
override and preflight behavior, deduplicate shared batch inspection, and
|
||||||
and every Stage 12–15 behavioral test passes unchanged.
|
preserve safe logical and effective model information during active execution
|
||||||
|
without adding a durable-provenance contract.
|
||||||
|
|
||||||
## Stage 17: Restore Batch Behavioral Coverage
|
## Stage 5: Publish Canonical Operator And Maintainer Documentation
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Re-establish confidence that morning and evening batches preserve their
|
Document the implemented feature once in each canonical owner and provide one
|
||||||
pre-migration behavior while sharing one Promptkit executor.
|
maintained, copyable local override example.
|
||||||
|
|
||||||
### Work
|
### Work
|
||||||
|
|
||||||
1. Add assembled batch tests proving:
|
1. Update `docs/config.md` to explain global profile selection versus
|
||||||
|
`profile_file`/`profile_dir` definition lookup and link to the maintained
|
||||||
- one executor factory call and one executor per CLI invocation;
|
example. Keep the field reference in this canonical document.
|
||||||
- inspection of the full candidate set before collection;
|
2. Update the Promptkit integration document with the logical profile catalog,
|
||||||
- one weather collection;
|
source precedence, exact prompt-version relationship, and safe active
|
||||||
- existing morning/evening planning and ordering;
|
inspection and execution contract. Avoid restating complete configuration
|
||||||
- sequential execution through the shared executor;
|
syntax or presenting transitional persistence as the target architecture.
|
||||||
- continuation after an independent report failure;
|
3. Update the report-registry, Promptkit adapter, app-orchestration, and state
|
||||||
- no retry after capacity rejection;
|
internal documents only where their implemented contracts changed.
|
||||||
- distinct identities and debug directories for multiple Daily dates; and
|
4. Update `docs/operations.md` with the normal local-override workflow and
|
||||||
- exact reached paths on successful and failed batch items.
|
`docs/troubleshooting.md` with malformed override, unavailable local
|
||||||
|
endpoint, missing credential, and unexpected effective-model diagnostics.
|
||||||
2. Restore notification coverage for disabled notification, suppressed
|
5. Add or update one secret-free file under `examples/` showing a
|
||||||
per-report notification, all-success batch notification, skipped
|
`weather-light` override for a local OpenAI-compatible endpoint. Choose one
|
||||||
notification after report failure, and persisted notification failure/status
|
supported form as the complete example and mention the other form only in
|
||||||
artifacts.
|
its canonical reference.
|
||||||
3. Restore output-directory, Today/Tomorrow naming, dynamic Daily planning,
|
6. Update the architecture policy only if implementation changed a normative
|
||||||
prior-snapshot, and managed-Markdown upload-source coverage.
|
boundary or invariant. Do not add future behavior to current-state docs.
|
||||||
4. Adapt useful tests from the deleted batch portions of the app and CLI suites.
|
7. Keep the feature roadmap and this plan in their pre-implementation statuses
|
||||||
Do not restore retired report cases or Scriptorium fakes.
|
until the final repository gate passes. Do not create release notes before
|
||||||
5. Fix only roadmap-defined batch regressions exposed by the restored tests.
|
a release version is chosen.
|
||||||
|
|
||||||
### Tests
|
### Tests
|
||||||
|
|
||||||
Run:
|
- Verify every changed repository-relative link and every profile/model ID.
|
||||||
|
- Validate maintained YAML examples through the same strict configuration or
|
||||||
|
Promptkit profile path used by production where practical.
|
||||||
|
- Run the focused tests that own any executable examples, followed by:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go test ./internal/app ./internal/cli
|
git diff --check
|
||||||
go test -race ./internal/app
|
```
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
### Exit Gate
|
||||||
|
|
||||||
Morning and evening batches are covered as assembled sequential workflows and
|
Users, operators, and maintainers can discover the tier defaults, precedence,
|
||||||
demonstrably preserve collection, planning, continuation, output, debug,
|
global override, local override, and failure behavior without duplicated or
|
||||||
artifact, and notification contracts with one Promptkit executor.
|
future-state documentation.
|
||||||
|
|
||||||
## Stage 18: Restore CLI And Inspection Coverage
|
## Stage 6: Complete Repository Verification And Roadmap Handoff
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Restore the user-facing command, summary, and historical inspection contracts
|
Demonstrate that the complete feature is coherent, offline-testable, and ready
|
||||||
removed with the old root test suite.
|
for review and a later release decision.
|
||||||
|
|
||||||
### Work
|
### Work
|
||||||
|
|
||||||
1. Add parser and resolver tests for all four generate commands, both batch
|
1. Review the complete diff against the roadmap, this plan, and all three
|
||||||
commands, shared flags, report-specific date rules, malformed input,
|
policy documents. Remove stale identifiers, temporary helpers, redundant
|
||||||
`--llm-debug-dir`, `--quiet`, output paths, and rejection of retired report
|
tests, and documentation duplication.
|
||||||
names.
|
2. Confirm `go.mod` and `go.sum` retain tagged Promptkit v0.5.0 without a local
|
||||||
2. Add assembled CLI tests for representative successful single and batch
|
replacement or dependency drift.
|
||||||
invocations using injected offline boundaries. Verify exactly one executor
|
3. Confirm only the four supported report products exist and no retired report
|
||||||
construction per action.
|
surfaces were reintroduced.
|
||||||
3. Cover pre-run errors with no invented run summary, successful and failed
|
4. Confirm the roadmap's completion criteria one by one. Change its status to
|
||||||
JSON summaries, quiet-mode behavior, safe human status output, partial paths,
|
implemented and this plan's status to completed only after every criterion
|
||||||
and omission of absent notification/debug fields.
|
and command below passes.
|
||||||
4. Restore inspection tests for report listing and v1/v2 metadata, modules,
|
5. Do not require a live provider for completion. If credentials and network
|
||||||
data packages, prior snapshots, and sources. Include failed v2 runs and v1
|
access are deliberately supplied by an operator, record live smoke results
|
||||||
fixtures using historical report IDs.
|
separately as release-candidate evidence rather than adding them to the
|
||||||
5. Assert that routine output never contains rendered prompts, schema bodies,
|
default suite.
|
||||||
data packages, generated bodies, endpoints, full parameters, credentials, or
|
|
||||||
secret-like dependency errors.
|
|
||||||
6. Keep tests at stable CLI/app boundaries; do not restore assertions about
|
|
||||||
private parser formatting or Scriptorium subprocess mechanics.
|
|
||||||
|
|
||||||
### Tests
|
### Verification
|
||||||
|
|
||||||
Run:
|
Run `gofmt -w` on every changed Go file, then run:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go test ./internal/cli ./internal/app ./internal/state
|
go test ./...
|
||||||
go run ./cmd/weatherreporter --help
|
|
||||||
git diff --check
|
|
||||||
```
|
|
||||||
|
|
||||||
### Exit Gate
|
|
||||||
|
|
||||||
The supported CLI surface, summaries, quiet mode, partial failures, executor
|
|
||||||
composition, and v1/v2 inspection behavior have deterministic offline coverage.
|
|
||||||
|
|
||||||
## Stage 19: Finalize Documentation And Repository Verification
|
|
||||||
|
|
||||||
Status: Completed.
|
|
||||||
|
|
||||||
### Goal
|
|
||||||
|
|
||||||
Close the audit remediation, make roadmap lifecycle state truthful, and verify
|
|
||||||
the repository against the complete target contract.
|
|
||||||
|
|
||||||
### Work
|
|
||||||
|
|
||||||
1. Update `docs/roadmap/promptkit.md` from future tense and “unimplemented”
|
|
||||||
statuses to a completed roadmap record. Describe its old seven-report and
|
|
||||||
Scriptorium material explicitly as the pre-migration baseline rather than
|
|
||||||
current behavior.
|
|
||||||
2. Mark Stages 12–19 and this implementation plan complete only after their
|
|
||||||
exit gates pass. Retain the concise completed-stage history unless the
|
|
||||||
documentation policy calls for archival in the same change.
|
|
||||||
3. Review canonical architecture, app, state, CLI, Promptkit integration,
|
|
||||||
operations, troubleshooting, configuration, and testing documentation
|
|
||||||
against the corrected implementation. Update only actual current-state
|
|
||||||
discrepancies; do not duplicate the roadmap.
|
|
||||||
4. Search current-state code, tests, examples, help, and non-roadmap
|
|
||||||
documentation for stale Scriptorium terms, retired reports, old artifact
|
|
||||||
fields, speculative-path descriptions, or claims of missing Promptkit
|
|
||||||
implementation.
|
|
||||||
5. Confirm examples contain no credentials or private infrastructure values
|
|
||||||
and load through config tests.
|
|
||||||
|
|
||||||
### Final Verification
|
|
||||||
|
|
||||||
Run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
gofmt -w <all changed Go files>
|
|
||||||
go mod tidy
|
|
||||||
go vet ./...
|
|
||||||
go test -count=1 ./...
|
|
||||||
go test -race ./...
|
go test -race ./...
|
||||||
go run ./cmd/weatherreporter --help
|
go run ./cmd/weatherreporter --help
|
||||||
git diff --check
|
git diff --check
|
||||||
|
git status --short
|
||||||
```
|
```
|
||||||
|
|
||||||
Then verify explicitly:
|
Also inspect all three logical profiles through the application's normal
|
||||||
|
preflight path using offline provider doubles, including one same-ID local
|
||||||
- `go list -m gitea.maximumdirect.net/eric/promptkit` reports `v0.4.0`;
|
override and one explicit global override.
|
||||||
- no committed `go.work`, `replace`, secret fixture, or live-provider test
|
|
||||||
exists;
|
|
||||||
- all four prompts inspect at exact version `1.0.0`;
|
|
||||||
- no runtime prompt requests repair attempts;
|
|
||||||
- v1 fixtures remain inspectable and new runs write only v2;
|
|
||||||
- normal artifacts and output contain no sensitive prompt/debug content;
|
|
||||||
- failed-run metadata, execution artifacts, app results, batch items, and CLI
|
|
||||||
summaries contain exactly the paths actually reached;
|
|
||||||
- help exposes only Daily, Today, Tomorrow, Hourly, morning, and evening; and
|
|
||||||
- managed Markdown remains the only Distributor upload source.
|
|
||||||
|
|
||||||
### Exit Gate
|
### Exit Gate
|
||||||
|
|
||||||
Every migration and audit-remediation criterion is demonstrably satisfied,
|
All roadmap completion criteria are satisfied, all verification commands pass,
|
||||||
the restored tests protect the consequential contracts, canonical
|
the working tree contains only intentional changes, and the canonical
|
||||||
documentation describes the corrected implementation, and both roadmap
|
documentation describes the implemented state. The feature is ready for code
|
||||||
documents are marked complete.
|
review and release preparation.
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
None. The roadmap and this completed plan record the decisions used for the
|
None. The model identifiers, profile settings, report assignments, version
|
||||||
audit remediation.
|
transition, precedence, compatibility behavior, test boundaries, and
|
||||||
|
documentation ownership are decision-complete.
|
||||||
|
|||||||
@@ -0,0 +1,6 @@
|
|||||||
|
id: weather-balanced
|
||||||
|
backend: openrouter
|
||||||
|
model: "~google/gemini-flash-latest"
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 240
|
||||||
|
service_tier: flex
|
||||||
6
internal/promptassets/assets/profiles/weather-deep.yml
Normal file
6
internal/promptassets/assets/profiles/weather-deep.yml
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
id: weather-deep
|
||||||
|
backend: openrouter
|
||||||
|
model: "~anthropic/claude-sonnet-latest"
|
||||||
|
reasoning_effort: high
|
||||||
|
timeout_seconds: 240
|
||||||
|
service_tier: flex
|
||||||
5
internal/promptassets/assets/profiles/weather-light.yml
Normal file
5
internal/promptassets/assets/profiles/weather-light.yml
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
id: weather-light
|
||||||
|
backend: openrouter
|
||||||
|
model: deepseek/deepseek-v4-flash
|
||||||
|
timeout_seconds: 180
|
||||||
|
service_tier: flex
|
||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"io/fs"
|
"io/fs"
|
||||||
)
|
)
|
||||||
|
|
||||||
//go:embed assets/prompts assets/schemas
|
//go:embed assets/prompts assets/profiles assets/schemas
|
||||||
var assets embed.FS
|
var assets embed.FS
|
||||||
|
|
||||||
var schemaPaths = map[string]string{
|
var schemaPaths = map[string]string{
|
||||||
@@ -35,6 +35,15 @@ func SchemaFS() fs.FS {
|
|||||||
return fsys
|
return fsys
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ProfileFS returns the embedded Weatherreporter Promptkit profile definitions.
|
||||||
|
func ProfileFS() fs.FS {
|
||||||
|
fsys, err := fs.Sub(assets, "assets/profiles")
|
||||||
|
if err != nil {
|
||||||
|
panic(fmt.Sprintf("embedded profile assets: %v", err))
|
||||||
|
}
|
||||||
|
return fsys
|
||||||
|
}
|
||||||
|
|
||||||
// Schema returns an independent copy of the canonical schema for id.
|
// Schema returns an independent copy of the canonical schema for id.
|
||||||
func Schema(id string) ([]byte, error) {
|
func Schema(id string) ([]byte, error) {
|
||||||
path, ok := schemaPaths[id]
|
path, ok := schemaPaths[id]
|
||||||
|
|||||||
@@ -140,6 +140,83 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
|
||||||
|
wantPaths := map[string]bool{
|
||||||
|
"weather-balanced.yml": false,
|
||||||
|
"weather-deep.yml": false,
|
||||||
|
"weather-light.yml": false,
|
||||||
|
}
|
||||||
|
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||||
|
if err != nil || entry.IsDir() {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, ok := wantPaths[path]; !ok {
|
||||||
|
t.Fatalf("unexpected embedded profile asset %q", path)
|
||||||
|
}
|
||||||
|
wantPaths[path] = true
|
||||||
|
return nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("walk embedded profiles: %v", err)
|
||||||
|
}
|
||||||
|
for path, found := range wantPaths {
|
||||||
|
if !found {
|
||||||
|
t.Errorf("missing embedded profile asset %q", path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||||
|
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
|
||||||
|
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
|
||||||
|
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewEngine() error = %v", err)
|
||||||
|
}
|
||||||
|
profiles := []struct {
|
||||||
|
id string
|
||||||
|
model string
|
||||||
|
timeoutSeconds int
|
||||||
|
reasoningEffort string
|
||||||
|
}{
|
||||||
|
{"weather-light", "deepseek/deepseek-v4-flash", 180, ""},
|
||||||
|
{"weather-balanced", "~google/gemini-flash-latest", 240, "high"},
|
||||||
|
{"weather-deep", "~anthropic/claude-sonnet-latest", 240, "high"},
|
||||||
|
}
|
||||||
|
for _, want := range profiles {
|
||||||
|
t.Run(want.id, func(t *testing.T) {
|
||||||
|
inspection, err := engine.InspectProfile(context.Background(), want.id)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v", err)
|
||||||
|
}
|
||||||
|
got := inspection.EffectiveModelParams
|
||||||
|
if inspection.ProfileID != want.id || got.BackendID != "openrouter" || got.Model != want.model || got.TimeoutSeconds != want.timeoutSeconds || got.ServiceTier != "flex" || got.ReasoningEffort != want.reasoningEffort {
|
||||||
|
t.Fatalf("inspection = %#v, want %q using openrouter model %q", inspection, want.id, want.model)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
|
||||||
|
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
|
||||||
|
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||||
|
if err != nil || entry.IsDir() {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, setting := range forbidden {
|
||||||
|
if strings.Contains(string(data), setting) {
|
||||||
|
t.Fatalf("%s contains forbidden profile setting %q", path, setting)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("walk embedded profiles: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
|
func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
|
||||||
if err := fs.WalkDir(promptassets.PromptFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
if err := fs.WalkDir(promptassets.PromptFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||||
if err != nil || entry.IsDir() {
|
if err != nil || entry.IsDir() {
|
||||||
|
|||||||
Reference in New Issue
Block a user