weatherreporter/docs/roadmap/implementation.md

Domain-Specific Prompt Profiles Implementation Plan

Status: Stages 1–7 completed; remediation Stage 8 ready.

Purpose And Authority

This document records the implementation and post-implementation remediation of the domain-specific prompt profiles roadmap. The roadmap is authoritative for scope, user intent, policy choices, and the intended end state. This plan records implementation sequence, verification, audit findings, and exit gates.

This plan follows the repository's architecture, documentation, and testing policies.

Completed Prerequisite

Weatherreporter is already pinned to Promptkit v0.5.0. That release provides the public WithFallbackProfileFS option and the required precedence across inspection, preparation, and execution. The dependency upgrade passed Weatherreporter's full offline test suite, race-enabled suite, CLI help check, and an operator generate hourly smoke test. Do not repeat or replace the dependency upgrade as part of these stages.

Locked Product Decisions

Implement these exact Weatherreporter-owned profiles:

Profile ID	Backend	Model	Reasoning effort	Timeout	Service tier
weather-light	openrouter	deepseek/deepseek-v4-flash	Omitted	180 seconds	flex
weather-balanced	openrouter	~google/gemini-flash-latest	high	240 seconds	flex
weather-deep	openrouter	~anthropic/claude-sonnet-latest	high	240 seconds	flex

Assign Hourly to weather-light; assign Daily, Today, and Tomorrow to weather-balanced; assign no report to weather-deep initially. Advance all four prompt definitions and matching report-registry entries from 1.0.1 to 1.1.0 when their defaults change.

The leading ~ in the Gemini and Claude model IDs is required and denotes an OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a dated model version. Do not add temperature, top_p, maximum-token, endpoint, or credential fields to the embedded definitions.

Prompt preparation and execution artifacts written at 1.0.1 are not required to remain readable after the transition to 1.1.0. Do not add a migration, compatibility shim, or weaker historical-artifact validation for this feature.

Definition lookup must remain:

1. explicit Promptkit in-memory profiles used by tests or an embedding consumer;
2. Weatherreporter's configured profile_file or profile_dir source;
3. Weatherreporter's embedded fallback profiles; and
4. Promptkit's built-in catalog.

Selection remains a separate concern: a nonblank global promptkit.profile selects the profile for every report in the invocation; otherwise the exact prompt definition's default_profile selects it. A malformed matching higher-precedence profile is an error and never falls through.

Continuing Invariants

• Keep all Promptkit types and mechanics inside internal/adapters/promptkit, its focused tests, and asset contract tests.
• Keep prompt inspection before weather collection and provider work.
• Keep one Promptkit engine per command action and one shared engine across a sequential batch.
• Preserve logical profile ID and effective backend/model information through active inspection and execution where the project-owned contract already exposes it. Do not add new durable-provenance fields or compatibility guarantees.
• Leave existing workspace persistence behavior otherwise unchanged. The accepted ephemeral-state roadmap owns its future removal and must not be partially implemented here.
• Do not expose endpoints, credentials, rendered messages, schemas, request bodies, response bodies, or complete parameter maps through ordinary errors, logs, summaries, or state.
• Keep the default suite deterministic, offline, and credential-free.
• Do not add endpoint discovery, health probing, provider failover, retries at a more expensive tier, profile merging, per-report configuration fields, or severity-driven model selection.
• Update canonical current-state documentation only in the stage where the corresponding behavior becomes implemented.
• Run git diff --check before completing every stage.

Stage 1: Add The Embedded Weather Profile Catalog

Goal

Create one repository-owned, embedded profile source containing exactly the three locked logical profiles.

Work

1. Add strict YAML profile assets beneath internal/promptassets using the exact IDs and definitions in this plan.
2. Extend internal/promptassets with a narrowly named accessor that returns the embedded profile fs.FS. Follow the existing prompt and schema asset pattern without exposing Promptkit types from the package.
3. Keep profile filenames and embed layout simple and deterministic. Do not duplicate Promptkit's built-in directory taxonomy unless the application assets require it.
4. Validate the assets through Promptkit's public engine/profile inspection surface rather than adding a second YAML parser or a Weatherreporter-owned profile representation.

Tests

• Extend the asset contract tests to assert exactly the three logical IDs, their exact effective model IDs, and the intentional parameters.

• Prove all three profiles inspect successfully offline when supplied as a fallback source and no operator source is present.

• Assert that the catalog contains no endpoints, credentials, temperature, top_p, or maximum-token settings.

• Run:

  go test ./internal/promptassets
  git diff --check
  

Exit Gate

The embedded catalog is complete, strictly valid, safe, and independently inspectable through Promptkit v0.5.0's public API.

Stage 2: Wire Fallback Resolution And Protect Precedence

Goal

Supply the embedded catalog through Promptkit's application fallback layer without changing existing operator configuration or application boundaries.

Work

1. Add promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".") to normal adapter engine construction.
2. Preserve existing profile_file, profile_dir, configured local backend, timeout, prompt filesystem, schema filesystem, and test-option behavior.
3. Ensure ordinary production construction and the adapter's test construction path exercise the same fallback wiring. Test-only explicit profiles may retain Promptkit's documented highest precedence.
4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays, existence checks, YAML parsing, or merge behavior to Weatherreporter.

Tests

• At the adapter boundary, prove fallback-only inspection of all three Weatherreporter profiles.

• Prove same-ID overrides through both configured profile_file and profile_dir, including resolution of the override's effective backend and model.

• Prove an absent operator match falls through, while a malformed matching operator definition fails without using the embedded profile.

• Prove a selected Promptkit built-in that is absent from both higher layers still resolves.

• Prove an explicit in-memory test profile retains highest precedence.

• Cover both local override forms required by the roadmap: an endpoint-only OpenAI-compatible weather-light profile and a backend: local profile using the configured local endpoint. No test may contact either endpoint.

• Run:

  go test ./internal/adapters/promptkit
  git diff --check
  

Exit Gate

Inspection and prepared execution use Promptkit's exact four-layer precedence, operator errors remain visible, and local overrides require no prompt or code changes.

Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0

Goal

Move operational prompts from provider-oriented defaults to the three-tier Weatherreporter policy with an exact, synchronized version transition.

Work

1. Change Hourly's default_profile to weather-light.
2. Change Daily, Today, and Tomorrow to weather-balanced.
3. Advance the exact version in all four prompt YAML assets from 1.0.1 to 1.1.0 without changing prompt text or generated-text schemas solely for this feature.
4. Advance the four matching report-registry prompt versions to 1.1.0 in the same change. Keep prompt IDs, report IDs, modules, periods, templates, and output contracts unchanged.
5. Update fixtures and expectations that intentionally assert the current prompt contract. Do not rewrite historical fixture versions or weaken tests that protect actual compatibility.

Tests

• Update asset and report-registry contract tests to require exact version 1.1.0 and the report-to-profile assignments locked in this plan.

• Inspect every exact prompt version through the real embedded prompt, schema, and fallback-profile filesystems.

• Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve Gemini Flash Latest, and weather-deep remains inspectable but unassigned.

• Run:

  go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit
  git diff --check
  

Exit Gate

Every operational prompt and registry definition agrees on exact version 1.1.0, selects its intended logical tier, and resolves its expected effective model offline.

Stage 4: Verify Application Selection And Batch Reuse

Goal

Protect the assembled application behavior created by the new defaults and confirm that logical identity is not lost during active effective-model resolution.

Work

1. Preserve the current pre-collection inspection order and fail-fast behavior for missing credentials, unknown profiles, malformed profiles, and unusable backends.
2. Preserve the global promptkit.profile all-report override. Do not add a second override mechanism or report-specific configuration fields.
3. Preserve batch preflight deduplication by selected effective profile ID: Today and Tomorrow in the same batch should inspect their shared weather-balanced selection once.
4. Preserve the selected logical profile ID and resolved backend/model through active inspection, preparation, and execution using the existing project-owned contract. Do not add state fields, expand persisted parameter detail, or create a new historical compatibility guarantee.

Tests

• Add or update representative app tests for default Hourly and day-scale selection, a global-profile override, and a morning/evening batch sharing weather-balanced.

• Assert inspection completes before weather collection and provider generation, including malformed same-ID operator overrides.

• Assert active inspection and execution expose the logical profile ID and effective model for both embedded and overridden profiles.

• Assert endpoints and credentials remain absent from errors, summaries, normal logs, and ordinary state.

• Use project-owned executor fakes or Promptkit provider fakes; do not make live provider calls.

• Run:

  go test ./internal/app ./internal/cli
  git diff --check
  

Exit Gate

Single-report and batch workflows select the intended tier, retain existing override and preflight behavior, deduplicate shared batch inspection, and preserve safe logical and effective model information during active execution without adding a durable-provenance contract.

Stage 5: Publish Canonical Operator And Maintainer Documentation

Goal

Document the implemented feature once in each canonical owner and provide one maintained, copyable local override example.

Work

1. Update docs/config.md to explain global profile selection versus profile_file/profile_dir definition lookup and link to the maintained example. Keep the field reference in this canonical document.
2. Update the Promptkit integration document with the logical profile catalog, source precedence, exact prompt-version relationship, and safe active inspection and execution contract. Avoid restating complete configuration syntax or presenting transitional persistence as the target architecture.
3. Update the report-registry, Promptkit adapter, app-orchestration, and state internal documents only where their implemented contracts changed.
4. Update docs/operations.md with the normal local-override workflow and docs/troubleshooting.md with malformed override, unavailable local endpoint, missing credential, and unexpected effective-model diagnostics.
5. Add or update one secret-free file under examples/ showing a weather-light override for a local OpenAI-compatible endpoint. Choose one supported form as the complete example and mention the other form only in its canonical reference.
6. Update the architecture policy only if implementation changed a normative boundary or invariant. Do not add future behavior to current-state docs.
7. Keep the feature roadmap and this plan in their pre-implementation statuses until the final repository gate passes. Do not create release notes before a release version is chosen.

Tests

• Verify every changed repository-relative link and every profile/model ID.

• Validate maintained YAML examples through the same strict configuration or Promptkit profile path used by production where practical.

• Run the focused tests that own any executable examples, followed by:

  git diff --check
  

Exit Gate

Users, operators, and maintainers can discover the tier defaults, precedence, global override, local override, and failure behavior without duplicated or future-state documentation.

Stage 6: Complete Repository Verification And Roadmap Handoff

Goal

Demonstrate that the complete feature is coherent, offline-testable, and ready for review and a later release decision.

Work

1. Review the complete diff against the roadmap, this plan, and all three policy documents. Remove stale identifiers, temporary helpers, redundant tests, and documentation duplication.
2. Confirm go.mod and go.sum retain tagged Promptkit v0.5.0 without a local replacement or dependency drift.
3. Confirm only the four supported report products exist and no retired report surfaces were reintroduced.
4. Confirm the roadmap's completion criteria one by one. Change its status to implemented and this plan's status to completed only after every criterion and command below passes.
5. Do not require a live provider for completion. If credentials and network access are deliberately supplied by an operator, record live smoke results separately as release-candidate evidence rather than adding them to the default suite.

Verification

Run gofmt -w on every changed Go file, then run:

go test ./...
go test -race ./...
go run ./cmd/weatherreporter --help
git diff --check
git status --short

Also inspect all three logical profiles through the application's normal preflight path using offline provider doubles, including one same-ID local override and one explicit global override.

Exit Gate

All roadmap completion criteria are satisfied, all verification commands pass, the working tree contains only intentional changes, and the canonical documentation describes the implemented state. The feature is ready for code review and release preparation.

Post-Implementation Review

Stages 1–6 implemented the intended production behavior and passed their offline verification gates. A subsequent review found no high-severity runtime defect, but identified three test-quality issues and one remaining validation obligation:

• one app test asserted durable preparation and execution artifact provenance, contrary to the active-execution boundary and accepted ephemeral-state direction;
• an adapter-package test depended upward on app orchestration and duplicated test ownership;
• embedded fallback profiles were inspected but not exercised through one prepared execution with a provider fake; and
• the roadmap's representative model-evaluation policy had no recorded evidence.

Stages 7 and 8 address those findings without changing the profile catalog, selection precedence, report assignments, prompt content, generated-text schemas, or default offline test contract.

Stage 7: Correct Test Ownership And Fallback Execution Coverage

Goal

Remove accidental durable-state and cross-layer test commitments while adding one focused offline execution test for the embedded fallback path.

Work

1. Rewrite TestGenerateDetailedPreservesSelectedProfileThroughExecution so it protects active workflow behavior only:

   • retain the Hourly default, day-scale default, and global-override cases;
   • assert the profile ID sent in promptexec.ExecuteRequest;
   • have the executor fake record the preparation and execution values it emits, then assert their logical profile ID and effective backend/model;
   • do not load preparation, execution, or metadata files to establish a durable profile-provenance contract; and
   • remove artifact-content scans whose fake inputs cannot contain an endpoint or credential.

2. Preserve meaningful safety coverage at the boundary that can expose the sensitive value:

   • retain adapter mapping coverage proving an endpoint from a real Promptkit profile does not enter promptexec.ProfileInspection;
   • retain app error coverage proving dependency errors containing an endpoint or credential are replaced by a bounded classified error; and
   • do not add profile endpoints or credentials to project-owned execution types merely to make a leakage test possible.

3. Remove internal/app, app configuration, and report-registry dependencies from internal/adapters/promptkit/adapter_test.go. Move the assembled application-preflight test to a new app-owned external integration test, such as internal/app/prompt_profile_integration_test.go with package app_test:

   • construct the real Promptkit adapter through its public New function;
   • call the public app prompt-inspection operation;
   • supply a deterministic credential lookup rather than reading the process environment; and
   • cover Hourly, one representative day-scale default, the explicit weather-deep global override, and a same-ID endpoint-only weather-light override. The asset contract tests already own the exact mapping for all four prompts, so the integration test need not repeat all four.

4. Add one adapter-owned, offline fake-client execution test using the real embedded Hourly prompt at 1.1.0 and selected profile weather-light. Execute through the normal prepared adapter path and assert:

   • the preparation callback runs before the fake provider;
   • preparation and execution report logical profile weather-light, backend openrouter, and model deepseek/deepseek-v4-flash;
   • the fake provider request targets deepseek/deepseek-v4-flash; and
   • schema validation completes without contacting a live service.

   One execution case is sufficient because Promptkit owns uniform source precedence and the adapter's inspection tests already cover fallback, operator file, operator directory, built-in, and explicit in-memory layers.

5. Reconcile the profile-related current-state documentation:

   • it may accurately describe fields present in current preparation and execution receipts;
   • it must not promise cross-version readability or characterize those receipts as the profile feature's durable target architecture; and
   • troubleshooting should prefer active command errors and explicit secure debug capture, mentioning current-version receipts only as transitional state if they remain useful before the ephemeral-state refactor.

6. Do not change production profile resolution, prompt definitions, state schemas, artifact validators, or the ephemeral-state roadmap in this stage.

Tests

Run:

go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli
go test -count=1 -race ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli
go test -count=1 ./...
go vet ./...
go run ./cmd/weatherreporter --help
git diff --check

Review the changed tests against the testing policy and confirm that adapter tests own adapter behavior, app tests own orchestration, and state tests remain the sole owner of durable artifact format and validation details.

Exit Gate

Active profile selection and effective-model propagation remain protected without adding a durable-provenance commitment; the adapter test package no longer imports the app layer; one embedded fallback profile completes prepared execution through a provider fake; and every required check passes offline.

Stage 8: Evaluate The Initial Model Ladder

Goal

Produce explicit release-candidate evidence that the selected models are acceptable for their intended report tiers and that a representative local override provides the promised operator experience.

This is an opt-in evaluation stage, not an ordinary automated-test stage. It requires operator-approved provider credentials, network access, and a local OpenAI-compatible endpoint. Do not mark it complete when those prerequisites are unavailable; report the missing prerequisite instead.

Corpus

Use four representative, secret-free YAML data packages: one each for Daily, Today, Tomorrow, and Hourly. The set must include at least one package with precipitation windows and at least one with none. Remove precise private location identifiers or other operationally sensitive values without changing the meteorological relationships being evaluated.

Record a SHA-256 hash and a short, non-sensitive description for each package. Do not commit full packages or generated prose unless the user separately approves them as repository fixtures.

Execution Matrix

Run these six evaluations from the exact package bytes:

Case	Package	Profile
Hourly default	Hourly	weather-light
Daily default	Daily	weather-balanced
Today default	Today	weather-balanced
Tomorrow default	Tomorrow	weather-balanced
Deep comparison	The same Daily package used above	weather-deep
Local override	The same Hourly package used above	operator-defined weather-light endpoint profile

After the successful local-override case, stop or deliberately address an unavailable test endpoint and repeat it as a negative control. Confirm that the request fails visibly and does not call or select an embedded remote profile. This negative control is not an additional quality-evaluation case.

Use a temporary, untracked evaluation harness beneath the module when exact package replay is needed. It should call the existing Promptkit adapter and project-owned execution contract rather than duplicate prompt loading, rendering, or schema validation. Remove the harness and all unapproved raw outputs before completing the stage. Never print or record credentials.

Evaluation Record

Add a concise ## Evaluation Record section to docs/roadmap/domain-profiles.md. For every case, record:

• evaluation date, logical profile, effective backend, and exact model reported by execution;
• corpus hash, validation outcome, latency, prompt/completion/total token use, and provider-reported or contemporaneously calculated cost;
• whether every generated claim is supported by the deterministic package;
• whether hazards, periods, uncertainty, and precipitation timing are used correctly;
• whether precipitation_timing is exactly an empty string for the no-window case;
• a short usefulness assessment for summary and forecast discussion; and
• any provider, alias, or local-endpoint caveat observed.

Do not include credentials, endpoints, complete effective parameter maps, full data packages, rendered prompts, or full generated responses in the record. The secure debug directory may be used temporarily for operator review and remains operator-managed.

Acceptance Rules

• Every case must complete strict JSON Schema validation without repair.
• Generated prose must contain no material unsupported weather claim or contradiction of deterministic hazards, periods, or uncertainty.
• Precipitation timing must agree with the deterministic windows and use the required empty-string representation when no window exists.
• The local override must select the operator model without modifying a prompt or application code and must not fall back to a remote profile when the local endpoint is unavailable.
• Latency, tokens, and cost must be recorded, but this initial evaluation does not impose an invented numeric threshold. The operator decides whether the observed tradeoff remains acceptable for the named tier.
• If a default case fails schema or factual acceptance, do not weaken the schema or prompt to accommodate the model. Reopen the concrete model or profile-setting decision in the feature roadmap and leave this stage incomplete.

Verification

After removing temporary evaluation material, run:

go test -count=1 ./...
git diff --check
git status --short

Confirm that the only intended repository change from this stage is the concise evaluation record and any roadmap status correction required by its result. Do not add live credentials, provider-dependent tests, a permanent benchmark framework, or release notes before a release version is selected.

Exit Gate

All six cases satisfy the acceptance rules, the roadmap contains concise and safe evaluation evidence, no temporary corpus or response material remains in the repository, and the default suite remains offline. Set this plan back to Status: Completed only after both Stages 7 and 8 have passed.

Open Questions

None. The model identifiers, profile settings, report assignments, version transition, precedence, compatibility behavior, test boundaries, and documentation ownership are decision-complete.