Clean up roadmap and troubleshooting documentation
This commit is contained in:
@@ -1,599 +0,0 @@
|
||||
# Domain-Specific Prompt Profiles Implementation Plan
|
||||
|
||||
Status: Stages 1–7 completed; remediation Stage 8 ready.
|
||||
|
||||
## Purpose And Authority
|
||||
|
||||
This document records the implementation and post-implementation remediation
|
||||
of the
|
||||
[domain-specific prompt profiles roadmap](domain-profiles.md). The roadmap is
|
||||
authoritative for scope, user intent, policy choices, and the intended end
|
||||
state. This plan records implementation sequence, verification, audit findings,
|
||||
and exit gates.
|
||||
|
||||
This plan follows the repository's
|
||||
[architecture](../policy/architecture.md),
|
||||
[documentation](../policy/documentation.md), and
|
||||
[testing](../policy/testing.md) policies.
|
||||
|
||||
## Completed Prerequisite
|
||||
|
||||
Weatherreporter is already pinned to Promptkit v0.5.0. That release provides
|
||||
the public `WithFallbackProfileFS` option and the required precedence across
|
||||
inspection, preparation, and execution. The dependency upgrade passed
|
||||
Weatherreporter's full offline test suite, race-enabled suite, CLI help check,
|
||||
and an operator `generate hourly` smoke test. Do not repeat or replace the
|
||||
dependency upgrade as part of these stages.
|
||||
|
||||
## Locked Product Decisions
|
||||
|
||||
Implement these exact Weatherreporter-owned profiles:
|
||||
|
||||
| Profile ID | Backend | Model | Reasoning effort | Timeout | Service tier |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `openrouter` | `deepseek/deepseek-v4-flash` | Omitted | 180 seconds | `flex` |
|
||||
| `weather-balanced` | `openrouter` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||
| `weather-deep` | `openrouter` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||
|
||||
Assign Hourly to `weather-light`; assign Daily, Today, and Tomorrow to
|
||||
`weather-balanced`; assign no report to `weather-deep` initially. Advance all
|
||||
four prompt definitions and matching report-registry entries from `1.0.1` to
|
||||
`1.1.0` when their defaults change.
|
||||
|
||||
The leading `~` in the Gemini and Claude model IDs is required and denotes an
|
||||
OpenRouter rolling alias. Do not substitute the unavailable non-tilde IDs or a
|
||||
dated model version. Do not add temperature, `top_p`, maximum-token, endpoint,
|
||||
or credential fields to the embedded definitions.
|
||||
|
||||
Prompt preparation and execution artifacts written at `1.0.1` are not required
|
||||
to remain readable after the transition to `1.1.0`. Do not add a migration,
|
||||
compatibility shim, or weaker historical-artifact validation for this feature.
|
||||
|
||||
Definition lookup must remain:
|
||||
|
||||
1. explicit Promptkit in-memory profiles used by tests or an embedding
|
||||
consumer;
|
||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's built-in catalog.
|
||||
|
||||
Selection remains a separate concern: a nonblank global `promptkit.profile`
|
||||
selects the profile for every report in the invocation; otherwise the exact
|
||||
prompt definition's `default_profile` selects it. A malformed matching
|
||||
higher-precedence profile is an error and never falls through.
|
||||
|
||||
## Continuing Invariants
|
||||
|
||||
- Keep all Promptkit types and mechanics inside
|
||||
`internal/adapters/promptkit`, its focused tests, and asset contract tests.
|
||||
- Keep prompt inspection before weather collection and provider work.
|
||||
- Keep one Promptkit engine per command action and one shared engine across a
|
||||
sequential batch.
|
||||
- Preserve logical profile ID and effective backend/model information through
|
||||
active inspection and execution where the project-owned contract already
|
||||
exposes it. Do not add new durable-provenance fields or compatibility
|
||||
guarantees.
|
||||
- Leave existing workspace persistence behavior otherwise unchanged. The
|
||||
accepted [ephemeral-state roadmap](ephemeral-state.md) owns its future
|
||||
removal and must not be partially implemented here.
|
||||
- Do not expose endpoints, credentials, rendered messages, schemas, request
|
||||
bodies, response bodies, or complete parameter maps through ordinary errors,
|
||||
logs, summaries, or state.
|
||||
- Keep the default suite deterministic, offline, and credential-free.
|
||||
- Do not add endpoint discovery, health probing, provider failover, retries at
|
||||
a more expensive tier, profile merging, per-report configuration fields, or
|
||||
severity-driven model selection.
|
||||
- Update canonical current-state documentation only in the stage where the
|
||||
corresponding behavior becomes implemented.
|
||||
- Run `git diff --check` before completing every stage.
|
||||
|
||||
## Stage 1: Add The Embedded Weather Profile Catalog
|
||||
|
||||
### Goal
|
||||
|
||||
Create one repository-owned, embedded profile source containing exactly the
|
||||
three locked logical profiles.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add strict YAML profile assets beneath `internal/promptassets` using the
|
||||
exact IDs and definitions in this plan.
|
||||
2. Extend `internal/promptassets` with a narrowly named accessor that returns
|
||||
the embedded profile `fs.FS`. Follow the existing prompt and schema asset
|
||||
pattern without exposing Promptkit types from the package.
|
||||
3. Keep profile filenames and embed layout simple and deterministic. Do not
|
||||
duplicate Promptkit's built-in directory taxonomy unless the application
|
||||
assets require it.
|
||||
4. Validate the assets through Promptkit's public engine/profile inspection
|
||||
surface rather than adding a second YAML parser or a Weatherreporter-owned
|
||||
profile representation.
|
||||
|
||||
### Tests
|
||||
|
||||
- Extend the asset contract tests to assert exactly the three logical IDs,
|
||||
their exact effective model IDs, and the intentional parameters.
|
||||
- Prove all three profiles inspect successfully offline when supplied as a
|
||||
fallback source and no operator source is present.
|
||||
- Assert that the catalog contains no endpoints, credentials, temperature,
|
||||
`top_p`, or maximum-token settings.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptassets
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
The embedded catalog is complete, strictly valid, safe, and independently
|
||||
inspectable through Promptkit v0.5.0's public API.
|
||||
|
||||
## Stage 2: Wire Fallback Resolution And Protect Precedence
|
||||
|
||||
### Goal
|
||||
|
||||
Supply the embedded catalog through Promptkit's application fallback layer
|
||||
without changing existing operator configuration or application boundaries.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add `promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), ".")` to
|
||||
normal adapter engine construction.
|
||||
2. Preserve existing `profile_file`, `profile_dir`, configured local backend,
|
||||
timeout, prompt filesystem, schema filesystem, and test-option behavior.
|
||||
3. Ensure ordinary production construction and the adapter's test
|
||||
construction path exercise the same fallback wiring. Test-only explicit
|
||||
profiles may retain Promptkit's documented highest precedence.
|
||||
4. Keep all fallback resolution in Promptkit. Do not add filesystem overlays,
|
||||
existence checks, YAML parsing, or merge behavior to Weatherreporter.
|
||||
|
||||
### Tests
|
||||
|
||||
- At the adapter boundary, prove fallback-only inspection of all three
|
||||
Weatherreporter profiles.
|
||||
- Prove same-ID overrides through both configured `profile_file` and
|
||||
`profile_dir`, including resolution of the override's effective backend and
|
||||
model.
|
||||
- Prove an absent operator match falls through, while a malformed matching
|
||||
operator definition fails without using the embedded profile.
|
||||
- Prove a selected Promptkit built-in that is absent from both higher layers
|
||||
still resolves.
|
||||
- Prove an explicit in-memory test profile retains highest precedence.
|
||||
- Cover both local override forms required by the roadmap: an endpoint-only
|
||||
OpenAI-compatible `weather-light` profile and a `backend: local` profile
|
||||
using the configured local endpoint. No test may contact either endpoint.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Inspection and prepared execution use Promptkit's exact four-layer precedence,
|
||||
operator errors remain visible, and local overrides require no prompt or code
|
||||
changes.
|
||||
|
||||
## Stage 3: Adopt Logical Defaults And Prompt Version 1.1.0
|
||||
|
||||
### Goal
|
||||
|
||||
Move operational prompts from provider-oriented defaults to the three-tier
|
||||
Weatherreporter policy with an exact, synchronized version transition.
|
||||
|
||||
### Work
|
||||
|
||||
1. Change Hourly's `default_profile` to `weather-light`.
|
||||
2. Change Daily, Today, and Tomorrow to `weather-balanced`.
|
||||
3. Advance the exact version in all four prompt YAML assets from `1.0.1` to
|
||||
`1.1.0` without changing prompt text or generated-text schemas solely for
|
||||
this feature.
|
||||
4. Advance the four matching report-registry prompt versions to `1.1.0` in the
|
||||
same change. Keep prompt IDs, report IDs, modules, periods, templates, and
|
||||
output contracts unchanged.
|
||||
5. Update fixtures and expectations that intentionally assert the current
|
||||
prompt contract. Do not rewrite historical fixture versions or weaken tests
|
||||
that protect actual compatibility.
|
||||
|
||||
### Tests
|
||||
|
||||
- Update asset and report-registry contract tests to require exact version
|
||||
`1.1.0` and the report-to-profile assignments locked in this plan.
|
||||
- Inspect every exact prompt version through the real embedded prompt, schema,
|
||||
and fallback-profile filesystems.
|
||||
- Prove Hourly resolves DeepSeek V4 Flash, the three day-scale reports resolve
|
||||
Gemini Flash Latest, and `weather-deep` remains inspectable but unassigned.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptassets ./internal/report ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Every operational prompt and registry definition agrees on exact version
|
||||
`1.1.0`, selects its intended logical tier, and resolves its expected effective
|
||||
model offline.
|
||||
|
||||
## Stage 4: Verify Application Selection And Batch Reuse
|
||||
|
||||
### Goal
|
||||
|
||||
Protect the assembled application behavior created by the new defaults and
|
||||
confirm that logical identity is not lost during active effective-model
|
||||
resolution.
|
||||
|
||||
### Work
|
||||
|
||||
1. Preserve the current pre-collection inspection order and fail-fast behavior
|
||||
for missing credentials, unknown profiles, malformed profiles, and unusable
|
||||
backends.
|
||||
2. Preserve the global `promptkit.profile` all-report override. Do not add a
|
||||
second override mechanism or report-specific configuration fields.
|
||||
3. Preserve batch preflight deduplication by selected effective profile ID:
|
||||
Today and Tomorrow in the same batch should inspect their shared
|
||||
`weather-balanced` selection once.
|
||||
4. Preserve the selected logical profile ID and resolved backend/model through
|
||||
active inspection, preparation, and execution using the existing
|
||||
project-owned contract. Do not add state fields, expand persisted parameter
|
||||
detail, or create a new historical compatibility guarantee.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add or update representative app tests for default Hourly and day-scale
|
||||
selection, a global-profile override, and a morning/evening batch sharing
|
||||
`weather-balanced`.
|
||||
- Assert inspection completes before weather collection and provider
|
||||
generation, including malformed same-ID operator overrides.
|
||||
- Assert active inspection and execution expose the logical profile ID and
|
||||
effective model for both embedded and overridden profiles.
|
||||
- Assert endpoints and credentials remain absent from errors, summaries,
|
||||
normal logs, and ordinary state.
|
||||
- Use project-owned executor fakes or Promptkit provider fakes; do not make live
|
||||
provider calls.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/app ./internal/cli
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Single-report and batch workflows select the intended tier, retain existing
|
||||
override and preflight behavior, deduplicate shared batch inspection, and
|
||||
preserve safe logical and effective model information during active execution
|
||||
without adding a durable-provenance contract.
|
||||
|
||||
## Stage 5: Publish Canonical Operator And Maintainer Documentation
|
||||
|
||||
### Goal
|
||||
|
||||
Document the implemented feature once in each canonical owner and provide one
|
||||
maintained, copyable local override example.
|
||||
|
||||
### Work
|
||||
|
||||
1. Update `docs/config.md` to explain global profile selection versus
|
||||
`profile_file`/`profile_dir` definition lookup and link to the maintained
|
||||
example. Keep the field reference in this canonical document.
|
||||
2. Update the Promptkit integration document with the logical profile catalog,
|
||||
source precedence, exact prompt-version relationship, and safe active
|
||||
inspection and execution contract. Avoid restating complete configuration
|
||||
syntax or presenting transitional persistence as the target architecture.
|
||||
3. Update the report-registry, Promptkit adapter, app-orchestration, and state
|
||||
internal documents only where their implemented contracts changed.
|
||||
4. Update `docs/operations.md` with the normal local-override workflow and
|
||||
`docs/troubleshooting.md` with malformed override, unavailable local
|
||||
endpoint, missing credential, and unexpected effective-model diagnostics.
|
||||
5. Add or update one secret-free file under `examples/` showing a
|
||||
`weather-light` override for a local OpenAI-compatible endpoint. Choose one
|
||||
supported form as the complete example and mention the other form only in
|
||||
its canonical reference.
|
||||
6. Update the architecture policy only if implementation changed a normative
|
||||
boundary or invariant. Do not add future behavior to current-state docs.
|
||||
7. Keep the feature roadmap and this plan in their pre-implementation statuses
|
||||
until the final repository gate passes. Do not create release notes before
|
||||
a release version is chosen.
|
||||
|
||||
### Tests
|
||||
|
||||
- Verify every changed repository-relative link and every profile/model ID.
|
||||
- Validate maintained YAML examples through the same strict configuration or
|
||||
Promptkit profile path used by production where practical.
|
||||
- Run the focused tests that own any executable examples, followed by:
|
||||
|
||||
```sh
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Users, operators, and maintainers can discover the tier defaults, precedence,
|
||||
global override, local override, and failure behavior without duplicated or
|
||||
future-state documentation.
|
||||
|
||||
## Stage 6: Complete Repository Verification And Roadmap Handoff
|
||||
|
||||
### Goal
|
||||
|
||||
Demonstrate that the complete feature is coherent, offline-testable, and ready
|
||||
for review and a later release decision.
|
||||
|
||||
### Work
|
||||
|
||||
1. Review the complete diff against the roadmap, this plan, and all three
|
||||
policy documents. Remove stale identifiers, temporary helpers, redundant
|
||||
tests, and documentation duplication.
|
||||
2. Confirm `go.mod` and `go.sum` retain tagged Promptkit v0.5.0 without a local
|
||||
replacement or dependency drift.
|
||||
3. Confirm only the four supported report products exist and no retired report
|
||||
surfaces were reintroduced.
|
||||
4. Confirm the roadmap's completion criteria one by one. Change its status to
|
||||
implemented and this plan's status to completed only after every criterion
|
||||
and command below passes.
|
||||
5. Do not require a live provider for completion. If credentials and network
|
||||
access are deliberately supplied by an operator, record live smoke results
|
||||
separately as release-candidate evidence rather than adding them to the
|
||||
default suite.
|
||||
|
||||
### Verification
|
||||
|
||||
Run `gofmt -w` on every changed Go file, then run:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Also inspect all three logical profiles through the application's normal
|
||||
preflight path using offline provider doubles, including one same-ID local
|
||||
override and one explicit global override.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All roadmap completion criteria are satisfied, all verification commands pass,
|
||||
the working tree contains only intentional changes, and the canonical
|
||||
documentation describes the implemented state. The feature is ready for code
|
||||
review and release preparation.
|
||||
|
||||
## Post-Implementation Review
|
||||
|
||||
Stages 1–6 implemented the intended production behavior and passed their
|
||||
offline verification gates. A subsequent review found no high-severity runtime
|
||||
defect, but identified three test-quality issues and one remaining validation
|
||||
obligation:
|
||||
|
||||
- one app test asserted durable preparation and execution artifact provenance,
|
||||
contrary to the active-execution boundary and accepted ephemeral-state
|
||||
direction;
|
||||
- an adapter-package test depended upward on app orchestration and duplicated
|
||||
test ownership;
|
||||
- embedded fallback profiles were inspected but not exercised through one
|
||||
prepared execution with a provider fake; and
|
||||
- the roadmap's representative model-evaluation policy had no recorded
|
||||
evidence.
|
||||
|
||||
Stages 7 and 8 address those findings without changing the profile catalog,
|
||||
selection precedence, report assignments, prompt content, generated-text
|
||||
schemas, or default offline test contract.
|
||||
|
||||
## Stage 7: Correct Test Ownership And Fallback Execution Coverage
|
||||
|
||||
### Goal
|
||||
|
||||
Remove accidental durable-state and cross-layer test commitments while adding
|
||||
one focused offline execution test for the embedded fallback path.
|
||||
|
||||
### Work
|
||||
|
||||
1. Rewrite `TestGenerateDetailedPreservesSelectedProfileThroughExecution` so
|
||||
it protects active workflow behavior only:
|
||||
|
||||
- retain the Hourly default, day-scale default, and global-override cases;
|
||||
- assert the profile ID sent in `promptexec.ExecuteRequest`;
|
||||
- have the executor fake record the preparation and execution values it
|
||||
emits, then assert their logical profile ID and effective backend/model;
|
||||
- do not load preparation, execution, or metadata files to establish a
|
||||
durable profile-provenance contract; and
|
||||
- remove artifact-content scans whose fake inputs cannot contain an endpoint
|
||||
or credential.
|
||||
|
||||
2. Preserve meaningful safety coverage at the boundary that can expose the
|
||||
sensitive value:
|
||||
|
||||
- retain adapter mapping coverage proving an endpoint from a real Promptkit
|
||||
profile does not enter `promptexec.ProfileInspection`;
|
||||
- retain app error coverage proving dependency errors containing an endpoint
|
||||
or credential are replaced by a bounded classified error; and
|
||||
- do not add profile endpoints or credentials to project-owned execution
|
||||
types merely to make a leakage test possible.
|
||||
|
||||
3. Remove `internal/app`, app configuration, and report-registry dependencies
|
||||
from `internal/adapters/promptkit/adapter_test.go`. Move the assembled
|
||||
application-preflight test to a new app-owned external integration test,
|
||||
such as `internal/app/prompt_profile_integration_test.go` with package
|
||||
`app_test`:
|
||||
|
||||
- construct the real Promptkit adapter through its public `New` function;
|
||||
- call the public app prompt-inspection operation;
|
||||
- supply a deterministic credential lookup rather than reading the process
|
||||
environment; and
|
||||
- cover Hourly, one representative day-scale default, the explicit
|
||||
`weather-deep` global override, and a same-ID endpoint-only
|
||||
`weather-light` override. The asset contract tests already own the exact
|
||||
mapping for all four prompts, so the integration test need not repeat all
|
||||
four.
|
||||
|
||||
4. Add one adapter-owned, offline fake-client execution test using the real
|
||||
embedded Hourly prompt at `1.1.0` and selected profile `weather-light`.
|
||||
Execute through the normal prepared adapter path and assert:
|
||||
|
||||
- the preparation callback runs before the fake provider;
|
||||
- preparation and execution report logical profile `weather-light`, backend
|
||||
`openrouter`, and model `deepseek/deepseek-v4-flash`;
|
||||
- the fake provider request targets `deepseek/deepseek-v4-flash`; and
|
||||
- schema validation completes without contacting a live service.
|
||||
|
||||
One execution case is sufficient because Promptkit owns uniform source
|
||||
precedence and the adapter's inspection tests already cover fallback,
|
||||
operator file, operator directory, built-in, and explicit in-memory layers.
|
||||
|
||||
5. Reconcile the profile-related current-state documentation:
|
||||
|
||||
- it may accurately describe fields present in current preparation and
|
||||
execution receipts;
|
||||
- it must not promise cross-version readability or characterize those
|
||||
receipts as the profile feature's durable target architecture; and
|
||||
- troubleshooting should prefer active command errors and explicit secure
|
||||
debug capture, mentioning current-version receipts only as transitional
|
||||
state if they remain useful before the ephemeral-state refactor.
|
||||
|
||||
6. Do not change production profile resolution, prompt definitions, state
|
||||
schemas, artifact validators, or the ephemeral-state roadmap in this stage.
|
||||
|
||||
### Tests
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||
go test -count=1 -race ./internal/promptassets ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||
go test -count=1 ./...
|
||||
go vet ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Review the changed tests against the testing policy and confirm that adapter
|
||||
tests own adapter behavior, app tests own orchestration, and state tests remain
|
||||
the sole owner of durable artifact format and validation details.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Active profile selection and effective-model propagation remain protected
|
||||
without adding a durable-provenance commitment; the adapter test package no
|
||||
longer imports the app layer; one embedded fallback profile completes prepared
|
||||
execution through a provider fake; and every required check passes offline.
|
||||
|
||||
## Stage 8: Evaluate The Initial Model Ladder
|
||||
|
||||
### Goal
|
||||
|
||||
Produce explicit release-candidate evidence that the selected models are
|
||||
acceptable for their intended report tiers and that a representative local
|
||||
override provides the promised operator experience.
|
||||
|
||||
This is an opt-in evaluation stage, not an ordinary automated-test stage. It
|
||||
requires operator-approved provider credentials, network access, and a local
|
||||
OpenAI-compatible endpoint. Do not mark it complete when those prerequisites
|
||||
are unavailable; report the missing prerequisite instead.
|
||||
|
||||
### Corpus
|
||||
|
||||
Use four representative, secret-free YAML data packages: one each for Daily,
|
||||
Today, Tomorrow, and Hourly. The set must include at least one package with
|
||||
precipitation windows and at least one with none. Remove precise private
|
||||
location identifiers or other operationally sensitive values without changing
|
||||
the meteorological relationships being evaluated.
|
||||
|
||||
Record a SHA-256 hash and a short, non-sensitive description for each package.
|
||||
Do not commit full packages or generated prose unless the user separately
|
||||
approves them as repository fixtures.
|
||||
|
||||
### Execution Matrix
|
||||
|
||||
Run these six evaluations from the exact package bytes:
|
||||
|
||||
| Case | Package | Profile |
|
||||
| --- | --- | --- |
|
||||
| Hourly default | Hourly | `weather-light` |
|
||||
| Daily default | Daily | `weather-balanced` |
|
||||
| Today default | Today | `weather-balanced` |
|
||||
| Tomorrow default | Tomorrow | `weather-balanced` |
|
||||
| Deep comparison | The same Daily package used above | `weather-deep` |
|
||||
| Local override | The same Hourly package used above | operator-defined `weather-light` endpoint profile |
|
||||
|
||||
After the successful local-override case, stop or deliberately address an
|
||||
unavailable test endpoint and repeat it as a negative control. Confirm that the
|
||||
request fails visibly and does not call or select an embedded remote profile.
|
||||
This negative control is not an additional quality-evaluation case.
|
||||
|
||||
Use a temporary, untracked evaluation harness beneath the module when exact
|
||||
package replay is needed. It should call the existing Promptkit adapter and
|
||||
project-owned execution contract rather than duplicate prompt loading,
|
||||
rendering, or schema validation. Remove the harness and all unapproved raw
|
||||
outputs before completing the stage. Never print or record credentials.
|
||||
|
||||
### Evaluation Record
|
||||
|
||||
Add a concise `## Evaluation Record` section to
|
||||
`docs/roadmap/domain-profiles.md`. For every case, record:
|
||||
|
||||
- evaluation date, logical profile, effective backend, and exact model
|
||||
reported by execution;
|
||||
- corpus hash, validation outcome, latency, prompt/completion/total token use,
|
||||
and provider-reported or contemporaneously calculated cost;
|
||||
- whether every generated claim is supported by the deterministic package;
|
||||
- whether hazards, periods, uncertainty, and precipitation timing are used
|
||||
correctly;
|
||||
- whether `precipitation_timing` is exactly an empty string for the no-window
|
||||
case;
|
||||
- a short usefulness assessment for summary and forecast discussion; and
|
||||
- any provider, alias, or local-endpoint caveat observed.
|
||||
|
||||
Do not include credentials, endpoints, complete effective parameter maps,
|
||||
full data packages, rendered prompts, or full generated responses in the
|
||||
record. The secure debug directory may be used temporarily for operator review
|
||||
and remains operator-managed.
|
||||
|
||||
### Acceptance Rules
|
||||
|
||||
- Every case must complete strict JSON Schema validation without repair.
|
||||
- Generated prose must contain no material unsupported weather claim or
|
||||
contradiction of deterministic hazards, periods, or uncertainty.
|
||||
- Precipitation timing must agree with the deterministic windows and use the
|
||||
required empty-string representation when no window exists.
|
||||
- The local override must select the operator model without modifying a prompt
|
||||
or application code and must not fall back to a remote profile when the local
|
||||
endpoint is unavailable.
|
||||
- Latency, tokens, and cost must be recorded, but this initial evaluation does
|
||||
not impose an invented numeric threshold. The operator decides whether the
|
||||
observed tradeoff remains acceptable for the named tier.
|
||||
- If a default case fails schema or factual acceptance, do not weaken the
|
||||
schema or prompt to accommodate the model. Reopen the concrete model or
|
||||
profile-setting decision in the feature roadmap and leave this stage
|
||||
incomplete.
|
||||
|
||||
### Verification
|
||||
|
||||
After removing temporary evaluation material, run:
|
||||
|
||||
```sh
|
||||
go test -count=1 ./...
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Confirm that the only intended repository change from this stage is the
|
||||
concise evaluation record and any roadmap status correction required by its
|
||||
result. Do not add live credentials, provider-dependent tests, a permanent
|
||||
benchmark framework, or release notes before a release version is selected.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All six cases satisfy the acceptance rules, the roadmap contains concise and
|
||||
safe evaluation evidence, no temporary corpus or response material remains in
|
||||
the repository, and the default suite remains offline. Set this plan back to
|
||||
`Status: Completed` only after both Stages 7 and 8 have passed.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The model identifiers, profile settings, report assignments, version
|
||||
transition, precedence, compatibility behavior, test boundaries, and
|
||||
documentation ownership are decision-complete.
|
||||
Reference in New Issue
Block a user