361 lines
14 KiB
Markdown
361 lines
14 KiB
Markdown
# Data Package Export Roadmap
|
|
|
|
## Purpose
|
|
|
|
This roadmap defines the target state for cleaning up module fields exposed in
|
|
YAML data packages. The goal is to keep report templates composable while making
|
|
LLM prompt inputs concise, readable, and free of template-only helper fields.
|
|
|
|
This feature is not implemented yet. Current data-package behavior remains
|
|
documented outside `docs/roadmap/`.
|
|
|
|
## Problem
|
|
|
|
Module output structs currently serve two different consumers:
|
|
|
|
- deterministic report templates, which benefit from presentation helpers such
|
|
as lower-case text, display labels, trend phrases, and hour labels;
|
|
- Scriptorium data packages, which should expose the clearest useful weather
|
|
facts to the LLM with minimal redundancy.
|
|
|
|
Those consumers now need different surfaces. Examples include:
|
|
|
|
- `current_conditions` exposes both `condition_text` and
|
|
`condition_text_lower`, plus both abbreviated and long-form wind-direction
|
|
fields.
|
|
- `hourly_forecast.periods[]` exposes both `period_begins` and `hour_label`,
|
|
and both `text_description` and `text_description_lower`.
|
|
- `derived_daypart_summaries` exposes numerous temperature and condition phrase
|
|
fields that are useful for deterministic template wording but noisy in the
|
|
prompt data package.
|
|
|
|
The cleanup should not weaken template composability. Templates should still be
|
|
able to use rich module values and helper fields.
|
|
|
|
The cleanup applies to every report that consumes these modules, including the
|
|
generated-template `today`, `tomorrow`, and `daily` reports. The same
|
|
`derived_daypart_summaries` prompt export should serve all three reports while
|
|
their templates continue to use rich daypart helper fields.
|
|
|
|
## Intent
|
|
|
|
Data packages should be curated prompt inputs, not a raw dump of every field
|
|
available to Go templates.
|
|
|
|
The intended architecture is:
|
|
|
|
- module builders produce rich internal/template module values;
|
|
- each module may define a prompt-facing export value for data-package use;
|
|
- prompt input construction serializes the prompt-facing export value;
|
|
- template rendering continues to use the full rich module value.
|
|
|
|
The result should let `weatherreporter` optimize separately for:
|
|
|
|
- precise deterministic Markdown rendering;
|
|
- compact, readable LLM input;
|
|
- stable internal module contracts.
|
|
|
|
## Locked Decisions
|
|
|
|
- Do not make the existing module structs smaller solely to clean up data
|
|
packages.
|
|
- Do not use per-module string field allowlists as the primary mechanism.
|
|
- Do not rely on reflection-heavy field filtering for nested module shapes.
|
|
- Do not use `json:"-"` or `yaml:"-"` on rich template fields as the main
|
|
boundary.
|
|
- Keep rich module outputs available for template rendering, inspection, tests,
|
|
and internal use.
|
|
- Add an explicit prompt/data-package export layer for module outputs.
|
|
- Simple modules may use default pass-through export behavior.
|
|
- No compatibility aliases are needed for removed prompt-facing fields because
|
|
the prompt schema is still pre-release.
|
|
- Bump the data-package schema version when implementing this change.
|
|
- Compute prompt export values during module snapshot construction and store the
|
|
runtime-only prompt value on `module.Output` alongside the rich `Value`.
|
|
- Do not persist prompt export values in module snapshot JSON; module snapshots
|
|
should continue to preserve rich module values.
|
|
|
|
## Target Architecture
|
|
|
|
Each module definition should be able to declare how its output is represented
|
|
in prompt data packages.
|
|
|
|
A possible shape is:
|
|
|
|
```go
|
|
type ModuleDefinition struct {
|
|
// existing fields...
|
|
PromptExporter ModulePromptExporter
|
|
}
|
|
|
|
type ModulePromptExporter func(value any) (any, error)
|
|
```
|
|
|
|
The exact API may differ if implementation discovers a cleaner fit, but the
|
|
contract should preserve these properties:
|
|
|
|
- the exporter is owned near the module definition or module builder;
|
|
- the exporter receives the rich module value and returns a prompt-facing value;
|
|
- missing exporters default to pass-through for modules whose rich value is
|
|
already prompt-appropriate;
|
|
- exporter errors include module ID and stanza context;
|
|
- promptinput uses exported prompt values instead of rich values;
|
|
- render contexts and templates continue using rich values.
|
|
|
|
The preferred implementation should avoid making `internal/promptinput` import
|
|
`internal/briefing` directly. If prompt export needs registry knowledge, either:
|
|
|
|
- record the prompt-facing value in `module.Output` when the module snapshot is
|
|
built; or
|
|
- pass an explicit export map/registry into prompt-input construction without
|
|
creating a package cycle.
|
|
|
|
The implementation should keep package boundaries consistent with existing
|
|
architecture: module output policy belongs with module definitions, and data
|
|
package serialization belongs in `internal/promptinput`.
|
|
|
|
## Prompt Export Contract
|
|
|
|
A module prompt export should be:
|
|
|
|
- **curated:** include fields useful to the LLM, omit fields used only for
|
|
deterministic sentence construction;
|
|
- **typed:** use small prompt-facing structs for modules that need reshaping;
|
|
- **stable:** keep field names intentional and avoid duplicating equivalent
|
|
facts under multiple names;
|
|
- **readable:** prefer fields that explain themselves in YAML;
|
|
- **loss-aware:** do not omit facts that the LLM needs to reason about timing,
|
|
severity, uncertainty, or practical impact;
|
|
- **module-owned:** keep each module responsible for its own prompt-facing
|
|
contract.
|
|
|
|
Prompt-facing structs may live next to the module that owns them, for example:
|
|
|
|
```go
|
|
type CurrentConditionsPromptExport struct {
|
|
ConditionText string `json:"condition_text,omitempty"`
|
|
TemperatureF *int `json:"temperature_f,omitempty"`
|
|
ApparentTemperatureF *int `json:"apparent_temperature_f,omitempty"`
|
|
RelativeHumidityPercent *int `json:"relative_humidity_percent,omitempty"`
|
|
WindSpeedMph *int `json:"wind_speed_mph,omitempty"`
|
|
WindDirection string `json:"wind_direction,omitempty"`
|
|
}
|
|
```
|
|
|
|
The names do not need to include `PromptExport` if implementation finds a
|
|
clearer convention, but they should distinguish data-package shape from
|
|
template-rendering shape.
|
|
|
|
## Initial Cleanup Targets
|
|
|
|
### Current Conditions
|
|
|
|
Keep prompt-facing fields that express current observed conditions directly:
|
|
|
|
- `condition_text`
|
|
- `is_day`
|
|
- temperature fields
|
|
- apparent temperature fields
|
|
- dewpoint fields
|
|
- relative humidity
|
|
- wind speed
|
|
- one wind direction field
|
|
|
|
Remove prompt-facing fields that are template-only duplicates:
|
|
|
|
- `condition_text_lower`
|
|
- duplicate wind-direction text when an equivalent `wind_direction` field is
|
|
present
|
|
|
|
The template surface may keep those helper fields.
|
|
|
|
### Hourly Forecast
|
|
|
|
Keep prompt-facing period fields that carry facts:
|
|
|
|
- `period_begins`
|
|
- `period_ends`
|
|
- `name`
|
|
- `is_day`
|
|
- condition code, if useful
|
|
- `text_description`
|
|
- temperature fields
|
|
- dewpoint, apparent temperature, humidity, wind, gust, pressure, visibility,
|
|
cloud cover, precipitation probability, precipitation amount, snowfall depth,
|
|
and UV index when provided by upstream data
|
|
|
|
Remove prompt-facing fields that duplicate or encode template logic:
|
|
|
|
- `hour_label`, because `period_begins` already gives the time in a friendly
|
|
local label;
|
|
- `text_description_lower`, because the LLM can interpret
|
|
`text_description`;
|
|
- `mention_precipitation`, because it is a template threshold helper when the
|
|
underlying precipitation probability is present.
|
|
|
|
The template surface may keep these helper fields.
|
|
|
|
### Derived Daypart Summaries
|
|
|
|
Keep prompt-facing fields that describe the daypart:
|
|
|
|
- `date`
|
|
- `display_name`
|
|
- `period_begins`
|
|
- `period_ends`
|
|
- temperature range or the best single temperature phrase
|
|
- apparent temperature range when useful
|
|
- maximum precipitation probability and time
|
|
- maximum wind gust and time
|
|
- dominant condition
|
|
- temperature trend
|
|
- notable conditions
|
|
- weather indicator booleans
|
|
- relevant alert count
|
|
|
|
Remove prompt-facing fields that mainly support deterministic sentence
|
|
construction:
|
|
|
|
- duplicate lower-case/display variants of the same dominant condition;
|
|
- multiple temperature phrase fragments when a smaller set can express the same
|
|
trend;
|
|
- duplicate time labels where one friendly time field is enough.
|
|
|
|
The exact retained daypart temperature fields should be chosen during
|
|
implementation with template needs and LLM readability in mind. The prompt
|
|
export should preserve the facts needed to understand whether temperatures are
|
|
rising, falling, peaking, or steady, but it does not need every phrase fragment
|
|
used by the Markdown template.
|
|
|
|
### Other Modules
|
|
|
|
Most existing modules may initially use pass-through export unless they expose
|
|
clear template-only helpers. During implementation, review at least:
|
|
|
|
- `narrative_forecast`
|
|
- `precip_timing`
|
|
- `outdoor_windows`
|
|
- `alert_digest`
|
|
- `spc_convective_outlooks`
|
|
- `spc_convective_discussion`
|
|
- `area_forecast_discussion`
|
|
- `weather_story`
|
|
- planning modules
|
|
|
|
Do not remove fields merely because they are verbose. Remove or reshape fields
|
|
when they are redundant, template-specific, or confusing in the context of LLM
|
|
input.
|
|
|
|
## Data Package Behavior
|
|
|
|
After implementation:
|
|
|
|
- saved YAML data packages should use prompt-facing module exports;
|
|
- saved module snapshots should continue preserving rich module output values;
|
|
- generated-text render contexts should continue preserving rich module values;
|
|
- Recent Changes should continue using structured module snapshots unless a
|
|
specific comparison should intentionally move to prompt-facing fields;
|
|
- inspection commands should make clear whether they are showing rich module
|
|
snapshots or prompt data packages.
|
|
- generated-template reports, including `today`, `tomorrow`, and `daily`, should
|
|
continue rendering from rich module values.
|
|
|
|
This roadmap does not require changing source warnings, report metadata,
|
|
collected facts, derived facts, or generated report artifacts.
|
|
|
|
## Schema And Versioning
|
|
|
|
This is a prompt-input schema cleanup. Because the project is pre-release, the
|
|
implementation may make a clean break in data-package field names without
|
|
compatibility aliases.
|
|
|
|
The data-package schema version should be bumped when this feature is
|
|
implemented because persisted data-package fields will be removed or renamed.
|
|
This makes artifact shape changes explicit and helps inspection tooling
|
|
distinguish old and new data packages.
|
|
|
|
## Documentation Guidance
|
|
|
|
After implementation, update implemented documentation only:
|
|
|
|
- `docs/internal/module.md`: describe the distinction between rich module output
|
|
and prompt-facing export values.
|
|
- `docs/internal/prompt-input.md`: document that data packages use curated
|
|
prompt exports, not full template module structs.
|
|
- `docs/templates.md`: clarify that templates may have richer fields than the
|
|
data package.
|
|
- Any module field examples in implemented docs should match the new
|
|
prompt-facing data package shape.
|
|
|
|
Do not document future module fields or unimplemented exporters outside
|
|
`docs/roadmap/`.
|
|
|
|
## Acceptance Criteria
|
|
|
|
The feature is complete when:
|
|
|
|
- prompt data packages serialize curated module exports instead of blindly
|
|
serializing rich module values;
|
|
- templates still render from rich module values without losing helper fields;
|
|
- `current_conditions` no longer exposes lower-case condition text or duplicate
|
|
wind-direction fields in data packages;
|
|
- `hourly_forecast.periods[]` no longer exposes `hour_label`,
|
|
`text_description_lower`, or `mention_precipitation` in data packages;
|
|
- `derived_daypart_summaries` no longer exposes redundant condition and
|
|
temperature phrase variants in data packages;
|
|
- `today`, `tomorrow`, and `daily` rendered reports continue to have access to
|
|
rich daypart helper fields for deterministic template wording;
|
|
- simple modules that do not need cleanup still export correctly through default
|
|
pass-through behavior;
|
|
- exporter errors include module/stanza context;
|
|
- YAML category ordering remains unchanged;
|
|
- module snapshot artifacts remain rich enough for templates, inspection, and
|
|
regression diagnosis;
|
|
- tests prove that removed prompt-facing fields are absent from saved YAML data
|
|
packages and still available to templates where needed.
|
|
|
|
## Testing Expectations
|
|
|
|
Implementation should add or update focused tests for:
|
|
|
|
- module registry validation for prompt exporters, if exporters are registered
|
|
there;
|
|
- promptinput construction using exported prompt values;
|
|
- pass-through behavior for simple modules;
|
|
- custom exports for current conditions, hourly forecast, and daypart summaries;
|
|
- data-package YAML output rejecting stale fields;
|
|
- template render tests proving template-only helper fields remain available for
|
|
`today`, `tomorrow`, and `daily`;
|
|
- app workflow tests proving saved data packages use curated exports while
|
|
render contexts keep rich values.
|
|
|
|
Suggested validation after implementation:
|
|
|
|
```sh
|
|
go test ./internal/module ./internal/briefing ./internal/promptinput
|
|
go test ./internal/generatedtext ./internal/reporttemplate ./internal/app
|
|
go test ./...
|
|
go run ./cmd/weatherreporter --help
|
|
git diff --check
|
|
```
|
|
|
|
## Deferred Work
|
|
|
|
Do not include these in the initial cleanup unless implementation reveals they
|
|
are necessary:
|
|
|
|
- user-configurable data-package field selection;
|
|
- per-prompt custom field profiles;
|
|
- reflection-based generic include/exclude lists;
|
|
- automatic schema generation for data-package exports;
|
|
- changing collected facts or derived facts contracts;
|
|
- changing Scriptorium invocation behavior;
|
|
- changing generated Markdown templates beyond preserving their current output.
|
|
|
|
## Open Questions
|
|
|
|
No open questions block implementation.
|
|
|
|
The implementation plan in `docs/roadmap/implementation.md` is the sequencing
|
|
authority for this feature.
|