Add feature roadmap and implementation plan to clean up and rationalize the fields provided to the data package
This commit is contained in:
360
docs/roadmap/data-package-exports.md
Normal file
360
docs/roadmap/data-package-exports.md
Normal file
@@ -0,0 +1,360 @@
|
||||
# Data Package Export Roadmap
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the target state for cleaning up module fields exposed in
|
||||
YAML data packages. The goal is to keep report templates composable while making
|
||||
LLM prompt inputs concise, readable, and free of template-only helper fields.
|
||||
|
||||
This feature is not implemented yet. Current data-package behavior remains
|
||||
documented outside `docs/roadmap/`.
|
||||
|
||||
## Problem
|
||||
|
||||
Module output structs currently serve two different consumers:
|
||||
|
||||
- deterministic report templates, which benefit from presentation helpers such
|
||||
as lower-case text, display labels, trend phrases, and hour labels;
|
||||
- Scriptorium data packages, which should expose the clearest useful weather
|
||||
facts to the LLM with minimal redundancy.
|
||||
|
||||
Those consumers now need different surfaces. Examples include:
|
||||
|
||||
- `current_conditions` exposes both `condition_text` and
|
||||
`condition_text_lower`, plus both abbreviated and long-form wind-direction
|
||||
fields.
|
||||
- `hourly_forecast.periods[]` exposes both `period_begins` and `hour_label`,
|
||||
and both `text_description` and `text_description_lower`.
|
||||
- `derived_daypart_summaries` exposes numerous temperature and condition phrase
|
||||
fields that are useful for deterministic template wording but noisy in the
|
||||
prompt data package.
|
||||
|
||||
The cleanup should not weaken template composability. Templates should still be
|
||||
able to use rich module values and helper fields.
|
||||
|
||||
The cleanup applies to every report that consumes these modules, including the
|
||||
generated-template `today`, `tomorrow`, and `daily` reports. The same
|
||||
`derived_daypart_summaries` prompt export should serve all three reports while
|
||||
their templates continue to use rich daypart helper fields.
|
||||
|
||||
## Intent
|
||||
|
||||
Data packages should be curated prompt inputs, not a raw dump of every field
|
||||
available to Go templates.
|
||||
|
||||
The intended architecture is:
|
||||
|
||||
- module builders produce rich internal/template module values;
|
||||
- each module may define a prompt-facing export value for data-package use;
|
||||
- prompt input construction serializes the prompt-facing export value;
|
||||
- template rendering continues to use the full rich module value.
|
||||
|
||||
The result should let `weatherreporter` optimize separately for:
|
||||
|
||||
- precise deterministic Markdown rendering;
|
||||
- compact, readable LLM input;
|
||||
- stable internal module contracts.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
- Do not make the existing module structs smaller solely to clean up data
|
||||
packages.
|
||||
- Do not use per-module string field allowlists as the primary mechanism.
|
||||
- Do not rely on reflection-heavy field filtering for nested module shapes.
|
||||
- Do not use `json:"-"` or `yaml:"-"` on rich template fields as the main
|
||||
boundary.
|
||||
- Keep rich module outputs available for template rendering, inspection, tests,
|
||||
and internal use.
|
||||
- Add an explicit prompt/data-package export layer for module outputs.
|
||||
- Simple modules may use default pass-through export behavior.
|
||||
- No compatibility aliases are needed for removed prompt-facing fields because
|
||||
the prompt schema is still pre-release.
|
||||
- Bump the data-package schema version when implementing this change.
|
||||
- Compute prompt export values during module snapshot construction and store the
|
||||
runtime-only prompt value on `module.Output` alongside the rich `Value`.
|
||||
- Do not persist prompt export values in module snapshot JSON; module snapshots
|
||||
should continue to preserve rich module values.
|
||||
|
||||
## Target Architecture
|
||||
|
||||
Each module definition should be able to declare how its output is represented
|
||||
in prompt data packages.
|
||||
|
||||
A possible shape is:
|
||||
|
||||
```go
|
||||
type ModuleDefinition struct {
|
||||
// existing fields...
|
||||
PromptExporter ModulePromptExporter
|
||||
}
|
||||
|
||||
type ModulePromptExporter func(value any) (any, error)
|
||||
```
|
||||
|
||||
The exact API may differ if implementation discovers a cleaner fit, but the
|
||||
contract should preserve these properties:
|
||||
|
||||
- the exporter is owned near the module definition or module builder;
|
||||
- the exporter receives the rich module value and returns a prompt-facing value;
|
||||
- missing exporters default to pass-through for modules whose rich value is
|
||||
already prompt-appropriate;
|
||||
- exporter errors include module ID and stanza context;
|
||||
- promptinput uses exported prompt values instead of rich values;
|
||||
- render contexts and templates continue using rich values.
|
||||
|
||||
The preferred implementation should avoid making `internal/promptinput` import
|
||||
`internal/briefing` directly. If prompt export needs registry knowledge, either:
|
||||
|
||||
- record the prompt-facing value in `module.Output` when the module snapshot is
|
||||
built; or
|
||||
- pass an explicit export map/registry into prompt-input construction without
|
||||
creating a package cycle.
|
||||
|
||||
The implementation should keep package boundaries consistent with existing
|
||||
architecture: module output policy belongs with module definitions, and data
|
||||
package serialization belongs in `internal/promptinput`.
|
||||
|
||||
## Prompt Export Contract
|
||||
|
||||
A module prompt export should be:
|
||||
|
||||
- **curated:** include fields useful to the LLM, omit fields used only for
|
||||
deterministic sentence construction;
|
||||
- **typed:** use small prompt-facing structs for modules that need reshaping;
|
||||
- **stable:** keep field names intentional and avoid duplicating equivalent
|
||||
facts under multiple names;
|
||||
- **readable:** prefer fields that explain themselves in YAML;
|
||||
- **loss-aware:** do not omit facts that the LLM needs to reason about timing,
|
||||
severity, uncertainty, or practical impact;
|
||||
- **module-owned:** keep each module responsible for its own prompt-facing
|
||||
contract.
|
||||
|
||||
Prompt-facing structs may live next to the module that owns them, for example:
|
||||
|
||||
```go
|
||||
type CurrentConditionsPromptExport struct {
|
||||
ConditionText string `json:"condition_text,omitempty"`
|
||||
TemperatureF *int `json:"temperature_f,omitempty"`
|
||||
ApparentTemperatureF *int `json:"apparent_temperature_f,omitempty"`
|
||||
RelativeHumidityPercent *int `json:"relative_humidity_percent,omitempty"`
|
||||
WindSpeedMph *int `json:"wind_speed_mph,omitempty"`
|
||||
WindDirection string `json:"wind_direction,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
The names do not need to include `PromptExport` if implementation finds a
|
||||
clearer convention, but they should distinguish data-package shape from
|
||||
template-rendering shape.
|
||||
|
||||
## Initial Cleanup Targets
|
||||
|
||||
### Current Conditions
|
||||
|
||||
Keep prompt-facing fields that express current observed conditions directly:
|
||||
|
||||
- `condition_text`
|
||||
- `is_day`
|
||||
- temperature fields
|
||||
- apparent temperature fields
|
||||
- dewpoint fields
|
||||
- relative humidity
|
||||
- wind speed
|
||||
- one wind direction field
|
||||
|
||||
Remove prompt-facing fields that are template-only duplicates:
|
||||
|
||||
- `condition_text_lower`
|
||||
- duplicate wind-direction text when an equivalent `wind_direction` field is
|
||||
present
|
||||
|
||||
The template surface may keep those helper fields.
|
||||
|
||||
### Hourly Forecast
|
||||
|
||||
Keep prompt-facing period fields that carry facts:
|
||||
|
||||
- `period_begins`
|
||||
- `period_ends`
|
||||
- `name`
|
||||
- `is_day`
|
||||
- condition code, if useful
|
||||
- `text_description`
|
||||
- temperature fields
|
||||
- dewpoint, apparent temperature, humidity, wind, gust, pressure, visibility,
|
||||
cloud cover, precipitation probability, precipitation amount, snowfall depth,
|
||||
and UV index when provided by upstream data
|
||||
|
||||
Remove prompt-facing fields that duplicate or encode template logic:
|
||||
|
||||
- `hour_label`, because `period_begins` already gives the time in a friendly
|
||||
local label;
|
||||
- `text_description_lower`, because the LLM can interpret
|
||||
`text_description`;
|
||||
- `mention_precipitation`, because it is a template threshold helper when the
|
||||
underlying precipitation probability is present.
|
||||
|
||||
The template surface may keep these helper fields.
|
||||
|
||||
### Derived Daypart Summaries
|
||||
|
||||
Keep prompt-facing fields that describe the daypart:
|
||||
|
||||
- `date`
|
||||
- `display_name`
|
||||
- `period_begins`
|
||||
- `period_ends`
|
||||
- temperature range or the best single temperature phrase
|
||||
- apparent temperature range when useful
|
||||
- maximum precipitation probability and time
|
||||
- maximum wind gust and time
|
||||
- dominant condition
|
||||
- temperature trend
|
||||
- notable conditions
|
||||
- weather indicator booleans
|
||||
- relevant alert count
|
||||
|
||||
Remove prompt-facing fields that mainly support deterministic sentence
|
||||
construction:
|
||||
|
||||
- duplicate lower-case/display variants of the same dominant condition;
|
||||
- multiple temperature phrase fragments when a smaller set can express the same
|
||||
trend;
|
||||
- duplicate time labels where one friendly time field is enough.
|
||||
|
||||
The exact retained daypart temperature fields should be chosen during
|
||||
implementation with template needs and LLM readability in mind. The prompt
|
||||
export should preserve the facts needed to understand whether temperatures are
|
||||
rising, falling, peaking, or steady, but it does not need every phrase fragment
|
||||
used by the Markdown template.
|
||||
|
||||
### Other Modules
|
||||
|
||||
Most existing modules may initially use pass-through export unless they expose
|
||||
clear template-only helpers. During implementation, review at least:
|
||||
|
||||
- `narrative_forecast`
|
||||
- `precip_timing`
|
||||
- `outdoor_windows`
|
||||
- `alert_digest`
|
||||
- `spc_convective_outlooks`
|
||||
- `spc_convective_discussion`
|
||||
- `area_forecast_discussion`
|
||||
- `weather_story`
|
||||
- planning modules
|
||||
|
||||
Do not remove fields merely because they are verbose. Remove or reshape fields
|
||||
when they are redundant, template-specific, or confusing in the context of LLM
|
||||
input.
|
||||
|
||||
## Data Package Behavior
|
||||
|
||||
After implementation:
|
||||
|
||||
- saved YAML data packages should use prompt-facing module exports;
|
||||
- saved module snapshots should continue preserving rich module output values;
|
||||
- generated-text render contexts should continue preserving rich module values;
|
||||
- Recent Changes should continue using structured module snapshots unless a
|
||||
specific comparison should intentionally move to prompt-facing fields;
|
||||
- inspection commands should make clear whether they are showing rich module
|
||||
snapshots or prompt data packages.
|
||||
- generated-template reports, including `today`, `tomorrow`, and `daily`, should
|
||||
continue rendering from rich module values.
|
||||
|
||||
This roadmap does not require changing source warnings, report metadata,
|
||||
collected facts, derived facts, or generated report artifacts.
|
||||
|
||||
## Schema And Versioning
|
||||
|
||||
This is a prompt-input schema cleanup. Because the project is pre-release, the
|
||||
implementation may make a clean break in data-package field names without
|
||||
compatibility aliases.
|
||||
|
||||
The data-package schema version should be bumped when this feature is
|
||||
implemented because persisted data-package fields will be removed or renamed.
|
||||
This makes artifact shape changes explicit and helps inspection tooling
|
||||
distinguish old and new data packages.
|
||||
|
||||
## Documentation Guidance
|
||||
|
||||
After implementation, update implemented documentation only:
|
||||
|
||||
- `docs/internal/module.md`: describe the distinction between rich module output
|
||||
and prompt-facing export values.
|
||||
- `docs/internal/prompt-input.md`: document that data packages use curated
|
||||
prompt exports, not full template module structs.
|
||||
- `docs/templates.md`: clarify that templates may have richer fields than the
|
||||
data package.
|
||||
- Any module field examples in implemented docs should match the new
|
||||
prompt-facing data package shape.
|
||||
|
||||
Do not document future module fields or unimplemented exporters outside
|
||||
`docs/roadmap/`.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The feature is complete when:
|
||||
|
||||
- prompt data packages serialize curated module exports instead of blindly
|
||||
serializing rich module values;
|
||||
- templates still render from rich module values without losing helper fields;
|
||||
- `current_conditions` no longer exposes lower-case condition text or duplicate
|
||||
wind-direction fields in data packages;
|
||||
- `hourly_forecast.periods[]` no longer exposes `hour_label`,
|
||||
`text_description_lower`, or `mention_precipitation` in data packages;
|
||||
- `derived_daypart_summaries` no longer exposes redundant condition and
|
||||
temperature phrase variants in data packages;
|
||||
- `today`, `tomorrow`, and `daily` rendered reports continue to have access to
|
||||
rich daypart helper fields for deterministic template wording;
|
||||
- simple modules that do not need cleanup still export correctly through default
|
||||
pass-through behavior;
|
||||
- exporter errors include module/stanza context;
|
||||
- YAML category ordering remains unchanged;
|
||||
- module snapshot artifacts remain rich enough for templates, inspection, and
|
||||
regression diagnosis;
|
||||
- tests prove that removed prompt-facing fields are absent from saved YAML data
|
||||
packages and still available to templates where needed.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Implementation should add or update focused tests for:
|
||||
|
||||
- module registry validation for prompt exporters, if exporters are registered
|
||||
there;
|
||||
- promptinput construction using exported prompt values;
|
||||
- pass-through behavior for simple modules;
|
||||
- custom exports for current conditions, hourly forecast, and daypart summaries;
|
||||
- data-package YAML output rejecting stale fields;
|
||||
- template render tests proving template-only helper fields remain available for
|
||||
`today`, `tomorrow`, and `daily`;
|
||||
- app workflow tests proving saved data packages use curated exports while
|
||||
render contexts keep rich values.
|
||||
|
||||
Suggested validation after implementation:
|
||||
|
||||
```sh
|
||||
go test ./internal/module ./internal/briefing ./internal/promptinput
|
||||
go test ./internal/generatedtext ./internal/reporttemplate ./internal/app
|
||||
go test ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
## Deferred Work
|
||||
|
||||
Do not include these in the initial cleanup unless implementation reveals they
|
||||
are necessary:
|
||||
|
||||
- user-configurable data-package field selection;
|
||||
- per-prompt custom field profiles;
|
||||
- reflection-based generic include/exclude lists;
|
||||
- automatic schema generation for data-package exports;
|
||||
- changing collected facts or derived facts contracts;
|
||||
- changing Scriptorium invocation behavior;
|
||||
- changing generated Markdown templates beyond preserving their current output.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No open questions block implementation.
|
||||
|
||||
The implementation plan in `docs/roadmap/implementation.md` is the sequencing
|
||||
authority for this feature.
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user