ejr/weatherfeeder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Clean up completed roadmap and documentation for the new outlook schema
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
Details

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-12 08:30:41 -05:00
parent 4358a7cdce
commit 74411e3f54
3 changed files with 1 additions and 744 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/roadmap/future.md
View File

@@ -31,7 +31,7 @@ Notes:
- Day 4-8 products have different semantics from Day 1-3 categorical/tornado/hail/wind products.
- Avoid forcing Day 4-8 assumptions into the current Day 1-3 model until the source shapes and consumer needs are reviewed.
- Prefer reusing `weather.outlook.v1` if the fields remain accurate; otherwise write a separate roadmap before changing the canonical contract.
- Prefer reusing `weather.outlook.v2` if the fields remain accurate; otherwise write a separate roadmap before changing the canonical contract.
### Degraded SPC Bundle Mode

388
docs/roadmap/implementation.md
View File

@@ -1,388 +0,0 @@
# Implement SPC Outlook V2 Location Filtering
## Summary
Implement the roadmap in `docs/roadmap/outlook.md` as a behavior-changing canonical contract update for SPC convective outlooks.
The implementation must make `weatherfeeder` emit location-relevant SPC outlook snapshots instead of all SPC polygons, and move SPC print-page prose from each polygon to run-level day discussions.
This plan is decision-complete. Do not preserve the old `weather.outlook.v1` output behavior in the SPC normalizer. Implement `weather.outlook.v2`, update the Postgres sink contract, add transition documentation, and update current-behavior docs after the code change lands.
## Required Target Behavior
- Raw SPC source behavior remains complete and unfiltered.
- Event kind remains `outlook`.
- Raw schema remains `raw.spc.convective_outlook.v1`.
- Canonical SPC outlook output changes from `weather.outlook.v1` to `weather.outlook.v2`.
- Canonical outlook runs include only polygons where the configured forecast location is inside or on the boundary.
- Canonical outlook runs are still emitted when no polygons apply locally.
- No local polygons means `outlooks: []` and `discussions: []`.
- Day discussions are run-level entries in `WeatherOutlookRun.discussions`.
- Discussions are included only for days represented by at least one retained outlook.
- Multiple retained outlooks for the same day share one run-level discussion entry.
- Polygon-level `headline`, `summary`, and `discussion` fields are removed from `model.WeatherOutlook`.
- `containsLocation` remains on `WeatherOutlook` and must be `true` for every v2 emitted outlook.
- Empty SPC `GeometryCollection` no-risk placeholders remain skipped and still contribute issue time.
- Supersession semantics are documented in permanent docs: latest-run semantics are preferred; historical row supersession uses `provider`, `product`, `outlookType`, `validFrom`, `validTo`, and greatest `issuedAt`; `day` and `label` are not identity fields.
## Guardrails
- Do not move point-in-polygon filtering into the source layer.
- Do not fetch additional SPC products, images, shapefiles, or Day 4-8 products.
- Do not make `weatherfeeder` an archive of national SPC polygons.
- Do not add a separate outlook-discussion event kind.
- Do not introduce feedkit changes.
- Do not use column-level compatibility migrations for the outlook table family. Existing SPC outlook history may be discarded, so the transition should drop and recreate outlook tables.
- Do not update non-roadmap current-behavior docs before the corresponding behavior is implemented.
- Keep implementation inside existing architecture boundaries: `model`, `standards`, SPC normalizer/provider helpers, and Postgres sink mapping.
## Stage 1: Standards And Canonical Model
### Code Changes
- Add `standards.SchemaWeatherOutlookV2 = "weather.outlook.v2"` in `standards/schema.go`.
- Keep `standards.SchemaWeatherOutlookV1` for historical references and any existing tests that still validate documented constants.
- Update `model/outlook.go`:
- add `WeatherOutlookDiscussion`;
- add `Discussions []WeatherOutlookDiscussion` to `WeatherOutlookRun` with JSON tag `json:"discussions"`;
- remove `Headline`, `Summary`, and `Discussion` fields from `WeatherOutlook`.
- Keep all other `WeatherOutlook` fields unchanged, including `ContainsLocation` and `Geometry`.
- Update any code or tests that construct `WeatherOutlook` values to remove the deleted polygon-level prose fields.
- Do not change raw SPC provider structs in this stage.
### Tests
- Update model documentation/consumer tests so `WeatherOutlookRun`, `WeatherOutlook`, and `WeatherOutlookDiscussion` are listed where applicable.
- Update standards documentation tests so `weather.outlook.v2` is expected.
- Add or update a JSON-shape test to verify:
- `WeatherOutlookRun` serializes `discussions`;
- `WeatherOutlook` no longer serializes polygon-level `headline`, `summary`, or `discussion`.
### Verification
```sh
go test ./model ./standards
```
## Stage 2: SPC Normalizer V2 Output
### Code Changes
- Update `internal/normalizers/spc/convective_outlook.go` so `Normalize` emits `standards.SchemaWeatherOutlookV2`.
- Keep matching raw input schema `standards.SchemaRawSPCConvectiveOutlookV1`.
- Keep parsing all required print-page discussions before mapping features.
- Keep decoding every configured GeoJSON product.
- Keep skipping empty `GeometryCollection` placeholders.
- Track latest GeoJSON issue time across all real features and empty placeholders, not only retained local polygons.
- For every real feature:
- parse required timestamps and label as today;
- compute `containsLocation` using existing `geo.ContainsPoint`;
- if `containsLocation == false`, do not append a canonical outlook;
- if `containsLocation == true`, append the canonical outlook with `ContainsLocation: true`.
- Do not attach `headline`, `summary`, or `discussion` to retained outlooks.
- After retained outlooks are built, build `WeatherOutlookRun.Discussions`:
- collect unique days present in retained outlooks;
- include one `WeatherOutlookDiscussion` per retained day;
- sort discussions by day ascending;
- map `Day`, `Headline`, `Summary`, `Discussion`, and `UpdatedAt` from parsed print-page data;
- omit discussions for days with no retained outlooks.
- Preserve existing product ordering for retained outlooks: day, outlook type order, then feature order.
- Preserve `asOf` and normalized event `effective_at` policy:
- latest valid GeoJSON issue time across the complete bundle;
- latest print-page update time;
- incoming event `effective_at`;
- incoming event `emitted_at`.
- Preserve `WeatherOutlookRun.IssuedAt` as latest valid GeoJSON issue time when available, even when `outlooks` is empty.
### Tests
Update `internal/normalizers/spc/convective_outlook_test.go`:
- Existing sample test should expect `out.Schema == standards.SchemaWeatherOutlookV2`.
- Existing in-location sample should expect only location-contained outlooks.
- Outside-location sample should normalize successfully with:
- `len(run.Outlooks) == 0`;
- `len(run.Discussions) == 0`;
- non-zero `run.AsOf` from latest product issue time;
- normalized event `EffectiveAt == run.AsOf`.
- Add a fixture/test case where only Day 2 contains the location and assert:
- all retained outlooks have `Day == 2`;
- `run.Discussions` length is 1;
- discussion day is 2;
- Day 1 and Day 3 discussions are absent.
- Add a fixture/test case where multiple Day 1 outlook types contain the location and assert:
- multiple retained Day 1 outlooks are present;
- exactly one Day 1 discussion is present.
- Assert every retained outlook has `ContainsLocation == true`.
- Keep the empty `GeometryCollection` regression test and update it for v2 shape.
- Keep malformed timestamp, malformed geometry, missing discussion, missing label, and product ordering tests.
- Replace old tests that expected all nine fixture polygons to be emitted with location-filtered expectations.
- Replace tests that asserted polygon-level discussion text with run-level discussion assertions.
### Verification
```sh
go test ./internal/providers/spc ./internal/normalizers/spc ./internal/geo
```
## Stage 3: Postgres Schema And Mapper
### Schema Changes
Update `internal/sinks/postgres/schema.go`:
- Add table constant `tableOutlookDiscussions = "outlook_discussions"`.
- Add `discussion_count INTEGER NOT NULL` to `outlook_runs`.
- Add `outlook_discussions` table:
- `run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL`;
- `discussion_index INTEGER NOT NULL`;
- `as_of TIMESTAMPTZ NOT NULL`;
- `day INTEGER NOT NULL`;
- `headline TEXT NULL`;
- `summary TEXT NULL`;
- `discussion TEXT NULL`;
- `updated_at TIMESTAMPTZ NULL`.
- Set primary key to `(run_event_id, discussion_index)`.
- Set prune column to `as_of`.
- Add index `idx_wf_outlook_discussions_day_as_of` on `(day, as_of)`.
- Add unique index `idx_wf_outlook_discussions_run_day` on `(run_event_id, day)` using `fksinks.PostgresIndex{Unique: true}`.
- Remove `headline`, `summary`, and `discussion` from the target `outlooks` schema. Do not carry these legacy nullable columns forward.
### Mapper Changes
Update `internal/sinks/postgres/map.go`:
- Route `standards.SchemaWeatherOutlookV2` to the outlook mapper.
- Remove `standards.SchemaWeatherOutlookV1` support from the active outlook mapper unless there is a compile-time reason to keep a clearly documented legacy test. Existing v1 outlook rows are intentionally discarded during the table reset and the SPC normalizer must not produce v1 events.
- Decode payload into the updated `model.WeatherOutlookRun`.
- Write `outlook_runs.discussion_count = len(run.Discussions)`.
- Write one `outlook_discussions` row for each `run.Discussions[i]`.
- Use `discussion_index` as the array position.
- Use parent `run.AsOf.UTC()` for `as_of`.
- Normalize `UpdatedAt` to UTC when present.
- Validate discussion entries before writing:
- `day` must be 1, 2, or 3;
- at least one of `headline`, `summary`, or `discussion` must be non-empty;
- duplicate discussion days in one run should fail before hitting the unique index.
- Continue validating outlook required fields:
- `id`, `provider`, `product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`, `expiresAt`, and `geometry` remain required.
- Add a v2-specific invariant validation: every persisted v2 outlook must have `ContainsLocation == true`. If the mapper continues to accept v1, apply this invariant only to v2.
- Continue compacting geometry JSON and normalizing all timestamps to UTC.
### Tests
Update `internal/sinks/postgres/schema_test.go`:
- Assert `outlook_runs` includes `discussion_count`.
- Assert `outlook_discussions` exists with expected columns, primary key, prune column, day/as-of index, and unique run/day index.
- Assert `outlooks` does not include `headline`, `summary`, or `discussion`.
Update `internal/sinks/postgres/map_test.go`:
- Assert v2 outlook run writes:
- one parent row;
- one outlook row per retained outlook;
- one discussion row per run discussion;
- correct `outlook_count` and `discussion_count`.
- Assert empty local run writes parent row with `outlook_count = 0` and `discussion_count = 0` and no child rows.
- Assert discussion rows map day/headline/summary/discussion/updatedAt correctly.
- Assert duplicate discussion days fail with useful context.
- Assert invalid discussion day fails with useful context.
- Assert empty discussion content fails with useful context.
- Assert v2 outlook with `ContainsLocation == false` fails.
- Update existing tests that used polygon-level `Headline`, `Summary`, or `Discussion`.
- Keep compact geometry and UTC normalization tests.
### Verification
```sh
go test ./internal/sinks/postgres
```
## Stage 4: Normalizer Registry And Cross-Package Consistency
### Code Changes
- Confirm SPC normalizer registration remains unchanged except for output schema.
- Update any package-level tests under `internal/normalizers` that assert canonical schema routing or supported schema names.
- Update any source/registry documentation consistency tests that expect canonical schema strings.
- Search for stale `SchemaWeatherOutlookV1` usage:
```sh
rg "SchemaWeatherOutlookV1|weather\.outlook\.v1|headline|summary|discussion" model standards internal docs/consumers docs/integrations docs/internal
```
- Keep `SchemaWeatherOutlookV1` only where intentionally retained for historical docs or standards compatibility. Do not route new SPC normalized events to v1.
- Ensure no SPC normalizer test, source test, or current v2 documentation path still claims all polygons are emitted.
### Tests
```sh
go test ./internal/normalizers ./internal/normalizers/...
go test ./internal/sources ./cmd/weatherfeeder
```
## Stage 5: Transition Documentation
### Roadmap Transition Doc
Create `docs/roadmap/outlook-schema-transition.md` with operator transition guidance.
The document must include:
- Purpose: reset existing Postgres outlook tables from the `weather.outlook.v1` storage shape to the v2-compatible storage shape.
- Scope: drops only the outlook table family and lets updated `weatherfeeder` recreate it. Other weather tables are not affected.
- Explicit warning: these commands delete stored SPC outlook history.
- Deployment order:
1. Stop `weatherfeeder`.
2. Drop existing outlook tables.
3. Deploy updated `weatherfeeder`.
4. Start `weatherfeeder` so the Postgres sink recreates the v2 tables.
5. Deploy updated downstream consumers such as `weatherapi`.
- Transition SQL:
```sql
DROP TABLE IF EXISTS outlook_discussions;
DROP TABLE IF EXISTS outlooks;
DROP TABLE IF EXISTS outlook_runs;
```
- Verification SQL examples:
```sql
SELECT table_name
FROM information_schema.tables
WHERE table_name IN ('outlook_runs', 'outlooks', 'outlook_discussions')
ORDER BY table_name;
```
After the updated daemon has started and recreated tables:
```sql
SELECT column_name, is_nullable, data_type
FROM information_schema.columns
WHERE table_name = 'outlook_runs'
AND column_name = 'discussion_count';
SELECT indexname
FROM pg_indexes
WHERE tablename = 'outlook_discussions'
ORDER BY indexname;
SELECT column_name
FROM information_schema.columns
WHERE table_name = 'outlooks'
AND column_name IN ('headline', 'summary', 'discussion');
```
- State that the final verification query for legacy `outlooks` prose columns should return zero rows.
- State that v1 outlook rows are intentionally removed and downstream readers should be updated intentionally.
### Tests
No code tests are required for the roadmap transition doc, but documentation consistency tests may need updates if they check table names or schema strings.
## Stage 6: Permanent Documentation Updates
After the implementation is complete, update permanent current-behavior documentation. Do not leave these as roadmap-only notes.
### Required Docs
Update `docs/policy/architecture.md`:
- Add a core design principle that `weatherfeeder` is location-focused and canonical weather events should represent weather data relevant to the configured forecast location or configured provider object; it is not intended to archive all provider data for all places.
Update `docs/integrations/events.md`:
- Document `weather.outlook.v2`.
- Document `WeatherOutlookRun.discussions` and `WeatherOutlookDiscussion` fields.
- Document that `outlooks[]` is location-filtered.
- Document `containsLocation` as an invariant that is true for emitted v2 outlooks.
- Document empty local runs with `outlooks: []` and `discussions: []`.
- Document supersession semantics:
- latest-run semantics preferred;
- historical supersession key is `provider`, `product`, `outlookType`, `validFrom`, `validTo`;
- greatest `issuedAt` wins;
- `day` and `label` are not identity;
- preserve all polygons from the selected latest `issuedAt` group.
- Keep any v1 notes clearly marked legacy/historical if retained.
Update `docs/integrations/spc.md`:
- State raw SPC bundles remain complete.
- State canonical output is location-filtered.
- State discussions are day-level and only included for retained outlook days.
- Remove or replace claims that all polygons are emitted or discussion text is attached to every polygon.
- Add supersession guidance or link to the canonical section in `docs/integrations/events.md`.
Update `docs/integrations/postgres.md`:
- Add `outlook_runs.discussion_count`.
- Add `outlook_discussions` table, columns, keys, indexes, prune column, and mapping source.
- Remove `outlooks.headline`, `outlooks.summary`, and `outlooks.discussion` from the current table contract.
- Document v2 reader reconstruction: read `outlook_runs`, `outlooks`, and `outlook_discussions` by `run_event_id` ordered by child indexes.
Update `docs/internal/postgres-sink.md` and `internal/sinks/postgres/doc.go`:
- Document the new table and mapper validation rules.
Update `docs/consumers/pkg-model.md` and `docs/consumers/api.md`:
- Add `WeatherOutlookDiscussion`.
- Update model examples for run-level discussions.
- Remove polygon-level prose from v2 consumer examples.
Update `docs/config.md` only if SPC prose still describes all-polygons behavior.
Update `docs/operations.md` or `docs/troubleshooting.md` only if operational behavior or recovery instructions need adjustment.
### Tests
- Update docs consistency tests for `weather.outlook.v2`.
- Update docs consistency tests for `outlook_discussions` if such tests exist or are practical.
- Ensure no permanent doc still describes v1 all-polygons behavior as current behavior:
```sh
rg "All outlook polygons|all polygons|attached to every outlook|weather\.outlook\.v1" docs README.md
```
## Stage 7: Full Verification And Release Readiness
### Focused Tests
```sh
go test ./model ./standards
go test ./internal/providers/spc ./internal/normalizers/spc ./internal/normalizers ./internal/geo
go test ./internal/sinks/postgres
go test ./internal/sources ./cmd/weatherfeeder
```
### Full Test Suite
```sh
go test ./...
```
### Manual Checks
- Confirm `cmd/weatherfeeder/config.yml` still loads and source registry tests pass.
- Confirm no current-behavior docs outside `docs/roadmap/` describe unimplemented behavior.
- Confirm `docs/roadmap/outlook-schema-transition.md` transition SQL and verification steps match the final table names and `internal/sinks/postgres/schema.go`.
- Confirm the SPC normalizer emits `weather.outlook.v2`, not `weather.outlook.v1`.
- Confirm a no-local-risk SPC bundle emits a canonical run with empty outlook and discussion arrays.
- Confirm a local-risk SPC bundle emits only containing polygons and only matching day discussions.
- Confirm retained v2 outlooks all have `containsLocation: true`.
## Weatherapi Coordination Notes
This stage is not implemented in `weatherfeeder`, but the weatherfeeder release should call it out for downstream work.
- Release `weatherfeeder` with `weather.outlook.v2` before updating `weatherapi` dependency.
- Update `weatherapi` Postgres reads to load `outlook_discussions`.
- Update weatherapi endpoint semantics to prefer latest-run behavior for current outlook endpoints.
- Remove or revise weatherapi filters that depend on historical `containsLocation=false` rows.
- Consider keeping `/outlooks/convective/location` as a compatibility alias for latest local outlooks if already exposed.
## Open Questions
None. This plan chooses the long-term maintainable options from `docs/roadmap/outlook.md`: schema v2, location-filtered canonical output, run-level discussions, destructive reset of the outlook table family, and permanent documentation of supersession semantics.

355
docs/roadmap/outlook.md
View File

@@ -1,355 +0,0 @@
# SPC Outlook Location Filtering And Discussion Refactor
## Summary
Revise SPC convective outlook normalization so `weatherfeeder` emits only outlook data relevant to the configured forecast location, and move SPC discussion text from each outlook polygon to day-level run discussions.
This is a canonical contract change. The current `weather.outlook.v1` contract preserves all SPC polygons and duplicates day discussion text on each polygon. The target contract should use a new canonical schema, `weather.outlook.v2`, so downstream consumers can distinguish the released all-polygons shape from the location-filtered shape.
## Locked Decisions
- `weatherfeeder` is a location-focused weather daemon, not a national weather archive.
- SPC sources should continue fetching complete upstream bundles for correctness and effective-time calculation.
- SPC raw events should remain complete and unfiltered.
- SPC normalizers should own location relevance policy.
- Canonical outlook runs should include only polygons that contain the configured forecast point.
- A canonical run should still be emitted when no polygons contain the configured point.
- When no polygons apply, the canonical run should contain `outlooks: []` and `discussions: []`.
- SPC discussion text should be modeled once per applicable outlook day, not duplicated on each polygon.
- If only Day 2 polygons apply, include only the Day 2 discussion.
- If multiple outlook types apply for the same day, include that day discussion once.
- Keep point-in-polygon boundary semantics unchanged: points on a polygon boundary count as contained.
- Keep empty SPC `GeometryCollection` no-risk placeholders skipped as non-polygons.
## Target Public Contract
### Event Kind And Schemas
- Event kind remains `outlook`.
- Raw schema remains `raw.spc.convective_outlook.v1`.
- Add canonical schema `weather.outlook.v2`.
- Stop producing new `weather.outlook.v1` events from the SPC normalizer after the transition.
- Keep `weather.outlook.v1` documentation as historical behavior if needed for consumers of already-stored events.
### Canonical Model
Update `model.WeatherOutlookRun` to include day-level discussions:
```go
type WeatherOutlookRun struct {
LocationID string `json:"locationId,omitempty"`
LocationName string `json:"locationName,omitempty"`
Latitude *float64 `json:"latitude,omitempty"`
Longitude *float64 `json:"longitude,omitempty"`
AsOf time.Time `json:"asOf"`
IssuedAt *time.Time `json:"issuedAt,omitempty"`
Outlooks []WeatherOutlook `json:"outlooks"`
Discussions []WeatherOutlookDiscussion `json:"discussions"`
}
```
Add `model.WeatherOutlookDiscussion`:
```go
type WeatherOutlookDiscussion struct {
Day int `json:"day"`
Headline string `json:"headline,omitempty"`
Summary string `json:"summary,omitempty"`
Discussion string `json:"discussion,omitempty"`
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
}
```
Update `model.WeatherOutlook`:
- Remove or stop populating polygon-level `headline`.
- Remove or stop populating polygon-level `summary`.
- Remove or stop populating polygon-level `discussion`.
- Keep `containsLocation` for compatibility and invariant visibility, but every emitted canonical outlook should have `containsLocation: true`.
Recommended implementation choice: remove the three polygon-level prose fields from the Go model and document them as `weather.outlook.v1` fields only. This is cleaner and consistent with the schema bump.
## Normalization Behavior
The SPC normalizer should:
1. Decode the complete raw SPC bundle.
2. Parse all required print-page discussions so missing or malformed required pages still fail normalization.
3. Decode all configured GeoJSON products.
4. Skip empty `GeometryCollection` no-risk placeholders.
5. Compute point-in-polygon for every real `Polygon` or `MultiPolygon` feature.
6. Append only outlooks where the configured forecast point is inside or on the boundary of the polygon.
7. Track latest valid GeoJSON `ISSUE_ISO` across all decoded features, including skipped empty placeholders, so `asOf` remains a snapshot of the checked upstream products.
8. Build `discussions` from the set of days represented by retained outlooks.
9. Preserve product ordering by day, then categorical, tornado, hail, and wind, and preserve feature order within each product.
10. Emit a canonical run even when no outlooks apply.
`asOf` and normalized event `effective_at` should continue using:
1. latest valid GeoJSON issue time across the complete bundle;
2. latest print-page update time;
3. incoming event `effective_at`;
4. incoming event `emitted_at`.
`issuedAt` on the run should continue representing the latest valid GeoJSON issue time when one exists, even if no polygons apply locally.
## Supersession Semantics
The permanent event and integration documentation must define SPC outlook
supersession semantics when `weather.outlook.v2` is implemented.
Required contract language:
- A `WeatherOutlookRun` is a point-in-time provider snapshot for the configured
forecast location as of `asOf`.
- Consumers that need current local outlooks should prefer latest-run semantics:
read the latest run and use that run's `outlooks` and `discussions` rather
than accumulating active outlook rows from older runs.
- `WeatherOutlook.issuedAt` is the authoritative version timestamp for an
outlook polygon.
- If a consumer intentionally queries across historical rows, later outlooks
supersede earlier outlooks with the same `provider`, `product`,
`outlookType`, `validFrom`, and `validTo`.
- `day` is an issuance-relative classification, not a stable supersession key.
A Day 3 outlook can become the Day 2 outlook and then the Day 1 outlook for
the same valid period.
- `label` is outlook content, not identity. A newer outlook with the same valid
period and type can legitimately change risk label.
- When multiple polygons from the same latest issuance share the same
supersession key, consumers must preserve all polygons from that latest
`issuedAt` group rather than collapsing to one row.
This should be documented in the permanent non-roadmap docs as part of the v2
implementation, especially:
- `docs/integrations/events.md`
- `docs/integrations/spc.md`
- `docs/integrations/postgres.md`
- `docs/consumers/api.md`
The roadmap should not update those current-behavior docs before the behavior is
implemented.
## Postgres Contract Changes
The Postgres sink needs a schema transition because discussion text moves from `outlooks` rows to day-level rows.
### Target Tables
Keep `outlook_runs` and `outlooks`.
Add a new child table, recommended name `outlook_discussions`:
- `run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL`
- `discussion_index INTEGER NOT NULL`
- `as_of TIMESTAMPTZ NOT NULL`
- `day INTEGER NOT NULL`
- `headline TEXT NULL`
- `summary TEXT NULL`
- `discussion TEXT NULL`
- `updated_at TIMESTAMPTZ NULL`
Recommended keys and indexes:
- Primary key: `(run_event_id, discussion_index)`
- Index: `(day, as_of)`
- Optional uniqueness: `(run_event_id, day)` if mapper guarantees one discussion per day and tests cover it.
Update `outlook_runs`:
- Keep `outlook_count`.
- Add `discussion_count INTEGER NOT NULL`.
Update `outlooks`:
- Remove `headline`, `summary`, and `discussion` from the target v2 table contract.
- Keep `contains_location BOOLEAN NOT NULL`; for v2 records this should always be true.
Recommended implementation choice: do not carry legacy nullable prose columns forward. Existing SPC outlook history does not need to be retained, so the transition should drop and recreate the outlook tables rather than preserving v1-only columns.
## Database Transition Document
Create a dedicated transitional document during implementation:
- `docs/roadmap/outlook-schema-transition.md`
That document should include operator-facing SQL for existing deployments. Because feedkit's Postgres sink uses `CREATE TABLE IF NOT EXISTS`, existing databases need manual intervention before the new schema can be created.
Recommended transition commands:
```sql
DROP TABLE IF EXISTS outlook_discussions;
DROP TABLE IF EXISTS outlooks;
DROP TABLE IF EXISTS outlook_runs;
```
After those tables are dropped, deploy the updated `weatherfeeder` and let the Postgres sink create the v2 outlook tables from `internal/sinks/postgres/schema.go`.
The transition document must state clearly that these commands delete stored SPC outlook history. That loss is acceptable for this feature because `weatherfeeder` is a location-focused current weather daemon, not an archive.
## Documentation Policy Update
A small architecture-policy revision is warranted. Add an explicit domain-scope invariant to `docs/policy/architecture.md`:
- `weatherfeeder` is location-focused. Canonical weather events should represent weather data relevant to the configured forecast location or configured provider object. It is not intended to archive all provider data for all places.
Suggested placement:
- Add this under `Core Design Principles` as a new bullet.
- Mention SPC outlooks as an example only if the implementation has shipped; otherwise keep the statement general.
Also update current-behavior docs only when implementation changes land:
- `docs/integrations/events.md`
- `docs/integrations/spc.md`
- `docs/integrations/postgres.md`
- `docs/internal/postgres-sink.md`
- `docs/consumers/pkg-model.md`
- `docs/consumers/api.md`
- `docs/config.md` if source/operator behavior descriptions mention all polygons
- `docs/operations.md` and `docs/troubleshooting.md` only if operational behavior changes need explicit mention
## Implementation Stages
### Stage 1: Standards And Model Contract
- Add `standards.SchemaWeatherOutlookV2 = "weather.outlook.v2"`.
- Update the SPC normalizer to emit `weather.outlook.v2`.
- Add `model.WeatherOutlookDiscussion`.
- Add `Discussions []WeatherOutlookDiscussion` to `model.WeatherOutlookRun` with JSON field `discussions`.
- Remove polygon-level prose fields from `model.WeatherOutlook`, or stop populating them if a softer source-compatible model transition is preferred.
- Update model documentation tests and consumer docs references.
Tests:
- Model JSON shape includes `discussions`.
- Model JSON shape no longer emits polygon-level prose fields if removed.
- Standards docs tests include `weather.outlook.v2`.
### Stage 2: Normalizer Filtering And Discussion Mapping
- Keep raw bundle decode complete and unfiltered.
- Compute `containsLocation` for each real geometry.
- Append only `containsLocation == true` outlooks.
- Set retained outlooks' `ContainsLocation` to true.
- Build run `Discussions` by collecting unique days from retained outlooks.
- Preserve discussion ordering by day ascending.
- Do not include discussions for days without retained outlooks.
- Ensure empty applicable outlook results produce `outlooks: []` and `discussions: []`.
- Continue using all products and print pages for timestamp fallback and validation.
Tests:
- Outside-location fixture normalizes successfully with `outlooks: []` and `discussions: []`.
- Day 2-only applicable polygon includes exactly one Day 2 discussion.
- Multiple Day 1 applicable outlook types include one Day 1 discussion.
- Retained outlooks always have `containsLocation == true`.
- Empty `GeometryCollection` placeholders remain skipped and still contribute issue time.
- Existing malformed timestamp, malformed geometry, missing discussion, and product ordering tests remain meaningful.
### Stage 3: Postgres Sink V2 Mapping
- Update schema definition for `outlook_runs.discussion_count`.
- Add `outlook_discussions` schema definition.
- Map `WeatherOutlookRun.Discussions` to `outlook_discussions` rows.
- Stop writing polygon-level prose fields for v2 outlook rows.
- Preserve event envelope fields and UTC normalization.
- Preserve compact GeoJSON storage.
- Validate required discussion fields:
- `day` is required and must be 1, 2, or 3.
- At least one of `headline`, `summary`, or `discussion` should be present for a discussion row.
- `updatedAt` remains optional.
- Preserve required outlook validation for retained polygons.
Tests:
- Schema includes `discussion_count` and `outlook_discussions`.
- Mapper writes one discussion row per run discussion.
- Mapper writes `discussion_count == len(payload.discussions)`.
- Mapper writes empty runs with `outlook_count = 0` and `discussion_count = 0`.
- Mapper rejects invalid discussion day.
- Mapper preserves compact geometry and UTC timestamps.
- Existing mapper tests for outlook IDs, provider, geometry, and required fields still pass.
### Stage 4: Transition Documentation
- Add `docs/roadmap/outlook-schema-transition.md` with the SQL commands from this roadmap, adjusted to match the final implemented schema.
- Clearly state deployment order:
1. Stop `weatherfeeder`.
2. Drop existing outlook tables.
3. Deploy updated `weatherfeeder`.
4. Start `weatherfeeder` so it recreates the outlook tables.
5. Deploy updated downstream consumers such as `weatherapi`.
- State that existing v1 outlook rows are intentionally deleted by this transition.
- State that new v2 rows store discussion text in `outlook_discussions`.
- Do not include `ALTER TABLE ... DROP COLUMN`; the chosen path is to drop and recreate the outlook table family.
### Stage 5: Current-Behavior Documentation Updates
After implementation lands, update current-behavior docs to describe v2 behavior:
- `docs/policy/architecture.md`: add the location-focused domain boundary invariant.
- `docs/integrations/events.md`: document `weather.outlook.v2`, `discussions[]`, and location-filtered `outlooks[]`.
- `docs/integrations/spc.md`: update mapping notes to say raw bundles are complete but canonical outlooks are location-filtered.
- `docs/integrations/postgres.md`: document `outlook_discussions` and `discussion_count`.
- `docs/internal/postgres-sink.md`: update mapper contract.
- `docs/consumers/pkg-model.md` and `docs/consumers/api.md`: update model examples.
- Keep any `weather.outlook.v1` notes explicitly marked as legacy/historical if retained.
Tests:
- Documentation consistency tests include `weather.outlook.v2`.
- Postgres docs mention `outlook_discussions`.
- SPC docs no longer claim all polygons are emitted or discussion text is attached to every polygon.
### Stage 6: Downstream Consumer Coordination
- Update `weatherapi` after `weatherfeeder` has a released version containing `weather.outlook.v2`.
- Update weatherapi Postgres reads to load `outlook_discussions`.
- Update weatherapi response docs to expose run-level `discussions`.
- Remove or adapt weatherapi filters that assumed all polygons were persisted and `containsLocation=false` rows existed.
- `/outlooks/convective/location` may become redundant if weatherapi only consumes v2 rows; keep it as an alias/filter endpoint only if useful for API stability.
## Verification
Focused tests during implementation:
```sh
go test ./model ./standards
go test ./internal/providers/spc ./internal/normalizers/spc ./internal/sources/spc ./internal/geo
go test ./internal/sinks/postgres
```
Full verification:
```sh
go test ./...
```
Manual operational validation with a live SPC no-risk scenario should confirm:
- poll succeeds;
- normalization succeeds;
- a canonical outlook run is emitted;
- `outlooks` is empty when no polygon contains the configured point;
- `discussions` is empty when no local outlook applies;
- `asOf` still reflects the latest checked SPC product issue time.
## Risks And Mitigations
| Risk | Mitigation |
| --- | --- |
| Downstream consumers expect `weather.outlook.v1` shape. | Use `weather.outlook.v2` and coordinate downstream updates. |
| Existing databases contain v1 outlook tables. | Stop the daemon, drop the outlook table family, deploy the v2 binary, and let the sink recreate the tables. |
| Empty local runs look like missing data. | Preserve `asOf`, `issuedAt`, location metadata, and empty arrays to indicate a successful checked snapshot. |
| Removing polygon-level discussion fields breaks code that reads `model.WeatherOutlook`. | Schema bump and release notes; update weatherapi immediately after weatherfeeder release. |
| Filtering hides useful national context. | Raw events remain complete for debugging; canonical events intentionally remain location-scoped. |
## Non-Goals
- Do not change SPC source polling URLs in this roadmap.
- Do not fetch SPC image assets or shapefiles.
- Do not add Day 4-8 outlooks.
- Do not make weatherfeeder a national SPC outlook archive.
- Do not add a separate outlook-discussion event kind unless a future roadmap establishes independent consumer value.
- Do not move point-in-polygon filtering into the source layer.