weatherfeeder/docs/roadmap/implementation.md

Implement SPC Outlook V2 Location Filtering

Summary

Implement the roadmap in docs/roadmap/outlook.md as a behavior-changing canonical contract update for SPC convective outlooks.

The implementation must make weatherfeeder emit location-relevant SPC outlook snapshots instead of all SPC polygons, and move SPC print-page prose from each polygon to run-level day discussions.

This plan is decision-complete. Do not preserve the old weather.outlook.v1 output behavior in the SPC normalizer. Implement weather.outlook.v2, update the Postgres sink contract, add transition documentation, and update current-behavior docs after the code change lands.

Required Target Behavior

• Raw SPC source behavior remains complete and unfiltered.
• Event kind remains outlook.
• Raw schema remains raw.spc.convective_outlook.v1.
• Canonical SPC outlook output changes from weather.outlook.v1 to weather.outlook.v2.
• Canonical outlook runs include only polygons where the configured forecast location is inside or on the boundary.
• Canonical outlook runs are still emitted when no polygons apply locally.
• No local polygons means outlooks: [] and discussions: [].
• Day discussions are run-level entries in WeatherOutlookRun.discussions.
• Discussions are included only for days represented by at least one retained outlook.
• Multiple retained outlooks for the same day share one run-level discussion entry.
• Polygon-level headline, summary, and discussion fields are removed from model.WeatherOutlook.
• containsLocation remains on WeatherOutlook and must be true for every v2 emitted outlook.
• Empty SPC GeometryCollection no-risk placeholders remain skipped and still contribute issue time.
• Supersession semantics are documented in permanent docs: latest-run semantics are preferred; historical row supersession uses provider, product, outlookType, validFrom, validTo, and greatest issuedAt; day and label are not identity fields.

Guardrails

• Do not move point-in-polygon filtering into the source layer.
• Do not fetch additional SPC products, images, shapefiles, or Day 4-8 products.
• Do not make weatherfeeder an archive of national SPC polygons.
• Do not add a separate outlook-discussion event kind.
• Do not introduce feedkit changes.
• Do not use column-level compatibility migrations for the outlook table family. Existing SPC outlook history may be discarded, so the transition should drop and recreate outlook tables.
• Do not update non-roadmap current-behavior docs before the corresponding behavior is implemented.
• Keep implementation inside existing architecture boundaries: model, standards, SPC normalizer/provider helpers, and Postgres sink mapping.

Stage 1: Standards And Canonical Model

Code Changes

• Add standards.SchemaWeatherOutlookV2 = "weather.outlook.v2" in standards/schema.go.
• Keep standards.SchemaWeatherOutlookV1 for historical references and any existing tests that still validate documented constants.
• Update model/outlook.go:
  • add WeatherOutlookDiscussion;
  • add Discussions []WeatherOutlookDiscussion to WeatherOutlookRun with JSON tag json:"discussions";
  • remove Headline, Summary, and Discussion fields from WeatherOutlook.
• Keep all other WeatherOutlook fields unchanged, including ContainsLocation and Geometry.
• Update any code or tests that construct WeatherOutlook values to remove the deleted polygon-level prose fields.
• Do not change raw SPC provider structs in this stage.

Tests

• Update model documentation/consumer tests so WeatherOutlookRun, WeatherOutlook, and WeatherOutlookDiscussion are listed where applicable.
• Update standards documentation tests so weather.outlook.v2 is expected.
• Add or update a JSON-shape test to verify:
  • WeatherOutlookRun serializes discussions;
  • WeatherOutlook no longer serializes polygon-level headline, summary, or discussion.

Verification

go test ./model ./standards

Stage 2: SPC Normalizer V2 Output

Code Changes

• Update internal/normalizers/spc/convective_outlook.go so Normalize emits standards.SchemaWeatherOutlookV2.
• Keep matching raw input schema standards.SchemaRawSPCConvectiveOutlookV1.
• Keep parsing all required print-page discussions before mapping features.
• Keep decoding every configured GeoJSON product.
• Keep skipping empty GeometryCollection placeholders.
• Track latest GeoJSON issue time across all real features and empty placeholders, not only retained local polygons.
• For every real feature:
  • parse required timestamps and label as today;
  • compute containsLocation using existing geo.ContainsPoint;
  • if containsLocation == false, do not append a canonical outlook;
  • if containsLocation == true, append the canonical outlook with ContainsLocation: true.
• Do not attach headline, summary, or discussion to retained outlooks.
• After retained outlooks are built, build WeatherOutlookRun.Discussions:
  • collect unique days present in retained outlooks;
  • include one WeatherOutlookDiscussion per retained day;
  • sort discussions by day ascending;
  • map Day, Headline, Summary, Discussion, and UpdatedAt from parsed print-page data;
  • omit discussions for days with no retained outlooks.
• Preserve existing product ordering for retained outlooks: day, outlook type order, then feature order.
• Preserve asOf and normalized event effective_at policy:
  • latest valid GeoJSON issue time across the complete bundle;
  • latest print-page update time;
  • incoming event effective_at;
  • incoming event emitted_at.
• Preserve WeatherOutlookRun.IssuedAt as latest valid GeoJSON issue time when available, even when outlooks is empty.

Tests

Update internal/normalizers/spc/convective_outlook_test.go:

• Existing sample test should expect out.Schema == standards.SchemaWeatherOutlookV2.
• Existing in-location sample should expect only location-contained outlooks.
• Outside-location sample should normalize successfully with:
  • len(run.Outlooks) == 0;
  • len(run.Discussions) == 0;
  • non-zero run.AsOf from latest product issue time;
  • normalized event EffectiveAt == run.AsOf.
• Add a fixture/test case where only Day 2 contains the location and assert:
  • all retained outlooks have Day == 2;
  • run.Discussions length is 1;
  • discussion day is 2;
  • Day 1 and Day 3 discussions are absent.
• Add a fixture/test case where multiple Day 1 outlook types contain the location and assert:
  • multiple retained Day 1 outlooks are present;
  • exactly one Day 1 discussion is present.
• Assert every retained outlook has ContainsLocation == true.
• Keep the empty GeometryCollection regression test and update it for v2 shape.
• Keep malformed timestamp, malformed geometry, missing discussion, missing label, and product ordering tests.
• Replace old tests that expected all nine fixture polygons to be emitted with location-filtered expectations.
• Replace tests that asserted polygon-level discussion text with run-level discussion assertions.

Verification

go test ./internal/providers/spc ./internal/normalizers/spc ./internal/geo

Stage 3: Postgres Schema And Mapper

Schema Changes

Update internal/sinks/postgres/schema.go:

• Add table constant tableOutlookDiscussions = "outlook_discussions".
• Add discussion_count INTEGER NOT NULL to outlook_runs.
• Add outlook_discussions table:
  • run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL;
  • discussion_index INTEGER NOT NULL;
  • as_of TIMESTAMPTZ NOT NULL;
  • day INTEGER NOT NULL;
  • headline TEXT NULL;
  • summary TEXT NULL;
  • discussion TEXT NULL;
  • updated_at TIMESTAMPTZ NULL.
• Set primary key to (run_event_id, discussion_index).
• Set prune column to as_of.
• Add index idx_wf_outlook_discussions_day_as_of on (day, as_of).
• Add unique index idx_wf_outlook_discussions_run_day on (run_event_id, day) using fksinks.PostgresIndex{Unique: true}.
• Remove headline, summary, and discussion from the target outlooks schema. Do not carry these legacy nullable columns forward.

Mapper Changes

Update internal/sinks/postgres/map.go:

• Route standards.SchemaWeatherOutlookV2 to the outlook mapper.
• Remove standards.SchemaWeatherOutlookV1 support from the active outlook mapper unless there is a compile-time reason to keep a clearly documented legacy test. Existing v1 outlook rows are intentionally discarded during the table reset and the SPC normalizer must not produce v1 events.
• Decode payload into the updated model.WeatherOutlookRun.
• Write outlook_runs.discussion_count = len(run.Discussions).
• Write one outlook_discussions row for each run.Discussions[i].
• Use discussion_index as the array position.
• Use parent run.AsOf.UTC() for as_of.
• Normalize UpdatedAt to UTC when present.
• Validate discussion entries before writing:
  • day must be 1, 2, or 3;
  • at least one of headline, summary, or discussion must be non-empty;
  • duplicate discussion days in one run should fail before hitting the unique index.
• Continue validating outlook required fields:
  • id, provider, product, day, outlookType, label, validFrom, validTo, issuedAt, expiresAt, and geometry remain required.
• Add a v2-specific invariant validation: every persisted v2 outlook must have ContainsLocation == true. If the mapper continues to accept v1, apply this invariant only to v2.
• Continue compacting geometry JSON and normalizing all timestamps to UTC.

Tests

Update internal/sinks/postgres/schema_test.go:

• Assert outlook_runs includes discussion_count.
• Assert outlook_discussions exists with expected columns, primary key, prune column, day/as-of index, and unique run/day index.
• Assert outlooks does not include headline, summary, or discussion.

Update internal/sinks/postgres/map_test.go:

• Assert v2 outlook run writes:
  • one parent row;
  • one outlook row per retained outlook;
  • one discussion row per run discussion;
  • correct outlook_count and discussion_count.
• Assert empty local run writes parent row with outlook_count = 0 and discussion_count = 0 and no child rows.
• Assert discussion rows map day/headline/summary/discussion/updatedAt correctly.
• Assert duplicate discussion days fail with useful context.
• Assert invalid discussion day fails with useful context.
• Assert empty discussion content fails with useful context.
• Assert v2 outlook with ContainsLocation == false fails.
• Update existing tests that used polygon-level Headline, Summary, or Discussion.
• Keep compact geometry and UTC normalization tests.

Verification

go test ./internal/sinks/postgres

Stage 4: Normalizer Registry And Cross-Package Consistency

Code Changes

• Confirm SPC normalizer registration remains unchanged except for output schema.
• Update any package-level tests under internal/normalizers that assert canonical schema routing or supported schema names.
• Update any source/registry documentation consistency tests that expect canonical schema strings.
• Search for stale SchemaWeatherOutlookV1 usage:

rg "SchemaWeatherOutlookV1|weather\.outlook\.v1|headline|summary|discussion" model standards internal docs/consumers docs/integrations docs/internal

• Keep SchemaWeatherOutlookV1 only where intentionally retained for historical docs or standards compatibility. Do not route new SPC normalized events to v1.
• Ensure no SPC normalizer test, source test, or current v2 documentation path still claims all polygons are emitted.

Tests

go test ./internal/normalizers ./internal/normalizers/...
go test ./internal/sources ./cmd/weatherfeeder

Stage 5: Transition Documentation

Roadmap Transition Doc

Create docs/roadmap/outlook-schema-transition.md with operator transition guidance.

The document must include:

• Purpose: reset existing Postgres outlook tables from the weather.outlook.v1 storage shape to the v2-compatible storage shape.
• Scope: drops only the outlook table family and lets updated weatherfeeder recreate it. Other weather tables are not affected.
• Explicit warning: these commands delete stored SPC outlook history.
• Deployment order:
  1. Stop weatherfeeder.
  2. Drop existing outlook tables.
  3. Deploy updated weatherfeeder.
  4. Start weatherfeeder so the Postgres sink recreates the v2 tables.
  5. Deploy updated downstream consumers such as weatherapi.
• Transition SQL:

DROP TABLE IF EXISTS outlook_discussions;
DROP TABLE IF EXISTS outlooks;
DROP TABLE IF EXISTS outlook_runs;

• Verification SQL examples:

SELECT table_name
FROM information_schema.tables
WHERE table_name IN ('outlook_runs', 'outlooks', 'outlook_discussions')
ORDER BY table_name;

After the updated daemon has started and recreated tables:

SELECT column_name, is_nullable, data_type
FROM information_schema.columns
WHERE table_name = 'outlook_runs'
  AND column_name = 'discussion_count';

SELECT indexname
FROM pg_indexes
WHERE tablename = 'outlook_discussions'
ORDER BY indexname;

SELECT column_name
FROM information_schema.columns
WHERE table_name = 'outlooks'
  AND column_name IN ('headline', 'summary', 'discussion');

• State that the final verification query for legacy outlooks prose columns should return zero rows.
• State that v1 outlook rows are intentionally removed and downstream readers should be updated intentionally.

Tests

No code tests are required for the roadmap transition doc, but documentation consistency tests may need updates if they check table names or schema strings.

Stage 6: Permanent Documentation Updates

After the implementation is complete, update permanent current-behavior documentation. Do not leave these as roadmap-only notes.

Required Docs

Update docs/policy/architecture.md:

• Add a core design principle that weatherfeeder is location-focused and canonical weather events should represent weather data relevant to the configured forecast location or configured provider object; it is not intended to archive all provider data for all places.

Update docs/integrations/events.md:

• Document weather.outlook.v2.
• Document WeatherOutlookRun.discussions and WeatherOutlookDiscussion fields.
• Document that outlooks[] is location-filtered.
• Document containsLocation as an invariant that is true for emitted v2 outlooks.
• Document empty local runs with outlooks: [] and discussions: [].
• Document supersession semantics:
  • latest-run semantics preferred;
  • historical supersession key is provider, product, outlookType, validFrom, validTo;
  • greatest issuedAt wins;
  • day and label are not identity;
  • preserve all polygons from the selected latest issuedAt group.
• Keep any v1 notes clearly marked legacy/historical if retained.

Update docs/integrations/spc.md:

• State raw SPC bundles remain complete.
• State canonical output is location-filtered.
• State discussions are day-level and only included for retained outlook days.
• Remove or replace claims that all polygons are emitted or discussion text is attached to every polygon.
• Add supersession guidance or link to the canonical section in docs/integrations/events.md.

Update docs/integrations/postgres.md:

• Add outlook_runs.discussion_count.
• Add outlook_discussions table, columns, keys, indexes, prune column, and mapping source.
• Remove outlooks.headline, outlooks.summary, and outlooks.discussion from the current table contract.
• Document v2 reader reconstruction: read outlook_runs, outlooks, and outlook_discussions by run_event_id ordered by child indexes.

Update docs/internal/postgres-sink.md and internal/sinks/postgres/doc.go:

• Document the new table and mapper validation rules.

Update docs/consumers/pkg-model.md and docs/consumers/api.md:

• Add WeatherOutlookDiscussion.
• Update model examples for run-level discussions.
• Remove polygon-level prose from v2 consumer examples.

Update docs/config.md only if SPC prose still describes all-polygons behavior.

Update docs/operations.md or docs/troubleshooting.md only if operational behavior or recovery instructions need adjustment.

Tests

• Update docs consistency tests for weather.outlook.v2.
• Update docs consistency tests for outlook_discussions if such tests exist or are practical.
• Ensure no permanent doc still describes v1 all-polygons behavior as current behavior:

rg "All outlook polygons|all polygons|attached to every outlook|weather\.outlook\.v1" docs README.md

Stage 7: Full Verification And Release Readiness

Focused Tests

go test ./model ./standards
go test ./internal/providers/spc ./internal/normalizers/spc ./internal/normalizers ./internal/geo
go test ./internal/sinks/postgres
go test ./internal/sources ./cmd/weatherfeeder

Full Test Suite

go test ./...

Manual Checks

• Confirm cmd/weatherfeeder/config.yml still loads and source registry tests pass.
• Confirm no current-behavior docs outside docs/roadmap/ describe unimplemented behavior.
• Confirm docs/roadmap/outlook-schema-transition.md transition SQL and verification steps match the final table names and internal/sinks/postgres/schema.go.
• Confirm the SPC normalizer emits weather.outlook.v2, not weather.outlook.v1.
• Confirm a no-local-risk SPC bundle emits a canonical run with empty outlook and discussion arrays.
• Confirm a local-risk SPC bundle emits only containing polygons and only matching day discussions.
• Confirm retained v2 outlooks all have containsLocation: true.

Weatherapi Coordination Notes

This stage is not implemented in weatherfeeder, but the weatherfeeder release should call it out for downstream work.

• Release weatherfeeder with weather.outlook.v2 before updating weatherapi dependency.
• Update weatherapi Postgres reads to load outlook_discussions.
• Update weatherapi endpoint semantics to prefer latest-run behavior for current outlook endpoints.
• Remove or revise weatherapi filters that depend on historical containsLocation=false rows.
• Consider keeping /outlooks/convective/location as a compatibility alias for latest local outlooks if already exposed.

Open Questions

None. This plan chooses the long-term maintainable options from docs/roadmap/outlook.md: schema v2, location-filtered canonical output, run-level discussions, destructive reset of the outlook table family, and permanent documentation of supersession semantics.