weatherfeeder/docs/roadmap/outlook.md

SPC Outlook Location Filtering And Discussion Refactor

Summary

Revise SPC convective outlook normalization so weatherfeeder emits only outlook data relevant to the configured forecast location, and move SPC discussion text from each outlook polygon to day-level run discussions.

This is a canonical contract change. The current weather.outlook.v1 contract preserves all SPC polygons and duplicates day discussion text on each polygon. The target contract should use a new canonical schema, weather.outlook.v2, so downstream consumers can distinguish the released all-polygons shape from the location-filtered shape.

Locked Decisions

• weatherfeeder is a location-focused weather daemon, not a national weather archive.
• SPC sources should continue fetching complete upstream bundles for correctness and effective-time calculation.
• SPC raw events should remain complete and unfiltered.
• SPC normalizers should own location relevance policy.
• Canonical outlook runs should include only polygons that contain the configured forecast point.
• A canonical run should still be emitted when no polygons contain the configured point.
• When no polygons apply, the canonical run should contain outlooks: [] and discussions: [].
• SPC discussion text should be modeled once per applicable outlook day, not duplicated on each polygon.
• If only Day 2 polygons apply, include only the Day 2 discussion.
• If multiple outlook types apply for the same day, include that day discussion once.
• Keep point-in-polygon boundary semantics unchanged: points on a polygon boundary count as contained.
• Keep empty SPC GeometryCollection no-risk placeholders skipped as non-polygons.

Target Public Contract

Event Kind And Schemas

• Event kind remains outlook.
• Raw schema remains raw.spc.convective_outlook.v1.
• Add canonical schema weather.outlook.v2.
• Stop producing new weather.outlook.v1 events from the SPC normalizer after the transition.
• Keep weather.outlook.v1 documentation as historical behavior if needed for consumers of already-stored events.

Canonical Model

Update model.WeatherOutlookRun to include day-level discussions:

type WeatherOutlookRun struct {
    LocationID   string                     `json:"locationId,omitempty"`
    LocationName string                     `json:"locationName,omitempty"`
    Latitude     *float64                   `json:"latitude,omitempty"`
    Longitude    *float64                   `json:"longitude,omitempty"`
    AsOf         time.Time                  `json:"asOf"`
    IssuedAt     *time.Time                 `json:"issuedAt,omitempty"`
    Outlooks     []WeatherOutlook           `json:"outlooks"`
    Discussions  []WeatherOutlookDiscussion `json:"discussions"`
}

Add model.WeatherOutlookDiscussion:

type WeatherOutlookDiscussion struct {
    Day        int        `json:"day"`
    Headline   string     `json:"headline,omitempty"`
    Summary    string     `json:"summary,omitempty"`
    Discussion string     `json:"discussion,omitempty"`
    UpdatedAt  *time.Time `json:"updatedAt,omitempty"`
}

Update model.WeatherOutlook:

• Remove or stop populating polygon-level headline.
• Remove or stop populating polygon-level summary.
• Remove or stop populating polygon-level discussion.
• Keep containsLocation for compatibility and invariant visibility, but every emitted canonical outlook should have containsLocation: true.

Recommended implementation choice: remove the three polygon-level prose fields from the Go model and document them as weather.outlook.v1 fields only. This is cleaner and consistent with the schema bump.

Normalization Behavior

The SPC normalizer should:

1. Decode the complete raw SPC bundle.
2. Parse all required print-page discussions so missing or malformed required pages still fail normalization.
3. Decode all configured GeoJSON products.
4. Skip empty GeometryCollection no-risk placeholders.
5. Compute point-in-polygon for every real Polygon or MultiPolygon feature.
6. Append only outlooks where the configured forecast point is inside or on the boundary of the polygon.
7. Track latest valid GeoJSON ISSUE_ISO across all decoded features, including skipped empty placeholders, so asOf remains a snapshot of the checked upstream products.
8. Build discussions from the set of days represented by retained outlooks.
9. Preserve product ordering by day, then categorical, tornado, hail, and wind, and preserve feature order within each product.
10. Emit a canonical run even when no outlooks apply.

asOf and normalized event effective_at should continue using:

1. latest valid GeoJSON issue time across the complete bundle;
2. latest print-page update time;
3. incoming event effective_at;
4. incoming event emitted_at.

issuedAt on the run should continue representing the latest valid GeoJSON issue time when one exists, even if no polygons apply locally.

Supersession Semantics

The permanent event and integration documentation must define SPC outlook supersession semantics when weather.outlook.v2 is implemented.

Required contract language:

• A WeatherOutlookRun is a point-in-time provider snapshot for the configured forecast location as of asOf.
• Consumers that need current local outlooks should prefer latest-run semantics: read the latest run and use that run's outlooks and discussions rather than accumulating active outlook rows from older runs.
• WeatherOutlook.issuedAt is the authoritative version timestamp for an outlook polygon.
• If a consumer intentionally queries across historical rows, later outlooks supersede earlier outlooks with the same provider, product, outlookType, validFrom, and validTo.
• day is an issuance-relative classification, not a stable supersession key. A Day 3 outlook can become the Day 2 outlook and then the Day 1 outlook for the same valid period.
• label is outlook content, not identity. A newer outlook with the same valid period and type can legitimately change risk label.
• When multiple polygons from the same latest issuance share the same supersession key, consumers must preserve all polygons from that latest issuedAt group rather than collapsing to one row.

This should be documented in the permanent non-roadmap docs as part of the v2 implementation, especially:

• docs/integrations/events.md
• docs/integrations/spc.md
• docs/integrations/postgres.md
• docs/consumers/api.md

The roadmap should not update those current-behavior docs before the behavior is implemented.

Postgres Contract Changes

The Postgres sink needs a schema transition because discussion text moves from outlooks rows to day-level rows.

Target Tables

Keep outlook_runs and outlooks.

Add a new child table, recommended name outlook_discussions:

• run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL
• discussion_index INTEGER NOT NULL
• as_of TIMESTAMPTZ NOT NULL
• day INTEGER NOT NULL
• headline TEXT NULL
• summary TEXT NULL
• discussion TEXT NULL
• updated_at TIMESTAMPTZ NULL

Recommended keys and indexes:

• Primary key: (run_event_id, discussion_index)
• Index: (day, as_of)
• Optional uniqueness: (run_event_id, day) if mapper guarantees one discussion per day and tests cover it.

Update outlook_runs:

• Keep outlook_count.
• Add discussion_count INTEGER NOT NULL.

Update outlooks:

• Remove headline, summary, and discussion from the target v2 table contract.
• Keep contains_location BOOLEAN NOT NULL; for v2 records this should always be true.

Recommended implementation choice: do not carry legacy nullable prose columns forward. Existing SPC outlook history does not need to be retained, so the transition should drop and recreate the outlook tables rather than preserving v1-only columns.

Database Transition Document

Create a dedicated transitional document during implementation:

• docs/roadmap/outlook-schema-transition.md

That document should include operator-facing SQL for existing deployments. Because feedkit's Postgres sink uses CREATE TABLE IF NOT EXISTS, existing databases need manual intervention before the new schema can be created.

Recommended transition commands:

DROP TABLE IF EXISTS outlook_discussions;
DROP TABLE IF EXISTS outlooks;
DROP TABLE IF EXISTS outlook_runs;

After those tables are dropped, deploy the updated weatherfeeder and let the Postgres sink create the v2 outlook tables from internal/sinks/postgres/schema.go.

The transition document must state clearly that these commands delete stored SPC outlook history. That loss is acceptable for this feature because weatherfeeder is a location-focused current weather daemon, not an archive.

Documentation Policy Update

A small architecture-policy revision is warranted. Add an explicit domain-scope invariant to docs/policy/architecture.md:

• weatherfeeder is location-focused. Canonical weather events should represent weather data relevant to the configured forecast location or configured provider object. It is not intended to archive all provider data for all places.

Suggested placement:

• Add this under Core Design Principles as a new bullet.
• Mention SPC outlooks as an example only if the implementation has shipped; otherwise keep the statement general.

Also update current-behavior docs only when implementation changes land:

• docs/integrations/events.md
• docs/integrations/spc.md
• docs/integrations/postgres.md
• docs/internal/postgres-sink.md
• docs/consumers/pkg-model.md
• docs/consumers/api.md
• docs/config.md if source/operator behavior descriptions mention all polygons
• docs/operations.md and docs/troubleshooting.md only if operational behavior changes need explicit mention

Implementation Stages

Stage 1: Standards And Model Contract

• Add standards.SchemaWeatherOutlookV2 = "weather.outlook.v2".
• Update the SPC normalizer to emit weather.outlook.v2.
• Add model.WeatherOutlookDiscussion.
• Add Discussions []WeatherOutlookDiscussion to model.WeatherOutlookRun with JSON field discussions.
• Remove polygon-level prose fields from model.WeatherOutlook, or stop populating them if a softer source-compatible model transition is preferred.
• Update model documentation tests and consumer docs references.

Tests:

• Model JSON shape includes discussions.
• Model JSON shape no longer emits polygon-level prose fields if removed.
• Standards docs tests include weather.outlook.v2.

Stage 2: Normalizer Filtering And Discussion Mapping

• Keep raw bundle decode complete and unfiltered.
• Compute containsLocation for each real geometry.
• Append only containsLocation == true outlooks.
• Set retained outlooks' ContainsLocation to true.
• Build run Discussions by collecting unique days from retained outlooks.
• Preserve discussion ordering by day ascending.
• Do not include discussions for days without retained outlooks.
• Ensure empty applicable outlook results produce outlooks: [] and discussions: [].
• Continue using all products and print pages for timestamp fallback and validation.

Tests:

• Outside-location fixture normalizes successfully with outlooks: [] and discussions: [].
• Day 2-only applicable polygon includes exactly one Day 2 discussion.
• Multiple Day 1 applicable outlook types include one Day 1 discussion.
• Retained outlooks always have containsLocation == true.
• Empty GeometryCollection placeholders remain skipped and still contribute issue time.
• Existing malformed timestamp, malformed geometry, missing discussion, and product ordering tests remain meaningful.

Stage 3: Postgres Sink V2 Mapping

• Update schema definition for outlook_runs.discussion_count.
• Add outlook_discussions schema definition.
• Map WeatherOutlookRun.Discussions to outlook_discussions rows.
• Stop writing polygon-level prose fields for v2 outlook rows.
• Preserve event envelope fields and UTC normalization.
• Preserve compact GeoJSON storage.
• Validate required discussion fields:
  • day is required and must be 1, 2, or 3.
  • At least one of headline, summary, or discussion should be present for a discussion row.
  • updatedAt remains optional.
• Preserve required outlook validation for retained polygons.

Tests:

• Schema includes discussion_count and outlook_discussions.
• Mapper writes one discussion row per run discussion.
• Mapper writes discussion_count == len(payload.discussions).
• Mapper writes empty runs with outlook_count = 0 and discussion_count = 0.
• Mapper rejects invalid discussion day.
• Mapper preserves compact geometry and UTC timestamps.
• Existing mapper tests for outlook IDs, provider, geometry, and required fields still pass.

Stage 4: Transition Documentation

• Add docs/roadmap/outlook-schema-transition.md with the SQL commands from this roadmap, adjusted to match the final implemented schema.
• Clearly state deployment order:
  1. Stop weatherfeeder.
  2. Drop existing outlook tables.
  3. Deploy updated weatherfeeder.
  4. Start weatherfeeder so it recreates the outlook tables.
  5. Deploy updated downstream consumers such as weatherapi.
• State that existing v1 outlook rows are intentionally deleted by this transition.
• State that new v2 rows store discussion text in outlook_discussions.
• Do not include ALTER TABLE ... DROP COLUMN; the chosen path is to drop and recreate the outlook table family.

Stage 5: Current-Behavior Documentation Updates

After implementation lands, update current-behavior docs to describe v2 behavior:

• docs/policy/architecture.md: add the location-focused domain boundary invariant.
• docs/integrations/events.md: document weather.outlook.v2, discussions[], and location-filtered outlooks[].
• docs/integrations/spc.md: update mapping notes to say raw bundles are complete but canonical outlooks are location-filtered.
• docs/integrations/postgres.md: document outlook_discussions and discussion_count.
• docs/internal/postgres-sink.md: update mapper contract.
• docs/consumers/pkg-model.md and docs/consumers/api.md: update model examples.
• Keep any weather.outlook.v1 notes explicitly marked as legacy/historical if retained.

Tests:

• Documentation consistency tests include weather.outlook.v2.
• Postgres docs mention outlook_discussions.
• SPC docs no longer claim all polygons are emitted or discussion text is attached to every polygon.

Stage 6: Downstream Consumer Coordination

• Update weatherapi after weatherfeeder has a released version containing weather.outlook.v2.
• Update weatherapi Postgres reads to load outlook_discussions.
• Update weatherapi response docs to expose run-level discussions.
• Remove or adapt weatherapi filters that assumed all polygons were persisted and containsLocation=false rows existed.
• /outlooks/convective/location may become redundant if weatherapi only consumes v2 rows; keep it as an alias/filter endpoint only if useful for API stability.

Verification

Focused tests during implementation:

go test ./model ./standards
go test ./internal/providers/spc ./internal/normalizers/spc ./internal/sources/spc ./internal/geo
go test ./internal/sinks/postgres

Full verification:

go test ./...

Manual operational validation with a live SPC no-risk scenario should confirm:

• poll succeeds;
• normalization succeeds;
• a canonical outlook run is emitted;
• outlooks is empty when no polygon contains the configured point;
• discussions is empty when no local outlook applies;
• asOf still reflects the latest checked SPC product issue time.

Risks And Mitigations

Risk	Mitigation
Downstream consumers expect weather.outlook.v1 shape.	Use weather.outlook.v2 and coordinate downstream updates.
Existing databases contain v1 outlook tables.	Stop the daemon, drop the outlook table family, deploy the v2 binary, and let the sink recreate the tables.
Empty local runs look like missing data.	Preserve asOf, issuedAt, location metadata, and empty arrays to indicate a successful checked snapshot.
Removing polygon-level discussion fields breaks code that reads model.WeatherOutlook.	Schema bump and release notes; update weatherapi immediately after weatherfeeder release.
Filtering hides useful national context.	Raw events remain complete for debugging; canonical events intentionally remain location-scoped.

Non-Goals

• Do not change SPC source polling URLs in this roadmap.
• Do not fetch SPC image assets or shapefiles.
• Do not add Day 4-8 outlooks.
• Do not make weatherfeeder a national SPC outlook archive.
• Do not add a separate outlook-discussion event kind unless a future roadmap establishes independent consumer value.
• Do not move point-in-polygon filtering into the source layer.