# SPC Outlook Location Filtering And Discussion Refactor

## Summary

Revise SPC convective outlook normalization so `weatherfeeder` emits only outlook data relevant to the configured forecast location, and move SPC discussion text from each outlook polygon to day-level run discussions.

This is a canonical contract change. The current `weather.outlook.v1` contract preserves all SPC polygons and duplicates day discussion text on each polygon. The target contract should use a new canonical schema, `weather.outlook.v2`, so downstream consumers can distinguish the released all-polygons shape from the location-filtered shape.

## Locked Decisions

- `weatherfeeder` is a location-focused weather daemon, not a national weather archive.
- SPC sources should continue fetching complete upstream bundles for correctness and effective-time calculation.
- SPC raw events should remain complete and unfiltered.
- SPC normalizers should own location relevance policy.
- Canonical outlook runs should include only polygons that contain the configured forecast point.
- A canonical run should still be emitted when no polygons contain the configured point.
- When no polygons apply, the canonical run should contain `outlooks: []` and `discussions: []`.
- SPC discussion text should be modeled once per applicable outlook day, not duplicated on each polygon.
- If only Day 2 polygons apply, include only the Day 2 discussion.
- If multiple outlook types apply for the same day, include that day discussion once.
- Keep point-in-polygon boundary semantics unchanged: points on a polygon boundary count as contained.
- Keep empty SPC `GeometryCollection` no-risk placeholders skipped as non-polygons.

## Target Public Contract

### Event Kind And Schemas

- Event kind remains `outlook`.
- Raw schema remains `raw.spc.convective_outlook.v1`.
- Add canonical schema `weather.outlook.v2`.
- Stop producing new `weather.outlook.v1` events from the SPC normalizer after the transition.
- Keep `weather.outlook.v1` documentation as historical behavior if needed for consumers of already-stored events.

### Canonical Model

Update `model.WeatherOutlookRun` to include day-level discussions:

```go
type WeatherOutlookRun struct {
    LocationID   string                     `json:"locationId,omitempty"`
    LocationName string                     `json:"locationName,omitempty"`
    Latitude     *float64                   `json:"latitude,omitempty"`
    Longitude    *float64                   `json:"longitude,omitempty"`
    AsOf         time.Time                  `json:"asOf"`
    IssuedAt     *time.Time                 `json:"issuedAt,omitempty"`
    Outlooks     []WeatherOutlook           `json:"outlooks"`
    Discussions  []WeatherOutlookDiscussion `json:"discussions"`
}
```

Add `model.WeatherOutlookDiscussion`:

```go
type WeatherOutlookDiscussion struct {
    Day        int        `json:"day"`
    Headline   string     `json:"headline,omitempty"`
    Summary    string     `json:"summary,omitempty"`
    Discussion string     `json:"discussion,omitempty"`
    UpdatedAt  *time.Time `json:"updatedAt,omitempty"`
}
```

Update `model.WeatherOutlook`:

- Remove or stop populating polygon-level `headline`.
- Remove or stop populating polygon-level `summary`.
- Remove or stop populating polygon-level `discussion`.
- Keep `containsLocation` for compatibility and invariant visibility, but every emitted canonical outlook should have `containsLocation: true`.

Recommended implementation choice: remove the three polygon-level prose fields from the Go model and document them as `weather.outlook.v1` fields only. This is cleaner and consistent with the schema bump.

## Normalization Behavior

The SPC normalizer should:

1. Decode the complete raw SPC bundle.
2. Parse all required print-page discussions so missing or malformed required pages still fail normalization.
3. Decode all configured GeoJSON products.
4. Skip empty `GeometryCollection` no-risk placeholders.
5. Compute point-in-polygon for every real `Polygon` or `MultiPolygon` feature.
6. Append only outlooks where the configured forecast point is inside or on the boundary of the polygon.
7. Track latest valid GeoJSON `ISSUE_ISO` across all decoded features, including skipped empty placeholders, so `asOf` remains a snapshot of the checked upstream products.
8. Build `discussions` from the set of days represented by retained outlooks.
9. Preserve product ordering by day, then categorical, tornado, hail, and wind, and preserve feature order within each product.
10. Emit a canonical run even when no outlooks apply.

`asOf` and normalized event `effective_at` should continue using:

1. latest valid GeoJSON issue time across the complete bundle;
2. latest print-page update time;
3. incoming event `effective_at`;
4. incoming event `emitted_at`.

`issuedAt` on the run should continue representing the latest valid GeoJSON issue time when one exists, even if no polygons apply locally.

## Supersession Semantics

The permanent event and integration documentation must define SPC outlook
supersession semantics when `weather.outlook.v2` is implemented.

Required contract language:

- A `WeatherOutlookRun` is a point-in-time provider snapshot for the configured
  forecast location as of `asOf`.
- Consumers that need current local outlooks should prefer latest-run semantics:
  read the latest run and use that run's `outlooks` and `discussions` rather
  than accumulating active outlook rows from older runs.
- `WeatherOutlook.issuedAt` is the authoritative version timestamp for an
  outlook polygon.
- If a consumer intentionally queries across historical rows, later outlooks
  supersede earlier outlooks with the same `provider`, `product`,
  `outlookType`, `validFrom`, and `validTo`.
- `day` is an issuance-relative classification, not a stable supersession key.
  A Day 3 outlook can become the Day 2 outlook and then the Day 1 outlook for
  the same valid period.
- `label` is outlook content, not identity. A newer outlook with the same valid
  period and type can legitimately change risk label.
- When multiple polygons from the same latest issuance share the same
  supersession key, consumers must preserve all polygons from that latest
  `issuedAt` group rather than collapsing to one row.

This should be documented in the permanent non-roadmap docs as part of the v2
implementation, especially:

- `docs/integrations/events.md`
- `docs/integrations/spc.md`
- `docs/integrations/postgres.md`
- `docs/consumers/api.md`

The roadmap should not update those current-behavior docs before the behavior is
implemented.

## Postgres Contract Changes

The Postgres sink needs a schema transition because discussion text moves from `outlooks` rows to day-level rows.

### Target Tables

Keep `outlook_runs` and `outlooks`.

Add a new child table, recommended name `outlook_discussions`:

- `run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL`
- `discussion_index INTEGER NOT NULL`
- `as_of TIMESTAMPTZ NOT NULL`
- `day INTEGER NOT NULL`
- `headline TEXT NULL`
- `summary TEXT NULL`
- `discussion TEXT NULL`
- `updated_at TIMESTAMPTZ NULL`

Recommended keys and indexes:

- Primary key: `(run_event_id, discussion_index)`
- Index: `(day, as_of)`
- Optional uniqueness: `(run_event_id, day)` if mapper guarantees one discussion per day and tests cover it.

Update `outlook_runs`:

- Keep `outlook_count`.
- Add `discussion_count INTEGER NOT NULL`.

Update `outlooks`:

- Remove `headline`, `summary`, and `discussion` from the target v2 table contract.
- Keep `contains_location BOOLEAN NOT NULL`; for v2 records this should always be true.

Recommended implementation choice: do not carry legacy nullable prose columns forward. Existing SPC outlook history does not need to be retained, so the transition should drop and recreate the outlook tables rather than preserving v1-only columns.

## Database Transition Document

Create a dedicated transitional document during implementation:

- `docs/roadmap/outlook-schema-transition.md`

That document should include operator-facing SQL for existing deployments. Because feedkit's Postgres sink uses `CREATE TABLE IF NOT EXISTS`, existing databases need manual intervention before the new schema can be created.

Recommended transition commands:

```sql
DROP TABLE IF EXISTS outlook_discussions;
DROP TABLE IF EXISTS outlooks;
DROP TABLE IF EXISTS outlook_runs;
```

After those tables are dropped, deploy the updated `weatherfeeder` and let the Postgres sink create the v2 outlook tables from `internal/sinks/postgres/schema.go`.

The transition document must state clearly that these commands delete stored SPC outlook history. That loss is acceptable for this feature because `weatherfeeder` is a location-focused current weather daemon, not an archive.

## Documentation Policy Update

A small architecture-policy revision is warranted. Add an explicit domain-scope invariant to `docs/policy/architecture.md`:

- `weatherfeeder` is location-focused. Canonical weather events should represent weather data relevant to the configured forecast location or configured provider object. It is not intended to archive all provider data for all places.

Suggested placement:

- Add this under `Core Design Principles` as a new bullet.
- Mention SPC outlooks as an example only if the implementation has shipped; otherwise keep the statement general.

Also update current-behavior docs only when implementation changes land:

- `docs/integrations/events.md`
- `docs/integrations/spc.md`
- `docs/integrations/postgres.md`
- `docs/internal/postgres-sink.md`
- `docs/consumers/pkg-model.md`
- `docs/consumers/api.md`
- `docs/config.md` if source/operator behavior descriptions mention all polygons
- `docs/operations.md` and `docs/troubleshooting.md` only if operational behavior changes need explicit mention

## Implementation Stages

### Stage 1: Standards And Model Contract

- Add `standards.SchemaWeatherOutlookV2 = "weather.outlook.v2"`.
- Update the SPC normalizer to emit `weather.outlook.v2`.
- Add `model.WeatherOutlookDiscussion`.
- Add `Discussions []WeatherOutlookDiscussion` to `model.WeatherOutlookRun` with JSON field `discussions`.
- Remove polygon-level prose fields from `model.WeatherOutlook`, or stop populating them if a softer source-compatible model transition is preferred.
- Update model documentation tests and consumer docs references.

Tests:

- Model JSON shape includes `discussions`.
- Model JSON shape no longer emits polygon-level prose fields if removed.
- Standards docs tests include `weather.outlook.v2`.

### Stage 2: Normalizer Filtering And Discussion Mapping

- Keep raw bundle decode complete and unfiltered.
- Compute `containsLocation` for each real geometry.
- Append only `containsLocation == true` outlooks.
- Set retained outlooks' `ContainsLocation` to true.
- Build run `Discussions` by collecting unique days from retained outlooks.
- Preserve discussion ordering by day ascending.
- Do not include discussions for days without retained outlooks.
- Ensure empty applicable outlook results produce `outlooks: []` and `discussions: []`.
- Continue using all products and print pages for timestamp fallback and validation.

Tests:

- Outside-location fixture normalizes successfully with `outlooks: []` and `discussions: []`.
- Day 2-only applicable polygon includes exactly one Day 2 discussion.
- Multiple Day 1 applicable outlook types include one Day 1 discussion.
- Retained outlooks always have `containsLocation == true`.
- Empty `GeometryCollection` placeholders remain skipped and still contribute issue time.
- Existing malformed timestamp, malformed geometry, missing discussion, and product ordering tests remain meaningful.

### Stage 3: Postgres Sink V2 Mapping

- Update schema definition for `outlook_runs.discussion_count`.
- Add `outlook_discussions` schema definition.
- Map `WeatherOutlookRun.Discussions` to `outlook_discussions` rows.
- Stop writing polygon-level prose fields for v2 outlook rows.
- Preserve event envelope fields and UTC normalization.
- Preserve compact GeoJSON storage.
- Validate required discussion fields:
  - `day` is required and must be 1, 2, or 3.
  - At least one of `headline`, `summary`, or `discussion` should be present for a discussion row.
  - `updatedAt` remains optional.
- Preserve required outlook validation for retained polygons.

Tests:

- Schema includes `discussion_count` and `outlook_discussions`.
- Mapper writes one discussion row per run discussion.
- Mapper writes `discussion_count == len(payload.discussions)`.
- Mapper writes empty runs with `outlook_count = 0` and `discussion_count = 0`.
- Mapper rejects invalid discussion day.
- Mapper preserves compact geometry and UTC timestamps.
- Existing mapper tests for outlook IDs, provider, geometry, and required fields still pass.

### Stage 4: Transition Documentation

- Add `docs/roadmap/outlook-schema-transition.md` with the SQL commands from this roadmap, adjusted to match the final implemented schema.
- Clearly state deployment order:
  1. Stop `weatherfeeder`.
  2. Drop existing outlook tables.
  3. Deploy updated `weatherfeeder`.
  4. Start `weatherfeeder` so it recreates the outlook tables.
  5. Deploy updated downstream consumers such as `weatherapi`.
- State that existing v1 outlook rows are intentionally deleted by this transition.
- State that new v2 rows store discussion text in `outlook_discussions`.
- Do not include `ALTER TABLE ... DROP COLUMN`; the chosen path is to drop and recreate the outlook table family.

### Stage 5: Current-Behavior Documentation Updates

After implementation lands, update current-behavior docs to describe v2 behavior:

- `docs/policy/architecture.md`: add the location-focused domain boundary invariant.
- `docs/integrations/events.md`: document `weather.outlook.v2`, `discussions[]`, and location-filtered `outlooks[]`.
- `docs/integrations/spc.md`: update mapping notes to say raw bundles are complete but canonical outlooks are location-filtered.
- `docs/integrations/postgres.md`: document `outlook_discussions` and `discussion_count`.
- `docs/internal/postgres-sink.md`: update mapper contract.
- `docs/consumers/pkg-model.md` and `docs/consumers/api.md`: update model examples.
- Keep any `weather.outlook.v1` notes explicitly marked as legacy/historical if retained.

Tests:

- Documentation consistency tests include `weather.outlook.v2`.
- Postgres docs mention `outlook_discussions`.
- SPC docs no longer claim all polygons are emitted or discussion text is attached to every polygon.

### Stage 6: Downstream Consumer Coordination

- Update `weatherapi` after `weatherfeeder` has a released version containing `weather.outlook.v2`.
- Update weatherapi Postgres reads to load `outlook_discussions`.
- Update weatherapi response docs to expose run-level `discussions`.
- Remove or adapt weatherapi filters that assumed all polygons were persisted and `containsLocation=false` rows existed.
- `/outlooks/convective/location` may become redundant if weatherapi only consumes v2 rows; keep it as an alias/filter endpoint only if useful for API stability.

## Verification

Focused tests during implementation:

```sh
go test ./model ./standards
go test ./internal/providers/spc ./internal/normalizers/spc ./internal/sources/spc ./internal/geo
go test ./internal/sinks/postgres
```

Full verification:

```sh
go test ./...
```

Manual operational validation with a live SPC no-risk scenario should confirm:

- poll succeeds;
- normalization succeeds;
- a canonical outlook run is emitted;
- `outlooks` is empty when no polygon contains the configured point;
- `discussions` is empty when no local outlook applies;
- `asOf` still reflects the latest checked SPC product issue time.

## Risks And Mitigations

| Risk | Mitigation |
| --- | --- |
| Downstream consumers expect `weather.outlook.v1` shape. | Use `weather.outlook.v2` and coordinate downstream updates. |
| Existing databases contain v1 outlook tables. | Stop the daemon, drop the outlook table family, deploy the v2 binary, and let the sink recreate the tables. |
| Empty local runs look like missing data. | Preserve `asOf`, `issuedAt`, location metadata, and empty arrays to indicate a successful checked snapshot. |
| Removing polygon-level discussion fields breaks code that reads `model.WeatherOutlook`. | Schema bump and release notes; update weatherapi immediately after weatherfeeder release. |
| Filtering hides useful national context. | Raw events remain complete for debugging; canonical events intentionally remain location-scoped. |

## Non-Goals

- Do not change SPC source polling URLs in this roadmap.
- Do not fetch SPC image assets or shapefiles.
- Do not add Day 4-8 outlooks.
- Do not make weatherfeeder a national SPC outlook archive.
- Do not add a separate outlook-discussion event kind unless a future roadmap establishes independent consumer value.
- Do not move point-in-polygon filtering into the source layer.