weatherfeeder/docs/roadmap/implementation.md

# Implement SPC Outlook V2 Location Filtering

## Summary

Implement the roadmap in `docs/roadmap/outlook.md` as a behavior-changing canonical contract update for SPC convective outlooks.

The implementation must make `weatherfeeder` emit location-relevant SPC outlook snapshots instead of all SPC polygons, and move SPC print-page prose from each polygon to run-level day discussions.

This plan is decision-complete. Do not preserve the old `weather.outlook.v1` output behavior in the SPC normalizer. Implement `weather.outlook.v2`, update the Postgres sink contract, add transition documentation, and update current-behavior docs after the code change lands.

## Required Target Behavior

- Raw SPC source behavior remains complete and unfiltered.
- Event kind remains `outlook`.
- Raw schema remains `raw.spc.convective_outlook.v1`.
- Canonical SPC outlook output changes from `weather.outlook.v1` to `weather.outlook.v2`.
- Canonical outlook runs include only polygons where the configured forecast location is inside or on the boundary.
- Canonical outlook runs are still emitted when no polygons apply locally.
- No local polygons means `outlooks: []` and `discussions: []`.
- Day discussions are run-level entries in `WeatherOutlookRun.discussions`.
- Discussions are included only for days represented by at least one retained outlook.
- Multiple retained outlooks for the same day share one run-level discussion entry.
- Polygon-level `headline`, `summary`, and `discussion` fields are removed from `model.WeatherOutlook`.
- `containsLocation` remains on `WeatherOutlook` and must be `true` for every v2 emitted outlook.
- Empty SPC `GeometryCollection` no-risk placeholders remain skipped and still contribute issue time.
- Supersession semantics are documented in permanent docs: latest-run semantics are preferred; historical row supersession uses `provider`, `product`, `outlookType`, `validFrom`, `validTo`, and greatest `issuedAt`; `day` and `label` are not identity fields.

## Guardrails

- Do not move point-in-polygon filtering into the source layer.
- Do not fetch additional SPC products, images, shapefiles, or Day 4-8 products.
- Do not make `weatherfeeder` an archive of national SPC polygons.
- Do not add a separate outlook-discussion event kind.
- Do not introduce feedkit changes.
- Do not use column-level compatibility migrations for the outlook table family. Existing SPC outlook history may be discarded, so the transition should drop and recreate outlook tables.
- Do not update non-roadmap current-behavior docs before the corresponding behavior is implemented.
- Keep implementation inside existing architecture boundaries: `model`, `standards`, SPC normalizer/provider helpers, and Postgres sink mapping.

## Stage 1: Standards And Canonical Model

### Code Changes

- Add `standards.SchemaWeatherOutlookV2 = "weather.outlook.v2"` in `standards/schema.go`.
- Keep `standards.SchemaWeatherOutlookV1` for historical references and any existing tests that still validate documented constants.
- Update `model/outlook.go`:
  - add `WeatherOutlookDiscussion`;
  - add `Discussions []WeatherOutlookDiscussion` to `WeatherOutlookRun` with JSON tag `json:"discussions"`;
  - remove `Headline`, `Summary`, and `Discussion` fields from `WeatherOutlook`.
- Keep all other `WeatherOutlook` fields unchanged, including `ContainsLocation` and `Geometry`.
- Update any code or tests that construct `WeatherOutlook` values to remove the deleted polygon-level prose fields.
- Do not change raw SPC provider structs in this stage.

### Tests

- Update model documentation/consumer tests so `WeatherOutlookRun`, `WeatherOutlook`, and `WeatherOutlookDiscussion` are listed where applicable.
- Update standards documentation tests so `weather.outlook.v2` is expected.
- Add or update a JSON-shape test to verify:
  - `WeatherOutlookRun` serializes `discussions`;
  - `WeatherOutlook` no longer serializes polygon-level `headline`, `summary`, or `discussion`.

### Verification

```sh
go test ./model ./standards
```

## Stage 2: SPC Normalizer V2 Output

### Code Changes

- Update `internal/normalizers/spc/convective_outlook.go` so `Normalize` emits `standards.SchemaWeatherOutlookV2`.
- Keep matching raw input schema `standards.SchemaRawSPCConvectiveOutlookV1`.
- Keep parsing all required print-page discussions before mapping features.
- Keep decoding every configured GeoJSON product.
- Keep skipping empty `GeometryCollection` placeholders.
- Track latest GeoJSON issue time across all real features and empty placeholders, not only retained local polygons.
- For every real feature:
  - parse required timestamps and label as today;
  - compute `containsLocation` using existing `geo.ContainsPoint`;
  - if `containsLocation == false`, do not append a canonical outlook;
  - if `containsLocation == true`, append the canonical outlook with `ContainsLocation: true`.
- Do not attach `headline`, `summary`, or `discussion` to retained outlooks.
- After retained outlooks are built, build `WeatherOutlookRun.Discussions`:
  - collect unique days present in retained outlooks;
  - include one `WeatherOutlookDiscussion` per retained day;
  - sort discussions by day ascending;
  - map `Day`, `Headline`, `Summary`, `Discussion`, and `UpdatedAt` from parsed print-page data;
  - omit discussions for days with no retained outlooks.
- Preserve existing product ordering for retained outlooks: day, outlook type order, then feature order.
- Preserve `asOf` and normalized event `effective_at` policy:
  - latest valid GeoJSON issue time across the complete bundle;
  - latest print-page update time;
  - incoming event `effective_at`;
  - incoming event `emitted_at`.
- Preserve `WeatherOutlookRun.IssuedAt` as latest valid GeoJSON issue time when available, even when `outlooks` is empty.

### Tests

Update `internal/normalizers/spc/convective_outlook_test.go`:

- Existing sample test should expect `out.Schema == standards.SchemaWeatherOutlookV2`.
- Existing in-location sample should expect only location-contained outlooks.
- Outside-location sample should normalize successfully with:
  - `len(run.Outlooks) == 0`;
  - `len(run.Discussions) == 0`;
  - non-zero `run.AsOf` from latest product issue time;
  - normalized event `EffectiveAt == run.AsOf`.
- Add a fixture/test case where only Day 2 contains the location and assert:
  - all retained outlooks have `Day == 2`;
  - `run.Discussions` length is 1;
  - discussion day is 2;
  - Day 1 and Day 3 discussions are absent.
- Add a fixture/test case where multiple Day 1 outlook types contain the location and assert:
  - multiple retained Day 1 outlooks are present;
  - exactly one Day 1 discussion is present.
- Assert every retained outlook has `ContainsLocation == true`.
- Keep the empty `GeometryCollection` regression test and update it for v2 shape.
- Keep malformed timestamp, malformed geometry, missing discussion, missing label, and product ordering tests.
- Replace old tests that expected all nine fixture polygons to be emitted with location-filtered expectations.
- Replace tests that asserted polygon-level discussion text with run-level discussion assertions.

### Verification

```sh
go test ./internal/providers/spc ./internal/normalizers/spc ./internal/geo
```

## Stage 3: Postgres Schema And Mapper

### Schema Changes

Update `internal/sinks/postgres/schema.go`:

- Add table constant `tableOutlookDiscussions = "outlook_discussions"`.
- Add `discussion_count INTEGER NOT NULL` to `outlook_runs`.
- Add `outlook_discussions` table:
  - `run_event_id TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE NOT NULL`;
  - `discussion_index INTEGER NOT NULL`;
  - `as_of TIMESTAMPTZ NOT NULL`;
  - `day INTEGER NOT NULL`;
  - `headline TEXT NULL`;
  - `summary TEXT NULL`;
  - `discussion TEXT NULL`;
  - `updated_at TIMESTAMPTZ NULL`.
- Set primary key to `(run_event_id, discussion_index)`.
- Set prune column to `as_of`.
- Add index `idx_wf_outlook_discussions_day_as_of` on `(day, as_of)`.
- Add unique index `idx_wf_outlook_discussions_run_day` on `(run_event_id, day)` using `fksinks.PostgresIndex{Unique: true}`.
- Remove `headline`, `summary`, and `discussion` from the target `outlooks` schema. Do not carry these legacy nullable columns forward.

### Mapper Changes

Update `internal/sinks/postgres/map.go`:

- Route `standards.SchemaWeatherOutlookV2` to the outlook mapper.
- Remove `standards.SchemaWeatherOutlookV1` support from the active outlook mapper unless there is a compile-time reason to keep a clearly documented legacy test. Existing v1 outlook rows are intentionally discarded during the table reset and the SPC normalizer must not produce v1 events.
- Decode payload into the updated `model.WeatherOutlookRun`.
- Write `outlook_runs.discussion_count = len(run.Discussions)`.
- Write one `outlook_discussions` row for each `run.Discussions[i]`.
- Use `discussion_index` as the array position.
- Use parent `run.AsOf.UTC()` for `as_of`.
- Normalize `UpdatedAt` to UTC when present.
- Validate discussion entries before writing:
  - `day` must be 1, 2, or 3;
  - at least one of `headline`, `summary`, or `discussion` must be non-empty;
  - duplicate discussion days in one run should fail before hitting the unique index.
- Continue validating outlook required fields:
  - `id`, `provider`, `product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`, `expiresAt`, and `geometry` remain required.
- Add a v2-specific invariant validation: every persisted v2 outlook must have `ContainsLocation == true`. If the mapper continues to accept v1, apply this invariant only to v2.
- Continue compacting geometry JSON and normalizing all timestamps to UTC.

### Tests

Update `internal/sinks/postgres/schema_test.go`:

- Assert `outlook_runs` includes `discussion_count`.
- Assert `outlook_discussions` exists with expected columns, primary key, prune column, day/as-of index, and unique run/day index.
- Assert `outlooks` does not include `headline`, `summary`, or `discussion`.

Update `internal/sinks/postgres/map_test.go`:

- Assert v2 outlook run writes:
  - one parent row;
  - one outlook row per retained outlook;
  - one discussion row per run discussion;
  - correct `outlook_count` and `discussion_count`.
- Assert empty local run writes parent row with `outlook_count = 0` and `discussion_count = 0` and no child rows.
- Assert discussion rows map day/headline/summary/discussion/updatedAt correctly.
- Assert duplicate discussion days fail with useful context.
- Assert invalid discussion day fails with useful context.
- Assert empty discussion content fails with useful context.
- Assert v2 outlook with `ContainsLocation == false` fails.
- Update existing tests that used polygon-level `Headline`, `Summary`, or `Discussion`.
- Keep compact geometry and UTC normalization tests.

### Verification

```sh
go test ./internal/sinks/postgres
```

## Stage 4: Normalizer Registry And Cross-Package Consistency

### Code Changes

- Confirm SPC normalizer registration remains unchanged except for output schema.
- Update any package-level tests under `internal/normalizers` that assert canonical schema routing or supported schema names.
- Update any source/registry documentation consistency tests that expect canonical schema strings.
- Search for stale `SchemaWeatherOutlookV1` usage:

```sh
rg "SchemaWeatherOutlookV1|weather\.outlook\.v1|headline|summary|discussion" model standards internal docs/consumers docs/integrations docs/internal
```

- Keep `SchemaWeatherOutlookV1` only where intentionally retained for historical docs or standards compatibility. Do not route new SPC normalized events to v1.
- Ensure no SPC normalizer test, source test, or current v2 documentation path still claims all polygons are emitted.

### Tests

```sh
go test ./internal/normalizers ./internal/normalizers/...
go test ./internal/sources ./cmd/weatherfeeder
```

## Stage 5: Transition Documentation

### Roadmap Transition Doc

Create `docs/roadmap/outlook-schema-transition.md` with operator transition guidance.

The document must include:

- Purpose: reset existing Postgres outlook tables from the `weather.outlook.v1` storage shape to the v2-compatible storage shape.
- Scope: drops only the outlook table family and lets updated `weatherfeeder` recreate it. Other weather tables are not affected.
- Explicit warning: these commands delete stored SPC outlook history.
- Deployment order:
  1. Stop `weatherfeeder`.
  2. Drop existing outlook tables.
  3. Deploy updated `weatherfeeder`.
  4. Start `weatherfeeder` so the Postgres sink recreates the v2 tables.
  5. Deploy updated downstream consumers such as `weatherapi`.
- Transition SQL:

```sql
DROP TABLE IF EXISTS outlook_discussions;
DROP TABLE IF EXISTS outlooks;
DROP TABLE IF EXISTS outlook_runs;
```

- Verification SQL examples:

```sql
SELECT table_name
FROM information_schema.tables
WHERE table_name IN ('outlook_runs', 'outlooks', 'outlook_discussions')
ORDER BY table_name;
```

After the updated daemon has started and recreated tables:

```sql
SELECT column_name, is_nullable, data_type
FROM information_schema.columns
WHERE table_name = 'outlook_runs'
  AND column_name = 'discussion_count';

SELECT indexname
FROM pg_indexes
WHERE tablename = 'outlook_discussions'
ORDER BY indexname;

SELECT column_name
FROM information_schema.columns
WHERE table_name = 'outlooks'
  AND column_name IN ('headline', 'summary', 'discussion');
```

- State that the final verification query for legacy `outlooks` prose columns should return zero rows.
- State that v1 outlook rows are intentionally removed and downstream readers should be updated intentionally.

### Tests

No code tests are required for the roadmap transition doc, but documentation consistency tests may need updates if they check table names or schema strings.

## Stage 6: Permanent Documentation Updates

After the implementation is complete, update permanent current-behavior documentation. Do not leave these as roadmap-only notes.

### Required Docs

Update `docs/policy/architecture.md`:

- Add a core design principle that `weatherfeeder` is location-focused and canonical weather events should represent weather data relevant to the configured forecast location or configured provider object; it is not intended to archive all provider data for all places.

Update `docs/integrations/events.md`:

- Document `weather.outlook.v2`.
- Document `WeatherOutlookRun.discussions` and `WeatherOutlookDiscussion` fields.
- Document that `outlooks[]` is location-filtered.
- Document `containsLocation` as an invariant that is true for emitted v2 outlooks.
- Document empty local runs with `outlooks: []` and `discussions: []`.
- Document supersession semantics:
  - latest-run semantics preferred;
  - historical supersession key is `provider`, `product`, `outlookType`, `validFrom`, `validTo`;
  - greatest `issuedAt` wins;
  - `day` and `label` are not identity;
  - preserve all polygons from the selected latest `issuedAt` group.
- Keep any v1 notes clearly marked legacy/historical if retained.

Update `docs/integrations/spc.md`:

- State raw SPC bundles remain complete.
- State canonical output is location-filtered.
- State discussions are day-level and only included for retained outlook days.
- Remove or replace claims that all polygons are emitted or discussion text is attached to every polygon.
- Add supersession guidance or link to the canonical section in `docs/integrations/events.md`.

Update `docs/integrations/postgres.md`:

- Add `outlook_runs.discussion_count`.
- Add `outlook_discussions` table, columns, keys, indexes, prune column, and mapping source.
- Remove `outlooks.headline`, `outlooks.summary`, and `outlooks.discussion` from the current table contract.
- Document v2 reader reconstruction: read `outlook_runs`, `outlooks`, and `outlook_discussions` by `run_event_id` ordered by child indexes.

Update `docs/internal/postgres-sink.md` and `internal/sinks/postgres/doc.go`:

- Document the new table and mapper validation rules.

Update `docs/consumers/pkg-model.md` and `docs/consumers/api.md`:

- Add `WeatherOutlookDiscussion`.
- Update model examples for run-level discussions.
- Remove polygon-level prose from v2 consumer examples.

Update `docs/config.md` only if SPC prose still describes all-polygons behavior.

Update `docs/operations.md` or `docs/troubleshooting.md` only if operational behavior or recovery instructions need adjustment.

### Tests

- Update docs consistency tests for `weather.outlook.v2`.
- Update docs consistency tests for `outlook_discussions` if such tests exist or are practical.
- Ensure no permanent doc still describes v1 all-polygons behavior as current behavior:

```sh
rg "All outlook polygons|all polygons|attached to every outlook|weather\.outlook\.v1" docs README.md
```

## Stage 7: Full Verification And Release Readiness

### Focused Tests

```sh
go test ./model ./standards
go test ./internal/providers/spc ./internal/normalizers/spc ./internal/normalizers ./internal/geo
go test ./internal/sinks/postgres
go test ./internal/sources ./cmd/weatherfeeder
```

### Full Test Suite

```sh
go test ./...
```

### Manual Checks

- Confirm `cmd/weatherfeeder/config.yml` still loads and source registry tests pass.
- Confirm no current-behavior docs outside `docs/roadmap/` describe unimplemented behavior.
- Confirm `docs/roadmap/outlook-schema-transition.md` transition SQL and verification steps match the final table names and `internal/sinks/postgres/schema.go`.
- Confirm the SPC normalizer emits `weather.outlook.v2`, not `weather.outlook.v1`.
- Confirm a no-local-risk SPC bundle emits a canonical run with empty outlook and discussion arrays.
- Confirm a local-risk SPC bundle emits only containing polygons and only matching day discussions.
- Confirm retained v2 outlooks all have `containsLocation: true`.

## Weatherapi Coordination Notes

This stage is not implemented in `weatherfeeder`, but the weatherfeeder release should call it out for downstream work.

- Release `weatherfeeder` with `weather.outlook.v2` before updating `weatherapi` dependency.
- Update `weatherapi` Postgres reads to load `outlook_discussions`.
- Update weatherapi endpoint semantics to prefer latest-run behavior for current outlook endpoints.
- Remove or revise weatherapi filters that depend on historical `containsLocation=false` rows.
- Consider keeping `/outlooks/convective/location` as a compatibility alias for latest local outlooks if already exposed.

## Open Questions

None. This plan chooses the long-term maintainable options from `docs/roadmap/outlook.md`: schema v2, location-filtered canonical output, run-level discussions, destructive reset of the outlook table family, and permanent documentation of supersession semantics.