# NWS AFD Section Parsing Resilience Implementation Plan

## Purpose

Complete the resilience end state in
[`afd-section-heading-variants.md`](afd-section-heading-variants.md) while
preserving the current canonical forecast-discussion contract. Stages 1-3 below
summarize completed work. Implement Stages 4-8 in order.

## Cross-Stage Constraints

- Keep all production parsing changes in
  `internal/providers/nws/forecast_discussion.go`.
- Use only the Go standard library and keep helpers unexported.
- Do not change `model`, `standards`, source configuration or polling, schemas,
  event-envelope behavior, Postgres mapping, sinks, consumer docs, or current
  integration contracts.
- Continue exposing only key messages, short term, and long term. All other
  structurally valid identities are boundary-only.
- Preserve first-occurrence behavior for mapped sections.
- Preserve raw body lines during structural scanning; apply provider-specific
  cleanup only when parsing a block's content.
- Keep parsing deterministic. Tests must use local strings and fixtures, never
  live NWS requests.
- Preserve all pre-existing user work and avoid unrelated refactors.

## Stage 1: Centralize Known Heading Parsing — Completed

The provider parser gained one heading classifier, explicit section and block
types, and shared discovery, boundary, and qualifier handling for the original
four identities. Key-message and text-section consumers were moved to the block
representation, and immediately adjacent recognized headings became valid
boundaries.

## Stage 2: Add Heading-Variant Regressions — Completed

Provider tests now cover ellipsis-first and slash-qualified headings, malformed
forms, empty bodies, mixed heading families, and slash-qualified aviation
termination. Normalizer coverage verifies canonical propagation and unchanged
wire shape.

## Stage 3: Validate and Record the Baseline — Completed

Focused and full tests passed, production changes remained inside the NWS
provider parser, and the original feature roadmap was marked implemented.

## Stage 4: Generalize Heading Syntax and Centralize Canonical Roles

Decouple structural heading recognition from canonical section selection.

1. In `internal/providers/nws/forecast_discussion.go`, replace the four-name
   regular-expression alternation with a generic heading parser. Keep the
   `forecastDiscussionSectionHeading` result, with normalized `section` and
   `qualifier` fields.
2. Implement the generic parser with these decisions:

   - trim outer whitespace and require a leading `.`;
   - try the slash-qualified form before the legacy ellipsis-first form so the
     terminal ellipsis of an all-uppercase slash heading cannot be mistaken for
     a legacy identity;
   - accept identities made only from uppercase ASCII letters, digits,
     horizontal whitespace, `/`, `&`, apostrophes, and hyphens, with at least
     one letter or digit;
   - trim the identity and collapse every internal run of spaces or tabs to one
     ASCII space before returning it or performing role lookup;
   - accept optional horizontal whitespace between the identity and the legacy
     `...` delimiter;
   - for legacy headings, store trimmed text after the first `...` as the
     qualifier, preserving parentheses and permitting an empty value;
   - for slash-qualified headings, remove the terminal `/...`, then use the
     first slash preceded by horizontal whitespace as the opening separator;
     trim the identity before that separator and the qualifier after it, reject
     an empty qualifier, and allow additional `/` characters inside the
     qualifier; slashes inside an identity must therefore be adjacent to its
     other identity characters rather than preceded by whitespace;
   - reject trailing text after a slash-qualified terminal, lowercase or mixed
     case identities, missing delimiters, and lines whose identity contains
     other punctuation; and
   - keep ASCII `...` as the only delimiter because that is the raw product
     convention; do not interpret a Unicode ellipsis.
3. Replace the duplicated identity constants and identity-pattern string with:

   - an unexported `forecastDiscussionSectionRole` enum for key messages, short
     term, and long term; and
   - one `map[string]forecastDiscussionSectionRole` containing exactly `KEY
     MESSAGES`, `SHORT TERM`, and `LONG TERM`.

   Unknown identities and `AVIATION` intentionally have no role entry; successful
   structural parsing is sufficient for boundary behavior.
4. Update the heading table test in
   `internal/providers/nws/forecast_discussion_test.go`. Preserve every legacy
   positive case and revise the old policy-specific negatives:

   - `.SYNOPSIS...`, `.UPDATE...`, `.MARINE...`, `.HYDROLOGY...`, an office
     watch/advisory heading, and `.PRELIMINARY POINT TEMPS/POPS ...` are valid
     generic headings;
   - a slash-qualified heading whose qualifier contains `/` is valid;
   - leading/trailing whitespace, multiple separator spaces, repeated internal
     identity whitespace, and whitespace before a legacy ellipsis are valid and
     produce the normalized identity; and
   - lowercase prose, an empty or punctuation-only identity, malformed slash
     terminals, missing ellipses, and unsupported identity punctuation remain
     invalid.
5. Add assertions that every role-registry key parses successfully, and confirm
   that no production switch or second collection repeats the mapped identity
   list.

Run and pass:

```sh
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
go test -count=1 ./internal/providers/nws
```

Do not proceed until generic syntax tests pass without changing canonical
models or schemas.

## Stage 5: Scan Ordered Blocks Once and Use Generic Boundaries

Replace repeated per-identity extraction with one structural pass.

1. Replace `extractForecastDiscussionSection` with
   `parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSectionBlock`.
   The scanner must:

   - traverse lines once in source order;
   - start a block on every structurally valid heading;
   - finish the active block before a new heading;
   - finish and clear the active block on `&&`;
   - finish the active block and stop scanning on `$$`;
   - retain the existing watch/advisory termination safeguard even though a
     normal dotted watch/advisory heading is structurally recognized;
   - ignore preamble lines before the first heading and signature lines after
     `$$`; and
   - append original, untrimmed body lines to the active block.
2. Refactor `ParseForecastDiscussionText` to iterate over the ordered blocks
   once and look up each heading identity in the role registry. Ignore blocks
   with no role. Populate each mapped output only if it has not already been
   populated, so the first occurrence wins. Track seen roles explicitly rather
   than inferring them from output values, because an empty first key-message
   block still counts as the first occurrence.
3. Preserve contextual errors when a mapped text block fails to parse; include
   the parsed heading identity in the wrapped error.
4. Replace extraction tests with scanner tests covering:

   - consecutive headings and empty bodies;
   - `&&`, `$$`, and watch/advisory termination;
   - unknown valid headings terminating short- or long-term content even when
     no `&&` is present;
   - malformed heading-like lines remaining inside the active body;
   - preamble and post-signature exclusion;
   - ordered block retention; and
   - duplicate mapped sections where the parser keeps the first canonical
     occurrence.

Run and pass:

```sh
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
go test -count=1 ./internal/providers/nws
```

## Stage 6: Normalize Multiline Preambles and NWS Presentation Markers

Handle real section-layout variation without weakening heading syntax.

1. Add an exact provider-local marker predicate for lines whose trimmed value is
   `-- Changed Discussion --` or `-- End Changed Discussion --`, compared
   case-insensitively. Add a helper that removes only those complete marker
   lines from a block body. Do not remove arbitrary dashed lines or bullet text.
2. Apply marker removal before parsing both key-message and text-section bodies.
3. Refactor text-section preamble parsing in this exact order:

   - trim blank lines after marker removal;
   - start with the qualifier parsed from the heading;
   - only when that qualifier is empty, consume the first content line as a
     qualifier if its trimmed value is a nonempty standalone parenthetical
     string beginning with `(` and ending with `)`; preserve the parentheses;
   - after the optional qualifier, consume an optional `Issued at` line and
     parse it into `ForecastDiscussionSection.IssuedAt`; and
   - pass only the remaining lines to existing signature trimming and paragraph
     joining.
4. Make recognition of the `Issued at` label ASCII case-insensitive while
   retaining the existing timestamp grammar and error behavior. The top-level
   header path, which passes an unlabeled timestamp, must remain compatible.
5. Before key-message bullet parsing, remove markers, trim blank lines, and
   discard at most one leading metadata line beginning case-insensitively with
   `Issued at` or `Updated at`. Do not discard timestamp-like lines after the
   first message begins.
6. Add focused tests for:

   - same-line legacy and slash qualifiers remaining unchanged;
   - next-line parenthesized qualifiers followed by `Issued at`;
   - uppercase `ISSUED AT`;
   - a heading qualifier taking precedence over a following parenthetical prose
     line;
   - empty sections;
   - invalid issue timestamps retaining contextual errors;
   - marker removal at the beginning and end of text sections;
   - key-message markers and a leading `Updated at` line not becoming messages;
     and
   - arbitrary dashed prose remaining content.

Run and pass:

```sh
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
go test -count=1 ./internal/providers/nws
```

## Stage 7: Add Cross-Office Fixtures and Normalizer Regressions

Prove the resilient parser against representative source shapes rather than only
synthetic heading replacement.

1. Retain the existing LSX fixture and mixed-format test for regression
   compatibility.
2. Add one compact, maintained HTML fixture under
   `internal/providers/nws/testdata/` representing a second real NWS formatting
   family. It must contain:

   - a valid AFD header and issue time;
   - key-message change markers plus a leading `Updated at` line;
   - ellipsis-first short- and long-term headings with qualifiers on the next
     line;
   - `Issued at` lines following those qualifiers;
   - at least one structurally valid boundary-only section; and
   - representative prose sufficient to detect metadata or adjacent-section
     leakage.

   Keep the fixture concise; include no unrelated webpage content, secrets, or
   private data.
3. Add a provider end-to-end test that parses the new fixture and asserts exact
   key messages, qualifiers, section issue times, representative prose, and
   absence of change markers, timestamp metadata, and boundary-only content.
4. Add a normalizer regression using the same fixture. Verify kind, canonical
   schema, envelope/effective-time behavior, short- and long-term mapping, and
   JSON wire shape without `aviation`, `discussion`, or generic `sections`
   fields.
5. Keep small parser and scanner edge cases table-driven. Do not multiply full
   fixtures for cases that a short line slice proves more clearly.

Run and pass:

```sh
gofmt -w internal/providers/nws/forecast_discussion_test.go internal/normalizers/nws/forecast_discussion_test.go
go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
```

## Stage 8: Reconcile Documentation and Perform Final Validation

Close the resilience follow-up only after all behavior is proven.

1. Review the final diff. Production changes must remain confined to the NWS
   provider parser; other Go changes should be tests in the owning provider and
   normalizer packages. Preserve all unrelated user work.
2. Confirm the public contract is unchanged. Do not edit canonical model,
   schema, Postgres, config, consumer, or integration documentation unless an
   actual contract change is discovered. If one appears necessary, stop rather
   than expanding this plan.
3. Run:

   ```sh
   gofmt -w \
     internal/providers/nws/forecast_discussion.go \
     internal/providers/nws/forecast_discussion_test.go \
     internal/normalizers/nws/forecast_discussion_test.go
   go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
   go test -count=1 ./...
   go vet ./...
   git diff --check
   ```
4. Verify every acceptance criterion in the feature roadmap has a corresponding
   passing automated test, including compatibility with all Stage 1-2 cases.
5. After all checks pass, change the feature roadmap status to `Implemented.`
   and rewrite any remaining future-tense statements that would misdescribe the
   completed parser. Do not mark the follow-up implemented earlier.

## Open Questions

None. This plan fixes the parser grammar, boundary policy, canonical role
selection, multiline preamble rules, presentation cleanup, fixture strategy,
and compatibility boundary. Generic or single-section discussion content remains
outside the canonical model by explicit policy.