285 lines
13 KiB
Markdown
285 lines
13 KiB
Markdown
# NWS AFD Section Parsing Resilience Implementation Plan
|
|
|
|
## Purpose
|
|
|
|
Complete the resilience end state in
|
|
[`afd-section-heading-variants.md`](afd-section-heading-variants.md) while
|
|
preserving the current canonical forecast-discussion contract. Stages 1-3 below
|
|
summarize completed work. Implement Stages 4-8 in order.
|
|
|
|
## Cross-Stage Constraints
|
|
|
|
- Keep all production parsing changes in
|
|
`internal/providers/nws/forecast_discussion.go`.
|
|
- Use only the Go standard library and keep helpers unexported.
|
|
- Do not change `model`, `standards`, source configuration or polling, schemas,
|
|
event-envelope behavior, Postgres mapping, sinks, consumer docs, or current
|
|
integration contracts.
|
|
- Continue exposing only key messages, short term, and long term. All other
|
|
structurally valid identities are boundary-only.
|
|
- Preserve first-occurrence behavior for mapped sections.
|
|
- Preserve raw body lines during structural scanning; apply provider-specific
|
|
cleanup only when parsing a block's content.
|
|
- Keep parsing deterministic. Tests must use local strings and fixtures, never
|
|
live NWS requests.
|
|
- Preserve all pre-existing user work and avoid unrelated refactors.
|
|
|
|
## Stage 1: Centralize Known Heading Parsing — Completed
|
|
|
|
The provider parser gained one heading classifier, explicit section and block
|
|
types, and shared discovery, boundary, and qualifier handling for the original
|
|
four identities. Key-message and text-section consumers were moved to the block
|
|
representation, and immediately adjacent recognized headings became valid
|
|
boundaries.
|
|
|
|
## Stage 2: Add Heading-Variant Regressions — Completed
|
|
|
|
Provider tests now cover ellipsis-first and slash-qualified headings, malformed
|
|
forms, empty bodies, mixed heading families, and slash-qualified aviation
|
|
termination. Normalizer coverage verifies canonical propagation and unchanged
|
|
wire shape.
|
|
|
|
## Stage 3: Validate and Record the Baseline — Completed
|
|
|
|
Focused and full tests passed, production changes remained inside the NWS
|
|
provider parser, and the original feature roadmap was marked implemented.
|
|
|
|
## Stage 4: Generalize Heading Syntax and Centralize Canonical Roles
|
|
|
|
Decouple structural heading recognition from canonical section selection.
|
|
|
|
1. In `internal/providers/nws/forecast_discussion.go`, replace the four-name
|
|
regular-expression alternation with a generic heading parser. Keep the
|
|
`forecastDiscussionSectionHeading` result, with normalized `section` and
|
|
`qualifier` fields.
|
|
2. Implement the generic parser with these decisions:
|
|
|
|
- trim outer whitespace and require a leading `.`;
|
|
- try the slash-qualified form before the legacy ellipsis-first form so the
|
|
terminal ellipsis of an all-uppercase slash heading cannot be mistaken for
|
|
a legacy identity;
|
|
- accept identities made only from uppercase ASCII letters, digits,
|
|
horizontal whitespace, `/`, `&`, apostrophes, and hyphens, with at least
|
|
one letter or digit;
|
|
- trim the identity and collapse every internal run of spaces or tabs to one
|
|
ASCII space before returning it or performing role lookup;
|
|
- accept optional horizontal whitespace between the identity and the legacy
|
|
`...` delimiter;
|
|
- for legacy headings, store trimmed text after the first `...` as the
|
|
qualifier, preserving parentheses and permitting an empty value;
|
|
- for slash-qualified headings, remove the terminal `/...`, then use the
|
|
first slash preceded by horizontal whitespace as the opening separator;
|
|
trim the identity before that separator and the qualifier after it, reject
|
|
an empty qualifier, and allow additional `/` characters inside the
|
|
qualifier; slashes inside an identity must therefore be adjacent to its
|
|
other identity characters rather than preceded by whitespace;
|
|
- reject trailing text after a slash-qualified terminal, lowercase or mixed
|
|
case identities, missing delimiters, and lines whose identity contains
|
|
other punctuation; and
|
|
- keep ASCII `...` as the only delimiter because that is the raw product
|
|
convention; do not interpret a Unicode ellipsis.
|
|
3. Replace the duplicated identity constants and identity-pattern string with:
|
|
|
|
- an unexported `forecastDiscussionSectionRole` enum for key messages, short
|
|
term, and long term; and
|
|
- one `map[string]forecastDiscussionSectionRole` containing exactly `KEY
|
|
MESSAGES`, `SHORT TERM`, and `LONG TERM`.
|
|
|
|
Unknown identities and `AVIATION` intentionally have no role entry; successful
|
|
structural parsing is sufficient for boundary behavior.
|
|
4. Update the heading table test in
|
|
`internal/providers/nws/forecast_discussion_test.go`. Preserve every legacy
|
|
positive case and revise the old policy-specific negatives:
|
|
|
|
- `.SYNOPSIS...`, `.UPDATE...`, `.MARINE...`, `.HYDROLOGY...`, an office
|
|
watch/advisory heading, and `.PRELIMINARY POINT TEMPS/POPS ...` are valid
|
|
generic headings;
|
|
- a slash-qualified heading whose qualifier contains `/` is valid;
|
|
- leading/trailing whitespace, multiple separator spaces, repeated internal
|
|
identity whitespace, and whitespace before a legacy ellipsis are valid and
|
|
produce the normalized identity; and
|
|
- lowercase prose, an empty or punctuation-only identity, malformed slash
|
|
terminals, missing ellipses, and unsupported identity punctuation remain
|
|
invalid.
|
|
5. Add assertions that every role-registry key parses successfully, and confirm
|
|
that no production switch or second collection repeats the mapped identity
|
|
list.
|
|
|
|
Run and pass:
|
|
|
|
```sh
|
|
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws
|
|
```
|
|
|
|
Do not proceed until generic syntax tests pass without changing canonical
|
|
models or schemas.
|
|
|
|
## Stage 5: Scan Ordered Blocks Once and Use Generic Boundaries
|
|
|
|
Replace repeated per-identity extraction with one structural pass.
|
|
|
|
1. Replace `extractForecastDiscussionSection` with
|
|
`parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSectionBlock`.
|
|
The scanner must:
|
|
|
|
- traverse lines once in source order;
|
|
- start a block on every structurally valid heading;
|
|
- finish the active block before a new heading;
|
|
- finish and clear the active block on `&&`;
|
|
- finish the active block and stop scanning on `$$`;
|
|
- retain the existing watch/advisory termination safeguard even though a
|
|
normal dotted watch/advisory heading is structurally recognized;
|
|
- ignore preamble lines before the first heading and signature lines after
|
|
`$$`; and
|
|
- append original, untrimmed body lines to the active block.
|
|
2. Refactor `ParseForecastDiscussionText` to iterate over the ordered blocks
|
|
once and look up each heading identity in the role registry. Ignore blocks
|
|
with no role. Populate each mapped output only if it has not already been
|
|
populated, so the first occurrence wins. Track seen roles explicitly rather
|
|
than inferring them from output values, because an empty first key-message
|
|
block still counts as the first occurrence.
|
|
3. Preserve contextual errors when a mapped text block fails to parse; include
|
|
the parsed heading identity in the wrapped error.
|
|
4. Replace extraction tests with scanner tests covering:
|
|
|
|
- consecutive headings and empty bodies;
|
|
- `&&`, `$$`, and watch/advisory termination;
|
|
- unknown valid headings terminating short- or long-term content even when
|
|
no `&&` is present;
|
|
- malformed heading-like lines remaining inside the active body;
|
|
- preamble and post-signature exclusion;
|
|
- ordered block retention; and
|
|
- duplicate mapped sections where the parser keeps the first canonical
|
|
occurrence.
|
|
|
|
Run and pass:
|
|
|
|
```sh
|
|
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws
|
|
```
|
|
|
|
## Stage 6: Normalize Multiline Preambles and NWS Presentation Markers
|
|
|
|
Handle real section-layout variation without weakening heading syntax.
|
|
|
|
1. Add an exact provider-local marker predicate for lines whose trimmed value is
|
|
`-- Changed Discussion --` or `-- End Changed Discussion --`, compared
|
|
case-insensitively. Add a helper that removes only those complete marker
|
|
lines from a block body. Do not remove arbitrary dashed lines or bullet text.
|
|
2. Apply marker removal before parsing both key-message and text-section bodies.
|
|
3. Refactor text-section preamble parsing in this exact order:
|
|
|
|
- trim blank lines after marker removal;
|
|
- start with the qualifier parsed from the heading;
|
|
- only when that qualifier is empty, consume the first content line as a
|
|
qualifier if its trimmed value is a nonempty standalone parenthetical
|
|
string beginning with `(` and ending with `)`; preserve the parentheses;
|
|
- after the optional qualifier, consume an optional `Issued at` line and
|
|
parse it into `ForecastDiscussionSection.IssuedAt`; and
|
|
- pass only the remaining lines to existing signature trimming and paragraph
|
|
joining.
|
|
4. Make recognition of the `Issued at` label ASCII case-insensitive while
|
|
retaining the existing timestamp grammar and error behavior. The top-level
|
|
header path, which passes an unlabeled timestamp, must remain compatible.
|
|
5. Before key-message bullet parsing, remove markers, trim blank lines, and
|
|
discard at most one leading metadata line beginning case-insensitively with
|
|
`Issued at` or `Updated at`. Do not discard timestamp-like lines after the
|
|
first message begins.
|
|
6. Add focused tests for:
|
|
|
|
- same-line legacy and slash qualifiers remaining unchanged;
|
|
- next-line parenthesized qualifiers followed by `Issued at`;
|
|
- uppercase `ISSUED AT`;
|
|
- a heading qualifier taking precedence over a following parenthetical prose
|
|
line;
|
|
- empty sections;
|
|
- invalid issue timestamps retaining contextual errors;
|
|
- marker removal at the beginning and end of text sections;
|
|
- key-message markers and a leading `Updated at` line not becoming messages;
|
|
and
|
|
- arbitrary dashed prose remaining content.
|
|
|
|
Run and pass:
|
|
|
|
```sh
|
|
gofmt -w internal/providers/nws/forecast_discussion.go internal/providers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws
|
|
```
|
|
|
|
## Stage 7: Add Cross-Office Fixtures and Normalizer Regressions
|
|
|
|
Prove the resilient parser against representative source shapes rather than only
|
|
synthetic heading replacement.
|
|
|
|
1. Retain the existing LSX fixture and mixed-format test for regression
|
|
compatibility.
|
|
2. Add one compact, maintained HTML fixture under
|
|
`internal/providers/nws/testdata/` representing a second real NWS formatting
|
|
family. It must contain:
|
|
|
|
- a valid AFD header and issue time;
|
|
- key-message change markers plus a leading `Updated at` line;
|
|
- ellipsis-first short- and long-term headings with qualifiers on the next
|
|
line;
|
|
- `Issued at` lines following those qualifiers;
|
|
- at least one structurally valid boundary-only section; and
|
|
- representative prose sufficient to detect metadata or adjacent-section
|
|
leakage.
|
|
|
|
Keep the fixture concise; include no unrelated webpage content, secrets, or
|
|
private data.
|
|
3. Add a provider end-to-end test that parses the new fixture and asserts exact
|
|
key messages, qualifiers, section issue times, representative prose, and
|
|
absence of change markers, timestamp metadata, and boundary-only content.
|
|
4. Add a normalizer regression using the same fixture. Verify kind, canonical
|
|
schema, envelope/effective-time behavior, short- and long-term mapping, and
|
|
JSON wire shape without `aviation`, `discussion`, or generic `sections`
|
|
fields.
|
|
5. Keep small parser and scanner edge cases table-driven. Do not multiply full
|
|
fixtures for cases that a short line slice proves more clearly.
|
|
|
|
Run and pass:
|
|
|
|
```sh
|
|
gofmt -w internal/providers/nws/forecast_discussion_test.go internal/normalizers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
|
|
```
|
|
|
|
## Stage 8: Reconcile Documentation and Perform Final Validation
|
|
|
|
Close the resilience follow-up only after all behavior is proven.
|
|
|
|
1. Review the final diff. Production changes must remain confined to the NWS
|
|
provider parser; other Go changes should be tests in the owning provider and
|
|
normalizer packages. Preserve all unrelated user work.
|
|
2. Confirm the public contract is unchanged. Do not edit canonical model,
|
|
schema, Postgres, config, consumer, or integration documentation unless an
|
|
actual contract change is discovered. If one appears necessary, stop rather
|
|
than expanding this plan.
|
|
3. Run:
|
|
|
|
```sh
|
|
gofmt -w \
|
|
internal/providers/nws/forecast_discussion.go \
|
|
internal/providers/nws/forecast_discussion_test.go \
|
|
internal/normalizers/nws/forecast_discussion_test.go
|
|
go test -count=1 ./internal/providers/nws ./internal/normalizers/nws
|
|
go test -count=1 ./...
|
|
go vet ./...
|
|
git diff --check
|
|
```
|
|
4. Verify every acceptance criterion in the feature roadmap has a corresponding
|
|
passing automated test, including compatibility with all Stage 1-2 cases.
|
|
5. After all checks pass, change the feature roadmap status to `Implemented.`
|
|
and rewrite any remaining future-tense statements that would misdescribe the
|
|
completed parser. Do not mark the follow-up implemented earlier.
|
|
|
|
## Open Questions
|
|
|
|
None. This plan fixes the parser grammar, boundary policy, canonical role
|
|
selection, multiline preamble rules, presentation cleanup, fixture strategy,
|
|
and compatibility boundary. Generic or single-section discussion content remains
|
|
outside the canonical model by explicit policy.
|