ejr/weatherfeeder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Generalize NWS forecast discussion heading parsing

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-02 22:53:00 +00:00
parent 29c65971eb
commit 2b19a121fa
4 changed files with 618 additions and 331 deletions
Download Patch File Download Diff File Expand all files Collapse all files

240
docs/roadmap/afd-section-heading-variants.md
View File

@@ -1,141 +1,171 @@
# NWS AFD Section Heading Variants
# NWS AFD Section Parsing Resilience
## Status
Implemented.
The original ellipsis-first and slash-qualified heading feature is implemented.
The resilience follow-up defined below is proposed and unimplemented.
## Problem
## Completed Baseline
The NWS Area Forecast Discussion parser recognizes supported section headings
only when the section name is followed immediately by an ellipsis, for example:
```text
.SHORT TERM... (Through Monday)
.LONG TERM... (Tuesday through Sunday)
.AVIATION... (For the 18z TAFs through 18z Monday)
```
NWS offices also publish slash-qualified headings:
```text
.SHORT TERM /THROUGH MONDAY/...
.LONG TERM /MONDAY NIGHT THROUGH SUNDAY/...
.AVIATION /18Z TAFS THROUGH 18Z MONDAY/...
```
The current exact-prefix discovery logic does not find slash-qualified short-
or long-term sections. Its boundary detection also does not recognize a
slash-qualified aviation heading, so preceding section text can absorb aviation
content. The forecast discussion can still normalize successfully, leaving
downstream consumers with missing or incorrectly bounded structured prose.
## Feature Objective
Make supported AFD section recognition consistent across section discovery,
section-boundary detection, and qualifier extraction while preserving the
existing canonical forecast-discussion contract.
## Targeted End State
One NWS-specific heading parser classifies a trimmed AFD line as either a
supported heading or a non-heading. For a supported heading it provides:
- the canonical section identity;
- the qualifier, if present; and
- enough information for discovery and boundary detection to use the same
recognition result.
The supported section identities remain:
- `KEY MESSAGES`;
- `SHORT TERM`;
- `LONG TERM`; and
- `AVIATION`.
`KEY MESSAGES`, `SHORT TERM`, and `LONG TERM` participate in the existing
canonical extraction behavior. `AVIATION` remains a recognized boundary only;
it does not become a canonical payload field.
For each supported identity, the parser accepts both heading families:
The NWS Area Forecast Discussion parser now recognizes these heading families
for key messages, short term, long term, and aviation:
```text
.<SECTION>...<optional qualifier>
.<SECTION> /<qualifier>/...
```
Qualifier handling is deterministic:
Discovery, qualifier extraction, and recognized-section boundary detection use
one provider-specific parser. Slash delimiters are removed from qualifiers,
legacy qualifier text is preserved, and aviation remains a boundary rather than
a canonical field. Focused provider and normalizer regressions cover the
implemented forms.
- leading and trailing whitespace is removed;
- the enclosing slash pair used by the slash-qualified form is removed;
- qualifier content otherwise retains its published text, including
parentheses in the existing ellipsis-first form; and
- a heading without a qualifier yields an empty qualifier.
## Remaining Problem
The same classification rules govern discovery and termination. A recognized
heading ends the preceding section without becoming part of its text. Unknown
section names, unrelated dotted lines, and malformed slash-qualified lines are
not treated as supported headings.
Current NWS bulletins vary beyond those two same-line forms. In particular:
## Scope of Work
- an ellipsis-first heading may put its qualifier on the next nonblank line;
- NWS presentation output can place `-- Changed Discussion --` markers and an
`Updated at` line around section content;
- valid AFDs can include `DISCUSSION`, `UPDATE`, `MARINE`, `HYDROLOGY`,
`CLIMATE`, `FIRE WEATHER`, office watch/advisory blocks, and other topic
headings; and
- minor punctuation or qualifier changes can produce headings that are
structurally valid but absent from the parser's identity whitelist.
The feature includes:
The current parser recognizes only four exact identities. Unknown headings do
not terminate the preceding section. It also expects a text section's optional
qualifier and `Issued at` line in a narrow order. Consequently, valid upstream
format variants can leave qualifier and section-time fields empty, leak
presentation metadata into prose or key messages, or allow one section to
absorb another.
- consolidating supported-heading recognition and qualifier extraction in
`internal/providers/nws`;
- applying that recognition consistently to section lookup and section
boundaries;
- preserving key-message parsing and short- and long-term section mapping;
- covering legacy, slash-qualified, mixed-format, and malformed headings with
focused provider-parser tests; and
- verifying that parsed short- and long-term sections continue through the NWS
normalizer into the existing canonical payload.
## Feature Objective
Existing ellipsis-first fixtures and behavior remain regression coverage. New
fixtures should be minimal and representative rather than copies of complete
upstream bulletins unless a full bulletin is needed to prove an interaction.
Make AFD parsing resilient to minor upstream format evolution by separating:
1. generic structural heading recognition;
2. section boundary scanning;
3. canonical section-role selection; and
4. section-preamble and presentation cleanup.
The parser should accept new structurally valid topic names as safe boundaries
without treating them as new canonical fields.
## Targeted End State
### Generic heading recognition
One NWS-specific heading parser recognizes structurally valid uppercase AFD
topic lines independently of the canonical model. It supports:
```text
.<IDENTITY>...<optional qualifier>
.<IDENTITY> /<qualifier>/...
```
The identity may contain uppercase ASCII letters, digits, horizontal whitespace,
`/`, `&`, apostrophes, and hyphens. Outer whitespace and whitespace immediately
before the ellipsis are ignored, and internal identity whitespace is collapsed
to one space for role lookup. Slash-qualified headings require a nonempty
qualifier but may contain slash characters inside that qualifier. Lowercase
prose, ordinary dotted lines, and malformed delimiters remain non-headings.
Every structurally valid heading terminates the preceding section. This applies
to known boundary-only sections such as aviation and to future or office-specific
topic identities that weatherfeeder does not map.
### Canonical role selection
A single provider-local role registry maps only these identities into existing
parsed fields:
- `KEY MESSAGES` to key messages;
- `SHORT TERM` to the short-term section; and
- `LONG TERM` to the long-term section.
All other identities are boundary-only. The first occurrence of each mapped
identity wins, preserving current behavior if an unusual bulletin repeats a
section. Heading syntax and canonical roles do not duplicate identity lists.
### Section scanning
The discussion text is scanned once into ordered section blocks. A new heading,
`&&`, `$$`, or the existing watch/advisory terminator ends the active block.
Original body lines are preserved until provider presentation cleanup is
applied. Preamble text before the first heading and post-`$$` signatures are not
treated as section content.
### Preamble and presentation handling
For short- and long-term sections:
- a qualifier on the heading line remains authoritative;
- when the heading has no qualifier, a standalone parenthesized first content
line becomes the qualifier and retains its parentheses;
- an optional case-insensitive `Issued at` line after the qualifier is parsed
into the existing section issue time; and
- exact NWS change-presentation marker lines are removed without removing
arbitrary dashed prose.
For key messages, exact change-presentation markers and one leading `Issued at`
or `Updated at` metadata line are removed before bullet parsing. Those metadata
lines never become key messages.
### Representative coverage
Tests include small structural tables and maintained local HTML fixtures for at
least two real NWS formatting families: slash-qualified same-line headings and
ellipsis-first headings with multiline qualifiers or change-presentation
markers. Tests remain deterministic and never contact live services.
## Compatibility and Contracts
This is a provider-parsing compatibility improvement. It does not change:
This remains a provider-parsing compatibility improvement. It does not change:
- event kinds or raw and canonical schema identifiers;
- canonical models or JSON field names;
- source configuration or polling behavior;
- source configuration, URLs, or polling behavior;
- event envelope or effective-time behavior;
- Postgres tables or event-to-row mapping; or
- downstream sink and consumer responsibilities.
Current-behavior documentation should be updated only if implementation reveals
an externally observable contract change beyond the scope defined here.
A single `DISCUSSION` section and other currently unmapped identities are
recognized as boundaries but are not forced into short- or long-term fields.
Exposing such content would require a separate canonical schema decision.
## Acceptance Criteria
The feature is complete when automated tests demonstrate that:
The resilience follow-up is complete when automated tests demonstrate that:
- all currently accepted ellipsis-first headings produce unchanged results;
- slash-qualified short- and long-term headings are discovered and their
qualifiers exclude slash delimiters;
- slash-qualified recognized headings correctly terminate a preceding section,
including an aviation heading following a long-term section;
- documents mixing the two heading families are parsed correctly;
- headings without qualifiers retain existing behavior;
- unknown or malformed heading-like lines do not create supported sections or
prematurely terminate one;
- the NWS forecast-discussion normalizer emits populated canonical short- and
long-term fields for representative slash-qualified input without adding an
aviation field; and
- the full repository test suite passes.
- every previously accepted heading and canonical result remains compatible;
- generic structurally valid identities terminate preceding content without
becoming canonical fields;
- malformed heading-like lines and lowercase prose remain body content;
- identities containing `/` and slash qualifiers containing `/` are parsed
without ambiguity;
- next-line parenthesized qualifiers and following `Issued at` lines populate
the existing short- and long-term metadata correctly;
- exact NWS change markers and leading key-message timestamps do not leak into
canonical prose or messages;
- repeated mapped sections preserve first-occurrence behavior;
- genuine cross-office fixture styles propagate correctly through the NWS
normalizer with unchanged wire shape; and
- focused tests, the full repository suite, and static analysis pass.
## Non-Goals
This feature does not:
This follow-up does not:
- add new canonical forecast-discussion sections;
- add aviation prose to the canonical model;
- recognize arbitrary or previously unsupported NWS section families;
- introduce heuristic section-name matching or AFD summarization;
- preserve complete raw AFD documents in canonical payloads; or
- change schemas, persistence contracts, configuration, or downstream APIs.
- add canonical `discussion`, aviation, marine, hydrology, climate, fire
weather, update, or arbitrary-section fields;
- infer short- or long-term semantics from an unknown heading;
- accept free-form or lowercase prose as a heading;
- introduce heuristic summarization;
- preserve complete raw AFD documents in canonical payloads;
- change schemas, persistence contracts, configuration, or downstream APIs; or
- fetch live NWS data during tests.
Support for additional section identities or broader AFD structure should be
driven by a separate consumer requirement and roadmap.
Canonical support for additional AFD section identities should be driven by a
separate consumer requirement and schema roadmap.