148 lines
5.6 KiB
Markdown
148 lines
5.6 KiB
Markdown
# NWS AFD Section Parsing Resilience
|
|
|
|
## Status
|
|
|
|
Implemented.
|
|
|
|
## Implemented Behavior
|
|
|
|
The NWS Area Forecast Discussion parser recognizes these heading families for
|
|
all structurally valid section identities:
|
|
|
|
```text
|
|
.<SECTION>...<optional qualifier>
|
|
.<SECTION> /<qualifier>/...
|
|
```
|
|
|
|
Discovery, qualifier extraction, and recognized-section boundary detection use
|
|
one provider-specific parser. Slash delimiters are removed from qualifiers,
|
|
legacy qualifier text is preserved, and aviation remains a boundary rather than
|
|
a canonical field.
|
|
|
|
The parser separates:
|
|
|
|
1. generic structural heading recognition;
|
|
2. section boundary scanning;
|
|
3. canonical section-role selection; and
|
|
4. section-preamble and presentation cleanup.
|
|
|
|
Structurally valid topic names act as safe boundaries without becoming new
|
|
canonical fields.
|
|
|
|
## Implemented Details
|
|
|
|
### Generic heading recognition
|
|
|
|
One NWS-specific heading parser recognizes structurally valid uppercase AFD
|
|
topic lines independently of the canonical model. It supports:
|
|
|
|
```text
|
|
.<IDENTITY>...<optional qualifier>
|
|
.<IDENTITY> /<qualifier>/...
|
|
```
|
|
|
|
The identity may contain uppercase ASCII letters, digits, horizontal whitespace,
|
|
`/`, `&`, apostrophes, and hyphens. Outer whitespace and whitespace immediately
|
|
before the ellipsis are ignored, and internal identity whitespace is collapsed
|
|
to one space for role lookup. Slash-qualified headings require a nonempty
|
|
qualifier but may contain slash characters inside that qualifier. Lowercase
|
|
prose, ordinary dotted lines, and malformed delimiters remain non-headings.
|
|
|
|
Every structurally valid heading terminates the preceding section. This applies
|
|
to known boundary-only sections such as aviation and to future or office-specific
|
|
topic identities that weatherfeeder does not map.
|
|
|
|
### Canonical role selection
|
|
|
|
A single provider-local role registry maps only these identities into existing
|
|
parsed fields:
|
|
|
|
- `KEY MESSAGES` to key messages;
|
|
- `SHORT TERM` to the short-term section; and
|
|
- `LONG TERM` to the long-term section.
|
|
|
|
All other identities are boundary-only. The first occurrence of each mapped
|
|
identity wins, preserving current behavior if an unusual bulletin repeats a
|
|
section. Heading syntax and canonical roles do not duplicate identity lists.
|
|
|
|
### Section scanning
|
|
|
|
The discussion text is scanned once into ordered section blocks. A new heading,
|
|
`&&`, `$$`, or the existing watch/advisory terminator ends the active block.
|
|
Original body lines are preserved until provider presentation cleanup is
|
|
applied. Preamble text before the first heading and post-`$$` signatures are not
|
|
treated as section content.
|
|
|
|
### Preamble and presentation handling
|
|
|
|
For short- and long-term sections:
|
|
|
|
- a qualifier on the heading line remains authoritative;
|
|
- when the heading has no qualifier, a standalone parenthesized first content
|
|
line becomes the qualifier and retains its parentheses;
|
|
- an optional case-insensitive `Issued at` line after the qualifier is parsed
|
|
into the existing section issue time; and
|
|
- exact NWS change-presentation marker lines are removed without removing
|
|
arbitrary dashed prose.
|
|
|
|
For key messages, exact change-presentation markers and one leading `Issued at`
|
|
or `Updated at` metadata line are removed before bullet parsing. Those metadata
|
|
lines never become key messages.
|
|
|
|
### Representative coverage
|
|
|
|
Tests include small structural tables and maintained local HTML fixtures for at
|
|
least two real NWS formatting families: slash-qualified same-line headings and
|
|
ellipsis-first headings with multiline qualifiers or change-presentation
|
|
markers. Tests remain deterministic and never contact live services.
|
|
|
|
## Compatibility and Contracts
|
|
|
|
This remains a provider-parsing compatibility improvement. It does not change:
|
|
|
|
- event kinds or raw and canonical schema identifiers;
|
|
- canonical models or JSON field names;
|
|
- source configuration, URLs, or polling behavior;
|
|
- event envelope or effective-time behavior;
|
|
- Postgres tables or event-to-row mapping; or
|
|
- downstream sink and consumer responsibilities.
|
|
|
|
A single `DISCUSSION` section and other currently unmapped identities are
|
|
recognized as boundaries but are not forced into short- or long-term fields.
|
|
Exposing such content would require a separate canonical schema decision.
|
|
|
|
## Verified Coverage
|
|
|
|
Automated tests demonstrate that:
|
|
|
|
- every previously accepted heading and canonical result remains compatible;
|
|
- generic structurally valid identities terminate preceding content without
|
|
becoming canonical fields;
|
|
- malformed heading-like lines and lowercase prose remain body content;
|
|
- identities containing `/` and slash qualifiers containing `/` are parsed
|
|
without ambiguity;
|
|
- next-line parenthesized qualifiers and following `Issued at` lines populate
|
|
the existing short- and long-term metadata correctly;
|
|
- exact NWS change markers and leading key-message timestamps do not leak into
|
|
canonical prose or messages;
|
|
- repeated mapped sections preserve first-occurrence behavior;
|
|
- genuine cross-office fixture styles propagate correctly through the NWS
|
|
normalizer with unchanged wire shape; and
|
|
- focused tests, the full repository suite, and static analysis pass.
|
|
|
|
## Non-Goals
|
|
|
|
This follow-up does not:
|
|
|
|
- add canonical `discussion`, aviation, marine, hydrology, climate, fire
|
|
weather, update, or arbitrary-section fields;
|
|
- infer short- or long-term semantics from an unknown heading;
|
|
- accept free-form or lowercase prose as a heading;
|
|
- introduce heuristic summarization;
|
|
- preserve complete raw AFD documents in canonical payloads;
|
|
- change schemas, persistence contracts, configuration, or downstream APIs; or
|
|
- fetch live NWS data during tests.
|
|
|
|
Additional canonical support for AFD section identities requires a separate
|
|
consumer requirement and schema roadmap.
|