172 lines
7.0 KiB
Markdown
172 lines
7.0 KiB
Markdown
# NWS AFD Section Parsing Resilience
|
|
|
|
## Status
|
|
|
|
The original ellipsis-first and slash-qualified heading feature is implemented.
|
|
The resilience follow-up defined below is proposed and unimplemented.
|
|
|
|
## Completed Baseline
|
|
|
|
The NWS Area Forecast Discussion parser now recognizes these heading families
|
|
for key messages, short term, long term, and aviation:
|
|
|
|
```text
|
|
.<SECTION>...<optional qualifier>
|
|
.<SECTION> /<qualifier>/...
|
|
```
|
|
|
|
Discovery, qualifier extraction, and recognized-section boundary detection use
|
|
one provider-specific parser. Slash delimiters are removed from qualifiers,
|
|
legacy qualifier text is preserved, and aviation remains a boundary rather than
|
|
a canonical field. Focused provider and normalizer regressions cover the
|
|
implemented forms.
|
|
|
|
## Remaining Problem
|
|
|
|
Current NWS bulletins vary beyond those two same-line forms. In particular:
|
|
|
|
- an ellipsis-first heading may put its qualifier on the next nonblank line;
|
|
- NWS presentation output can place `-- Changed Discussion --` markers and an
|
|
`Updated at` line around section content;
|
|
- valid AFDs can include `DISCUSSION`, `UPDATE`, `MARINE`, `HYDROLOGY`,
|
|
`CLIMATE`, `FIRE WEATHER`, office watch/advisory blocks, and other topic
|
|
headings; and
|
|
- minor punctuation or qualifier changes can produce headings that are
|
|
structurally valid but absent from the parser's identity whitelist.
|
|
|
|
The current parser recognizes only four exact identities. Unknown headings do
|
|
not terminate the preceding section. It also expects a text section's optional
|
|
qualifier and `Issued at` line in a narrow order. Consequently, valid upstream
|
|
format variants can leave qualifier and section-time fields empty, leak
|
|
presentation metadata into prose or key messages, or allow one section to
|
|
absorb another.
|
|
|
|
## Feature Objective
|
|
|
|
Make AFD parsing resilient to minor upstream format evolution by separating:
|
|
|
|
1. generic structural heading recognition;
|
|
2. section boundary scanning;
|
|
3. canonical section-role selection; and
|
|
4. section-preamble and presentation cleanup.
|
|
|
|
The parser should accept new structurally valid topic names as safe boundaries
|
|
without treating them as new canonical fields.
|
|
|
|
## Targeted End State
|
|
|
|
### Generic heading recognition
|
|
|
|
One NWS-specific heading parser recognizes structurally valid uppercase AFD
|
|
topic lines independently of the canonical model. It supports:
|
|
|
|
```text
|
|
.<IDENTITY>...<optional qualifier>
|
|
.<IDENTITY> /<qualifier>/...
|
|
```
|
|
|
|
The identity may contain uppercase ASCII letters, digits, horizontal whitespace,
|
|
`/`, `&`, apostrophes, and hyphens. Outer whitespace and whitespace immediately
|
|
before the ellipsis are ignored, and internal identity whitespace is collapsed
|
|
to one space for role lookup. Slash-qualified headings require a nonempty
|
|
qualifier but may contain slash characters inside that qualifier. Lowercase
|
|
prose, ordinary dotted lines, and malformed delimiters remain non-headings.
|
|
|
|
Every structurally valid heading terminates the preceding section. This applies
|
|
to known boundary-only sections such as aviation and to future or office-specific
|
|
topic identities that weatherfeeder does not map.
|
|
|
|
### Canonical role selection
|
|
|
|
A single provider-local role registry maps only these identities into existing
|
|
parsed fields:
|
|
|
|
- `KEY MESSAGES` to key messages;
|
|
- `SHORT TERM` to the short-term section; and
|
|
- `LONG TERM` to the long-term section.
|
|
|
|
All other identities are boundary-only. The first occurrence of each mapped
|
|
identity wins, preserving current behavior if an unusual bulletin repeats a
|
|
section. Heading syntax and canonical roles do not duplicate identity lists.
|
|
|
|
### Section scanning
|
|
|
|
The discussion text is scanned once into ordered section blocks. A new heading,
|
|
`&&`, `$$`, or the existing watch/advisory terminator ends the active block.
|
|
Original body lines are preserved until provider presentation cleanup is
|
|
applied. Preamble text before the first heading and post-`$$` signatures are not
|
|
treated as section content.
|
|
|
|
### Preamble and presentation handling
|
|
|
|
For short- and long-term sections:
|
|
|
|
- a qualifier on the heading line remains authoritative;
|
|
- when the heading has no qualifier, a standalone parenthesized first content
|
|
line becomes the qualifier and retains its parentheses;
|
|
- an optional case-insensitive `Issued at` line after the qualifier is parsed
|
|
into the existing section issue time; and
|
|
- exact NWS change-presentation marker lines are removed without removing
|
|
arbitrary dashed prose.
|
|
|
|
For key messages, exact change-presentation markers and one leading `Issued at`
|
|
or `Updated at` metadata line are removed before bullet parsing. Those metadata
|
|
lines never become key messages.
|
|
|
|
### Representative coverage
|
|
|
|
Tests include small structural tables and maintained local HTML fixtures for at
|
|
least two real NWS formatting families: slash-qualified same-line headings and
|
|
ellipsis-first headings with multiline qualifiers or change-presentation
|
|
markers. Tests remain deterministic and never contact live services.
|
|
|
|
## Compatibility and Contracts
|
|
|
|
This remains a provider-parsing compatibility improvement. It does not change:
|
|
|
|
- event kinds or raw and canonical schema identifiers;
|
|
- canonical models or JSON field names;
|
|
- source configuration, URLs, or polling behavior;
|
|
- event envelope or effective-time behavior;
|
|
- Postgres tables or event-to-row mapping; or
|
|
- downstream sink and consumer responsibilities.
|
|
|
|
A single `DISCUSSION` section and other currently unmapped identities are
|
|
recognized as boundaries but are not forced into short- or long-term fields.
|
|
Exposing such content would require a separate canonical schema decision.
|
|
|
|
## Acceptance Criteria
|
|
|
|
The resilience follow-up is complete when automated tests demonstrate that:
|
|
|
|
- every previously accepted heading and canonical result remains compatible;
|
|
- generic structurally valid identities terminate preceding content without
|
|
becoming canonical fields;
|
|
- malformed heading-like lines and lowercase prose remain body content;
|
|
- identities containing `/` and slash qualifiers containing `/` are parsed
|
|
without ambiguity;
|
|
- next-line parenthesized qualifiers and following `Issued at` lines populate
|
|
the existing short- and long-term metadata correctly;
|
|
- exact NWS change markers and leading key-message timestamps do not leak into
|
|
canonical prose or messages;
|
|
- repeated mapped sections preserve first-occurrence behavior;
|
|
- genuine cross-office fixture styles propagate correctly through the NWS
|
|
normalizer with unchanged wire shape; and
|
|
- focused tests, the full repository suite, and static analysis pass.
|
|
|
|
## Non-Goals
|
|
|
|
This follow-up does not:
|
|
|
|
- add canonical `discussion`, aviation, marine, hydrology, climate, fire
|
|
weather, update, or arbitrary-section fields;
|
|
- infer short- or long-term semantics from an unknown heading;
|
|
- accept free-form or lowercase prose as a heading;
|
|
- introduce heuristic summarization;
|
|
- preserve complete raw AFD documents in canonical payloads;
|
|
- change schemas, persistence contracts, configuration, or downstream APIs; or
|
|
- fetch live NWS data during tests.
|
|
|
|
Canonical support for additional AFD section identities should be driven by a
|
|
separate consumer requirement and schema roadmap.
|