208 lines
8.7 KiB
Markdown
208 lines
8.7 KiB
Markdown
# NWS AFD Section Parsing Resilience
|
|
|
|
## Status
|
|
|
|
Implemented.
|
|
|
|
The original heading-variant work and both resilience follow-ups are
|
|
implemented without changing the canonical forecast-discussion contract.
|
|
|
|
## Completed Baseline
|
|
|
|
The NWS Area Forecast Discussion parser currently separates:
|
|
|
|
1. generic structural heading recognition;
|
|
2. one-pass section-boundary scanning;
|
|
3. canonical section-role selection; and
|
|
4. section-preamble and presentation cleanup.
|
|
|
|
It recognizes generic uppercase identities in ellipsis-first and
|
|
slash-qualified headings, treats structurally recognized but unmapped sections
|
|
as boundaries, supports same-line and next-line qualifiers, removes exact NWS
|
|
change markers, and keeps the first occurrence of each mapped role. Provider and
|
|
normalizer tests cover two office-format families while preserving the existing
|
|
wire contract.
|
|
|
|
## Implemented Extensions
|
|
|
|
The parser now handles common AFD forms that were outside the completed
|
|
baseline:
|
|
|
|
- a qualifier may appear between the identity and terminal ellipsis, as in
|
|
`.DISCUSSION (Today through Thursday)...`;
|
|
- the same key-message concept may be headed `KEY POINTS` rather than `KEY
|
|
MESSAGES`;
|
|
- key-message items may use hyphens, asterisks, numeric markers such as `1)` or
|
|
`1.`, or unmarked paragraphs;
|
|
- a bare prefix check for `Issued at` or `Updated at` can misclassify ordinary
|
|
prose such as “Updated atmospheric conditions...” as metadata; and
|
|
- offices may publish semantically distinct `NEAR TERM`, `SHORT TERM`, and
|
|
`LONG TERM` sections, so wording flexibility cannot safely rely on treating
|
|
every similar identity as an alias.
|
|
|
|
These provider-local extensions preserve the architecture while strengthening
|
|
syntax, semantic-alias, and list-tokenization handling. Compact cross-office
|
|
fixtures exercise each added format family.
|
|
|
|
## Implementation Approach
|
|
|
|
Common, minor NWS presentation changes remain local and inexpensive to support
|
|
while strict structural recognition and the existing canonical schema are
|
|
preserved. Each observed heading form uses an isolated grammar helper, a true
|
|
synonym uses one role-registry entry, and each list marker uses one
|
|
marker-classifier case. None requires changes to the scanner, canonical model,
|
|
normalizer architecture, or downstream contracts.
|
|
|
|
## Implemented Behavior
|
|
|
|
### Heading syntax
|
|
|
|
The provider-local heading parser recognizes these three explicit families:
|
|
|
|
```text
|
|
.<IDENTITY>...<optional qualifier>
|
|
.<IDENTITY> (<nonempty qualifier>)...
|
|
.<IDENTITY> /<nonempty qualifier>/...
|
|
```
|
|
|
|
The existing identity grammar remains unchanged: uppercase ASCII letters,
|
|
digits, horizontal whitespace, `/`, `&`, apostrophes, and hyphens, with at least
|
|
one letter or digit. Identity whitespace is normalized to one ASCII space.
|
|
|
|
The parenthesized-terminal form:
|
|
|
|
- requires horizontal whitespace between the identity and opening `(`;
|
|
- requires a nonempty standalone parenthetical qualifier immediately before
|
|
the terminal ASCII `...`;
|
|
- retains the outer parentheses in the parsed qualifier;
|
|
- permits ordinary qualifier punctuation inside the parentheses; and
|
|
- rejects trailing text after the terminal ellipsis.
|
|
|
|
Each family is parsed by a small, ordered helper. Slash-qualified parsing remains
|
|
first because its terminal is otherwise ambiguous with the ellipsis-first form;
|
|
parenthesized-terminal parsing runs second, followed by the existing
|
|
ellipsis-first form. Malformed heading-like lines remain body content.
|
|
|
|
Every successfully parsed heading is a section boundary regardless of whether
|
|
its identity has a canonical role. This includes `DISCUSSION`, `NEAR TERM`,
|
|
aviation, marine, hydrology, office-specific sections, and other structurally
|
|
compatible identities.
|
|
|
|
### Canonical role aliases
|
|
|
|
One provider-local identity-to-role registry remains the sole semantic mapping
|
|
source. It maps:
|
|
|
|
- `KEY MESSAGES` and `KEY POINTS` to key messages;
|
|
- `SHORT TERM` to the short-term section; and
|
|
- `LONG TERM` to the long-term section.
|
|
|
|
Aliases are explicit and evidence-based; identity similarity is never inferred.
|
|
The first encountered identity for a role wins, so `KEY MESSAGES` and `KEY
|
|
POINTS` participate in the same first-occurrence policy.
|
|
|
|
`NEAR TERM` remains boundary-only. It is a distinct section that may coexist
|
|
with `SHORT TERM`, so silently relabeling it would lose meaning. Exposing it
|
|
requires a separate canonical schema roadmap. `DISCUSSION` and other unmapped
|
|
sections likewise remain boundary-only.
|
|
|
|
### Key-message normalization
|
|
|
|
After exact presentation-marker removal and blank-line trimming, the parser
|
|
removes at most one leading metadata line only when all of these conditions are
|
|
true:
|
|
|
|
- the label is exactly `Issued at` or `Updated at`, compared ASCII
|
|
case-insensitively;
|
|
- the label is followed by horizontal whitespace; and
|
|
- the remainder is a valid timestamp under the existing NWS issue-time grammar.
|
|
|
|
An invalid or merely prefix-matching line remains content. Metadata recognition
|
|
does not consume later message prose.
|
|
|
|
Key-message item recognition supports:
|
|
|
|
- hyphen markers;
|
|
- asterisk markers;
|
|
- positive ASCII numeric markers followed by `)` or `.`, then either end of line
|
|
or a horizontal-space boundary;
|
|
- composite hyphen-or-asterisk plus numeric markers such as `- 1.`; and
|
|
- unmarked, blank-line-separated paragraphs when the block contains no
|
|
recognized list marker.
|
|
|
|
Marker text is removed from canonical messages. Wrapped nonempty lines remain
|
|
continuations of the current item, and blank lines between marked items do not
|
|
create empty messages. If marked and unmarked content are mixed, nonempty prose
|
|
before the first marker is preserved as its own message and later unmarked lines
|
|
continue the active marked item. The parser never interprets numeric or
|
|
asterisk markers outside a mapped key-message block.
|
|
|
|
### Representative coverage
|
|
|
|
Tests combine small table-driven grammar and tokenizer cases with maintained
|
|
local HTML fixtures for distinct real NWS format families. In addition to the
|
|
existing LSX and BOU coverage, fixtures cover:
|
|
|
|
- numbered `KEY MESSAGES` followed by a boundary-only `DISCUSSION` section; and
|
|
- asterisk `KEY POINTS` plus a parenthesized-terminal `DISCUSSION` heading.
|
|
|
|
Fixtures remain compact, deterministic, attributable in test comments to the
|
|
format family they represent, and free of live network dependencies. Provider
|
|
and normalizer tests prove canonical values, boundary isolation, envelope
|
|
behavior, and unchanged JSON wire shape.
|
|
|
|
## Compatibility and Contracts
|
|
|
|
This remains a provider-parsing compatibility improvement. It does not change:
|
|
|
|
- event kinds or raw and canonical schema identifiers;
|
|
- canonical models or JSON field names;
|
|
- source configuration, URLs, or polling behavior;
|
|
- event envelope or effective-time behavior;
|
|
- Postgres tables or event-to-row mapping; or
|
|
- downstream sink and consumer responsibilities.
|
|
|
|
The implementation remains confined to the NWS provider parser plus owning
|
|
provider and normalizer tests. It uses the Go standard library and introduces no
|
|
runtime configuration or general parser framework.
|
|
|
|
## Acceptance Coverage
|
|
|
|
Automated tests demonstrate that:
|
|
|
|
- all previously supported heading, scanning, preamble, marker, and canonical
|
|
results remain compatible;
|
|
- parenthesized-terminal headings are parsed with their qualifier and terminate
|
|
preceding content;
|
|
- malformed variants of that heading family remain body content;
|
|
- `KEY POINTS` populates the existing key-message field and shares
|
|
first-occurrence behavior with `KEY MESSAGES`;
|
|
- `NEAR TERM` and `DISCUSSION` remain boundary-only and never populate short- or
|
|
long-term fields;
|
|
- hyphen, asterisk, `N)`, `N.`, and composite key-message lists produce distinct
|
|
ordered messages with wrapped continuations;
|
|
- unmarked key-message paragraphs produce distinct ordered messages;
|
|
- valid leading issue/update metadata is removed, while prefix collisions,
|
|
malformed timestamps, and later timestamp-like prose are retained;
|
|
- representative additional office-format fixtures parse end to end through the
|
|
provider and normalizer without adjacent-section leakage; and
|
|
- focused tests, the full repository suite, static analysis, and diff checks
|
|
pass with no public contract changes.
|
|
|
|
## Non-Goals
|
|
|
|
This follow-up does not:
|
|
|
|
- add canonical `nearTerm`, `discussion`, aviation, marine, hydrology, climate,
|
|
fire-weather, update, or arbitrary-section fields;
|
|
- map `NEAR TERM` to `SHORT TERM` or infer roles from similar words;
|
|
- parse arbitrary lowercase or free-form prose as headings;
|
|
- accept arbitrary punctuation as list markers;
|
|
- remove malformed metadata-like prose;
|
|
- introduce heuristic summarization or preserve complete raw AFD documents;
|
|
- change schemas, persistence contracts, configuration, or downstream APIs; or
|
|
- fetch live NWS data during tests.
|
|
|
|
Additional canonical support for AFD section identities remains a separate
|
|
consumer and schema decision.
|