ejr/weatherfeeder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Support parenthesized NWS discussion headings

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-03 00:13:58 +00:00
parent a62cb87b78
commit b8c6708439
4 changed files with 526 additions and 296 deletions
Download Patch File Download Diff File Expand all files Collapse all files

238
docs/roadmap/afd-section-heading-variants.md
View File

@@ -2,99 +2,154 @@
## Status
Implemented.
The original heading-variant work and the first resilience follow-up are
implemented. A second resilience follow-up is proposed to cover additional
observed NWS heading and key-message formats without changing the canonical
forecast-discussion contract.
## Implemented Behavior
## Completed Baseline
The NWS Area Forecast Discussion parser recognizes these heading families for
all structurally valid section identities:
```text
.<SECTION>...<optional qualifier>
.<SECTION> /<qualifier>/...
```
Discovery, qualifier extraction, and recognized-section boundary detection use
one provider-specific parser. Slash delimiters are removed from qualifiers,
legacy qualifier text is preserved, and aviation remains a boundary rather than
a canonical field.
The parser separates:
The NWS Area Forecast Discussion parser currently separates:
1. generic structural heading recognition;
2. section boundary scanning;
2. one-pass section-boundary scanning;
3. canonical section-role selection; and
4. section-preamble and presentation cleanup.
Structurally valid topic names act as safe boundaries without becoming new
canonical fields.
It recognizes generic uppercase identities in ellipsis-first and
slash-qualified headings, treats structurally recognized but unmapped sections
as boundaries, supports same-line and next-line qualifiers, removes exact NWS
change markers, and keeps the first occurrence of each mapped role. Provider and
normalizer tests cover two office-format families while preserving the existing
wire contract.
## Implemented Details
## Remaining Problem
### Generic heading recognition
Observed AFDs still expose common forms outside the completed baseline:
One NWS-specific heading parser recognizes structurally valid uppercase AFD
topic lines independently of the canonical model. It supports:
- a qualifier may appear between the identity and terminal ellipsis, as in
`.DISCUSSION (Today through Thursday)...`;
- the same key-message concept may be headed `KEY POINTS` rather than `KEY
MESSAGES`;
- key-message items may use hyphens, asterisks, numeric markers such as `1)` or
`1.`, or unmarked paragraphs;
- a bare prefix check for `Issued at` or `Updated at` can misclassify ordinary
prose such as “Updated atmospheric conditions...” as metadata; and
- offices may publish semantically distinct `NEAR TERM`, `SHORT TERM`, and
`LONG TERM` sections, so wording flexibility cannot safely rely on treating
every similar identity as an alias.
These gaps do not undermine the provider-local architecture, but they mean the
parser is still brittle at its syntax, semantic-alias, and list-tokenization
extension points. The current compact cross-office fixture does not exercise
these format families.
## Objective
Make common, minor NWS presentation changes local and inexpensive to support
while preserving strict structural recognition and the existing canonical
schema. A new observed heading form should require an isolated grammar helper;
a true synonym should require one role-registry entry; and a new list marker
should require one marker-classifier case. None should require changes to the
scanner, canonical model, normalizer architecture, or downstream contracts.
## Target End State
### Heading syntax
The provider-local heading parser recognizes these three explicit families:
```text
.<IDENTITY>...<optional qualifier>
.<IDENTITY> /<qualifier>/...
.<IDENTITY> (<nonempty qualifier>)...
.<IDENTITY> /<nonempty qualifier>/...
```
The identity may contain uppercase ASCII letters, digits, horizontal whitespace,
`/`, `&`, apostrophes, and hyphens. Outer whitespace and whitespace immediately
before the ellipsis are ignored, and internal identity whitespace is collapsed
to one space for role lookup. Slash-qualified headings require a nonempty
qualifier but may contain slash characters inside that qualifier. Lowercase
prose, ordinary dotted lines, and malformed delimiters remain non-headings.
The existing identity grammar remains unchanged: uppercase ASCII letters,
digits, horizontal whitespace, `/`, `&`, apostrophes, and hyphens, with at least
one letter or digit. Identity whitespace is normalized to one ASCII space.
Every structurally valid heading terminates the preceding section. This applies
to known boundary-only sections such as aviation and to future or office-specific
topic identities that weatherfeeder does not map.
The parenthesized-terminal form:
### Canonical role selection
- requires horizontal whitespace between the identity and opening `(`;
- requires a nonempty standalone parenthetical qualifier immediately before
the terminal ASCII `...`;
- retains the outer parentheses in the parsed qualifier;
- permits ordinary qualifier punctuation inside the parentheses; and
- rejects trailing text after the terminal ellipsis.
A single provider-local role registry maps only these identities into existing
parsed fields:
Each family is parsed by a small, ordered helper. Slash-qualified parsing remains
first because its terminal is otherwise ambiguous with the ellipsis-first form;
parenthesized-terminal parsing runs second, followed by the existing
ellipsis-first form. Malformed heading-like lines remain body content.
- `KEY MESSAGES` to key messages;
Every successfully parsed heading is a section boundary regardless of whether
its identity has a canonical role. This includes `DISCUSSION`, `NEAR TERM`,
aviation, marine, hydrology, office-specific sections, and future structurally
compatible identities.
### Canonical role aliases
One provider-local identity-to-role registry remains the sole semantic mapping
source. It maps:
- `KEY MESSAGES` and `KEY POINTS` to key messages;
- `SHORT TERM` to the short-term section; and
- `LONG TERM` to the long-term section.
All other identities are boundary-only. The first occurrence of each mapped
identity wins, preserving current behavior if an unusual bulletin repeats a
section. Heading syntax and canonical roles do not duplicate identity lists.
Aliases are explicit and evidence-based; identity similarity is never inferred.
The first encountered identity for a role wins, so `KEY MESSAGES` and `KEY
POINTS` participate in the same first-occurrence policy.
### Section scanning
`NEAR TERM` remains boundary-only. It is a distinct section that may coexist
with `SHORT TERM`, so silently relabeling it would lose meaning. Exposing it
requires a separate canonical schema roadmap. `DISCUSSION` and other unmapped
sections likewise remain boundary-only.
The discussion text is scanned once into ordered section blocks. A new heading,
`&&`, `$$`, or the existing watch/advisory terminator ends the active block.
Original body lines are preserved until provider presentation cleanup is
applied. Preamble text before the first heading and post-`$$` signatures are not
treated as section content.
### Key-message normalization
### Preamble and presentation handling
After exact presentation-marker removal and blank-line trimming, the parser
removes at most one leading metadata line only when all of these conditions are
true:
For short- and long-term sections:
- the label is exactly `Issued at` or `Updated at`, compared ASCII
case-insensitively;
- the label is followed by horizontal whitespace; and
- the remainder is a valid timestamp under the existing NWS issue-time grammar.
- a qualifier on the heading line remains authoritative;
- when the heading has no qualifier, a standalone parenthesized first content
line becomes the qualifier and retains its parentheses;
- an optional case-insensitive `Issued at` line after the qualifier is parsed
into the existing section issue time; and
- exact NWS change-presentation marker lines are removed without removing
arbitrary dashed prose.
An invalid or merely prefix-matching line remains content. Metadata recognition
does not consume later message prose.
For key messages, exact change-presentation markers and one leading `Issued at`
or `Updated at` metadata line are removed before bullet parsing. Those metadata
lines never become key messages.
Key-message item recognition supports:
- hyphen markers;
- asterisk markers;
- positive ASCII numeric markers followed by `)` or `.`, then either end of line
or a horizontal-space boundary;
- composite hyphen-or-asterisk plus numeric markers such as `- 1.`; and
- unmarked, blank-line-separated paragraphs when the block contains no
recognized list marker.
Marker text is removed from canonical messages. Wrapped nonempty lines remain
continuations of the current item, and blank lines between marked items do not
create empty messages. If marked and unmarked content are mixed, nonempty prose
before the first marker is preserved as its own message and later unmarked lines
continue the active marked item. The parser never interprets numeric or
asterisk markers outside a mapped key-message block.
### Representative coverage
Tests include small structural tables and maintained local HTML fixtures for at
least two real NWS formatting families: slash-qualified same-line headings and
ellipsis-first headings with multiline qualifiers or change-presentation
markers. Tests remain deterministic and never contact live services.
Tests combine small table-driven grammar and tokenizer cases with maintained
local HTML fixtures for distinct real NWS format families. In addition to the
existing LSX and BOU coverage, fixtures cover:
- numbered `KEY MESSAGES` followed by a boundary-only `DISCUSSION` section; and
- asterisk `KEY POINTS` plus a parenthesized-terminal `DISCUSSION` heading.
Fixtures remain compact, deterministic, attributable in test comments to the
format family they represent, and free of live network dependencies. Provider
and normalizer tests prove canonical values, boundary isolation, envelope
behavior, and unchanged JSON wire shape.
## Compatibility and Contracts
@@ -107,41 +162,46 @@ This remains a provider-parsing compatibility improvement. It does not change:
- Postgres tables or event-to-row mapping; or
- downstream sink and consumer responsibilities.
A single `DISCUSSION` section and other currently unmapped identities are
recognized as boundaries but are not forced into short- or long-term fields.
Exposing such content would require a separate canonical schema decision.
The implementation remains confined to the NWS provider parser plus owning
provider and normalizer tests. It uses the Go standard library and introduces no
runtime configuration or general parser framework.
## Verified Coverage
## Acceptance Criteria
Automated tests demonstrate that:
The follow-up is complete when automated tests demonstrate that:
- every previously accepted heading and canonical result remains compatible;
- generic structurally valid identities terminate preceding content without
becoming canonical fields;
- malformed heading-like lines and lowercase prose remain body content;
- identities containing `/` and slash qualifiers containing `/` are parsed
without ambiguity;
- next-line parenthesized qualifiers and following `Issued at` lines populate
the existing short- and long-term metadata correctly;
- exact NWS change markers and leading key-message timestamps do not leak into
canonical prose or messages;
- repeated mapped sections preserve first-occurrence behavior;
- genuine cross-office fixture styles propagate correctly through the NWS
normalizer with unchanged wire shape; and
- focused tests, the full repository suite, and static analysis pass.
- all previously supported heading, scanning, preamble, marker, and canonical
results remain compatible;
- parenthesized-terminal headings are parsed with their qualifier and terminate
preceding content;
- malformed variants of that heading family remain body content;
- `KEY POINTS` populates the existing key-message field and shares
first-occurrence behavior with `KEY MESSAGES`;
- `NEAR TERM` and `DISCUSSION` remain boundary-only and never populate short- or
long-term fields;
- hyphen, asterisk, `N)`, `N.`, and composite key-message lists produce distinct
ordered messages with wrapped continuations;
- unmarked key-message paragraphs produce distinct ordered messages;
- valid leading issue/update metadata is removed, while prefix collisions,
malformed timestamps, and later timestamp-like prose are retained;
- representative additional office-format fixtures parse end to end through the
provider and normalizer without adjacent-section leakage; and
- focused tests, the full repository suite, static analysis, and diff checks
pass with no public contract changes.
## Non-Goals
This follow-up does not:
- add canonical `discussion`, aviation, marine, hydrology, climate, fire
weather, update, or arbitrary-section fields;
- infer short- or long-term semantics from an unknown heading;
- accept free-form or lowercase prose as a heading;
- introduce heuristic summarization;
- preserve complete raw AFD documents in canonical payloads;
- add canonical `nearTerm`, `discussion`, aviation, marine, hydrology, climate,
fire-weather, update, or arbitrary-section fields;
- map `NEAR TERM` to `SHORT TERM` or infer roles from similar words;
- parse arbitrary lowercase or free-form prose as headings;
- accept arbitrary punctuation as list markers;
- remove malformed metadata-like prose;
- introduce heuristic summarization or preserve complete raw AFD documents;
- change schemas, persistence contracts, configuration, or downstream APIs; or
- fetch live NWS data during tests.
Additional canonical support for AFD section identities requires a separate
consumer requirement and schema roadmap.
Additional canonical support for AFD section identities remains a separate
consumer and schema decision.