weatherfeeder/docs/roadmap/afd-section-heading-variants.md

NWS AFD Section Parsing Resilience

Status

Implemented.

The original heading-variant work and both resilience follow-ups are implemented without changing the canonical forecast-discussion contract.

Completed Baseline

The NWS Area Forecast Discussion parser currently separates:

1. generic structural heading recognition;
2. one-pass section-boundary scanning;
3. canonical section-role selection; and
4. section-preamble and presentation cleanup.

It recognizes generic uppercase identities in ellipsis-first and slash-qualified headings, treats structurally recognized but unmapped sections as boundaries, supports same-line and next-line qualifiers, removes exact NWS change markers, and keeps the first occurrence of each mapped role. Provider and normalizer tests cover two office-format families while preserving the existing wire contract.

Implemented Extensions

The parser now handles common AFD forms that were outside the completed baseline:

• a qualifier may appear between the identity and terminal ellipsis, as in .DISCUSSION (Today through Thursday)...;
• the same key-message concept may be headed KEY POINTS rather than KEY MESSAGES;
• key-message items may use hyphens, asterisks, numeric markers such as 1) or 1., or unmarked paragraphs;
• a bare prefix check for Issued at or Updated at can misclassify ordinary prose such as “Updated atmospheric conditions...” as metadata; and
• offices may publish semantically distinct NEAR TERM, SHORT TERM, and LONG TERM sections, so wording flexibility cannot safely rely on treating every similar identity as an alias.

These provider-local extensions preserve the architecture while strengthening syntax, semantic-alias, and list-tokenization handling. Compact cross-office fixtures exercise each added format family.

Implementation Approach

Common, minor NWS presentation changes remain local and inexpensive to support while strict structural recognition and the existing canonical schema are preserved. Each observed heading form uses an isolated grammar helper, a true synonym uses one role-registry entry, and each list marker uses one marker-classifier case. None requires changes to the scanner, canonical model, normalizer architecture, or downstream contracts.

Implemented Behavior

Heading syntax

The provider-local heading parser recognizes these three explicit families:

.<IDENTITY>...<optional qualifier>
.<IDENTITY> (<nonempty qualifier>)...
.<IDENTITY> /<nonempty qualifier>/...

The existing identity grammar remains unchanged: uppercase ASCII letters, digits, horizontal whitespace, /, &, apostrophes, and hyphens, with at least one letter or digit. Identity whitespace is normalized to one ASCII space.

The parenthesized-terminal form:

• requires horizontal whitespace between the identity and opening (;
• requires a nonempty standalone parenthetical qualifier immediately before the terminal ASCII ...;
• retains the outer parentheses in the parsed qualifier;
• permits ordinary qualifier punctuation inside the parentheses; and
• rejects trailing text after the terminal ellipsis.

Each family is parsed by a small, ordered helper. Slash-qualified parsing remains first because its terminal is otherwise ambiguous with the ellipsis-first form; parenthesized-terminal parsing runs second, followed by the existing ellipsis-first form. Malformed heading-like lines remain body content.

Every successfully parsed heading is a section boundary regardless of whether its identity has a canonical role. This includes DISCUSSION, NEAR TERM, aviation, marine, hydrology, office-specific sections, and other structurally compatible identities.

Canonical role aliases

One provider-local identity-to-role registry remains the sole semantic mapping source. It maps:

• KEY MESSAGES and KEY POINTS to key messages;
• SHORT TERM to the short-term section; and
• LONG TERM to the long-term section.

Aliases are explicit and evidence-based; identity similarity is never inferred. The first encountered identity for a role wins, so KEY MESSAGES and KEY POINTS participate in the same first-occurrence policy.

NEAR TERM remains boundary-only. It is a distinct section that may coexist with SHORT TERM, so silently relabeling it would lose meaning. Exposing it requires a separate canonical schema roadmap. DISCUSSION and other unmapped sections likewise remain boundary-only.

Key-message normalization

After exact presentation-marker removal and blank-line trimming, the parser removes at most one leading metadata line only when all of these conditions are true:

• the label is exactly Issued at or Updated at, compared ASCII case-insensitively;
• the label is followed by horizontal whitespace; and
• the remainder is a valid timestamp under the existing NWS issue-time grammar.

An invalid or merely prefix-matching line remains content. Metadata recognition does not consume later message prose.

Key-message item recognition supports:

• hyphen markers;
• asterisk markers;
• positive ASCII numeric markers followed by ) or ., then either end of line or a horizontal-space boundary;
• composite hyphen-or-asterisk plus numeric markers such as - 1.; and
• unmarked, blank-line-separated paragraphs when the block contains no recognized list marker.

Marker text is removed from canonical messages. Wrapped nonempty lines remain continuations of the current item, and blank lines between marked items do not create empty messages. If marked and unmarked content are mixed, nonempty prose before the first marker is preserved as its own message and later unmarked lines continue the active marked item. The parser never interprets numeric or asterisk markers outside a mapped key-message block.

Representative coverage

Tests combine small table-driven grammar and tokenizer cases with maintained local HTML fixtures for distinct real NWS format families. In addition to the existing LSX and BOU coverage, fixtures cover:

• numbered KEY MESSAGES followed by a boundary-only DISCUSSION section; and
• asterisk KEY POINTS plus a parenthesized-terminal DISCUSSION heading.

Fixtures remain compact, deterministic, attributable in test comments to the format family they represent, and free of live network dependencies. Provider and normalizer tests prove canonical values, boundary isolation, envelope behavior, and unchanged JSON wire shape.

Compatibility and Contracts

This remains a provider-parsing compatibility improvement. It does not change:

• event kinds or raw and canonical schema identifiers;
• canonical models or JSON field names;
• source configuration, URLs, or polling behavior;
• event envelope or effective-time behavior;
• Postgres tables or event-to-row mapping; or
• downstream sink and consumer responsibilities.

The implementation remains confined to the NWS provider parser plus owning provider and normalizer tests. It uses the Go standard library and introduces no runtime configuration or general parser framework.

Acceptance Coverage

Automated tests demonstrate that:

• all previously supported heading, scanning, preamble, marker, and canonical results remain compatible;
• parenthesized-terminal headings are parsed with their qualifier and terminate preceding content;
• malformed variants of that heading family remain body content;
• KEY POINTS populates the existing key-message field and shares first-occurrence behavior with KEY MESSAGES;
• NEAR TERM and DISCUSSION remain boundary-only and never populate short- or long-term fields;
• hyphen, asterisk, N), N., and composite key-message lists produce distinct ordered messages with wrapped continuations;
• unmarked key-message paragraphs produce distinct ordered messages;
• valid leading issue/update metadata is removed, while prefix collisions, malformed timestamps, and later timestamp-like prose are retained;
• representative additional office-format fixtures parse end to end through the provider and normalizer without adjacent-section leakage; and
• focused tests, the full repository suite, static analysis, and diff checks pass with no public contract changes.

Non-Goals

This follow-up does not:

• add canonical nearTerm, discussion, aviation, marine, hydrology, climate, fire-weather, update, or arbitrary-section fields;
• map NEAR TERM to SHORT TERM or infer roles from similar words;
• parse arbitrary lowercase or free-form prose as headings;
• accept arbitrary punctuation as list markers;
• remove malformed metadata-like prose;
• introduce heuristic summarization or preserve complete raw AFD documents;
• change schemas, persistence contracts, configuration, or downstream APIs; or
• fetch live NWS data during tests.

Additional canonical support for AFD section identities remains a separate consumer and schema decision.