weatherfeeder/docs/roadmap/afd-section-heading-variants.md

NWS AFD Section Parsing Resilience

Status

The original ellipsis-first and slash-qualified heading feature is implemented. The resilience follow-up defined below is proposed and unimplemented.

Completed Baseline

The NWS Area Forecast Discussion parser now recognizes these heading families for key messages, short term, long term, and aviation:

.<SECTION>...<optional qualifier>
.<SECTION> /<qualifier>/...

Discovery, qualifier extraction, and recognized-section boundary detection use one provider-specific parser. Slash delimiters are removed from qualifiers, legacy qualifier text is preserved, and aviation remains a boundary rather than a canonical field. Focused provider and normalizer regressions cover the implemented forms.

Remaining Problem

Current NWS bulletins vary beyond those two same-line forms. In particular:

• an ellipsis-first heading may put its qualifier on the next nonblank line;
• NWS presentation output can place -- Changed Discussion -- markers and an Updated at line around section content;
• valid AFDs can include DISCUSSION, UPDATE, MARINE, HYDROLOGY, CLIMATE, FIRE WEATHER, office watch/advisory blocks, and other topic headings; and
• minor punctuation or qualifier changes can produce headings that are structurally valid but absent from the parser's identity whitelist.

The current parser recognizes only four exact identities. Unknown headings do not terminate the preceding section. It also expects a text section's optional qualifier and Issued at line in a narrow order. Consequently, valid upstream format variants can leave qualifier and section-time fields empty, leak presentation metadata into prose or key messages, or allow one section to absorb another.

Feature Objective

Make AFD parsing resilient to minor upstream format evolution by separating:

1. generic structural heading recognition;
2. section boundary scanning;
3. canonical section-role selection; and
4. section-preamble and presentation cleanup.

The parser should accept new structurally valid topic names as safe boundaries without treating them as new canonical fields.

Targeted End State

Generic heading recognition

One NWS-specific heading parser recognizes structurally valid uppercase AFD topic lines independently of the canonical model. It supports:

.<IDENTITY>...<optional qualifier>
.<IDENTITY> /<qualifier>/...

The identity may contain uppercase ASCII letters, digits, horizontal whitespace, /, &, apostrophes, and hyphens. Outer whitespace and whitespace immediately before the ellipsis are ignored, and internal identity whitespace is collapsed to one space for role lookup. Slash-qualified headings require a nonempty qualifier but may contain slash characters inside that qualifier. Lowercase prose, ordinary dotted lines, and malformed delimiters remain non-headings.

Every structurally valid heading terminates the preceding section. This applies to known boundary-only sections such as aviation and to future or office-specific topic identities that weatherfeeder does not map.

Canonical role selection

A single provider-local role registry maps only these identities into existing parsed fields:

• KEY MESSAGES to key messages;
• SHORT TERM to the short-term section; and
• LONG TERM to the long-term section.

All other identities are boundary-only. The first occurrence of each mapped identity wins, preserving current behavior if an unusual bulletin repeats a section. Heading syntax and canonical roles do not duplicate identity lists.

Section scanning

The discussion text is scanned once into ordered section blocks. A new heading, &&, $$, or the existing watch/advisory terminator ends the active block. Original body lines are preserved until provider presentation cleanup is applied. Preamble text before the first heading and post-$$ signatures are not treated as section content.

Preamble and presentation handling

For short- and long-term sections:

• a qualifier on the heading line remains authoritative;
• when the heading has no qualifier, a standalone parenthesized first content line becomes the qualifier and retains its parentheses;
• an optional case-insensitive Issued at line after the qualifier is parsed into the existing section issue time; and
• exact NWS change-presentation marker lines are removed without removing arbitrary dashed prose.

For key messages, exact change-presentation markers and one leading Issued at or Updated at metadata line are removed before bullet parsing. Those metadata lines never become key messages.

Representative coverage

Tests include small structural tables and maintained local HTML fixtures for at least two real NWS formatting families: slash-qualified same-line headings and ellipsis-first headings with multiline qualifiers or change-presentation markers. Tests remain deterministic and never contact live services.

Compatibility and Contracts

This remains a provider-parsing compatibility improvement. It does not change:

• event kinds or raw and canonical schema identifiers;
• canonical models or JSON field names;
• source configuration, URLs, or polling behavior;
• event envelope or effective-time behavior;
• Postgres tables or event-to-row mapping; or
• downstream sink and consumer responsibilities.

A single DISCUSSION section and other currently unmapped identities are recognized as boundaries but are not forced into short- or long-term fields. Exposing such content would require a separate canonical schema decision.

Acceptance Criteria

The resilience follow-up is complete when automated tests demonstrate that:

• every previously accepted heading and canonical result remains compatible;
• generic structurally valid identities terminate preceding content without becoming canonical fields;
• malformed heading-like lines and lowercase prose remain body content;
• identities containing / and slash qualifiers containing / are parsed without ambiguity;
• next-line parenthesized qualifiers and following Issued at lines populate the existing short- and long-term metadata correctly;
• exact NWS change markers and leading key-message timestamps do not leak into canonical prose or messages;
• repeated mapped sections preserve first-occurrence behavior;
• genuine cross-office fixture styles propagate correctly through the NWS normalizer with unchanged wire shape; and
• focused tests, the full repository suite, and static analysis pass.

Non-Goals

This follow-up does not:

• add canonical discussion, aviation, marine, hydrology, climate, fire weather, update, or arbitrary-section fields;
• infer short- or long-term semantics from an unknown heading;
• accept free-form or lowercase prose as a heading;
• introduce heuristic summarization;
• preserve complete raw AFD documents in canonical payloads;
• change schemas, persistence contracts, configuration, or downstream APIs; or
• fetch live NWS data during tests.

Canonical support for additional AFD section identities should be driven by a separate consumer requirement and schema roadmap.