notarius/docs/roadmap/archive/warning-signal-and-presentation-audit.md

Warning Signal And Presentation Audit

Executive Assessment

Notarius warning execution is mechanically stronger than its operator-facing presentation. Terminal-attempt promotion, stable ordering after concurrent work, checkpoint replay, validation summaries, and debug retention are all substantially correct. The audit found no general duplicate-append defect in the extract, merge, or normalize handoffs and no leakage of abandoned-attempt warnings into a successful result.

The warning channel itself is not coherent. One flat contracts.Warning type currently represents at least four materially different concepts:

• actionable degradation or incomplete validation;
• heuristic data-quality doubt;
• successful but potentially reviewable fallback; and
• routine canonicalization, ordering, and deduplication observations.

That conflation is the primary reason successful runs produce a count that is large but operationally weak. The maintained complete D&D example demonstrates the problem without a live provider: an approved run with no rejected outputs published 12 warning records, all from three advisory relatedness checks. An operator separately reported a successful complete D&D run with 10 outputs, one rejection, and 85 warnings. The production bundle for that run was not available in this environment, so its reason-code distribution could not be measured.

The current implementation also has four correctness or robustness gaps:

1. warning records lose stage, step, lane, module, validator, and chunk provenance when promoted, which makes safe aggregation and diagnosis impossible from warnings.json alone;
2. there is no framework-level validation or aggregate bound, and the NPC- and spell-relatedness validators bypass the D&D warning limiter entirely;
3. warnings returned by an output encoder are added after warnings.json has already been encoded, so the receipt, stderr, debug bundle, and published warning file can disagree; and
4. a skipped validator contributes to incomplete validation but does not receive the warning generated for an exhausted validator failure.

The recommended end state is a structured diagnostic contract with explicit disposition, category, origin, occurrence count, and bounded samples. Warnings are reserved for process-level degradation or incompleteness. LLM-judged or deterministically inferred extraction-quality signals are advisories, never warnings, and routine normalization observations remain inspectable without being reported as top-level warnings. An ordinary successful run in which all configured work completes normally should therefore report zero warnings. This is an architectural and durable-contract change, not merely revised CLI prose.

Evidence And Limits

The audit used:

• a complete static search of production contracts.Warning constructors, reason-code constants, result fields, and promotion sites under internal/;
• call-path inspection through producer attempts, validators, lane coordination, chunk-plan reuse, checkpoints, output encoding, debug output, CLI presentation, and run-result construction;
• the maintained complete and minimal D&D examples with offline fake LLMs;
• focused deterministic tests for warning bounds, semantic-reconciliation fallback, warn_continue, semantic retries, terminal rejection, concurrency ordering, and checkpoint reuse; and
• the operator-provided observation of an 85-warning complete D&D run.

No provider-backed production run was attempted because this environment has no API key. Consequently, the audit can establish warning mechanics, possible multiplicity, synthetic volume, and obvious heuristic limitations, but cannot estimate production frequency or the real false-positive rate of individual D&D advisories. Those measurements are not required to choose the recommended architecture; they are required before strengthening any heuristic advisory into a rejection or setting a numerical production acceptance target.

Complete Warning-Producer Inventory

Framework And Generic Boundaries

Producer	Reason code	Trigger and consequence	Multiplicity and bound	Current surfaces and coverage
Reference materialization in internal/framework/pipeline/references.go	empty_reference	A bound external reference is a valid, accepted media type but contains zero bytes. The prompt may receive materially incomplete context.	One per empty bound file; finite by configuration but no shared run-level cap.	Enters RunInput.Warnings; reference tests protect contextual scope.
Producer-attempt policy in internal/framework/pipeline/producer_attempts.go	validator_execution_incomplete	An applicable validator exhausted its execution budget and warn_continue accepted the otherwise valid candidate.	One per failed validator on each terminal candidate. An extract chain can multiply this by chunks and lanes. There is no global cap.	Durable warning, receipt count, stderr, debug, and validation summary. TestWarnContinueRecordsOneWarningForEachExhaustedValidator covers failures.
Chunk, extract, merge, normalize, and output module result contracts	Module-defined	A module may return arbitrary warnings with its successful candidate.	No contract validation, message limit, per-result cap, or global cap. Current production modules are inventoried below.	Terminal-attempt filtering and concurrency ordering are well tested.
Production JSON output encoder	None	The encoder copies incoming warnings into warnings.json; it does not currently create warnings.	Same count as its input.	JSON encoder and assembled-pipeline tests compare the incoming run warnings with the published file.
Output encoder result contract	Module-defined	Any output encoder may return warnings discovered during encoding.	Unbounded by contract. No production encoder currently exercises this capability.	Appended to final RunOutput only after logical files were encoded; this is the cross-surface defect described in AUD-WARN-004.

Input adapters and production mergers do not currently have independent warning producers. Chunk-plan and checkpoint decisions are structured manifest or debug provenance rather than warnings. Cancellation and hard persistence, reference, parsing, serialization, and provider failures remain errors.

D&D Extraction Gates

Producer	Reason code	Trigger and consequence	Multiplicity and bound
dnd/combat-turns extractor	scene_classification_unavailable	The chunk has no exact matching scene-description classification. The extractor returns an empty accepted result and skips the LLM, so combat-turn output may be incomplete.	At most one per chunk for this lane.
dnd/enemy-events extractor	scene_classification_unavailable	The same missing or mismatched scene gate causes accepted empty enemy-event output.	At most one per chunk for this lane.

An exact non-combat classification produces an intentional empty result without a warning. An exact combat classification proceeds normally. The two producers share a code and operator consequence but use different messages; their module origins are not retained in the final warning record.

D&D Source-Relatedness Validators

All ten relatedness validators are deterministic advisories: they approve the candidate and warn when contextual prose or an entity name is not lexically present in cited text. Shape and source-reference failures are deliberately left to blocking validators earlier in the chain. The same relatedness validator is registered in both the extract and normalize default chain for each artifact family in internal/modules/dnd/register/chains.go.

Artifact family	Warning reason	Per-record trigger	Local bound	Omission reason
Combat turns	combat_turn_not_near_source	Actor token sequence absent	20 per validator invocation	combat_turn_relatedness_warnings_omitted
Enemy events	enemy_event_not_near_source	Subject token sequence absent	20	enemy_event_relatedness_warnings_omitted
Item occurrences	item_occurrence_source_unrelated	Item name token sequence absent	20	item_occurrence_relatedness_warnings_omitted
Item registry	item_not_near_source	Item name token sequence absent	20	item_relatedness_warnings_omitted
Location occurrences	location_occurrence_not_near_source	Location name token sequence absent	20	location_occurrence_relatedness_warnings_omitted
Location registry	location_not_near_source	Location name token sequence absent	20	location_relatedness_warnings_omitted
NPC occurrences	npc_occurrence_not_near_source	NPC name token sequence absent	20	npc_occurrence_relatedness_warnings_omitted
NPC registry	npc_not_near_source	NPC name token sequence absent	Unbounded	None
Scene descriptions	scene_description_not_near_source	No significant title or summary token appears; up to two findings per scene	20	scene_description_relatedness_warnings_omitted
Spells	spell_not_near_source	Spell-name token sequence absent	Unbounded	None

The eight limiter-generated omission records are presentation artifacts, not new source-relatedness conditions. They occupy a warning slot and make list length differ from the actual occurrence count.

D&D Normalizers

Every production D&D normalizer bounds its returned warning slice to 20 through internal/modules/dnd/shared/diagnostics, including a final omission record when needed. Registry semantic retries reserve space for their fallback warning. The following table is complete by semantically distinct condition; codes listed together are parallel artifact-family variants.

Condition	Reason codes	Result impact	Current classification assessment
Display or field whitespace/name canonicalization	npc_fields_normalized, item_fields_normalized, location_fields_normalized, spell_name_canonicalized, combat_actor_canonicalized, enemy_event_name_canonicalized, item_occurrence_name_canonicalized, location_occurrence_name_canonicalized, scene_description_prose_normalized	Deterministic successful mutation. The item-occurrence code can also describe from/to whitespace, not only the item name.	Routine observation.
Durable ID recomputation	npc_id_recomputed, item_id_recomputed, location_id_recomputed	Restores the deterministic name-derived ID.	Routine observation; invalid identity is separately rejected by default chains.
Source-reference sorting or deduplication	source_references_normalized	Sorts and removes exact duplicate references while deliberately preserving invalid references for their validators.	Routine observation. Shared code is useful but ambiguous without producer origin.
Canonical record ordering	combat_turns_reordered, enemy_events_reordered, item_occurrences_reordered, location_occurrences_reordered, npc_occurrences_reordered, scene_description_order_normalized	Deterministic order changes only.	Routine observation.
Exact or approved semantic duplicate consolidation	duplicate_npc_collapsed, duplicate_item_collapsed, duplicate_location_collapsed, duplicate_spell_cast_collapsed, duplicate_combat_turn_collapsed, duplicate_enemy_event_collapsed, duplicate_item_occurrence_collapsed, duplicate_location_occurrence_collapsed, duplicate_npc_occurrence_collapsed, scene_description_duplicate_collapsed	Removes duplicate records and preserves or combines canonical evidence according to the artifact policy. Registry codes cover both exact and accepted semantic consolidation.	Durable normalization observation; not normally operator-actionable.
Unresolved external membership	spell_name_unresolved, item_occurrence_unknown_item_id, location_occurrence_unknown_location_id	The value is preserved but is not grounded in the effective catalog or registry. Default chains normally reject the same condition before normalization; it remains reachable with validator overrides or defensive direct use.	Actionable data-quality warning.
Unsafe currency consolidation proposal	item_semantic_proposal_invalid	The proposed group is rejected and all records are preserved because denominations or currency/non-currency members are incompatible. The same code is also used internally as a retry reason.	Advisory about model proposal quality; no accepted-data loss. The control and diagnostic meanings should be separated.
Semantic reconciliation unavailable or exhausted	npc_semantic_reconciliation_exhausted, item_semantic_reconciliation_exhausted, location_semantic_reconciliation_exhausted	The safe deterministic result is accepted, but possible semantic duplicates remain.	Actionable fallback warning.
Local warning truncation	npc_normalization_warnings_omitted, item_normalization_warnings_omitted, location_normalization_warnings_omitted, spell_normalization_warnings_omitted, combat_turn_normalization_warnings_omitted, enemy_event_normalization_warnings_omitted, item_occurrence_normalization_warnings_omitted, location_occurrence_normalization_warnings_omitted, npc_occurrence_normalization_warnings_omitted, scene_description_normalization_warnings_omitted	Reports that individual records were omitted from presentation.	Group metadata, not an independent warning.

No production D&D normalization warning exposes raw model responses, correction guidance, or provider errors. Most dynamic names are quoted and truncated by the shared helper. That local discipline is not enforced by the generic warning contract, and the spell relatedness message does not use the shared truncation helper.

Warning Propagation And Surface Map

external-reference warnings -----------------------------+
                                                           |
module candidate warnings -> validation chain warnings     |
             |                      |                       |
             +---- producer-attempt terminal policy -------+
                                  |                        |
                   accepted / terminal rejection only      |
                                  |                        |
              chunk or lane result in canonical order      |
                                  |                        |
       checkpoint record/replay and ordered step merge     |
                                  |                        |
                            RunOutput.Warnings <------------+
                                  |
                   OutputRequest -> output encoder
                                  |             |
                         warnings.json     OutputResult.Warnings
                                                |
                              appended to final RunOutput only
                                  |
                 receipt, stderr, final debug warning summary

Attempts And Validation

• runProducerAttempts promotes only the terminal accepted or terminal rejected candidate's module and completed-validator warnings. Operational, structural, semantic, and module-directed attempts that are superseded are retained in attempt debug artifacts but not in the final collection.
• A module-directed semantic-reconciliation retry adds its fallback warning only when no retry remains. Earlier attempt warnings are discarded.
• On warn_continue, warnings from the otherwise accepted candidate and completed approved or rejected validators are retained. One fixed, non-sensitive validator_execution_incomplete warning is added for every failed validator. Skipped validators affect the validation summary and final incomplete status but do not receive such a warning.
• A semantic terminal rejection retains only warnings from that rejected attempt. Structural rejection after producer failure cannot retain a candidate warning because no valid candidate result exists.

These behaviors are protected by the producer-attempt, extract-handoff, rejection-warning, normalize-retry, and attempt-debug tests. They are the right foundation for the redesign and should not be replaced with early-exit or all-attempt accumulation.

Concurrency And Ordering

Extract jobs are dispatched chunk-first and lane-second. Results are stored by chunk index, finalized in ascending chunk order, and lane continuations are merged into a slice indexed by configured lane order. Pipeline steps run in configured order. The resulting public order is therefore:

1. pre-run reference warnings;
2. chunk-stage warnings;
3. step order;
4. configured lane order within each step;
5. chunk order within extract;
6. merge warnings; then
7. normalize warnings; followed by any output-result warnings.

TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion exercises warning order under reversed completion. No completion-order leak was found.

Chunk Plans And Checkpoints

• A reusable chunk plan stores producer warnings only. Current validators run again, and their current warnings are appended. Warnings from a cached plan candidate that fails current validation are discarded before regeneration.
• Accepted extract, merge, and normalize checkpoints store the terminal warnings for their stage. Reuse loads and appends those warnings once at the same logical handoff. Tests compare fresh and resumed warning collections and preserve their order.
• Validation-incomplete accepted outputs are not reusable, preventing a later run from silently treating incomplete validation as complete.
• Required accepted-normalize hydration replays that normalize checkpoint's warnings; checkpoint decisions separately expose that reuse occurred.

The recommended redesign should keep fresh and resumed logical diagnostics equivalent. Whether a result was reused belongs in checkpoint provenance, not in the diagnostic grouping key; adding a reused distinction would fragment groups and make equivalent runs present differently.

Terminal Surfaces

Surface	Current content	Audience	Audit result
RunOutput.Warnings	Flat final slice	Framework and CLI	Canonical in-memory list, but lacks origin and bounds.
Published warnings.json	Object containing the warnings passed into the output encoder	Durable consumers	Exact for the production JSON encoder unless the encoder itself returns warnings.
index.json	Path to warnings.json	Durable consumers	Stable discovery path; no separate diagnostic-detail path.
Run-result v1	warning_count = len(final RunOutput.Warnings)	Subprocess callers	Count only; no group/occurrence distinction.
Human stderr	run completed with N warning(s)	Operators	Count only and no direct detail path. Successful exit remains zero.
Manifest	Validation and rejection summaries, no warning collection	Durable provenance	Correctly avoids duplicating the flat list.
Debug summary warnings.json	Raw final warning array	Operators/developers	Includes final output-result warnings and can therefore differ from published warnings.json.
Debug run report	Final warning count	Operators/developers	Same final slice length as receipt and stderr.
Attempt/stage debug	Candidate-local warning detail and origin in path/envelope	Forensics	Sufficient to diagnose provenance, but debug capture is optional and is not a durable consumer contract.

Empirical Measurements

Offline And Synthetic Runs

Scenario	Result	What it establishes
Maintained complete D&D config and transcript with the repository's offline fake LLM	Approved, 10 normalized outputs, 0 rejected outputs, 12 warnings; receipt, stderr, and published file all reported 12	An ordinary structurally successful workflow can be noisy without fallback or incomplete validation.
Same complete run, grouped after publication	Three reason codes, seven exact (reason, scope, message) tuples, maximum exact-tuple repetition of three	The flat count materially overstates distinct operator conditions. Scope resets within chunks and does not identify origin.
Maintained focused scene-description workflow	Approved, one normalized output, 0 warnings	The warning channel can be quiet when synthetic model text is lexically grounded.
Generic warning publication contract	One warning reaches successful stderr, durable output, and debug summary	The ordinary pre-output path is consistent.
NPC semantic-reconciliation candidate-limit fallback	No LLM call, all records preserved, one exhaustion warning, total warnings no greater than 20	Fallback is bounded and materially different from routine normalization.
warn_continue with two failed validators and one skipped validator	Validation status contains all three incomplete validators; warning slice contains two execution-incomplete records	Current warning count does not describe all incomplete validation.
Retrying extract candidate	Two producer attempts; only the accepted attempt's one warning is final	Retry does not amplify abandoned warnings.
Terminal semantic rejection	Only the final rejected attempt's operation and validator warnings are final	Rejection diagnostics are retained without retaining superseded warnings.
Fresh versus reused extract checkpoint	Warning collections are deeply equal	Checkpoint replay does not itself amplify warnings.
Spell normalizer with 21 unresolved entries	20 records: 19 samples plus one omission record saying two additional warnings were omitted	warning_count is neither exact occurrence count nor distinct-condition count.

The focused audit tests passed in internal/cli, internal/framework/pipeline, the NPC-registry and spell normalizers, and all D&D packages.

Bounded Sample Review

The complete offline D&D run produced:

Reason	Count	Sample	Review
location_not_near_source	4	Moon Gate was absent from cited text	Correctly identifies deliberately unsupported fake output. Three records shared the same exact tuple because chunk and stage origin were lost.
location_occurrence_not_near_source	4	A Moon Gate visit was absent from cited text	Correctly identifies the same unsupported registry-driven occurrence, but repeats the same operator concern across extraction and normalization.
scene_description_not_near_source	4	A title or summary had no significant exact token in cited text	Mixed value. Generic session scene prose is ungrounded, while Arrival versus transcript arrive illustrates an expected lexical false positive.

This fake workflow is an integration fixture, not a model-quality benchmark. It nonetheless proves that the checks carry useful evidence while being too imprecise and repetitive to serve as one-warning-per-record operator alerts.

Production Evidence Still Needed

The reported 85-warning run establishes that high volume occurs in practice, but the following remain unknown:

• dominant production reason codes and stage/lane sources;
• unique group count versus repeated occurrence count;
• false-positive rate for each relatedness family;
• how much volume comes from normalization observations versus advisories;
• whether fresh and resumed production runs remain equivalent; and
• a defensible numerical acceptance target.

If further data is worthwhile, the operator can supply the v1 receipt, warnings.json, and manifest validation summaries without supplying transcript or lane artifacts. An initial privacy-preserving report should group by reason code and normalized scope family, count exact repeated tuples, and omit message text. Reviewing heuristic precision requires a separately approved bounded sample with its cited source context.

Classification Of Current Conditions

Target disposition	Current families	Result impact	Operator action	Durable placement
Warning	Empty reference; validator failure or skip accepted under warn_continue; unavailable required scene classification; exhausted semantic reconciliation	A configured process completed under an allowed degraded or incomplete policy rather than completing normally	Correct reference/configuration, inspect provider/validator, or rerun	Actionable warnings.json, receipt summary, stderr summary, debug
Advisory	Source-relatedness heuristics; unresolved spell or registry membership; guarded invalid semantic proposal; any future LLM-judged uncertainty or extraction-quality signal	Uncertain data quality or poor model proposal, but accepted data is structurally valid and deterministic guards prevented unsafe mutation	Optional model/source review; no routine action for every record	Durable diagnostic detail and debug; never a top-level warning
Observation	Whitespace/name/ID/source-reference canonicalization; canonical ordering; exact and approved semantic duplicate consolidation	Successful intended normalization	None under normal operation	Durable bounded normalization diagnostics or debug; no stderr warning
Not a diagnostic	Rejection, invalid structure, cancellation, persistence error, provider failure under fail-run policy	Candidate or run did not complete according to policy	Inspect rejection/error and retry or correct input/configuration	Existing rejection, validation summary, error, and debug contracts

Exact and semantic duplicate consolidation should remain distinguishable in category or reason metadata even though both are observations. Semantic reconciliation exhaustion remains a warning because a capability was not applied; successful approved consolidation is an observation because it is the normalizer's intended work.

Ranked Findings

AUD-WARN-001 — The Flat Warning Type Destroys Signal Quality

• Priority: High operator impact; high implementation leverage.
• Evidence: contracts.Warning has only scope, reason, and message. Routine normalizer changes, heuristic doubt, fallback, and incomplete validation all enter the same slice and the same CLI count. The offline complete run's 12 records were all advisories; the operator observed 85 records in a successful real run.
• Impact: Operators cannot tell whether a warning requires a rerun, a configuration repair, optional review, or no action. Repeated routine output trains them to ignore the channel.
• Recommendation: Replace the flat semantic contract with explicit warning, advisory, and observation dispositions plus a small category vocabulary. Do not infer disposition from message text or require every downstream consumer to maintain a reason-code policy table.

AUD-WARN-002 — Warning Records Lose The Origin Needed For Diagnosis And Aggregation

• Priority: High correctness and usability impact.
• Evidence: The runner knows stage, step, lane, module, validator, chunk ID, and chunk index at promotion time, but terminalWarnings flattens module and validator records into []contracts.Warning. Per-chunk scopes such as locations[0] and occurrences[0] then repeat without identifying their chunk or producer. source_references_normalized is intentionally shared across families and is therefore especially ambiguous.
• Impact: warnings.json cannot answer which stage or module produced a record. Message- or scope-based deduplication would merge unrelated findings or retain accidental duplicates.
• Recommendation: Keep module findings free of framework context, then have the framework add a structured origin envelope before promotion. Validator findings must retain validator identity instead of passing through validationReport.Warnings() as a flat slice.

AUD-WARN-003 — Warning Volume Is Not End-To-End Bounded Or Validated

• Priority: High robustness impact; medium immediate likelihood.
• Evidence: D&D's LimitWarnings caps most individual producers at 20, but NPC- and spell-relatedness return one warning per record without the helper. Every extract validator is invoked per chunk, all ten relatedness checks run again after normalization, and there is no run-level collector. The generic contract validates neither disposition nor reason/message size, UTF-8, blankness, or total records.
• Impact: Warning memory and output grow with chunks, lanes, configured validators, and record counts. Local omission records lose exact occurrence semantics while still incrementing warning_count.
• Recommendation: Add a generic bounded diagnostic collector that preserves exact occurrence counts and bounded samples. Validate all diagnostic fields at the module/framework boundary. Immediately bring NPC and spell relatedness under the existing cap if the full redesign is staged.

AUD-WARN-004 — Output Encoder Warnings Make Durable Surfaces Disagree

• Priority: Medium current impact; high contract correctness risk.
• Evidence: Runner.Run passes existing warnings to encoder.Encode, then the production encoder serializes warnings.json. Only after encoding does the runner append OutputResult.Warnings. The receipt, stderr, debug summary, and debug run report see the final slice; the already-created published file cannot. No production encoder currently returns a warning, so ordinary JSON runs do not trigger the defect.
• Impact: A valid output-module implementation can violate the documented claim that warning_count describes the published warning collection.
• Recommendation: Remove successful output warnings from the output-module contract unless a demonstrated use case requires them; encoding failures should be errors and optional encoder observations should be debug data. A two-phase finalize API is the viable but more complex alternative.

AUD-WARN-005 — Validation Skips Are Incomplete But Not Warned

• Priority: Medium operator/correctness impact.
• Evidence: firstIncompleteValidation treats failed and skipped validators alike, and validation summaries include both. incompleteValidationWarnings emits records only for validationFailed. The focused test demonstrates three incomplete validators but two warnings.
• Impact: A successful run can have validation_status: incomplete while its warning count understates or even omits the affected validators. A caller that checks only warnings receives a weaker signal than the manifest and receipt status.
• Recommendation: Produce one aggregated incomplete-validation warning group whose occurrences cover both failure and skip, while retaining typed outcome and safe reason metadata in the validation summary. Do not expose provider errors or arbitrary skip prose in model or operator messages.

AUD-WARN-006 — Relatedness Checks Are Useful But Repetitive And Lexically Weak

• Priority: Medium operator impact; low acceptance-policy urgency.
• Evidence: Every family runs the advisory in both extract and normalize chains. The complete fixture contains exact repeated tuples, and the checks rely on exact normalized token sequences or a minimal significant-token overlap. Reason naming drifts between *_not_near_source and *_source_unrelated.
• Impact: The checks can catch unsupported entities, but aliases, pronouns, inflection, and generic scene prose create predictable false positives. Flat per-record presentation magnifies them.
• Recommendation: Retain the validators and their stage-local execution, but classify and aggregate them as advisories. Normalize reason-code naming when the diagnostic contract changes. Do not strengthen them into rejection rules without a human-reviewed production evaluation.

AUD-WARN-007 — warning_count Has No Stable Operational Meaning

• Priority: High downstream-contract impact.
• Evidence: The receipt and CLI use len(output.Warnings). One list element can be an omission summary representing several hidden occurrences; repeated records can represent the same condition; and skipped validators can be absent. A 21-occurrence spell test produces a list length of 20.
• Impact: The value is neither an exact occurrence count nor a distinct warning-group count. Consumers cannot set policy or present a trustworthy summary from it.
• Recommendation: Introduce explicit warning-group and warning-occurrence counts in a versioned receipt. Do not silently redefine the v1 field.

Recommended Target Contract And Presentation Model

Diagnostic Model

Use one validated internal diagnostic model with these concepts:

• disposition: warning, advisory, or observation;
• category: a small enum such as configuration, degradation, validation_incomplete, data_quality, fallback, or normalization;
• reason code: stable semantic identity owned by the producer;
• origin: framework-added phase/stage, step ID, lane ID, module key, validator name, chunk ID, and chunk index when applicable;
• occurrence count: exact number of matching findings;
• samples: a small deterministic list of bounded scope/message pairs; and
• omitted sample count: occurrence_count - len(samples), represented as metadata rather than another diagnostic record.

Errors and rejected outputs must not become diagnostic dispositions. A warning means that the run completed under policy despite a process-level degradation or incomplete configured operation. Advisory and observation dispositions can describe accepted artifact quality and transformation provenance, but no LLM-judged extraction-quality signal may be promoted to a warning. Validation status remains authoritative for approval, rejection, and incomplete validation.

Aggregation

The framework runner should own aggregation after it enriches findings with origin and before public output construction. Modules and validators retain semantic ownership of disposition, category, reason, scope, and message; they must not own CLI or file presentation.

The default stable key should be:

disposition + category + reason_code
+ phase/stage + step_id + lane_id + module_key + validator_name

Chunk, record scope, and message text belong in samples and must not be part of the group key. This groups repeated per-chunk findings without merging the same code across distinct producers or pipeline locations. Group order should be the first occurrence in the runner's existing canonical order; sample order should follow the same order. A final canonical sort by the complete origin key is also viable, but completion timing must never choose either order.

Aggregation must be incremental and bounded. Producers should use a shared collector that counts every occurrence while retaining only bounded samples; the framework then merges producer groups without reconstructing counts from omission prose. A global maximum group count is also required, with overflow represented by structured aggregate metadata and with actionable groups given priority over lower dispositions.

Durable Files

Keep one canonical home for each class:

• warnings.json should contain versioned, grouped actionable warnings only;
• a new diagnostics.json should contain versioned advisory and observation groups only, avoiding duplication of warning groups;
• index.json should link both files;
• rejected.json and manifest validation summaries should retain their current separate responsibilities; and
• debug bundles should retain candidate-attempt detail plus the final grouped projections.

This is preferable to keeping all detail in warnings.json and filtering only the CLI: downstream consumers would otherwise continue to receive a semantically mixed warning contract, and routine observations would still dominate the durable file.

CLI And Receipt

For a successful human run with actionable warnings, print a concise summary such as:

notarius: run completed with 2 warning groups (7 occurrences); details=/.../warnings.json

Advisories and observations should not produce the warning line. Their durable path remains discoverable through index.json; a concise non-warning count can be added to the ordinary success line only if operator testing shows value. An ordinary successful run with no process degradation should write nothing to the warning stream even when it publishes quality advisories or normalization observations.

Create notarius.run-result.v2 rather than redefining v1. It should expose at least:

• warning_group_count;
• warning_occurrence_count; and
• diagnostic_group_count for non-warning durable groups.

The receipt should continue to expose validation status, validation summaries, and rejected-output count independently. Process exit behavior should not change as part of warning presentation reform.

Checkpoint Semantics

Store the structured terminal diagnostic groups with accepted checkpoints and replay them exactly once at their logical stage. Fresh and reused runs should produce the same public groups and counts. Checkpoint events and debug records, not diagnostic identity, should disclose whether computation was reused.

Output Encoder Boundary

Prefer removing OutputResult.Warnings. A successful output encoder should either return the complete logical files or fail. If future encoders genuinely need to produce durable post-encoding warnings, introduce an explicit two-phase prepare/finalize contract so those warnings can be included in the same published collection. Do not retain the current self-inconsistent one-pass capability.

Resolution Of Required Design Questions

Question	Recommendation	Viable alternative and tradeoff
Explicit severity/disposition or external reason mapping?	Put validated disposition and category in the contract.	A central reason-code registry avoids payload fields but makes new modules depend on a second synchronized policy table and leaves downstream meaning implicit.
Keep all detail in warnings.json or separate it?	Separate grouped actionable warnings from grouped advisories/observations in diagnostics.json.	Keep the flat durable list and aggregate only CLI output; simpler migration, but it preserves the noisy downstream contract and ambiguous count.
Who owns aggregation?	Framework runner/coordinator after origin enrichment.	Output module aggregation keeps framework types smaller but duplicates policy across encoders and cannot repair missing validator origin.
Stable aggregation key?	Disposition, category, reason code, and full producer origin; exclude chunk/scope/message.	Explicit producer-supplied grouping keys offer flexibility but add another identity that can drift from reason codes. Message-template grouping is brittle and unsafe.
Samples and omissions?	Exact occurrence count plus deterministic bounded samples and numeric omitted-sample count.	Omission warning records preserve the current representation but inflate group counts and require prose parsing.
warning_count semantics?	Version receipt and replace ambiguity with group and occurrence counts.	Keep v1 count as published record length and add optional fields; compatible, but two competing warning counts remain easy to misuse.
Which normalization changes remain warnings?	Only exhausted process fallback. Unresolved membership is a data-quality advisory; successful canonicalization, reordering, ID repair, source-ref dedupe, and duplicate consolidation are observations.	Treat unresolved membership or semantic duplicate consolidation as warnings because they affect grounding or cardinality; more conservative, but it violates the process-only warning rule and reports accepted artifact quality as an operational failure.
Source-relatedness disposition?	Grouped advisory by default; preserve current approve behavior.	Retain warning disposition or make rejection configurable. Rejection requires production precision evidence; current lexical rules are not strong enough.
Checkpoint-loaded warnings?	Present the same logical groups as fresh execution and use checkpoint events for reuse provenance.	Mark groups as replayed; aids forensics but fragments aggregation and makes semantically equivalent runs differ.
ADR and schema versions?	Add an ADR and version the run receipt and diagnostic files.	Treat the work as CLI-only presentation and avoid an ADR; insufficient because module contracts, output files, checkpoint payloads, and downstream fields change.

Compatibility, Documentation, And ADR Implications

The target alters public and internal contracts enough to require a new ADR. It should record:

• the distinction among warnings, advisories, observations, rejections, and errors;
• the invariant that warnings are process-level signals, LLM-judged extraction quality is never a warning, and ordinary non-degraded success has zero warnings;
• module semantic ownership versus framework origin/aggregation ownership;
• bounded group and sample semantics;
• fresh/checkpoint equivalence; and
• the output-encoder decision.

Implementation should introduce notarius.run-result.v2. The grouped warning and diagnostic envelopes should each carry their own schema version. Because the content of warnings.json changes incompatibly from a flat array wrapper to groups, release notes and the published JSON integration contract must call out the migration. index.json gains the diagnostic file path.

Canonical documentation updates belong in:

• docs/cli.md for stderr presentation only;
• docs/operations.md for operator review and debug workflow;
• docs/integrations/json-output.md for warning and diagnostic file schemas;
• docs/integrations/run-result.md for v2 fields and compatibility;
• docs/consumers/subprocess.md and docs/consumers/dnd-pipeline.md for downstream policy checks;
• docs/internal/pipeline.md for promotion, aggregation, retry, and checkpoint mechanics;
• docs/internal/modules.md and docs/internal/dnd.md for producer rules and the D&D classification matrix; and
• docs/policy/architecture.md for the durable ownership invariant after the ADR is accepted and implemented.

No configuration knob is required for the first implementation. A fixed, well-documented taxonomy is easier to reason about than per-reason display overrides. Configurable escalation or suppression can be considered only after production review demonstrates a concrete operator need.

Test Coverage Assessment

Existing coverage worth preserving includes:

• accepted-attempt and terminal-rejection warning promotion;
• validator failure retry exhaustion and warn_continue;
• module semantic retry fallback;
• deterministic warning order under concurrent lane completion;
• chunk-plan invalidation and discarded-cache warning behavior;
• fresh/checkpoint warning equivalence;
• local D&D warning caps and safe dynamic-message quoting;
• JSON warning-file publication; and
• CLI stderr, debug, and receipt counts.

Material gaps are:

• no bound test for NPC- or spell-relatedness warnings;
• no generic warning-field or result-size validation;
• no test for output encoder warnings versus published warnings.json;
• no operator-level aggregation or bounded-sample tests;
• no fresh/resume test for grouped counts because groups do not yet exist; and
• no production evaluation of advisory precision.

Tests should protect the semantic relationships: exact occurrence counts, bounded samples, deterministic group order, actionable-only warning presentation, and cross-surface equality. They should not assert one exact warning count for every complete D&D run or treat message wording as a public API unless the wording itself enforces a security boundary.

Audit Conclusion

The application is in a good position for warning reform. Its retry, validation, checkpoint, and concurrency mechanics provide reliable points at which to attach structured diagnostics. The most valuable change is not to suppress individual reason codes; it is to replace the semantically flat, origin-free collection with bounded typed groups and to reserve the word “warning” for conditions that merit operator attention.

Provider-backed runs would improve prioritization and help tune the D&D advisories, but they are not necessary to conclude that routine normalization and heuristic doubt should not dominate stderr or the durable warning contract. They should be gathered before changing heuristic acceptance policy or adopting a numerical production warning-volume target.