Plan warning and diagnostic reform

docs/roadmap/audit.md

@@ -0,0 +1,648 @@
+# Warning Signal And Presentation Audit
+
+## Executive Assessment
+
+Notarius warning execution is mechanically stronger than its operator-facing
+presentation. Terminal-attempt promotion, stable ordering after concurrent
+work, checkpoint replay, validation summaries, and debug retention are all
+substantially correct. The audit found no general duplicate-append defect in
+the extract, merge, or normalize handoffs and no leakage of abandoned-attempt
+warnings into a successful result.
+
+The warning channel itself is not coherent. One flat `contracts.Warning` type
+currently represents at least four materially different concepts:
+
+- actionable degradation or incomplete validation;
+- heuristic data-quality doubt;
+- successful but potentially reviewable fallback; and
+- routine canonicalization, ordering, and deduplication observations.
+
+That conflation is the primary reason successful runs produce a count that is
+large but operationally weak. The maintained complete D&D example demonstrates
+the problem without a live provider: an approved run with no rejected outputs
+published 12 warning records, all from three advisory relatedness checks. An
+operator separately reported a successful complete D&D run with 10 outputs,
+one rejection, and 85 warnings. The production bundle for that run was not
+available in this environment, so its reason-code distribution could not be
+measured.
+
+The current implementation also has four correctness or robustness gaps:
+
+1. warning records lose stage, step, lane, module, validator, and chunk
+   provenance when promoted, which makes safe aggregation and diagnosis
+   impossible from `warnings.json` alone;
+2. there is no framework-level validation or aggregate bound, and the NPC- and
+   spell-relatedness validators bypass the D&D warning limiter entirely;
+3. warnings returned by an output encoder are added after `warnings.json` has
+   already been encoded, so the receipt, stderr, debug bundle, and published
+   warning file can disagree; and
+4. a skipped validator contributes to `incomplete` validation but does not
+   receive the warning generated for an exhausted validator failure.
+
+The recommended end state is a structured diagnostic contract with explicit
+disposition, category, origin, occurrence count, and bounded samples. Warnings
+are reserved for process-level degradation or incompleteness. LLM-judged or
+deterministically inferred extraction-quality signals are advisories, never
+warnings, and routine normalization observations remain inspectable without
+being reported as top-level warnings. An ordinary successful run in which all
+configured work completes normally should therefore report zero warnings. This
+is an architectural and durable-contract change, not merely revised CLI prose.
+
+## Evidence And Limits
+
+The audit used:
+
+- a complete static search of production `contracts.Warning` constructors,
+  reason-code constants, result fields, and promotion sites under `internal/`;
+- call-path inspection through producer attempts, validators, lane
+  coordination, chunk-plan reuse, checkpoints, output encoding, debug output,
+  CLI presentation, and run-result construction;
+- the maintained complete and minimal D&D examples with offline fake LLMs;
+- focused deterministic tests for warning bounds, semantic-reconciliation
+  fallback, `warn_continue`, semantic retries, terminal rejection, concurrency
+  ordering, and checkpoint reuse; and
+- the operator-provided observation of an 85-warning complete D&D run.
+
+No provider-backed production run was attempted because this environment has
+no API key. Consequently, the audit can establish warning mechanics, possible
+multiplicity, synthetic volume, and obvious heuristic limitations, but cannot
+estimate production frequency or the real false-positive rate of individual
+D&D advisories. Those measurements are not required to choose the recommended
+architecture; they are required before strengthening any heuristic advisory
+into a rejection or setting a numerical production acceptance target.
+
+## Complete Warning-Producer Inventory
+
+### Framework And Generic Boundaries
+
+| Producer | Reason code | Trigger and consequence | Multiplicity and bound | Current surfaces and coverage |
+| --- | --- | --- | --- | --- |
+| Reference materialization in `internal/framework/pipeline/references.go` | `empty_reference` | A bound external reference is a valid, accepted media type but contains zero bytes. The prompt may receive materially incomplete context. | One per empty bound file; finite by configuration but no shared run-level cap. | Enters `RunInput.Warnings`; reference tests protect contextual scope. |
+| Producer-attempt policy in `internal/framework/pipeline/producer_attempts.go` | `validator_execution_incomplete` | An applicable validator exhausted its execution budget and `warn_continue` accepted the otherwise valid candidate. | One per failed validator on each terminal candidate. An extract chain can multiply this by chunks and lanes. There is no global cap. | Durable warning, receipt count, stderr, debug, and validation summary. `TestWarnContinueRecordsOneWarningForEachExhaustedValidator` covers failures. |
+| Chunk, extract, merge, normalize, and output module result contracts | Module-defined | A module may return arbitrary warnings with its successful candidate. | No contract validation, message limit, per-result cap, or global cap. Current production modules are inventoried below. | Terminal-attempt filtering and concurrency ordering are well tested. |
+| Production JSON output encoder | None | The encoder copies incoming warnings into `warnings.json`; it does not currently create warnings. | Same count as its input. | JSON encoder and assembled-pipeline tests compare the incoming run warnings with the published file. |
+| Output encoder result contract | Module-defined | Any output encoder may return warnings discovered during encoding. | Unbounded by contract. No production encoder currently exercises this capability. | Appended to final `RunOutput` only after logical files were encoded; this is the cross-surface defect described in AUD-WARN-004. |
+
+Input adapters and production mergers do not currently have independent
+warning producers. Chunk-plan and checkpoint decisions are structured manifest
+or debug provenance rather than warnings. Cancellation and hard persistence,
+reference, parsing, serialization, and provider failures remain errors.
+
+### D&D Extraction Gates
+
+| Producer | Reason code | Trigger and consequence | Multiplicity and bound |
+| --- | --- | --- | --- |
+| `dnd/combat-turns` extractor | `scene_classification_unavailable` | The chunk has no exact matching scene-description classification. The extractor returns an empty accepted result and skips the LLM, so combat-turn output may be incomplete. | At most one per chunk for this lane. |
+| `dnd/enemy-events` extractor | `scene_classification_unavailable` | The same missing or mismatched scene gate causes accepted empty enemy-event output. | At most one per chunk for this lane. |
+
+An exact non-combat classification produces an intentional empty result without
+a warning. An exact combat classification proceeds normally. The two producers
+share a code and operator consequence but use different messages; their module
+origins are not retained in the final warning record.
+
+### D&D Source-Relatedness Validators
+
+All ten relatedness validators are deterministic advisories: they approve the
+candidate and warn when contextual prose or an entity name is not lexically
+present in cited text. Shape and source-reference failures are deliberately
+left to blocking validators earlier in the chain. The same relatedness
+validator is registered in both the extract and normalize default chain for
+each artifact family in `internal/modules/dnd/register/chains.go`.
+
+| Artifact family | Warning reason | Per-record trigger | Local bound | Omission reason |
+| --- | --- | --- | --- | --- |
+| Combat turns | `combat_turn_not_near_source` | Actor token sequence absent | 20 per validator invocation | `combat_turn_relatedness_warnings_omitted` |
+| Enemy events | `enemy_event_not_near_source` | Subject token sequence absent | 20 | `enemy_event_relatedness_warnings_omitted` |
+| Item occurrences | `item_occurrence_source_unrelated` | Item name token sequence absent | 20 | `item_occurrence_relatedness_warnings_omitted` |
+| Item registry | `item_not_near_source` | Item name token sequence absent | 20 | `item_relatedness_warnings_omitted` |
+| Location occurrences | `location_occurrence_not_near_source` | Location name token sequence absent | 20 | `location_occurrence_relatedness_warnings_omitted` |
+| Location registry | `location_not_near_source` | Location name token sequence absent | 20 | `location_relatedness_warnings_omitted` |
+| NPC occurrences | `npc_occurrence_not_near_source` | NPC name token sequence absent | 20 | `npc_occurrence_relatedness_warnings_omitted` |
+| NPC registry | `npc_not_near_source` | NPC name token sequence absent | **Unbounded** | None |
+| Scene descriptions | `scene_description_not_near_source` | No significant title or summary token appears; up to two findings per scene | 20 | `scene_description_relatedness_warnings_omitted` |
+| Spells | `spell_not_near_source` | Spell-name token sequence absent | **Unbounded** | None |
+
+The eight limiter-generated omission records are presentation artifacts, not
+new source-relatedness conditions. They occupy a warning slot and make list
+length differ from the actual occurrence count.
+
+### D&D Normalizers
+
+Every production D&D normalizer bounds its returned warning slice to 20 through
+`internal/modules/dnd/shared/diagnostics`, including a final omission record
+when needed. Registry semantic retries reserve space for their fallback
+warning. The following table is complete by semantically distinct condition;
+codes listed together are parallel artifact-family variants.
+
+| Condition | Reason codes | Result impact | Current classification assessment |
+| --- | --- | --- | --- |
+| Display or field whitespace/name canonicalization | `npc_fields_normalized`, `item_fields_normalized`, `location_fields_normalized`, `spell_name_canonicalized`, `combat_actor_canonicalized`, `enemy_event_name_canonicalized`, `item_occurrence_name_canonicalized`, `location_occurrence_name_canonicalized`, `scene_description_prose_normalized` | Deterministic successful mutation. The item-occurrence code can also describe `from`/`to` whitespace, not only the item name. | Routine observation. |
+| Durable ID recomputation | `npc_id_recomputed`, `item_id_recomputed`, `location_id_recomputed` | Restores the deterministic name-derived ID. | Routine observation; invalid identity is separately rejected by default chains. |
+| Source-reference sorting or deduplication | `source_references_normalized` | Sorts and removes exact duplicate references while deliberately preserving invalid references for their validators. | Routine observation. Shared code is useful but ambiguous without producer origin. |
+| Canonical record ordering | `combat_turns_reordered`, `enemy_events_reordered`, `item_occurrences_reordered`, `location_occurrences_reordered`, `npc_occurrences_reordered`, `scene_description_order_normalized` | Deterministic order changes only. | Routine observation. |
+| Exact or approved semantic duplicate consolidation | `duplicate_npc_collapsed`, `duplicate_item_collapsed`, `duplicate_location_collapsed`, `duplicate_spell_cast_collapsed`, `duplicate_combat_turn_collapsed`, `duplicate_enemy_event_collapsed`, `duplicate_item_occurrence_collapsed`, `duplicate_location_occurrence_collapsed`, `duplicate_npc_occurrence_collapsed`, `scene_description_duplicate_collapsed` | Removes duplicate records and preserves or combines canonical evidence according to the artifact policy. Registry codes cover both exact and accepted semantic consolidation. | Durable normalization observation; not normally operator-actionable. |
+| Unresolved external membership | `spell_name_unresolved`, `item_occurrence_unknown_item_id`, `location_occurrence_unknown_location_id` | The value is preserved but is not grounded in the effective catalog or registry. Default chains normally reject the same condition before normalization; it remains reachable with validator overrides or defensive direct use. | Actionable data-quality warning. |
+| Unsafe currency consolidation proposal | `item_semantic_proposal_invalid` | The proposed group is rejected and all records are preserved because denominations or currency/non-currency members are incompatible. The same code is also used internally as a retry reason. | Advisory about model proposal quality; no accepted-data loss. The control and diagnostic meanings should be separated. |
+| Semantic reconciliation unavailable or exhausted | `npc_semantic_reconciliation_exhausted`, `item_semantic_reconciliation_exhausted`, `location_semantic_reconciliation_exhausted` | The safe deterministic result is accepted, but possible semantic duplicates remain. | Actionable fallback warning. |
+| Local warning truncation | `npc_normalization_warnings_omitted`, `item_normalization_warnings_omitted`, `location_normalization_warnings_omitted`, `spell_normalization_warnings_omitted`, `combat_turn_normalization_warnings_omitted`, `enemy_event_normalization_warnings_omitted`, `item_occurrence_normalization_warnings_omitted`, `location_occurrence_normalization_warnings_omitted`, `npc_occurrence_normalization_warnings_omitted`, `scene_description_normalization_warnings_omitted` | Reports that individual records were omitted from presentation. | Group metadata, not an independent warning. |
+
+No production D&D normalization warning exposes raw model responses,
+correction guidance, or provider errors. Most dynamic names are quoted and
+truncated by the shared helper. That local discipline is not enforced by the
+generic warning contract, and the spell relatedness message does not use the
+shared truncation helper.
+
+## Warning Propagation And Surface Map
+
+```text
+external-reference warnings -----------------------------+
+                                                           |
+module candidate warnings -> validation chain warnings     |
+             |                      |                       |
+             +---- producer-attempt terminal policy -------+
+                                  |                        |
+                   accepted / terminal rejection only      |
+                                  |                        |
+              chunk or lane result in canonical order      |
+                                  |                        |
+       checkpoint record/replay and ordered step merge     |
+                                  |                        |
+                            RunOutput.Warnings <------------+
+                                  |
+                   OutputRequest -> output encoder
+                                  |             |
+                         warnings.json     OutputResult.Warnings
+                                                |
+                              appended to final RunOutput only
+                                  |
+                 receipt, stderr, final debug warning summary
+```
+
+### Attempts And Validation
+
+- `runProducerAttempts` promotes only the terminal accepted or terminal
+  rejected candidate's module and completed-validator warnings. Operational,
+  structural, semantic, and module-directed attempts that are superseded are
+  retained in attempt debug artifacts but not in the final collection.
+- A module-directed semantic-reconciliation retry adds its fallback warning
+  only when no retry remains. Earlier attempt warnings are discarded.
+- On `warn_continue`, warnings from the otherwise accepted candidate and
+  completed approved or rejected validators are retained. One fixed,
+  non-sensitive `validator_execution_incomplete` warning is added for every
+  failed validator. Skipped validators affect the validation summary and final
+  `incomplete` status but do not receive such a warning.
+- A semantic terminal rejection retains only warnings from that rejected
+  attempt. Structural rejection after producer failure cannot retain a
+  candidate warning because no valid candidate result exists.
+
+These behaviors are protected by the producer-attempt, extract-handoff,
+rejection-warning, normalize-retry, and attempt-debug tests. They are the right
+foundation for the redesign and should not be replaced with early-exit or
+all-attempt accumulation.
+
+### Concurrency And Ordering
+
+Extract jobs are dispatched chunk-first and lane-second. Results are stored by
+chunk index, finalized in ascending chunk order, and lane continuations are
+merged into a slice indexed by configured lane order. Pipeline steps run in
+configured order. The resulting public order is therefore:
+
+1. pre-run reference warnings;
+2. chunk-stage warnings;
+3. step order;
+4. configured lane order within each step;
+5. chunk order within extract;
+6. merge warnings; then
+7. normalize warnings; followed by any output-result warnings.
+
+`TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion` exercises warning
+order under reversed completion. No completion-order leak was found.
+
+### Chunk Plans And Checkpoints
+
+- A reusable chunk plan stores producer warnings only. Current validators run
+  again, and their current warnings are appended. Warnings from a cached plan
+  candidate that fails current validation are discarded before regeneration.
+- Accepted extract, merge, and normalize checkpoints store the terminal
+  warnings for their stage. Reuse loads and appends those warnings once at the
+  same logical handoff. Tests compare fresh and resumed warning collections and
+  preserve their order.
+- Validation-incomplete accepted outputs are not reusable, preventing a later
+  run from silently treating incomplete validation as complete.
+- Required accepted-normalize hydration replays that normalize checkpoint's
+  warnings; checkpoint decisions separately expose that reuse occurred.
+
+The recommended redesign should keep fresh and resumed logical diagnostics
+equivalent. Whether a result was reused belongs in checkpoint provenance, not
+in the diagnostic grouping key; adding a `reused` distinction would fragment
+groups and make equivalent runs present differently.
+
+### Terminal Surfaces
+
+| Surface | Current content | Audience | Audit result |
+| --- | --- | --- | --- |
+| `RunOutput.Warnings` | Flat final slice | Framework and CLI | Canonical in-memory list, but lacks origin and bounds. |
+| Published `warnings.json` | Object containing the warnings passed into the output encoder | Durable consumers | Exact for the production JSON encoder unless the encoder itself returns warnings. |
+| `index.json` | Path to `warnings.json` | Durable consumers | Stable discovery path; no separate diagnostic-detail path. |
+| Run-result v1 | `warning_count = len(final RunOutput.Warnings)` | Subprocess callers | Count only; no group/occurrence distinction. |
+| Human stderr | `run completed with N warning(s)` | Operators | Count only and no direct detail path. Successful exit remains zero. |
+| Manifest | Validation and rejection summaries, no warning collection | Durable provenance | Correctly avoids duplicating the flat list. |
+| Debug summary `warnings.json` | Raw final warning array | Operators/developers | Includes final output-result warnings and can therefore differ from published `warnings.json`. |
+| Debug run report | Final warning count | Operators/developers | Same final slice length as receipt and stderr. |
+| Attempt/stage debug | Candidate-local warning detail and origin in path/envelope | Forensics | Sufficient to diagnose provenance, but debug capture is optional and is not a durable consumer contract. |
+
+## Empirical Measurements
+
+### Offline And Synthetic Runs
+
+| Scenario | Result | What it establishes |
+| --- | --- | --- |
+| Maintained complete D&D config and transcript with the repository's offline fake LLM | Approved, 10 normalized outputs, 0 rejected outputs, 12 warnings; receipt, stderr, and published file all reported 12 | An ordinary structurally successful workflow can be noisy without fallback or incomplete validation. |
+| Same complete run, grouped after publication | Three reason codes, seven exact `(reason, scope, message)` tuples, maximum exact-tuple repetition of three | The flat count materially overstates distinct operator conditions. Scope resets within chunks and does not identify origin. |
+| Maintained focused scene-description workflow | Approved, one normalized output, 0 warnings | The warning channel can be quiet when synthetic model text is lexically grounded. |
+| Generic warning publication contract | One warning reaches successful stderr, durable output, and debug summary | The ordinary pre-output path is consistent. |
+| NPC semantic-reconciliation candidate-limit fallback | No LLM call, all records preserved, one exhaustion warning, total warnings no greater than 20 | Fallback is bounded and materially different from routine normalization. |
+| `warn_continue` with two failed validators and one skipped validator | Validation status contains all three incomplete validators; warning slice contains two execution-incomplete records | Current warning count does not describe all incomplete validation. |
+| Retrying extract candidate | Two producer attempts; only the accepted attempt's one warning is final | Retry does not amplify abandoned warnings. |
+| Terminal semantic rejection | Only the final rejected attempt's operation and validator warnings are final | Rejection diagnostics are retained without retaining superseded warnings. |
+| Fresh versus reused extract checkpoint | Warning collections are deeply equal | Checkpoint replay does not itself amplify warnings. |
+| Spell normalizer with 21 unresolved entries | 20 records: 19 samples plus one omission record saying two additional warnings were omitted | `warning_count` is neither exact occurrence count nor distinct-condition count. |
+
+The focused audit tests passed in `internal/cli`,
+`internal/framework/pipeline`, the NPC-registry and spell normalizers, and all
+D&D packages.
+
+### Bounded Sample Review
+
+The complete offline D&D run produced:
+
+| Reason | Count | Sample | Review |
+| --- | ---: | --- | --- |
+| `location_not_near_source` | 4 | `Moon Gate` was absent from cited text | Correctly identifies deliberately unsupported fake output. Three records shared the same exact tuple because chunk and stage origin were lost. |
+| `location_occurrence_not_near_source` | 4 | A `Moon Gate` visit was absent from cited text | Correctly identifies the same unsupported registry-driven occurrence, but repeats the same operator concern across extraction and normalization. |
+| `scene_description_not_near_source` | 4 | A title or summary had no significant exact token in cited text | Mixed value. Generic `session scene` prose is ungrounded, while `Arrival` versus transcript `arrive` illustrates an expected lexical false positive. |
+
+This fake workflow is an integration fixture, not a model-quality benchmark.
+It nonetheless proves that the checks carry useful evidence while being too
+imprecise and repetitive to serve as one-warning-per-record operator alerts.
+
+### Production Evidence Still Needed
+
+The reported 85-warning run establishes that high volume occurs in practice,
+but the following remain unknown:
+
+- dominant production reason codes and stage/lane sources;
+- unique group count versus repeated occurrence count;
+- false-positive rate for each relatedness family;
+- how much volume comes from normalization observations versus advisories;
+- whether fresh and resumed production runs remain equivalent; and
+- a defensible numerical acceptance target.
+
+If further data is worthwhile, the operator can supply the v1 receipt,
+`warnings.json`, and manifest validation summaries without supplying transcript
+or lane artifacts. An initial privacy-preserving report should group by reason
+code and normalized scope family, count exact repeated tuples, and omit message
+text. Reviewing heuristic precision requires a separately approved bounded
+sample with its cited source context.
+
+## Classification Of Current Conditions
+
+| Target disposition | Current families | Result impact | Operator action | Durable placement |
+| --- | --- | --- | --- | --- |
+| **Warning** | Empty reference; validator failure or skip accepted under `warn_continue`; unavailable required scene classification; exhausted semantic reconciliation | A configured process completed under an allowed degraded or incomplete policy rather than completing normally | Correct reference/configuration, inspect provider/validator, or rerun | Actionable `warnings.json`, receipt summary, stderr summary, debug |
+| **Advisory** | Source-relatedness heuristics; unresolved spell or registry membership; guarded invalid semantic proposal; any future LLM-judged uncertainty or extraction-quality signal | Uncertain data quality or poor model proposal, but accepted data is structurally valid and deterministic guards prevented unsafe mutation | Optional model/source review; no routine action for every record | Durable diagnostic detail and debug; never a top-level warning |
+| **Observation** | Whitespace/name/ID/source-reference canonicalization; canonical ordering; exact and approved semantic duplicate consolidation | Successful intended normalization | None under normal operation | Durable bounded normalization diagnostics or debug; no stderr warning |
+| **Not a diagnostic** | Rejection, invalid structure, cancellation, persistence error, provider failure under fail-run policy | Candidate or run did not complete according to policy | Inspect rejection/error and retry or correct input/configuration | Existing rejection, validation summary, error, and debug contracts |
+
+Exact and semantic duplicate consolidation should remain distinguishable in
+category or reason metadata even though both are observations. Semantic
+reconciliation exhaustion remains a warning because a capability was not
+applied; successful approved consolidation is an observation because it is the
+normalizer's intended work.
+
+## Ranked Findings
+
+### AUD-WARN-001 — The Flat Warning Type Destroys Signal Quality
+
+- **Priority:** High operator impact; high implementation leverage.
+- **Evidence:** `contracts.Warning` has only scope, reason, and message. Routine
+  normalizer changes, heuristic doubt, fallback, and incomplete validation all
+  enter the same slice and the same CLI count. The offline complete run's 12
+  records were all advisories; the operator observed 85 records in a successful
+  real run.
+- **Impact:** Operators cannot tell whether a warning requires a rerun, a
+  configuration repair, optional review, or no action. Repeated routine output
+  trains them to ignore the channel.
+- **Recommendation:** Replace the flat semantic contract with explicit
+  `warning`, `advisory`, and `observation` dispositions plus a small category
+  vocabulary. Do not infer disposition from message text or require every
+  downstream consumer to maintain a reason-code policy table.
+
+### AUD-WARN-002 — Warning Records Lose The Origin Needed For Diagnosis And Aggregation
+
+- **Priority:** High correctness and usability impact.
+- **Evidence:** The runner knows stage, step, lane, module, validator, chunk ID,
+  and chunk index at promotion time, but `terminalWarnings` flattens module and
+  validator records into `[]contracts.Warning`. Per-chunk scopes such as
+  `locations[0]` and `occurrences[0]` then repeat without identifying their
+  chunk or producer. `source_references_normalized` is intentionally shared
+  across families and is therefore especially ambiguous.
+- **Impact:** `warnings.json` cannot answer which stage or module produced a
+  record. Message- or scope-based deduplication would merge unrelated findings
+  or retain accidental duplicates.
+- **Recommendation:** Keep module findings free of framework context, then have
+  the framework add a structured origin envelope before promotion. Validator
+  findings must retain validator identity instead of passing through
+  `validationReport.Warnings()` as a flat slice.
+
+### AUD-WARN-003 — Warning Volume Is Not End-To-End Bounded Or Validated
+
+- **Priority:** High robustness impact; medium immediate likelihood.
+- **Evidence:** D&D's `LimitWarnings` caps most individual producers at 20, but
+  NPC- and spell-relatedness return one warning per record without the helper.
+  Every extract validator is invoked per chunk, all ten relatedness checks run
+  again after normalization, and there is no run-level collector. The generic
+  contract validates neither disposition nor reason/message size, UTF-8,
+  blankness, or total records.
+- **Impact:** Warning memory and output grow with chunks, lanes, configured
+  validators, and record counts. Local omission records lose exact occurrence
+  semantics while still incrementing `warning_count`.
+- **Recommendation:** Add a generic bounded diagnostic collector that preserves
+  exact occurrence counts and bounded samples. Validate all diagnostic fields
+  at the module/framework boundary. Immediately bring NPC and spell
+  relatedness under the existing cap if the full redesign is staged.
+
+### AUD-WARN-004 — Output Encoder Warnings Make Durable Surfaces Disagree
+
+- **Priority:** Medium current impact; high contract correctness risk.
+- **Evidence:** `Runner.Run` passes existing warnings to `encoder.Encode`, then
+  the production encoder serializes `warnings.json`. Only after encoding does
+  the runner append `OutputResult.Warnings`. The receipt, stderr, debug summary,
+  and debug run report see the final slice; the already-created published file
+  cannot. No production encoder currently returns a warning, so ordinary JSON
+  runs do not trigger the defect.
+- **Impact:** A valid output-module implementation can violate the documented
+  claim that `warning_count` describes the published warning collection.
+- **Recommendation:** Remove successful output warnings from the output-module
+  contract unless a demonstrated use case requires them; encoding failures
+  should be errors and optional encoder observations should be debug data. A
+  two-phase finalize API is the viable but more complex alternative.
+
+### AUD-WARN-005 — Validation Skips Are Incomplete But Not Warned
+
+- **Priority:** Medium operator/correctness impact.
+- **Evidence:** `firstIncompleteValidation` treats failed and skipped validators
+  alike, and validation summaries include both. `incompleteValidationWarnings`
+  emits records only for `validationFailed`. The focused test demonstrates
+  three incomplete validators but two warnings.
+- **Impact:** A successful run can have `validation_status: incomplete` while
+  its warning count understates or even omits the affected validators. A caller
+  that checks only warnings receives a weaker signal than the manifest and
+  receipt status.
+- **Recommendation:** Produce one aggregated incomplete-validation warning
+  group whose occurrences cover both failure and skip, while retaining typed
+  outcome and safe reason metadata in the validation summary. Do not expose
+  provider errors or arbitrary skip prose in model or operator messages.
+
+### AUD-WARN-006 — Relatedness Checks Are Useful But Repetitive And Lexically Weak
+
+- **Priority:** Medium operator impact; low acceptance-policy urgency.
+- **Evidence:** Every family runs the advisory in both extract and normalize
+  chains. The complete fixture contains exact repeated tuples, and the checks
+  rely on exact normalized token sequences or a minimal significant-token
+  overlap. Reason naming drifts between `*_not_near_source` and
+  `*_source_unrelated`.
+- **Impact:** The checks can catch unsupported entities, but aliases, pronouns,
+  inflection, and generic scene prose create predictable false positives. Flat
+  per-record presentation magnifies them.
+- **Recommendation:** Retain the validators and their stage-local execution,
+  but classify and aggregate them as advisories. Normalize reason-code naming
+  when the diagnostic contract changes. Do not strengthen them into rejection
+  rules without a human-reviewed production evaluation.
+
+### AUD-WARN-007 — `warning_count` Has No Stable Operational Meaning
+
+- **Priority:** High downstream-contract impact.
+- **Evidence:** The receipt and CLI use `len(output.Warnings)`. One list element
+  can be an omission summary representing several hidden occurrences; repeated
+  records can represent the same condition; and skipped validators can be
+  absent. A 21-occurrence spell test produces a list length of 20.
+- **Impact:** The value is neither an exact occurrence count nor a distinct
+  warning-group count. Consumers cannot set policy or present a trustworthy
+  summary from it.
+- **Recommendation:** Introduce explicit warning-group and warning-occurrence
+  counts in a versioned receipt. Do not silently redefine the v1 field.
+
+## Recommended Target Contract And Presentation Model
+
+### Diagnostic Model
+
+Use one validated internal diagnostic model with these concepts:
+
+- **disposition:** `warning`, `advisory`, or `observation`;
+- **category:** a small enum such as `configuration`, `degradation`,
+  `validation_incomplete`, `data_quality`, `fallback`, or `normalization`;
+- **reason code:** stable semantic identity owned by the producer;
+- **origin:** framework-added phase/stage, step ID, lane ID, module key,
+  validator name, chunk ID, and chunk index when applicable;
+- **occurrence count:** exact number of matching findings;
+- **samples:** a small deterministic list of bounded scope/message pairs; and
+- **omitted sample count:** `occurrence_count - len(samples)`, represented as
+  metadata rather than another diagnostic record.
+
+Errors and rejected outputs must not become diagnostic dispositions. A warning
+means that the run completed under policy despite a process-level degradation
+or incomplete configured operation. Advisory and observation dispositions can
+describe accepted artifact quality and transformation provenance, but no
+LLM-judged extraction-quality signal may be promoted to a warning. Validation
+status remains authoritative for approval, rejection, and incomplete
+validation.
+
+### Aggregation
+
+The framework runner should own aggregation after it enriches findings with
+origin and before public output construction. Modules and validators retain
+semantic ownership of disposition, category, reason, scope, and message; they
+must not own CLI or file presentation.
+
+The default stable key should be:
+
+```text
+disposition + category + reason_code
++ phase/stage + step_id + lane_id + module_key + validator_name
+```
+
+Chunk, record scope, and message text belong in samples and must not be part of
+the group key. This groups repeated per-chunk findings without merging the same
+code across distinct producers or pipeline locations. Group order should be
+the first occurrence in the runner's existing canonical order; sample order
+should follow the same order. A final canonical sort by the complete origin key
+is also viable, but completion timing must never choose either order.
+
+Aggregation must be incremental and bounded. Producers should use a shared
+collector that counts every occurrence while retaining only bounded samples;
+the framework then merges producer groups without reconstructing counts from
+omission prose. A global maximum group count is also required, with overflow
+represented by structured aggregate metadata and with actionable groups given
+priority over lower dispositions.
+
+### Durable Files
+
+Keep one canonical home for each class:
+
+- `warnings.json` should contain versioned, grouped actionable warnings only;
+- a new `diagnostics.json` should contain versioned advisory and observation
+  groups only, avoiding duplication of warning groups;
+- `index.json` should link both files;
+- `rejected.json` and manifest validation summaries should retain their current
+  separate responsibilities; and
+- debug bundles should retain candidate-attempt detail plus the final grouped
+  projections.
+
+This is preferable to keeping all detail in `warnings.json` and filtering only
+the CLI: downstream consumers would otherwise continue to receive a semantically
+mixed warning contract, and routine observations would still dominate the
+durable file.
+
+### CLI And Receipt
+
+For a successful human run with actionable warnings, print a concise summary
+such as:
+
+```text
+notarius: run completed with 2 warning groups (7 occurrences); details=/.../warnings.json
+```
+
+Advisories and observations should not produce the warning line. Their durable
+path remains discoverable through `index.json`; a concise non-warning count can
+be added to the ordinary success line only if operator testing shows value. An
+ordinary successful run with no process degradation should write nothing to
+the warning stream even when it publishes quality advisories or normalization
+observations.
+
+Create `notarius.run-result.v2` rather than redefining v1. It should expose at
+least:
+
+- `warning_group_count`;
+- `warning_occurrence_count`; and
+- `diagnostic_group_count` for non-warning durable groups.
+
+The receipt should continue to expose validation status, validation summaries,
+and rejected-output count independently. Process exit behavior should not
+change as part of warning presentation reform.
+
+### Checkpoint Semantics
+
+Store the structured terminal diagnostic groups with accepted checkpoints and
+replay them exactly once at their logical stage. Fresh and reused runs should
+produce the same public groups and counts. Checkpoint events and debug records,
+not diagnostic identity, should disclose whether computation was reused.
+
+### Output Encoder Boundary
+
+Prefer removing `OutputResult.Warnings`. A successful output encoder should
+either return the complete logical files or fail. If future encoders genuinely
+need to produce durable post-encoding warnings, introduce an explicit
+two-phase prepare/finalize contract so those warnings can be included in the
+same published collection. Do not retain the current self-inconsistent
+one-pass capability.
+
+## Resolution Of Required Design Questions
+
+| Question | Recommendation | Viable alternative and tradeoff |
+| --- | --- | --- |
+| Explicit severity/disposition or external reason mapping? | Put validated disposition and category in the contract. | A central reason-code registry avoids payload fields but makes new modules depend on a second synchronized policy table and leaves downstream meaning implicit. |
+| Keep all detail in `warnings.json` or separate it? | Separate grouped actionable warnings from grouped advisories/observations in `diagnostics.json`. | Keep the flat durable list and aggregate only CLI output; simpler migration, but it preserves the noisy downstream contract and ambiguous count. |
+| Who owns aggregation? | Framework runner/coordinator after origin enrichment. | Output module aggregation keeps framework types smaller but duplicates policy across encoders and cannot repair missing validator origin. |
+| Stable aggregation key? | Disposition, category, reason code, and full producer origin; exclude chunk/scope/message. | Explicit producer-supplied grouping keys offer flexibility but add another identity that can drift from reason codes. Message-template grouping is brittle and unsafe. |
+| Samples and omissions? | Exact occurrence count plus deterministic bounded samples and numeric omitted-sample count. | Omission warning records preserve the current representation but inflate group counts and require prose parsing. |
+| `warning_count` semantics? | Version receipt and replace ambiguity with group and occurrence counts. | Keep v1 count as published record length and add optional fields; compatible, but two competing warning counts remain easy to misuse. |
+| Which normalization changes remain warnings? | Only exhausted process fallback. Unresolved membership is a data-quality advisory; successful canonicalization, reordering, ID repair, source-ref dedupe, and duplicate consolidation are observations. | Treat unresolved membership or semantic duplicate consolidation as warnings because they affect grounding or cardinality; more conservative, but it violates the process-only warning rule and reports accepted artifact quality as an operational failure. |
+| Source-relatedness disposition? | Grouped advisory by default; preserve current approve behavior. | Retain warning disposition or make rejection configurable. Rejection requires production precision evidence; current lexical rules are not strong enough. |
+| Checkpoint-loaded warnings? | Present the same logical groups as fresh execution and use checkpoint events for reuse provenance. | Mark groups as replayed; aids forensics but fragments aggregation and makes semantically equivalent runs differ. |
+| ADR and schema versions? | Add an ADR and version the run receipt and diagnostic files. | Treat the work as CLI-only presentation and avoid an ADR; insufficient because module contracts, output files, checkpoint payloads, and downstream fields change. |
+
+## Compatibility, Documentation, And ADR Implications
+
+The target alters public and internal contracts enough to require a new ADR.
+It should record:
+
+- the distinction among warnings, advisories, observations, rejections, and
+  errors;
+- the invariant that warnings are process-level signals, LLM-judged extraction
+  quality is never a warning, and ordinary non-degraded success has zero
+  warnings;
+- module semantic ownership versus framework origin/aggregation ownership;
+- bounded group and sample semantics;
+- fresh/checkpoint equivalence; and
+- the output-encoder decision.
+
+Implementation should introduce `notarius.run-result.v2`. The grouped warning
+and diagnostic envelopes should each carry their own schema version. Because
+the content of `warnings.json` changes incompatibly from a flat array wrapper
+to groups, release notes and the published JSON integration contract must call
+out the migration. `index.json` gains the diagnostic file path.
+
+Canonical documentation updates belong in:
+
+- `docs/cli.md` for stderr presentation only;
+- `docs/operations.md` for operator review and debug workflow;
+- `docs/integrations/json-output.md` for warning and diagnostic file schemas;
+- `docs/integrations/run-result.md` for v2 fields and compatibility;
+- `docs/consumers/subprocess.md` and `docs/consumers/dnd-pipeline.md` for
+  downstream policy checks;
+- `docs/internal/pipeline.md` for promotion, aggregation, retry, and checkpoint
+  mechanics;
+- `docs/internal/modules.md` and `docs/internal/dnd.md` for producer rules and
+  the D&D classification matrix; and
+- `docs/policy/architecture.md` for the durable ownership invariant after the
+  ADR is accepted and implemented.
+
+No configuration knob is required for the first implementation. A fixed,
+well-documented taxonomy is easier to reason about than per-reason display
+overrides. Configurable escalation or suppression can be considered only after
+production review demonstrates a concrete operator need.
+
+## Test Coverage Assessment
+
+Existing coverage worth preserving includes:
+
+- accepted-attempt and terminal-rejection warning promotion;
+- validator failure retry exhaustion and `warn_continue`;
+- module semantic retry fallback;
+- deterministic warning order under concurrent lane completion;
+- chunk-plan invalidation and discarded-cache warning behavior;
+- fresh/checkpoint warning equivalence;
+- local D&D warning caps and safe dynamic-message quoting;
+- JSON warning-file publication; and
+- CLI stderr, debug, and receipt counts.
+
+Material gaps are:
+
+- no bound test for NPC- or spell-relatedness warnings;
+- no generic warning-field or result-size validation;
+- no test for output encoder warnings versus published `warnings.json`;
+- no operator-level aggregation or bounded-sample tests;
+- no fresh/resume test for grouped counts because groups do not yet exist; and
+- no production evaluation of advisory precision.
+
+Tests should protect the semantic relationships: exact occurrence counts,
+bounded samples, deterministic group order, actionable-only warning
+presentation, and cross-surface equality. They should not assert one exact
+warning count for every complete D&D run or treat message wording as a public
+API unless the wording itself enforces a security boundary.
+
+## Audit Conclusion
+
+The application is in a good position for warning reform. Its retry,
+validation, checkpoint, and concurrency mechanics provide reliable points at
+which to attach structured diagnostics. The most valuable change is not to
+suppress individual reason codes; it is to replace the semantically flat,
+origin-free collection with bounded typed groups and to reserve the word
+“warning” for conditions that merit operator attention.
+
+Provider-backed runs would improve prioritization and help tune the D&D
+advisories, but they are not necessary to conclude that routine normalization
+and heuristic doubt should not dominate stderr or the durable warning
+contract. They should be gathered before changing heuristic acceptance policy
+or adopting a numerical production warning-volume target.