From 610dd3d7c301dcd2389ade34ebcec6a95ae54a2a Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 27 Aug 2026 15:06:43 +0000 Subject: [PATCH] Plan warning and diagnostic reform --- docs/roadmap/audit-plan.md | 399 ++++++++++++++++++ docs/roadmap/audit.md | 648 ++++++++++++++++++++++++++++ docs/roadmap/implementation.md | 742 +++++++++++++++++++++++++++++++++ 3 files changed, 1789 insertions(+) create mode 100644 docs/roadmap/audit-plan.md create mode 100644 docs/roadmap/audit.md create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/audit-plan.md b/docs/roadmap/audit-plan.md new file mode 100644 index 00000000..bc609e6b --- /dev/null +++ b/docs/roadmap/audit-plan.md @@ -0,0 +1,399 @@ +# Warning Signal And Presentation Audit Plan + +## Purpose + +This document defines the audit required before redesigning Notarius warning +semantics and presentation. The audit must explain why an ordinary successful +D&D run can produce dozens of warnings, distinguish actionable degradation +from routine diagnostic observations, and recommend a bounded, deterministic +warning contract that operators will actually review. + +This is an audit plan, not a feature design or implementation plan. The audit +must first establish the current producers, propagation behavior, empirical +volume, public contracts, and compatibility constraints. The completed audit +selects the target behavior from those findings; `implementation.md` owns the +subsequent implementation sequence. + +## Context + +Notarius currently uses one `contracts.Warning` shape—optional `scope`, required +`reason_code`, and required `message`—for findings from modules, validators, +normalizers, reference preparation, retries, and framework policy. Accepted +warnings are collected by the pipeline, counted by the CLI and machine-readable +run receipt, and published as a flat array in `warnings.json`. + +The architecture already guarantees deterministic public ordering and bounded +warning production at several local boundaries. Recent validation work also +gave incomplete validation explicit provenance. Those guarantees must be +preserved. The observed problem is signal quality and aggregate volume: a +successful run can report many warnings even when most describe routine, +expected cleanup or weak heuristic observations rather than conditions that +require operator action. + +Initial repository inspection identifies several likely warning families: + +- D&D source-relatedness validators can produce one warning per artifact + record when a contextual name is not found in its cited text; +- deterministic normalizers report canonicalization, whitespace cleanup, + ordering changes, source-reference cleanup, and duplicate consolidation; +- semantic registry reconciliation reports discarded proposals, exhausted + fallback, and duplicate consolidation; +- optional reference materialization can report unavailable or skipped + reference inputs; +- the producer-attempt state machine reports validators that exhausted their + execution budgets under `warn_continue`; and +- producer modules can return their own warnings, which are promoted only from + the terminal accepted or rejected attempt. + +These are audit hypotheses. The audit must verify exact behavior and frequency +from current code and representative runs rather than assume that every family +is noisy or incorrectly classified. + +## Policy And Architecture Constraints + +The audit and its recommendations must follow: + +- `docs/policy/architecture.md` for module/framework ownership, deterministic + ordering, validation semantics, durable provenance, and sensitive-data + boundaries; +- `docs/policy/documentation.md` for canonical documentation ownership and the + distinction between current and future behavior; +- `docs/policy/testing.md` for risk-based, behavior-oriented test + recommendations; and +- ADR-0014 for the distinction between semantic rejection, validator failure, + correction guidance, and durable diagnostic provenance. + +In particular: + +- modules and validators may report findings, but the framework and output + boundaries own aggregation and presentation; +- reducing warning volume must not silently change artifact acceptance, + validation policy, retry behavior, rejection behavior, or process exit + status; +- genuine validator execution failures and validation-incomplete outcomes must + remain visible and auditable; +- public ordering must remain deterministic despite concurrent lane execution; +- aggregation must be bounded and must not leak raw model responses, + correction guidance, credentials, private reference content, or unnecessary + transcript text; and +- detailed diagnostics may move to a more appropriate durable or debug surface, + but must not be discarded when they are needed to understand a lossy or + degraded result. + +## Audit Objectives + +The audit must answer five questions. + +1. Which code paths produce warnings, under what conditions, and with what + expected multiplicity? +2. How are warnings cloned, bounded, replayed, ordered, promoted, persisted, + counted, and presented from their producer through the final CLI and output + bundle? +3. Which warnings identify actionable degradation or data-quality risk, and + which merely describe routine successful transformations or advisory + heuristics? +4. What aggregation and presentation model would materially reduce operator + noise without hiding genuine failures, lossy fallback, or incomplete + validation? +5. Which proposed changes affect only presentation, and which would alter a + public receipt, durable output contract, validation invariant, or other + compatibility boundary? + +## Scope + +### Warning Producers + +Inventory every production warning constructor and direct warning literal +under `internal/`. Include warnings originating from: + +- input, chunk, extract, merge, normalize, and output modules; +- typed, chunk, and serialized validators; +- semantic reconciliation and normalizer fallback; +- external and generated reference materialization; +- chunk-plan and checkpoint reuse or fallback; +- producer-attempt exhaustion and validation-incomplete continuation; +- pipeline orchestration and cancellation handling; and +- CLI or publication logic, if it creates warnings rather than only presenting + them. + +Do not infer completeness from one textual search. Inspect shared constructors, +returned result types, reason-code constants, registration paths, and tests that +exercise warning behavior. + +### Warning Propagation + +Trace each warning family through: + +- stage result and validation result contracts; +- producer attempts, including abandoned attempts, semantic retries, structural + retries, terminal rejection, and `warn_continue`; +- concurrent lane collection and stable public ordering; +- merge and normalize continuation; +- chunk-plan caching and checkpoint recording or hydration; +- ordered step output merging and generated-reference handoff; +- `RunOutput`, run manifests, validation summaries, debug records, and CLI + result construction; +- `warnings.json`, `manifest.json`, the run-result receipt, human-readable + standard error, and debug bundles. + +For every boundary, determine whether warnings are copied, filtered, +deduplicated, bounded, summarized, replayed from reusable state, or dropped. +Pay particular attention to amplification across chunks, lanes, validators, +retries, and resumed runs. + +### D&D Warning Semantics + +Review every implemented D&D artifact family. At minimum, distinguish: + +- evidence/source-relatedness advisories; +- deterministic canonicalization or whitespace changes; +- order and source-reference normalization; +- exact and semantic duplicate consolidation; +- unresolved catalog or registry membership; +- semantic-reconciliation proposal failure or fallback; and +- validator execution incompleteness. + +Determine whether the reason-code vocabulary is consistent across artifact +families, whether scopes are sufficiently contextual, and whether equivalent +conditions produce near-duplicate warnings with different codes or prose. + +### Public And Operator Surfaces + +Review the implemented contracts and documentation for: + +- CLI success output and warning-count output; +- `notarius.run-result.v1` and its `warning_count` and validation fields; +- `warnings.json` and the published JSON index; +- manifest validation summaries and rejection summaries; +- debug summary and detailed debug artifacts; and +- downstream subprocess guidance, especially the complete D&D consumer + workflow. + +Identify which surfaces are intended for immediate operator attention, durable +machine consumption, forensic detail, or debugging. Record any places where +the same flat count or warning list is being asked to serve incompatible +audiences. + +### Tests And Documentation + +Inventory tests that protect warning production, bounds, ordering, retry +promotion, checkpoint replay, JSON publication, receipt counts, and CLI stream +behavior. Identify meaningful gaps, redundant exact-prose assertions, and tests +that would unnecessarily obstruct a taxonomy or aggregation redesign. + +Review `docs/cli.md`, `docs/config.md`, `docs/operations.md`, +`docs/integrations/json-output.md`, `docs/integrations/run-result.md`, +`docs/consumers/`, and relevant internal documentation for current warning +claims. Record the canonical document that would own each future contract +change; do not rewrite those documents during the audit. + +## Out Of Scope + +The audit must not: + +- implement warning filtering, severity levels, aggregation, or new CLI flags; +- change validator decisions, default chains, retry budgets, or terminal + validation policy; +- suppress warnings merely to meet a numerical target; +- redesign rejections, errors, logs, metrics, or debug bundles except where + their boundary with warnings must be clarified; +- add the planned D&D combat-scene semantic validator; +- create deterministic tests that assert one exact global warning count for + all future runs; or +- use live paid LLM calls as part of the default automated test suite. + +## Inventory Method + +Create a warning inventory with one row per semantically distinct production +condition. Each row should record: + +| Field | Required analysis | +| --- | --- | +| Producer | Package, function, module or validator key, stage, and artifact family. | +| Trigger | Exact condition that emits the warning and whether it follows successful mutation, heuristic doubt, fallback, or failure. | +| Identity | Reason code, scope format, and whether either is stable enough for aggregation or machine use. | +| Multiplicity | Maximum per record, chunk, lane, validator, attempt, step, and run. | +| Lifecycle | Whether abandoned attempts are discarded, terminal warnings promoted, and cached or checkpointed warnings replayed. | +| Consequence | Whether data changed, evidence is questionable, output is incomplete, fallback occurred, or no externally meaningful consequence exists. | +| Actionability | What an operator can reasonably do in response, if anything. | +| Surfaces | CLI, receipt count, `warnings.json`, manifest, rejection, checkpoint, or debug presence. | +| Bounds | Existing local caps, message limits, omission records, and any missing aggregate bound. | +| Sensitivity | Whether scope or message can contain source-derived or otherwise sensitive content. | +| Coverage | Existing tests and the meaningful regression risk they protect. | + +Treat different reason codes that represent the same operator condition as +potential consolidation candidates, but do not merge them in the audit +document without explaining lost diagnostic information. + +## Propagation Analysis + +Produce a compact propagation map from warning creation to each terminal +surface. The analysis must explicitly verify: + +- only the terminal candidate's warnings are promoted after retries; +- whether rejected candidates retain warnings and where; +- whether `warn_continue` creates one warning per exhausted validator and how + that relates to validation summaries; +- whether cached chunk plans or hydrated checkpoints replay historical warnings + into the current run; +- whether the same warning can be appended at more than one handoff boundary; +- how concurrent completion is reordered before publication; +- whether local per-module caps compose into an unbounded or excessively large + run-level result; and +- whether warning counts on stderr, receipts, manifests, and `warnings.json` + refer to exactly the same collection. + +Any suspected duplicate append or unstable ordering is a correctness finding, +not merely a presentation concern. + +## Empirical D&D Run Analysis + +Static inspection must be supplemented with representative run evidence. Use +at least: + +- one ordinary successful complete D&D run known to produce high warning + volume; +- one smaller maintained example or synthetic run; +- one run with semantic registry reconciliation activity; +- one run with a validator execution failure allowed through + `warn_continue`, using an offline test double where practical; and +- one retrying producer case to verify abandoned-attempt warning treatment. + +For real campaign runs, analyze only bounded metadata unless the operator +explicitly provides source content for review. Record counts grouped by stage, +lane, module or validator, reason code, and scope family. Also record unique +reason-code count, repeated-message count, maximum group size, validation +status, rejected-output count, and whether each warning led to a plausible +operator action. + +Compare the same logical run under ordinary execution and checkpoint resume +when practical. The audit must distinguish warning volume caused by actual data +conditions from volume caused by orchestration or replay. + +Do not make a model-quality judgment solely from warning frequency. Manually +inspect a bounded sample from each high-volume reason code to estimate false +positive rate and operational value. + +## Classification Rubric + +Classify each warning condition along independent dimensions rather than force +an immediate single severity enum: + +- **result impact:** none, routine mutation, lossy mutation, uncertain data + quality, fallback, or incomplete validation; +- **operator action:** none, informational review, configuration or reference + correction, source/model review, or rerun required; +- **scope:** record, chunk, lane, step, pipeline, or infrastructure; +- **persistence need:** top-level attention, durable detail, debug-only detail, + or metric/trace candidate; and +- **confidence:** deterministic fact, heuristic advisory, or execution failure. + +The audit should then test whether a small durable taxonomy can represent the +meaningful combinations. A promising starting hypothesis is that top-level +operator warnings should be limited to actionable degradation, incomplete +validation, lossy fallback, and material data-quality risk, while routine +successful normalization observations remain available as lower-level durable +diagnostics. The audit must validate or revise that hypothesis from evidence. + +## Design Questions The Audit Must Resolve + +The findings must give a recommendation, with at least one viable alternative +and tradeoffs, for each of these questions: + +1. Should warning severity or disposition become an explicit contract field, + or should stable reason-code metadata drive presentation policy outside the + warning payload? +2. Should `warnings.json` remain the complete durable detail while the CLI and + receipt expose an aggregated actionable summary, or should durable warnings + themselves be separated from routine observations? +3. Where should global deduplication and aggregation live so modules retain + semantic ownership but concurrent pipeline results remain deterministic? +4. What is the stable aggregation key: reason code, stage/lane/module identity, + normalized scope, message template, or an explicit structured grouping key? +5. How should bounded samples and omitted counts be represented without + converting a summary record into another warning that inflates the count? +6. Should `warning_count` continue to mean the length of `warnings.json`, or + should a new receipt or schema field distinguish actionable warning groups + from detailed observations? +7. Which normalization changes are sufficiently lossy or surprising to remain + warnings, and which are ordinary provenance that belongs in manifest or + debug data? +8. Should heuristic source-relatedness findings remain warnings, become + grouped data-quality advisories, or be strengthened into configurable + validation decisions only after demonstrated precision? +9. How should warnings loaded from checkpoints be identified or aggregated + relative to newly produced warnings? +10. Does the chosen target alter durable or validation semantics enough to + require a new ADR or a versioned run-result/output contract? + +## Required Audit Deliverable + +Write the completed findings to `docs/roadmap/audit.md`. It should contain: + +1. an executive assessment of current warning quality and risk; +2. the complete warning-producer inventory; +3. the warning propagation and surface map; +4. empirical measurements and bounded representative samples; +5. findings ranked by operator impact, correctness risk, and implementation + leverage; +6. a recommended target taxonomy and presentation model; +7. compatibility, documentation, ADR, and migration implications; +8. implementation implications and dependencies sufficient to support a + separate implementation plan; and +9. open decisions only where repository evidence cannot support a responsible + recommendation. + +Each finding should identify the supporting code paths, tests, documentation, +and run evidence. Separate observed facts from recommendations and avoid +changelog or development-history framing. + +## High-Level Audit Sequence + +The detailed execution sequence should be written separately if needed. At a +high level, perform the audit in this order: + +1. **Static inventory:** enumerate warning producers, reason codes, scopes, + bounds, and existing tests. +2. **Propagation audit:** trace promotion, ordering, replay, persistence, + counting, and presentation across the framework and CLI. +3. **Empirical analysis:** measure representative D&D runs and inspect bounded + samples from high-volume warning groups. +4. **Classification:** apply the rubric, identify duplicate concepts and + misplaced routine diagnostics, and evaluate public-contract options. +5. **Synthesis:** rank findings and recommend a decision-complete target for a + subsequent feature roadmap. + +Static inventory and propagation may be performed as separate focused agent +prompts. Empirical analysis should be isolated because it may require operator +artifacts or opt-in provider execution. Classification and synthesis should +consume the earlier written evidence rather than rediscover the repository. + +## Validation Of The Audit + +Before considering the audit complete, verify that: + +- every production warning literal or constructor is represented in the + inventory; +- every reason code observed in representative `warnings.json` files maps to a + known producer or is recorded as an unexplained finding; +- counts agree across the runner result, CLI receipt, stderr summary, and + published warning collection for each sampled run; +- retry, rejection, incomplete-validation, cache, checkpoint, concurrency, and + ordered-step paths are covered; +- recommended aggregation preserves deterministic ordering and bounded memory; +- recommendations distinguish warnings from errors, rejections, validation + summaries, logs, and debug diagnostics; +- no recommendation hides a condition that changes output completeness or + correctness; +- public compatibility and schema-version consequences are explicit; and +- proposed tests protect meaningful behavior without asserting incidental + prose or one permanently fixed global warning count. + +## Completion Criteria + +The audit is ready to become a feature roadmap when it can explain the current +high warning count quantitatively, identify the dominant producers and any +amplification defects, classify every warning family by consequence and +actionability, and recommend where each class should appear. The findings must +be specific enough that a later roadmap can define the target contract without +repeating the discovery work. diff --git a/docs/roadmap/audit.md b/docs/roadmap/audit.md new file mode 100644 index 00000000..c9bbc634 --- /dev/null +++ b/docs/roadmap/audit.md @@ -0,0 +1,648 @@ +# Warning Signal And Presentation Audit + +## Executive Assessment + +Notarius warning execution is mechanically stronger than its operator-facing +presentation. Terminal-attempt promotion, stable ordering after concurrent +work, checkpoint replay, validation summaries, and debug retention are all +substantially correct. The audit found no general duplicate-append defect in +the extract, merge, or normalize handoffs and no leakage of abandoned-attempt +warnings into a successful result. + +The warning channel itself is not coherent. One flat `contracts.Warning` type +currently represents at least four materially different concepts: + +- actionable degradation or incomplete validation; +- heuristic data-quality doubt; +- successful but potentially reviewable fallback; and +- routine canonicalization, ordering, and deduplication observations. + +That conflation is the primary reason successful runs produce a count that is +large but operationally weak. The maintained complete D&D example demonstrates +the problem without a live provider: an approved run with no rejected outputs +published 12 warning records, all from three advisory relatedness checks. An +operator separately reported a successful complete D&D run with 10 outputs, +one rejection, and 85 warnings. The production bundle for that run was not +available in this environment, so its reason-code distribution could not be +measured. + +The current implementation also has four correctness or robustness gaps: + +1. warning records lose stage, step, lane, module, validator, and chunk + provenance when promoted, which makes safe aggregation and diagnosis + impossible from `warnings.json` alone; +2. there is no framework-level validation or aggregate bound, and the NPC- and + spell-relatedness validators bypass the D&D warning limiter entirely; +3. warnings returned by an output encoder are added after `warnings.json` has + already been encoded, so the receipt, stderr, debug bundle, and published + warning file can disagree; and +4. a skipped validator contributes to `incomplete` validation but does not + receive the warning generated for an exhausted validator failure. + +The recommended end state is a structured diagnostic contract with explicit +disposition, category, origin, occurrence count, and bounded samples. Warnings +are reserved for process-level degradation or incompleteness. LLM-judged or +deterministically inferred extraction-quality signals are advisories, never +warnings, and routine normalization observations remain inspectable without +being reported as top-level warnings. An ordinary successful run in which all +configured work completes normally should therefore report zero warnings. This +is an architectural and durable-contract change, not merely revised CLI prose. + +## Evidence And Limits + +The audit used: + +- a complete static search of production `contracts.Warning` constructors, + reason-code constants, result fields, and promotion sites under `internal/`; +- call-path inspection through producer attempts, validators, lane + coordination, chunk-plan reuse, checkpoints, output encoding, debug output, + CLI presentation, and run-result construction; +- the maintained complete and minimal D&D examples with offline fake LLMs; +- focused deterministic tests for warning bounds, semantic-reconciliation + fallback, `warn_continue`, semantic retries, terminal rejection, concurrency + ordering, and checkpoint reuse; and +- the operator-provided observation of an 85-warning complete D&D run. + +No provider-backed production run was attempted because this environment has +no API key. Consequently, the audit can establish warning mechanics, possible +multiplicity, synthetic volume, and obvious heuristic limitations, but cannot +estimate production frequency or the real false-positive rate of individual +D&D advisories. Those measurements are not required to choose the recommended +architecture; they are required before strengthening any heuristic advisory +into a rejection or setting a numerical production acceptance target. + +## Complete Warning-Producer Inventory + +### Framework And Generic Boundaries + +| Producer | Reason code | Trigger and consequence | Multiplicity and bound | Current surfaces and coverage | +| --- | --- | --- | --- | --- | +| Reference materialization in `internal/framework/pipeline/references.go` | `empty_reference` | A bound external reference is a valid, accepted media type but contains zero bytes. The prompt may receive materially incomplete context. | One per empty bound file; finite by configuration but no shared run-level cap. | Enters `RunInput.Warnings`; reference tests protect contextual scope. | +| Producer-attempt policy in `internal/framework/pipeline/producer_attempts.go` | `validator_execution_incomplete` | An applicable validator exhausted its execution budget and `warn_continue` accepted the otherwise valid candidate. | One per failed validator on each terminal candidate. An extract chain can multiply this by chunks and lanes. There is no global cap. | Durable warning, receipt count, stderr, debug, and validation summary. `TestWarnContinueRecordsOneWarningForEachExhaustedValidator` covers failures. | +| Chunk, extract, merge, normalize, and output module result contracts | Module-defined | A module may return arbitrary warnings with its successful candidate. | No contract validation, message limit, per-result cap, or global cap. Current production modules are inventoried below. | Terminal-attempt filtering and concurrency ordering are well tested. | +| Production JSON output encoder | None | The encoder copies incoming warnings into `warnings.json`; it does not currently create warnings. | Same count as its input. | JSON encoder and assembled-pipeline tests compare the incoming run warnings with the published file. | +| Output encoder result contract | Module-defined | Any output encoder may return warnings discovered during encoding. | Unbounded by contract. No production encoder currently exercises this capability. | Appended to final `RunOutput` only after logical files were encoded; this is the cross-surface defect described in AUD-WARN-004. | + +Input adapters and production mergers do not currently have independent +warning producers. Chunk-plan and checkpoint decisions are structured manifest +or debug provenance rather than warnings. Cancellation and hard persistence, +reference, parsing, serialization, and provider failures remain errors. + +### D&D Extraction Gates + +| Producer | Reason code | Trigger and consequence | Multiplicity and bound | +| --- | --- | --- | --- | +| `dnd/combat-turns` extractor | `scene_classification_unavailable` | The chunk has no exact matching scene-description classification. The extractor returns an empty accepted result and skips the LLM, so combat-turn output may be incomplete. | At most one per chunk for this lane. | +| `dnd/enemy-events` extractor | `scene_classification_unavailable` | The same missing or mismatched scene gate causes accepted empty enemy-event output. | At most one per chunk for this lane. | + +An exact non-combat classification produces an intentional empty result without +a warning. An exact combat classification proceeds normally. The two producers +share a code and operator consequence but use different messages; their module +origins are not retained in the final warning record. + +### D&D Source-Relatedness Validators + +All ten relatedness validators are deterministic advisories: they approve the +candidate and warn when contextual prose or an entity name is not lexically +present in cited text. Shape and source-reference failures are deliberately +left to blocking validators earlier in the chain. The same relatedness +validator is registered in both the extract and normalize default chain for +each artifact family in `internal/modules/dnd/register/chains.go`. + +| Artifact family | Warning reason | Per-record trigger | Local bound | Omission reason | +| --- | --- | --- | --- | --- | +| Combat turns | `combat_turn_not_near_source` | Actor token sequence absent | 20 per validator invocation | `combat_turn_relatedness_warnings_omitted` | +| Enemy events | `enemy_event_not_near_source` | Subject token sequence absent | 20 | `enemy_event_relatedness_warnings_omitted` | +| Item occurrences | `item_occurrence_source_unrelated` | Item name token sequence absent | 20 | `item_occurrence_relatedness_warnings_omitted` | +| Item registry | `item_not_near_source` | Item name token sequence absent | 20 | `item_relatedness_warnings_omitted` | +| Location occurrences | `location_occurrence_not_near_source` | Location name token sequence absent | 20 | `location_occurrence_relatedness_warnings_omitted` | +| Location registry | `location_not_near_source` | Location name token sequence absent | 20 | `location_relatedness_warnings_omitted` | +| NPC occurrences | `npc_occurrence_not_near_source` | NPC name token sequence absent | 20 | `npc_occurrence_relatedness_warnings_omitted` | +| NPC registry | `npc_not_near_source` | NPC name token sequence absent | **Unbounded** | None | +| Scene descriptions | `scene_description_not_near_source` | No significant title or summary token appears; up to two findings per scene | 20 | `scene_description_relatedness_warnings_omitted` | +| Spells | `spell_not_near_source` | Spell-name token sequence absent | **Unbounded** | None | + +The eight limiter-generated omission records are presentation artifacts, not +new source-relatedness conditions. They occupy a warning slot and make list +length differ from the actual occurrence count. + +### D&D Normalizers + +Every production D&D normalizer bounds its returned warning slice to 20 through +`internal/modules/dnd/shared/diagnostics`, including a final omission record +when needed. Registry semantic retries reserve space for their fallback +warning. The following table is complete by semantically distinct condition; +codes listed together are parallel artifact-family variants. + +| Condition | Reason codes | Result impact | Current classification assessment | +| --- | --- | --- | --- | +| Display or field whitespace/name canonicalization | `npc_fields_normalized`, `item_fields_normalized`, `location_fields_normalized`, `spell_name_canonicalized`, `combat_actor_canonicalized`, `enemy_event_name_canonicalized`, `item_occurrence_name_canonicalized`, `location_occurrence_name_canonicalized`, `scene_description_prose_normalized` | Deterministic successful mutation. The item-occurrence code can also describe `from`/`to` whitespace, not only the item name. | Routine observation. | +| Durable ID recomputation | `npc_id_recomputed`, `item_id_recomputed`, `location_id_recomputed` | Restores the deterministic name-derived ID. | Routine observation; invalid identity is separately rejected by default chains. | +| Source-reference sorting or deduplication | `source_references_normalized` | Sorts and removes exact duplicate references while deliberately preserving invalid references for their validators. | Routine observation. Shared code is useful but ambiguous without producer origin. | +| Canonical record ordering | `combat_turns_reordered`, `enemy_events_reordered`, `item_occurrences_reordered`, `location_occurrences_reordered`, `npc_occurrences_reordered`, `scene_description_order_normalized` | Deterministic order changes only. | Routine observation. | +| Exact or approved semantic duplicate consolidation | `duplicate_npc_collapsed`, `duplicate_item_collapsed`, `duplicate_location_collapsed`, `duplicate_spell_cast_collapsed`, `duplicate_combat_turn_collapsed`, `duplicate_enemy_event_collapsed`, `duplicate_item_occurrence_collapsed`, `duplicate_location_occurrence_collapsed`, `duplicate_npc_occurrence_collapsed`, `scene_description_duplicate_collapsed` | Removes duplicate records and preserves or combines canonical evidence according to the artifact policy. Registry codes cover both exact and accepted semantic consolidation. | Durable normalization observation; not normally operator-actionable. | +| Unresolved external membership | `spell_name_unresolved`, `item_occurrence_unknown_item_id`, `location_occurrence_unknown_location_id` | The value is preserved but is not grounded in the effective catalog or registry. Default chains normally reject the same condition before normalization; it remains reachable with validator overrides or defensive direct use. | Actionable data-quality warning. | +| Unsafe currency consolidation proposal | `item_semantic_proposal_invalid` | The proposed group is rejected and all records are preserved because denominations or currency/non-currency members are incompatible. The same code is also used internally as a retry reason. | Advisory about model proposal quality; no accepted-data loss. The control and diagnostic meanings should be separated. | +| Semantic reconciliation unavailable or exhausted | `npc_semantic_reconciliation_exhausted`, `item_semantic_reconciliation_exhausted`, `location_semantic_reconciliation_exhausted` | The safe deterministic result is accepted, but possible semantic duplicates remain. | Actionable fallback warning. | +| Local warning truncation | `npc_normalization_warnings_omitted`, `item_normalization_warnings_omitted`, `location_normalization_warnings_omitted`, `spell_normalization_warnings_omitted`, `combat_turn_normalization_warnings_omitted`, `enemy_event_normalization_warnings_omitted`, `item_occurrence_normalization_warnings_omitted`, `location_occurrence_normalization_warnings_omitted`, `npc_occurrence_normalization_warnings_omitted`, `scene_description_normalization_warnings_omitted` | Reports that individual records were omitted from presentation. | Group metadata, not an independent warning. | + +No production D&D normalization warning exposes raw model responses, +correction guidance, or provider errors. Most dynamic names are quoted and +truncated by the shared helper. That local discipline is not enforced by the +generic warning contract, and the spell relatedness message does not use the +shared truncation helper. + +## Warning Propagation And Surface Map + +```text +external-reference warnings -----------------------------+ + | +module candidate warnings -> validation chain warnings | + | | | + +---- producer-attempt terminal policy -------+ + | | + accepted / terminal rejection only | + | | + chunk or lane result in canonical order | + | | + checkpoint record/replay and ordered step merge | + | | + RunOutput.Warnings <------------+ + | + OutputRequest -> output encoder + | | + warnings.json OutputResult.Warnings + | + appended to final RunOutput only + | + receipt, stderr, final debug warning summary +``` + +### Attempts And Validation + +- `runProducerAttempts` promotes only the terminal accepted or terminal + rejected candidate's module and completed-validator warnings. Operational, + structural, semantic, and module-directed attempts that are superseded are + retained in attempt debug artifacts but not in the final collection. +- A module-directed semantic-reconciliation retry adds its fallback warning + only when no retry remains. Earlier attempt warnings are discarded. +- On `warn_continue`, warnings from the otherwise accepted candidate and + completed approved or rejected validators are retained. One fixed, + non-sensitive `validator_execution_incomplete` warning is added for every + failed validator. Skipped validators affect the validation summary and final + `incomplete` status but do not receive such a warning. +- A semantic terminal rejection retains only warnings from that rejected + attempt. Structural rejection after producer failure cannot retain a + candidate warning because no valid candidate result exists. + +These behaviors are protected by the producer-attempt, extract-handoff, +rejection-warning, normalize-retry, and attempt-debug tests. They are the right +foundation for the redesign and should not be replaced with early-exit or +all-attempt accumulation. + +### Concurrency And Ordering + +Extract jobs are dispatched chunk-first and lane-second. Results are stored by +chunk index, finalized in ascending chunk order, and lane continuations are +merged into a slice indexed by configured lane order. Pipeline steps run in +configured order. The resulting public order is therefore: + +1. pre-run reference warnings; +2. chunk-stage warnings; +3. step order; +4. configured lane order within each step; +5. chunk order within extract; +6. merge warnings; then +7. normalize warnings; followed by any output-result warnings. + +`TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion` exercises warning +order under reversed completion. No completion-order leak was found. + +### Chunk Plans And Checkpoints + +- A reusable chunk plan stores producer warnings only. Current validators run + again, and their current warnings are appended. Warnings from a cached plan + candidate that fails current validation are discarded before regeneration. +- Accepted extract, merge, and normalize checkpoints store the terminal + warnings for their stage. Reuse loads and appends those warnings once at the + same logical handoff. Tests compare fresh and resumed warning collections and + preserve their order. +- Validation-incomplete accepted outputs are not reusable, preventing a later + run from silently treating incomplete validation as complete. +- Required accepted-normalize hydration replays that normalize checkpoint's + warnings; checkpoint decisions separately expose that reuse occurred. + +The recommended redesign should keep fresh and resumed logical diagnostics +equivalent. Whether a result was reused belongs in checkpoint provenance, not +in the diagnostic grouping key; adding a `reused` distinction would fragment +groups and make equivalent runs present differently. + +### Terminal Surfaces + +| Surface | Current content | Audience | Audit result | +| --- | --- | --- | --- | +| `RunOutput.Warnings` | Flat final slice | Framework and CLI | Canonical in-memory list, but lacks origin and bounds. | +| Published `warnings.json` | Object containing the warnings passed into the output encoder | Durable consumers | Exact for the production JSON encoder unless the encoder itself returns warnings. | +| `index.json` | Path to `warnings.json` | Durable consumers | Stable discovery path; no separate diagnostic-detail path. | +| Run-result v1 | `warning_count = len(final RunOutput.Warnings)` | Subprocess callers | Count only; no group/occurrence distinction. | +| Human stderr | `run completed with N warning(s)` | Operators | Count only and no direct detail path. Successful exit remains zero. | +| Manifest | Validation and rejection summaries, no warning collection | Durable provenance | Correctly avoids duplicating the flat list. | +| Debug summary `warnings.json` | Raw final warning array | Operators/developers | Includes final output-result warnings and can therefore differ from published `warnings.json`. | +| Debug run report | Final warning count | Operators/developers | Same final slice length as receipt and stderr. | +| Attempt/stage debug | Candidate-local warning detail and origin in path/envelope | Forensics | Sufficient to diagnose provenance, but debug capture is optional and is not a durable consumer contract. | + +## Empirical Measurements + +### Offline And Synthetic Runs + +| Scenario | Result | What it establishes | +| --- | --- | --- | +| Maintained complete D&D config and transcript with the repository's offline fake LLM | Approved, 10 normalized outputs, 0 rejected outputs, 12 warnings; receipt, stderr, and published file all reported 12 | An ordinary structurally successful workflow can be noisy without fallback or incomplete validation. | +| Same complete run, grouped after publication | Three reason codes, seven exact `(reason, scope, message)` tuples, maximum exact-tuple repetition of three | The flat count materially overstates distinct operator conditions. Scope resets within chunks and does not identify origin. | +| Maintained focused scene-description workflow | Approved, one normalized output, 0 warnings | The warning channel can be quiet when synthetic model text is lexically grounded. | +| Generic warning publication contract | One warning reaches successful stderr, durable output, and debug summary | The ordinary pre-output path is consistent. | +| NPC semantic-reconciliation candidate-limit fallback | No LLM call, all records preserved, one exhaustion warning, total warnings no greater than 20 | Fallback is bounded and materially different from routine normalization. | +| `warn_continue` with two failed validators and one skipped validator | Validation status contains all three incomplete validators; warning slice contains two execution-incomplete records | Current warning count does not describe all incomplete validation. | +| Retrying extract candidate | Two producer attempts; only the accepted attempt's one warning is final | Retry does not amplify abandoned warnings. | +| Terminal semantic rejection | Only the final rejected attempt's operation and validator warnings are final | Rejection diagnostics are retained without retaining superseded warnings. | +| Fresh versus reused extract checkpoint | Warning collections are deeply equal | Checkpoint replay does not itself amplify warnings. | +| Spell normalizer with 21 unresolved entries | 20 records: 19 samples plus one omission record saying two additional warnings were omitted | `warning_count` is neither exact occurrence count nor distinct-condition count. | + +The focused audit tests passed in `internal/cli`, +`internal/framework/pipeline`, the NPC-registry and spell normalizers, and all +D&D packages. + +### Bounded Sample Review + +The complete offline D&D run produced: + +| Reason | Count | Sample | Review | +| --- | ---: | --- | --- | +| `location_not_near_source` | 4 | `Moon Gate` was absent from cited text | Correctly identifies deliberately unsupported fake output. Three records shared the same exact tuple because chunk and stage origin were lost. | +| `location_occurrence_not_near_source` | 4 | A `Moon Gate` visit was absent from cited text | Correctly identifies the same unsupported registry-driven occurrence, but repeats the same operator concern across extraction and normalization. | +| `scene_description_not_near_source` | 4 | A title or summary had no significant exact token in cited text | Mixed value. Generic `session scene` prose is ungrounded, while `Arrival` versus transcript `arrive` illustrates an expected lexical false positive. | + +This fake workflow is an integration fixture, not a model-quality benchmark. +It nonetheless proves that the checks carry useful evidence while being too +imprecise and repetitive to serve as one-warning-per-record operator alerts. + +### Production Evidence Still Needed + +The reported 85-warning run establishes that high volume occurs in practice, +but the following remain unknown: + +- dominant production reason codes and stage/lane sources; +- unique group count versus repeated occurrence count; +- false-positive rate for each relatedness family; +- how much volume comes from normalization observations versus advisories; +- whether fresh and resumed production runs remain equivalent; and +- a defensible numerical acceptance target. + +If further data is worthwhile, the operator can supply the v1 receipt, +`warnings.json`, and manifest validation summaries without supplying transcript +or lane artifacts. An initial privacy-preserving report should group by reason +code and normalized scope family, count exact repeated tuples, and omit message +text. Reviewing heuristic precision requires a separately approved bounded +sample with its cited source context. + +## Classification Of Current Conditions + +| Target disposition | Current families | Result impact | Operator action | Durable placement | +| --- | --- | --- | --- | --- | +| **Warning** | Empty reference; validator failure or skip accepted under `warn_continue`; unavailable required scene classification; exhausted semantic reconciliation | A configured process completed under an allowed degraded or incomplete policy rather than completing normally | Correct reference/configuration, inspect provider/validator, or rerun | Actionable `warnings.json`, receipt summary, stderr summary, debug | +| **Advisory** | Source-relatedness heuristics; unresolved spell or registry membership; guarded invalid semantic proposal; any future LLM-judged uncertainty or extraction-quality signal | Uncertain data quality or poor model proposal, but accepted data is structurally valid and deterministic guards prevented unsafe mutation | Optional model/source review; no routine action for every record | Durable diagnostic detail and debug; never a top-level warning | +| **Observation** | Whitespace/name/ID/source-reference canonicalization; canonical ordering; exact and approved semantic duplicate consolidation | Successful intended normalization | None under normal operation | Durable bounded normalization diagnostics or debug; no stderr warning | +| **Not a diagnostic** | Rejection, invalid structure, cancellation, persistence error, provider failure under fail-run policy | Candidate or run did not complete according to policy | Inspect rejection/error and retry or correct input/configuration | Existing rejection, validation summary, error, and debug contracts | + +Exact and semantic duplicate consolidation should remain distinguishable in +category or reason metadata even though both are observations. Semantic +reconciliation exhaustion remains a warning because a capability was not +applied; successful approved consolidation is an observation because it is the +normalizer's intended work. + +## Ranked Findings + +### AUD-WARN-001 — The Flat Warning Type Destroys Signal Quality + +- **Priority:** High operator impact; high implementation leverage. +- **Evidence:** `contracts.Warning` has only scope, reason, and message. Routine + normalizer changes, heuristic doubt, fallback, and incomplete validation all + enter the same slice and the same CLI count. The offline complete run's 12 + records were all advisories; the operator observed 85 records in a successful + real run. +- **Impact:** Operators cannot tell whether a warning requires a rerun, a + configuration repair, optional review, or no action. Repeated routine output + trains them to ignore the channel. +- **Recommendation:** Replace the flat semantic contract with explicit + `warning`, `advisory`, and `observation` dispositions plus a small category + vocabulary. Do not infer disposition from message text or require every + downstream consumer to maintain a reason-code policy table. + +### AUD-WARN-002 — Warning Records Lose The Origin Needed For Diagnosis And Aggregation + +- **Priority:** High correctness and usability impact. +- **Evidence:** The runner knows stage, step, lane, module, validator, chunk ID, + and chunk index at promotion time, but `terminalWarnings` flattens module and + validator records into `[]contracts.Warning`. Per-chunk scopes such as + `locations[0]` and `occurrences[0]` then repeat without identifying their + chunk or producer. `source_references_normalized` is intentionally shared + across families and is therefore especially ambiguous. +- **Impact:** `warnings.json` cannot answer which stage or module produced a + record. Message- or scope-based deduplication would merge unrelated findings + or retain accidental duplicates. +- **Recommendation:** Keep module findings free of framework context, then have + the framework add a structured origin envelope before promotion. Validator + findings must retain validator identity instead of passing through + `validationReport.Warnings()` as a flat slice. + +### AUD-WARN-003 — Warning Volume Is Not End-To-End Bounded Or Validated + +- **Priority:** High robustness impact; medium immediate likelihood. +- **Evidence:** D&D's `LimitWarnings` caps most individual producers at 20, but + NPC- and spell-relatedness return one warning per record without the helper. + Every extract validator is invoked per chunk, all ten relatedness checks run + again after normalization, and there is no run-level collector. The generic + contract validates neither disposition nor reason/message size, UTF-8, + blankness, or total records. +- **Impact:** Warning memory and output grow with chunks, lanes, configured + validators, and record counts. Local omission records lose exact occurrence + semantics while still incrementing `warning_count`. +- **Recommendation:** Add a generic bounded diagnostic collector that preserves + exact occurrence counts and bounded samples. Validate all diagnostic fields + at the module/framework boundary. Immediately bring NPC and spell + relatedness under the existing cap if the full redesign is staged. + +### AUD-WARN-004 — Output Encoder Warnings Make Durable Surfaces Disagree + +- **Priority:** Medium current impact; high contract correctness risk. +- **Evidence:** `Runner.Run` passes existing warnings to `encoder.Encode`, then + the production encoder serializes `warnings.json`. Only after encoding does + the runner append `OutputResult.Warnings`. The receipt, stderr, debug summary, + and debug run report see the final slice; the already-created published file + cannot. No production encoder currently returns a warning, so ordinary JSON + runs do not trigger the defect. +- **Impact:** A valid output-module implementation can violate the documented + claim that `warning_count` describes the published warning collection. +- **Recommendation:** Remove successful output warnings from the output-module + contract unless a demonstrated use case requires them; encoding failures + should be errors and optional encoder observations should be debug data. A + two-phase finalize API is the viable but more complex alternative. + +### AUD-WARN-005 — Validation Skips Are Incomplete But Not Warned + +- **Priority:** Medium operator/correctness impact. +- **Evidence:** `firstIncompleteValidation` treats failed and skipped validators + alike, and validation summaries include both. `incompleteValidationWarnings` + emits records only for `validationFailed`. The focused test demonstrates + three incomplete validators but two warnings. +- **Impact:** A successful run can have `validation_status: incomplete` while + its warning count understates or even omits the affected validators. A caller + that checks only warnings receives a weaker signal than the manifest and + receipt status. +- **Recommendation:** Produce one aggregated incomplete-validation warning + group whose occurrences cover both failure and skip, while retaining typed + outcome and safe reason metadata in the validation summary. Do not expose + provider errors or arbitrary skip prose in model or operator messages. + +### AUD-WARN-006 — Relatedness Checks Are Useful But Repetitive And Lexically Weak + +- **Priority:** Medium operator impact; low acceptance-policy urgency. +- **Evidence:** Every family runs the advisory in both extract and normalize + chains. The complete fixture contains exact repeated tuples, and the checks + rely on exact normalized token sequences or a minimal significant-token + overlap. Reason naming drifts between `*_not_near_source` and + `*_source_unrelated`. +- **Impact:** The checks can catch unsupported entities, but aliases, pronouns, + inflection, and generic scene prose create predictable false positives. Flat + per-record presentation magnifies them. +- **Recommendation:** Retain the validators and their stage-local execution, + but classify and aggregate them as advisories. Normalize reason-code naming + when the diagnostic contract changes. Do not strengthen them into rejection + rules without a human-reviewed production evaluation. + +### AUD-WARN-007 — `warning_count` Has No Stable Operational Meaning + +- **Priority:** High downstream-contract impact. +- **Evidence:** The receipt and CLI use `len(output.Warnings)`. One list element + can be an omission summary representing several hidden occurrences; repeated + records can represent the same condition; and skipped validators can be + absent. A 21-occurrence spell test produces a list length of 20. +- **Impact:** The value is neither an exact occurrence count nor a distinct + warning-group count. Consumers cannot set policy or present a trustworthy + summary from it. +- **Recommendation:** Introduce explicit warning-group and warning-occurrence + counts in a versioned receipt. Do not silently redefine the v1 field. + +## Recommended Target Contract And Presentation Model + +### Diagnostic Model + +Use one validated internal diagnostic model with these concepts: + +- **disposition:** `warning`, `advisory`, or `observation`; +- **category:** a small enum such as `configuration`, `degradation`, + `validation_incomplete`, `data_quality`, `fallback`, or `normalization`; +- **reason code:** stable semantic identity owned by the producer; +- **origin:** framework-added phase/stage, step ID, lane ID, module key, + validator name, chunk ID, and chunk index when applicable; +- **occurrence count:** exact number of matching findings; +- **samples:** a small deterministic list of bounded scope/message pairs; and +- **omitted sample count:** `occurrence_count - len(samples)`, represented as + metadata rather than another diagnostic record. + +Errors and rejected outputs must not become diagnostic dispositions. A warning +means that the run completed under policy despite a process-level degradation +or incomplete configured operation. Advisory and observation dispositions can +describe accepted artifact quality and transformation provenance, but no +LLM-judged extraction-quality signal may be promoted to a warning. Validation +status remains authoritative for approval, rejection, and incomplete +validation. + +### Aggregation + +The framework runner should own aggregation after it enriches findings with +origin and before public output construction. Modules and validators retain +semantic ownership of disposition, category, reason, scope, and message; they +must not own CLI or file presentation. + +The default stable key should be: + +```text +disposition + category + reason_code ++ phase/stage + step_id + lane_id + module_key + validator_name +``` + +Chunk, record scope, and message text belong in samples and must not be part of +the group key. This groups repeated per-chunk findings without merging the same +code across distinct producers or pipeline locations. Group order should be +the first occurrence in the runner's existing canonical order; sample order +should follow the same order. A final canonical sort by the complete origin key +is also viable, but completion timing must never choose either order. + +Aggregation must be incremental and bounded. Producers should use a shared +collector that counts every occurrence while retaining only bounded samples; +the framework then merges producer groups without reconstructing counts from +omission prose. A global maximum group count is also required, with overflow +represented by structured aggregate metadata and with actionable groups given +priority over lower dispositions. + +### Durable Files + +Keep one canonical home for each class: + +- `warnings.json` should contain versioned, grouped actionable warnings only; +- a new `diagnostics.json` should contain versioned advisory and observation + groups only, avoiding duplication of warning groups; +- `index.json` should link both files; +- `rejected.json` and manifest validation summaries should retain their current + separate responsibilities; and +- debug bundles should retain candidate-attempt detail plus the final grouped + projections. + +This is preferable to keeping all detail in `warnings.json` and filtering only +the CLI: downstream consumers would otherwise continue to receive a semantically +mixed warning contract, and routine observations would still dominate the +durable file. + +### CLI And Receipt + +For a successful human run with actionable warnings, print a concise summary +such as: + +```text +notarius: run completed with 2 warning groups (7 occurrences); details=/.../warnings.json +``` + +Advisories and observations should not produce the warning line. Their durable +path remains discoverable through `index.json`; a concise non-warning count can +be added to the ordinary success line only if operator testing shows value. An +ordinary successful run with no process degradation should write nothing to +the warning stream even when it publishes quality advisories or normalization +observations. + +Create `notarius.run-result.v2` rather than redefining v1. It should expose at +least: + +- `warning_group_count`; +- `warning_occurrence_count`; and +- `diagnostic_group_count` for non-warning durable groups. + +The receipt should continue to expose validation status, validation summaries, +and rejected-output count independently. Process exit behavior should not +change as part of warning presentation reform. + +### Checkpoint Semantics + +Store the structured terminal diagnostic groups with accepted checkpoints and +replay them exactly once at their logical stage. Fresh and reused runs should +produce the same public groups and counts. Checkpoint events and debug records, +not diagnostic identity, should disclose whether computation was reused. + +### Output Encoder Boundary + +Prefer removing `OutputResult.Warnings`. A successful output encoder should +either return the complete logical files or fail. If future encoders genuinely +need to produce durable post-encoding warnings, introduce an explicit +two-phase prepare/finalize contract so those warnings can be included in the +same published collection. Do not retain the current self-inconsistent +one-pass capability. + +## Resolution Of Required Design Questions + +| Question | Recommendation | Viable alternative and tradeoff | +| --- | --- | --- | +| Explicit severity/disposition or external reason mapping? | Put validated disposition and category in the contract. | A central reason-code registry avoids payload fields but makes new modules depend on a second synchronized policy table and leaves downstream meaning implicit. | +| Keep all detail in `warnings.json` or separate it? | Separate grouped actionable warnings from grouped advisories/observations in `diagnostics.json`. | Keep the flat durable list and aggregate only CLI output; simpler migration, but it preserves the noisy downstream contract and ambiguous count. | +| Who owns aggregation? | Framework runner/coordinator after origin enrichment. | Output module aggregation keeps framework types smaller but duplicates policy across encoders and cannot repair missing validator origin. | +| Stable aggregation key? | Disposition, category, reason code, and full producer origin; exclude chunk/scope/message. | Explicit producer-supplied grouping keys offer flexibility but add another identity that can drift from reason codes. Message-template grouping is brittle and unsafe. | +| Samples and omissions? | Exact occurrence count plus deterministic bounded samples and numeric omitted-sample count. | Omission warning records preserve the current representation but inflate group counts and require prose parsing. | +| `warning_count` semantics? | Version receipt and replace ambiguity with group and occurrence counts. | Keep v1 count as published record length and add optional fields; compatible, but two competing warning counts remain easy to misuse. | +| Which normalization changes remain warnings? | Only exhausted process fallback. Unresolved membership is a data-quality advisory; successful canonicalization, reordering, ID repair, source-ref dedupe, and duplicate consolidation are observations. | Treat unresolved membership or semantic duplicate consolidation as warnings because they affect grounding or cardinality; more conservative, but it violates the process-only warning rule and reports accepted artifact quality as an operational failure. | +| Source-relatedness disposition? | Grouped advisory by default; preserve current approve behavior. | Retain warning disposition or make rejection configurable. Rejection requires production precision evidence; current lexical rules are not strong enough. | +| Checkpoint-loaded warnings? | Present the same logical groups as fresh execution and use checkpoint events for reuse provenance. | Mark groups as replayed; aids forensics but fragments aggregation and makes semantically equivalent runs differ. | +| ADR and schema versions? | Add an ADR and version the run receipt and diagnostic files. | Treat the work as CLI-only presentation and avoid an ADR; insufficient because module contracts, output files, checkpoint payloads, and downstream fields change. | + +## Compatibility, Documentation, And ADR Implications + +The target alters public and internal contracts enough to require a new ADR. +It should record: + +- the distinction among warnings, advisories, observations, rejections, and + errors; +- the invariant that warnings are process-level signals, LLM-judged extraction + quality is never a warning, and ordinary non-degraded success has zero + warnings; +- module semantic ownership versus framework origin/aggregation ownership; +- bounded group and sample semantics; +- fresh/checkpoint equivalence; and +- the output-encoder decision. + +Implementation should introduce `notarius.run-result.v2`. The grouped warning +and diagnostic envelopes should each carry their own schema version. Because +the content of `warnings.json` changes incompatibly from a flat array wrapper +to groups, release notes and the published JSON integration contract must call +out the migration. `index.json` gains the diagnostic file path. + +Canonical documentation updates belong in: + +- `docs/cli.md` for stderr presentation only; +- `docs/operations.md` for operator review and debug workflow; +- `docs/integrations/json-output.md` for warning and diagnostic file schemas; +- `docs/integrations/run-result.md` for v2 fields and compatibility; +- `docs/consumers/subprocess.md` and `docs/consumers/dnd-pipeline.md` for + downstream policy checks; +- `docs/internal/pipeline.md` for promotion, aggregation, retry, and checkpoint + mechanics; +- `docs/internal/modules.md` and `docs/internal/dnd.md` for producer rules and + the D&D classification matrix; and +- `docs/policy/architecture.md` for the durable ownership invariant after the + ADR is accepted and implemented. + +No configuration knob is required for the first implementation. A fixed, +well-documented taxonomy is easier to reason about than per-reason display +overrides. Configurable escalation or suppression can be considered only after +production review demonstrates a concrete operator need. + +## Test Coverage Assessment + +Existing coverage worth preserving includes: + +- accepted-attempt and terminal-rejection warning promotion; +- validator failure retry exhaustion and `warn_continue`; +- module semantic retry fallback; +- deterministic warning order under concurrent lane completion; +- chunk-plan invalidation and discarded-cache warning behavior; +- fresh/checkpoint warning equivalence; +- local D&D warning caps and safe dynamic-message quoting; +- JSON warning-file publication; and +- CLI stderr, debug, and receipt counts. + +Material gaps are: + +- no bound test for NPC- or spell-relatedness warnings; +- no generic warning-field or result-size validation; +- no test for output encoder warnings versus published `warnings.json`; +- no operator-level aggregation or bounded-sample tests; +- no fresh/resume test for grouped counts because groups do not yet exist; and +- no production evaluation of advisory precision. + +Tests should protect the semantic relationships: exact occurrence counts, +bounded samples, deterministic group order, actionable-only warning +presentation, and cross-surface equality. They should not assert one exact +warning count for every complete D&D run or treat message wording as a public +API unless the wording itself enforces a security boundary. + +## Audit Conclusion + +The application is in a good position for warning reform. Its retry, +validation, checkpoint, and concurrency mechanics provide reliable points at +which to attach structured diagnostics. The most valuable change is not to +suppress individual reason codes; it is to replace the semantically flat, +origin-free collection with bounded typed groups and to reserve the word +“warning” for conditions that merit operator attention. + +Provider-backed runs would improve prioritization and help tune the D&D +advisories, but they are not necessary to conclude that routine normalization +and heuristic doubt should not dominate stderr or the durable warning +contract. They should be gathered before changing heuristic acceptance policy +or adopting a numerical production warning-volume target. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 00000000..9a0ede8c --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,742 @@ +# Warning Signal And Presentation Implementation Plan + +## Purpose + +This document is the ordered implementation plan for the warning and diagnostic +reform defined by [the completed audit](audit.md). It is intended to be executed +stage by stage by a `gpt-5.6-terra` coding agent. Each numbered stage is one +implementation prompt and must leave the repository compiling, internally +coherent, and covered at the narrowest durable test boundaries relevant to that +stage. + +The audit is the canonical source for findings, evidence, and target rationale. +This document is the canonical source for implementation order and task +breakdown. + +## Governing Decisions + +The following decisions are final for this work set. + +1. **Warnings are process signals.** A warning means that the run completed + under policy despite process-level degradation or incomplete configured + work. Examples are an empty configured reference, an applicable validator + that could not complete under `warn_continue`, an unavailable required + upstream classification, or exhausted semantic-reconciliation fallback. +2. **Extraction-quality signals are not warnings.** LLM-judged uncertainty, + lexical source-relatedness findings, unresolved entity grounding, and other + accepted-artifact quality signals are advisories. They must never be + promoted to warnings merely because a model or heuristic expressed doubt. +3. **Routine successful transformations are observations.** Canonicalization, + sorting, source-reference cleanup, ID repair, and accepted duplicate + consolidation remain inspectable but do not require operator action. +4. **Ordinary successful runs have zero warnings.** A successful-but-degraded + run may have warnings when policy permits continuation. Advisory or + observation volume alone must not produce warning stderr or a nonzero + warning count. +5. **Errors and rejections remain separate.** This work must not change + validation approval, rejection, retry budgets, exit status, or error policy. + A corrected superseded attempt leaves no final warning. An exhausted + rejection remains a rejection or run failure according to existing policy. +6. **Modules own meaning; the framework owns context and presentation.** A + producer chooses disposition, category, reason code, scope, and safe message. + The framework attaches stage and pipeline origin, aggregates deterministically, + enforces bounds, and supplies the final collections to output and CLI code. +7. **Durable contracts are versioned.** The incompatible grouped warning file + is `notarius.warnings.v2`, the new diagnostic file is + `notarius.diagnostics.v1`, and the machine-readable run receipt becomes + `notarius.run-result.v2`. Do not silently redefine the v1 receipt or warning + payload. +8. **There is no first-release configuration surface for presentation policy.** + Classification, sample bounds, and aggregation are fixed application + policy. Do not add suppression, escalation, verbosity, or per-reason config + in this work set. + +## Target Contract + +Use the following model unless existing Go naming requires a narrowly scoped +variation. Any naming variation must preserve the specified fields and +semantics. + +### Producer Diagnostic + +Add a framework contract representing a locally grouped producer diagnostic: + +- `disposition`: `warning`, `advisory`, or `observation`; +- `category`: one of `configuration`, `degradation`, + `validation_incomplete`, `fallback`, `data_quality`, or `normalization`; +- `reason_code`: stable, nonblank producer-owned identity; +- `occurrence_count`: exact positive number of represented occurrences; +- `samples`: deterministic bounded samples containing safe `scope` and + `message`; and +- `omitted_sample_count`: exactly `occurrence_count - len(samples)`. + +Validate these combinations: + +| Disposition | Allowed categories | +| --- | --- | +| `warning` | `configuration`, `degradation`, `validation_incomplete`, `fallback` | +| `advisory` | `data_quality` | +| `observation` | `normalization` | + +This strict matrix is intentional. It makes the process-only warning rule a +contract invariant rather than a convention inferred from reason codes. + +Use these fixed limits: + +- reason code: 128 UTF-8 bytes; +- scope: 512 UTF-8 bytes; +- sample message: 4 KiB of valid UTF-8; +- retained distinct samples per group: 3; and +- groups returned by one producer or validator result: 64. + +Blank or invalid required fields, invalid disposition/category combinations, +invalid UTF-8, inconsistent counts, excessive samples, or excessive local +groups make the producer result invalid. They must return a bounded contextual +error rather than be silently repaired. The three-sample and 64-group bounds +are public operational safeguards and may be asserted directly in their owning +contract tests. + +The shared collector must count every occurrence and retain the first three +distinct samples in producer order. Repeated identical samples still increase +`occurrence_count`. The omission count is numeric metadata, never another +diagnostic record. + +### Framework Origin And Aggregation + +The framework adds this origin before final aggregation: + +- stage: `references`, `chunk`, `extract`, `merge`, or `normalize`; +- step ID, when applicable; +- lane ID, when applicable; +- module key, when applicable; and +- validator key, for validator-produced diagnostics. + +Chunk ID and zero-based chunk index belong on samples, not group origin, +because otherwise identical per-chunk findings cannot aggregate. Use an +optional integer representation that preserves chunk index zero. + +The stable aggregation key is: + +```text +disposition + category + reason_code ++ stage + step_id + lane_id + module_key + validator_key +``` + +Scope, message, chunk ID, and chunk index are excluded from the key. Groups and +samples retain first-occurrence order from the runner's existing canonical +ordering; completion timing must never affect the result. When groups merge, +sum exact occurrence counts and retain the first three distinct samples. + +Use separate global bounds: + +- at most 128 actionable warning groups; exceeding this limit is a framework + error because Notarius must not hide process degradation; and +- at most 256 advisory/observation groups. Additional non-warning groups are + omitted from representation while their occurrences contribute to an exact + `unrepresented_occurrence_count` and set `truncated: true`. + +The non-warning envelope's group count means represented groups. Its occurrence +count includes represented and unrepresented occurrences. The warning +collection is never truncated, so both warning counts are exact. + +### Classification Matrix + +Migrate existing producers according to this matrix. + +| Disposition and category | Existing conditions | +| --- | --- | +| Warning / `configuration` | `empty_reference` | +| Warning / `validation_incomplete` | `validator_execution_incomplete`, covering both exhausted failure and skip when `warn_continue` advances the candidate | +| Warning / `degradation` | `scene_classification_unavailable` from combat-turn and enemy-event extraction gates | +| Warning / `fallback` | `npc_semantic_reconciliation_exhausted`, `item_semantic_reconciliation_exhausted`, `location_semantic_reconciliation_exhausted` | +| Advisory / `data_quality` | All ten source-relatedness families; `spell_name_unresolved`; `item_occurrence_unknown_item_id`; `location_occurrence_unknown_location_id`; guarded invalid semantic-consolidation proposals | +| Observation / `normalization` | Field and whitespace normalization; durable ID recomputation; source-reference sorting/deduplication; canonical record ordering; exact or approved semantic duplicate consolidation | + +Rename `item_occurrence_source_unrelated` to +`item_occurrence_not_near_source` when it moves into the new contract. Keep +other existing reason codes unless this plan explicitly changes them. Split the +item registry's internal retry reason from its accepted advisory: use +`item_semantic_retry_proposal_invalid` for the retry directive and retain +`item_semantic_proposal_invalid` only for the accepted data-quality advisory. + +Local `*_warnings_omitted` reason codes disappear. Omission is represented by +group counts. Rejections, structural failures, provider failures, cancellation, +and successful corrected retries do not receive a diagnostic solely to mirror +their existing error, rejection, validation, or attempt-debug records. + +### Durable And CLI Contracts + +Always publish both companion files, including for empty collections: + +- `warnings.json` with schema version `notarius.warnings.v2`, actionable groups, + exact `group_count`, and exact `occurrence_count`; +- `diagnostics.json` with schema version `notarius.diagnostics.v1`, advisory and + observation groups, represented `group_count`, total `occurrence_count`, + `truncated`, and `unrepresented_occurrence_count`; and +- `index.json` with both `warnings_file` and `diagnostics_file`. + +The run receipt becomes `notarius.run-result.v2` and replaces ambiguous +`warning_count` with: + +- `warning_group_count`; +- `warning_occurrence_count`; +- `diagnostic_group_count`; +- `diagnostic_occurrence_count`; and +- `diagnostics_truncated`. + +The human CLI writes a warning summary to stderr only when +`warning_group_count > 0`. It reports group and occurrence counts plus the +durable warning path. Advisory and observation counts do not create a warning +line. Preserve the existing successful stdout summary and process exit rules. + +Remove `OutputResult.Warnings`. An output encoder either returns its complete +logical files or returns an error. It cannot discover a warning after the +warning file has already been serialized. + +## Stage 1 — Record The Decision And Add Core Diagnostic Primitives + +### Goal + +Create the durable decision record and the generic, independently tested +diagnostic model without changing current runtime behavior. + +### Work + +- Add `docs/adr/0015-separate-process-warnings-from-quality-diagnostics.md` + using the repository ADR format. Record the governing decisions, origin and + aggregation ownership, bounded samples, fresh/resume equivalence, versioned + public contracts, and removal of output-result warnings. Mark it accepted; + do not claim implementation is complete. +- Add the producer diagnostic, sample, disposition, category, origin, group, + and collection types under `internal/framework/contracts/`. +- Add contract validation with the exact allowed combinations and limits in + this plan. +- Add a small generic collector under `internal/framework/diagnostics/` that + groups producer-local occurrences by disposition, category, and reason code, + preserves exact counts, and retains three distinct samples. +- Keep `contracts.Warning` and existing result fields temporarily. Do not alter + current warning output in this stage. + +### Tests + +Use table-driven contract tests for valid classifications, invalid UTF-8, +blank/oversized fields, invalid counts, the disposition/category matrix, sample +bounds, exact repeated-occurrence counting, and deterministic distinct-sample +selection. Test behavior through exported package contracts rather than private +collector structure. + +### Acceptance Criteria + +- Existing application behavior and public JSON remain unchanged. +- The new primitives cannot represent an LLM-quality warning because + `warning` plus `data_quality` is rejected. +- `go test ./internal/framework/contracts ./internal/framework/diagnostics` + passes. + +This stage is appropriately sized for one `gpt-5.6-terra` prompt. + +## Stage 2 — Add Framework Diagnostic Transport And Transitional Projection + +### Goal + +Carry structured diagnostics through every framework result boundary and add +origin-aware aggregation while preserving the current public warning contract +temporarily. + +### Work + +- Add structured diagnostic fields alongside legacy warning fields in chunk + plan, typed extract, typed merge, typed normalize, normalize-retry fallback, + validation, run input/output, and output request contracts. +- Extend erased typed results, producer-attempt state, lane result collection, + ordered-step handoff, and run finalization to transport diagnostics. +- Preserve validator identity in validation reports instead of flattening new + diagnostics through a warning-only helper. +- Attach current stage, step, lane, module, validator, and sample-level chunk + context at promotion time. +- Add a run-level aggregator implementing the stable key, canonical + first-occurrence ordering, count merging, and bounds from this plan. +- Preserve terminal-attempt semantics: diagnostics from superseded attempts + remain debug-only; only terminal accepted or terminal rejected candidate + diagnostics are promoted. +- Add a private transitional projection from each structured diagnostic sample + to the existing legacy `Warning` collection so public surfaces remain + unchanged until Stage 8. Mark it explicitly temporary and ensure it does not + duplicate diagnostics already supplied through the legacy path. + +### Tests + +Extend focused framework tests for terminal attempt promotion, terminal +rejection, reversed concurrent completion, stage/lane/chunk origin, stable +grouping across chunks, and global bounds. Use controlled test dispositions and +origins; do not assert incidental diagnostic prose. + +### Acceptance Criteria + +- A producer may return either legacy warnings or new structured diagnostics + during migration, but not produce duplicate public records through both. +- Structured groups are deterministic under concurrent completion. +- Superseded attempt diagnostics never reach final groups. +- Existing public warning tests still pass through the transitional projection. +- Focused framework tests and `go test ./internal/framework/pipeline` pass. + +This stage is appropriately sized for one high-reasoning +`gpt-5.6-terra` prompt. Do not combine it with D&D migration. + +## Stage 3 — Migrate Framework Process Signals And Close The Output Boundary + +### Goal + +Move framework-owned warning conditions into the new process-only contract and +remove the output-encoder consistency defect. + +### Work + +- Convert `empty_reference` to warning/configuration with reference origin and + bounded, non-sensitive sample content. +- Convert incomplete validation to warning/validation-incomplete whenever + `warn_continue` advances a candidate after an applicable validator either + exhausts with failure or skips. Retain validator key and typed failure/skip + detail in validation summaries; do not copy provider errors or arbitrary + validator prose into diagnostic samples. +- Aggregate all incomplete-validator occurrences rather than generating + omission warning records. +- Validate module and validator diagnostic results before promotion. An invalid + diagnostic contract is a bounded contextual framework error. +- Remove `Warnings` from `contracts.OutputResult`, the output adapter, test + doubles, and runner append logic. Keep `OutputRequest` diagnostic input; an + encoder failure remains an error. +- Add a regression test proving no post-encoding result can make receipt/debug + warning state disagree with the already encoded logical files. + +### Acceptance Criteria + +- Failed and skipped validators are both visible when incomplete validation is + allowed to continue. +- A fail-run validation policy still fails rather than converting the failure + into a warning. +- Output encoders have no successful post-encoding warning capability. +- Existing validation decisions, retry budgets, and exit behavior are + unchanged. +- `go test ./internal/framework/... ./internal/modules/generic/output/json` + passes. + +This stage is appropriately sized for one `gpt-5.6-terra` prompt. + +## Stage 4 — Version Diagnostic-Bearing Cache And Checkpoint State + +### Goal + +Persist and replay structured producer diagnostics safely without allowing old +warning-only state to be interpreted as complete new state. + +### Work + +- Store producer-local structured diagnostics in chunk-plan and extract, + merge, and normalize checkpoint envelopes. Do not persist current pipeline + origin in source-keyed chunk-plan state. +- On reuse, attach the current run's origin at the same logical promotion point + used by fresh execution, then aggregate once. +- Bump the checkpoint workspace schema from v3 to + `notarius.workspace.v4` and the chunk-plan schema from v2 to + `notarius.chunk-plan.v3`. +- Treat earlier workspace and chunk-plan versions as incompatible cache misses, + never as corrupt fatal state and never as reusable diagnostic-complete state. +- Preserve the rule that validation-incomplete results and their dependents are + not reusable. + +### Tests + +Add or adapt behavioral tests for old-version invalidation, structured +diagnostic round trips, fresh/resume equality of groups and counts, exact-once +replay, and current-validator diagnostics on a reused chunk plan. + +### Acceptance Criteria + +- Fresh and resumed logical runs produce deeply equal diagnostic groups and + counts. +- Reuse provenance remains in checkpoint events and does not alter diagnostic + identity. +- Old cache data is safely bypassed. +- `go test ./internal/framework/checkpoint ./internal/framework/pipeline` + passes. + +This stage is appropriately sized for one `gpt-5.6-terra` prompt. + +## Stage 5 — Migrate All D&D Source-Relatedness Validators + +### Goal + +Move all ten heuristic relatedness families from warnings to bounded +data-quality advisories. + +### Work + +- Migrate combat turns, enemy events, item occurrences, item registry, + location occurrences, location registry, NPC occurrences, NPC registry, + scene descriptions, and spells source-relatedness validators. +- Use the shared generic collector so every family returns exact occurrence + counts and at most three distinct samples. Remove each relatedness omission + reason code. +- Bring NPC and spell validators under the same contract bounds as every other + family. +- Rename `item_occurrence_source_unrelated` to + `item_occurrence_not_near_source`. +- Preserve current approval behavior, validator chain registration and order, + lexical algorithms, scope selection, and messages except for changes needed + to satisfy generic safety bounds. + +### Tests + +Consolidate repetitive limiter tests where a shared collector contract already +owns the behavior. Retain family tests for realistic triggering and +non-triggering cases. Prove that a relatedness finding is advisory/data-quality +and cannot increment process warning groups. + +### Acceptance Criteria + +- All ten relatedness validators use the same structured advisory convention. +- No relatedness validator returns a legacy warning or an omission record. +- High-cardinality NPC and spell results remain bounded while reporting exact + occurrence counts. +- `go test ./internal/modules/dnd/validate/...` passes. + +This repetitive but cohesive migration is appropriately sized for one +`gpt-5.6-terra` prompt. Do not combine it with normalizer migration. + +## Stage 6 — Migrate D&D Registry Normalizers And Semantic Reconciliation + +### Goal + +Apply the new taxonomy to NPC, item, and location registry normalization and +semantic-reconciliation outcomes. + +### Work + +- Migrate deterministic field cleanup, ID recomputation, source-reference + normalization, ordering, and accepted duplicate consolidation to + observation/normalization groups. +- Migrate exhausted NPC, item, and location semantic reconciliation to + warning/fallback groups. +- Migrate guarded invalid item semantic proposals to advisory/data-quality. +- Split the item retry directive reason to + `item_semantic_retry_proposal_invalid`; keep correction control data separate + from the accepted advisory reason and from model-facing text. +- Replace local warning limiters and omission records with the shared + diagnostic collector while preserving artifact values, safe fallback, and + retry budgets. + +### Tests + +Protect exact artifact outcomes, classification, retry exhaustion, safe +currency behavior, occurrence counts, and bounded samples. Do not preserve +old warning slice lengths or omission prose. + +### Acceptance Criteria + +- Successful registry cleanup and consolidation produce observations only. +- A guarded model proposal produces an advisory, not a warning. +- Exhausted semantic reconciliation is the only registry-normalizer process + warning family. +- Semantic retry behavior and artifacts are unchanged. +- Registry normalizer and semantic-reconciliation tests pass. + +This stage is appropriately sized for one high-reasoning +`gpt-5.6-terra` prompt. + +## Stage 7 — Migrate Remaining D&D Producers + +### Goal + +Complete D&D diagnostic classification across spells, occurrences, combat, +enemy events, scenes, and extraction gates. + +### Work + +- Migrate spell, combat-turn, enemy-event, item-occurrence, + location-occurrence, NPC-occurrence, and scene-description normalizers. +- Classify deterministic cleanup, ordering, source-reference normalization, + and duplicate consolidation as observation/normalization. +- Classify unresolved spell, item, or location membership as + advisory/data-quality, never warning. +- Convert combat-turn and enemy-event `scene_classification_unavailable` + extraction gates to warning/degradation with module and chunk provenance. +- Remove all remaining D&D local warning omission reason codes and all uses of + `LimitWarnings`; retain text-safety helpers that still have value. + +### Tests + +Adapt family tests to assert artifact invariants and classification. Add one +assembled D&D test demonstrating that quality advisories and normalization +observations can be present while the warning group count remains zero. + +### Acceptance Criteria + +- No D&D producer returns a legacy `contracts.Warning`. +- No accepted-artifact uncertainty or unresolved grounding appears as a + process warning. +- Missing required scene classification remains an actionable process warning. +- All D&D package tests pass. + +This stage is appropriately sized for one high-reasoning +`gpt-5.6-terra` prompt. + +## Stage 8 — Remove The Legacy Warning Path And Finalize Aggregation + +### Goal + +Make the structured diagnostic collection the sole in-memory signal path. + +### Work + +- Migrate any remaining generic test modules and framework fixtures from + `contracts.Warning` to structured diagnostics. +- Remove `contracts.Warning`, legacy `Warnings` fields, clone helpers, warning + append helpers, transitional projections, and obsolete D&D warning limiters. +- Make `RunOutput` and `OutputRequest` carry the finalized warning and + non-warning grouped collections derived from one aggregator result. +- Enforce the 128-warning-group error bound and 256-non-warning-group truncation + behavior. Ensure occurrence counts remain exact and truncation metadata is + deterministic. +- Verify all producers are validated at their stable framework boundary. + +### Tests + +Add focused behavioral coverage for warning overflow failure, non-warning +truncation, exact occurrence totals, distinct sample retention, stable group +order, and no legacy double-counting. Remove tests whose only purpose was to +assert old flat-list caps or omission prose. + +### Acceptance Criteria + +- Repository search finds no production `contracts.Warning`, legacy warning + result field, or `*_warnings_omitted` reason code. +- One finalized collection supplies all later surfaces. +- An advisory-only successful run has zero warning groups. +- `go test ./internal/framework/... ./internal/modules/dnd/...` passes. + +This stage is appropriately sized for one `gpt-5.6-terra` prompt. + +## Stage 9 — Publish Versioned Warning And Diagnostic Files + +### Goal + +Replace the flat durable warning payload with the two target grouped contracts. + +### Work + +- Update the production JSON encoder to always emit grouped `warnings.json` + (`notarius.warnings.v2`) and `diagnostics.json` + (`notarius.diagnostics.v1`). +- Include exact counts and diagnostic truncation metadata specified above. +- Add `diagnostics_file` to `index.json`; retain `warnings_file`. +- Ensure warning groups appear only in `warnings.json` and advisory/observation + groups appear only in `diagnostics.json`. +- Update `docs/integrations/json-output.md` in the same stage. It owns file + names, envelope schemas, count meanings, group/sample fields, bounds, + truncation, and index discovery. Link rather than duplicate CLI behavior. +- Update maintained output examples or test fixtures only where they encode + implemented output contracts. + +### Tests + +Use encoder contract tests for empty and populated files, schema versions, +partitioning, counts, truncation, index paths, deterministic JSON ordering, and +newline/valid-JSON conventions. Update assembled pipeline tests to compare both +durable files with the finalized in-memory collections. + +### Acceptance Criteria + +- Every production JSON bundle contains both companion files and index paths. +- No diagnostic appears in both files. +- Published counts agree with the in-memory collection. +- Output encoder and maintained example contract tests pass. + +This stage is appropriately sized for one `gpt-5.6-terra` prompt. + +## Stage 10 — Introduce Run Result V2 And Quiet CLI Presentation + +### Goal + +Give subprocess consumers unambiguous counts and make an ordinary successful +run quiet on the warning stream. + +### Work + +- Change the emitted receipt to `notarius.run-result.v2` with the five fields + specified in the target contract. Remove v1 `warning_count`; do not emit two + competing count models. +- Derive receipt counts from the same finalized collection used by the JSON + encoder. +- Print a stderr warning summary only when actionable warning groups exist. It + must include group count, occurrence count, and the output-relative or + absolute details path consistent with current CLI path conventions. +- Do not mention advisory/observation counts as warnings. Preserve ordinary + stdout completion output and all exit classifications. +- Update `docs/cli.md`, `docs/integrations/run-result.md`, + `docs/consumers/subprocess.md`, and `docs/consumers/dnd-pipeline.md` in the + same stage. Each document must keep to its canonical scope and link to the + grouped JSON contract rather than duplicating schemas. + +### Tests + +Cover zero-warning approved success, advisory-only success, degraded success +with warnings, rejected output, JSON receipt delivery, and non-production +output modules. Assert semantic fields and stream choice, not complete prose. + +### Acceptance Criteria + +- Advisory-only and observation-only successful runs write no warning stderr. +- Degraded success writes one concise actionable warning summary. +- Receipt counts and durable files agree. +- Receipt, CLI command, and consumer contract tests pass. + +This stage is appropriately sized for one `gpt-5.6-terra` prompt. + +## Stage 11 — Align Debug, Manifest, And Resume Surfaces + +### Goal + +Ensure forensic and provenance surfaces reflect the new model without creating +another competing warning contract. + +### Work + +- Replace final flat warning debug summaries with final grouped warning and + non-warning diagnostic projections, using separate clearly named files or + one versioned diagnostic summary envelope consistently with existing debug + layout. +- Keep attempt-local candidate diagnostics in detailed debug records with + stage/attempt provenance. Preserve redaction and the rule that raw model + responses and correction messages appear only in detailed debug. +- Keep manifest validation and rejection summaries authoritative for validation + status; do not duplicate full diagnostic groups into the manifest. +- Ensure checkpoint events, not diagnostic origin, identify reuse. +- Update `docs/operations.md`, `docs/internal/state.md`, and the debug portions + of `docs/internal/pipeline.md` in the same stage. + +### Tests + +Cover successful debug capture, partial failure debug capture, +terminal-attempt-only final grouping, redaction, and fresh/resume equality. +Retain existing validation summary tests rather than duplicating every outcome +in diagnostic tests. + +### Acceptance Criteria + +- Debug consumers can distinguish process warnings, quality advisories, and + observations. +- Debug final counts match receipt and durable output when output is published. +- Attempt detail remains bounded and sensitive-data rules are preserved. +- Debug, state, and resume-focused tests pass. + +This stage is appropriately sized for one `gpt-5.6-terra` prompt. + +## Stage 12 — Update Architecture And Internal Producer Documentation + +### Goal + +Make the implemented ownership and classification rules durable for future +modules without duplicating public schemas. + +### Work + +- Update `docs/policy/architecture.md` to state the process-only warning + invariant, module semantic ownership, framework origin/aggregation ownership, + and ordinary-success-zero-warnings expectation. Link to ADR-0015 for + rationale. +- Update `docs/internal/pipeline.md` for terminal promotion, origin enrichment, + aggregation, bounds, concurrency ordering, checkpoint replay, and output + handoff. +- Update `docs/internal/modules.md` with the generic producer contract and the + prohibition on using warnings for extraction quality. +- Update `docs/internal/dnd.md` with the implemented classification matrix and + shared collector convention. Link to integration contracts for durable + schemas rather than restating them. +- Update `docs/internal/overview.md` only if the new generic diagnostics package + changes the component inventory. +- Remove stale terminology from current docs outside `docs/roadmap/`, but do + not write release history or document unimplemented configuration. + +### Acceptance Criteria + +- Each volatile fact has one canonical owner under the documentation policy. +- Future module authors can determine which disposition/category to use and + where origin and aggregation are attached. +- Current documentation contains no claim that heuristic quality doubt is a + warning. +- Relative Markdown links resolve. + +This stage is appropriately sized for one `gpt-5.6-terra` prompt. + +## Stage 13 — Final Behavioral Verification And Cleanup + +### Goal + +Verify the complete migration, remove transitional residue, and establish that +the audit findings are fully addressed. + +### Work + +- Run repository searches for legacy warning types, warning omission reason + codes, old run-result v1 fields, old flat warning JSON assumptions, and + `OutputResult.Warnings`. +- Review every producer inventoried in `audit.md` against the final + classification matrix. +- Run the maintained minimal and complete D&D workflows with offline fake LLM + clients. Do not require provider credentials or add live calls to the default + suite. +- Verify these relationships end to end: + - ordinary approved success and advisory-only success have zero warnings; + - successful process degradation has a warning; + - corrected retries retain only terminal diagnostics; + - rejected and failed runs retain their established semantics; + - warning and diagnostic group order is deterministic under concurrency; + - fresh and resumed runs are equivalent; + - global group and sample bounds hold; and + - receipt, stderr, durable files, index, and debug counts agree. +- Remove obsolete helpers and redundant tests made unnecessary by stronger + package-level collector or end-to-end contract tests. + +### Validation Commands + +Run at minimum: + +```sh +go test ./... +go vet ./... +go build ./cmd/notarius +``` + +Also run any repository documentation or link checker discovered during the +stage. If none exists, perform a focused relative-link review for the documents +changed by Stages 9–12. + +### Acceptance Criteria + +- All three repository-wide Go commands pass offline. +- No live API key is required. +- No legacy flat-warning code path or public v1 warning-count claim remains in + current-behavior documentation. +- All seven audit findings are addressed without changing artifact values, + validator chain order, retry budgets, rejection policy, or exit semantics. +- The ordinary maintained successful workflow emits zero actionable warnings; + non-warning diagnostics remain available in `diagnostics.json`. + +This final verification is appropriately sized for one `gpt-5.6-terra` +prompt. + +## Deferred Work + +The following are deliberately outside this implementation plan: + +- provider-backed measurement of production advisory precision or frequency; +- configurable warning suppression, escalation, verbosity, or reason filters; +- converting heuristic relatedness checks into rejection rules; +- the planned LLM-backed D&D combat-scene validator; +- metrics or telemetry export beyond the specified durable and debug files; +- a two-phase output encoder protocol; and +- release preparation or release-note creation. + +Provider-backed production data may inform later advisory tuning, but it is not +required to implement or validate this architecture.