eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan warning and diagnostic reform

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-27 15:06:43 +00:00
parent e92bcfa74c
commit 610dd3d7c3
3 changed files with 1789 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

399
docs/roadmap/audit-plan.md Normal file
View File

@@ -0,0 +1,399 @@
# Warning Signal And Presentation Audit Plan
## Purpose
This document defines the audit required before redesigning Notarius warning
semantics and presentation. The audit must explain why an ordinary successful
D&D run can produce dozens of warnings, distinguish actionable degradation
from routine diagnostic observations, and recommend a bounded, deterministic
warning contract that operators will actually review.
This is an audit plan, not a feature design or implementation plan. The audit
must first establish the current producers, propagation behavior, empirical
volume, public contracts, and compatibility constraints. The completed audit
selects the target behavior from those findings; `implementation.md` owns the
subsequent implementation sequence.
## Context
Notarius currently uses one `contracts.Warning` shape—optional `scope`, required
`reason_code`, and required `message`—for findings from modules, validators,
normalizers, reference preparation, retries, and framework policy. Accepted
warnings are collected by the pipeline, counted by the CLI and machine-readable
run receipt, and published as a flat array in `warnings.json`.
The architecture already guarantees deterministic public ordering and bounded
warning production at several local boundaries. Recent validation work also
gave incomplete validation explicit provenance. Those guarantees must be
preserved. The observed problem is signal quality and aggregate volume: a
successful run can report many warnings even when most describe routine,
expected cleanup or weak heuristic observations rather than conditions that
require operator action.
Initial repository inspection identifies several likely warning families:
- D&D source-relatedness validators can produce one warning per artifact
record when a contextual name is not found in its cited text;
- deterministic normalizers report canonicalization, whitespace cleanup,
ordering changes, source-reference cleanup, and duplicate consolidation;
- semantic registry reconciliation reports discarded proposals, exhausted
fallback, and duplicate consolidation;
- optional reference materialization can report unavailable or skipped
reference inputs;
- the producer-attempt state machine reports validators that exhausted their
execution budgets under `warn_continue`; and
- producer modules can return their own warnings, which are promoted only from
the terminal accepted or rejected attempt.
These are audit hypotheses. The audit must verify exact behavior and frequency
from current code and representative runs rather than assume that every family
is noisy or incorrectly classified.
## Policy And Architecture Constraints
The audit and its recommendations must follow:
- `docs/policy/architecture.md` for module/framework ownership, deterministic
ordering, validation semantics, durable provenance, and sensitive-data
boundaries;
- `docs/policy/documentation.md` for canonical documentation ownership and the
distinction between current and future behavior;
- `docs/policy/testing.md` for risk-based, behavior-oriented test
recommendations; and
- ADR-0014 for the distinction between semantic rejection, validator failure,
correction guidance, and durable diagnostic provenance.
In particular:
- modules and validators may report findings, but the framework and output
boundaries own aggregation and presentation;
- reducing warning volume must not silently change artifact acceptance,
validation policy, retry behavior, rejection behavior, or process exit
status;
- genuine validator execution failures and validation-incomplete outcomes must
remain visible and auditable;
- public ordering must remain deterministic despite concurrent lane execution;
- aggregation must be bounded and must not leak raw model responses,
correction guidance, credentials, private reference content, or unnecessary
transcript text; and
- detailed diagnostics may move to a more appropriate durable or debug surface,
but must not be discarded when they are needed to understand a lossy or
degraded result.
## Audit Objectives
The audit must answer five questions.
1. Which code paths produce warnings, under what conditions, and with what
expected multiplicity?
2. How are warnings cloned, bounded, replayed, ordered, promoted, persisted,
counted, and presented from their producer through the final CLI and output
bundle?
3. Which warnings identify actionable degradation or data-quality risk, and
which merely describe routine successful transformations or advisory
heuristics?
4. What aggregation and presentation model would materially reduce operator
noise without hiding genuine failures, lossy fallback, or incomplete
validation?
5. Which proposed changes affect only presentation, and which would alter a
public receipt, durable output contract, validation invariant, or other
compatibility boundary?
## Scope
### Warning Producers
Inventory every production warning constructor and direct warning literal
under `internal/`. Include warnings originating from:
- input, chunk, extract, merge, normalize, and output modules;
- typed, chunk, and serialized validators;
- semantic reconciliation and normalizer fallback;
- external and generated reference materialization;
- chunk-plan and checkpoint reuse or fallback;
- producer-attempt exhaustion and validation-incomplete continuation;
- pipeline orchestration and cancellation handling; and
- CLI or publication logic, if it creates warnings rather than only presenting
them.
Do not infer completeness from one textual search. Inspect shared constructors,
returned result types, reason-code constants, registration paths, and tests that
exercise warning behavior.
### Warning Propagation
Trace each warning family through:
- stage result and validation result contracts;
- producer attempts, including abandoned attempts, semantic retries, structural
retries, terminal rejection, and `warn_continue`;
- concurrent lane collection and stable public ordering;
- merge and normalize continuation;
- chunk-plan caching and checkpoint recording or hydration;
- ordered step output merging and generated-reference handoff;
- `RunOutput`, run manifests, validation summaries, debug records, and CLI
result construction;
- `warnings.json`, `manifest.json`, the run-result receipt, human-readable
standard error, and debug bundles.
For every boundary, determine whether warnings are copied, filtered,
deduplicated, bounded, summarized, replayed from reusable state, or dropped.
Pay particular attention to amplification across chunks, lanes, validators,
retries, and resumed runs.
### D&D Warning Semantics
Review every implemented D&D artifact family. At minimum, distinguish:
- evidence/source-relatedness advisories;
- deterministic canonicalization or whitespace changes;
- order and source-reference normalization;
- exact and semantic duplicate consolidation;
- unresolved catalog or registry membership;
- semantic-reconciliation proposal failure or fallback; and
- validator execution incompleteness.
Determine whether the reason-code vocabulary is consistent across artifact
families, whether scopes are sufficiently contextual, and whether equivalent
conditions produce near-duplicate warnings with different codes or prose.
### Public And Operator Surfaces
Review the implemented contracts and documentation for:
- CLI success output and warning-count output;
- `notarius.run-result.v1` and its `warning_count` and validation fields;
- `warnings.json` and the published JSON index;
- manifest validation summaries and rejection summaries;
- debug summary and detailed debug artifacts; and
- downstream subprocess guidance, especially the complete D&D consumer
workflow.
Identify which surfaces are intended for immediate operator attention, durable
machine consumption, forensic detail, or debugging. Record any places where
the same flat count or warning list is being asked to serve incompatible
audiences.
### Tests And Documentation
Inventory tests that protect warning production, bounds, ordering, retry
promotion, checkpoint replay, JSON publication, receipt counts, and CLI stream
behavior. Identify meaningful gaps, redundant exact-prose assertions, and tests
that would unnecessarily obstruct a taxonomy or aggregation redesign.
Review `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
`docs/integrations/json-output.md`, `docs/integrations/run-result.md`,
`docs/consumers/`, and relevant internal documentation for current warning
claims. Record the canonical document that would own each future contract
change; do not rewrite those documents during the audit.
## Out Of Scope
The audit must not:
- implement warning filtering, severity levels, aggregation, or new CLI flags;
- change validator decisions, default chains, retry budgets, or terminal
validation policy;
- suppress warnings merely to meet a numerical target;
- redesign rejections, errors, logs, metrics, or debug bundles except where
their boundary with warnings must be clarified;
- add the planned D&D combat-scene semantic validator;
- create deterministic tests that assert one exact global warning count for
all future runs; or
- use live paid LLM calls as part of the default automated test suite.
## Inventory Method
Create a warning inventory with one row per semantically distinct production
condition. Each row should record:
| Field | Required analysis |
| --- | --- |
| Producer | Package, function, module or validator key, stage, and artifact family. |
| Trigger | Exact condition that emits the warning and whether it follows successful mutation, heuristic doubt, fallback, or failure. |
| Identity | Reason code, scope format, and whether either is stable enough for aggregation or machine use. |
| Multiplicity | Maximum per record, chunk, lane, validator, attempt, step, and run. |
| Lifecycle | Whether abandoned attempts are discarded, terminal warnings promoted, and cached or checkpointed warnings replayed. |
| Consequence | Whether data changed, evidence is questionable, output is incomplete, fallback occurred, or no externally meaningful consequence exists. |
| Actionability | What an operator can reasonably do in response, if anything. |
| Surfaces | CLI, receipt count, `warnings.json`, manifest, rejection, checkpoint, or debug presence. |
| Bounds | Existing local caps, message limits, omission records, and any missing aggregate bound. |
| Sensitivity | Whether scope or message can contain source-derived or otherwise sensitive content. |
| Coverage | Existing tests and the meaningful regression risk they protect. |
Treat different reason codes that represent the same operator condition as
potential consolidation candidates, but do not merge them in the audit
document without explaining lost diagnostic information.
## Propagation Analysis
Produce a compact propagation map from warning creation to each terminal
surface. The analysis must explicitly verify:
- only the terminal candidate's warnings are promoted after retries;
- whether rejected candidates retain warnings and where;
- whether `warn_continue` creates one warning per exhausted validator and how
that relates to validation summaries;
- whether cached chunk plans or hydrated checkpoints replay historical warnings
into the current run;
- whether the same warning can be appended at more than one handoff boundary;
- how concurrent completion is reordered before publication;
- whether local per-module caps compose into an unbounded or excessively large
run-level result; and
- whether warning counts on stderr, receipts, manifests, and `warnings.json`
refer to exactly the same collection.
Any suspected duplicate append or unstable ordering is a correctness finding,
not merely a presentation concern.
## Empirical D&D Run Analysis
Static inspection must be supplemented with representative run evidence. Use
at least:
- one ordinary successful complete D&D run known to produce high warning
volume;
- one smaller maintained example or synthetic run;
- one run with semantic registry reconciliation activity;
- one run with a validator execution failure allowed through
`warn_continue`, using an offline test double where practical; and
- one retrying producer case to verify abandoned-attempt warning treatment.
For real campaign runs, analyze only bounded metadata unless the operator
explicitly provides source content for review. Record counts grouped by stage,
lane, module or validator, reason code, and scope family. Also record unique
reason-code count, repeated-message count, maximum group size, validation
status, rejected-output count, and whether each warning led to a plausible
operator action.
Compare the same logical run under ordinary execution and checkpoint resume
when practical. The audit must distinguish warning volume caused by actual data
conditions from volume caused by orchestration or replay.
Do not make a model-quality judgment solely from warning frequency. Manually
inspect a bounded sample from each high-volume reason code to estimate false
positive rate and operational value.
## Classification Rubric
Classify each warning condition along independent dimensions rather than force
an immediate single severity enum:
- **result impact:** none, routine mutation, lossy mutation, uncertain data
quality, fallback, or incomplete validation;
- **operator action:** none, informational review, configuration or reference
correction, source/model review, or rerun required;
- **scope:** record, chunk, lane, step, pipeline, or infrastructure;
- **persistence need:** top-level attention, durable detail, debug-only detail,
or metric/trace candidate; and
- **confidence:** deterministic fact, heuristic advisory, or execution failure.
The audit should then test whether a small durable taxonomy can represent the
meaningful combinations. A promising starting hypothesis is that top-level
operator warnings should be limited to actionable degradation, incomplete
validation, lossy fallback, and material data-quality risk, while routine
successful normalization observations remain available as lower-level durable
diagnostics. The audit must validate or revise that hypothesis from evidence.
## Design Questions The Audit Must Resolve
The findings must give a recommendation, with at least one viable alternative
and tradeoffs, for each of these questions:
1. Should warning severity or disposition become an explicit contract field,
or should stable reason-code metadata drive presentation policy outside the
warning payload?
2. Should `warnings.json` remain the complete durable detail while the CLI and
receipt expose an aggregated actionable summary, or should durable warnings
themselves be separated from routine observations?
3. Where should global deduplication and aggregation live so modules retain
semantic ownership but concurrent pipeline results remain deterministic?
4. What is the stable aggregation key: reason code, stage/lane/module identity,
normalized scope, message template, or an explicit structured grouping key?
5. How should bounded samples and omitted counts be represented without
converting a summary record into another warning that inflates the count?
6. Should `warning_count` continue to mean the length of `warnings.json`, or
should a new receipt or schema field distinguish actionable warning groups
from detailed observations?
7. Which normalization changes are sufficiently lossy or surprising to remain
warnings, and which are ordinary provenance that belongs in manifest or
debug data?
8. Should heuristic source-relatedness findings remain warnings, become
grouped data-quality advisories, or be strengthened into configurable
validation decisions only after demonstrated precision?
9. How should warnings loaded from checkpoints be identified or aggregated
relative to newly produced warnings?
10. Does the chosen target alter durable or validation semantics enough to
require a new ADR or a versioned run-result/output contract?
## Required Audit Deliverable
Write the completed findings to `docs/roadmap/audit.md`. It should contain:
1. an executive assessment of current warning quality and risk;
2. the complete warning-producer inventory;
3. the warning propagation and surface map;
4. empirical measurements and bounded representative samples;
5. findings ranked by operator impact, correctness risk, and implementation
leverage;
6. a recommended target taxonomy and presentation model;
7. compatibility, documentation, ADR, and migration implications;
8. implementation implications and dependencies sufficient to support a
separate implementation plan; and
9. open decisions only where repository evidence cannot support a responsible
recommendation.
Each finding should identify the supporting code paths, tests, documentation,
and run evidence. Separate observed facts from recommendations and avoid
changelog or development-history framing.
## High-Level Audit Sequence
The detailed execution sequence should be written separately if needed. At a
high level, perform the audit in this order:
1. **Static inventory:** enumerate warning producers, reason codes, scopes,
bounds, and existing tests.
2. **Propagation audit:** trace promotion, ordering, replay, persistence,
counting, and presentation across the framework and CLI.
3. **Empirical analysis:** measure representative D&D runs and inspect bounded
samples from high-volume warning groups.
4. **Classification:** apply the rubric, identify duplicate concepts and
misplaced routine diagnostics, and evaluate public-contract options.
5. **Synthesis:** rank findings and recommend a decision-complete target for a
subsequent feature roadmap.
Static inventory and propagation may be performed as separate focused agent
prompts. Empirical analysis should be isolated because it may require operator
artifacts or opt-in provider execution. Classification and synthesis should
consume the earlier written evidence rather than rediscover the repository.
## Validation Of The Audit
Before considering the audit complete, verify that:
- every production warning literal or constructor is represented in the
inventory;
- every reason code observed in representative `warnings.json` files maps to a
known producer or is recorded as an unexplained finding;
- counts agree across the runner result, CLI receipt, stderr summary, and
published warning collection for each sampled run;
- retry, rejection, incomplete-validation, cache, checkpoint, concurrency, and
ordered-step paths are covered;
- recommended aggregation preserves deterministic ordering and bounded memory;
- recommendations distinguish warnings from errors, rejections, validation
summaries, logs, and debug diagnostics;
- no recommendation hides a condition that changes output completeness or
correctness;
- public compatibility and schema-version consequences are explicit; and
- proposed tests protect meaningful behavior without asserting incidental
prose or one permanently fixed global warning count.
## Completion Criteria
The audit is ready to become a feature roadmap when it can explain the current
high warning count quantitatively, identify the dominant producers and any
amplification defects, classify every warning family by consequence and
actionability, and recommend where each class should appear. The findings must
be specific enough that a later roadmap can define the target contract without
repeating the discovery work.

648
docs/roadmap/audit.md Normal file
View File

@@ -0,0 +1,648 @@
# Warning Signal And Presentation Audit
## Executive Assessment
Notarius warning execution is mechanically stronger than its operator-facing
presentation. Terminal-attempt promotion, stable ordering after concurrent
work, checkpoint replay, validation summaries, and debug retention are all
substantially correct. The audit found no general duplicate-append defect in
the extract, merge, or normalize handoffs and no leakage of abandoned-attempt
warnings into a successful result.
The warning channel itself is not coherent. One flat `contracts.Warning` type
currently represents at least four materially different concepts:
- actionable degradation or incomplete validation;
- heuristic data-quality doubt;
- successful but potentially reviewable fallback; and
- routine canonicalization, ordering, and deduplication observations.
That conflation is the primary reason successful runs produce a count that is
large but operationally weak. The maintained complete D&D example demonstrates
the problem without a live provider: an approved run with no rejected outputs
published 12 warning records, all from three advisory relatedness checks. An
operator separately reported a successful complete D&D run with 10 outputs,
one rejection, and 85 warnings. The production bundle for that run was not
available in this environment, so its reason-code distribution could not be
measured.
The current implementation also has four correctness or robustness gaps:
1. warning records lose stage, step, lane, module, validator, and chunk
provenance when promoted, which makes safe aggregation and diagnosis
impossible from `warnings.json` alone;
2. there is no framework-level validation or aggregate bound, and the NPC- and
spell-relatedness validators bypass the D&D warning limiter entirely;
3. warnings returned by an output encoder are added after `warnings.json` has
already been encoded, so the receipt, stderr, debug bundle, and published
warning file can disagree; and
4. a skipped validator contributes to `incomplete` validation but does not
receive the warning generated for an exhausted validator failure.
The recommended end state is a structured diagnostic contract with explicit
disposition, category, origin, occurrence count, and bounded samples. Warnings
are reserved for process-level degradation or incompleteness. LLM-judged or
deterministically inferred extraction-quality signals are advisories, never
warnings, and routine normalization observations remain inspectable without
being reported as top-level warnings. An ordinary successful run in which all
configured work completes normally should therefore report zero warnings. This
is an architectural and durable-contract change, not merely revised CLI prose.
## Evidence And Limits
The audit used:
- a complete static search of production `contracts.Warning` constructors,
reason-code constants, result fields, and promotion sites under `internal/`;
- call-path inspection through producer attempts, validators, lane
coordination, chunk-plan reuse, checkpoints, output encoding, debug output,
CLI presentation, and run-result construction;
- the maintained complete and minimal D&D examples with offline fake LLMs;
- focused deterministic tests for warning bounds, semantic-reconciliation
fallback, `warn_continue`, semantic retries, terminal rejection, concurrency
ordering, and checkpoint reuse; and
- the operator-provided observation of an 85-warning complete D&D run.
No provider-backed production run was attempted because this environment has
no API key. Consequently, the audit can establish warning mechanics, possible
multiplicity, synthetic volume, and obvious heuristic limitations, but cannot
estimate production frequency or the real false-positive rate of individual
D&D advisories. Those measurements are not required to choose the recommended
architecture; they are required before strengthening any heuristic advisory
into a rejection or setting a numerical production acceptance target.
## Complete Warning-Producer Inventory
### Framework And Generic Boundaries
| Producer | Reason code | Trigger and consequence | Multiplicity and bound | Current surfaces and coverage |
| --- | --- | --- | --- | --- |
| Reference materialization in `internal/framework/pipeline/references.go` | `empty_reference` | A bound external reference is a valid, accepted media type but contains zero bytes. The prompt may receive materially incomplete context. | One per empty bound file; finite by configuration but no shared run-level cap. | Enters `RunInput.Warnings`; reference tests protect contextual scope. |
| Producer-attempt policy in `internal/framework/pipeline/producer_attempts.go` | `validator_execution_incomplete` | An applicable validator exhausted its execution budget and `warn_continue` accepted the otherwise valid candidate. | One per failed validator on each terminal candidate. An extract chain can multiply this by chunks and lanes. There is no global cap. | Durable warning, receipt count, stderr, debug, and validation summary. `TestWarnContinueRecordsOneWarningForEachExhaustedValidator` covers failures. |
| Chunk, extract, merge, normalize, and output module result contracts | Module-defined | A module may return arbitrary warnings with its successful candidate. | No contract validation, message limit, per-result cap, or global cap. Current production modules are inventoried below. | Terminal-attempt filtering and concurrency ordering are well tested. |
| Production JSON output encoder | None | The encoder copies incoming warnings into `warnings.json`; it does not currently create warnings. | Same count as its input. | JSON encoder and assembled-pipeline tests compare the incoming run warnings with the published file. |
| Output encoder result contract | Module-defined | Any output encoder may return warnings discovered during encoding. | Unbounded by contract. No production encoder currently exercises this capability. | Appended to final `RunOutput` only after logical files were encoded; this is the cross-surface defect described in AUD-WARN-004. |
Input adapters and production mergers do not currently have independent
warning producers. Chunk-plan and checkpoint decisions are structured manifest
or debug provenance rather than warnings. Cancellation and hard persistence,
reference, parsing, serialization, and provider failures remain errors.
### D&D Extraction Gates
| Producer | Reason code | Trigger and consequence | Multiplicity and bound |
| --- | --- | --- | --- |
| `dnd/combat-turns` extractor | `scene_classification_unavailable` | The chunk has no exact matching scene-description classification. The extractor returns an empty accepted result and skips the LLM, so combat-turn output may be incomplete. | At most one per chunk for this lane. |
| `dnd/enemy-events` extractor | `scene_classification_unavailable` | The same missing or mismatched scene gate causes accepted empty enemy-event output. | At most one per chunk for this lane. |
An exact non-combat classification produces an intentional empty result without
a warning. An exact combat classification proceeds normally. The two producers
share a code and operator consequence but use different messages; their module
origins are not retained in the final warning record.
### D&D Source-Relatedness Validators
All ten relatedness validators are deterministic advisories: they approve the
candidate and warn when contextual prose or an entity name is not lexically
present in cited text. Shape and source-reference failures are deliberately
left to blocking validators earlier in the chain. The same relatedness
validator is registered in both the extract and normalize default chain for
each artifact family in `internal/modules/dnd/register/chains.go`.
| Artifact family | Warning reason | Per-record trigger | Local bound | Omission reason |
| --- | --- | --- | --- | --- |
| Combat turns | `combat_turn_not_near_source` | Actor token sequence absent | 20 per validator invocation | `combat_turn_relatedness_warnings_omitted` |
| Enemy events | `enemy_event_not_near_source` | Subject token sequence absent | 20 | `enemy_event_relatedness_warnings_omitted` |
| Item occurrences | `item_occurrence_source_unrelated` | Item name token sequence absent | 20 | `item_occurrence_relatedness_warnings_omitted` |
| Item registry | `item_not_near_source` | Item name token sequence absent | 20 | `item_relatedness_warnings_omitted` |
| Location occurrences | `location_occurrence_not_near_source` | Location name token sequence absent | 20 | `location_occurrence_relatedness_warnings_omitted` |
| Location registry | `location_not_near_source` | Location name token sequence absent | 20 | `location_relatedness_warnings_omitted` |
| NPC occurrences | `npc_occurrence_not_near_source` | NPC name token sequence absent | 20 | `npc_occurrence_relatedness_warnings_omitted` |
| NPC registry | `npc_not_near_source` | NPC name token sequence absent | **Unbounded** | None |
| Scene descriptions | `scene_description_not_near_source` | No significant title or summary token appears; up to two findings per scene | 20 | `scene_description_relatedness_warnings_omitted` |
| Spells | `spell_not_near_source` | Spell-name token sequence absent | **Unbounded** | None |
The eight limiter-generated omission records are presentation artifacts, not
new source-relatedness conditions. They occupy a warning slot and make list
length differ from the actual occurrence count.
### D&D Normalizers
Every production D&D normalizer bounds its returned warning slice to 20 through
`internal/modules/dnd/shared/diagnostics`, including a final omission record
when needed. Registry semantic retries reserve space for their fallback
warning. The following table is complete by semantically distinct condition;
codes listed together are parallel artifact-family variants.
| Condition | Reason codes | Result impact | Current classification assessment |
| --- | --- | --- | --- |
| Display or field whitespace/name canonicalization | `npc_fields_normalized`, `item_fields_normalized`, `location_fields_normalized`, `spell_name_canonicalized`, `combat_actor_canonicalized`, `enemy_event_name_canonicalized`, `item_occurrence_name_canonicalized`, `location_occurrence_name_canonicalized`, `scene_description_prose_normalized` | Deterministic successful mutation. The item-occurrence code can also describe `from`/`to` whitespace, not only the item name. | Routine observation. |
| Durable ID recomputation | `npc_id_recomputed`, `item_id_recomputed`, `location_id_recomputed` | Restores the deterministic name-derived ID. | Routine observation; invalid identity is separately rejected by default chains. |
| Source-reference sorting or deduplication | `source_references_normalized` | Sorts and removes exact duplicate references while deliberately preserving invalid references for their validators. | Routine observation. Shared code is useful but ambiguous without producer origin. |
| Canonical record ordering | `combat_turns_reordered`, `enemy_events_reordered`, `item_occurrences_reordered`, `location_occurrences_reordered`, `npc_occurrences_reordered`, `scene_description_order_normalized` | Deterministic order changes only. | Routine observation. |
| Exact or approved semantic duplicate consolidation | `duplicate_npc_collapsed`, `duplicate_item_collapsed`, `duplicate_location_collapsed`, `duplicate_spell_cast_collapsed`, `duplicate_combat_turn_collapsed`, `duplicate_enemy_event_collapsed`, `duplicate_item_occurrence_collapsed`, `duplicate_location_occurrence_collapsed`, `duplicate_npc_occurrence_collapsed`, `scene_description_duplicate_collapsed` | Removes duplicate records and preserves or combines canonical evidence according to the artifact policy. Registry codes cover both exact and accepted semantic consolidation. | Durable normalization observation; not normally operator-actionable. |
| Unresolved external membership | `spell_name_unresolved`, `item_occurrence_unknown_item_id`, `location_occurrence_unknown_location_id` | The value is preserved but is not grounded in the effective catalog or registry. Default chains normally reject the same condition before normalization; it remains reachable with validator overrides or defensive direct use. | Actionable data-quality warning. |
| Unsafe currency consolidation proposal | `item_semantic_proposal_invalid` | The proposed group is rejected and all records are preserved because denominations or currency/non-currency members are incompatible. The same code is also used internally as a retry reason. | Advisory about model proposal quality; no accepted-data loss. The control and diagnostic meanings should be separated. |
| Semantic reconciliation unavailable or exhausted | `npc_semantic_reconciliation_exhausted`, `item_semantic_reconciliation_exhausted`, `location_semantic_reconciliation_exhausted` | The safe deterministic result is accepted, but possible semantic duplicates remain. | Actionable fallback warning. |
| Local warning truncation | `npc_normalization_warnings_omitted`, `item_normalization_warnings_omitted`, `location_normalization_warnings_omitted`, `spell_normalization_warnings_omitted`, `combat_turn_normalization_warnings_omitted`, `enemy_event_normalization_warnings_omitted`, `item_occurrence_normalization_warnings_omitted`, `location_occurrence_normalization_warnings_omitted`, `npc_occurrence_normalization_warnings_omitted`, `scene_description_normalization_warnings_omitted` | Reports that individual records were omitted from presentation. | Group metadata, not an independent warning. |
No production D&D normalization warning exposes raw model responses,
correction guidance, or provider errors. Most dynamic names are quoted and
truncated by the shared helper. That local discipline is not enforced by the
generic warning contract, and the spell relatedness message does not use the
shared truncation helper.
## Warning Propagation And Surface Map
```text
external-reference warnings -----------------------------+
|
module candidate warnings -> validation chain warnings |
| | |
+---- producer-attempt terminal policy -------+
| |
accepted / terminal rejection only |
| |
chunk or lane result in canonical order |
| |
checkpoint record/replay and ordered step merge |
| |
RunOutput.Warnings <------------+
|
OutputRequest -> output encoder
| |
warnings.json OutputResult.Warnings
|
appended to final RunOutput only
|
receipt, stderr, final debug warning summary
```
### Attempts And Validation
- `runProducerAttempts` promotes only the terminal accepted or terminal
rejected candidate's module and completed-validator warnings. Operational,
structural, semantic, and module-directed attempts that are superseded are
retained in attempt debug artifacts but not in the final collection.
- A module-directed semantic-reconciliation retry adds its fallback warning
only when no retry remains. Earlier attempt warnings are discarded.
- On `warn_continue`, warnings from the otherwise accepted candidate and
completed approved or rejected validators are retained. One fixed,
non-sensitive `validator_execution_incomplete` warning is added for every
failed validator. Skipped validators affect the validation summary and final
`incomplete` status but do not receive such a warning.
- A semantic terminal rejection retains only warnings from that rejected
attempt. Structural rejection after producer failure cannot retain a
candidate warning because no valid candidate result exists.
These behaviors are protected by the producer-attempt, extract-handoff,
rejection-warning, normalize-retry, and attempt-debug tests. They are the right
foundation for the redesign and should not be replaced with early-exit or
all-attempt accumulation.
### Concurrency And Ordering
Extract jobs are dispatched chunk-first and lane-second. Results are stored by
chunk index, finalized in ascending chunk order, and lane continuations are
merged into a slice indexed by configured lane order. Pipeline steps run in
configured order. The resulting public order is therefore:
1. pre-run reference warnings;
2. chunk-stage warnings;
3. step order;
4. configured lane order within each step;
5. chunk order within extract;
6. merge warnings; then
7. normalize warnings; followed by any output-result warnings.
`TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion` exercises warning
order under reversed completion. No completion-order leak was found.
### Chunk Plans And Checkpoints
- A reusable chunk plan stores producer warnings only. Current validators run
again, and their current warnings are appended. Warnings from a cached plan
candidate that fails current validation are discarded before regeneration.
- Accepted extract, merge, and normalize checkpoints store the terminal
warnings for their stage. Reuse loads and appends those warnings once at the
same logical handoff. Tests compare fresh and resumed warning collections and
preserve their order.
- Validation-incomplete accepted outputs are not reusable, preventing a later
run from silently treating incomplete validation as complete.
- Required accepted-normalize hydration replays that normalize checkpoint's
warnings; checkpoint decisions separately expose that reuse occurred.
The recommended redesign should keep fresh and resumed logical diagnostics
equivalent. Whether a result was reused belongs in checkpoint provenance, not
in the diagnostic grouping key; adding a `reused` distinction would fragment
groups and make equivalent runs present differently.
### Terminal Surfaces
| Surface | Current content | Audience | Audit result |
| --- | --- | --- | --- |
| `RunOutput.Warnings` | Flat final slice | Framework and CLI | Canonical in-memory list, but lacks origin and bounds. |
| Published `warnings.json` | Object containing the warnings passed into the output encoder | Durable consumers | Exact for the production JSON encoder unless the encoder itself returns warnings. |
| `index.json` | Path to `warnings.json` | Durable consumers | Stable discovery path; no separate diagnostic-detail path. |
| Run-result v1 | `warning_count = len(final RunOutput.Warnings)` | Subprocess callers | Count only; no group/occurrence distinction. |
| Human stderr | `run completed with N warning(s)` | Operators | Count only and no direct detail path. Successful exit remains zero. |
| Manifest | Validation and rejection summaries, no warning collection | Durable provenance | Correctly avoids duplicating the flat list. |
| Debug summary `warnings.json` | Raw final warning array | Operators/developers | Includes final output-result warnings and can therefore differ from published `warnings.json`. |
| Debug run report | Final warning count | Operators/developers | Same final slice length as receipt and stderr. |
| Attempt/stage debug | Candidate-local warning detail and origin in path/envelope | Forensics | Sufficient to diagnose provenance, but debug capture is optional and is not a durable consumer contract. |
## Empirical Measurements
### Offline And Synthetic Runs
| Scenario | Result | What it establishes |
| --- | --- | --- |
| Maintained complete D&D config and transcript with the repository's offline fake LLM | Approved, 10 normalized outputs, 0 rejected outputs, 12 warnings; receipt, stderr, and published file all reported 12 | An ordinary structurally successful workflow can be noisy without fallback or incomplete validation. |
| Same complete run, grouped after publication | Three reason codes, seven exact `(reason, scope, message)` tuples, maximum exact-tuple repetition of three | The flat count materially overstates distinct operator conditions. Scope resets within chunks and does not identify origin. |
| Maintained focused scene-description workflow | Approved, one normalized output, 0 warnings | The warning channel can be quiet when synthetic model text is lexically grounded. |
| Generic warning publication contract | One warning reaches successful stderr, durable output, and debug summary | The ordinary pre-output path is consistent. |
| NPC semantic-reconciliation candidate-limit fallback | No LLM call, all records preserved, one exhaustion warning, total warnings no greater than 20 | Fallback is bounded and materially different from routine normalization. |
| `warn_continue` with two failed validators and one skipped validator | Validation status contains all three incomplete validators; warning slice contains two execution-incomplete records | Current warning count does not describe all incomplete validation. |
| Retrying extract candidate | Two producer attempts; only the accepted attempt's one warning is final | Retry does not amplify abandoned warnings. |
| Terminal semantic rejection | Only the final rejected attempt's operation and validator warnings are final | Rejection diagnostics are retained without retaining superseded warnings. |
| Fresh versus reused extract checkpoint | Warning collections are deeply equal | Checkpoint replay does not itself amplify warnings. |
| Spell normalizer with 21 unresolved entries | 20 records: 19 samples plus one omission record saying two additional warnings were omitted | `warning_count` is neither exact occurrence count nor distinct-condition count. |
The focused audit tests passed in `internal/cli`,
`internal/framework/pipeline`, the NPC-registry and spell normalizers, and all
D&D packages.
### Bounded Sample Review
The complete offline D&D run produced:
| Reason | Count | Sample | Review |
| --- | ---: | --- | --- |
| `location_not_near_source` | 4 | `Moon Gate` was absent from cited text | Correctly identifies deliberately unsupported fake output. Three records shared the same exact tuple because chunk and stage origin were lost. |
| `location_occurrence_not_near_source` | 4 | A `Moon Gate` visit was absent from cited text | Correctly identifies the same unsupported registry-driven occurrence, but repeats the same operator concern across extraction and normalization. |
| `scene_description_not_near_source` | 4 | A title or summary had no significant exact token in cited text | Mixed value. Generic `session scene` prose is ungrounded, while `Arrival` versus transcript `arrive` illustrates an expected lexical false positive. |
This fake workflow is an integration fixture, not a model-quality benchmark.
It nonetheless proves that the checks carry useful evidence while being too
imprecise and repetitive to serve as one-warning-per-record operator alerts.
### Production Evidence Still Needed
The reported 85-warning run establishes that high volume occurs in practice,
but the following remain unknown:
- dominant production reason codes and stage/lane sources;
- unique group count versus repeated occurrence count;
- false-positive rate for each relatedness family;
- how much volume comes from normalization observations versus advisories;
- whether fresh and resumed production runs remain equivalent; and
- a defensible numerical acceptance target.
If further data is worthwhile, the operator can supply the v1 receipt,
`warnings.json`, and manifest validation summaries without supplying transcript
or lane artifacts. An initial privacy-preserving report should group by reason
code and normalized scope family, count exact repeated tuples, and omit message
text. Reviewing heuristic precision requires a separately approved bounded
sample with its cited source context.
## Classification Of Current Conditions
| Target disposition | Current families | Result impact | Operator action | Durable placement |
| --- | --- | --- | --- | --- |
| **Warning** | Empty reference; validator failure or skip accepted under `warn_continue`; unavailable required scene classification; exhausted semantic reconciliation | A configured process completed under an allowed degraded or incomplete policy rather than completing normally | Correct reference/configuration, inspect provider/validator, or rerun | Actionable `warnings.json`, receipt summary, stderr summary, debug |
| **Advisory** | Source-relatedness heuristics; unresolved spell or registry membership; guarded invalid semantic proposal; any future LLM-judged uncertainty or extraction-quality signal | Uncertain data quality or poor model proposal, but accepted data is structurally valid and deterministic guards prevented unsafe mutation | Optional model/source review; no routine action for every record | Durable diagnostic detail and debug; never a top-level warning |
| **Observation** | Whitespace/name/ID/source-reference canonicalization; canonical ordering; exact and approved semantic duplicate consolidation | Successful intended normalization | None under normal operation | Durable bounded normalization diagnostics or debug; no stderr warning |
| **Not a diagnostic** | Rejection, invalid structure, cancellation, persistence error, provider failure under fail-run policy | Candidate or run did not complete according to policy | Inspect rejection/error and retry or correct input/configuration | Existing rejection, validation summary, error, and debug contracts |
Exact and semantic duplicate consolidation should remain distinguishable in
category or reason metadata even though both are observations. Semantic
reconciliation exhaustion remains a warning because a capability was not
applied; successful approved consolidation is an observation because it is the
normalizer's intended work.
## Ranked Findings
### AUD-WARN-001 — The Flat Warning Type Destroys Signal Quality
- **Priority:** High operator impact; high implementation leverage.
- **Evidence:** `contracts.Warning` has only scope, reason, and message. Routine
normalizer changes, heuristic doubt, fallback, and incomplete validation all
enter the same slice and the same CLI count. The offline complete run's 12
records were all advisories; the operator observed 85 records in a successful
real run.
- **Impact:** Operators cannot tell whether a warning requires a rerun, a
configuration repair, optional review, or no action. Repeated routine output
trains them to ignore the channel.
- **Recommendation:** Replace the flat semantic contract with explicit
`warning`, `advisory`, and `observation` dispositions plus a small category
vocabulary. Do not infer disposition from message text or require every
downstream consumer to maintain a reason-code policy table.
### AUD-WARN-002 — Warning Records Lose The Origin Needed For Diagnosis And Aggregation
- **Priority:** High correctness and usability impact.
- **Evidence:** The runner knows stage, step, lane, module, validator, chunk ID,
and chunk index at promotion time, but `terminalWarnings` flattens module and
validator records into `[]contracts.Warning`. Per-chunk scopes such as
`locations[0]` and `occurrences[0]` then repeat without identifying their
chunk or producer. `source_references_normalized` is intentionally shared
across families and is therefore especially ambiguous.
- **Impact:** `warnings.json` cannot answer which stage or module produced a
record. Message- or scope-based deduplication would merge unrelated findings
or retain accidental duplicates.
- **Recommendation:** Keep module findings free of framework context, then have
the framework add a structured origin envelope before promotion. Validator
findings must retain validator identity instead of passing through
`validationReport.Warnings()` as a flat slice.
### AUD-WARN-003 — Warning Volume Is Not End-To-End Bounded Or Validated
- **Priority:** High robustness impact; medium immediate likelihood.
- **Evidence:** D&D's `LimitWarnings` caps most individual producers at 20, but
NPC- and spell-relatedness return one warning per record without the helper.
Every extract validator is invoked per chunk, all ten relatedness checks run
again after normalization, and there is no run-level collector. The generic
contract validates neither disposition nor reason/message size, UTF-8,
blankness, or total records.
- **Impact:** Warning memory and output grow with chunks, lanes, configured
validators, and record counts. Local omission records lose exact occurrence
semantics while still incrementing `warning_count`.
- **Recommendation:** Add a generic bounded diagnostic collector that preserves
exact occurrence counts and bounded samples. Validate all diagnostic fields
at the module/framework boundary. Immediately bring NPC and spell
relatedness under the existing cap if the full redesign is staged.
### AUD-WARN-004 — Output Encoder Warnings Make Durable Surfaces Disagree
- **Priority:** Medium current impact; high contract correctness risk.
- **Evidence:** `Runner.Run` passes existing warnings to `encoder.Encode`, then
the production encoder serializes `warnings.json`. Only after encoding does
the runner append `OutputResult.Warnings`. The receipt, stderr, debug summary,
and debug run report see the final slice; the already-created published file
cannot. No production encoder currently returns a warning, so ordinary JSON
runs do not trigger the defect.
- **Impact:** A valid output-module implementation can violate the documented
claim that `warning_count` describes the published warning collection.
- **Recommendation:** Remove successful output warnings from the output-module
contract unless a demonstrated use case requires them; encoding failures
should be errors and optional encoder observations should be debug data. A
two-phase finalize API is the viable but more complex alternative.
### AUD-WARN-005 — Validation Skips Are Incomplete But Not Warned
- **Priority:** Medium operator/correctness impact.
- **Evidence:** `firstIncompleteValidation` treats failed and skipped validators
alike, and validation summaries include both. `incompleteValidationWarnings`
emits records only for `validationFailed`. The focused test demonstrates
three incomplete validators but two warnings.
- **Impact:** A successful run can have `validation_status: incomplete` while
its warning count understates or even omits the affected validators. A caller
that checks only warnings receives a weaker signal than the manifest and
receipt status.
- **Recommendation:** Produce one aggregated incomplete-validation warning
group whose occurrences cover both failure and skip, while retaining typed
outcome and safe reason metadata in the validation summary. Do not expose
provider errors or arbitrary skip prose in model or operator messages.
### AUD-WARN-006 — Relatedness Checks Are Useful But Repetitive And Lexically Weak
- **Priority:** Medium operator impact; low acceptance-policy urgency.
- **Evidence:** Every family runs the advisory in both extract and normalize
chains. The complete fixture contains exact repeated tuples, and the checks
rely on exact normalized token sequences or a minimal significant-token
overlap. Reason naming drifts between `*_not_near_source` and
`*_source_unrelated`.
- **Impact:** The checks can catch unsupported entities, but aliases, pronouns,
inflection, and generic scene prose create predictable false positives. Flat
per-record presentation magnifies them.
- **Recommendation:** Retain the validators and their stage-local execution,
but classify and aggregate them as advisories. Normalize reason-code naming
when the diagnostic contract changes. Do not strengthen them into rejection
rules without a human-reviewed production evaluation.
### AUD-WARN-007 — `warning_count` Has No Stable Operational Meaning
- **Priority:** High downstream-contract impact.
- **Evidence:** The receipt and CLI use `len(output.Warnings)`. One list element
can be an omission summary representing several hidden occurrences; repeated
records can represent the same condition; and skipped validators can be
absent. A 21-occurrence spell test produces a list length of 20.
- **Impact:** The value is neither an exact occurrence count nor a distinct
warning-group count. Consumers cannot set policy or present a trustworthy
summary from it.
- **Recommendation:** Introduce explicit warning-group and warning-occurrence
counts in a versioned receipt. Do not silently redefine the v1 field.
## Recommended Target Contract And Presentation Model
### Diagnostic Model
Use one validated internal diagnostic model with these concepts:
- **disposition:** `warning`, `advisory`, or `observation`;
- **category:** a small enum such as `configuration`, `degradation`,
`validation_incomplete`, `data_quality`, `fallback`, or `normalization`;
- **reason code:** stable semantic identity owned by the producer;
- **origin:** framework-added phase/stage, step ID, lane ID, module key,
validator name, chunk ID, and chunk index when applicable;
- **occurrence count:** exact number of matching findings;
- **samples:** a small deterministic list of bounded scope/message pairs; and
- **omitted sample count:** `occurrence_count - len(samples)`, represented as
metadata rather than another diagnostic record.
Errors and rejected outputs must not become diagnostic dispositions. A warning
means that the run completed under policy despite a process-level degradation
or incomplete configured operation. Advisory and observation dispositions can
describe accepted artifact quality and transformation provenance, but no
LLM-judged extraction-quality signal may be promoted to a warning. Validation
status remains authoritative for approval, rejection, and incomplete
validation.
### Aggregation
The framework runner should own aggregation after it enriches findings with
origin and before public output construction. Modules and validators retain
semantic ownership of disposition, category, reason, scope, and message; they
must not own CLI or file presentation.
The default stable key should be:
```text
disposition + category + reason_code
+ phase/stage + step_id + lane_id + module_key + validator_name
```
Chunk, record scope, and message text belong in samples and must not be part of
the group key. This groups repeated per-chunk findings without merging the same
code across distinct producers or pipeline locations. Group order should be
the first occurrence in the runner's existing canonical order; sample order
should follow the same order. A final canonical sort by the complete origin key
is also viable, but completion timing must never choose either order.
Aggregation must be incremental and bounded. Producers should use a shared
collector that counts every occurrence while retaining only bounded samples;
the framework then merges producer groups without reconstructing counts from
omission prose. A global maximum group count is also required, with overflow
represented by structured aggregate metadata and with actionable groups given
priority over lower dispositions.
### Durable Files
Keep one canonical home for each class:
- `warnings.json` should contain versioned, grouped actionable warnings only;
- a new `diagnostics.json` should contain versioned advisory and observation
groups only, avoiding duplication of warning groups;
- `index.json` should link both files;
- `rejected.json` and manifest validation summaries should retain their current
separate responsibilities; and
- debug bundles should retain candidate-attempt detail plus the final grouped
projections.
This is preferable to keeping all detail in `warnings.json` and filtering only
the CLI: downstream consumers would otherwise continue to receive a semantically
mixed warning contract, and routine observations would still dominate the
durable file.
### CLI And Receipt
For a successful human run with actionable warnings, print a concise summary
such as:
```text
notarius: run completed with 2 warning groups (7 occurrences); details=/.../warnings.json
```
Advisories and observations should not produce the warning line. Their durable
path remains discoverable through `index.json`; a concise non-warning count can
be added to the ordinary success line only if operator testing shows value. An
ordinary successful run with no process degradation should write nothing to
the warning stream even when it publishes quality advisories or normalization
observations.
Create `notarius.run-result.v2` rather than redefining v1. It should expose at
least:
- `warning_group_count`;
- `warning_occurrence_count`; and
- `diagnostic_group_count` for non-warning durable groups.
The receipt should continue to expose validation status, validation summaries,
and rejected-output count independently. Process exit behavior should not
change as part of warning presentation reform.
### Checkpoint Semantics
Store the structured terminal diagnostic groups with accepted checkpoints and
replay them exactly once at their logical stage. Fresh and reused runs should
produce the same public groups and counts. Checkpoint events and debug records,
not diagnostic identity, should disclose whether computation was reused.
### Output Encoder Boundary
Prefer removing `OutputResult.Warnings`. A successful output encoder should
either return the complete logical files or fail. If future encoders genuinely
need to produce durable post-encoding warnings, introduce an explicit
two-phase prepare/finalize contract so those warnings can be included in the
same published collection. Do not retain the current self-inconsistent
one-pass capability.
## Resolution Of Required Design Questions
| Question | Recommendation | Viable alternative and tradeoff |
| --- | --- | --- |
| Explicit severity/disposition or external reason mapping? | Put validated disposition and category in the contract. | A central reason-code registry avoids payload fields but makes new modules depend on a second synchronized policy table and leaves downstream meaning implicit. |
| Keep all detail in `warnings.json` or separate it? | Separate grouped actionable warnings from grouped advisories/observations in `diagnostics.json`. | Keep the flat durable list and aggregate only CLI output; simpler migration, but it preserves the noisy downstream contract and ambiguous count. |
| Who owns aggregation? | Framework runner/coordinator after origin enrichment. | Output module aggregation keeps framework types smaller but duplicates policy across encoders and cannot repair missing validator origin. |
| Stable aggregation key? | Disposition, category, reason code, and full producer origin; exclude chunk/scope/message. | Explicit producer-supplied grouping keys offer flexibility but add another identity that can drift from reason codes. Message-template grouping is brittle and unsafe. |
| Samples and omissions? | Exact occurrence count plus deterministic bounded samples and numeric omitted-sample count. | Omission warning records preserve the current representation but inflate group counts and require prose parsing. |
| `warning_count` semantics? | Version receipt and replace ambiguity with group and occurrence counts. | Keep v1 count as published record length and add optional fields; compatible, but two competing warning counts remain easy to misuse. |
| Which normalization changes remain warnings? | Only exhausted process fallback. Unresolved membership is a data-quality advisory; successful canonicalization, reordering, ID repair, source-ref dedupe, and duplicate consolidation are observations. | Treat unresolved membership or semantic duplicate consolidation as warnings because they affect grounding or cardinality; more conservative, but it violates the process-only warning rule and reports accepted artifact quality as an operational failure. |
| Source-relatedness disposition? | Grouped advisory by default; preserve current approve behavior. | Retain warning disposition or make rejection configurable. Rejection requires production precision evidence; current lexical rules are not strong enough. |
| Checkpoint-loaded warnings? | Present the same logical groups as fresh execution and use checkpoint events for reuse provenance. | Mark groups as replayed; aids forensics but fragments aggregation and makes semantically equivalent runs differ. |
| ADR and schema versions? | Add an ADR and version the run receipt and diagnostic files. | Treat the work as CLI-only presentation and avoid an ADR; insufficient because module contracts, output files, checkpoint payloads, and downstream fields change. |
## Compatibility, Documentation, And ADR Implications
The target alters public and internal contracts enough to require a new ADR.
It should record:
- the distinction among warnings, advisories, observations, rejections, and
errors;
- the invariant that warnings are process-level signals, LLM-judged extraction
quality is never a warning, and ordinary non-degraded success has zero
warnings;
- module semantic ownership versus framework origin/aggregation ownership;
- bounded group and sample semantics;
- fresh/checkpoint equivalence; and
- the output-encoder decision.
Implementation should introduce `notarius.run-result.v2`. The grouped warning
and diagnostic envelopes should each carry their own schema version. Because
the content of `warnings.json` changes incompatibly from a flat array wrapper
to groups, release notes and the published JSON integration contract must call
out the migration. `index.json` gains the diagnostic file path.
Canonical documentation updates belong in:
- `docs/cli.md` for stderr presentation only;
- `docs/operations.md` for operator review and debug workflow;
- `docs/integrations/json-output.md` for warning and diagnostic file schemas;
- `docs/integrations/run-result.md` for v2 fields and compatibility;
- `docs/consumers/subprocess.md` and `docs/consumers/dnd-pipeline.md` for
downstream policy checks;
- `docs/internal/pipeline.md` for promotion, aggregation, retry, and checkpoint
mechanics;
- `docs/internal/modules.md` and `docs/internal/dnd.md` for producer rules and
the D&D classification matrix; and
- `docs/policy/architecture.md` for the durable ownership invariant after the
ADR is accepted and implemented.
No configuration knob is required for the first implementation. A fixed,
well-documented taxonomy is easier to reason about than per-reason display
overrides. Configurable escalation or suppression can be considered only after
production review demonstrates a concrete operator need.
## Test Coverage Assessment
Existing coverage worth preserving includes:
- accepted-attempt and terminal-rejection warning promotion;
- validator failure retry exhaustion and `warn_continue`;
- module semantic retry fallback;
- deterministic warning order under concurrent lane completion;
- chunk-plan invalidation and discarded-cache warning behavior;
- fresh/checkpoint warning equivalence;
- local D&D warning caps and safe dynamic-message quoting;
- JSON warning-file publication; and
- CLI stderr, debug, and receipt counts.
Material gaps are:
- no bound test for NPC- or spell-relatedness warnings;
- no generic warning-field or result-size validation;
- no test for output encoder warnings versus published `warnings.json`;
- no operator-level aggregation or bounded-sample tests;
- no fresh/resume test for grouped counts because groups do not yet exist; and
- no production evaluation of advisory precision.
Tests should protect the semantic relationships: exact occurrence counts,
bounded samples, deterministic group order, actionable-only warning
presentation, and cross-surface equality. They should not assert one exact
warning count for every complete D&D run or treat message wording as a public
API unless the wording itself enforces a security boundary.
## Audit Conclusion
The application is in a good position for warning reform. Its retry,
validation, checkpoint, and concurrency mechanics provide reliable points at
which to attach structured diagnostics. The most valuable change is not to
suppress individual reason codes; it is to replace the semantically flat,
origin-free collection with bounded typed groups and to reserve the word
“warning” for conditions that merit operator attention.
Provider-backed runs would improve prioritization and help tune the D&D
advisories, but they are not necessary to conclude that routine normalization
and heuristic doubt should not dominate stderr or the durable warning
contract. They should be gathered before changing heuristic acceptance policy
or adopting a numerical production warning-volume target.

742
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,742 @@
# Warning Signal And Presentation Implementation Plan
## Purpose
This document is the ordered implementation plan for the warning and diagnostic
reform defined by [the completed audit](audit.md). It is intended to be executed
stage by stage by a `gpt-5.6-terra` coding agent. Each numbered stage is one
implementation prompt and must leave the repository compiling, internally
coherent, and covered at the narrowest durable test boundaries relevant to that
stage.
The audit is the canonical source for findings, evidence, and target rationale.
This document is the canonical source for implementation order and task
breakdown.
## Governing Decisions
The following decisions are final for this work set.
1. **Warnings are process signals.** A warning means that the run completed
under policy despite process-level degradation or incomplete configured
work. Examples are an empty configured reference, an applicable validator
that could not complete under `warn_continue`, an unavailable required
upstream classification, or exhausted semantic-reconciliation fallback.
2. **Extraction-quality signals are not warnings.** LLM-judged uncertainty,
lexical source-relatedness findings, unresolved entity grounding, and other
accepted-artifact quality signals are advisories. They must never be
promoted to warnings merely because a model or heuristic expressed doubt.
3. **Routine successful transformations are observations.** Canonicalization,
sorting, source-reference cleanup, ID repair, and accepted duplicate
consolidation remain inspectable but do not require operator action.
4. **Ordinary successful runs have zero warnings.** A successful-but-degraded
run may have warnings when policy permits continuation. Advisory or
observation volume alone must not produce warning stderr or a nonzero
warning count.
5. **Errors and rejections remain separate.** This work must not change
validation approval, rejection, retry budgets, exit status, or error policy.
A corrected superseded attempt leaves no final warning. An exhausted
rejection remains a rejection or run failure according to existing policy.
6. **Modules own meaning; the framework owns context and presentation.** A
producer chooses disposition, category, reason code, scope, and safe message.
The framework attaches stage and pipeline origin, aggregates deterministically,
enforces bounds, and supplies the final collections to output and CLI code.
7. **Durable contracts are versioned.** The incompatible grouped warning file
is `notarius.warnings.v2`, the new diagnostic file is
`notarius.diagnostics.v1`, and the machine-readable run receipt becomes
`notarius.run-result.v2`. Do not silently redefine the v1 receipt or warning
payload.
8. **There is no first-release configuration surface for presentation policy.**
Classification, sample bounds, and aggregation are fixed application
policy. Do not add suppression, escalation, verbosity, or per-reason config
in this work set.
## Target Contract
Use the following model unless existing Go naming requires a narrowly scoped
variation. Any naming variation must preserve the specified fields and
semantics.
### Producer Diagnostic
Add a framework contract representing a locally grouped producer diagnostic:
- `disposition`: `warning`, `advisory`, or `observation`;
- `category`: one of `configuration`, `degradation`,
`validation_incomplete`, `fallback`, `data_quality`, or `normalization`;
- `reason_code`: stable, nonblank producer-owned identity;
- `occurrence_count`: exact positive number of represented occurrences;
- `samples`: deterministic bounded samples containing safe `scope` and
`message`; and
- `omitted_sample_count`: exactly `occurrence_count - len(samples)`.
Validate these combinations:
| Disposition | Allowed categories |
| --- | --- |
| `warning` | `configuration`, `degradation`, `validation_incomplete`, `fallback` |
| `advisory` | `data_quality` |
| `observation` | `normalization` |
This strict matrix is intentional. It makes the process-only warning rule a
contract invariant rather than a convention inferred from reason codes.
Use these fixed limits:
- reason code: 128 UTF-8 bytes;
- scope: 512 UTF-8 bytes;
- sample message: 4 KiB of valid UTF-8;
- retained distinct samples per group: 3; and
- groups returned by one producer or validator result: 64.
Blank or invalid required fields, invalid disposition/category combinations,
invalid UTF-8, inconsistent counts, excessive samples, or excessive local
groups make the producer result invalid. They must return a bounded contextual
error rather than be silently repaired. The three-sample and 64-group bounds
are public operational safeguards and may be asserted directly in their owning
contract tests.
The shared collector must count every occurrence and retain the first three
distinct samples in producer order. Repeated identical samples still increase
`occurrence_count`. The omission count is numeric metadata, never another
diagnostic record.
### Framework Origin And Aggregation
The framework adds this origin before final aggregation:
- stage: `references`, `chunk`, `extract`, `merge`, or `normalize`;
- step ID, when applicable;
- lane ID, when applicable;
- module key, when applicable; and
- validator key, for validator-produced diagnostics.
Chunk ID and zero-based chunk index belong on samples, not group origin,
because otherwise identical per-chunk findings cannot aggregate. Use an
optional integer representation that preserves chunk index zero.
The stable aggregation key is:
```text
disposition + category + reason_code
+ stage + step_id + lane_id + module_key + validator_key
```
Scope, message, chunk ID, and chunk index are excluded from the key. Groups and
samples retain first-occurrence order from the runner's existing canonical
ordering; completion timing must never affect the result. When groups merge,
sum exact occurrence counts and retain the first three distinct samples.
Use separate global bounds:
- at most 128 actionable warning groups; exceeding this limit is a framework
error because Notarius must not hide process degradation; and
- at most 256 advisory/observation groups. Additional non-warning groups are
omitted from representation while their occurrences contribute to an exact
`unrepresented_occurrence_count` and set `truncated: true`.
The non-warning envelope's group count means represented groups. Its occurrence
count includes represented and unrepresented occurrences. The warning
collection is never truncated, so both warning counts are exact.
### Classification Matrix
Migrate existing producers according to this matrix.
| Disposition and category | Existing conditions |
| --- | --- |
| Warning / `configuration` | `empty_reference` |
| Warning / `validation_incomplete` | `validator_execution_incomplete`, covering both exhausted failure and skip when `warn_continue` advances the candidate |
| Warning / `degradation` | `scene_classification_unavailable` from combat-turn and enemy-event extraction gates |
| Warning / `fallback` | `npc_semantic_reconciliation_exhausted`, `item_semantic_reconciliation_exhausted`, `location_semantic_reconciliation_exhausted` |
| Advisory / `data_quality` | All ten source-relatedness families; `spell_name_unresolved`; `item_occurrence_unknown_item_id`; `location_occurrence_unknown_location_id`; guarded invalid semantic-consolidation proposals |
| Observation / `normalization` | Field and whitespace normalization; durable ID recomputation; source-reference sorting/deduplication; canonical record ordering; exact or approved semantic duplicate consolidation |
Rename `item_occurrence_source_unrelated` to
`item_occurrence_not_near_source` when it moves into the new contract. Keep
other existing reason codes unless this plan explicitly changes them. Split the
item registry's internal retry reason from its accepted advisory: use
`item_semantic_retry_proposal_invalid` for the retry directive and retain
`item_semantic_proposal_invalid` only for the accepted data-quality advisory.
Local `*_warnings_omitted` reason codes disappear. Omission is represented by
group counts. Rejections, structural failures, provider failures, cancellation,
and successful corrected retries do not receive a diagnostic solely to mirror
their existing error, rejection, validation, or attempt-debug records.
### Durable And CLI Contracts
Always publish both companion files, including for empty collections:
- `warnings.json` with schema version `notarius.warnings.v2`, actionable groups,
exact `group_count`, and exact `occurrence_count`;
- `diagnostics.json` with schema version `notarius.diagnostics.v1`, advisory and
observation groups, represented `group_count`, total `occurrence_count`,
`truncated`, and `unrepresented_occurrence_count`; and
- `index.json` with both `warnings_file` and `diagnostics_file`.
The run receipt becomes `notarius.run-result.v2` and replaces ambiguous
`warning_count` with:
- `warning_group_count`;
- `warning_occurrence_count`;
- `diagnostic_group_count`;
- `diagnostic_occurrence_count`; and
- `diagnostics_truncated`.
The human CLI writes a warning summary to stderr only when
`warning_group_count > 0`. It reports group and occurrence counts plus the
durable warning path. Advisory and observation counts do not create a warning
line. Preserve the existing successful stdout summary and process exit rules.
Remove `OutputResult.Warnings`. An output encoder either returns its complete
logical files or returns an error. It cannot discover a warning after the
warning file has already been serialized.
## Stage 1 — Record The Decision And Add Core Diagnostic Primitives
### Goal
Create the durable decision record and the generic, independently tested
diagnostic model without changing current runtime behavior.
### Work
- Add `docs/adr/0015-separate-process-warnings-from-quality-diagnostics.md`
using the repository ADR format. Record the governing decisions, origin and
aggregation ownership, bounded samples, fresh/resume equivalence, versioned
public contracts, and removal of output-result warnings. Mark it accepted;
do not claim implementation is complete.
- Add the producer diagnostic, sample, disposition, category, origin, group,
and collection types under `internal/framework/contracts/`.
- Add contract validation with the exact allowed combinations and limits in
this plan.
- Add a small generic collector under `internal/framework/diagnostics/` that
groups producer-local occurrences by disposition, category, and reason code,
preserves exact counts, and retains three distinct samples.
- Keep `contracts.Warning` and existing result fields temporarily. Do not alter
current warning output in this stage.
### Tests
Use table-driven contract tests for valid classifications, invalid UTF-8,
blank/oversized fields, invalid counts, the disposition/category matrix, sample
bounds, exact repeated-occurrence counting, and deterministic distinct-sample
selection. Test behavior through exported package contracts rather than private
collector structure.
### Acceptance Criteria
- Existing application behavior and public JSON remain unchanged.
- The new primitives cannot represent an LLM-quality warning because
`warning` plus `data_quality` is rejected.
- `go test ./internal/framework/contracts ./internal/framework/diagnostics`
passes.
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
## Stage 2 — Add Framework Diagnostic Transport And Transitional Projection
### Goal
Carry structured diagnostics through every framework result boundary and add
origin-aware aggregation while preserving the current public warning contract
temporarily.
### Work
- Add structured diagnostic fields alongside legacy warning fields in chunk
plan, typed extract, typed merge, typed normalize, normalize-retry fallback,
validation, run input/output, and output request contracts.
- Extend erased typed results, producer-attempt state, lane result collection,
ordered-step handoff, and run finalization to transport diagnostics.
- Preserve validator identity in validation reports instead of flattening new
diagnostics through a warning-only helper.
- Attach current stage, step, lane, module, validator, and sample-level chunk
context at promotion time.
- Add a run-level aggregator implementing the stable key, canonical
first-occurrence ordering, count merging, and bounds from this plan.
- Preserve terminal-attempt semantics: diagnostics from superseded attempts
remain debug-only; only terminal accepted or terminal rejected candidate
diagnostics are promoted.
- Add a private transitional projection from each structured diagnostic sample
to the existing legacy `Warning` collection so public surfaces remain
unchanged until Stage 8. Mark it explicitly temporary and ensure it does not
duplicate diagnostics already supplied through the legacy path.
### Tests
Extend focused framework tests for terminal attempt promotion, terminal
rejection, reversed concurrent completion, stage/lane/chunk origin, stable
grouping across chunks, and global bounds. Use controlled test dispositions and
origins; do not assert incidental diagnostic prose.
### Acceptance Criteria
- A producer may return either legacy warnings or new structured diagnostics
during migration, but not produce duplicate public records through both.
- Structured groups are deterministic under concurrent completion.
- Superseded attempt diagnostics never reach final groups.
- Existing public warning tests still pass through the transitional projection.
- Focused framework tests and `go test ./internal/framework/pipeline` pass.
This stage is appropriately sized for one high-reasoning
`gpt-5.6-terra` prompt. Do not combine it with D&D migration.
## Stage 3 — Migrate Framework Process Signals And Close The Output Boundary
### Goal
Move framework-owned warning conditions into the new process-only contract and
remove the output-encoder consistency defect.
### Work
- Convert `empty_reference` to warning/configuration with reference origin and
bounded, non-sensitive sample content.
- Convert incomplete validation to warning/validation-incomplete whenever
`warn_continue` advances a candidate after an applicable validator either
exhausts with failure or skips. Retain validator key and typed failure/skip
detail in validation summaries; do not copy provider errors or arbitrary
validator prose into diagnostic samples.
- Aggregate all incomplete-validator occurrences rather than generating
omission warning records.
- Validate module and validator diagnostic results before promotion. An invalid
diagnostic contract is a bounded contextual framework error.
- Remove `Warnings` from `contracts.OutputResult`, the output adapter, test
doubles, and runner append logic. Keep `OutputRequest` diagnostic input; an
encoder failure remains an error.
- Add a regression test proving no post-encoding result can make receipt/debug
warning state disagree with the already encoded logical files.
### Acceptance Criteria
- Failed and skipped validators are both visible when incomplete validation is
allowed to continue.
- A fail-run validation policy still fails rather than converting the failure
into a warning.
- Output encoders have no successful post-encoding warning capability.
- Existing validation decisions, retry budgets, and exit behavior are
unchanged.
- `go test ./internal/framework/... ./internal/modules/generic/output/json`
passes.
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
## Stage 4 — Version Diagnostic-Bearing Cache And Checkpoint State
### Goal
Persist and replay structured producer diagnostics safely without allowing old
warning-only state to be interpreted as complete new state.
### Work
- Store producer-local structured diagnostics in chunk-plan and extract,
merge, and normalize checkpoint envelopes. Do not persist current pipeline
origin in source-keyed chunk-plan state.
- On reuse, attach the current run's origin at the same logical promotion point
used by fresh execution, then aggregate once.
- Bump the checkpoint workspace schema from v3 to
`notarius.workspace.v4` and the chunk-plan schema from v2 to
`notarius.chunk-plan.v3`.
- Treat earlier workspace and chunk-plan versions as incompatible cache misses,
never as corrupt fatal state and never as reusable diagnostic-complete state.
- Preserve the rule that validation-incomplete results and their dependents are
not reusable.
### Tests
Add or adapt behavioral tests for old-version invalidation, structured
diagnostic round trips, fresh/resume equality of groups and counts, exact-once
replay, and current-validator diagnostics on a reused chunk plan.
### Acceptance Criteria
- Fresh and resumed logical runs produce deeply equal diagnostic groups and
counts.
- Reuse provenance remains in checkpoint events and does not alter diagnostic
identity.
- Old cache data is safely bypassed.
- `go test ./internal/framework/checkpoint ./internal/framework/pipeline`
passes.
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
## Stage 5 — Migrate All D&D Source-Relatedness Validators
### Goal
Move all ten heuristic relatedness families from warnings to bounded
data-quality advisories.
### Work
- Migrate combat turns, enemy events, item occurrences, item registry,
location occurrences, location registry, NPC occurrences, NPC registry,
scene descriptions, and spells source-relatedness validators.
- Use the shared generic collector so every family returns exact occurrence
counts and at most three distinct samples. Remove each relatedness omission
reason code.
- Bring NPC and spell validators under the same contract bounds as every other
family.
- Rename `item_occurrence_source_unrelated` to
`item_occurrence_not_near_source`.
- Preserve current approval behavior, validator chain registration and order,
lexical algorithms, scope selection, and messages except for changes needed
to satisfy generic safety bounds.
### Tests
Consolidate repetitive limiter tests where a shared collector contract already
owns the behavior. Retain family tests for realistic triggering and
non-triggering cases. Prove that a relatedness finding is advisory/data-quality
and cannot increment process warning groups.
### Acceptance Criteria
- All ten relatedness validators use the same structured advisory convention.
- No relatedness validator returns a legacy warning or an omission record.
- High-cardinality NPC and spell results remain bounded while reporting exact
occurrence counts.
- `go test ./internal/modules/dnd/validate/...` passes.
This repetitive but cohesive migration is appropriately sized for one
`gpt-5.6-terra` prompt. Do not combine it with normalizer migration.
## Stage 6 — Migrate D&D Registry Normalizers And Semantic Reconciliation
### Goal
Apply the new taxonomy to NPC, item, and location registry normalization and
semantic-reconciliation outcomes.
### Work
- Migrate deterministic field cleanup, ID recomputation, source-reference
normalization, ordering, and accepted duplicate consolidation to
observation/normalization groups.
- Migrate exhausted NPC, item, and location semantic reconciliation to
warning/fallback groups.
- Migrate guarded invalid item semantic proposals to advisory/data-quality.
- Split the item retry directive reason to
`item_semantic_retry_proposal_invalid`; keep correction control data separate
from the accepted advisory reason and from model-facing text.
- Replace local warning limiters and omission records with the shared
diagnostic collector while preserving artifact values, safe fallback, and
retry budgets.
### Tests
Protect exact artifact outcomes, classification, retry exhaustion, safe
currency behavior, occurrence counts, and bounded samples. Do not preserve
old warning slice lengths or omission prose.
### Acceptance Criteria
- Successful registry cleanup and consolidation produce observations only.
- A guarded model proposal produces an advisory, not a warning.
- Exhausted semantic reconciliation is the only registry-normalizer process
warning family.
- Semantic retry behavior and artifacts are unchanged.
- Registry normalizer and semantic-reconciliation tests pass.
This stage is appropriately sized for one high-reasoning
`gpt-5.6-terra` prompt.
## Stage 7 — Migrate Remaining D&D Producers
### Goal
Complete D&D diagnostic classification across spells, occurrences, combat,
enemy events, scenes, and extraction gates.
### Work
- Migrate spell, combat-turn, enemy-event, item-occurrence,
location-occurrence, NPC-occurrence, and scene-description normalizers.
- Classify deterministic cleanup, ordering, source-reference normalization,
and duplicate consolidation as observation/normalization.
- Classify unresolved spell, item, or location membership as
advisory/data-quality, never warning.
- Convert combat-turn and enemy-event `scene_classification_unavailable`
extraction gates to warning/degradation with module and chunk provenance.
- Remove all remaining D&D local warning omission reason codes and all uses of
`LimitWarnings`; retain text-safety helpers that still have value.
### Tests
Adapt family tests to assert artifact invariants and classification. Add one
assembled D&D test demonstrating that quality advisories and normalization
observations can be present while the warning group count remains zero.
### Acceptance Criteria
- No D&D producer returns a legacy `contracts.Warning`.
- No accepted-artifact uncertainty or unresolved grounding appears as a
process warning.
- Missing required scene classification remains an actionable process warning.
- All D&D package tests pass.
This stage is appropriately sized for one high-reasoning
`gpt-5.6-terra` prompt.
## Stage 8 — Remove The Legacy Warning Path And Finalize Aggregation
### Goal
Make the structured diagnostic collection the sole in-memory signal path.
### Work
- Migrate any remaining generic test modules and framework fixtures from
`contracts.Warning` to structured diagnostics.
- Remove `contracts.Warning`, legacy `Warnings` fields, clone helpers, warning
append helpers, transitional projections, and obsolete D&D warning limiters.
- Make `RunOutput` and `OutputRequest` carry the finalized warning and
non-warning grouped collections derived from one aggregator result.
- Enforce the 128-warning-group error bound and 256-non-warning-group truncation
behavior. Ensure occurrence counts remain exact and truncation metadata is
deterministic.
- Verify all producers are validated at their stable framework boundary.
### Tests
Add focused behavioral coverage for warning overflow failure, non-warning
truncation, exact occurrence totals, distinct sample retention, stable group
order, and no legacy double-counting. Remove tests whose only purpose was to
assert old flat-list caps or omission prose.
### Acceptance Criteria
- Repository search finds no production `contracts.Warning`, legacy warning
result field, or `*_warnings_omitted` reason code.
- One finalized collection supplies all later surfaces.
- An advisory-only successful run has zero warning groups.
- `go test ./internal/framework/... ./internal/modules/dnd/...` passes.
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
## Stage 9 — Publish Versioned Warning And Diagnostic Files
### Goal
Replace the flat durable warning payload with the two target grouped contracts.
### Work
- Update the production JSON encoder to always emit grouped `warnings.json`
(`notarius.warnings.v2`) and `diagnostics.json`
(`notarius.diagnostics.v1`).
- Include exact counts and diagnostic truncation metadata specified above.
- Add `diagnostics_file` to `index.json`; retain `warnings_file`.
- Ensure warning groups appear only in `warnings.json` and advisory/observation
groups appear only in `diagnostics.json`.
- Update `docs/integrations/json-output.md` in the same stage. It owns file
names, envelope schemas, count meanings, group/sample fields, bounds,
truncation, and index discovery. Link rather than duplicate CLI behavior.
- Update maintained output examples or test fixtures only where they encode
implemented output contracts.
### Tests
Use encoder contract tests for empty and populated files, schema versions,
partitioning, counts, truncation, index paths, deterministic JSON ordering, and
newline/valid-JSON conventions. Update assembled pipeline tests to compare both
durable files with the finalized in-memory collections.
### Acceptance Criteria
- Every production JSON bundle contains both companion files and index paths.
- No diagnostic appears in both files.
- Published counts agree with the in-memory collection.
- Output encoder and maintained example contract tests pass.
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
## Stage 10 — Introduce Run Result V2 And Quiet CLI Presentation
### Goal
Give subprocess consumers unambiguous counts and make an ordinary successful
run quiet on the warning stream.
### Work
- Change the emitted receipt to `notarius.run-result.v2` with the five fields
specified in the target contract. Remove v1 `warning_count`; do not emit two
competing count models.
- Derive receipt counts from the same finalized collection used by the JSON
encoder.
- Print a stderr warning summary only when actionable warning groups exist. It
must include group count, occurrence count, and the output-relative or
absolute details path consistent with current CLI path conventions.
- Do not mention advisory/observation counts as warnings. Preserve ordinary
stdout completion output and all exit classifications.
- Update `docs/cli.md`, `docs/integrations/run-result.md`,
`docs/consumers/subprocess.md`, and `docs/consumers/dnd-pipeline.md` in the
same stage. Each document must keep to its canonical scope and link to the
grouped JSON contract rather than duplicating schemas.
### Tests
Cover zero-warning approved success, advisory-only success, degraded success
with warnings, rejected output, JSON receipt delivery, and non-production
output modules. Assert semantic fields and stream choice, not complete prose.
### Acceptance Criteria
- Advisory-only and observation-only successful runs write no warning stderr.
- Degraded success writes one concise actionable warning summary.
- Receipt counts and durable files agree.
- Receipt, CLI command, and consumer contract tests pass.
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
## Stage 11 — Align Debug, Manifest, And Resume Surfaces
### Goal
Ensure forensic and provenance surfaces reflect the new model without creating
another competing warning contract.
### Work
- Replace final flat warning debug summaries with final grouped warning and
non-warning diagnostic projections, using separate clearly named files or
one versioned diagnostic summary envelope consistently with existing debug
layout.
- Keep attempt-local candidate diagnostics in detailed debug records with
stage/attempt provenance. Preserve redaction and the rule that raw model
responses and correction messages appear only in detailed debug.
- Keep manifest validation and rejection summaries authoritative for validation
status; do not duplicate full diagnostic groups into the manifest.
- Ensure checkpoint events, not diagnostic origin, identify reuse.
- Update `docs/operations.md`, `docs/internal/state.md`, and the debug portions
of `docs/internal/pipeline.md` in the same stage.
### Tests
Cover successful debug capture, partial failure debug capture,
terminal-attempt-only final grouping, redaction, and fresh/resume equality.
Retain existing validation summary tests rather than duplicating every outcome
in diagnostic tests.
### Acceptance Criteria
- Debug consumers can distinguish process warnings, quality advisories, and
observations.
- Debug final counts match receipt and durable output when output is published.
- Attempt detail remains bounded and sensitive-data rules are preserved.
- Debug, state, and resume-focused tests pass.
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
## Stage 12 — Update Architecture And Internal Producer Documentation
### Goal
Make the implemented ownership and classification rules durable for future
modules without duplicating public schemas.
### Work
- Update `docs/policy/architecture.md` to state the process-only warning
invariant, module semantic ownership, framework origin/aggregation ownership,
and ordinary-success-zero-warnings expectation. Link to ADR-0015 for
rationale.
- Update `docs/internal/pipeline.md` for terminal promotion, origin enrichment,
aggregation, bounds, concurrency ordering, checkpoint replay, and output
handoff.
- Update `docs/internal/modules.md` with the generic producer contract and the
prohibition on using warnings for extraction quality.
- Update `docs/internal/dnd.md` with the implemented classification matrix and
shared collector convention. Link to integration contracts for durable
schemas rather than restating them.
- Update `docs/internal/overview.md` only if the new generic diagnostics package
changes the component inventory.
- Remove stale terminology from current docs outside `docs/roadmap/`, but do
not write release history or document unimplemented configuration.
### Acceptance Criteria
- Each volatile fact has one canonical owner under the documentation policy.
- Future module authors can determine which disposition/category to use and
where origin and aggregation are attached.
- Current documentation contains no claim that heuristic quality doubt is a
warning.
- Relative Markdown links resolve.
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
## Stage 13 — Final Behavioral Verification And Cleanup
### Goal
Verify the complete migration, remove transitional residue, and establish that
the audit findings are fully addressed.
### Work
- Run repository searches for legacy warning types, warning omission reason
codes, old run-result v1 fields, old flat warning JSON assumptions, and
`OutputResult.Warnings`.
- Review every producer inventoried in `audit.md` against the final
classification matrix.
- Run the maintained minimal and complete D&D workflows with offline fake LLM
clients. Do not require provider credentials or add live calls to the default
suite.
- Verify these relationships end to end:
- ordinary approved success and advisory-only success have zero warnings;
- successful process degradation has a warning;
- corrected retries retain only terminal diagnostics;
- rejected and failed runs retain their established semantics;
- warning and diagnostic group order is deterministic under concurrency;
- fresh and resumed runs are equivalent;
- global group and sample bounds hold; and
- receipt, stderr, durable files, index, and debug counts agree.
- Remove obsolete helpers and redundant tests made unnecessary by stronger
package-level collector or end-to-end contract tests.
### Validation Commands
Run at minimum:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
```
Also run any repository documentation or link checker discovered during the
stage. If none exists, perform a focused relative-link review for the documents
changed by Stages 9–12.
### Acceptance Criteria
- All three repository-wide Go commands pass offline.
- No live API key is required.
- No legacy flat-warning code path or public v1 warning-count claim remains in
current-behavior documentation.
- All seven audit findings are addressed without changing artifact values,
validator chain order, retry budgets, rejection policy, or exit semantics.
- The ordinary maintained successful workflow emits zero actionable warnings;
non-warning diagnostics remain available in `diagnostics.json`.
This final verification is appropriately sized for one `gpt-5.6-terra`
prompt.
## Deferred Work
The following are deliberately outside this implementation plan:
- provider-backed measurement of production advisory precision or frequency;
- configurable warning suppression, escalation, verbosity, or reason filters;
- converting heuristic relatedness checks into rejection rules;
- the planned LLM-backed D&D combat-scene validator;
- metrics or telemetry export beyond the specified durable and debug files;
- a two-phase output encoder protocol; and
- release preparation or release-note creation.
Provider-backed production data may inform later advisory tuning, but it is not
required to implement or validate this architecture.