notarius/docs/roadmap/audit-plan.md

# Warning Signal And Presentation Audit Plan

## Purpose

This document defines the audit required before redesigning Notarius warning
semantics and presentation. The audit must explain why an ordinary successful
D&D run can produce dozens of warnings, distinguish actionable degradation
from routine diagnostic observations, and recommend a bounded, deterministic
warning contract that operators will actually review.

This is an audit plan, not a feature design or implementation plan. The audit
must first establish the current producers, propagation behavior, empirical
volume, public contracts, and compatibility constraints. The completed audit
selects the target behavior from those findings; `implementation.md` owns the
subsequent implementation sequence.

## Context

Notarius currently uses one `contracts.Warning` shape—optional `scope`, required
`reason_code`, and required `message`—for findings from modules, validators,
normalizers, reference preparation, retries, and framework policy. Accepted
warnings are collected by the pipeline, counted by the CLI and machine-readable
run receipt, and published as a flat array in `warnings.json`.

The architecture already guarantees deterministic public ordering and bounded
warning production at several local boundaries. Recent validation work also
gave incomplete validation explicit provenance. Those guarantees must be
preserved. The observed problem is signal quality and aggregate volume: a
successful run can report many warnings even when most describe routine,
expected cleanup or weak heuristic observations rather than conditions that
require operator action.

Initial repository inspection identifies several likely warning families:

- D&D source-relatedness validators can produce one warning per artifact
  record when a contextual name is not found in its cited text;
- deterministic normalizers report canonicalization, whitespace cleanup,
  ordering changes, source-reference cleanup, and duplicate consolidation;
- semantic registry reconciliation reports discarded proposals, exhausted
  fallback, and duplicate consolidation;
- optional reference materialization can report unavailable or skipped
  reference inputs;
- the producer-attempt state machine reports validators that exhausted their
  execution budgets under `warn_continue`; and
- producer modules can return their own warnings, which are promoted only from
  the terminal accepted or rejected attempt.

These are audit hypotheses. The audit must verify exact behavior and frequency
from current code and representative runs rather than assume that every family
is noisy or incorrectly classified.

## Policy And Architecture Constraints

The audit and its recommendations must follow:

- `docs/policy/architecture.md` for module/framework ownership, deterministic
  ordering, validation semantics, durable provenance, and sensitive-data
  boundaries;
- `docs/policy/documentation.md` for canonical documentation ownership and the
  distinction between current and future behavior;
- `docs/policy/testing.md` for risk-based, behavior-oriented test
  recommendations; and
- ADR-0014 for the distinction between semantic rejection, validator failure,
  correction guidance, and durable diagnostic provenance.

In particular:

- modules and validators may report findings, but the framework and output
  boundaries own aggregation and presentation;
- reducing warning volume must not silently change artifact acceptance,
  validation policy, retry behavior, rejection behavior, or process exit
  status;
- genuine validator execution failures and validation-incomplete outcomes must
  remain visible and auditable;
- public ordering must remain deterministic despite concurrent lane execution;
- aggregation must be bounded and must not leak raw model responses,
  correction guidance, credentials, private reference content, or unnecessary
  transcript text; and
- detailed diagnostics may move to a more appropriate durable or debug surface,
  but must not be discarded when they are needed to understand a lossy or
  degraded result.

## Audit Objectives

The audit must answer five questions.

1. Which code paths produce warnings, under what conditions, and with what
   expected multiplicity?
2. How are warnings cloned, bounded, replayed, ordered, promoted, persisted,
   counted, and presented from their producer through the final CLI and output
   bundle?
3. Which warnings identify actionable degradation or data-quality risk, and
   which merely describe routine successful transformations or advisory
   heuristics?
4. What aggregation and presentation model would materially reduce operator
   noise without hiding genuine failures, lossy fallback, or incomplete
   validation?
5. Which proposed changes affect only presentation, and which would alter a
   public receipt, durable output contract, validation invariant, or other
   compatibility boundary?

## Scope

### Warning Producers

Inventory every production warning constructor and direct warning literal
under `internal/`. Include warnings originating from:

- input, chunk, extract, merge, normalize, and output modules;
- typed, chunk, and serialized validators;
- semantic reconciliation and normalizer fallback;
- external and generated reference materialization;
- chunk-plan and checkpoint reuse or fallback;
- producer-attempt exhaustion and validation-incomplete continuation;
- pipeline orchestration and cancellation handling; and
- CLI or publication logic, if it creates warnings rather than only presenting
  them.

Do not infer completeness from one textual search. Inspect shared constructors,
returned result types, reason-code constants, registration paths, and tests that
exercise warning behavior.

### Warning Propagation

Trace each warning family through:

- stage result and validation result contracts;
- producer attempts, including abandoned attempts, semantic retries, structural
  retries, terminal rejection, and `warn_continue`;
- concurrent lane collection and stable public ordering;
- merge and normalize continuation;
- chunk-plan caching and checkpoint recording or hydration;
- ordered step output merging and generated-reference handoff;
- `RunOutput`, run manifests, validation summaries, debug records, and CLI
  result construction;
- `warnings.json`, `manifest.json`, the run-result receipt, human-readable
  standard error, and debug bundles.

For every boundary, determine whether warnings are copied, filtered,
deduplicated, bounded, summarized, replayed from reusable state, or dropped.
Pay particular attention to amplification across chunks, lanes, validators,
retries, and resumed runs.

### D&D Warning Semantics

Review every implemented D&D artifact family. At minimum, distinguish:

- evidence/source-relatedness advisories;
- deterministic canonicalization or whitespace changes;
- order and source-reference normalization;
- exact and semantic duplicate consolidation;
- unresolved catalog or registry membership;
- semantic-reconciliation proposal failure or fallback; and
- validator execution incompleteness.

Determine whether the reason-code vocabulary is consistent across artifact
families, whether scopes are sufficiently contextual, and whether equivalent
conditions produce near-duplicate warnings with different codes or prose.

### Public And Operator Surfaces

Review the implemented contracts and documentation for:

- CLI success output and warning-count output;
- `notarius.run-result.v1` and its `warning_count` and validation fields;
- `warnings.json` and the published JSON index;
- manifest validation summaries and rejection summaries;
- debug summary and detailed debug artifacts; and
- downstream subprocess guidance, especially the complete D&D consumer
  workflow.

Identify which surfaces are intended for immediate operator attention, durable
machine consumption, forensic detail, or debugging. Record any places where
the same flat count or warning list is being asked to serve incompatible
audiences.

### Tests And Documentation

Inventory tests that protect warning production, bounds, ordering, retry
promotion, checkpoint replay, JSON publication, receipt counts, and CLI stream
behavior. Identify meaningful gaps, redundant exact-prose assertions, and tests
that would unnecessarily obstruct a taxonomy or aggregation redesign.

Review `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
`docs/integrations/json-output.md`, `docs/integrations/run-result.md`,
`docs/consumers/`, and relevant internal documentation for current warning
claims. Record the canonical document that would own each future contract
change; do not rewrite those documents during the audit.

## Out Of Scope

The audit must not:

- implement warning filtering, severity levels, aggregation, or new CLI flags;
- change validator decisions, default chains, retry budgets, or terminal
  validation policy;
- suppress warnings merely to meet a numerical target;
- redesign rejections, errors, logs, metrics, or debug bundles except where
  their boundary with warnings must be clarified;
- add the planned D&D combat-scene semantic validator;
- create deterministic tests that assert one exact global warning count for
  all future runs; or
- use live paid LLM calls as part of the default automated test suite.

## Inventory Method

Create a warning inventory with one row per semantically distinct production
condition. Each row should record:

| Field | Required analysis |
| --- | --- |
| Producer | Package, function, module or validator key, stage, and artifact family. |
| Trigger | Exact condition that emits the warning and whether it follows successful mutation, heuristic doubt, fallback, or failure. |
| Identity | Reason code, scope format, and whether either is stable enough for aggregation or machine use. |
| Multiplicity | Maximum per record, chunk, lane, validator, attempt, step, and run. |
| Lifecycle | Whether abandoned attempts are discarded, terminal warnings promoted, and cached or checkpointed warnings replayed. |
| Consequence | Whether data changed, evidence is questionable, output is incomplete, fallback occurred, or no externally meaningful consequence exists. |
| Actionability | What an operator can reasonably do in response, if anything. |
| Surfaces | CLI, receipt count, `warnings.json`, manifest, rejection, checkpoint, or debug presence. |
| Bounds | Existing local caps, message limits, omission records, and any missing aggregate bound. |
| Sensitivity | Whether scope or message can contain source-derived or otherwise sensitive content. |
| Coverage | Existing tests and the meaningful regression risk they protect. |

Treat different reason codes that represent the same operator condition as
potential consolidation candidates, but do not merge them in the audit
document without explaining lost diagnostic information.

## Propagation Analysis

Produce a compact propagation map from warning creation to each terminal
surface. The analysis must explicitly verify:

- only the terminal candidate's warnings are promoted after retries;
- whether rejected candidates retain warnings and where;
- whether `warn_continue` creates one warning per exhausted validator and how
  that relates to validation summaries;
- whether cached chunk plans or hydrated checkpoints replay historical warnings
  into the current run;
- whether the same warning can be appended at more than one handoff boundary;
- how concurrent completion is reordered before publication;
- whether local per-module caps compose into an unbounded or excessively large
  run-level result; and
- whether warning counts on stderr, receipts, manifests, and `warnings.json`
  refer to exactly the same collection.

Any suspected duplicate append or unstable ordering is a correctness finding,
not merely a presentation concern.

## Empirical D&D Run Analysis

Static inspection must be supplemented with representative run evidence. Use
at least:

- one ordinary successful complete D&D run known to produce high warning
  volume;
- one smaller maintained example or synthetic run;
- one run with semantic registry reconciliation activity;
- one run with a validator execution failure allowed through
  `warn_continue`, using an offline test double where practical; and
- one retrying producer case to verify abandoned-attempt warning treatment.

For real campaign runs, analyze only bounded metadata unless the operator
explicitly provides source content for review. Record counts grouped by stage,
lane, module or validator, reason code, and scope family. Also record unique
reason-code count, repeated-message count, maximum group size, validation
status, rejected-output count, and whether each warning led to a plausible
operator action.

Compare the same logical run under ordinary execution and checkpoint resume
when practical. The audit must distinguish warning volume caused by actual data
conditions from volume caused by orchestration or replay.

Do not make a model-quality judgment solely from warning frequency. Manually
inspect a bounded sample from each high-volume reason code to estimate false
positive rate and operational value.

## Classification Rubric

Classify each warning condition along independent dimensions rather than force
an immediate single severity enum:

- **result impact:** none, routine mutation, lossy mutation, uncertain data
  quality, fallback, or incomplete validation;
- **operator action:** none, informational review, configuration or reference
  correction, source/model review, or rerun required;
- **scope:** record, chunk, lane, step, pipeline, or infrastructure;
- **persistence need:** top-level attention, durable detail, debug-only detail,
  or metric/trace candidate; and
- **confidence:** deterministic fact, heuristic advisory, or execution failure.

The audit should then test whether a small durable taxonomy can represent the
meaningful combinations. A promising starting hypothesis is that top-level
operator warnings should be limited to actionable degradation, incomplete
validation, lossy fallback, and material data-quality risk, while routine
successful normalization observations remain available as lower-level durable
diagnostics. The audit must validate or revise that hypothesis from evidence.

## Design Questions The Audit Must Resolve

The findings must give a recommendation, with at least one viable alternative
and tradeoffs, for each of these questions:

1. Should warning severity or disposition become an explicit contract field,
   or should stable reason-code metadata drive presentation policy outside the
   warning payload?
2. Should `warnings.json` remain the complete durable detail while the CLI and
   receipt expose an aggregated actionable summary, or should durable warnings
   themselves be separated from routine observations?
3. Where should global deduplication and aggregation live so modules retain
   semantic ownership but concurrent pipeline results remain deterministic?
4. What is the stable aggregation key: reason code, stage/lane/module identity,
   normalized scope, message template, or an explicit structured grouping key?
5. How should bounded samples and omitted counts be represented without
   converting a summary record into another warning that inflates the count?
6. Should `warning_count` continue to mean the length of `warnings.json`, or
   should a new receipt or schema field distinguish actionable warning groups
   from detailed observations?
7. Which normalization changes are sufficiently lossy or surprising to remain
   warnings, and which are ordinary provenance that belongs in manifest or
   debug data?
8. Should heuristic source-relatedness findings remain warnings, become
   grouped data-quality advisories, or be strengthened into configurable
   validation decisions only after demonstrated precision?
9. How should warnings loaded from checkpoints be identified or aggregated
   relative to newly produced warnings?
10. Does the chosen target alter durable or validation semantics enough to
    require a new ADR or a versioned run-result/output contract?

## Required Audit Deliverable

Write the completed findings to `docs/roadmap/audit.md`. It should contain:

1. an executive assessment of current warning quality and risk;
2. the complete warning-producer inventory;
3. the warning propagation and surface map;
4. empirical measurements and bounded representative samples;
5. findings ranked by operator impact, correctness risk, and implementation
   leverage;
6. a recommended target taxonomy and presentation model;
7. compatibility, documentation, ADR, and migration implications;
8. implementation implications and dependencies sufficient to support a
   separate implementation plan; and
9. open decisions only where repository evidence cannot support a responsible
   recommendation.

Each finding should identify the supporting code paths, tests, documentation,
and run evidence. Separate observed facts from recommendations and avoid
changelog or development-history framing.

## High-Level Audit Sequence

The detailed execution sequence should be written separately if needed. At a
high level, perform the audit in this order:

1. **Static inventory:** enumerate warning producers, reason codes, scopes,
   bounds, and existing tests.
2. **Propagation audit:** trace promotion, ordering, replay, persistence,
   counting, and presentation across the framework and CLI.
3. **Empirical analysis:** measure representative D&D runs and inspect bounded
   samples from high-volume warning groups.
4. **Classification:** apply the rubric, identify duplicate concepts and
   misplaced routine diagnostics, and evaluate public-contract options.
5. **Synthesis:** rank findings and recommend a decision-complete target for a
   subsequent feature roadmap.

Static inventory and propagation may be performed as separate focused agent
prompts. Empirical analysis should be isolated because it may require operator
artifacts or opt-in provider execution. Classification and synthesis should
consume the earlier written evidence rather than rediscover the repository.

## Validation Of The Audit

Before considering the audit complete, verify that:

- every production warning literal or constructor is represented in the
  inventory;
- every reason code observed in representative `warnings.json` files maps to a
  known producer or is recorded as an unexplained finding;
- counts agree across the runner result, CLI receipt, stderr summary, and
  published warning collection for each sampled run;
- retry, rejection, incomplete-validation, cache, checkpoint, concurrency, and
  ordered-step paths are covered;
- recommended aggregation preserves deterministic ordering and bounded memory;
- recommendations distinguish warnings from errors, rejections, validation
  summaries, logs, and debug diagnostics;
- no recommendation hides a condition that changes output completeness or
  correctness;
- public compatibility and schema-version consequences are explicit; and
- proposed tests protect meaningful behavior without asserting incidental
  prose or one permanently fixed global warning count.

## Completion Criteria

The audit is ready to become a feature roadmap when it can explain the current
high warning count quantitatively, identify the dominant producers and any
amplification defects, classify every warning family by consequence and
actionability, and recommend where each class should appear. The findings must
be specific enough that a later roadmap can define the target contract without
repeating the discovery work.