Plan warning and diagnostic reform
This commit is contained in:
399
docs/roadmap/audit-plan.md
Normal file
399
docs/roadmap/audit-plan.md
Normal file
@@ -0,0 +1,399 @@
|
||||
# Warning Signal And Presentation Audit Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the audit required before redesigning Notarius warning
|
||||
semantics and presentation. The audit must explain why an ordinary successful
|
||||
D&D run can produce dozens of warnings, distinguish actionable degradation
|
||||
from routine diagnostic observations, and recommend a bounded, deterministic
|
||||
warning contract that operators will actually review.
|
||||
|
||||
This is an audit plan, not a feature design or implementation plan. The audit
|
||||
must first establish the current producers, propagation behavior, empirical
|
||||
volume, public contracts, and compatibility constraints. The completed audit
|
||||
selects the target behavior from those findings; `implementation.md` owns the
|
||||
subsequent implementation sequence.
|
||||
|
||||
## Context
|
||||
|
||||
Notarius currently uses one `contracts.Warning` shape—optional `scope`, required
|
||||
`reason_code`, and required `message`—for findings from modules, validators,
|
||||
normalizers, reference preparation, retries, and framework policy. Accepted
|
||||
warnings are collected by the pipeline, counted by the CLI and machine-readable
|
||||
run receipt, and published as a flat array in `warnings.json`.
|
||||
|
||||
The architecture already guarantees deterministic public ordering and bounded
|
||||
warning production at several local boundaries. Recent validation work also
|
||||
gave incomplete validation explicit provenance. Those guarantees must be
|
||||
preserved. The observed problem is signal quality and aggregate volume: a
|
||||
successful run can report many warnings even when most describe routine,
|
||||
expected cleanup or weak heuristic observations rather than conditions that
|
||||
require operator action.
|
||||
|
||||
Initial repository inspection identifies several likely warning families:
|
||||
|
||||
- D&D source-relatedness validators can produce one warning per artifact
|
||||
record when a contextual name is not found in its cited text;
|
||||
- deterministic normalizers report canonicalization, whitespace cleanup,
|
||||
ordering changes, source-reference cleanup, and duplicate consolidation;
|
||||
- semantic registry reconciliation reports discarded proposals, exhausted
|
||||
fallback, and duplicate consolidation;
|
||||
- optional reference materialization can report unavailable or skipped
|
||||
reference inputs;
|
||||
- the producer-attempt state machine reports validators that exhausted their
|
||||
execution budgets under `warn_continue`; and
|
||||
- producer modules can return their own warnings, which are promoted only from
|
||||
the terminal accepted or rejected attempt.
|
||||
|
||||
These are audit hypotheses. The audit must verify exact behavior and frequency
|
||||
from current code and representative runs rather than assume that every family
|
||||
is noisy or incorrectly classified.
|
||||
|
||||
## Policy And Architecture Constraints
|
||||
|
||||
The audit and its recommendations must follow:
|
||||
|
||||
- `docs/policy/architecture.md` for module/framework ownership, deterministic
|
||||
ordering, validation semantics, durable provenance, and sensitive-data
|
||||
boundaries;
|
||||
- `docs/policy/documentation.md` for canonical documentation ownership and the
|
||||
distinction between current and future behavior;
|
||||
- `docs/policy/testing.md` for risk-based, behavior-oriented test
|
||||
recommendations; and
|
||||
- ADR-0014 for the distinction between semantic rejection, validator failure,
|
||||
correction guidance, and durable diagnostic provenance.
|
||||
|
||||
In particular:
|
||||
|
||||
- modules and validators may report findings, but the framework and output
|
||||
boundaries own aggregation and presentation;
|
||||
- reducing warning volume must not silently change artifact acceptance,
|
||||
validation policy, retry behavior, rejection behavior, or process exit
|
||||
status;
|
||||
- genuine validator execution failures and validation-incomplete outcomes must
|
||||
remain visible and auditable;
|
||||
- public ordering must remain deterministic despite concurrent lane execution;
|
||||
- aggregation must be bounded and must not leak raw model responses,
|
||||
correction guidance, credentials, private reference content, or unnecessary
|
||||
transcript text; and
|
||||
- detailed diagnostics may move to a more appropriate durable or debug surface,
|
||||
but must not be discarded when they are needed to understand a lossy or
|
||||
degraded result.
|
||||
|
||||
## Audit Objectives
|
||||
|
||||
The audit must answer five questions.
|
||||
|
||||
1. Which code paths produce warnings, under what conditions, and with what
|
||||
expected multiplicity?
|
||||
2. How are warnings cloned, bounded, replayed, ordered, promoted, persisted,
|
||||
counted, and presented from their producer through the final CLI and output
|
||||
bundle?
|
||||
3. Which warnings identify actionable degradation or data-quality risk, and
|
||||
which merely describe routine successful transformations or advisory
|
||||
heuristics?
|
||||
4. What aggregation and presentation model would materially reduce operator
|
||||
noise without hiding genuine failures, lossy fallback, or incomplete
|
||||
validation?
|
||||
5. Which proposed changes affect only presentation, and which would alter a
|
||||
public receipt, durable output contract, validation invariant, or other
|
||||
compatibility boundary?
|
||||
|
||||
## Scope
|
||||
|
||||
### Warning Producers
|
||||
|
||||
Inventory every production warning constructor and direct warning literal
|
||||
under `internal/`. Include warnings originating from:
|
||||
|
||||
- input, chunk, extract, merge, normalize, and output modules;
|
||||
- typed, chunk, and serialized validators;
|
||||
- semantic reconciliation and normalizer fallback;
|
||||
- external and generated reference materialization;
|
||||
- chunk-plan and checkpoint reuse or fallback;
|
||||
- producer-attempt exhaustion and validation-incomplete continuation;
|
||||
- pipeline orchestration and cancellation handling; and
|
||||
- CLI or publication logic, if it creates warnings rather than only presenting
|
||||
them.
|
||||
|
||||
Do not infer completeness from one textual search. Inspect shared constructors,
|
||||
returned result types, reason-code constants, registration paths, and tests that
|
||||
exercise warning behavior.
|
||||
|
||||
### Warning Propagation
|
||||
|
||||
Trace each warning family through:
|
||||
|
||||
- stage result and validation result contracts;
|
||||
- producer attempts, including abandoned attempts, semantic retries, structural
|
||||
retries, terminal rejection, and `warn_continue`;
|
||||
- concurrent lane collection and stable public ordering;
|
||||
- merge and normalize continuation;
|
||||
- chunk-plan caching and checkpoint recording or hydration;
|
||||
- ordered step output merging and generated-reference handoff;
|
||||
- `RunOutput`, run manifests, validation summaries, debug records, and CLI
|
||||
result construction;
|
||||
- `warnings.json`, `manifest.json`, the run-result receipt, human-readable
|
||||
standard error, and debug bundles.
|
||||
|
||||
For every boundary, determine whether warnings are copied, filtered,
|
||||
deduplicated, bounded, summarized, replayed from reusable state, or dropped.
|
||||
Pay particular attention to amplification across chunks, lanes, validators,
|
||||
retries, and resumed runs.
|
||||
|
||||
### D&D Warning Semantics
|
||||
|
||||
Review every implemented D&D artifact family. At minimum, distinguish:
|
||||
|
||||
- evidence/source-relatedness advisories;
|
||||
- deterministic canonicalization or whitespace changes;
|
||||
- order and source-reference normalization;
|
||||
- exact and semantic duplicate consolidation;
|
||||
- unresolved catalog or registry membership;
|
||||
- semantic-reconciliation proposal failure or fallback; and
|
||||
- validator execution incompleteness.
|
||||
|
||||
Determine whether the reason-code vocabulary is consistent across artifact
|
||||
families, whether scopes are sufficiently contextual, and whether equivalent
|
||||
conditions produce near-duplicate warnings with different codes or prose.
|
||||
|
||||
### Public And Operator Surfaces
|
||||
|
||||
Review the implemented contracts and documentation for:
|
||||
|
||||
- CLI success output and warning-count output;
|
||||
- `notarius.run-result.v1` and its `warning_count` and validation fields;
|
||||
- `warnings.json` and the published JSON index;
|
||||
- manifest validation summaries and rejection summaries;
|
||||
- debug summary and detailed debug artifacts; and
|
||||
- downstream subprocess guidance, especially the complete D&D consumer
|
||||
workflow.
|
||||
|
||||
Identify which surfaces are intended for immediate operator attention, durable
|
||||
machine consumption, forensic detail, or debugging. Record any places where
|
||||
the same flat count or warning list is being asked to serve incompatible
|
||||
audiences.
|
||||
|
||||
### Tests And Documentation
|
||||
|
||||
Inventory tests that protect warning production, bounds, ordering, retry
|
||||
promotion, checkpoint replay, JSON publication, receipt counts, and CLI stream
|
||||
behavior. Identify meaningful gaps, redundant exact-prose assertions, and tests
|
||||
that would unnecessarily obstruct a taxonomy or aggregation redesign.
|
||||
|
||||
Review `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
|
||||
`docs/integrations/json-output.md`, `docs/integrations/run-result.md`,
|
||||
`docs/consumers/`, and relevant internal documentation for current warning
|
||||
claims. Record the canonical document that would own each future contract
|
||||
change; do not rewrite those documents during the audit.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
The audit must not:
|
||||
|
||||
- implement warning filtering, severity levels, aggregation, or new CLI flags;
|
||||
- change validator decisions, default chains, retry budgets, or terminal
|
||||
validation policy;
|
||||
- suppress warnings merely to meet a numerical target;
|
||||
- redesign rejections, errors, logs, metrics, or debug bundles except where
|
||||
their boundary with warnings must be clarified;
|
||||
- add the planned D&D combat-scene semantic validator;
|
||||
- create deterministic tests that assert one exact global warning count for
|
||||
all future runs; or
|
||||
- use live paid LLM calls as part of the default automated test suite.
|
||||
|
||||
## Inventory Method
|
||||
|
||||
Create a warning inventory with one row per semantically distinct production
|
||||
condition. Each row should record:
|
||||
|
||||
| Field | Required analysis |
|
||||
| --- | --- |
|
||||
| Producer | Package, function, module or validator key, stage, and artifact family. |
|
||||
| Trigger | Exact condition that emits the warning and whether it follows successful mutation, heuristic doubt, fallback, or failure. |
|
||||
| Identity | Reason code, scope format, and whether either is stable enough for aggregation or machine use. |
|
||||
| Multiplicity | Maximum per record, chunk, lane, validator, attempt, step, and run. |
|
||||
| Lifecycle | Whether abandoned attempts are discarded, terminal warnings promoted, and cached or checkpointed warnings replayed. |
|
||||
| Consequence | Whether data changed, evidence is questionable, output is incomplete, fallback occurred, or no externally meaningful consequence exists. |
|
||||
| Actionability | What an operator can reasonably do in response, if anything. |
|
||||
| Surfaces | CLI, receipt count, `warnings.json`, manifest, rejection, checkpoint, or debug presence. |
|
||||
| Bounds | Existing local caps, message limits, omission records, and any missing aggregate bound. |
|
||||
| Sensitivity | Whether scope or message can contain source-derived or otherwise sensitive content. |
|
||||
| Coverage | Existing tests and the meaningful regression risk they protect. |
|
||||
|
||||
Treat different reason codes that represent the same operator condition as
|
||||
potential consolidation candidates, but do not merge them in the audit
|
||||
document without explaining lost diagnostic information.
|
||||
|
||||
## Propagation Analysis
|
||||
|
||||
Produce a compact propagation map from warning creation to each terminal
|
||||
surface. The analysis must explicitly verify:
|
||||
|
||||
- only the terminal candidate's warnings are promoted after retries;
|
||||
- whether rejected candidates retain warnings and where;
|
||||
- whether `warn_continue` creates one warning per exhausted validator and how
|
||||
that relates to validation summaries;
|
||||
- whether cached chunk plans or hydrated checkpoints replay historical warnings
|
||||
into the current run;
|
||||
- whether the same warning can be appended at more than one handoff boundary;
|
||||
- how concurrent completion is reordered before publication;
|
||||
- whether local per-module caps compose into an unbounded or excessively large
|
||||
run-level result; and
|
||||
- whether warning counts on stderr, receipts, manifests, and `warnings.json`
|
||||
refer to exactly the same collection.
|
||||
|
||||
Any suspected duplicate append or unstable ordering is a correctness finding,
|
||||
not merely a presentation concern.
|
||||
|
||||
## Empirical D&D Run Analysis
|
||||
|
||||
Static inspection must be supplemented with representative run evidence. Use
|
||||
at least:
|
||||
|
||||
- one ordinary successful complete D&D run known to produce high warning
|
||||
volume;
|
||||
- one smaller maintained example or synthetic run;
|
||||
- one run with semantic registry reconciliation activity;
|
||||
- one run with a validator execution failure allowed through
|
||||
`warn_continue`, using an offline test double where practical; and
|
||||
- one retrying producer case to verify abandoned-attempt warning treatment.
|
||||
|
||||
For real campaign runs, analyze only bounded metadata unless the operator
|
||||
explicitly provides source content for review. Record counts grouped by stage,
|
||||
lane, module or validator, reason code, and scope family. Also record unique
|
||||
reason-code count, repeated-message count, maximum group size, validation
|
||||
status, rejected-output count, and whether each warning led to a plausible
|
||||
operator action.
|
||||
|
||||
Compare the same logical run under ordinary execution and checkpoint resume
|
||||
when practical. The audit must distinguish warning volume caused by actual data
|
||||
conditions from volume caused by orchestration or replay.
|
||||
|
||||
Do not make a model-quality judgment solely from warning frequency. Manually
|
||||
inspect a bounded sample from each high-volume reason code to estimate false
|
||||
positive rate and operational value.
|
||||
|
||||
## Classification Rubric
|
||||
|
||||
Classify each warning condition along independent dimensions rather than force
|
||||
an immediate single severity enum:
|
||||
|
||||
- **result impact:** none, routine mutation, lossy mutation, uncertain data
|
||||
quality, fallback, or incomplete validation;
|
||||
- **operator action:** none, informational review, configuration or reference
|
||||
correction, source/model review, or rerun required;
|
||||
- **scope:** record, chunk, lane, step, pipeline, or infrastructure;
|
||||
- **persistence need:** top-level attention, durable detail, debug-only detail,
|
||||
or metric/trace candidate; and
|
||||
- **confidence:** deterministic fact, heuristic advisory, or execution failure.
|
||||
|
||||
The audit should then test whether a small durable taxonomy can represent the
|
||||
meaningful combinations. A promising starting hypothesis is that top-level
|
||||
operator warnings should be limited to actionable degradation, incomplete
|
||||
validation, lossy fallback, and material data-quality risk, while routine
|
||||
successful normalization observations remain available as lower-level durable
|
||||
diagnostics. The audit must validate or revise that hypothesis from evidence.
|
||||
|
||||
## Design Questions The Audit Must Resolve
|
||||
|
||||
The findings must give a recommendation, with at least one viable alternative
|
||||
and tradeoffs, for each of these questions:
|
||||
|
||||
1. Should warning severity or disposition become an explicit contract field,
|
||||
or should stable reason-code metadata drive presentation policy outside the
|
||||
warning payload?
|
||||
2. Should `warnings.json` remain the complete durable detail while the CLI and
|
||||
receipt expose an aggregated actionable summary, or should durable warnings
|
||||
themselves be separated from routine observations?
|
||||
3. Where should global deduplication and aggregation live so modules retain
|
||||
semantic ownership but concurrent pipeline results remain deterministic?
|
||||
4. What is the stable aggregation key: reason code, stage/lane/module identity,
|
||||
normalized scope, message template, or an explicit structured grouping key?
|
||||
5. How should bounded samples and omitted counts be represented without
|
||||
converting a summary record into another warning that inflates the count?
|
||||
6. Should `warning_count` continue to mean the length of `warnings.json`, or
|
||||
should a new receipt or schema field distinguish actionable warning groups
|
||||
from detailed observations?
|
||||
7. Which normalization changes are sufficiently lossy or surprising to remain
|
||||
warnings, and which are ordinary provenance that belongs in manifest or
|
||||
debug data?
|
||||
8. Should heuristic source-relatedness findings remain warnings, become
|
||||
grouped data-quality advisories, or be strengthened into configurable
|
||||
validation decisions only after demonstrated precision?
|
||||
9. How should warnings loaded from checkpoints be identified or aggregated
|
||||
relative to newly produced warnings?
|
||||
10. Does the chosen target alter durable or validation semantics enough to
|
||||
require a new ADR or a versioned run-result/output contract?
|
||||
|
||||
## Required Audit Deliverable
|
||||
|
||||
Write the completed findings to `docs/roadmap/audit.md`. It should contain:
|
||||
|
||||
1. an executive assessment of current warning quality and risk;
|
||||
2. the complete warning-producer inventory;
|
||||
3. the warning propagation and surface map;
|
||||
4. empirical measurements and bounded representative samples;
|
||||
5. findings ranked by operator impact, correctness risk, and implementation
|
||||
leverage;
|
||||
6. a recommended target taxonomy and presentation model;
|
||||
7. compatibility, documentation, ADR, and migration implications;
|
||||
8. implementation implications and dependencies sufficient to support a
|
||||
separate implementation plan; and
|
||||
9. open decisions only where repository evidence cannot support a responsible
|
||||
recommendation.
|
||||
|
||||
Each finding should identify the supporting code paths, tests, documentation,
|
||||
and run evidence. Separate observed facts from recommendations and avoid
|
||||
changelog or development-history framing.
|
||||
|
||||
## High-Level Audit Sequence
|
||||
|
||||
The detailed execution sequence should be written separately if needed. At a
|
||||
high level, perform the audit in this order:
|
||||
|
||||
1. **Static inventory:** enumerate warning producers, reason codes, scopes,
|
||||
bounds, and existing tests.
|
||||
2. **Propagation audit:** trace promotion, ordering, replay, persistence,
|
||||
counting, and presentation across the framework and CLI.
|
||||
3. **Empirical analysis:** measure representative D&D runs and inspect bounded
|
||||
samples from high-volume warning groups.
|
||||
4. **Classification:** apply the rubric, identify duplicate concepts and
|
||||
misplaced routine diagnostics, and evaluate public-contract options.
|
||||
5. **Synthesis:** rank findings and recommend a decision-complete target for a
|
||||
subsequent feature roadmap.
|
||||
|
||||
Static inventory and propagation may be performed as separate focused agent
|
||||
prompts. Empirical analysis should be isolated because it may require operator
|
||||
artifacts or opt-in provider execution. Classification and synthesis should
|
||||
consume the earlier written evidence rather than rediscover the repository.
|
||||
|
||||
## Validation Of The Audit
|
||||
|
||||
Before considering the audit complete, verify that:
|
||||
|
||||
- every production warning literal or constructor is represented in the
|
||||
inventory;
|
||||
- every reason code observed in representative `warnings.json` files maps to a
|
||||
known producer or is recorded as an unexplained finding;
|
||||
- counts agree across the runner result, CLI receipt, stderr summary, and
|
||||
published warning collection for each sampled run;
|
||||
- retry, rejection, incomplete-validation, cache, checkpoint, concurrency, and
|
||||
ordered-step paths are covered;
|
||||
- recommended aggregation preserves deterministic ordering and bounded memory;
|
||||
- recommendations distinguish warnings from errors, rejections, validation
|
||||
summaries, logs, and debug diagnostics;
|
||||
- no recommendation hides a condition that changes output completeness or
|
||||
correctness;
|
||||
- public compatibility and schema-version consequences are explicit; and
|
||||
- proposed tests protect meaningful behavior without asserting incidental
|
||||
prose or one permanently fixed global warning count.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The audit is ready to become a feature roadmap when it can explain the current
|
||||
high warning count quantitatively, identify the dominant producers and any
|
||||
amplification defects, classify every warning family by consequence and
|
||||
actionability, and recommend where each class should appear. The findings must
|
||||
be specific enough that a later roadmap can define the target contract without
|
||||
repeating the discovery work.
|
||||
Reference in New Issue
Block a user