743 lines
32 KiB
Markdown
743 lines
32 KiB
Markdown
# Warning Signal And Presentation Implementation Plan
|
||
|
||
## Purpose
|
||
|
||
This document is the ordered implementation plan for the warning and diagnostic
|
||
reform defined by [the completed audit](audit.md). It is intended to be executed
|
||
stage by stage by a `gpt-5.6-terra` coding agent. Each numbered stage is one
|
||
implementation prompt and must leave the repository compiling, internally
|
||
coherent, and covered at the narrowest durable test boundaries relevant to that
|
||
stage.
|
||
|
||
The audit is the canonical source for findings, evidence, and target rationale.
|
||
This document is the canonical source for implementation order and task
|
||
breakdown.
|
||
|
||
## Governing Decisions
|
||
|
||
The following decisions are final for this work set.
|
||
|
||
1. **Warnings are process signals.** A warning means that the run completed
|
||
under policy despite process-level degradation or incomplete configured
|
||
work. Examples are an empty configured reference, an applicable validator
|
||
that could not complete under `warn_continue`, an unavailable required
|
||
upstream classification, or exhausted semantic-reconciliation fallback.
|
||
2. **Extraction-quality signals are not warnings.** LLM-judged uncertainty,
|
||
lexical source-relatedness findings, unresolved entity grounding, and other
|
||
accepted-artifact quality signals are advisories. They must never be
|
||
promoted to warnings merely because a model or heuristic expressed doubt.
|
||
3. **Routine successful transformations are observations.** Canonicalization,
|
||
sorting, source-reference cleanup, ID repair, and accepted duplicate
|
||
consolidation remain inspectable but do not require operator action.
|
||
4. **Ordinary successful runs have zero warnings.** A successful-but-degraded
|
||
run may have warnings when policy permits continuation. Advisory or
|
||
observation volume alone must not produce warning stderr or a nonzero
|
||
warning count.
|
||
5. **Errors and rejections remain separate.** This work must not change
|
||
validation approval, rejection, retry budgets, exit status, or error policy.
|
||
A corrected superseded attempt leaves no final warning. An exhausted
|
||
rejection remains a rejection or run failure according to existing policy.
|
||
6. **Modules own meaning; the framework owns context and presentation.** A
|
||
producer chooses disposition, category, reason code, scope, and safe message.
|
||
The framework attaches stage and pipeline origin, aggregates deterministically,
|
||
enforces bounds, and supplies the final collections to output and CLI code.
|
||
7. **Durable contracts are versioned.** The incompatible grouped warning file
|
||
is `notarius.warnings.v2`, the new diagnostic file is
|
||
`notarius.diagnostics.v1`, and the machine-readable run receipt becomes
|
||
`notarius.run-result.v2`. Do not silently redefine the v1 receipt or warning
|
||
payload.
|
||
8. **There is no first-release configuration surface for presentation policy.**
|
||
Classification, sample bounds, and aggregation are fixed application
|
||
policy. Do not add suppression, escalation, verbosity, or per-reason config
|
||
in this work set.
|
||
|
||
## Target Contract
|
||
|
||
Use the following model unless existing Go naming requires a narrowly scoped
|
||
variation. Any naming variation must preserve the specified fields and
|
||
semantics.
|
||
|
||
### Producer Diagnostic
|
||
|
||
Add a framework contract representing a locally grouped producer diagnostic:
|
||
|
||
- `disposition`: `warning`, `advisory`, or `observation`;
|
||
- `category`: one of `configuration`, `degradation`,
|
||
`validation_incomplete`, `fallback`, `data_quality`, or `normalization`;
|
||
- `reason_code`: stable, nonblank producer-owned identity;
|
||
- `occurrence_count`: exact positive number of represented occurrences;
|
||
- `samples`: deterministic bounded samples containing safe `scope` and
|
||
`message`; and
|
||
- `omitted_sample_count`: exactly `occurrence_count - len(samples)`.
|
||
|
||
Validate these combinations:
|
||
|
||
| Disposition | Allowed categories |
|
||
| --- | --- |
|
||
| `warning` | `configuration`, `degradation`, `validation_incomplete`, `fallback` |
|
||
| `advisory` | `data_quality` |
|
||
| `observation` | `normalization` |
|
||
|
||
This strict matrix is intentional. It makes the process-only warning rule a
|
||
contract invariant rather than a convention inferred from reason codes.
|
||
|
||
Use these fixed limits:
|
||
|
||
- reason code: 128 UTF-8 bytes;
|
||
- scope: 512 UTF-8 bytes;
|
||
- sample message: 4 KiB of valid UTF-8;
|
||
- retained distinct samples per group: 3; and
|
||
- groups returned by one producer or validator result: 64.
|
||
|
||
Blank or invalid required fields, invalid disposition/category combinations,
|
||
invalid UTF-8, inconsistent counts, excessive samples, or excessive local
|
||
groups make the producer result invalid. They must return a bounded contextual
|
||
error rather than be silently repaired. The three-sample and 64-group bounds
|
||
are public operational safeguards and may be asserted directly in their owning
|
||
contract tests.
|
||
|
||
The shared collector must count every occurrence and retain the first three
|
||
distinct samples in producer order. Repeated identical samples still increase
|
||
`occurrence_count`. The omission count is numeric metadata, never another
|
||
diagnostic record.
|
||
|
||
### Framework Origin And Aggregation
|
||
|
||
The framework adds this origin before final aggregation:
|
||
|
||
- stage: `references`, `chunk`, `extract`, `merge`, or `normalize`;
|
||
- step ID, when applicable;
|
||
- lane ID, when applicable;
|
||
- module key, when applicable; and
|
||
- validator key, for validator-produced diagnostics.
|
||
|
||
Chunk ID and zero-based chunk index belong on samples, not group origin,
|
||
because otherwise identical per-chunk findings cannot aggregate. Use an
|
||
optional integer representation that preserves chunk index zero.
|
||
|
||
The stable aggregation key is:
|
||
|
||
```text
|
||
disposition + category + reason_code
|
||
+ stage + step_id + lane_id + module_key + validator_key
|
||
```
|
||
|
||
Scope, message, chunk ID, and chunk index are excluded from the key. Groups and
|
||
samples retain first-occurrence order from the runner's existing canonical
|
||
ordering; completion timing must never affect the result. When groups merge,
|
||
sum exact occurrence counts and retain the first three distinct samples.
|
||
|
||
Use separate global bounds:
|
||
|
||
- at most 128 actionable warning groups; exceeding this limit is a framework
|
||
error because Notarius must not hide process degradation; and
|
||
- at most 256 advisory/observation groups. Additional non-warning groups are
|
||
omitted from representation while their occurrences contribute to an exact
|
||
`unrepresented_occurrence_count` and set `truncated: true`.
|
||
|
||
The non-warning envelope's group count means represented groups. Its occurrence
|
||
count includes represented and unrepresented occurrences. The warning
|
||
collection is never truncated, so both warning counts are exact.
|
||
|
||
### Classification Matrix
|
||
|
||
Migrate existing producers according to this matrix.
|
||
|
||
| Disposition and category | Existing conditions |
|
||
| --- | --- |
|
||
| Warning / `configuration` | `empty_reference` |
|
||
| Warning / `validation_incomplete` | `validator_execution_incomplete`, covering both exhausted failure and skip when `warn_continue` advances the candidate |
|
||
| Warning / `degradation` | `scene_classification_unavailable` from combat-turn and enemy-event extraction gates |
|
||
| Warning / `fallback` | `npc_semantic_reconciliation_exhausted`, `item_semantic_reconciliation_exhausted`, `location_semantic_reconciliation_exhausted` |
|
||
| Advisory / `data_quality` | All ten source-relatedness families; `spell_name_unresolved`; `item_occurrence_unknown_item_id`; `location_occurrence_unknown_location_id`; guarded invalid semantic-consolidation proposals |
|
||
| Observation / `normalization` | Field and whitespace normalization; durable ID recomputation; source-reference sorting/deduplication; canonical record ordering; exact or approved semantic duplicate consolidation |
|
||
|
||
Rename `item_occurrence_source_unrelated` to
|
||
`item_occurrence_not_near_source` when it moves into the new contract. Keep
|
||
other existing reason codes unless this plan explicitly changes them. Split the
|
||
item registry's internal retry reason from its accepted advisory: use
|
||
`item_semantic_retry_proposal_invalid` for the retry directive and retain
|
||
`item_semantic_proposal_invalid` only for the accepted data-quality advisory.
|
||
|
||
Local `*_warnings_omitted` reason codes disappear. Omission is represented by
|
||
group counts. Rejections, structural failures, provider failures, cancellation,
|
||
and successful corrected retries do not receive a diagnostic solely to mirror
|
||
their existing error, rejection, validation, or attempt-debug records.
|
||
|
||
### Durable And CLI Contracts
|
||
|
||
Always publish both companion files, including for empty collections:
|
||
|
||
- `warnings.json` with schema version `notarius.warnings.v2`, actionable groups,
|
||
exact `group_count`, and exact `occurrence_count`;
|
||
- `diagnostics.json` with schema version `notarius.diagnostics.v1`, advisory and
|
||
observation groups, represented `group_count`, total `occurrence_count`,
|
||
`truncated`, and `unrepresented_occurrence_count`; and
|
||
- `index.json` with both `warnings_file` and `diagnostics_file`.
|
||
|
||
The run receipt becomes `notarius.run-result.v2` and replaces ambiguous
|
||
`warning_count` with:
|
||
|
||
- `warning_group_count`;
|
||
- `warning_occurrence_count`;
|
||
- `diagnostic_group_count`;
|
||
- `diagnostic_occurrence_count`; and
|
||
- `diagnostics_truncated`.
|
||
|
||
The human CLI writes a warning summary to stderr only when
|
||
`warning_group_count > 0`. It reports group and occurrence counts plus the
|
||
durable warning path. Advisory and observation counts do not create a warning
|
||
line. Preserve the existing successful stdout summary and process exit rules.
|
||
|
||
Remove `OutputResult.Warnings`. An output encoder either returns its complete
|
||
logical files or returns an error. It cannot discover a warning after the
|
||
warning file has already been serialized.
|
||
|
||
## Stage 1 ✅ — Record The Decision And Add Core Diagnostic Primitives
|
||
|
||
### Goal
|
||
|
||
Create the durable decision record and the generic, independently tested
|
||
diagnostic model without changing current runtime behavior.
|
||
|
||
### Work
|
||
|
||
- Add `docs/adr/0015-separate-process-warnings-from-quality-diagnostics.md`
|
||
using the repository ADR format. Record the governing decisions, origin and
|
||
aggregation ownership, bounded samples, fresh/resume equivalence, versioned
|
||
public contracts, and removal of output-result warnings. Mark it accepted;
|
||
do not claim implementation is complete.
|
||
- Add the producer diagnostic, sample, disposition, category, origin, group,
|
||
and collection types under `internal/framework/contracts/`.
|
||
- Add contract validation with the exact allowed combinations and limits in
|
||
this plan.
|
||
- Add a small generic collector under `internal/framework/diagnostics/` that
|
||
groups producer-local occurrences by disposition, category, and reason code,
|
||
preserves exact counts, and retains three distinct samples.
|
||
- Keep `contracts.Warning` and existing result fields temporarily. Do not alter
|
||
current warning output in this stage.
|
||
|
||
### Tests
|
||
|
||
Use table-driven contract tests for valid classifications, invalid UTF-8,
|
||
blank/oversized fields, invalid counts, the disposition/category matrix, sample
|
||
bounds, exact repeated-occurrence counting, and deterministic distinct-sample
|
||
selection. Test behavior through exported package contracts rather than private
|
||
collector structure.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Existing application behavior and public JSON remain unchanged.
|
||
- The new primitives cannot represent an LLM-quality warning because
|
||
`warning` plus `data_quality` is rejected.
|
||
- `go test ./internal/framework/contracts ./internal/framework/diagnostics`
|
||
passes.
|
||
|
||
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
|
||
|
||
## Stage 2 ✅ — Add Framework Diagnostic Transport And Transitional Projection
|
||
|
||
### Goal
|
||
|
||
Carry structured diagnostics through every framework result boundary and add
|
||
origin-aware aggregation while preserving the current public warning contract
|
||
temporarily.
|
||
|
||
### Work
|
||
|
||
- Add structured diagnostic fields alongside legacy warning fields in chunk
|
||
plan, typed extract, typed merge, typed normalize, normalize-retry fallback,
|
||
validation, run input/output, and output request contracts.
|
||
- Extend erased typed results, producer-attempt state, lane result collection,
|
||
ordered-step handoff, and run finalization to transport diagnostics.
|
||
- Preserve validator identity in validation reports instead of flattening new
|
||
diagnostics through a warning-only helper.
|
||
- Attach current stage, step, lane, module, validator, and sample-level chunk
|
||
context at promotion time.
|
||
- Add a run-level aggregator implementing the stable key, canonical
|
||
first-occurrence ordering, count merging, and bounds from this plan.
|
||
- Preserve terminal-attempt semantics: diagnostics from superseded attempts
|
||
remain debug-only; only terminal accepted or terminal rejected candidate
|
||
diagnostics are promoted.
|
||
- Add a private transitional projection from each structured diagnostic sample
|
||
to the existing legacy `Warning` collection so public surfaces remain
|
||
unchanged until Stage 8. Mark it explicitly temporary and ensure it does not
|
||
duplicate diagnostics already supplied through the legacy path.
|
||
|
||
### Tests
|
||
|
||
Extend focused framework tests for terminal attempt promotion, terminal
|
||
rejection, reversed concurrent completion, stage/lane/chunk origin, stable
|
||
grouping across chunks, and global bounds. Use controlled test dispositions and
|
||
origins; do not assert incidental diagnostic prose.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- A producer may return either legacy warnings or new structured diagnostics
|
||
during migration, but not produce duplicate public records through both.
|
||
- Structured groups are deterministic under concurrent completion.
|
||
- Superseded attempt diagnostics never reach final groups.
|
||
- Existing public warning tests still pass through the transitional projection.
|
||
- Focused framework tests and `go test ./internal/framework/pipeline` pass.
|
||
|
||
This stage is appropriately sized for one high-reasoning
|
||
`gpt-5.6-terra` prompt. Do not combine it with D&D migration.
|
||
|
||
## Stage 3 ✅ — Migrate Framework Process Signals And Close The Output Boundary
|
||
|
||
### Goal
|
||
|
||
Move framework-owned warning conditions into the new process-only contract and
|
||
remove the output-encoder consistency defect.
|
||
|
||
### Work
|
||
|
||
- Convert `empty_reference` to warning/configuration with reference origin and
|
||
bounded, non-sensitive sample content.
|
||
- Convert incomplete validation to warning/validation-incomplete whenever
|
||
`warn_continue` advances a candidate after an applicable validator either
|
||
exhausts with failure or skips. Retain validator key and typed failure/skip
|
||
detail in validation summaries; do not copy provider errors or arbitrary
|
||
validator prose into diagnostic samples.
|
||
- Aggregate all incomplete-validator occurrences rather than generating
|
||
omission warning records.
|
||
- Validate module and validator diagnostic results before promotion. An invalid
|
||
diagnostic contract is a bounded contextual framework error.
|
||
- Remove `Warnings` from `contracts.OutputResult`, the output adapter, test
|
||
doubles, and runner append logic. Keep `OutputRequest` diagnostic input; an
|
||
encoder failure remains an error.
|
||
- Add a regression test proving no post-encoding result can make receipt/debug
|
||
warning state disagree with the already encoded logical files.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Failed and skipped validators are both visible when incomplete validation is
|
||
allowed to continue.
|
||
- A fail-run validation policy still fails rather than converting the failure
|
||
into a warning.
|
||
- Output encoders have no successful post-encoding warning capability.
|
||
- Existing validation decisions, retry budgets, and exit behavior are
|
||
unchanged.
|
||
- `go test ./internal/framework/... ./internal/modules/generic/output/json`
|
||
passes.
|
||
|
||
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
|
||
|
||
## Stage 4 ✅ — Version Diagnostic-Bearing Cache And Checkpoint State
|
||
|
||
### Goal
|
||
|
||
Persist and replay structured producer diagnostics safely without allowing old
|
||
warning-only state to be interpreted as complete new state.
|
||
|
||
### Work
|
||
|
||
- Store producer-local structured diagnostics in chunk-plan and extract,
|
||
merge, and normalize checkpoint envelopes. Do not persist current pipeline
|
||
origin in source-keyed chunk-plan state.
|
||
- On reuse, attach the current run's origin at the same logical promotion point
|
||
used by fresh execution, then aggregate once.
|
||
- Bump the checkpoint workspace schema from v3 to
|
||
`notarius.workspace.v4` and the chunk-plan schema from v2 to
|
||
`notarius.chunk-plan.v3`.
|
||
- Treat earlier workspace and chunk-plan versions as incompatible cache misses,
|
||
never as corrupt fatal state and never as reusable diagnostic-complete state.
|
||
- Preserve the rule that validation-incomplete results and their dependents are
|
||
not reusable.
|
||
|
||
### Tests
|
||
|
||
Add or adapt behavioral tests for old-version invalidation, structured
|
||
diagnostic round trips, fresh/resume equality of groups and counts, exact-once
|
||
replay, and current-validator diagnostics on a reused chunk plan.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Fresh and resumed logical runs produce deeply equal diagnostic groups and
|
||
counts.
|
||
- Reuse provenance remains in checkpoint events and does not alter diagnostic
|
||
identity.
|
||
- Old cache data is safely bypassed.
|
||
- `go test ./internal/framework/checkpoint ./internal/framework/pipeline`
|
||
passes.
|
||
|
||
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
|
||
|
||
## Stage 5 ✅ — Migrate All D&D Source-Relatedness Validators
|
||
|
||
### Goal
|
||
|
||
Move all ten heuristic relatedness families from warnings to bounded
|
||
data-quality advisories.
|
||
|
||
### Work
|
||
|
||
- Migrate combat turns, enemy events, item occurrences, item registry,
|
||
location occurrences, location registry, NPC occurrences, NPC registry,
|
||
scene descriptions, and spells source-relatedness validators.
|
||
- Use the shared generic collector so every family returns exact occurrence
|
||
counts and at most three distinct samples. Remove each relatedness omission
|
||
reason code.
|
||
- Bring NPC and spell validators under the same contract bounds as every other
|
||
family.
|
||
- Rename `item_occurrence_source_unrelated` to
|
||
`item_occurrence_not_near_source`.
|
||
- Preserve current approval behavior, validator chain registration and order,
|
||
lexical algorithms, scope selection, and messages except for changes needed
|
||
to satisfy generic safety bounds.
|
||
|
||
### Tests
|
||
|
||
Consolidate repetitive limiter tests where a shared collector contract already
|
||
owns the behavior. Retain family tests for realistic triggering and
|
||
non-triggering cases. Prove that a relatedness finding is advisory/data-quality
|
||
and cannot increment process warning groups.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- All ten relatedness validators use the same structured advisory convention.
|
||
- No relatedness validator returns a legacy warning or an omission record.
|
||
- High-cardinality NPC and spell results remain bounded while reporting exact
|
||
occurrence counts.
|
||
- `go test ./internal/modules/dnd/validate/...` passes.
|
||
|
||
This repetitive but cohesive migration is appropriately sized for one
|
||
`gpt-5.6-terra` prompt. Do not combine it with normalizer migration.
|
||
|
||
## Stage 6 — Migrate D&D Registry Normalizers And Semantic Reconciliation ✅
|
||
|
||
### Goal
|
||
|
||
Apply the new taxonomy to NPC, item, and location registry normalization and
|
||
semantic-reconciliation outcomes.
|
||
|
||
### Work
|
||
|
||
- Migrate deterministic field cleanup, ID recomputation, source-reference
|
||
normalization, ordering, and accepted duplicate consolidation to
|
||
observation/normalization groups.
|
||
- Migrate exhausted NPC, item, and location semantic reconciliation to
|
||
warning/fallback groups.
|
||
- Migrate guarded invalid item semantic proposals to advisory/data-quality.
|
||
- Split the item retry directive reason to
|
||
`item_semantic_retry_proposal_invalid`; keep correction control data separate
|
||
from the accepted advisory reason and from model-facing text.
|
||
- Replace local warning limiters and omission records with the shared
|
||
diagnostic collector while preserving artifact values, safe fallback, and
|
||
retry budgets.
|
||
|
||
### Tests
|
||
|
||
Protect exact artifact outcomes, classification, retry exhaustion, safe
|
||
currency behavior, occurrence counts, and bounded samples. Do not preserve
|
||
old warning slice lengths or omission prose.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Successful registry cleanup and consolidation produce observations only.
|
||
- A guarded model proposal produces an advisory, not a warning.
|
||
- Exhausted semantic reconciliation is the only registry-normalizer process
|
||
warning family.
|
||
- Semantic retry behavior and artifacts are unchanged.
|
||
- Registry normalizer and semantic-reconciliation tests pass.
|
||
|
||
This stage is appropriately sized for one high-reasoning
|
||
`gpt-5.6-terra` prompt.
|
||
|
||
## Stage 7 — Migrate Remaining D&D Producers ✅
|
||
|
||
### Goal
|
||
|
||
Complete D&D diagnostic classification across spells, occurrences, combat,
|
||
enemy events, scenes, and extraction gates.
|
||
|
||
### Work
|
||
|
||
- Migrate spell, combat-turn, enemy-event, item-occurrence,
|
||
location-occurrence, NPC-occurrence, and scene-description normalizers.
|
||
- Classify deterministic cleanup, ordering, source-reference normalization,
|
||
and duplicate consolidation as observation/normalization.
|
||
- Classify unresolved spell, item, or location membership as
|
||
advisory/data-quality, never warning.
|
||
- Convert combat-turn and enemy-event `scene_classification_unavailable`
|
||
extraction gates to warning/degradation with module and chunk provenance.
|
||
- Remove all remaining D&D local warning omission reason codes and all uses of
|
||
`LimitWarnings`; retain text-safety helpers that still have value.
|
||
|
||
### Tests
|
||
|
||
Adapt family tests to assert artifact invariants and classification. Add one
|
||
assembled D&D test demonstrating that quality advisories and normalization
|
||
observations can be present while the warning group count remains zero.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- No D&D producer returns a legacy `contracts.Warning`.
|
||
- No accepted-artifact uncertainty or unresolved grounding appears as a
|
||
process warning.
|
||
- Missing required scene classification remains an actionable process warning.
|
||
- All D&D package tests pass.
|
||
|
||
This stage is appropriately sized for one high-reasoning
|
||
`gpt-5.6-terra` prompt.
|
||
|
||
## Stage 8 ✅ — Remove The Legacy Warning Path And Finalize Aggregation
|
||
|
||
### Goal
|
||
|
||
Make the structured diagnostic collection the sole in-memory signal path.
|
||
|
||
### Work
|
||
|
||
- Migrate any remaining generic test modules and framework fixtures from
|
||
`contracts.Warning` to structured diagnostics.
|
||
- Remove `contracts.Warning`, legacy `Warnings` fields, clone helpers, warning
|
||
append helpers, transitional projections, and obsolete D&D warning limiters.
|
||
- Make `RunOutput` and `OutputRequest` carry the finalized warning and
|
||
non-warning grouped collections derived from one aggregator result.
|
||
- Enforce the 128-warning-group error bound and 256-non-warning-group truncation
|
||
behavior. Ensure occurrence counts remain exact and truncation metadata is
|
||
deterministic.
|
||
- Verify all producers are validated at their stable framework boundary.
|
||
|
||
### Tests
|
||
|
||
Add focused behavioral coverage for warning overflow failure, non-warning
|
||
truncation, exact occurrence totals, distinct sample retention, stable group
|
||
order, and no legacy double-counting. Remove tests whose only purpose was to
|
||
assert old flat-list caps or omission prose.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Repository search finds no production `contracts.Warning`, legacy warning
|
||
result field, or `*_warnings_omitted` reason code.
|
||
- One finalized collection supplies all later surfaces.
|
||
- An advisory-only successful run has zero warning groups.
|
||
- `go test ./internal/framework/... ./internal/modules/dnd/...` passes.
|
||
|
||
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
|
||
|
||
## Stage 9 ✅ — Publish Versioned Warning And Diagnostic Files
|
||
|
||
### Goal
|
||
|
||
Replace the flat durable warning payload with the two target grouped contracts.
|
||
|
||
### Work
|
||
|
||
- Update the production JSON encoder to always emit grouped `warnings.json`
|
||
(`notarius.warnings.v2`) and `diagnostics.json`
|
||
(`notarius.diagnostics.v1`).
|
||
- Include exact counts and diagnostic truncation metadata specified above.
|
||
- Add `diagnostics_file` to `index.json`; retain `warnings_file`.
|
||
- Ensure warning groups appear only in `warnings.json` and advisory/observation
|
||
groups appear only in `diagnostics.json`.
|
||
- Update `docs/integrations/json-output.md` in the same stage. It owns file
|
||
names, envelope schemas, count meanings, group/sample fields, bounds,
|
||
truncation, and index discovery. Link rather than duplicate CLI behavior.
|
||
- Update maintained output examples or test fixtures only where they encode
|
||
implemented output contracts.
|
||
|
||
### Tests
|
||
|
||
Use encoder contract tests for empty and populated files, schema versions,
|
||
partitioning, counts, truncation, index paths, deterministic JSON ordering, and
|
||
newline/valid-JSON conventions. Update assembled pipeline tests to compare both
|
||
durable files with the finalized in-memory collections.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Every production JSON bundle contains both companion files and index paths.
|
||
- No diagnostic appears in both files.
|
||
- Published counts agree with the in-memory collection.
|
||
- Output encoder and maintained example contract tests pass.
|
||
|
||
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
|
||
|
||
## Stage 10 — Introduce Run Result V2 And Quiet CLI Presentation
|
||
|
||
### Goal
|
||
|
||
Give subprocess consumers unambiguous counts and make an ordinary successful
|
||
run quiet on the warning stream.
|
||
|
||
### Work
|
||
|
||
- Change the emitted receipt to `notarius.run-result.v2` with the five fields
|
||
specified in the target contract. Remove v1 `warning_count`; do not emit two
|
||
competing count models.
|
||
- Derive receipt counts from the same finalized collection used by the JSON
|
||
encoder.
|
||
- Print a stderr warning summary only when actionable warning groups exist. It
|
||
must include group count, occurrence count, and the output-relative or
|
||
absolute details path consistent with current CLI path conventions.
|
||
- Do not mention advisory/observation counts as warnings. Preserve ordinary
|
||
stdout completion output and all exit classifications.
|
||
- Update `docs/cli.md`, `docs/integrations/run-result.md`,
|
||
`docs/consumers/subprocess.md`, and `docs/consumers/dnd-pipeline.md` in the
|
||
same stage. Each document must keep to its canonical scope and link to the
|
||
grouped JSON contract rather than duplicating schemas.
|
||
|
||
### Tests
|
||
|
||
Cover zero-warning approved success, advisory-only success, degraded success
|
||
with warnings, rejected output, JSON receipt delivery, and non-production
|
||
output modules. Assert semantic fields and stream choice, not complete prose.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Advisory-only and observation-only successful runs write no warning stderr.
|
||
- Degraded success writes one concise actionable warning summary.
|
||
- Receipt counts and durable files agree.
|
||
- Receipt, CLI command, and consumer contract tests pass.
|
||
|
||
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
|
||
|
||
## Stage 11 — Align Debug, Manifest, And Resume Surfaces
|
||
|
||
### Goal
|
||
|
||
Ensure forensic and provenance surfaces reflect the new model without creating
|
||
another competing warning contract.
|
||
|
||
### Work
|
||
|
||
- Replace final flat warning debug summaries with final grouped warning and
|
||
non-warning diagnostic projections, using separate clearly named files or
|
||
one versioned diagnostic summary envelope consistently with existing debug
|
||
layout.
|
||
- Keep attempt-local candidate diagnostics in detailed debug records with
|
||
stage/attempt provenance. Preserve redaction and the rule that raw model
|
||
responses and correction messages appear only in detailed debug.
|
||
- Keep manifest validation and rejection summaries authoritative for validation
|
||
status; do not duplicate full diagnostic groups into the manifest.
|
||
- Ensure checkpoint events, not diagnostic origin, identify reuse.
|
||
- Update `docs/operations.md`, `docs/internal/state.md`, and the debug portions
|
||
of `docs/internal/pipeline.md` in the same stage.
|
||
|
||
### Tests
|
||
|
||
Cover successful debug capture, partial failure debug capture,
|
||
terminal-attempt-only final grouping, redaction, and fresh/resume equality.
|
||
Retain existing validation summary tests rather than duplicating every outcome
|
||
in diagnostic tests.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Debug consumers can distinguish process warnings, quality advisories, and
|
||
observations.
|
||
- Debug final counts match receipt and durable output when output is published.
|
||
- Attempt detail remains bounded and sensitive-data rules are preserved.
|
||
- Debug, state, and resume-focused tests pass.
|
||
|
||
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
|
||
|
||
## Stage 12 — Update Architecture And Internal Producer Documentation
|
||
|
||
### Goal
|
||
|
||
Make the implemented ownership and classification rules durable for future
|
||
modules without duplicating public schemas.
|
||
|
||
### Work
|
||
|
||
- Update `docs/policy/architecture.md` to state the process-only warning
|
||
invariant, module semantic ownership, framework origin/aggregation ownership,
|
||
and ordinary-success-zero-warnings expectation. Link to ADR-0015 for
|
||
rationale.
|
||
- Update `docs/internal/pipeline.md` for terminal promotion, origin enrichment,
|
||
aggregation, bounds, concurrency ordering, checkpoint replay, and output
|
||
handoff.
|
||
- Update `docs/internal/modules.md` with the generic producer contract and the
|
||
prohibition on using warnings for extraction quality.
|
||
- Update `docs/internal/dnd.md` with the implemented classification matrix and
|
||
shared collector convention. Link to integration contracts for durable
|
||
schemas rather than restating them.
|
||
- Update `docs/internal/overview.md` only if the new generic diagnostics package
|
||
changes the component inventory.
|
||
- Remove stale terminology from current docs outside `docs/roadmap/`, but do
|
||
not write release history or document unimplemented configuration.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- Each volatile fact has one canonical owner under the documentation policy.
|
||
- Future module authors can determine which disposition/category to use and
|
||
where origin and aggregation are attached.
|
||
- Current documentation contains no claim that heuristic quality doubt is a
|
||
warning.
|
||
- Relative Markdown links resolve.
|
||
|
||
This stage is appropriately sized for one `gpt-5.6-terra` prompt.
|
||
|
||
## Stage 13 — Final Behavioral Verification And Cleanup
|
||
|
||
### Goal
|
||
|
||
Verify the complete migration, remove transitional residue, and establish that
|
||
the audit findings are fully addressed.
|
||
|
||
### Work
|
||
|
||
- Run repository searches for legacy warning types, warning omission reason
|
||
codes, old run-result v1 fields, old flat warning JSON assumptions, and
|
||
`OutputResult.Warnings`.
|
||
- Review every producer inventoried in `audit.md` against the final
|
||
classification matrix.
|
||
- Run the maintained minimal and complete D&D workflows with offline fake LLM
|
||
clients. Do not require provider credentials or add live calls to the default
|
||
suite.
|
||
- Verify these relationships end to end:
|
||
- ordinary approved success and advisory-only success have zero warnings;
|
||
- successful process degradation has a warning;
|
||
- corrected retries retain only terminal diagnostics;
|
||
- rejected and failed runs retain their established semantics;
|
||
- warning and diagnostic group order is deterministic under concurrency;
|
||
- fresh and resumed runs are equivalent;
|
||
- global group and sample bounds hold; and
|
||
- receipt, stderr, durable files, index, and debug counts agree.
|
||
- Remove obsolete helpers and redundant tests made unnecessary by stronger
|
||
package-level collector or end-to-end contract tests.
|
||
|
||
### Validation Commands
|
||
|
||
Run at minimum:
|
||
|
||
```sh
|
||
go test ./...
|
||
go vet ./...
|
||
go build ./cmd/notarius
|
||
```
|
||
|
||
Also run any repository documentation or link checker discovered during the
|
||
stage. If none exists, perform a focused relative-link review for the documents
|
||
changed by Stages 9–12.
|
||
|
||
### Acceptance Criteria
|
||
|
||
- All three repository-wide Go commands pass offline.
|
||
- No live API key is required.
|
||
- No legacy flat-warning code path or public v1 warning-count claim remains in
|
||
current-behavior documentation.
|
||
- All seven audit findings are addressed without changing artifact values,
|
||
validator chain order, retry budgets, rejection policy, or exit semantics.
|
||
- The ordinary maintained successful workflow emits zero actionable warnings;
|
||
non-warning diagnostics remain available in `diagnostics.json`.
|
||
|
||
This final verification is appropriately sized for one `gpt-5.6-terra`
|
||
prompt.
|
||
|
||
## Deferred Work
|
||
|
||
The following are deliberately outside this implementation plan:
|
||
|
||
- provider-backed measurement of production advisory precision or frequency;
|
||
- configurable warning suppression, escalation, verbosity, or reason filters;
|
||
- converting heuristic relatedness checks into rejection rules;
|
||
- the planned LLM-backed D&D combat-scene validator;
|
||
- metrics or telemetry export beyond the specified durable and debug files;
|
||
- a two-phase output encoder protocol; and
|
||
- release preparation or release-note creation.
|
||
|
||
Provider-backed production data may inform later advisory tuning, but it is not
|
||
required to implement or validate this architecture.
|