# Pre-1.0 Deduplication Implementation Plan

This plan turns `docs/roadmap/audit.md` into staged, prompt-sized cleanup work for an LLM coding agent. Each stage should be implemented in order and kept small enough to review as an independent commit.

## Operating rules

- Read `docs/roadmap/audit.md` before starting any stage.
- Preserve public CLI, report, diagnostics, config precedence, prompt metadata, and output-schema behavior unless a stage explicitly calls out an intended behavior change.
- Keep the four production module packages separate: `glossary`, `homophones`, `spoken_word`, and `grammar`.
- Do not introduce plugin systems, generic workflow engines, broad CLI framework rewrites, reflection-heavy config mappers, or merged module packages.
- Prefer narrow helpers, catalogs, constants, and pure mapping functions over broad abstractions.
- Run the targeted tests listed in each stage before moving to the next stage.
- Run `go test ./...` before declaring the full sequence complete.
- Ignore unrelated worktree changes, including the existing deletion of `docs/roadmap/publish.md`, unless the user explicitly asks to handle them.
- Do not reduce parity, release-fixture, subprocess, or module-specific behavior coverage while consolidating helpers.

## Stages

### Stage 1: Diagnostics artifact constants and metadata paths

Goal:

- Centralize diagnostics artifact names and report diagnostics metadata path construction without changing any filenames or report fields.

Key edits:

- Define constants in `internal/core/diagnostics` for:
  - `source-transcript.json`
  - `source-transcript-parsed.json`
  - `normalized-transcript.json`
  - `normalization-summary.json`
  - `chunking-summary.json`
  - `utilization-diagnostics.json`
  - `correction-ledger.json`
  - `invocation.json`
  - `effective-config.json`
  - `report.json`
  - `error.log`
- Add a diagnostics helper that builds `reporting.DiagnosticsMetadata` from a run directory path and failure/success status.
- Update `RunDirectory` methods to use the constants.
- Update CLI report assembly and utilization/correction-ledger writes to use the constants/helper instead of raw strings.

Behavior changes:

- None. All artifact names, report JSON keys, and path values must remain byte-for-byte compatible except for normal timestamp/order differences in existing outputs.

Tests:

- Add or update `internal/core/diagnostics` tests proving metadata helper paths match the artifact constants.
- Run `go test ./internal/core/diagnostics ./internal/core/reporting ./internal/cli`.
- Run any existing CLI report/diagnostics tests touched by this stage.

Acceptance criteria:

- No raw core diagnostics artifact filename strings remain in CLI report metadata assembly.
- Existing success and failure reports still point to files that are actually written.
- Retention behavior is unchanged.

### Stage 2: Output schema validation and module catalog

Goal:

- Move public key validation to small canonical catalogs so config validation, runtime resolution, and factory behavior cannot drift.

Key edits:

- Add `SupportedKeys`, `IsSupported`, or an equivalent validation helper to `internal/core/outputschema`.
- Update `config.Validate` to use `internal/core/outputschema` for output schema validation.
- Add a small canonical module key catalog that is importable by:
  - `internal/core/config`
  - `internal/framework/modules`
  - `internal/validators`
  - `internal/framework/validators`
- Use the module catalog for default module key constants, known-key checks, validator chain keys, and confidence-threshold lookup.
- Keep module construction in `internal/framework/modules`; the catalog must not construct modules.

Behavior changes:

- Intended behavior change: unsupported configured module keys should fail during config validation, including `audita config validate`.
- Repeated supported module keys remain valid.
- Output schema behavior remains unchanged for `bare-segments`, `audita-v1`, and unsupported names.

Tests:

- Add `internal/core/config` tests for unsupported module keys and repeated supported module keys.
- Add config validation tests that every supported output schema validates.
- Add or update output schema registry tests for supported and unsupported schemas.
- Update module registry and validator chain tests to use the shared catalog where appropriate.
- Run `go test ./internal/core/config ./internal/core/outputschema ./internal/framework/modules ./internal/framework/validators ./internal/validators/... ./internal/cli`.

Acceptance criteria:

- Unknown modules fail before runner setup in config validation paths.
- No duplicated hardcoded output schema support list remains in config validation.
- No import cycle is introduced.

### Stage 3: Effective config loading context

Goal:

- Centralize config path resolution and defaults+file+env loading while keeping command-specific CLI overrides explicit.

Key edits:

- Move config path resolution from `internal/cli` into `internal/core/config` or add an equivalent exported helper there.
- Add an effective config loader that returns:
  - effective `config.Config`
  - config path
  - config source (`flag`, `env`, `default`, or empty)
  - config version pointer when a file was loaded
- Use the shared loader in `audita process` before applying CLI overrides.
- Use the shared loader in `audita config print-effective`.
- Keep `audita config validate` as file-only: load file, apply to defaults, validate, and do not apply environment overrides.

Behavior changes:

- None. Preserve existing precedence:
  - `process`: defaults, file config, environment, CLI flags
  - `config print-effective`: defaults, file config, environment
  - `config validate`: file config applied to defaults only
- Preserve explicit config path failure behavior and missing default path non-fatal behavior.

Tests:

- Add table-driven config loader tests for:
  - explicit `--config`
  - `AUDITA_CONFIG`
  - default search paths
  - missing explicit path
  - missing env path
  - missing default paths
- Add or update CLI tests proving `process` and `config print-effective` share file+env behavior.
- Add or update CLI tests proving `config validate` ignores environment overrides.
- Run `go test ./internal/core/config ./internal/cli ./cmd/audita`.

Acceptance criteria:

- Config precedence is unchanged.
- Config source/path/version metadata in invocation and reports is unchanged.
- Config command stdout/stderr and exit-code behavior is unchanged except for the intended unknown-module validation from Stage 2.

### Stage 4: Prompt/schema metadata and stage-name helpers

Goal:

- Centralize diagnostics-visible metadata and stage-name construction without changing production diagnostics names.

Key edits:

- Add a helper or method in `internal/prompts` that returns the stable prompt metadata diagnostics shape currently expanded by call sites.
- Add a helper or method in `internal/framework/responseschema` that returns the stable response schema metadata diagnostics shape currently expanded by call sites.
- Add shared proposal and validator stage-name helpers in the lowest package that avoids import cycles.
- Use the helpers in proposal generation, LLM validators, and production modules.

Behavior changes:

- None. Preserve current production stage names:
  - module proposal stages keep their existing `proposal` naming form;
  - validator batch stages keep their existing validator/batch naming form.
- Preserve all prompt metadata and response schema metadata field names and values.

Tests:

- Add prompt metadata helper tests covering every registered prompt.
- Add response schema metadata helper tests covering every registered response schema.
- Add stage-name helper tests for no-section, section, and validator batch cases.
- Run `go test ./internal/prompts ./internal/framework/responseschema ./internal/framework/proposal_generation ./internal/framework/validators ./internal/modules/...`.

Acceptance criteria:

- No manual prompt metadata map expansion remains in production module proposal plumbing.
- No duplicated response schema metadata map construction remains in proposal generation and LLM validators.
- Existing diagnostics fixture/path assertions still pass.

### Stage 5: Shared module proposal and prompt payload plumbing

Goal:

- Remove duplicated proposal execution and transcript-section prompt payload construction while preserving module-specific domain behavior.

Key edits:

- Add a narrow shared proposal execution helper, preferably in `internal/framework/proposal_generation` unless import cycles require a small module helper package.
- The helper should own:
  - transcript description extraction from config;
  - `GenerateCandidates` request construction;
  - prompt metadata attachment;
  - stage-name selection;
  - conversion from generated corrections/warnings to `contracts.ProposalResult`.
- Add shared transcript-section prompt payload construction in `internal/framework/promptcontext`.
- Update each production module to provide only:
  - module key;
  - replacement policy;
  - validator chain;
  - prompt ID;
  - domain-specific `BuildProposalMessages` call or message builder.
- Remove each module's redundant section transcript filtering if the runner already passes section-limited transcripts.

Behavior changes:

- None. Preserve module keys, replacement policies, validator chains, prompt IDs, diagnostics directories, proposal indexes, warning behavior, and correction mapping.

Tests:

- Add promptcontext tests for transcript section payload shape, empty transcript handling, section index, and category copying.
- Keep one module-specific prompt test per production module for domain wording and constraints.
- Add or update module proposal tests proving diagnostics are still written under the same module instance directory.
- Run `go test ./internal/framework/promptcontext ./internal/framework/proposal_generation ./internal/modules/... ./internal/cli`.

Acceptance criteria:

- Four production modules share proposal execution plumbing.
- Module packages remain separate and readable.
- CLI parity and release fixture behavior is unchanged.

### Stage 6: Validator classification and malformed LLM output policy

Goal:

- Use one source of truth for validator execution class and one shared classifier for malformed structured-output errors.

Key edits:

- Make validator execution class resolvable by stable validator key and by validator instance.
- Replace the correction-ledger hardcoded LLM-backed validator map with the canonical metadata source.
- Remove redundant validator metadata fields only after all call sites use the canonical source.
- Add a shared malformed structured-output classifier in `internal/framework/llm` or another low-level framework package.
- Update proposal generation and LLM validators to use the shared classifier while preserving their different handling outcomes.

Behavior changes:

- None. Proposal-generation malformed payloads still downgrade to warnings with zero proposals for affected sections.
- Validator malformed payloads still reject affected batches with warnings.
- Correction-ledger deterministic vs LLM validator sections should be unchanged for current validators.

Tests:

- Add validator metadata tests proving every registered validator has the expected execution class by key and instance.
- Add correction-ledger tests proving deterministic and LLM-backed decisions are classified through canonical metadata.
- Add shared malformed-output classifier tests covering current adapter malformed-output messages.
- Update proposal-generation and validator tests to assert representative malformed adapter errors are still downgraded.
- Run `go test ./internal/validators/... ./internal/framework/validators ./internal/framework/proposal_generation ./internal/framework/llm ./internal/cli`.

Acceptance criteria:

- No local hardcoded LLM-backed validator map remains in correction-ledger construction.
- Proposal-generation and validator malformed-output classifier lists cannot drift.
- Existing runner validator ordering is unchanged.

### Stage 7: Redaction and adapter workflow cleanup

Goal:

- Reduce duplicated secret extraction/redaction setup while preserving all no-secret-leak guarantees.

Key edits:

- Add a shared helper that extracts all configured LLM secret values from `config.Config`.
- Use the helper in proposal-generation diagnostics and validator diagnostics setup.
- Keep config structural redaction (`Config.Redacted`) separate from byte/string payload redaction.
- Keep adapter error redaction behavior compatible with current surfaced errors.
- Move runner adapter shims only if Stage 6 or this stage makes them materially larger; otherwise leave them in runner.

Behavior changes:

- None. Redaction token and no-secret-leak behavior remain unchanged.

Tests:

- Add or update tests proving proposal diagnostics, validator diagnostics, effective config artifacts, and surfaced adapter errors redact the same configured secrets.
- Keep existing subprocess no-secret-leak tests.
- Run `go test ./internal/core/config ./internal/framework/llm ./internal/framework/proposal_generation ./internal/framework/validators ./internal/cli ./cmd/audita`.

Acceptance criteria:

- Secret-list assembly is no longer duplicated between proposal and validator paths.
- No plaintext configured API key appears in diagnostics, reports, stdout, or stderr in existing redaction tests.
- No unrelated adapter behavior changes.

### Stage 8: Test helper cleanup and dead-code sweep

Goal:

- Consolidate test-only duplication and remove dead/redundant code left by prior stages.

Key edits:

- Consolidate package-local fake LLM clients, fixture readers, diagnostics glob helpers, and run-directory helpers where duplication is clear.
- Use cross-package test support only if it does not obscure test intent or introduce awkward imports.
- Remove redundant metadata fields, constants, or helper functions made obsolete by earlier stages.
- Keep module-specific prompt and behavior assertions local to each module package.

Behavior changes:

- None.

Tests:

- Run all package tests touched by helper cleanup.
- Run `go test ./internal/modules/... ./internal/framework/... ./internal/cli ./cmd/audita`.
- Run `go test ./...` before completing the full sequence.

Acceptance criteria:

- Test helpers are simpler without reducing coverage.
- No parity or release fixture assertions are removed unless replaced by equivalent or stronger assertions.
- No production behavior changes.

## Final verification

Before declaring the staged cleanup complete:

- Run:
  - `go test ./internal/core/config ./internal/core/outputschema`
  - `go test ./internal/core/diagnostics ./internal/core/reporting`
  - `go test ./internal/framework/proposal_generation ./internal/framework/validators ./internal/framework/runner`
  - `go test ./internal/validators/...`
  - `go test ./internal/modules/...`
  - `go test ./internal/cli ./cmd/audita`
  - `go test ./...`
- Inspect `git diff` for accidental public CLI, config, report, diagnostics, prompt metadata, stage-name, or output-schema changes.
- Update docs only when behavior intentionally changes, especially the intended Stage 2 unknown-module validation change.
- Keep commits stage-sized and mention behavior-preservation tests in each commit message or PR description.

## Assumptions

- Unknown configured module keys should become config-validation failures before 1.0.
- Diagnostics filenames and stage names are public enough to preserve unless a stage explicitly says otherwise.
- Each stage should be implemented and reviewed separately.