330 lines
15 KiB
Markdown
330 lines
15 KiB
Markdown
# Pre-1.0 Deduplication Implementation Plan
|
|
|
|
This plan turns `docs/roadmap/audit.md` into staged, prompt-sized cleanup work for an LLM coding agent. Each stage should be implemented in order and kept small enough to review as an independent commit.
|
|
|
|
## Operating rules
|
|
|
|
- Read `docs/roadmap/audit.md` before starting any stage.
|
|
- Preserve public CLI, report, diagnostics, config precedence, prompt metadata, and output-schema behavior unless a stage explicitly calls out an intended behavior change.
|
|
- Keep the four production module packages separate: `glossary`, `homophones`, `spoken_word`, and `grammar`.
|
|
- Do not introduce plugin systems, generic workflow engines, broad CLI framework rewrites, reflection-heavy config mappers, or merged module packages.
|
|
- Prefer narrow helpers, catalogs, constants, and pure mapping functions over broad abstractions.
|
|
- Run the targeted tests listed in each stage before moving to the next stage.
|
|
- Run `go test ./...` before declaring the full sequence complete.
|
|
- Ignore unrelated worktree changes, including the existing deletion of `docs/roadmap/publish.md`, unless the user explicitly asks to handle them.
|
|
- Do not reduce parity, release-fixture, subprocess, or module-specific behavior coverage while consolidating helpers.
|
|
|
|
## Stages
|
|
|
|
### Stage 1: Diagnostics artifact constants and metadata paths
|
|
|
|
Goal:
|
|
|
|
- Centralize diagnostics artifact names and report diagnostics metadata path construction without changing any filenames or report fields.
|
|
|
|
Key edits:
|
|
|
|
- Define constants in `internal/core/diagnostics` for:
|
|
- `source-transcript.json`
|
|
- `source-transcript-parsed.json`
|
|
- `normalized-transcript.json`
|
|
- `normalization-summary.json`
|
|
- `chunking-summary.json`
|
|
- `utilization-diagnostics.json`
|
|
- `correction-ledger.json`
|
|
- `invocation.json`
|
|
- `effective-config.json`
|
|
- `report.json`
|
|
- `error.log`
|
|
- Add a diagnostics helper that builds `reporting.DiagnosticsMetadata` from a run directory path and failure/success status.
|
|
- Update `RunDirectory` methods to use the constants.
|
|
- Update CLI report assembly and utilization/correction-ledger writes to use the constants/helper instead of raw strings.
|
|
|
|
Behavior changes:
|
|
|
|
- None. All artifact names, report JSON keys, and path values must remain byte-for-byte compatible except for normal timestamp/order differences in existing outputs.
|
|
|
|
Tests:
|
|
|
|
- Add or update `internal/core/diagnostics` tests proving metadata helper paths match the artifact constants.
|
|
- Run `go test ./internal/core/diagnostics ./internal/core/reporting ./internal/cli`.
|
|
- Run any existing CLI report/diagnostics tests touched by this stage.
|
|
|
|
Acceptance criteria:
|
|
|
|
- No raw core diagnostics artifact filename strings remain in CLI report metadata assembly.
|
|
- Existing success and failure reports still point to files that are actually written.
|
|
- Retention behavior is unchanged.
|
|
|
|
### Stage 2: Output schema validation and module catalog
|
|
|
|
Goal:
|
|
|
|
- Move public key validation to small canonical catalogs so config validation, runtime resolution, and factory behavior cannot drift.
|
|
|
|
Key edits:
|
|
|
|
- Add `SupportedKeys`, `IsSupported`, or an equivalent validation helper to `internal/core/outputschema`.
|
|
- Update `config.Validate` to use `internal/core/outputschema` for output schema validation.
|
|
- Add a small canonical module key catalog that is importable by:
|
|
- `internal/core/config`
|
|
- `internal/framework/modules`
|
|
- `internal/validators`
|
|
- `internal/framework/validators`
|
|
- Use the module catalog for default module key constants, known-key checks, validator chain keys, and confidence-threshold lookup.
|
|
- Keep module construction in `internal/framework/modules`; the catalog must not construct modules.
|
|
|
|
Behavior changes:
|
|
|
|
- Intended behavior change: unsupported configured module keys should fail during config validation, including `audita config validate`.
|
|
- Repeated supported module keys remain valid.
|
|
- Output schema behavior remains unchanged for `bare-segments`, `audita-v1`, and unsupported names.
|
|
|
|
Tests:
|
|
|
|
- Add `internal/core/config` tests for unsupported module keys and repeated supported module keys.
|
|
- Add config validation tests that every supported output schema validates.
|
|
- Add or update output schema registry tests for supported and unsupported schemas.
|
|
- Update module registry and validator chain tests to use the shared catalog where appropriate.
|
|
- Run `go test ./internal/core/config ./internal/core/outputschema ./internal/framework/modules ./internal/framework/validators ./internal/validators/... ./internal/cli`.
|
|
|
|
Acceptance criteria:
|
|
|
|
- Unknown modules fail before runner setup in config validation paths.
|
|
- No duplicated hardcoded output schema support list remains in config validation.
|
|
- No import cycle is introduced.
|
|
|
|
### Stage 3: Effective config loading context
|
|
|
|
Goal:
|
|
|
|
- Centralize config path resolution and defaults+file+env loading while keeping command-specific CLI overrides explicit.
|
|
|
|
Key edits:
|
|
|
|
- Move config path resolution from `internal/cli` into `internal/core/config` or add an equivalent exported helper there.
|
|
- Add an effective config loader that returns:
|
|
- effective `config.Config`
|
|
- config path
|
|
- config source (`flag`, `env`, `default`, or empty)
|
|
- config version pointer when a file was loaded
|
|
- Use the shared loader in `audita process` before applying CLI overrides.
|
|
- Use the shared loader in `audita config print-effective`.
|
|
- Keep `audita config validate` as file-only: load file, apply to defaults, validate, and do not apply environment overrides.
|
|
|
|
Behavior changes:
|
|
|
|
- None. Preserve existing precedence:
|
|
- `process`: defaults, file config, environment, CLI flags
|
|
- `config print-effective`: defaults, file config, environment
|
|
- `config validate`: file config applied to defaults only
|
|
- Preserve explicit config path failure behavior and missing default path non-fatal behavior.
|
|
|
|
Tests:
|
|
|
|
- Add table-driven config loader tests for:
|
|
- explicit `--config`
|
|
- `AUDITA_CONFIG`
|
|
- default search paths
|
|
- missing explicit path
|
|
- missing env path
|
|
- missing default paths
|
|
- Add or update CLI tests proving `process` and `config print-effective` share file+env behavior.
|
|
- Add or update CLI tests proving `config validate` ignores environment overrides.
|
|
- Run `go test ./internal/core/config ./internal/cli ./cmd/audita`.
|
|
|
|
Acceptance criteria:
|
|
|
|
- Config precedence is unchanged.
|
|
- Config source/path/version metadata in invocation and reports is unchanged.
|
|
- Config command stdout/stderr and exit-code behavior is unchanged except for the intended unknown-module validation from Stage 2.
|
|
|
|
### Stage 4: Prompt/schema metadata and stage-name helpers
|
|
|
|
Goal:
|
|
|
|
- Centralize diagnostics-visible metadata and stage-name construction without changing production diagnostics names.
|
|
|
|
Key edits:
|
|
|
|
- Add a helper or method in `internal/prompts` that returns the stable prompt metadata diagnostics shape currently expanded by call sites.
|
|
- Add a helper or method in `internal/framework/responseschema` that returns the stable response schema metadata diagnostics shape currently expanded by call sites.
|
|
- Add shared proposal and validator stage-name helpers in the lowest package that avoids import cycles.
|
|
- Use the helpers in proposal generation, LLM validators, and production modules.
|
|
|
|
Behavior changes:
|
|
|
|
- None. Preserve current production stage names:
|
|
- module proposal stages keep their existing `proposal` naming form;
|
|
- validator batch stages keep their existing validator/batch naming form.
|
|
- Preserve all prompt metadata and response schema metadata field names and values.
|
|
|
|
Tests:
|
|
|
|
- Add prompt metadata helper tests covering every registered prompt.
|
|
- Add response schema metadata helper tests covering every registered response schema.
|
|
- Add stage-name helper tests for no-section, section, and validator batch cases.
|
|
- Run `go test ./internal/prompts ./internal/framework/responseschema ./internal/framework/proposal_generation ./internal/framework/validators ./internal/modules/...`.
|
|
|
|
Acceptance criteria:
|
|
|
|
- No manual prompt metadata map expansion remains in production module proposal plumbing.
|
|
- No duplicated response schema metadata map construction remains in proposal generation and LLM validators.
|
|
- Existing diagnostics fixture/path assertions still pass.
|
|
|
|
### Stage 5: Shared module proposal and prompt payload plumbing
|
|
|
|
Goal:
|
|
|
|
- Remove duplicated proposal execution and transcript-section prompt payload construction while preserving module-specific domain behavior.
|
|
|
|
Key edits:
|
|
|
|
- Add a narrow shared proposal execution helper, preferably in `internal/framework/proposal_generation` unless import cycles require a small module helper package.
|
|
- The helper should own:
|
|
- transcript description extraction from config;
|
|
- `GenerateCandidates` request construction;
|
|
- prompt metadata attachment;
|
|
- stage-name selection;
|
|
- conversion from generated corrections/warnings to `contracts.ProposalResult`.
|
|
- Add shared transcript-section prompt payload construction in `internal/framework/promptcontext`.
|
|
- Update each production module to provide only:
|
|
- module key;
|
|
- replacement policy;
|
|
- validator chain;
|
|
- prompt ID;
|
|
- domain-specific `BuildProposalMessages` call or message builder.
|
|
- Remove each module's redundant section transcript filtering if the runner already passes section-limited transcripts.
|
|
|
|
Behavior changes:
|
|
|
|
- None. Preserve module keys, replacement policies, validator chains, prompt IDs, diagnostics directories, proposal indexes, warning behavior, and correction mapping.
|
|
|
|
Tests:
|
|
|
|
- Add promptcontext tests for transcript section payload shape, empty transcript handling, section index, and category copying.
|
|
- Keep one module-specific prompt test per production module for domain wording and constraints.
|
|
- Add or update module proposal tests proving diagnostics are still written under the same module instance directory.
|
|
- Run `go test ./internal/framework/promptcontext ./internal/framework/proposal_generation ./internal/modules/... ./internal/cli`.
|
|
|
|
Acceptance criteria:
|
|
|
|
- Four production modules share proposal execution plumbing.
|
|
- Module packages remain separate and readable.
|
|
- CLI parity and release fixture behavior is unchanged.
|
|
|
|
### Stage 6: Validator classification and malformed LLM output policy
|
|
|
|
Goal:
|
|
|
|
- Use one source of truth for validator execution class and one shared classifier for malformed structured-output errors.
|
|
|
|
Key edits:
|
|
|
|
- Make validator execution class resolvable by stable validator key and by validator instance.
|
|
- Replace the correction-ledger hardcoded LLM-backed validator map with the canonical metadata source.
|
|
- Remove redundant validator metadata fields only after all call sites use the canonical source.
|
|
- Add a shared malformed structured-output classifier in `internal/framework/llm` or another low-level framework package.
|
|
- Update proposal generation and LLM validators to use the shared classifier while preserving their different handling outcomes.
|
|
|
|
Behavior changes:
|
|
|
|
- None. Proposal-generation malformed payloads still downgrade to warnings with zero proposals for affected sections.
|
|
- Validator malformed payloads still reject affected batches with warnings.
|
|
- Correction-ledger deterministic vs LLM validator sections should be unchanged for current validators.
|
|
|
|
Tests:
|
|
|
|
- Add validator metadata tests proving every registered validator has the expected execution class by key and instance.
|
|
- Add correction-ledger tests proving deterministic and LLM-backed decisions are classified through canonical metadata.
|
|
- Add shared malformed-output classifier tests covering current adapter malformed-output messages.
|
|
- Update proposal-generation and validator tests to assert representative malformed adapter errors are still downgraded.
|
|
- Run `go test ./internal/validators/... ./internal/framework/validators ./internal/framework/proposal_generation ./internal/framework/llm ./internal/cli`.
|
|
|
|
Acceptance criteria:
|
|
|
|
- No local hardcoded LLM-backed validator map remains in correction-ledger construction.
|
|
- Proposal-generation and validator malformed-output classifier lists cannot drift.
|
|
- Existing runner validator ordering is unchanged.
|
|
|
|
### Stage 7: Redaction and adapter workflow cleanup
|
|
|
|
Goal:
|
|
|
|
- Reduce duplicated secret extraction/redaction setup while preserving all no-secret-leak guarantees.
|
|
|
|
Key edits:
|
|
|
|
- Add a shared helper that extracts all configured LLM secret values from `config.Config`.
|
|
- Use the helper in proposal-generation diagnostics and validator diagnostics setup.
|
|
- Keep config structural redaction (`Config.Redacted`) separate from byte/string payload redaction.
|
|
- Keep adapter error redaction behavior compatible with current surfaced errors.
|
|
- Move runner adapter shims only if Stage 6 or this stage makes them materially larger; otherwise leave them in runner.
|
|
|
|
Behavior changes:
|
|
|
|
- None. Redaction token and no-secret-leak behavior remain unchanged.
|
|
|
|
Tests:
|
|
|
|
- Add or update tests proving proposal diagnostics, validator diagnostics, effective config artifacts, and surfaced adapter errors redact the same configured secrets.
|
|
- Keep existing subprocess no-secret-leak tests.
|
|
- Run `go test ./internal/core/config ./internal/framework/llm ./internal/framework/proposal_generation ./internal/framework/validators ./internal/cli ./cmd/audita`.
|
|
|
|
Acceptance criteria:
|
|
|
|
- Secret-list assembly is no longer duplicated between proposal and validator paths.
|
|
- No plaintext configured API key appears in diagnostics, reports, stdout, or stderr in existing redaction tests.
|
|
- No unrelated adapter behavior changes.
|
|
|
|
### Stage 8: Test helper cleanup and dead-code sweep
|
|
|
|
Goal:
|
|
|
|
- Consolidate test-only duplication and remove dead/redundant code left by prior stages.
|
|
|
|
Key edits:
|
|
|
|
- Consolidate package-local fake LLM clients, fixture readers, diagnostics glob helpers, and run-directory helpers where duplication is clear.
|
|
- Use cross-package test support only if it does not obscure test intent or introduce awkward imports.
|
|
- Remove redundant metadata fields, constants, or helper functions made obsolete by earlier stages.
|
|
- Keep module-specific prompt and behavior assertions local to each module package.
|
|
|
|
Behavior changes:
|
|
|
|
- None.
|
|
|
|
Tests:
|
|
|
|
- Run all package tests touched by helper cleanup.
|
|
- Run `go test ./internal/modules/... ./internal/framework/... ./internal/cli ./cmd/audita`.
|
|
- Run `go test ./...` before completing the full sequence.
|
|
|
|
Acceptance criteria:
|
|
|
|
- Test helpers are simpler without reducing coverage.
|
|
- No parity or release fixture assertions are removed unless replaced by equivalent or stronger assertions.
|
|
- No production behavior changes.
|
|
|
|
## Final verification
|
|
|
|
Before declaring the staged cleanup complete:
|
|
|
|
- Run:
|
|
- `go test ./internal/core/config ./internal/core/outputschema`
|
|
- `go test ./internal/core/diagnostics ./internal/core/reporting`
|
|
- `go test ./internal/framework/proposal_generation ./internal/framework/validators ./internal/framework/runner`
|
|
- `go test ./internal/validators/...`
|
|
- `go test ./internal/modules/...`
|
|
- `go test ./internal/cli ./cmd/audita`
|
|
- `go test ./...`
|
|
- Inspect `git diff` for accidental public CLI, config, report, diagnostics, prompt metadata, stage-name, or output-schema changes.
|
|
- Update docs only when behavior intentionally changes, especially the intended Stage 2 unknown-module validation change.
|
|
- Keep commits stage-sized and mention behavior-preservation tests in each commit message or PR description.
|
|
|
|
## Assumptions
|
|
|
|
- Unknown configured module keys should become config-validation failures before 1.0.
|
|
- Diagnostics filenames and stage names are public enough to preserve unless a stage explicitly says otherwise.
|
|
- Each stage should be implemented and reviewed separately.
|