Complete Phase 16 default pipeline integration

docs/architecture.md

@@ -43,10 +43,19 @@ Implemented today:
 - Explicit runtime support for `--modules spoken_word` through the production runner path.
 
 Not implemented in CLI runtime path today:
-- End-to-end transcript polishing with the full default module sequence.
+- Python parity fixture suite and parity verification workflow.
+- Operational hardening tasks beyond current runtime/reporting/diagnostics behavior.
+- Rollout and Python retirement work.
 
 Current reality:
-- all production modules exist and are explicitly runnable by `--modules`, but default full-sequence integration remains a later phase.
+- all production modules exist and are wired into the default runtime path.
+- a normal `audita process` run without `--modules` now executes the full sequence:
+  - `glossary`
+  - `homophones`
+  - `glossary`
+  - `spoken_word`
+  - `grammar`
+- repeated glossary stages are deterministic and reported distinctly as `glossary_1` and `glossary_2`.
 
 Phase sequencing note:
 - Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
@@ -56,7 +65,8 @@ Phase sequencing note:
 - Phase 13 glossary module and protected-term behavior are complete;
 - Phase 14 homophones module implementation and explicit runtime wiring are complete;
 - Phase 15 spoken-word module implementation and explicit runtime wiring are complete;
-- next recommended phase is Phase 16 (default full pipeline integration).
+- Phase 16 default full pipeline integration is complete;
+- next recommended phase is Phase 17 (Python parity fixture suite).
 
 ## Actual Go package layout
 
@@ -165,19 +175,20 @@ Current runtime flow (`internal/cli/run.go`):
 9. Write normalized transcript and normalization summary artifacts.
 10. Chunk normalized transcript and compute chunk summaries.
 11. Write chunking summary artifact.
-12. Execute runner modules sequentially when:
-    - `--modules` is explicitly provided (production grammar/glossary/homophones/spoken_word paths); or
-    - a test/injected module factory is provided.
+12. Execute runner modules sequentially:
+    - default run path uses configured default sequence (`glossary,homophones,glossary,spoken_word,grammar`);
+    - explicit `--modules` overrides the default sequence;
+    - test/injected module factory path remains available for deterministic runtime tests.
 13. Output working transcript to `--output` file or stdout.
-14. Build process report (`phase` currently set to `phase15-spoken-word-module`).
+14. Build process report (`phase` currently set to `phase16-default-pipeline-integration`).
 15. Optionally write `--report-json`; always write run-dir `report.json`.
 16. Apply work-dir retention.
 
 Important behavior details:
 - Glossary is validated and is used for explicit glossary/grammar/homophones/spoken_word module correction paths.
-- Default production CLI behavior remains deterministic normalization/chunking/reporting unless modules are explicitly selected with `--modules`.
-- Explicit `--modules grammar`, `--modules glossary`, `--modules homophones`, and `--modules spoken_word` run production module paths with LLM-backed proposal generation and validator-chain execution.
-- Default runs (without explicit module selection) do not perform LLM calls.
+- Default production CLI behavior now executes the full production module sequence unless `--modules` override is supplied.
+- Explicit `--modules grammar`, `--modules glossary`, `--modules homophones`, and `--modules spoken_word` continue to run production module paths with LLM-backed proposal generation and validator-chain execution.
+- Default runs (without explicit module selection) perform LLM calls through production module and validator paths.
 - Success path is generally quiet on stderr.
 - Source IDs are preserved into a canonical transcript before normalization; normalization then reassigns output IDs sequentially from `1`.
 
@@ -225,7 +236,7 @@ Implemented config surfaces include:
 - work-dir and retention mode
 
 Current caveat:
-- LLM/module-related settings are active for explicit grammar/glossary/homophones/spoken_word runs; the default non-explicit path remains deterministic.
+- LLM/module-related settings are active for default and explicit module-run paths.
 
 ## Implemented structured LLM infrastructure
 `internal/framework/contracts` now defines a typed structured-completion contract:
@@ -241,8 +252,8 @@ Current caveat:
 - API-key redaction in adapter-returned errors.
 
 Current runtime boundary:
-- the default CLI runtime path (without explicit module selection) still does not instantiate the full production module sequence.
-- LLM calls are exercised in production when `--modules grammar`, `--modules glossary`, `--modules homophones`, or `--modules spoken_word` is explicitly requested and in tests when fake/injected clients are used.
+- the default CLI runtime path (without explicit module selection) instantiates the full production module sequence.
+- LLM calls are exercised in production in both default full-pipeline runs and explicit `--modules` runs, and in tests when fake/injected clients are used.
 
 `internal/framework/llm` also provides:
 - a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;

docs/rewrite-notes.md

@@ -71,7 +71,18 @@ Implemented:
 - Production spoken_word module package with Python-aligned prompt intent and guardrails.
 - Explicit `--modules spoken_word` runtime path through runner, shared proposal generation, validators, application, reporting, and diagnostics.
 - Focused multi-module runtime tests for already-implemented interoperability (for example `spoken_word,grammar`) without claiming full default-pipeline completion.
-- All production modules now exist (`glossary`, `homophones`, `spoken_word`, `grammar`), but default full-sequence integration remains Phase 16 work.
+- All production modules now exist (`glossary`, `homophones`, `spoken_word`, `grammar`) and are integrated into the default full-sequence runtime path.
+- Default runtime sequence is now active and ordered as:
+  - `glossary`
+  - `homophones`
+  - `glossary`
+  - `spoken_word`
+  - `grammar`
+- Repeated glossary stages resolve and report deterministically as `glossary_1` and `glossary_2`.
+- Full-pipeline module reports and run-level summaries aggregate applied/skipped/failed metadata across all module instances.
+- Mid-pipeline failure reporting preserves partial progress and failed-module metadata.
+- Full-pipeline diagnostics include proposal/validator prompt-response artifacts with redaction.
+- Skip-aware retention uses actual module skipped/rejected correction data.
 - Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
 - Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
 - `internal/framework/llm` instructor-go-backed adapter with:
@@ -87,7 +98,9 @@ Implemented:
 - Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
 
 Not yet implemented in runtime pipeline:
-- End-to-end transcript polishing behavior with the full default module sequence.
+- Python parity fixture suite and parity verification workflow.
+- Operational hardening beyond current Phase 16 runtime/reporting/diagnostics scope.
+- Rollout/Python retirement work.
 
 ## Completed phases
 
@@ -192,7 +205,7 @@ Not implemented in Phase 7 (by design):
 - End-to-end transcript polishing.
 
 Current runtime behavior note:
-- Default user-facing CLI behavior remains deterministic normalization/chunking output unless test-only module injection is used during tests.
+- Default user-facing CLI behavior now executes the full production module sequence unless `--modules` explicitly overrides it.
 
 ## Phase 8: Runtime validator framework and deterministic validators
 Completed.
@@ -225,7 +238,7 @@ Not implemented in Phase 8 (by design):
 
 ## Remaining work plan
 
-Next recommended phase: **Phase 16 (default full pipeline integration)**.
+Next recommended phase: **Phase 17 (Python parity fixture suite)**.
 
 ## Phase 9: Structured LLM client and scheduler infrastructure
 
@@ -446,6 +459,8 @@ Not implemented in Phase 15 (by design):
 
 ## Phase 16: Default full pipeline integration
 
+Completed.
+
 ### Purpose
 
 Enable and harden the full default module sequence in the Go runtime path.
@@ -489,6 +504,27 @@ Running `audita process transcript.json --glossary glossary.yaml --output correc
 - CLI stdout/stderr behavior remains subprocess-safe.
 - `go test ./...` passes without requiring external LLM credentials.
 
+### Phase 16 completion status
+
+Implemented:
+- Normal `audita process` runs without `--modules` now execute the full sequence:
+  - `glossary`
+  - `homophones`
+  - `glossary`
+  - `spoken_word`
+  - `grammar`
+- Explicit `--modules` still overrides the default sequence.
+- Repeated glossary stages are deterministic (`glossary_1`, `glossary_2`) and reported distinctly.
+- Full-pipeline module reports aggregate applied changes, application skips, validator rejections, and failed-module metadata.
+- Full-pipeline diagnostics include prompt/response artifacts for module proposal generation and validator LLM interactions with secret redaction.
+- Mid-pipeline failure preserves partial module progress in reports and retains diagnostics + `error.log`.
+- Auto-retention keeps successful runs with actual skipped/rejected corrections and always retains failed runs.
+
+Intentionally deferred:
+- Phase 17 parity fixture suite.
+- Phase 18 operational hardening.
+- Phase 19 rollout/Python retirement work.
+
 ## Phase 17: Python parity fixture suite
 
 ### Purpose