Complete Phase 11 proposal generation framework
This commit is contained in:
@@ -29,17 +29,21 @@ Implemented today:
|
||||
- LLM-backed validator models, prompt builders, batching, and runtime execution.
|
||||
- Runner wiring for LLM validators via the internal structured LLM abstraction and scheduler hooks.
|
||||
- LLM validator diagnostics artifacts and report-level decision metadata paths.
|
||||
- Shared LLM proposal-generation helper with structured correction-set parsing.
|
||||
- Deterministic proposal-index assignment and enriched proposal mapping for shared generation.
|
||||
- Proposal-generation diagnostics artifacts with secret redaction.
|
||||
- Production module registry scaffolding with known-key recognition and explicit unsupported/unimplemented errors.
|
||||
|
||||
Not implemented in CLI runtime path today:
|
||||
- Real module execution pipeline (`glossary`, `homophones`, `spoken_word`, `grammar`).
|
||||
- Structured LLM proposal generation.
|
||||
- Shared module proposal-generation framework and module registry for real modules.
|
||||
- Real domain proposal prompts for production modules.
|
||||
- End-to-end transcript polishing with real module behavior.
|
||||
|
||||
Phase sequencing note:
|
||||
- Phase 9 LLM infrastructure is complete (structured client, scheduler, effective config resolution, diagnostics primitives);
|
||||
- Phase 10 LLM-backed validator runtime integration is complete;
|
||||
- shared module proposal-generation and module registry work remain Phase 11.
|
||||
- Phase 11 shared proposal-generation framework and module-registry scaffolding are complete;
|
||||
- next recommended phase is Phase 12 (grammar module).
|
||||
|
||||
## Actual Go package layout
|
||||
|
||||
@@ -92,6 +96,12 @@ internal/framework/proposals/
|
||||
internal/framework/runner/
|
||||
runner.go
|
||||
|
||||
internal/framework/proposal_generation/
|
||||
generate.go
|
||||
|
||||
internal/framework/modules/
|
||||
registry.go
|
||||
|
||||
internal/framework/validators/
|
||||
models.go
|
||||
deterministic.go
|
||||
@@ -126,9 +136,9 @@ Current runtime flow (`internal/cli/run.go`):
|
||||
9. Write normalized transcript and normalization summary artifacts.
|
||||
10. Chunk normalized transcript and compute chunk summaries.
|
||||
11. Write chunking summary artifact.
|
||||
12. Optionally execute runner modules sequentially when an injected module registry/factory is available (used by deterministic tests today).
|
||||
12. Optionally execute runner modules sequentially when a module factory is injected (tests currently use this path; production defaults still avoid real module execution).
|
||||
13. Output working transcript to `--output` file or stdout.
|
||||
14. Build process report (`phase` currently set to `phase10-llm-validators`).
|
||||
14. Build process report (`phase` currently set to `phase11-proposal-generation-framework`).
|
||||
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
||||
16. Apply work-dir retention.
|
||||
|
||||
@@ -183,7 +193,7 @@ Implemented config surfaces include:
|
||||
- work-dir and retention mode
|
||||
|
||||
Current caveat:
|
||||
- LLM/module-related settings are mostly infrastructure-only today; runtime path does not execute LLM or modules.
|
||||
- LLM/module-related settings are mostly infrastructure-only today; default runtime path does not execute real modules.
|
||||
|
||||
## Implemented structured LLM infrastructure
|
||||
`internal/framework/contracts` now defines a typed structured-completion contract:
|
||||
@@ -280,6 +290,35 @@ Validator rejections are reported distinctly from proposal-application skips.
|
||||
- bounded scheduler hooks for validator call execution;
|
||||
- diagnostics writer hooks for machine-readable prompt/response artifacts with secret redaction.
|
||||
|
||||
## Implemented shared proposal-generation infrastructure
|
||||
`internal/framework/proposal_generation` provides a reusable, prompt-agnostic helper for future real modules:
|
||||
- structured request model including module key/instance, replacement policy, working transcript context, optional section metadata, glossary, config, and diagnostics context;
|
||||
- structured correction-set response model (`corrections`) mapped into existing `proposals.CorrectionProposal` and `proposals.EnrichedCorrectionProposal` models;
|
||||
- deterministic proposal-index assignment through a caller-provided `start_index`;
|
||||
- structured LLM calls through `contracts.StructuredLLMClient` only (no direct provider calls);
|
||||
- optional bounded execution through scheduler hooks (`contracts.LLMScheduler`);
|
||||
- prompt/response diagnostics artifact writing via the generic `internal/framework/llm` diagnostics primitives with redaction of API keys/secrets.
|
||||
|
||||
This helper only produces candidate proposals; validator-chain execution and proposal application remain runner responsibilities.
|
||||
|
||||
## Implemented production module-registry scaffolding
|
||||
`internal/framework/modules` now provides a production registry scaffold:
|
||||
- recognizes intended module keys:
|
||||
- `glossary`
|
||||
- `homophones`
|
||||
- `spoken_word`
|
||||
- `grammar`
|
||||
- supports explicit constructor registration with dependency injection for:
|
||||
- run spec
|
||||
- config
|
||||
- glossary
|
||||
- proposal/validation structured LLM clients
|
||||
- proposal/validation schedulers
|
||||
- diagnostics directory context
|
||||
- returns explicit errors for unknown keys (`unsupported_module`) and recognized-but-unimplemented keys (`unimplemented_module`).
|
||||
|
||||
No real production correction modules are registered yet.
|
||||
|
||||
## Reports and diagnostics (implemented)
|
||||
Current per-run artifacts include:
|
||||
- `source-transcript.json`
|
||||
@@ -321,7 +360,7 @@ Current runtime note:
|
||||
- real module execution is not implemented yet, so normal successful runs generally have no skipped corrections and `auto` typically removes clean successful run directories.
|
||||
|
||||
Intentionally deferred to module/LLM phases:
|
||||
- module proposal-generation prompt/response diagnostics remain tied to later real-module phases.
|
||||
- real domain proposal prompts and production module implementations remain tied to later module phases.
|
||||
|
||||
## Current tests and quality posture
|
||||
Implemented tests currently cover:
|
||||
@@ -337,8 +376,10 @@ Implemented tests currently cover:
|
||||
- CLI runner integration through injected fake module factories (`internal/cli/run_test.go`)
|
||||
- validator models, cardinality enforcement, and deterministic validators (`internal/framework/validators/*_test.go`)
|
||||
- LLM-backed validator batching, prompt builders, structured-response safety, scheduler hooks, and diagnostics redaction (`internal/framework/validators/*_test.go`, `internal/framework/runner/*_test.go`)
|
||||
- shared proposal-generation request/response parsing, deterministic indexing, scheduler hooks, and diagnostics redaction (`internal/framework/proposal_generation/*_test.go`, `internal/framework/runner/*_test.go`)
|
||||
- production module-registry known-key recognition and unsupported/unimplemented error behavior (`internal/framework/modules/*_test.go`, `internal/cli/run_test.go`)
|
||||
|
||||
Not covered yet (because not implemented): shared module proposal generation, real module implementations, and full transcript-polishing runtime behavior.
|
||||
Not covered yet (because not implemented): real production module implementations and full transcript-polishing runtime behavior.
|
||||
|
||||
## Intended final architecture (not yet implemented)
|
||||
The intended end-state still matches the rewrite plan:
|
||||
|
||||
@@ -55,6 +55,10 @@ Implemented:
|
||||
- LLM validator batching by validation prompt-token budget.
|
||||
- LLM validator runtime integration through structured LLM client abstraction and scheduler hooks.
|
||||
- LLM validator prompt/response diagnostics artifact wiring with secret redaction.
|
||||
- Shared LLM proposal-generation helper with structured correction-set parsing.
|
||||
- Deterministic proposal-index assignment for shared proposal generation.
|
||||
- Proposal-generation prompt/response diagnostics artifact wiring with secret redaction.
|
||||
- Production module-registry scaffolding with known key recognition and explicit unsupported/unimplemented errors.
|
||||
- Broad deterministic and CLI/subprocess test coverage for implemented phases through `go test ./...`.
|
||||
- Internal typed structured LLM contract (`StructuredLLMClient.CompleteStructured(ctx, req, out)`).
|
||||
- `internal/framework/llm` instructor-go-backed adapter with:
|
||||
@@ -71,7 +75,7 @@ Implemented:
|
||||
|
||||
Not yet implemented in runtime pipeline:
|
||||
- Real correction modules.
|
||||
- Shared module proposal generation and module registry wiring.
|
||||
- Domain proposal prompts for real modules.
|
||||
- End-to-end transcript polishing behavior.
|
||||
|
||||
## Completed phases
|
||||
@@ -210,7 +214,7 @@ Not implemented in Phase 8 (by design):
|
||||
|
||||
## Remaining work plan
|
||||
|
||||
Next recommended phase: **Phase 11 (shared LLM proposal generation framework and module registry)**.
|
||||
Next recommended phase: **Phase 12 (grammar module)**.
|
||||
|
||||
## Phase 9: Structured LLM client and scheduler infrastructure
|
||||
|
||||
@@ -261,7 +265,7 @@ Do not implement:
|
||||
|
||||
### Expected behavior at end of phase
|
||||
|
||||
The codebase has tested Phase 9 LLM infrastructure, but default CLI runtime behavior remains deterministic preprocessing/reporting because real modules and LLM-backed validators are not implemented.
|
||||
At the end of Phase 9, the codebase had tested LLM infrastructure primitives, while default CLI runtime behavior remained deterministic preprocessing/reporting because real modules were not implemented yet.
|
||||
|
||||
### Definition of done status
|
||||
|
||||
@@ -298,48 +302,40 @@ Implemented:
|
||||
|
||||
Not implemented in Phase 10 (by design):
|
||||
- Real correction modules (`glossary`, `homophones`, `spoken_word`, `grammar`).
|
||||
- Shared module proposal generation and module registry work (Phase 11).
|
||||
- Real module implementation and full runtime wiring (Phase 12+).
|
||||
- Domain proposal prompts.
|
||||
- Default CLI end-to-end transcript polishing behavior.
|
||||
|
||||
## Phase 11: Shared LLM proposal generation framework and module registry
|
||||
|
||||
### Purpose
|
||||
Completed.
|
||||
|
||||
Create the reusable proposal-generation layer used by all real modules, and establish the real module registry without yet requiring all modules to be fully implemented.
|
||||
Implemented:
|
||||
- Shared proposal-generation package `internal/framework/proposal_generation`.
|
||||
- Reusable request model for proposal generation including:
|
||||
- module key/instance
|
||||
- replacement policy
|
||||
- working transcript context
|
||||
- optional section metadata
|
||||
- glossary/config context
|
||||
- diagnostics context
|
||||
- injected structured LLM client/scheduler dependencies.
|
||||
- Structured correction-set response model and parsing into existing proposal models:
|
||||
- `proposals.CorrectionProposal`
|
||||
- `proposals.EnrichedCorrectionProposal`.
|
||||
- Deterministic proposal-index assignment via caller-provided start index.
|
||||
- Proposal-generation diagnostics artifact writing using generic LLM diagnostics primitives with secret redaction.
|
||||
- Scheduler-aware proposal generation through the internal LLM scheduler interface.
|
||||
- Production module-registry scaffolding in `internal/framework/modules` with:
|
||||
- known module-key recognition for `glossary`, `homophones`, `spoken_word`, `grammar`
|
||||
- constructor registration and dependency-injection path
|
||||
- explicit unsupported and recognized-but-unimplemented module errors.
|
||||
- Runner/CLI injection-path tests showing shared proposal generation can flow through runner validation/application semantics using fake modules/clients.
|
||||
|
||||
### Scope
|
||||
|
||||
Implement:
|
||||
- Shared LLM proposal generation helper.
|
||||
- Proposal prompt request/response models.
|
||||
- Structured correction set response parsing.
|
||||
- Proposal index assignment.
|
||||
- Module prompt/response diagnostics.
|
||||
- Module registry package for real module keys.
|
||||
- Module construction from run specs.
|
||||
- Clean unsupported-module behavior.
|
||||
- Shared module test harness using fake LLM responses.
|
||||
- One minimal real module may be implemented as a proof of the proposal-generation path if that keeps the phase coherent, but only if it does not blur scope.
|
||||
|
||||
Do not implement:
|
||||
- All real modules.
|
||||
- Full default pipeline parity.
|
||||
- Prompt improvements beyond faithful porting of Python behavior.
|
||||
|
||||
### Expected behavior at end of phase
|
||||
|
||||
The framework can support real LLM proposal generation, and modules can be registered and instantiated consistently. At least the infrastructure for real modules exists, even if most modules are implemented in later phases.
|
||||
|
||||
### Definition of done
|
||||
|
||||
- Shared proposal-generation helper exists.
|
||||
- Proposal prompt/response diagnostics are written for module proposal calls.
|
||||
- Module registry resolves known module keys deterministically.
|
||||
- Unsupported modules fail cleanly.
|
||||
- Fake module tests exercise shared proposal-generation behavior.
|
||||
- Module reports include proposal-generation failures where applicable.
|
||||
- `go test ./...` passes.
|
||||
Not implemented in Phase 11 (by design):
|
||||
- Real production `glossary`, `homophones`, `spoken_word`, and `grammar` modules.
|
||||
- Domain proposal prompts for production modules.
|
||||
- Default CLI end-to-end transcript polishing behavior.
|
||||
|
||||
## Phase 12: Grammar module
|
||||
|
||||
|
||||
Reference in New Issue
Block a user