Complete Phase 17 parity fixture suite
This commit is contained in:
@@ -43,7 +43,6 @@ Implemented today:
|
||||
- Explicit runtime support for `--modules spoken_word` through the production runner path.
|
||||
|
||||
Not implemented in CLI runtime path today:
|
||||
- Python parity fixture suite and parity verification workflow.
|
||||
- Operational hardening tasks beyond current runtime/reporting/diagnostics behavior.
|
||||
- Rollout and Python retirement work.
|
||||
|
||||
@@ -66,7 +65,8 @@ Phase sequencing note:
|
||||
- Phase 14 homophones module implementation and explicit runtime wiring are complete;
|
||||
- Phase 15 spoken-word module implementation and explicit runtime wiring are complete;
|
||||
- Phase 16 default full pipeline integration is complete;
|
||||
- next recommended phase is Phase 17 (Python parity fixture suite).
|
||||
- Phase 17 parity fixture suite is complete;
|
||||
- next recommended phase is Phase 18 (operational hardening and subprocess integration).
|
||||
|
||||
## Actual Go package layout
|
||||
|
||||
@@ -184,6 +184,12 @@ Current runtime flow (`internal/cli/run.go`):
|
||||
15. Optionally write `--report-json`; always write run-dir `report.json`.
|
||||
16. Apply work-dir retention.
|
||||
|
||||
Parity fixture status:
|
||||
- representative Python-parity fixture coverage exists under `internal/cli/testdata/parity`;
|
||||
- parity tests use fake structured LLM responses for deterministic behavior, including default full-pipeline shape assertions;
|
||||
- parity comparisons intentionally ignore nondeterministic metadata (timestamps, run IDs, temp paths, token usage) and remain strict for deterministic contract fields (transcript content, module order/instance naming, applied/skipped/rejected counts, and status).
|
||||
- intentional Python-vs-Go differences and open parity gaps are documented in `docs/python-parity.md`.
|
||||
|
||||
Important behavior details:
|
||||
- Glossary is validated and is used for explicit glossary/grammar/homophones/spoken_word module correction paths.
|
||||
- Default production CLI behavior now executes the full production module sequence unless `--modules` override is supplied.
|
||||
@@ -254,6 +260,7 @@ Current caveat:
|
||||
Current runtime boundary:
|
||||
- the default CLI runtime path (without explicit module selection) instantiates the full production module sequence.
|
||||
- LLM calls are exercised in production in both default full-pipeline runs and explicit `--modules` runs, and in tests when fake/injected clients are used.
|
||||
- normal `go test ./...` does not require real LLM credentials or Python dependencies.
|
||||
|
||||
`internal/framework/llm` also provides:
|
||||
- a bounded `Scheduler` for controlled concurrent LLM calls with reliable permit release;
|
||||
|
||||
78
docs/python-parity.md
Normal file
78
docs/python-parity.md
Normal file
@@ -0,0 +1,78 @@
|
||||
# Python vs Go Parity Notes
|
||||
|
||||
This document tracks Phase 17 parity-fixture coverage and differences between the original Python implementation and the Go rewrite.
|
||||
|
||||
## Scope
|
||||
|
||||
- Uses deterministic fixture-driven tests in `internal/cli/testdata/parity`.
|
||||
- Uses fake structured LLM responses for proposal generation and LLM validators.
|
||||
- Verifies functional contract fields (module order, instance naming, applied/skipped counts, report status, diagnostics presence).
|
||||
- Does not require real LLM credentials or Python dependencies during `go test ./...`.
|
||||
|
||||
## Intentional Differences
|
||||
|
||||
The following differences are expected and treated as intentional unless they break contract behavior:
|
||||
|
||||
1. JSON formatting and field ordering
|
||||
- Serialized JSON whitespace and object key order may differ.
|
||||
- Parity tests compare JSON semantically, not byte-for-byte.
|
||||
|
||||
2. Diagnostics path values
|
||||
- Absolute run-directory paths, run IDs, and temp directory roots differ by runtime and platform.
|
||||
- Parity checks assert artifact presence/shape, not exact absolute paths.
|
||||
|
||||
3. Time-variant metadata
|
||||
- Timestamps (`started_at`, `completed_at`) and generated run IDs are runtime-specific.
|
||||
- Parity checks ignore exact timestamp/run-id values.
|
||||
|
||||
4. Provider metadata/token accounting
|
||||
- Provider/token usage metadata may vary by adapter behavior and is not asserted as strict parity fields.
|
||||
|
||||
5. Internal adapter implementation details
|
||||
- Go uses its own structured LLM adapter implementation details while preserving the same high-level contract semantics.
|
||||
|
||||
## Current Fixture Coverage
|
||||
|
||||
Current parity fixtures cover:
|
||||
|
||||
- Transcript schema handling failure path.
|
||||
- Glossary schema handling failure path.
|
||||
- Default full module sequence shape:
|
||||
- `glossary_1`, `homophones`, `glossary_2`, `spoken_word`, `grammar`.
|
||||
- Mutable transcript handoff across default stages.
|
||||
- Module-specific behavior inside the default sequence:
|
||||
- glossary correction
|
||||
- homophone-style correction
|
||||
- spoken-word cleanup
|
||||
- grammar cleanup
|
||||
- Protected glossary-term guardrail behavior.
|
||||
- Deterministic validator rejection behavior.
|
||||
- LLM validator decision/rejection behavior.
|
||||
- Application-level skip behavior (`ambiguous_original_text`).
|
||||
- Mid-pipeline failure with partial progress preserved in reports.
|
||||
- Diagnostics artifact presence and secret-redaction checks.
|
||||
|
||||
## Open Parity Gaps (Not Intentional)
|
||||
|
||||
These are known Phase 17 expansion opportunities and should not be labeled as intentional compatibility differences:
|
||||
|
||||
1. Broader Python fixture import
|
||||
- The current Go parity fixtures are native fixture cases; they do not yet ingest all existing Python test fixtures directly.
|
||||
|
||||
2. Side-by-side runner command
|
||||
- No repository-standard Python+Go side-by-side parity command is required or enforced yet.
|
||||
|
||||
3. Wider transcript corpus
|
||||
- Current fixtures are representative but not exhaustive across all transcript/glossary edge combinations.
|
||||
|
||||
## How To Extend
|
||||
|
||||
1. Add a new `*.case.json` file under `internal/cli/testdata/parity`.
|
||||
2. Add referenced transcript/glossary/fake-LLM response files.
|
||||
3. Encode deterministic expectations in the case:
|
||||
- module order and instance names
|
||||
- transcript output
|
||||
- applied/skipped/rejected counts
|
||||
- report status and failure metadata
|
||||
- diagnostics artifact presence/redaction markers
|
||||
4. Run `go test ./...`.
|
||||
@@ -98,7 +98,6 @@ Implemented:
|
||||
- Generic JSON diagnostics primitives for LLM interactions (request metadata, request payload, response payload, optional error payload) with secret redaction.
|
||||
|
||||
Not yet implemented in runtime pipeline:
|
||||
- Python parity fixture suite and parity verification workflow.
|
||||
- Operational hardening beyond current Phase 16 runtime/reporting/diagnostics scope.
|
||||
- Rollout/Python retirement work.
|
||||
|
||||
@@ -238,7 +237,7 @@ Not implemented in Phase 8 (by design):
|
||||
|
||||
## Remaining work plan
|
||||
|
||||
Next recommended phase: **Phase 17 (Python parity fixture suite)**.
|
||||
Next recommended phase: **Phase 18 (operational hardening and subprocess integration)**.
|
||||
|
||||
## Phase 9: Structured LLM client and scheduler infrastructure
|
||||
|
||||
@@ -527,6 +526,8 @@ Intentionally deferred:
|
||||
|
||||
## Phase 17: Python parity fixture suite
|
||||
|
||||
Completed.
|
||||
|
||||
### Purpose
|
||||
|
||||
Establish confidence that the Go implementation matches the behavior and safety posture of the initial Python implementation.
|
||||
@@ -566,6 +567,35 @@ The repository has a durable test suite demonstrating that the Go implementation
|
||||
- Unintentional compatibility breaks are fixed.
|
||||
- `go test ./...` passes.
|
||||
|
||||
### Phase 17 completion status
|
||||
|
||||
Implemented:
|
||||
- Durable parity fixture harness in Go test path (`internal/cli/parity_test.go`).
|
||||
- Representative parity fixture corpus under `internal/cli/testdata/parity`.
|
||||
- Fake LLM proposal/validator fixtures driving deterministic parity tests.
|
||||
- Default full-pipeline parity coverage including sequence/order and repeated glossary naming:
|
||||
- `glossary_1`
|
||||
- `homophones`
|
||||
- `glossary_2`
|
||||
- `spoken_word`
|
||||
- `grammar`
|
||||
- Parity checks that are strict for deterministic contract fields:
|
||||
- transcript content
|
||||
- module order and instance names
|
||||
- applied/skipped/rejected counts
|
||||
- report status and failed-module metadata
|
||||
- Parity checks that ignore nondeterministic metadata fields:
|
||||
- timestamps
|
||||
- run IDs
|
||||
- temp/absolute paths
|
||||
- provider token usage details
|
||||
- Documentation of intentional Python-vs-Go differences and honest open parity gaps in `docs/python-parity.md`.
|
||||
- Normal `go test ./...` path remains independent of real LLM credentials and Python dependencies.
|
||||
|
||||
Intentionally deferred:
|
||||
- Phase 18 operational hardening and subprocess integration expansion.
|
||||
- Phase 19 rollout and Python retirement work.
|
||||
|
||||
## Phase 18: Operational hardening and subprocess integration
|
||||
|
||||
### Purpose
|
||||
|
||||
Reference in New Issue
Block a user