Complete pre-1.0 release hardening
This commit is contained in:
@@ -100,6 +100,7 @@ internal/framework/proposals/
|
||||
apply.go
|
||||
|
||||
internal/framework/runner/
|
||||
observability.go
|
||||
runner.go
|
||||
|
||||
internal/framework/proposal_generation/
|
||||
@@ -149,6 +150,18 @@ internal/framework/llm/
|
||||
scheduler.go
|
||||
effective_config.go
|
||||
diagnostics.go
|
||||
|
||||
internal/framework/responseschema/
|
||||
registry.go
|
||||
registry_test.go
|
||||
|
||||
internal/cli/
|
||||
review_artifacts.go
|
||||
parity_test.go
|
||||
release_fixtures_test.go
|
||||
testdata/
|
||||
parity/
|
||||
release/
|
||||
```
|
||||
|
||||
## Current CLI behavior
|
||||
@@ -771,6 +784,7 @@ Implemented tests currently cover:
|
||||
- default full-pipeline runtime shape and ordering (`internal/cli/run_test.go`, `cmd/audita/main_integration_test.go`, `internal/cli/parity_test.go`)
|
||||
- subprocess operational hardening behavior including large-input, failure-mode, timeout/cancellation, backend-failure, and partial-progress paths (`cmd/audita/main_integration_test.go`)
|
||||
- report/diagnostics redaction and artifact-shape behavior across success and failure paths (`internal/cli/run_test.go`, `cmd/audita/main_integration_test.go`)
|
||||
- curated release-fixture and idempotence-oriented readiness checks using fake structured LLM responses (`internal/cli/release_fixtures_test.go`, `internal/cli/testdata/release`)
|
||||
|
||||
## Operational hardening status
|
||||
The runtime now includes hardened subprocess behavior for parent-process callers:
|
||||
|
||||
@@ -99,6 +99,9 @@ diagnostics:
|
||||
retention: auto
|
||||
```
|
||||
|
||||
`context.description` provides background-only transcript context for prompts.
|
||||
If both config and CLI provide a description, `--transcript-description` takes precedence.
|
||||
|
||||
`output.schema` supports the built-in output schema registry values:
|
||||
- `bare-segments` (default)
|
||||
- `audita-v1`
|
||||
|
||||
122
docs/release-checklist.md
Normal file
122
docs/release-checklist.md
Normal file
@@ -0,0 +1,122 @@
|
||||
# Audita Release Checklist
|
||||
|
||||
Use this checklist before cutting a pre-1.0 or 1.0 release candidate.
|
||||
|
||||
## Core test pass
|
||||
|
||||
- Run:
|
||||
- `go test ./...`
|
||||
- Confirm tests pass without live LLM credentials and without Python dependencies.
|
||||
|
||||
## Config validation and precedence
|
||||
|
||||
- Validate a representative config:
|
||||
- `audita config validate --config <path>`
|
||||
- Inspect redacted effective config:
|
||||
- `audita config print-effective --config <path>`
|
||||
- Confirm precedence behavior:
|
||||
- defaults -> file config -> environment -> CLI.
|
||||
- Confirm missing `/etc/audita/config.yml` is non-fatal when `--config`/`AUDITA_CONFIG` are unset.
|
||||
|
||||
## Output schema checks
|
||||
|
||||
- Verify default output schema remains `bare-segments`.
|
||||
- Verify `--output-schema audita-v1` emits object payload with `schema` and `version`.
|
||||
- Verify unknown schema (for example `seriatim-intermediate`) fails clearly.
|
||||
|
||||
## Subprocess contract checks
|
||||
|
||||
- With `--output`, verify stdout is empty on success.
|
||||
- Without `--output`, verify stdout contains transcript JSON only.
|
||||
- Verify `--report-json` writes file output and does not write report JSON to stdout.
|
||||
- Verify failure stderr remains human-readable and includes diagnostics path when available.
|
||||
- Verify nonzero exit on failures.
|
||||
|
||||
## Structured LLM checks
|
||||
|
||||
- Verify runtime uses the Audita-owned OpenAI-compatible adapter.
|
||||
- Verify structured response schemas are attached via `response_format.type=json_schema`.
|
||||
- Verify diagnostics metadata includes structured schema `id/version/name/sha256`.
|
||||
- Verify provider output is still locally decoded/validated before use.
|
||||
|
||||
## Report and diagnostics schema checks
|
||||
|
||||
- Verify report metadata fields:
|
||||
- `report_schema_name`
|
||||
- `report_schema_version`
|
||||
- `output_schema`
|
||||
- `config_version` when file config is used.
|
||||
- Verify diagnostics artifact references exist in reports:
|
||||
- transcript/normalization/chunking/invocation/effective-config artifacts
|
||||
- utilization diagnostics artifact
|
||||
- correction ledger artifact
|
||||
- error log on failures.
|
||||
|
||||
## Redaction checks
|
||||
|
||||
- Verify secrets are redacted from:
|
||||
- `effective-config.json`
|
||||
- run-dir and `--report-json` reports
|
||||
- LLM request/response/error diagnostics payloads.
|
||||
- Verify no API keys/bearer tokens leak into fixtures or outputs.
|
||||
|
||||
## Prompt and validator metadata checks
|
||||
|
||||
- Verify prompt metadata appears in LLM request metadata diagnostics:
|
||||
- `prompt_id`, `prompt_version`, `prompt_source`, `embedded_path`, `sha256`.
|
||||
- Verify stable validator keys appear in report decisions/rejections.
|
||||
- Verify built-in validator chains resolve and execute for default and explicit module runs.
|
||||
|
||||
## Utilization diagnostics checks
|
||||
|
||||
- Verify `utilization-diagnostics.json` exists on successful runs.
|
||||
- Verify partial utilization artifact behavior on controlled failure paths.
|
||||
- Verify utilization fields are structurally present and nonnegative:
|
||||
- effective concurrency
|
||||
- run timing
|
||||
- module timing summaries
|
||||
- per-validator timing summaries.
|
||||
|
||||
## Correction ledger checks
|
||||
|
||||
- Verify `correction-ledger.json` exists on successful runs.
|
||||
- Verify report references ledger artifact path.
|
||||
- Verify ledger dispositions include applied/rejected and skipped/failed where exercised.
|
||||
- Verify validator rejection and proposal-application skip remain distinct.
|
||||
|
||||
## Pipeline behavior checks
|
||||
|
||||
- Verify default full pipeline run remains:
|
||||
- `glossary`, `homophones`, `glossary`, `spoken_word`, `grammar`
|
||||
- with deterministic repeated instance naming (`glossary_1`, `glossary_2`).
|
||||
- Verify explicit module runs (`--modules`) still work.
|
||||
|
||||
## Failure and cancellation checks
|
||||
|
||||
- Verify controlled failure paths retain diagnostics and produce best-effort failure reports.
|
||||
- Verify timeout/cancellation paths exit nonzero, do not hang, and retain failure diagnostics when initialized.
|
||||
|
||||
## Release fixture/idempotence checks
|
||||
|
||||
- Run release fixtures (`internal/cli/testdata/release`) through `go test ./...`.
|
||||
- Confirm fixture checks cover:
|
||||
- must-apply and must-not-apply expectations
|
||||
- protected-term survival
|
||||
- report and diagnostics contracts
|
||||
- output-schema checks
|
||||
- prompt/schema metadata diagnostics
|
||||
- utilization/ledger artifacts
|
||||
- idempotence-oriented second pass no-op behavior with deterministic fake responses.
|
||||
|
||||
## Deferred-feature guardrail
|
||||
|
||||
- Confirm release docs do not claim support for deferred items:
|
||||
- filesystem prompt overrides
|
||||
- user-configurable validator chains
|
||||
- arbitrary user-supplied output schemas
|
||||
- resume/start-at/stop-after execution
|
||||
- diff/check/propose-only modes
|
||||
- generated transcript descriptions enabled by default
|
||||
- interactive review UI
|
||||
- UI/server wrapper
|
||||
- provider benchmarking harness.
|
||||
@@ -897,6 +897,26 @@ This ledger should not replace existing reports. It should provide a flattened r
|
||||
|
||||
Finish 1.0 by turning the new architecture into a documented, tested, stable release candidate.
|
||||
|
||||
## Implementation status (2026-05-13)
|
||||
|
||||
This workstream is now implemented:
|
||||
- architecture, contract, configuration, output-schema, structured-LLM, validator, prompt, diagnostics, and subprocess docs are aligned to the current runtime;
|
||||
- a practical release checklist document exists (`docs/release-checklist.md`);
|
||||
- curated release fixtures exist under `internal/cli/testdata/release` and run under `go test ./...` using fake structured LLM responses only;
|
||||
- release fixture coverage includes:
|
||||
- must-apply and must-not-apply expectations;
|
||||
- protected-term survival;
|
||||
- default module sequence and stable instance naming;
|
||||
- report schema metadata;
|
||||
- diagnostics artifact references;
|
||||
- output schema checks (`bare-segments`, `audita-v1`);
|
||||
- prompt/structured-schema diagnostics metadata presence;
|
||||
- stable validator keys;
|
||||
- utilization diagnostics and correction-ledger artifacts;
|
||||
- idempotence-oriented second-pass no-op expectations with deterministic fake responses.
|
||||
|
||||
Deferred work remains deferred (no filesystem prompt overrides, no user-configurable validator chains, no arbitrary user-supplied output schemas, no resume/start-at/stop-after execution, no diff/check/propose-only modes, no generated transcript descriptions enabled by default, no interactive review UI, no UI/server wrapper, and no provider-specific benchmarking harness).
|
||||
|
||||
## Documentation updates
|
||||
|
||||
Update or add:
|
||||
|
||||
@@ -21,6 +21,12 @@ Recommended additions:
|
||||
- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs explicit LLM throughput controls.
|
||||
- `--modules ...` only when intentionally overriding the default full sequence.
|
||||
|
||||
For config-driven orchestration, validate config files in CI/preflight:
|
||||
|
||||
```sh
|
||||
audita config validate --config <path>
|
||||
```
|
||||
|
||||
## Stdout behavior
|
||||
|
||||
- With `--output`: stdout is expected to be empty on success.
|
||||
|
||||
Reference in New Issue
Block a user