From 14ef59aaed74ec11a0e8e20d01e5b2c751165ee2 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 10 Aug 2026 15:04:05 +0000 Subject: [PATCH] Document test suite policy audit conclusions --- docs/roadmap/audit-findings.md | 328 +++++++++++++++++++++++++++++++-- 1 file changed, 315 insertions(+), 13 deletions(-) diff --git a/docs/roadmap/audit-findings.md b/docs/roadmap/audit-findings.md index 1e36d74..029ca32 100644 --- a/docs/roadmap/audit-findings.md +++ b/docs/roadmap/audit-findings.md @@ -36,8 +36,8 @@ and this section must record the new revision. Coverage ranged from 69.8% (`internal/manifest`) to 100% (`internal/logging`) among tested packages. `cmd/narratio` reported 0% because it has no tests. The -remaining package results ranged from 70.1% to 88.1%. Stage 12 owns the -risk-based interpretation; these numbers are inventory signals only. +remaining package results ranged from 70.1% to 88.1%. The Stage 12 risk-based +interpretation is recorded below; these numbers are diagnostic signals only. ### Code Graph Freshness And Structural Inventory @@ -109,7 +109,7 @@ These are prioritization signals, not findings: | 9: extraction | complete | Configuration, transcript resolution, fingerprinting, Notarius execution, receipt/index/lane validation, immutable promotion, manifest advertisement, catalog hydration, resume, and explicit analyze/publish consumption were traced as one slice. The exact focused command passed. Confirmed `COR-020`; added `DUP-007`, `SIM-002`, and `TST-009`, and refined `ARC-001`, `RSK-013`, and scenario 3. | | 10: analyze and dependencies | complete | All five source-policy families, seven built-in catalog entries, configured-artifact execution/reuse, dependency validation/order, previous-cache locality, lifecycle, and publish selection were traced as one slice. The exact focused command passed. Confirmed `COR-021` through `COR-024` and `RSK-015`; added `ARC-007`, `DUP-008`, `SIM-003`, `COM-005`, and `TST-010`; and refined `ARC-001`, `ARC-002`, `RSK-013`, and scenario 9. | | 11: maintainability | complete | Production graph metrics, change coupling, dead-code/static patterns, all structural candidates, efficiency workloads, comments, dependencies, and platform assumptions reviewed. Confirmed the narrow `ARC`, `DUP`, and `SIM` corrections recorded below, merged lifecycle wording and extraction proof candidates, rejected generic abstractions and micro-optimizations, added `SIM-004` and `COM-006`, and passed the full normal test suite and vet. | -| 12: test policy | not_started | Seeded by intended owners and baseline execution observations. | +| 12: test policy | complete | All 749 tests were inventoried by behavior owner and consequential risk; every prior `TST` candidate was classified, five suite-wide candidates were added, and coverage, doubles, helpers, determinism, offline behavior, runtime, fuzzing, and automation were assessed. The prescribed shuffled run exposed `TST-011`; the prescribed race run reproduced only `TST-001`. | | 13: synthesis | not_started | No final ranking or accepted-risk decisions yet. | ## Area Coverage And Ownership @@ -133,7 +133,7 @@ has been mapped but not behaviorally audited. | File operations | `internal/fileops`, `internal/pathsafe`, local artifact store | 3 (promotion vertical slice in 9) | reviewed | | External adapters and storage | `internal/adapters`, `internal/audio` | 7 | reviewed | | Shared models and diagnostics | `internal/artifactmodel`, `internal/contracts`, `internal/logging` | 7 (maintainability revisited in 11) | reviewed | -| Tests, examples, and automation | package test owners, `examples/`, `.woodpecker/` | 12 | assigned | +| Tests, examples, and automation | package test owners, `examples/`, `.woodpecker/` | 12 | reviewed | ## Package And Interface Ownership Map @@ -397,9 +397,10 @@ no-output metadata so the pipeline can continue to publish; it is durably reused until forced, closing analyze's behavior portion of `ARC-002` and leaving the wording gap in `COM-005`. -## Preliminary Risk-To-Test Matrix +## Intended Risk-To-Test Ownership Matrix -This matrix identifies intended owners only. It makes no sufficiency judgment. +This Stage 1 matrix identifies intended owners. The final Stage 12 sufficiency +assessment follows the accumulated behavior-pass observations below. | Architectural invariant or risk | Implementation owner | Intended test owner | | --- | --- | --- | @@ -469,6 +470,119 @@ Stage 10 test observations for this matrix: | Previous-session locality and guidance | Requirement collection covers enabled/disabled, deduplication, required-wins, and stable ordering. Prepare/restore/app tests cover planned requirements; analyze proves manifest/fallback local resolution and explicitly asserts no object-store call. | Selection is absent from the collector API (`COR-022`), while the guidance test asserts only a fragment of the malformed command and therefore preserves `COR-023`. | | Scriptorium request and output materialization | Stage tests inspect exact named input paths, vars, generated/reused metadata, render-debug, logs/configuration, and canonical materialization. Adapter tests assert deterministic flags and generated invocation configuration. | Input `artifact`/`path` fields have no consumer or request representation (`COR-024`); no stage-side link/non-regular/oversized result case covers analyze, extending `RSK-013`; and the broad execution/resolution shape remains `SIM-003`. | +## Test-Suite Policy Conclusions + +The graph inventory contains 904 functions in 95 test files, including 749 +`Test` functions and no fuzz tests or benchmarks. Review grouped those tests by +the policy owner they protect rather than by filename. Parsing and validation +are concentrated in `config`, `artifactpolicy`, manifest decoding, and adapter +protocol tests; domain and durable state in `manifest`, `artifacts`, and app +lifecycle tests; filesystem safety in `pathsafe`, `fileops`, and the local +store; adapter contracts in their owning packages; orchestration and CLI in +`app`; and representative assembled behavior in app/stage workflow tests. + +### Final risk-to-test sufficiency matrix + +| Consequential invariant or risk | Current protection | Proper owner and realistic protected defect | Missing modes or cross-layer overlap | Sufficiency conclusion | +| --- | --- | --- | --- | --- | +| Canonical order, planning, and lifecycle transitions | Registry/planner tables, manifest transition tests, and app first-run/skip/force/failure/rerun cases | App lifecycle tests protect durable state-machine outcomes; manifest tests own field mutation | Run-store failures and resume-validation terminalization cannot be injected; several assembled tests repeat per-stage metadata | Strong ordinary protection; add the narrow persistence seam in `TST-002` and consolidate overlap under `TST-015` | +| Session/run identity and immutable invocation audit | Per-invocation runner tests and typed manifest load/save tests | App owns identity synchronization; manifest owns serialization | Derived identity conflicts and disagreement states remain visible only through `COR-001`/`TST-002` | Insufficient at the cross-manifest boundary; `TST-002` is the intended addition | +| Strict configuration, defaults, identity, and examples | Extensive strict-load/default/validation tables plus maintained-example loading | Config tests protect rejected language and normalized values; one example test protects assembly | A 949-line loader/validator table repeats complete YAML and broad default assertions already owned elsewhere | Behavior breadth is strong; restructure for clearer ownership under `TST-014` without reducing contract cases | +| Confined identities, paths, locks, and atomic persistence | `pathsafe`, `fileops`, artifact, cleanup, and local-store tests | Low-level owners protect no-escape/no-follow/atomicity; one app or stage case protects composition | Unsafe identity, destination-ancestor links, directory sync, stale/release lock faults, and concurrent runner cases are absent; duplicating each at every caller would be wasteful | Insufficient for confirmed safety roots; add one owner-level case per root plus representative composition under `TST-003` | +| Remote publish commit and cleanup recovery | Publish source/order/failure tests and cleanup path/effect tests | Publish owns pointer-last snapshot visibility; cleanup owns durable retry evidence | Current doubles record calls but do not model prior readable versions, accepted-with-error, barriers, or cleanup reload/retry | Insufficient for recovery/idempotency/concurrency; stateful fake and focused cases in `TST-004` | +| Restore committed-state authority and partial replacement | Restore plan/execution/workflow tables, conflict/force, malformed manifest, cache tests | Restore package owns snapshot selection, conflict policy, manifest-last commit, and resumable failure | No generation change, committed-vs-stray scope, partial forced overwrite, plan/lock race, same-size replacement, or foreign-path case | Insufficient at durable transition boundaries; package-level stateful cases in `TST-005` | +| Composition has one resolved configuration authority | Production assembly is exercised; injected environments generally leave `Env.Config` nil | App composition tests protect production-shaped dependency injection | No test intentionally diverges the two configs, so the impossible split fixture remains allowed | Insufficient seam fidelity; one authority test in `TST-006` | +| Adapter protocol, cancellation, resource, output, and secret boundaries | Strong argument/schema, retry/status, direct-child timeout, ordinary output, S3 one-page, and override-secret tests | Each adapter owns protocol behavior; shared subprocess/file acquisition owns mechanical limits and redaction | Descendant kill, inherited/raw secret leakage, scheme restriction, streaming cancellation, close observation, pagination progress, and non-regular/oversized outputs are absent | Insufficient at adversarial boundaries; targeted additions in `TST-007`, not duplicated protocol matrices | +| Bounded transcription concurrency and cancellation | Concurrency bound/order and ordinary adapter-error tests | Transcribe owns dispatch/cancel completion; WhisperX fake must honor concurrent interface use | The fake races and no barrier-controlled mid-dispatch cancellation exists | Insufficient; `TST-001` restores race signal and `TST-008` protects cancellation | +| Ordinary-stage input transitions and run-local output authority | Broad prepare and transcript-stage behavior suites | Prepare owns source replacement/deduplication; stage/shared acquisition owns requested path and multi-output commit behavior | Stale previous state, duplicate audio, alternate adapter path, and partial multi-output materialization are absent; schema tables already overlap heavily | Add only the transition/authority cases in `TST-008`; current schema and protocol coverage is otherwise sufficient | +| Extraction identity, immutable promotion, and consumer evidence | Strong adapter validation, lifecycle/reuse, catalog hydration, explicit-consumer, and low-level promotion tests | Extract owns direct-input fingerprint and lifecycle; artifact/file owners protect evidence and promotion | Direct input mutation and one assembled unsafe-root case are absent. Post-install orphan residue is not advertised authority and would require private choreography | Add the two marginal cases in `TST-009`; reject a dedicated orphan-residue test and reuse `TST-002`/`TST-003` owners | +| Analyze effective selection, optional inputs, dependencies, and passthrough contract | Strong source-family, catalog, order/cycle, reuse, local-previous, and publish-filter tests | Analyze/app composition owns effective selection; artifact owners retain source parsing | Optional built-ins, selected-disabled prerequisites, simultaneous errors, actionable guidance, and accepted passthrough fields are not protected | Insufficient at cross-owner seams; focused additions in `TST-010`, without repeating all source spellings | +| Artifact/source identifiers and remote/local mapping reject hostile structured input | Deterministic tables cover representative traversal, source families, and mappings | `pathsafe` and `artifactpolicy` are pure security-sensitive owners; restore mapping is a secondary seed source | No fuzz target probes arbitrary separators, normalization idempotence, round-trip mapping, or no-escape properties | Add focused seeded property fuzzing under `TST-013`; generic YAML/JSON parser fuzzing has lower marginal value | +| CLI parsing and operator rendering remain compatible and actionable | Command tables cover valid/invalid flags, dispatch, status/validate rendering, and semantic error fragments | App command tests own supported invocation language and actionable output | A few broad workflow cases repeat downstream policy, but no consequential CLI gap was established beyond `COR-023` guidance | Sufficient once `TST-010` asserts repaired guidance; retain semantic fragments rather than exact full prose | +| Representative assembled workflows prove boundary composition | App/stage tests run real filesystem/config/manifest collaborators with adapter fakes | App owns a small number of end-to-end success/failure cases; focused packages own details | Stage metadata checklists and a six-adapter generic failure matrix repeat focused suites and generic runner behavior | Overprotected and costly in places; consolidate under `TST-015`, retaining one assembled success and one generic failure | +| Default suite is deterministic, isolated, offline, and credential-free | Normal full suite passes in about 3.5 seconds using temp dirs, loopback HTTP, and test-binary subprocesses | Each test owns cleanup of process-global state; repository execution owns the aggregate signal | A config-secret test leaves environment values behind and fails under repeated execution; no tests use `t.Parallel` | Offline/credential isolation is sufficient; determinism is not until `TST-011` is fixed | +| Race diagnostics remain trustworthy | Baseline and shuffled race runs exercise all packages | Concurrent consumers and their doubles jointly own race-safe fixtures | WhisperX fake request capture races, causing the only observed race failure | Insufficient until `TST-001`; no second production race was observed | +| Repository automation enforces supported validation | Tag release automation cross-builds Linux, macOS, and Windows binaries | Normal change automation should protect test/vet/build compatibility before release | No PR/push test, vet, or build job exists; tag publishing is not tied to a validated revision | Insufficient; record the proportional enforcement decision in `TST-012` | + +### Doubles, helpers, redundancy, and brittleness + +Most tests follow the preferred collaborator order. Pure validation uses real +values; filesystem and workflow tests use real temporary directories and local +manifests; HTTP tests use loopback servers; and protocol subprocess tests run +the current test binary. Those subprocess argument assertions are interaction +contracts, not mock choreography, and should remain. Storage and publish are +the important exception: call-recording stubs cannot express version +visibility, accepted-with-error outcomes, pagination progress, or concurrent +commit barriers. `TST-004` and `TST-005` therefore call for one shared stateful +object-store fake. The WhisperX request-capture fake is the only double proven +unsafe for its actual concurrent consumer (`TST-001`); sequential fakes do not +need blanket synchronization. + +The largest test, `config.TestLoadAndValidate`, spans roughly 949 lines and +combines strict YAML loading, normalization/default checks, and validation in +one repeated full-document table. It matches semantic error fragments rather +than exact full messages, but its fixture duplication obscures which layer +failed and raises change cost; `TST-014` owns a split into strict-load cases, +constructed validator tables, and a small assembly sample. By contrast, the +large artifact requirement table and protocol helper-process tests express +meaningful contract cases and should remain table-driven. Existing helpers are +mostly domain-specific setup; no generic fixture framework is justified. + +The broad stage metadata checklist and six-adapter assembled failure matrix +repeat focused per-stage protocol/behavior suites and generic runner result +mapping. `TST-015` names the stronger protection that must remain before those +rows are deleted: focused owners plus one representative assembled success and +one representative terminal failure. No oversized snapshots or golden files +were found. Exact full `err.Error()` equality is rare; semantic fragments, +`errors.Is`, typed errors, and externally visible protocol arguments dominate, +so no repository-wide error-assertion rewrite is warranted. + +### Determinism, coverage, fuzzing, and automation + +`go test -count=1 -cover ./...` passed all packages in 3.49 seconds. Package +coverage ranged from 69.8% in `internal/manifest` to 100% in +`internal/logging`, with `cmd/narratio` at 0% because it has no test files. +These percentages are not a quality score: manifest's lowest result aligns +with the uninjectable save-failure branch in `TST-002`, while logging's 100% +does not justify more tests and the placeholder notification path remains an +architectural decision in `ARC-004`, not a coverage target. + +The default suite uses no live services, paid APIs, ambient credentials, or +fixed external ports. It uses temporary paths, loopback servers, and helper +subprocesses, and no test calls `t.Parallel`; normal runtime does not justify +parallelizing process-global fixtures. A few bounded sleeps exercise actual +time/process behavior, but the observed order defect is instead deterministic: +`go test -shuffle=on -count=3 ./...` failed in 4.71 seconds with seed +`1786373771816345415` because +`TestLoadSecretsFromConfigLoadsValidFiles` leaves two secret environment +variables set. The isolated command +`go test -shuffle=1786373771816345415 -count=3 -run '^TestLoadSecretsFromConfigLoadsValidFiles$' ./internal/app` +reproduced failures on repetitions two and three. `TST-011` owns restoration of +the prior environment state. + +`go test -race -shuffle=on -count=1 ./...` failed in 54.21 seconds with seed +`1786373816980315094`; the only reported race was the known WhisperX fake in +`TST-001`, now reached by `TestTranscribeStageConcurrencyBounded`. Every other +package passed. Barrier-controlled cancellation/concurrency tests should +replace additional timing dependence where `TST-008` touches that path, but no +general flakiness conclusion follows from the bounded evidence. + +There are no fuzz tests. The highest-value additions are seeded property tests +for confined path normalization/join and artifact source identifiers, with +properties such as no panic, no root escape, stable normalization, and valid +remote/local round trips (`TST-013`). Existing YAML/JSON tables already protect +Narratio's own strict schemas; indiscriminate fuzzing of standard-library +decoders or every manifest/config wrapper would add less marginal value. + +`.woodpecker/release.yml` runs only for tags and cross-builds release binaries +with Go 1.25 before publishing them. No automation runs the repository's local +test, vet, or build requirements on ordinary changes. `TST-012` recommends a +normal validation workflow and requires a release to consume or repeat the +validated revision. The 54-second race suite belongs in automation after +`TST-001` is fixed, at a frequency chosen against its cost; repeated shuffled +runs are suitable for scheduled/audit diagnostics rather than necessarily +every change. + ## Path, Artifact, Filesystem, And Lock Conclusions ### Canonical ownership and normalization @@ -2890,11 +3004,12 @@ credible workload large enough to justify a performance finding. ## Candidate Register -The `TST` entries remain candidates for Stage 12. Structural entries retain -their original locations for traceability, but their Stage 11 classifications -are final for this investigative pass: confirmed recommendations remain for -Stage 13 prioritization, merged entries defer to the named stronger root, and -rejected entries require new evidence before reopening. +The `TST` entries now carry their Stage 12 classifications. Structural entries +retain their original locations for traceability, but their Stage 11 +classifications are also final for this investigative pass: confirmed +recommendations remain for Stage 13 prioritization, merged entries defer to the +named stronger root, and rejected entries require new evidence before +reopening. ### `ARC-001`: `IODecl` is not a complete or consistently classified stage contract @@ -3082,6 +3197,9 @@ rejected entries require new evidence before reopening. valid. - Confirmation owner: Stage 12 should classify suite impact and the smallest durable fake fix. Do not change the fake during this investigative stage. +- Stage 12 classification: confirmed. Protect request capture with the fake's + own synchronization and expose a safe snapshot accessor. This restores the + full race suite without changing legitimate production concurrency. ### `TST-002`: runner tests cannot exercise invocation-manifest save failures @@ -3097,6 +3215,10 @@ rejected entries require new evidence before reopening. - Confirmation owner: Stage 12 should decide the smallest persistence seam and representative boundary cases; avoid exhaustive choreography tests for every mechanically identical save call. +- Stage 12 classification: confirmed. Inject the run-store/terminal-operation + boundary and retain one session-first terminal save failure plus one + resume-validation terminalization case. Repeating every `SaveRun` call order + would test private choreography rather than a distinct defect. ### `DUP-001`: session save duplicates the shared atomic JSON writer @@ -3302,6 +3424,12 @@ rejected entries require new evidence before reopening. - Confirmation owner: Stage 12 should select one narrow behavior-level case per distinct invariant and reuse shared low-level tests across callers. Avoid duplicating every path spelling or persistence call sequence. +- Stage 12 classification: confirmed and narrowed. Put unsafe identity and + symlink-ancestor cases at their low-level policy owners, retain one + destructive and one writer composition case, add a durability interaction + case at the injectable filesystem capability, and add stale/release failure + plus one concurrent assembled runner case. Do not repeat the same ancestry + table at every caller. ### `TST-004`: publish protocol tests do not preserve prior current state or exercise recovery @@ -3322,6 +3450,11 @@ rejected entries require new evidence before reopening. with barriers/version semantics and one behavior-level case per distinct invariant. Prefer extending current order/cleanup fixtures over duplicating all source-family tables, which already have good focused coverage. +- Stage 12 classification: confirmed. One versioned object-store fake should + prove prior-pair readability across the manifest-before-pointer window and + failure, accepted-with-error ambiguity, retry, cleanup evidence durability, + and concurrent mutation. Existing source-family tables remain the stronger + protection for selection and payload composition. ### `TST-005`: restore tests omit committed-snapshot and partial-transition invariants @@ -3345,6 +3478,10 @@ rejected entries require new evidence before reopening. consolidate caller rendering matrices. Avoid a mock for every mechanically identical download boundary; retain the existing manifest-invalid case as the representative pre-rename validation failure. +- Stage 12 classification: confirmed. Reuse the stateful store and barriers + from `TST-004` for one package-level case per committed-scope, generation, + conflict/partial-transition, identity, and recovery root. Keep the existing + manifest-invalid case; do not mock every download. ### `TST-006`: injected runner environments can split configuration authority @@ -3362,6 +3499,10 @@ rejected entries require new evidence before reopening. - Confirmation owner: Stage 12 should inventory intentional divergent fixtures and either reject differing configs, always rebind to the resolved config, or replace the dual input with a constructor that makes ownership singular. +- Stage 12 classification: confirmed. No intentional divergent fixture was + found. One app composition test should prove that injected collaborators + cannot introduce a second configuration authority; production should always + bind the environment to the resolved configuration. ### `TST-007`: adapter tests omit adversarial liveness and output-trust boundaries @@ -3385,6 +3526,11 @@ rejected entries require new evidence before reopening. retain protocol-specific argument/schema assertions. It should classify automatic fake materialization by the stage risks it protects rather than mechanically rewriting every fake. +- Stage 12 classification: confirmed. Add one focused case per process-tree, + inherited/raw secret, URL scheme, streaming cancellation, bounded regular + output, response closure, and pagination-progress root. Retain exact protocol + argument/schema assertions; they are observable contracts, not brittle mock + choreography. ### `TST-008`: ordinary-stage tests omit transition, cancellation, and output-authority boundaries @@ -3406,6 +3552,10 @@ rejected entries require new evidence before reopening. transcribe cancellation, and one cross-adapter output-authority contract. Reuse shared safe-output acquisition tests from `TST-007`; do not duplicate every schema error or stage fixture already covered. +- Stage 12 classification: confirmed. Add exactly those transition, + duplicate-input, barrier-cancellation, requested-path, and representative + multi-output partial-materialization cases. Safe output shape/size stays at + the shared owner from `TST-007`; existing schema tables remain sufficient. ### `TST-009`: extraction tests omit direct-input identity and assembled late-failure boundaries @@ -3431,6 +3581,12 @@ rejected entries require new evidence before reopening. downstream invalidation and catalog fixtures rather than creating a broad new end-to-end framework; coordinate manifest-persistence faults with `TST-002` and destination ancestry with `TST-003`/`COR-003`. +- Stage 12 classification: confirmed with one rejection. Add the direct-input + lifecycle regression and one assembled unsafe-root case; keep lane size at + the shared bounded reader. Reject a dedicated post-install orphan-residue + assertion because an unadvertised orphan is not authority and exposing the + point would test private choreography. Persistence and ancestry remain owned + by `TST-002` and `TST-003`. ### `TST-010`: analyze tests omit effective-selection and optional built-in boundaries @@ -3456,6 +3612,92 @@ rejected entries require new evidence before reopening. rejection for the passthrough fields after their contract is chosen. Reuse existing fixtures for publish/source success and do not retest every artifact-policy spelling. +- Stage 12 classification: confirmed. Add the optional-built-in table, one + selected-disabled assembled workflow, deterministic simultaneous-error + ordering, semantic actionable guidance, and the chosen passthrough behavior. + Existing source-family and publish tables remain the stronger protection and + should not be repeated end to end. + +### `TST-011`: filesystem-secret test leaks process environment across repetitions + +- Category: test-suite determinism defect. +- Stage 12 classification: confirmed by isolated reproduction. +- Evidence: `TestLoadSecretsFromConfigLoadsValidFiles` invokes the production + loader, which sets `NARRATIO_TEST_SECRET_A` and + `NARRATIO_TEST_SECRET_B`, but the test does not restore their prior state. + `go test -shuffle=on -count=3 ./...` failed with seed + `1786373771816345415`; an isolated same-seed, three-count invocation failed + on its second and third repetitions because the variables were already + present and correctly reported as preserved rather than loaded. +- Realistic defect and marginal value: leaked secrets make the package depend + on repetition/order and can conceal whether later cases loaded or preserved + values. Restore the exact prior environment state in cleanup, including + unset-versus-empty semantics. One cleanup repair protects the behavior; no + production change or broad environment fixture is warranted. + +### `TST-012`: automation does not enforce repository validation before release + +- Category: test-suite execution/automation risk. +- Stage 12 classification: confirmed. +- Evidence: `.woodpecker/release.yml` is tag-only and cross-builds/publishes + binaries. No repository automation runs `go test ./...`, `go vet ./...`, or a + normal build for pushes or proposed changes, and release publication is not + tied to a revision that passed those checks. +- Realistic defect and marginal value: a change or tagged revision can ship + despite breaking tests, static analysis, or a supported build. Add ordinary + test/vet/build validation and make release consume or repeat the validated + revision. Add race execution after `TST-001` is repaired at a frequency + justified by its roughly 54-second cost; reserve repeated shuffle runs for a + scheduled/audit diagnostic unless CI capacity supports them. + +### `TST-013`: security-sensitive path and source parsers have no property fuzzing + +- Category: test-suite sufficiency opportunity. +- Stage 12 classification: confirmed narrow addition. +- Evidence: no fuzz target exists. Deterministic tables cover known separators, + traversal spellings, source families, and remote/local mappings, but the + `pathsafe` and `artifactpolicy` boundaries accept attacker- or + configuration-controlled strings whose combinations are larger than those + tables. +- Realistic defect and marginal value: an unanticipated separator, + normalization sequence, or source token can escape a root, classify + inconsistently, or break a round trip. Seed the existing tables and assert no + panic, no root escape, normalization stability, and valid mapping round + trips. Do not blanket-fuzz standard YAML/JSON decoders or every typed manifest + wrapper; their Narratio-specific schema cases already have stronger tables. + +### `TST-014`: configuration tests conflate loader, defaults, and validator ownership + +- Category: test-suite redundancy and maintainability issue. +- Stage 12 classification: confirmed consolidation. +- Evidence: `config.TestLoadAndValidate` is roughly 949 lines with a large table + of repeated complete YAML documents followed by broad normalized/default + assertions and validator checks. It already uses semantic error substrings, + so exact prose is not the defect; fixture breadth and mixed ownership are. +- Realistic defect and marginal value: changing an unrelated required/default + field forces many validation fixtures to change and a failure does not + identify whether strict decoding, defaulting, or domain validation owns the + contract. Retain small strict-load cases, construct typed configurations for + validator tables, and keep one representative load/default/validate assembly + case. Those focused owners are the stronger remaining protection; do not + introduce a generic fixture framework. + +### `TST-015`: assembled stage matrices duplicate focused behavior owners + +- Category: test-suite redundancy and brittleness issue. +- Stage 12 classification: confirmed consolidation. +- Evidence: `TestStagesReturnExpectedMetadata` is a broad per-stage metadata + checklist that overlaps focused stage suites and incidental metadata, while + `TestAdapterBackedStageFailureMarksManifestFailed` repeats six adapters to + prove the runner's generic terminal-failure mapping. Similar broad success + assertions exist in `TestExecuteStagesPlaceholderSuccessUpdatesManifest`. +- Realistic defect and marginal value: a harmless metadata/default change + causes several layers to fail while real stage defects remain better + diagnosed by focused owners. Retain focused stage/adapter protocol and + behavior tests, one representative assembled success proving composition and + durable manifests, and one representative adapter failure proving generic + runner terminalization. Delete the per-stage checklist and repeated generic + failure rows only after those stronger protections are explicit. ### `SIM-001`: runner terminalization and persistence ordering lack a narrow owner @@ -3800,8 +4042,9 @@ rejected entries require new evidence before reopening. digest that covers profiles, prompts, modules, references, executable build, and other transitive inputs, or is documented operator-forced invalidation the intended long-term contract? -- Which native CI runner limitations explain the absence of validation jobs in - the tag-only release workflow? Stage 12 owns the automation conclusion. +- Which automation service and cadence should carry the validation required by + `TST-012`, particularly the roughly 54-second race run, and should tagged + release builds consume an already validated revision or repeat validation? No accepted risks or final audit conclusions are recorded yet. @@ -4275,3 +4518,62 @@ No accepted risks or final audit conclusions are recorded yet. test-policy audit. Stage 13 owns prioritization, dependency ordering, and accepted-risk decisions; it should treat the merged/rejected structural dispositions above as resolved rather than reopening them from metric shape. + +### Stage 12 + +- Entry revision: `f387222` (`Document maintainability audit conclusions`). + Commits since the pinned audit revision modify audit documentation only, so + implementation/test evidence remains pinned to the identity recorded above. +- Policy and scope reviewed: development guide, exact Stage 12 sequence, every + repository policy document, the complete risk-to-test ledger and prior + behavior-pass observations, all ten accumulated `TST` candidates, and the + repository's test/example/automation inventory. +- Graph/source evidence: 904 test-file functions, 749 `Test` functions, 95 test + files, no fuzz tests, no benchmarks, no `t.Parallel` use, test-size and + complexity hotspots, process-global environment/directory mutation, + time-dependent cases, test doubles, helper subprocesses, loopback HTTP, and + assembled workflow overlap. Exact source inspection followed the graph for + large/mixed-owner tests, secret loading, fakes, and release automation. +- Coverage diagnostic: `go test -count=1 -cover ./...` passed all packages in + 3.49 s. Tested-package coverage ranged from 69.8% to 100%; `cmd/narratio` + reported 0% because it has no tests. Percentages were used only to revisit + weak consequential branches and high-coverage duplication. Manifest's low + end aligns with `TST-002`; logging's 100% and the notification placeholder do + not justify percentage-driven additions. +- Determinism validation: `go test -shuffle=on -count=3 ./...` failed in 4.71 s + with seed `1786373771816345415` because + `TestLoadSecretsFromConfigLoadsValidFiles` leaked two environment values. The + isolated same-seed, same-test, three-count command reproduced failure on + repetitions two and three, confirming `TST-011` rather than random flakiness. +- Race validation: `go test -race -shuffle=on -count=1 ./...` failed in 54.21 s + with seed `1786373816980315094`. The only race was the already recorded + WhisperX fake request-slice mutation in `TST-001`, reached by the bounded + concurrent transcribe test; every other package passed and no second + production race was observed. +- Final documentation-only verification: `go test -count=1 ./...` passed all + packages in 3.50 s and `go vet ./...` passed in 0.78 s. `git diff --check` + reported no whitespace errors. +- Suite conclusions: normal execution is fast, offline, credential-free, and + based on temp files, loopback services, or the current test binary. Current + focused tests strongly protect ordinary parsing, protocol, lifecycle, and + source-family behavior. Confirmed additions target durable disagreement, + destructive recovery, adversarial output/resource boundaries, cancellation, + effective selection, and direct input identity. Stateful object-store + behavior is preferable to additional call-recording mocks; protocol argument + assertions remain contractual. +- Candidate disposition: confirmed `TST-001` through `TST-008` and `TST-010`; + confirmed `TST-009` after rejecting its low-value orphan-residue subcase; and + added confirmed `TST-011` through `TST-015` for environment isolation, + automation, focused fuzzing, configuration-test ownership, and assembled-test + consolidation. Every proposed addition names its realistic defect and owner; + every consolidation names the focused/representative protection that remains. +- Explicit no-finding conclusions: no live network, paid-service, ambient + credential, fixed-port, oversized snapshot, golden-file, pervasive exact- + error, generic fixture-framework, blanket fake-synchronization, or general + parallel-test opportunity was established. Existing YAML/JSON tables do not + justify indiscriminate parser fuzzing, and bounded evidence did not establish + flakiness beyond the reproducible environment leak and known fake race. +- Later-stage assignment: Stage 13 owns only deduplication, ranking, + remediation ordering, accepted-risk decisions, and final audit closeout. It + should not reopen the risk-based test ownership and marginal-value decisions + recorded here without new evidence.