diff --git a/docs/roadmap/audita-debug.md b/docs/roadmap/audita-debug.md new file mode 100644 index 0000000..132c7a0 --- /dev/null +++ b/docs/roadmap/audita-debug.md @@ -0,0 +1,10 @@ +# Optional Audita Debug Reporting + +Status: Deferred; not a v3.0.0 release prerequisite. + +Potential future work: enable Audita debug reporting through Narratio's testing +overlay and collect Audita's debug-only execution artifacts. Scope and design +remain undecided; this is not an implementation commitment. + +Current reporting behavior is documented in the +[Audita integration contract](../integrations/audita.md). diff --git a/docs/roadmap/audita.md b/docs/roadmap/audita.md deleted file mode 100644 index ac64e39..0000000 --- a/docs/roadmap/audita.md +++ /dev/null @@ -1,288 +0,0 @@ -# Adopt Audita v2 and ConfigKit v0.4.0 - -Status: implemented (2026-09-25); release pending. - -## Goal - -Migrate Narratio's `polish` integration from Audita's former command/configuration -contract to Audita v2.0.0. Keep Narratio responsible for orchestration and Audita -responsible for correction behavior and LLM execution configuration. - -Adopt a consistent provider-named credential list for both Audita and Notarius, -so operators define each secret once and explicitly share it with the downstream -applications that need it. This is a clean cutover without backward-compatibility -shims. - -Adopt ConfigKit v0.4.0 and use “overlay” consistently for Narratio's selected -pipeline configuration, distinct from downstream LLM model profiles. - -Production runs use Audita's default reporting configuration: Narratio passes -neither `--debug` nor `--reporting=false`. Normal reports remain available without -enabling debug diagnostics. Adding debug reporting through Narratio's testing -profile is deferred. - -This document defines the intended end state, migration scope, and policy. -The [implementation plan](implementation.md) owns implementation sequencing and -verification steps. - -## Baseline and upstream contract - -The baseline is Narratio v2.2.1 and the Audita checkout tagged v2.0.0. Upstream -references are Audita's `docs/migrations/v2.md`, `docs/cli.md`, -`docs/integrations/subprocess.md`, and `docs/integrations/output-files.md`, -available in the sibling `../audita/` checkout. Narratio's current contract is -documented in [Integration: Audita](../integrations/audita.md). - -The current integration is not compatible with Audita v2: - -- It requests `--report-json` by default; that flag has been removed. -- It can pass retired endpoint, model, concurrency, and work-directory-retention - flags. -- It expects one JSON report instead of the normal v2 report bundle. -- Its single forwarded `AUDITA_LLM_API_KEY` has no implicit meaning in v2; - an Audita backend must explicitly reference that environment variable. -- Its semantic fingerprint sorts modules, although Audita treats module order - and repetitions as meaningful. - -Audita v2 requires a version-2 configuration file. An explicit `--config` selects -one; otherwise Audita performs its own documented discovery. Backend definitions -own endpoints, credentials, and capacity. PromptKit profiles own models and -per-attempt controls. Audita owns retry/repair policy and operation deadlines. -Narratio's generated invocation YAML is a diagnostic record, not a substitute -for that Audita configuration file. - -## Target architecture - -Retain the existing subprocess boundary and `polish` stage. Do not introduce a -second correction engine, an in-process Audita dependency, or a generic external -tool configuration framework. - -Narratio owns: - -- Selection of the executable and the overall invocation timeout/cancellation. -- Transcript and glossary inputs, run-local output placement, validation of the - corrected transcript, and promotion to the existing polished transcript - identity. -- Explicit configuration/profile selections and supported domain overrides. -- Deliberate credential forwarding, secret-safe invocation diagnostics, and - manifest references to results and reports. -- Existing stage completion, resume, force, publish, restore, and cleanup - behavior. - -Audita owns its configuration schema and discovery, backend/model configuration, -provider execution, correction sequence, recovery behavior, and report contents. -Narratio does not translate old settings into a generated v2 configuration or -duplicate Audita's complete validation logic. - -## Configuration and invocation migration - -### ConfigKit and pipeline overlays - -Upgrade Narratio's ConfigKit dependency from v0.3.1 to v0.4.0. ConfigKit's -selected-configuration terminology changes from profile to overlay: the Go API -uses `Options.Overlay`, `LayerOverlay`, and `CodeOverlay*`; source layers and -diagnostic codes use `overlay` and `overlay.*`. Composition, precedence, secret -resolution, and lifecycle semantics otherwise remain unchanged. - -Narratio's pipeline-selection interface becomes `--overlay`, selecting -`overlays/.yml` beside the root pipeline file. Remove `--profile` and old -selected-directory discovery without aliases or fallbacks. Preserve existing -base selection for omission, explicit empty selection, and `default`. -Configuration inspection, diff descriptions, internal selection names, and -newly written selection provenance use overlay terminology as well. - -Maintain existing value-based digest and stage-reuse semantics; renaming -informational selection provenance is not a reason to invalidate completed -work or automatically rewrite historical manifests. Strict manifest decoding -rejects retired provenance fields; operators must deliberately migrate that -informational member on a backed-up manifest, preserving execution/checkpoint -state. Update maintained examples and operator migration guidance for the -directory, flag, and provenance changes. - -This rename does not apply to downstream model profiles, Scriptorium -`profile_id`, Audita's proposal/validation profiles, or Notarius's -`config_profile`. Nor does it prohibit ordinary explicit imports from a directory -named `profiles`. Narratio overlays do not automatically select downstream -overlays or profiles. These are independently owned choices. - -### Audita v2 configuration - -Keep binary, timeout, explicit config-path selection, and surviving explicit -domain overrides such as transcript description, output schema, and modules. -Allow Audita's native configuration discovery when no config path is selected; -do not impose an additional requirement to supply an explicit path. - -Provide optional Audita overlay, proposal-profile, and validation-profile -selectors, passed through to their corresponding CLI flags. Omission leaves -Audita's own defaults and validation-profile inheritance in control. These are -Audita selectors, not automatic translations of Narratio overlay names. - -Retire the Narratio settings that mirror removed Audita endpoint/model/capacity -flags and work-directory retention. Move their operational meaning into the -operator-maintained Audita configuration and profiles. Remove stale adapter -fields, validation, invocation metadata, examples, and tests along with them. -Remove the old report toggle and both adapters' singular `llm_api_key` fields -from accepted Narratio configuration. Do not retain deprecated aliases, -automatic conversion, ignored fields, or a v1 execution path. Existing strict -configuration decoding rejects removed fields. Provide migration guidance and a -release compatibility notice rather than runtime compatibility machinery. - -Preserve the distinction between omitted modules and an explicitly empty module -list: omission requests Audita's configured/default sequence; an empty list -requests no correction modules. Preserve order and repetitions rather than -sorting or deduplicating. Do not broaden this into a change to unrelated -adapters' module semantics. - -Normal invocations no longer pass `--report-json`, `--work-dir-retention`, or -debug-work-directory flags as a way to obtain reports. They need no reporting -override: Audita writes its ordinary report bundle by default. The old -Narratio `report` toggle must not silently become `--reporting=false`. - -The production Audita configuration must also retain reporting enabled and debug -disabled. Omitting CLI flags does not override an operator's explicit external -configuration. Document this distinction rather than silently adding forced -boolean overrides or rejecting otherwise valid Audita configuration. - -## Credentials - -Define secrets once under provider-oriented environment-variable names, such as -`OPENROUTER_API_KEY` and `RAKESTRAWHOME_API_KEY`, through the process environment -or Narratio's existing secrets-directory bootstrap. These names are examples, -not a fixed provider registry; application-specific names are not required. - -Add `pipeline.audita.credentials` and `pipeline.notarius.credentials` as lists -of Narratio typed secret references. Each entry names the environment variable -whose captured value is forwarded under that exact name. Omission or an empty -list forwards no credentials. The same reference may appear in both adapters' -lists without duplicating its stored secret value. An illustrative configuration -fragment is: - -```yaml -pipeline: - audita: - credentials: - - OPENROUTER_API_KEY - - RAKESTRAWHOME_API_KEY - notarius: - credentials: - - OPENROUTER_API_KEY -``` - -Resolve references through the existing configuration/secret-loading mechanism -before adapter execution, preserving its nonempty-value requirement. Forward -the captured values rather than rereading the parent environment during a run. -Retain the minimal child execution environment plus each adapter's explicit -credential list; do not forward every secret found during bootstrap. - -Audita backends select these names through `credential`; the current Notarius -PromptKit profiles select them through `api_key_env`. Narratio neither inspects -downstream configuration to infer required credentials nor translates names, -registers providers, or duplicates backend configuration. Audita resolves -credentials for every surviving configured backend, even one not selected by -a profile, so its list must cover all those references. - -Generalize Notarius's current single-name forwarding and replace Audita's fixed -`AUDITA_LLM_API_KEY` destination with this shared convention. Use existing shared -secret and subprocess facilities, extending narrow helpers where needed rather -than introducing a new secret-management framework. Register every forwarded -credential with captured-output redaction, including custom names and multiple -keys. Values must not enter arguments, generated configuration, manifests, -logs, or examples. As today, downstream tools remain responsible for files they -write directly; this does not add arbitrary downstream-file scanning. - -## Ordinary reports and result handling - -Replace the old `audita.report.json` handling with Audita's normal `errors.yml`, -`recoveries.yml`, and `report.md` bundle. With a file output, Audita writes these -beside the corrected transcript; an arbitrary report destination is not supplied -through the removed report flag. - -Discover the reports at their defined run-local locations and retain them as -diagnostics using Narratio's existing manifest and diagnostic lifecycle. Keep -them distinguishable from the canonical polished transcript. Adapt report -references, publication/restore coverage, and operator inspection guidance so -the bundle is not lost or confused with an older run's JSON report. Do not -fabricate a legacy JSON report or delete historical diagnostics as part of the -migration. - -Retain existing bounded, confined file handling. Replace JSON-specific report -handling with only the validation needed for consuming the v2 bundle; do not -introduce a second comprehensive report-schema validator or make incidental -Markdown wording a contract. Report locations must be discoverable without -parsing human-readable stderr. Detailed report-counter ingestion is not required -for this migration. - -Audita's exit status remains authoritative. Exit zero with a valid corrected -transcript may include recoverable LLM failures, including a normalized result -with no model corrections. Preserve the reports so operators can assess that -outcome; do not add an unrequested fail-on-recovery or minimum-correction policy. - -Conversely, a nonzero exit must fail polish even if a transcript file exists: -Audita can fail while writing required reports after writing the transcript. -Preserve available failure diagnostics without requiring early failures to have -produced a complete report bundle. Failed output must not become the canonical -successful result. - -## Resume and semantic evidence - -Update polish's semantic evidence for the new result-affecting selections and -remove retired endpoint/model fields. Include explicit config/overlay/profile -selection and preserve ordered module semantics, including omitted versus empty -selection. Version the changed evidence so incompatible older evidence is not -mistaken for a valid match when polish is considered for reuse. - -Keep the existing boundary for opaque downstream configuration: Narratio does -not recursively hash Audita configuration, overlays, and profile contents. -Operators force polish when changing contents behind unchanged selectors. -Timeouts, credential values, reporting details, and other execution-only settings -must not become blanket semantic invalidation keys. Do not rerun unrelated or -unselected stages merely because the integration was upgraded. - -## Migration guidance and verification - -Update the canonical [configuration reference](../config.md), -[Audita integration contract](../integrations/audita.md), -[Notarius integration contract](../integrations/notarius.md), -[polish internals](../internal/stage-polish.md), maintained examples, and relevant -operations/troubleshooting guidance when implementation lands. Cover: - -- Moving selected Narratio pipeline files to `overlays/`, replacing `--profile` - with `--overlay`, and distinguishing configuration overlays from model profiles. -- Updating ConfigKit API/source attribution and verifying unchanged effective - values retain existing semantic reuse behavior. -- Preparing and validating an Audita v2 configuration and its profiles before - switching the executable; copying an old v1 configuration is insufficient. -- Moving endpoint, model, capacity, and credential selections to their v2 owners. -- Replacing both singular `llm_api_key` settings with explicit `credentials` - lists, storing each provider key once, and matching downstream credential - references to those names. State clearly that old fields are removed. -- Using normal reports to inspect recoveries and failures without debug mode. -- Force/resume expectations and the treatment of existing completed sessions. -- The explicitly deferred nature of Narratio testing-profile debug support. - -Verification should exercise the real adapter boundary with offline subprocess -fixtures and focused stage/configuration tests: accepted v2 arguments, omitted -defaults, ordered/empty modules, secret-safe forwarding, successful report -capture, recoverable-error success, nonzero exit with an existing transcript, -failure diagnostics, and semantic reuse/invalidation. Update existing behavioral -tests rather than retaining assertions for removed flags or exact stderr prose. -Where practical, verify invocation against the tagged Audita CLI without live -provider calls. Follow the repository's testing and documentation policies. - -Credential tests must cover both adapters, multiple custom names, reuse of one -secret by both tools, no forwarding of unselected secrets, captured-value -semantics, and redaction of every forwarded value. Verify removal of obsolete -configuration fields without preserving their compatibility behavior. - -## Scope boundaries - -No changes to pipeline order, transcript identity, downstream artifact -definitions, or provider algorithms are intended. No automatic v1 configuration -converter, Audita-internal resume mechanism, runtime binary-version negotiation, -or live-service release gate is required. Debug-profile integration and ingestion -of Audita's debug-only execution artifacts are future work. - -The Notarius change is limited to credential configuration, forwarding, and -associated diagnostics/documentation/tests; extraction and cache behavior are -unchanged. This migration breaks existing pipeline configurations using removed -fields; account for that explicitly in Narratio release versioning. diff --git a/docs/roadmap/feature-request-notarius-config-profile.yml b/docs/roadmap/feature-request-notarius-config-profile.yml deleted file mode 100644 index b8f4f17..0000000 --- a/docs/roadmap/feature-request-notarius-config-profile.yml +++ /dev/null @@ -1,102 +0,0 @@ ---- -title: Select a Notarius configuration profile through Narratio -status: proposed -consumer: rakestraw.dnd Ansible collection and its pipeline role - -use_case: >- - The Ansible pipeline role deploys Narratio and Notarius as independently - configured applications. Operators select Narratio's testing profile to - develop prompts and other pipeline assets with inexpensive models while - retaining production settings in the base configuration. That selection - currently cannot select a Notarius configuration profile, leaving extraction - on its production settings. - -desired_outcome: >- - Narratio's testing overlay can explicitly select Notarius's testing overlay. - Notarius owns the resulting model choices and any other testing-specific - settings. The operator selects testing once when invoking Narratio, without - editing production files or separately invoking Notarius. - -rationale: >- - Both applications already use ConfigKit configuration overlays. Selecting - the downstream overlay preserves each application's configuration ownership - and supports future Notarius testing adjustments without adding a Narratio - configuration field for every downstream setting. This is explicit profile - composition, not a requirement for arbitrary recursive orchestration. - -requested_interface: - field: pipeline.notarius.config_profile - value: Optional Notarius ConfigKit configuration profile name - supplied: >- - Invoke Notarius with --config-profile and the selected name when extraction - executes. The setting is available in Narratio's base configuration and - its ordinary configuration overlays. - omitted_or_null: >- - Do not supply a profile-selection flag. Preserve existing Notarius - configuration selection behavior. - downstream_semantics: >- - Retain Notarius's existing selector semantics, including the default - selector for its base configuration. Notarius resolves profiles relative - to its own configuration root, not Narratio's root. - explicit_selection: >- - Do not automatically forward Narratio's profile name. The selected - Notarius profile may have the same name or a different name. - distinction: >- - This selects a ConfigKit configuration overlay through --config-profile, - not a PromptKit LLM profile through --llm-profile. The Notarius overlay - can itself select the desired LLM profiles using native Notarius settings. - -illustrative_consumer_configuration: - note: >- - These are separate files managed by the Ansible suite, not a proposed - combined configuration document. The Notarius pipeline and LLM profile - names are illustrative; the referenced LLM profile must be available to - Notarius. - narratio_profiles_testing_yml: | - notarius: - config_profile: testing - notarius_profiles_testing_yml: | - pipelines: - dnd-session: - llm_profile: gemini-flash-latest - operator_workflow: >- - Select --profile testing on the ordinary Narratio invocation. Narratio - selects Notarius's testing configuration overlay when it runs extraction; - other Narratio testing overrides continue to apply normally. - -observable_behavior: - - >- - Notarius owns overlay discovery, composition, validation, and the effective - downstream settings. A missing or invalid selected overlay must not silently - fall back to production or be reported as successful extraction. - - >- - Changing the selected Notarius configuration profile must not silently - reuse Narratio's completed extraction from the previous selection. - Existing downstream invalidation behavior applies when extraction changes. - - >- - Editing Notarius overlay contents behind an unchanged selector retains - the existing operator responsibility to force extraction when needed. - This request does not require inspecting or tracking private Notarius files. - - >- - Existing credential forwarding remains independently configurable through - notarius.llm_api_key, including in the Narratio testing overlay when the - selected Notarius settings require a different credential reference. - -consumer_follow_up: >- - After Narratio releases this interface, the standalone Narratio Ansible role - can expose an optional variable and omit the field when unset. The suite role - can deploy Notarius's testing overlay and select it from Narratio's testing - overlay, then distribute the changes through the collection. These consumer - updates are separate from this upstream feature request. - -scope_boundaries: - - No Notarius change is currently required; v0.7.0 supports --config-profile. - - No separate Narratio-side Notarius LLM-profile selector is requested. - - No automatic profile-name propagation or downstream configuration rewriting. - - No new credential discovery, copying, or provider-name remapping. - - This request specifies consumer behavior, not internal Narratio code changes. - -references: - narratio_configuration: ../config.md - narratio_notarius_integration: ../integrations/notarius.md - notarius_cli: https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.7.0/docs/cli.md diff --git a/docs/roadmap/fix-publish-output-destination-resolution.md b/docs/roadmap/fix-publish-output-destination-resolution.md deleted file mode 100644 index cfa35a6..0000000 --- a/docs/roadmap/fix-publish-output-destination-resolution.md +++ /dev/null @@ -1,171 +0,0 @@ -# Bugfix Request: Preserve Publish Destination Derivation with Explicit Rules - -Status: Implemented; release target v2.1.1 - -## Summary - -Narratio should successfully derive a publish destination when a valid -`publish.outputs` rule omits `dest`, regardless of whether that rule explicitly -sets `required`. Narratio v2.1.0 currently rejects this valid configuration in -some cases with a ConfigKit `resolve.unrecorded_change` diagnostic. - -This is a configuration-resolution bookkeeping defect, not a YAML formatting -restriction. The fix should preserve the existing public configuration -contract: operators may omit a derivable destination and may independently set -`required` to either `true` or `false`. - -## Observed Behavior - -A pipeline containing a rule equivalent to the following fails during native -configuration loading: - -```yaml -publish: - outputs: [{"source": "narratio.transcript.final", "required": true}] -``` - -Narratio reports: - -```text -error [resolve.unrecorded_change] /publish/outputs: application resolution changed a value without a covering record -``` - -The same failure is possible with ordinary block-style YAML when `required` is -explicit and `dest` is omitted. Flow-style YAML is valid input and is not the -cause of the defect. - -The existing destination-derivation tests do not expose this case because -their rules also omit `required`. Narratio then supplies the default value for -`required` and records a change covering the complete `publish.outputs` -sequence. That broader record happens to cover the destination mutation as -well, masking the missing resolution record that occurs when `required` is -already explicit. - -## Expected Contract - -The current configuration contract remains unchanged: - -- `publish.outputs[].source` is required; -- `publish.outputs[].dest` may be omitted when Narratio can derive it from the - selected built-in or configured artifact source; -- `publish.outputs[].required` defaults to `true`, but an operator may set it - explicitly to either `true` or `false`; and -- equivalent valid YAML representations have equivalent configuration - semantics. - -For example, both rules below should load and validate, retain their explicit -requiredness, and receive the canonical derived destination: - -```yaml -publish: - outputs: - - source: narratio.transcript.final - required: true - - source: narratio.transcript.final_markdown - required: false -``` - -## Root Cause - -ConfigKit intentionally treats sequences as atomic when comparing values -before and after application resolution. Changing a field within one -`publish.outputs` entry is therefore observed as a change to the -`/publish/outputs` sequence. - -Narratio currently derives each missing destination by recording a -normalization at the individual `publish.outputs[index].dest` path. That child -record does not cover the atomic parent-sequence change. A separate -container-level record is currently created only when Narratio also defaults -an omitted `required` value, so configurations with explicit `required` values -lose that incidental coverage. - -## Recommended Fix - -Make Narratio's publish-output resolution bookkeeping conform to ConfigKit's -atomic-sequence contract. Whenever resolution derives one or more destinations -inside `publish.outputs`, the recorded resolution changes must cover the -`/publish/outputs` container. Destination derivation must not depend on an -unrelated `required` default being applied in the same list. - -The implementation may retain more specific provenance where useful, but it -must provide valid container-level coverage for the sequence mutation. Do not -change ConfigKit's sequence comparison semantics merely to accommodate this -call site. - -## Scope - -- Correct destination derivation for publish rules that explicitly set - `required` and omit a derivable `dest`. -- Preserve explicit `required: true` and `required: false` values. -- Preserve canonical destination derivation for built-in transcript sources - and configured artifact sources. -- Add focused regression coverage through Narratio's real configuration load - and validation boundary. -- Prepare a backward-compatible patch release after the fix passes the normal - repository validation. - -## Non-goals - -- Changing the `publish.outputs` schema, defaults, source vocabulary, or - destination derivation rules. -- Requiring operators or configuration generators to supply `dest` when it is - derivable. -- Requiring operators to omit explicit `required: true` values. -- Treating flow-style YAML or JSON-compatible YAML values as unsupported. -- Changing ConfigKit's documented atomic treatment of sequences. -- Adding storage, publication, or network integration behavior. - -## Regression Coverage - -Add one focused package-level behavioral test that loads and validates a -pipeline with omitted destinations and explicit requiredness. A representative -list should include both `required: true` and `required: false`, then assert -that: - -- loading and validation succeed; -- each destination is derived from its source; and -- each explicit `required` value is preserved. - -Using the compact flow-style representation that exposed the integration -failure is reasonable, while the existing block-style cases continue to cover -omitted `required`. The test should exercise the public configuration behavior, -not ConfigKit call choreography or Narratio's private recording mechanism. - -No network service, object store, credential, or test double is required. - -## Target End State - -All valid publish rules with derivable sources behave consistently across -these cases: - -| `dest` | `required` | Result | -| --- | --- | --- | -| omitted | omitted | destination derived; required defaults to `true` | -| omitted | `true` | destination derived; explicit `true` retained | -| omitted | `false` | destination derived; explicit `false` retained | -| explicit | either value | explicit destination and requiredness retained | - -`narratio config validate`, `narratio config show`, and ordinary pipeline -loading accept the corrected cases without an unrecorded-change diagnostic. -No operator configuration migration or workaround is required. - -## Release - -This is a backward-compatible correction to documented configuration behavior -and should be released as Narratio `v2.1.1`. The release note should identify -the corrected explicit-required/derived-destination interaction and state that -no special upgrade action is required. - -## Acceptance Criteria - -- The focused regression test fails against v2.1.0 behavior and passes after - the fix. -- Publish rules with explicit `required: true` or `required: false` and omitted - derivable destinations load and validate successfully. -- Derived destinations and explicit requiredness appear correctly in the safe - effective configuration. -- Existing destination derivation, publish validation, imports, profiles, and - configuration inspection tests remain green. -- The complete repository validation and release-candidate checks pass. -- The correction is published as the immutable `v2.1.1` tag with the required - release note. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 8e51617..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,393 +0,0 @@ -# Implementation Plan: Standard Repository Policy Alignment - -Status: Complete (2026-09-28); release pending. - -## Scope and execution instructions - -Implement [the policy roadmap](policy.md). This replaces the completed Audita -migration plan, retained in Git history. This planning pass changes no executable -code or adopted policies. The instructions below govern the later implementation. - -Complete Stage 1 through Stage 7 in order, one bounded implementation prompt per -stage for gpt-5.6-terra. Record actual completion and verification under each stage -and commit each coherent completed unit with a plain-English message. Intermediate -states are not release candidates. Do not publish, tag, push, change dependencies, -or rewrite released material as part of this work. - -Before each stage, read root AGENTS.md, the development reading guide, the roadmap, -and the policies applicable to that stage. Reuse unchanged policy text already in -context. Agents and documentation apply throughout; meta governs adoption; OSS -governs incorporated material; architecture/testing/release apply to the script -and test stages. Do not impose whole-suite policy reading on unrelated tasks. - -Use upstream commit `d84a38ebab26f048cafbe59a2246700c35d29843` throughout. Start -with its [adoption guide](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/adoption.md). -Read source documents at that revision, not mutable sibling worktree content. -Public access or a verified Git object is sufficient. Do not reconstruct prior -adoption history. If an exact source/notice cannot be verified, report the concrete -gap rather than inventing a commit, copyright, or license expression. - -## Selected decisions - -### Common policy language and provenance - -Adopt non-project-specific upstream wording verbatim. Remove template-only -sections, customize required project content, and retain only the locally -significant exceptions identified in the roadmap. Preserve Narratio's actual -application contracts; this is not an architecture redesign or new safety audit. - -Every adopted policy receives its actual Base Policy Version, actual local -adoption date, and an Upstream Source URL with the full reviewed commit and its -own `docs/policy/.md` path. Use current branch URLs only for public -discovery/adoption-guide navigation. Preserve commit-pinned upstream example -links; do not duplicate the example collection locally. No sync checker or -metadata registry is required. - -### Attribution scope and presentation - -Use OSS v1.8's exact House Format. Attribution contains factual entries, not -research notes. Put concrete exception reasoning in OSS policy or an appropriate -decision record. Credit copied policy, agent-orientation, and reading-guide -material accurately, distinguishing adapted portions from unrelated local text. -Keep adapted material CC-BY-SA-4.0 and application code GPL-3.0-or-later. - -Inventory selected modules and shipped runtime material, not every historical -go.sum entry or development executable. Include the test-only go-cmp module as -part of the maintained module inventory with factual scope identification. Do not -include separately installed Audita, Notarius, PromptKit, or other subprocess -applications merely because Narratio invokes them. Include any actual copied -material discovered in the repository. Combine related modules into a component -entry only when their source revisions and notices can all be represented -accurately; do not invent a single revision for independently tagged modules. - -### Final release validation and mechanics - -Keep the current `scripts/release.sh VERSION` interface and source-push-before-tag -workflow. The optional `check-release-candidate.sh VERSION` remains diagnostic; -normal release instructions invoke only publication, which calls the checker once. -No persistent validation receipt/cache or skip-validation escape hatch is added. - -The final checker runs these checks once per invocation: - -| Check | Selected treatment and reason | -| --- | --- | -| Stable version and nonempty matching note/title | Retain existing identity checks. Remove fixed section-heading checks; content and footer are preparation responsibilities, not a new prose parser. | -| `go mod tidy -diff` | Retain: detects an incomplete checked-in module graph without modifying source. Module downloads are preparation, not application-service calls. | -| Tracked Go formatting and diff whitespace | Retain existing inexpensive source hygiene checks; do not add whole-history scans. | -| `go test -race -count=1 ./...` | The single full suite. It includes doccheck and maintained examples. Race instrumentation adds distinct coverage for Narratio's concurrent I/O, cancellation, and shared state. Remove the separate ordinary full-suite invocation. | -| `go vet ./...` | Retain distinct static checks not established by runtime tests. | -| Release asset builder | Retain three existing targets and embedded-version smoke check. Remove redundant `go build ./...`; package tests compile packages and the builder validates deliverables. | -| Extra focused doc/example test invocations | Remove: covered by the full suite. Remain available for development diagnosis. | - -Keep GOWORK=off for owned Go commands. Use explicit `-mod=readonly` on owned -test/vet/build commands, including the asset builder, so incidental vendor trees -are not selected and validation cannot silently edit the module requirements. -Do not overwrite all caller GOFLAGS just to set this option. Keep tidy's own -`-diff` mode. These choices preserve the existing module-based release model. - -Remove the blanket tracked-go.work/go.work.sum rejection: they are inert with -GOWORK=off. Remove the mere-existence vendor-directory rejection. Narrow the -existing all-replacements ban to filesystem replacements: release inputs remain -versioned modules rather than machine-local replacement paths; allow versioned -module replacements. Use Go's module metadata, not regex parsing of replace -blocks. A `go list -m` template inspecting `.Replace.Version` can report local -replacement module names. Do not build a new dependency-policy framework. -This narrows an existing prohibition; supporting repository-local replacement -modules is not introduced by this work. - -Retain exact-candidate guards, origin/main matching, clean tracked/untracked state -as reported by Git, fresh versions, lightweight tags, explicit-ref push, and -builder destination protections. Ignored local files alone do not fail release. -Do not add successful-push network read-back, CI polling, native macOS checks, -new artifacts, or application credential requirements. Cheap mutable publication -state checks before and after validation remain justified. - -## Stage 1: Adopt common policy, provenance, and orientation - -Status: Complete (2026-09-28). - -Verification: Compared all adopted policy text with the pinned Git objects; -agents, testing, and meta match after template removal, metadata insertion, and -whitespace cleanup. Reviewed the Narratio-specific documentation, architecture, -and OSS deviations. The CC-BY-SA-4.0 text is byte-identical to the pinned -source. `go test ./internal/doccheck` and `git diff --check` passed. Dependency -attribution and release-policy/procedure alignment remain for later stages. - -Read the pinned adoption guide, all seven pinned policy templates, the attribution -example for policies, and current local counterparts. Review the roadmap's local -customizations, root AGENTS.md, and docs/development.md. - -1. Adopt agents v1.3, testing v1.3, documentation v1.11, architecture v1.3, - meta v1.0, and OSS v1.8. Preserve upstream common wording and apply only - necessary project customizations. Leave release policy adoption to Stage 5, - when executable behavior will match it. Do not falsely stamp that file now. -2. Preserve Narratio's architecture-specific stage, manifest, storage, credential, - group-workspace, and platform contracts. Retain troubleshooting ownership; - omit unused public API responsibilities. Remove redundant local general-agent - elaborations rather than preserving stylistic variants. -3. Add accurate per-file metadata and adapt upstream agents/AGENTS.md plus the - supplied selective policy-reading guidance. Preserve existing task routes. -4. Create attribution with the canonical policy entry and local CC-BY-SA-4.0 - text. Describe exactly the material adopted so far, including adapted reading - guidance. Add COPYRIGHT's limited document-license scope. Root LICENSE and - the application grant remain unchanged. Do not imply the not-yet-updated - release policy was adopted from this commit. -5. Keep public discovery links in meta and source/example links pinned. While - release policy still has its old headings, make the OSS correction-policy - cross-reference resolve to the existing heading; restore the standard target - when Stage 5 lands. Do not create dead local example links. - -Verification: compare each adopted file to its pinned source, checking that each -remaining substantive deviation is justified. Run doccheck and diff checks. -Record that dependency attribution and release alignment remain incomplete. - -## Stage 2: Attribute core modules and shipped Go material - -Status: Complete (2026-09-28). - -Verification: Resolved selected module tags through Go download metadata and -upstream Git records; verified ConfigKit and both YAML notices against their -selected source revisions. Resolved the release-CI Go 1.26.7 tag to its source -commit. Four new standard license texts match the pinned reference copies by -SHA-256; review remediation restored BSD 3-Clause's original trailing whitespace -so that text is also byte-identical to the pinned source. All added commit-pinned -source/notice links returned HTTP 200. Reviewed -tracked source and example material for additional incorporated third-party -notices without finding another concrete item. `go test ./internal/doccheck` -and `git diff --check` passed. AWS module attribution remains for Stage 3. - -Read adopted OSS/documentation policy, go.mod, the actual selected module source -licenses/notices, and the build toolchain declarations. Focus this prompt on -ConfigKit, both YAML module generations, go-cmp, x/sync, x/sys, and Go runtime -material. AWS modules are explicitly deferred to Stage 3. - -1. Resolve selected versions to exact upstream commits using module download - metadata or authoritative upstream Git records. Read licenses, notices, - copyright statements, exceptions, and companion-text requirements at those - commits. Do not rely on today's default branch or copy another project's - inventory without verifying the versions and material. -2. Add factual House Format entries and required standard texts. ConfigKit's - LGPLv3 entry must make both LGPLv3 and GPLv3 texts reachable. Preserve distinct - YAML v3/v4 notices and actual expressions rather than assuming alternatives. -3. Attribute the runtime/toolchain material incorporated in the declared binary - builds using the pinned release-CI toolchain source revision. Identify that - scope factually; do not claim every locally built binary uses that compiler. - Do not add a new toolchain pin or runtime scanner. -4. Inspect repository material for copied snippets/assets requiring attribution, - distinguishing consultation from incorporation. Include verified entries; - record any concrete unresolved material before claiming inventory completion. - -Verification: check statements against selected sources, exact-commit links, -verbatim text copies, companion links, and local doccheck. Do not add generated -SBOM tooling, research prose to attribution, or runtime dependency upgrades. - -## Stage 3: Complete AWS attribution and public navigation - -Status: Complete (2026-09-28). - -Verification: Reconciled `go list -m all` with the modules compiled into -Narratio or its maintained tests: all 24 incorporated modules are represented, -including indirect AWS modules. The YAML dependency-test-only -`gopkg.in/check.v1` is selected in the broader module graph but is not compiled -into Narratio or its tests. Go module metadata resolved sixteen SDK modules to -one commit, S3 to a distinct commit, and smithy-go to its own commit. Compared -selected Apache licenses and AWS/smithy notices to their source revisions; -verified new pinned notice and embedded singleflight license links. README -navigation, `go test ./internal/doccheck`, and `git diff --check` passed. No -dependency or released material changed. - -Read adopted OSS policy and selected AWS SDK/smithy module license material. -Use go.mod plus Go's selected module graph, including indirect AWS modules. - -1. Resolve and document AWS components at their actual selected revisions. - Monorepo modules may share an entry only with accurate module/revision scope - and all necessary notice links. Include smithy separately when its source - and notices differ. Preserve Apache license and relevant NOTICE material. -2. Reconcile the complete attribution document against the module graph, runtime - scope, and copied-material review. Ensure no dependency was skipped merely - because it is indirect, and no external subprocess application was included - solely because it is invoked. Resolve concrete gaps; do not mark them covered - through generic wildcard prose. -3. End README with the two prescribed grant/attribution statements. Keep the - inventory factual and centralized; ensure all applicable local texts exist. - Do not copy unused reference licenses or example collections. - -Verification: review coverage and source identity, run doccheck, compare copied -texts to their authoritative sources, and check the diff for accidental secrets -or unrelated source changes. No new legal-approval or network release gate. - -## Stage 4: Simplify candidate validation and its behavioral tests - -Status: Complete (2026-09-28). - -Verification: The checker retains version/title, module hygiene, formatting, -whitespace, one race-enabled full suite, vet, and the asset builder while -removing duplicate test/build invocations. Go module metadata distinguishes -filesystem from versioned replacements; workspace and vendor presence are -inert under GOWORK=off and readonly module selection. Focused release-check -tests cover selected command counts, failure propagation, note handling, -replacement behavior, builder targets/version, and destination protections. -`go test ./internal/releasecheck`, shell syntax checks, `go test -./internal/doccheck`, and `git diff --check` passed. No publication ran. - -Read scripts/check-release-candidate.sh, scripts/build-release-assets.sh, -scripts/release-lib.sh, and internal/releasecheck/assets_test.go. Read testing, -architecture, OSS, and both current and target release policy. This stage changes -the checker; Stage 5 completes adopted release policy and operator documentation. - -1. Implement the exact selected validation matrix above. Remove fixed release - note sections, duplicate full/focused tests, and redundant generic build. - Retain existing note title/version checks without adding a footer parser. -2. Use the selected module mode/workspace handling in checker and builder. - Replace blanket file/directory guards and replacement parsing as specified. - Preserve output-directory protections and temporary cleanup ownership. -3. Update candidate test fixtures to accept a concise note without the old - headings; missing/empty notes and mismatched titles still fail. Test inert - workspace/vendor presence, local replacement rejection, and versioned - replacement acceptance using representative metadata through a small Go stub. -4. Update the existing owned-command tests to prove the new gate selection and - one invocation of each required suite/build, rather than retaining minimum - command-count assertions. Command observation is appropriate here because - duplicate validation is the specific regression being prevented. Exercise - failure propagation without duplicating every unrelated module-tool behavior. -5. Preserve builder tests for three named assets, version injection, GOWORK=off, - and destination protection. Cover readonly module selection without expanding - into a new cross-platform infrastructure matrix. - -Verification: focused internal/releasecheck tests, shell syntax checks, doccheck, -and diff checks. Do not run the real publication script against origin. Avoid -recursive real full-suite execution from within checker test fixtures. - -## Stage 5: Align publication behavior, policy, and operator procedure - -Status: Complete (2026-09-28). - -Verification: Adopted pinned release policy v1.4 with only metadata and the -selected Narratio validation rationale added to common wording. Publication -reports failed/uncertain tag pushes without asserting nonpublication; offline -bare-remote tests cover one checker call, ignored files, retained local tag, -and the existing exact-ref/branch/candidate guards. The runbook now has one -normal publication command, immutable-tag footer URLs, separate asynchronous -delivery status, and ambiguity/recovery guidance. Updated attribution and -release navigation without changing historical notes. `go test -./internal/releasecheck ./internal/doccheck`, shell syntax checks, and -`git diff --check` passed. No tag, push, or release ran. - -Read pinned release v1.4, current docs/policy/release.md and docs/release.md, -scripts/release.sh, and internal/releasecheck/publication_test.go. - -1. Adopt release v1.4 verbatim except documented project choices. Record the - selected extra validation checks and their distinct rationale in a concise - local section; do not retain obsolete exception/compensating-control prose - for tag-push completion or authoritative-origin scope. -2. Keep the existing publication interface and pre/post-validation guards. - Fix push-failure reporting so it says publication failed or is unconfirmed, - not definitively unpublished: a failed transport can have an ambiguous - outcome. Keep the local tag for investigation; do not auto-delete or retry. -3. Rewrite the runbook around one publication command. Standalone validation is - optional diagnosis, not a prerequisite. Keep source commit push before tag, - exact candidate identity, current three artifacts/checksums, and lightweight - tagging. No real tag or release is created in this stage. -4. Include the exact three footer links using Narratio Gitea URLs at the intended - release tag. Preparing the footer and checking target paths locally is enough; - do not require the future tag to resolve remotely. Preserve hosted note reuse. -5. Explain ambiguous-push investigation and separate artifact delivery status. - Transient failures can be retried from the immutable tag; already-published - contents/checksums must not be overwritten. Source/content defects need a new - release. Do not add an automated retry subsystem. -6. Update policy attribution material scope for the now-adopted release policy - and the OSS correction-policy link. Update related development guidance so - it does not retain the old mandatory duplicate release sequence. -7. Preserve offline local-bare-remote tests. Add/adjust checks that the checker - is called once, ignored files alone do not block publication, a rejected push - leaves its local tag, and ambiguous failure wording makes no false success or - nonpublication claim. Keep exact-candidate, branch, dirty-state, uniqueness, - upstream-change, and explicit-ref protections. No live forge or CI tests. - -Verification: focused release/doc tests and shell syntax; inspect all current -release guidance for conflicting requirements. Historical release notes stay -unchanged. Adopt metadata only for the actual document revision incorporated. - -## Stage 6: Align CI and contributor validation references - -Status: Complete (2026-09-28). - -Verification: Removed only the redundant push/PR documentation-and-examples -job, with cross-build dependencies repaired. The full suite still exercises -both checks, while focused commands remain documented for developer diagnosis. -Tag CI, scheduled shuffle/race testing, runner images, build targets, and -non-overwrite publication settings remain unchanged. `go test -./internal/doccheck ./internal/releasecheck` and `git diff --check` passed; -manual review found no CI wait or broad-ref publication path. - -Read .woodpecker/verify.yml, .woodpecker/release.yml, .woodpecker/shuffle.yml, -the doccheck workflow graph tests, and docs/development.md. - -1. Preserve asynchronous tag CI and its reuse of the candidate checker and - checked-in note. Revalidation on a separate CI runner is independent delivery - evidence, not a second synchronous release prerequisite. No CI-result polling. -2. Keep existing push/PR and scheduled race/shuffle responsibilities; broad CI - optimization is outside this scope. Remove only redundant standalone - documentation/example steps already covered by that workflow's full suite, - repairing depends_on edges and corresponding documentation/tests. Do not - change runner labels, platform targets, or toolchain versions. -3. Keep tag builder/publication dependencies and non-overwrite settings intact. - The tag checker may build temporary validation assets and the subsequent - build step creates upload assets; do not introduce cross-job artifact caching - or a new combined publisher just to remove this delivery separation. -4. Distinguish ordinary developer checks, optional focused diagnosis, and the - normal release entry point in documentation. Do not imply checks removed - from the release sequence are forbidden during development. - -Verification: doccheck workflow graph validation, relevant release tests, and -manual inspection that no execution path waits for CI or publishes broad refs. -No production CI configuration experiments or network publication are authorized. - -## Stage 7: Verify final alignment and close the roadmap - -Status: Complete (2026-09-28); release pending. - -Verification: Compared all seven policies with pinned upstream Git objects: -testing and meta common text match after metadata/template removal; release -common text matches with only the selected local validation section; agents, -documentation, architecture, and OSS differences are the reviewed local -customizations. Confirmed seven per-file source/version/date stamps, no retained -template sections, attribution material scope including release policy, local -license texts and pinned notices, README navigation, and immutable-tag footer -instructions. The retained checker and publication tests cover selected gates, -candidate identity, one validation call, exact ref publication, failure -uncertainty, and asset safety. `GOWORK=off go test -mod=readonly -race -count=1 -./...`, `GOWORK=off go vet -mod=readonly ./...`, `GOWORK=off go mod tidy --diff`, tracked Go formatting, shell syntax, documentation checks through the -suite, `git diff --check`, and a synthetic three-target asset build with host -version smoke all passed. No real publication, tag, push, or historical release -rewrite occurred. - -1. Compare all seven adopted policies against their exact pinned upstream files. - Confirm removed template instructions, correct source/version/date metadata, - verbatim common wording, and justified local content. Check orientation and - ownership, not just file presence. No metadata-only claim of full adoption. -2. Review attribution scope, modification descriptions, public source/notice - commits, license texts, and README/release navigation. Ensure upstream example - collections/guides were linked rather than copied. Do not expand the release - gate set to automate this one-time adoption review. -3. Run the complete race-enabled suite, vet, shell syntax checks, diff checks, - and the existing asset builder with its supported synthetic test version into - a temporary directory. Doccheck and maintained examples run through the suite; - reuse successful checks for unchanged content. Run tidy -diff and formatting - checks as selected above. This is verification, not publication: do not create - a fabricated release note or invoke release.sh on the real repository. -4. Audit test meaning: retained protections still cover candidate identity and - publication safety; removed checks are redundant or incidental; no new - contracts, approvals, provider calls, or live-service tests slipped in. -5. Mark this plan and the feature roadmap implemented with actual evidence and - note release pending. Do not remove unrelated roadmaps or historical audits. - Commit final coherent documentation/test corrections and report clean status. - -## Open Questions - -None requiring a product decision. The gate set and local customizations above -resolve the roadmap's planning choices. If source inspection reveals a concrete -rights question outside the adopted arrangements, stop only the affected item, -record the evidence, recommend a treatment and a viable alternative, and obtain -direction rather than inventing attribution or adding a blanket prohibition. diff --git a/docs/roadmap/notarius-config-profile.md b/docs/roadmap/notarius-config-profile.md deleted file mode 100644 index 41f8cba..0000000 --- a/docs/roadmap/notarius-config-profile.md +++ /dev/null @@ -1,151 +0,0 @@ -# Notarius Configuration Profile Selection - -Status: Implemented. - -## Goal and Authority - -Implement the consumer behavior in the -[feature request](feature-request-notarius-config-profile.yml): let a Narratio -configuration overlay explicitly select a Notarius configuration overlay. -Operators can select testing once in Narratio while each application retains -ownership of its own model choices and other settings. - -This roadmap describes the target state and required change areas, not an -implementation sequence; the [implementation plan](implementation.md) owns -the ordered work and implementation decisions. The feature request is the scope -authority; the mechanisms suggested below do not establish additional -user-facing contracts. - -## Underlying Issue - -Narratio can select its own testing profile, but extraction currently invokes -Notarius without a configuration-profile selector. Consequently, a Narratio -testing invocation cannot explicitly select the independently deployed Notarius -testing overlay through pipeline configuration. - -Adding individual downstream model or tuning fields would duplicate Notarius's -configuration responsibilities. Passing its existing profile selector provides -the requested flexibility without that duplication. - -## Targeted End State - -### Explicit Configuration Composition - -`pipeline.notarius.config_profile` is an optional Notarius configuration-profile -name, usable in Narratio's base configuration and ordinary overlays. For -example, Narratio's `profiles/testing.yml` may contain: - -```yaml -notarius: - config_profile: testing -``` - -Selecting Narratio's `--profile testing` then selects that explicit downstream -profile whenever extraction executes. The two profile names need not match. -Narratio does not automatically forward its own profile name. - -When supplied, the selection is passed through Notarius's `--config-profile` -argument. Omitted or null values add no profile-selection flag and preserve -existing downstream selection behavior. Notarius's selector semantics remain -authoritative, including empty and `default` selections for the composed base -configuration. Narratio does not introduce a stricter profile-name policy. - -This field selects a ConfigKit configuration overlay, not a PromptKit profile -through `--llm-profile`. The Notarius overlay owns any resulting LLM-profile -selection. Notarius resolves overlays relative to its own configuration root; -Narratio neither resolves those files against its own root nor rewrites them. - -### Execution, Reuse, and Failure - -A changed downstream profile selection must not silently reuse Narratio's -completed extraction from the previous selection. Existing dependent-stage -invalidation applies when extraction changes; the feature does not change the -pipeline order or transcript-generation behavior. - -Notarius continues to own downstream overlay discovery, composition, validation, -and cache compatibility. A missing or invalid selected overlay must fail -extraction through the existing subprocess error path, without silently falling -back to production or reporting successful extraction. - -Existing resume and chunk-cache controls, including force precedence, remain -in place. Profile selection does not itself mean forced regeneration or cache -deletion. Narratio's decision to execute extraction and Notarius's decision to -reuse compatible work remain separate responsibilities. - -Changes to private Notarius configuration behind an unchanged selector retain -the existing operator responsibility to force extraction when needed. There is -no new downstream file inspection or content-tracking requirement. - -Credential forwarding remains independently configured through -`notarius.llm_api_key`. A Narratio testing overlay may select a different -credential reference alongside the downstream profile; no new credential -discovery or provider-name mapping is introduced. - -## Required Change Areas - -### Configuration and Inspection - -Extend the typed Notarius configuration and its existing safe effective-YAML -projection to carry the optional field. Preserve the ordinary configuration -composition behavior for base values, overlay overrides, and null values. -Configuration inspection must reflect the selected field without exposing -credential values. No replacement configuration loader or downstream preflight -loader is needed. - -### Stage and Adapter Boundary - -Carry the selection from resolved configuration through the extraction request -to the subprocess adapter. Construct the optional CLI flag using the existing -argument-vector mechanism, alongside the current configuration path, references, -credential forwarding, and cache controls. Retain normal Notarius error handling. - -### Extraction Reuse Evidence - -Account for the downstream selection in the existing extraction reuse evidence. -The current semantic-configuration and invocation fingerprints are the relevant -change points. Keep their treatment of selection consistent so that a profile -change cannot bypass the required rerun and downstream invalidation. - -This roadmap does not require a new manifest schema, separate profile-indexed -workspaces, historical per-profile result retention, or an upgrade-wide forced -rerun. - -### Verification and Documentation - -Use focused behavioral coverage for configuration composition and inspection, -explicit argument forwarding and omission, changed-selection reuse decisions, -and ordinary subprocess failure handling. Reuse existing test coverage where -it already protects unchanged credentials, force behavior, and invalidation; -do not duplicate Notarius's complete profile-validation or cache test suites. - -When implemented, update the canonical -[configuration reference](../config.md), -[Notarius integration contract](../integrations/notarius.md), -[extract internals](../internal/stage-extract.md), and relevant maintained -examples. The development workflow in [Operations](../operations.md) should -explain explicit profile composition and the existing need to force extraction -after private downstream edits. Clarify that profile selection does not create -an isolated testing workspace or separate current extraction state. - -## Compatibility and Scope Boundaries - -Notarius v0.7.0 already documents `--config-profile`; no upstream change is -required. Document the capability needed when the option is used, without adding -an unconditional runtime version gate for existing unconfigured deployments. - -This feature adds no Narratio CLI selector, automatic profile propagation, -Notarius LLM-profile selector, downstream configuration rewriting, new -credential mechanism, or generalized recursive orchestration. Existing output -and receipt contracts remain unchanged. - -The consumer's Ansible role and suite updates are follow-up work outside this -repository. They can deploy downstream overlays and populate this field after -the interface is released. - -## Scope Review - -No stronger requirements or new invariants are proposed beyond the feature -request and existing repository contracts. In particular, minimizing one-time -upgrade invalidation and avoiding reruns between equivalent base selectors are -implementation considerations, not additional acceptance guarantees imposed -by this roadmap. No user decisions are currently required before planning. diff --git a/docs/roadmap/policy.md b/docs/roadmap/policy.md deleted file mode 100644 index bf8d9ad..0000000 --- a/docs/roadmap/policy.md +++ /dev/null @@ -1,217 +0,0 @@ -# Align Narratio With Standard Repository Policies - -Status: Implemented (2026-09-28); release pending. - -Implementation evidence: The [implementation plan](implementation.md) records -completion and verification of each bounded unit. Seven adopted policies are -credited at the pinned source revision; the local release checker, runbook, -publication tests, CI references, and OSS notices now follow the selected -workflow. The full race-enabled suite, vet, module/format/shell checks, and -synthetic release assets passed. No source release or asynchronous asset -delivery was initiated by this implementation work. - -## Goal and Approved Scope - -Align Narratio's adopted policy language with Standard Repository Policies. -Use upstream wording verbatim for non-substantive differences. Retain variations -only for substantive, locally significant requirements or decisions and explain -their justification. Follow template adoption instructions and customize required -project-specific sections rather than copying instructional placeholders. - -This document defines the target and scope; the [implementation plan](implementation.md) -owns sequencing and concrete validation choices. It -supersedes the earlier licensing-only roadmap: public upstream discovery, -copied-policy attribution, and coordinated release-workflow alignment are now -included. The earlier prohibition on policy-project references and deferral of -broader release alignment no longer apply. - -## Fixed Review Revision and Baseline - -The approved review compared current local documents directly with upstream -commit `d84a38ebab26f048cafbe59a2246700c35d29843`, verified against published -`main` on 2026-09-28. Historical adoption baselines need not be reconstructed. -Do not imply that this revision was previously adopted. - -Follow the [adoption guide at the reviewed revision](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/adoption.md). -Use this fixed revision for implementation; incorporating a later revision -requires deliberate comparison rather than an implicit refresh. - -Narratio has five adopted policies without per-document Upstream Source links. -It lacks `docs/policy/meta.md`, `docs/policy/oss.md`, and the OSS attribution -collection. Its development guide has task routes but not the standard selective -policy-reading guidance. COPYRIGHT already grants GPL-3.0-or-later and LICENSE -contains GPLv3; only the separate policy-document licensing scope needs adding. - -The release process already ends at successful tag publication to `origin`, uses -lightweight tags and source-push-before-tag, and publishes Gitea binaries and -checksums asynchronously. Current guidance requires standalone validation before -publication repeats it. The checker requires fixed note sections and repeats -documentation/example tests included in the full suite. These are alignment -work, not automatic justifications for different policy language. - -## Policy Target - -| Document | Target version | Required treatment | -| --- | --- | --- | -| `docs/policy/agents.md` | 1.3 | Adopt upstream wording. Remove redundant general-purpose local elaborations; retain application safeguards in architecture. Existing local-commit behavior remains. | -| `docs/policy/testing.md` | 1.3 | Adopt upstream wording after template removal. Protect requested behavior and established contracts without turning risk categories or examples into new requirements. No local exception is needed. | -| `docs/policy/documentation.md` | 1.11 | Adopt general wording and customize the ownership table. Retain troubleshooting ownership and materially useful stage/integration terminology; omit inapplicable public HTTP/API responsibilities. Add provenance and licensing owners. | -| `docs/policy/architecture.md` | 1.3 | Adopt general framing and customize application-specific sections as instructed. Preserve Narratio's established contracts rather than substituting generic examples. | -| `docs/policy/release.md` | 1.4 | Adopt upstream wording with coordinated executable changes. Tag-push completion and authoritative-remote scope are now standard, not local exceptions requiring compensating controls. | -| `docs/policy/meta.md` | 1.0 | Add local authority, public discovery, adoption-guide navigation, and per-document provenance guidance. | -| `docs/policy/oss.md` | 1.8 | Add current House Format, copied-document treatment, maintenance, and reference-example links; customize project/distribution facts. | - -No existing policy is wholly unchanged because provenance and wording require -updates. Narratio's architectural decisions and agent commit behavior need no -substantive change. Wording preferences alone do not justify local variants. - -## Provenance and Orientation - -Remove each adopted document's `Base Policy Template` section and replace other -template-only guidance with substantive project content where required. Retain -the actual Base Policy Version, set Last Updated to the adoption date, and add -Upstream Source pointing to that individual file at the reviewed commit: - -```text -https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/FILE.md -``` - -Substitute the correct filename for each policy. Preserve accurate provenance in -later edits; Git records suffice without a new adoption registry or sync service. - -Adapt upstream `agents/AGENTS.md` into root `AGENTS.md`, preserving applicable -existing instructions. Do not copy upstream's root agent instructions, which -govern work on that repository. Incorporate the supplied selective Policy Reading -Guide into `docs/development.md`: select by task, read before governed decisions, -reuse available context, follow relevant cross-references, expand reading when -scope expands, and read affected provisions when revising policies. Do not make -reading the whole suite mandatory. Preserve useful Narratio-specific task routes. - -Meta links to the [public project](https://gitea.maximumdirect.net/eric/policy) -and [current adoption guide](https://gitea.maximumdirect.net/eric/policy/src/branch/main/docs/adoption.md). -These are current-discovery links; adopted-source and example links stay -commit-pinned. Reading triggers belong in development guidance and formal credit -in attribution, not duplicated in meta. - -## Justified Local Customizations - -Preserve established stage/adapter boundaries, manifest-driven resume/restore, -artifact/path contracts, publication and lock semantics, credential protections, -group-owned workspace behavior, and platform support. These express Narratio's -operational responsibilities; policy alignment does not authorize changing them. - -Retain Linux amd64/arm64 and best-effort macOS arm64 deliverables, without new -native-platform gates or Windows/Intel Mac support. Preserve `origin/main`, -lightweight tags, and source-push-before-tag in the release procedure. These are -permitted project-specific mechanics, not exceptions to tag-push completion. - -Customize documentation ownership for real Narratio interfaces and responsibilities, -including troubleshooting; do not create empty API/consumer documents. Keep common -prose verbatim around these customizations and explain substantive local choices -in their canonical owner rather than maintaining a parallel general policy essay. - -## Licensing, Attribution, and Local Materials - -Create `docs/oss/attribution.md` with the prescribed title followed directly by -factual component entries. Use the canonical Standard Repository Policies entry -from the reviewed commit. Identify actual copied/adapted material, including policy -files, adapted agent instructions, and copied reading-guide material as applicable. -Preserve creator, copyright, license, disclaimer, and modification notices. Pin -source, COPYRIGHT, and LICENSE links to the reviewed full commit. - -Copy CC-BY-SA-4.0 into `docs/oss/licenses/`. Add the limited policy-document scope -to COPYRIGHT with an attribution link: adapted policy material remains CC-BY-SA-4.0; -Narratio's application code remains GPL-3.0-or-later. Root LICENSE needs no -replacement. Original upstream licensing-example formatting is CC0 and alone -does not require attribution. - -Complete the previously agreed inventory of incorporated/distributed material: -relevant transitive dependencies, Go runtime material in shipped binaries, and -copied code, documentation, or assets. Inspect selected sources and actual use; -neither direct dependencies alone nor every development tool defines scope. -ConfigKit v0.4.1 is a dependency. PromptKit is not in Narratio's module graph; -invoking a separately installed downstream application does not itself incorporate -that application's dependencies into Narratio. - -Follow OSS v1.8's ordered fields and presentation: project heading, applicable -faithful copyright statements, applicable SPDX expression, license/notice links, -and a link labeled `Source` to the exact commit, ending with a period. -Optional identification remains factual. Preserve -alternatives, exceptions, and required notices; do not invent identifiers or add -research narratives, compliance conclusions, or exception reasoning to entries. - -Copy applicable complete standard texts locally and make companion texts directly -reachable. Link project-specific notices and sources at exact imported commits, -not moving branches or tags. Apply the House interpretation of notice links -without blanket additional local-copy requirements. Distinct concrete obligations -still receive appropriate treatment; preserve notices in incorporated source. - -Keep upstream's README, root AGENTS.md, development/adoption guides, and example -collection upstream. Retain commit-pinned example links in the adopted OSS policy; -create actual local notices instead of copying the example collection. Adopted -policies, project orientation, actual attribution, and applicable standard texts -belong locally. Do not create placeholders for template-only links. - -End README with the prescribed grant and attribution links. Future release notes -and hosted pages end with `COPYRIGHT | LICENSE | ATTRIBUTION` links at their -immutable release tag or source revision. The procedure owns concrete Gitea URLs; -continue using the checked-in note as the hosted body. Maintain attribution with -relevant dependency/material changes through ordinary review. - -## Coordinated Release Alignment - -Update release policy, `docs/release.md`, relevant scripts, tests, and development -guidance together so rules and executable behavior agree: - -- Make publication the normal validating-and-publishing entry point. Standalone - validation remains optional rather than a mandatory duplicate prior pass. -- Remove fixed release-note section requirements. Preserve version-matched notes - and titles, relevant compatibility/upgrade information, and the licensing footer. -- Remove duplicate documentation/example test invocations covered by the full - suite. Avoid repeated validation of the same final candidate; no persistent - validation cache is required. -- Select additional gates by concrete risk and distinct protection. Do not keep - race, static, module-hygiene, formatting, or other checks merely because they - exist; document reasons for retained checks. Do not remove useful checks merely - to minimize command count. The implementation plan records the selected checks - and their distinct justification under these criteria. -- Keep offline default tests, required build targets, fresh-version checks, - clean candidate/exact source/tag guards, and explicit-ref publication. -- Treat ordinary review and roadmap retirement as preparation, not new human - approval rounds. Evaluate incidental file prohibitions by their impact on tagged - source rather than banning ignored local files solely because they exist. -- End at successful tag push, without CI polling, asset-availability gates, or - successful-push read-back requirements. Resolve failed/ambiguous pushes from - remote evidence and report asynchronous delivery separately. -- Align recovery guidance: transient delivery failures alone do not require new - versions. Retries/missing-asset additions must preserve published content and - checksum consistency; source/content corrections require a new release. - -Expected implementation surfaces include `scripts/check-release-candidate.sh`, -`scripts/release.sh`, shared helpers/builders where needed, and release tests. -Review workflow references for consistency without expanding into unrelated CI -repair or application behavior changes. - -## Boundaries and Completion Evidence - -No application relicensing, automatic copyright-year advancement, blanket headers, -dependency upgrade, runtime feature, or platform expansion is included. Do not add -mandatory scanners, a full SBOM, recurring legal sign-offs, publication-time network -checks, or release approval gates. New downloadable license assets or packaging -formats are not required merely to implement the conventions; concrete component -obligations still require their applicable treatment. - -Apply formatting prospectively. Preserve published tags, notes, binaries, and -checksums. Address any discovered historical omission deliberately under the -applicable terms and correction policy, not by silently rewriting releases. - -Completion requires comparison with the pinned source, accurate per-file metadata -and material attribution, justified variations, substantive supporting documents, -valid navigation, and complete applicable texts. Release procedures and tests must -prove the selected workflow and guards without duplicate mandatory validation or -CI waiting. Documentation checks and relevant release tests must pass. Merely -creating Markdown files is not evidence of alignment. - -No product-level decision remains open from the approved review. Concrete rights -questions or gate tradeoffs unresolved by the risk-based approach must be surfaced -with evidence during planning, not silently converted into stronger requirements. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index 80af85a..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,227 +0,0 @@ -# Audita v2 and ConfigKit Migration Review Remediation - -Status: Complete. - -Reviewed `83eaa96` through `fae18bf` against the approved clean-cutover decisions, -[feature roadmap](audita.md), and [implementation plan](implementation.md). -Review date: 2026-09-25. No production changes were made. The previous completed -transcription review is retained below as historical context. - -## 1. Preserve omitted modules in direct YAML serialization - -Priority: P2. Status: Complete. - -The safe YAML projection now omits nil modules while retaining explicit empty -and ordered/repeated selections. A round-trip regression failed before the fix -and passes afterward. Config package tests, including credential-safety coverage, -and its race tests pass. - -Location: `internal/config/credential_projection.go:25` (`AuditaConfig.MarshalYAML`), -especially the plain slice field at line 30 and assignment at line 40. - -The direct safe YAML projection passes a nil module slice to yaml.v3, which -serializes it as `modules: []`. Loading that output produces a non-nil empty -slice. Under the new Audita contract these are different instructions: nil -delegates to Audita's configuration/default sequence; empty passes an explicit -empty module argument and disables correction modules. Serialization therefore -silently changes behavior for a configuration that is subsequently reloaded. - -Evidence: a temporary package test marshaled -`AuditaConfig{Binary: "audita", Timeout: "3h"}`, unmarshaled it into another -`AuditaConfig`, and required `Modules == nil`. It failed with output containing -`modules: []`. The test was removed after reproduction. This concerns direct -`yaml.Marshal` of the typed configuration; ordinary CLI `config show` uses -`MarshalEffectivePipeline` through a separate JSON/value-tree projection and -does not use this faulty path. - -Required remediation: - -- Make the direct safe YAML projection preserve nil as null or omission and - explicit empty as `[]`. Do not simply add `omitempty` to the slice: that would - instead erase the explicit empty selection. -- Keep credential serialization name-only and preserve module order/repetitions. -- Add a small round-trip regression covering nil, explicit empty, and an ordered - repeated selection; retain effective-output inspection coverage without - duplicating its whole test matrix. - -## 2. Remove the unapproved manifest compatibility decoder - -Priority: P2. Status: Complete. - -Removed the permissive custom decoder; ordinary strict typed decoding now -applies to provenance as well. Replaced historical-acceptance coverage with -retired/unknown-field rejection tests, all of which failed before removal. -Current provenance round-trip coverage remains intact. Corrected the roadmap, -plan, internal contract, and operator guidance for narrow backed-up migration -without discarding execution/checkpoint state or changing resume authority. - -Location: `internal/manifest/manifest.go:79-89`, -`internal/manifest/store_test.go:78`, and `docs/internal/manifest.md`'s decoding -contract. Also correct the conflicting migration language in the plan/roadmap. - -The new `EffectiveConfigProvenance.UnmarshalJSON` bypasses the manifest store's -`DisallowUnknownFields` by using permissive `json.Unmarshal` for the entire -provenance object. It accepts retired `selected_profile` data and silently -discards it. It also silently accepts unknown current provenance fields, -including unknown nested selection fields, rather than limiting behavior to -the renamed label. - -The user expressly selected a clean cutover without backward-compatibility -shims. This adds precisely such a shim and a new historical-input acceptance -guarantee. The regression test explicitly requires the old selection to load, -and the internal guide now declares permissive informational-field decoding. - -The plan contributed to this divergence: it said existing generic decoding -would handle historical optional provenance and required old manifests to load. -That assumption was false; the baseline store already strictly decoded these -fields. Repetition in the plan, tests, and documentation does not authorize -overriding the user's clean-cutover decision. - -Required remediation: - -- Remove the custom compatibility unmarshaler and retain ordinary strict typed - decoding for the new `selected_overlay` representation. Do not add dual fields, - special retired-field handling, automatic manifest rewrites, or a format-version - bump just for this rename. -- Replace the historical-acceptance test with coverage for current provenance - round trips and unknown/retired provenance rejection through the real store. -- Correct the plan and current documentation to acknowledge the upgrade effect: - manifests with retired provenance require deliberate operator migration before - loading. Document narrow removal/renaming of retired informational provenance - on a backed-up manifest, preserving execution/checkpoint state; do not advise - deleting all session progress or rerunning everything. -- Keep provenance informational: this correction does not authorize adding it - to semantic resume keys or invalidating work based on overlay identity alone. - -If transparent historical-manifest acceptance is desired instead, obtain an -explicit exception to the clean-cutover decision before retaining a compatibility -reader. Do not treat that exception as already granted. - -## Review verification - -The following records the original review, before remediation. - -- Existing `go test ./...` passed. -- Race tests passed for subprocess, Audita, Notarius, config, stage, app, and - manifest packages after removing the temporary reproduction. -- `go vet ./...` passed. -- The temporary direct-YAML regression failed as described above. -- Source inspection and the new historical-provenance test establish finding 2; - `LocalStore.LoadReader` still uses strict decoding outside the custom hook. - -The primary architecture is otherwise in place: ConfigKit v0.4.0 overlays, -shared named credential forwarding/redaction, a v2-only Audita subprocess -adapter, ordinary reports as diagnostics, and versioned ordered-module polish -evidence. No live model/provider runs were performed during this review. - -## Migration remediation verification - -Completed on 2026-09-25. Both new regressions failed before their respective -fixes. After remediation, `go test ./...`, `go test -race ./...`, -`go vet ./...`, `go build ./...`, documentation checks, maintained configuration -example checks, and `git diff --check` passed. No live provider runs were needed. - -## Historical completed transcription review - -Status: Complete. - -Reviewed commits `7b0bf99` through `96b4e73` against the speaker-checkpoint and -upload-replay roadmap and the original user request. No production changes were -made during review. The two failures below were reproduced with temporary tests -using the existing stage fixtures; those temporary tests were removed afterward. - -### 1. Reconcile Aggregate State Even When No Speaker Needs HTTP Work - -Priority: P2. Status: Complete. - -Implemented read-only aggregate identity/evidence checks before resume skips. -Regression tests cover removed inputs, missing/duplicate/mismatched outputs, -repair without HTTP requests, downstream invalidation, and exclusion of removed -speakers from merge without deleting their files. Equivalent path forms and -output ordering remain reusable. Focused app/stage tests and documentation -checks pass. - -Location: `internal/stage/transcribe_resume.go:21` and the reconciliation result -consumed there. - -`ValidateResume` returns reusable whenever `reconciliation.work` is empty. That -only establishes that the remaining speakers need no transcription; it does -not establish that the stored speaker collection and aggregate outputs match -the current input set. Removing a speaker while the remaining inputs are -unchanged produces an empty work list, so a succeeded record can be skipped -without pruning the removed speaker. Its old aggregate output can then remain -available to merge. Likewise, missing/inconsistent aggregate output references -are not repaired when all individual checkpoints remain valid. A prepare rerun -may already stale transcribe in some workflows, but the resume validator cannot -rely on that to satisfy its own current-input/output checks. - -Reproduction: create complete alice/bob checkpoint records and aggregate -outputs with the existing fixture, mark the stage succeeded, remove bob from -the manifest's audio inputs, and call `ValidateResume`. It returns reusable -despite still retaining bob in the stored collection and aggregate outputs. - -Required change: - -- Distinguish no HTTP work from no reconciliation work. Validate that the - committed speaker set and aggregate transcript references represent the - reconciled current set, including relevant identity/evidence, before skipping. -- When only collection/output repair is needed, return non-resumable and let - normal execution checkpoint the pruned collection and rebuild aggregate - outputs without retranscribing reusable speakers. Preserve existing dependent - invalidation; do not force otherwise-current speaker jobs. -- Keep resume inspection read-only. Do not fix this by deleting incidental - transcript files or changing stage selection rules. - -Verification: add removed-only input coverage (not just removal combined with -an added speaker), plus incomplete/inconsistent aggregate output coverage. -Exercise the runner skip path and assert zero additional WhisperX requests, -correct rebuilt outputs, and no removed speaker reaching merge. - -### 2. Remove the Unrequested Positive Audio-Size Requirement - -Priority: P2. Status: Complete. - -Audio evidence now permits zero size while still rejecting negative values. -Regression tests verify zero-size manifest round trips, successful service -results being checkpointed and reused, and service rejection being propagated -without creating a completion record. Nonempty valid-JSON output validation is -unchanged. Both new success regressions failed before the fix. - -Location: `internal/manifest/transcribe_state.go:95`. - -The new record validator requires `AudioSize > 0`. Previously Narratio accepted -a regular `.flac` file and delegated transcription acceptance to WhisperX; -neither the user request nor the roadmap introduced minimum audio size or a new -audio-content validation policy. Zero-byte content still has a valid checksum -and can be represented by the checkpoint mechanism. The current check runs -after a successful response and output materialization, causing the entire -stage to fail instead of retaining the completed work. - -Reproduction: prepare one speaker, replace its prepared audio with an empty -regular file, and use the existing successful fake WhisperX client. Stage -execution fails at checkpoint validation with `audio_size must be positive`. -This demonstrates the extra Narratio restriction; it does not assert that a -particular production WhisperX deployment accepts empty audio. - -Required change: - -- Permit zero audio size and reject negative sizes. Keep byte hashing and - size equality checks. Let the downstream service decide whether the input - can be transcribed, as before. -- Do not remove the nonempty valid-JSON output requirement, which has a separate - existing justification. Do not add a new early empty-audio rejection instead. -- Update any new tests/documentation that promote strictly positive input size - to a contract. - -Verification: cover a zero-size record round-trip and successful adapter result -being checkpointed/reused for that input; retain negative-size rejection and -ordinary downstream-failure tests. - -### Verification After Remediation - -Completed on 2026-09-22 with Go 1.26.8: focused stage/application tests, -changed-package race tests, `go test ./...`, `go test -race ./...`, -`go vet ./...`, `go build ./...`, `go test ./internal/doccheck`, -`go test ./internal/config -run '^TestExamplesLoadAndValidate$'`, and -`git diff --check` all passed. Existing replay and cancellation regression -tests remain intact; the HTTP adapter required no changes. diff --git a/docs/roadmap/transcribe-checkpoints-and-replay.md b/docs/roadmap/transcribe-checkpoints-and-replay.md deleted file mode 100644 index 98959d1..0000000 --- a/docs/roadmap/transcribe-checkpoints-and-replay.md +++ /dev/null @@ -1,183 +0,0 @@ -# Resumable Speaker Transcription and Replayable Uploads - -Status: Implemented. - -## Goal and User Intent - -Preserve completed speaker transcriptions when another speaker fails, so an -ordinary run or resume does not repeat expensive successful work. Separately, -allow Go's HTTP transport to replay WhisperX uploads when it encounters a -recoverable connection or stream failure. - -The motivating incident exhausted four transcription attempts for one speaker -after three other speakers had completed. The final error reported an HTTP/2 -peer `PROTOCOL_ERROR` and an inability to replay the request body without -`Request.GetBody`. - -These are related reliability improvements, not a promise to eliminate all -service failures. This document records the target state and scope. The -completed implementation sequence is retained in Git history; current behavior -is documented in the canonical guides linked below. - -## Pre-implementation Limitations - -The [transcribe stage](../internal/stage-transcribe.md) writes successful -responses into the invocation's run directory. It installs canonical transcripts -and returns its per-file metadata only after every speaker succeeds. A failure -therefore leaves no durable per-speaker progress that ordinary resume can use, -even though some completed JSON files may remain on disk. - -The [WhisperX adapter](../integrations/whisperx.md) already retries requests at -the application level, reopening audio for each attempt. Its multipart request -body is streamed through a pipe and has no replay callback. Transport-level -recovery within an attempt cannot reconstruct a consumed body. Providing that -capability addresses the replay limitation, not the underlying peer protocol -error; persistent failures can still require service or proxy investigation. - -## Targeted End State - -### Durable Per-Speaker Completion - -Each successfully transcribed speaker file is validated, saved, and recorded -durably before that work is treated as reusable. The session manifest remains -the progress authority. Follow the existing analyze-stage checkpoint pattern -without turning speakers into pipeline stages or introducing a general task -engine. - -Ordinary execution reconciles the current prepared audio set with recorded -speaker results. It reuses valid completed work and schedules failed, -unattempted, missing, or obsolete results. A result is not reusable merely -because a matching filename exists. Evidence binds it to the speaker/audio -identity, audio content, Narratio-visible recognition settings, and the saved -transcript's integrity. - -The relevant recognition settings remain the WhisperX service URL and language. -Retry count/delay, concurrency, timeout, credentials, and diagnostic locations -remain operational settings rather than reasons to discard completed results. -Changes to private service models or configuration remain an operator reason -to force transcription; Narratio does not inspect the service's private state. - -Changes to individual audio inputs require replacement of affected results, -not automatic retranscription of unrelated unchanged speakers. New speakers -need transcription; removed speakers do not remain in the current aggregate -output set. Records must describe the inputs actually used, not grant reuse to -old bytes under a changed identity. - -### Aggregate Completion, Failure, and Force - -The overall stage succeeds only when every currently required speaker has a -validated, durably recorded result. Incomplete per-speaker progress does not -make the aggregate stage successful or authorize downstream consumption of an -incomplete collection. Existing dependent-stage invalidation remains in effect. - -Retain bounded concurrency and the current fail-fast behavior: a speaker -failure cancels outstanding work. Results already checkpointed remain reusable -after failure, cancellation, or process interruption. A later ordinary run or -resume retries the remaining work; it does not need a new CLI command. - -Checkpoint writes must be serialized so concurrently finishing speakers cannot -overwrite each other's progress. A failed persistence operation must not -advertise an unrecorded result as complete. Previously committed progress remains -the recovery authority, and checkpoint errors must be surfaced rather than -reported as successful transcription. - -`--force` on transcribe regenerates all currently required speakers instead of -reusing their prior completions. Successful replacements are checkpointed too, -so an ordinary retry after a failed forced invocation can retain the newly -completed work. Old results for speakers not successfully regenerated must not -satisfy that unfinished forced replacement on the next ordinary retry. - -This deliberately relaxes the current all-speakers-before-canonical-installation -implementation: individual durable results may exist while the stage remains -failed or incomplete. The all-speakers-before-stage-success boundary remains. - -### Replayable WhisperX Requests - -Provide `Request.GetBody` so the HTTP transport can obtain a fresh body for the -same request. Replay reopens the audio and reproduces the multipart boundary, -filename, fields, and framing used by the original request. Independent bodies -must not share a consumed reader or mutable read position. - -Preserve bounded-memory streaming rather than buffering entire audio files in -memory. Correctly close every original and replay body and its audio resources; -preserve source-read errors and cancellation, including partial uploads and -early server responses. Successful response handling must not accidentally wait -on, or report errors solely from, an abandoned body instead of the replay that -produced the response. - -Keep the existing application-level retry count, delay, classification, and -per-attempt timeout semantics. Transport replays occur within an application -attempt and its context; they are not additional Narratio-configured attempts. -Do not add another custom retry loop, silently increase retries, or disable -HTTP/2 globally as a substitute for replay support. - -The adapter still validates and installs only successful JSON responses. -Request replay does not provide exactly-once server execution: a lost response -can cause repeated transcription work, as can existing application retries. -No new service-side idempotency protocol is required by this feature. - -## Required Change Areas - -- **Manifest and persistence:** Add transcribe-owned per-speaker progress and - evidence, with safe checkpoint updates and preservation through runner failure - handling. Review manifest validation, copying, serialization, and existing - publish/restore handling wherever the new records pass through those paths. -- **Stage execution and reuse:** Reconcile speaker records before scheduling, - checkpoint validated completions, rebuild aggregate outputs from the current - set, and account for partial progress in ordinary and forced execution. - Review successful-stage reuse as well as failed-stage retries so stale audio - or missing/corrupt transcript evidence is not bypassed by an aggregate skip. -- **Artifact lifecycle:** Use existing path, installation, integrity, and manifest - mechanisms. Ensure checkpointed results do not depend on an ephemeral run - directory surviving cleanup. Do not introduce a second authoritative ledger - or infer completion by scanning abandoned runs. -- **HTTP adapter:** Make multipart bodies reproducible and correctly manage - every replay's lifetime, without changing the WhisperX HTTP schema or requiring - an upstream service change. -- **Tests:** Cover per-speaker recovery after a later failure, interruption and - checkpoint failure; changed/missing inputs or outputs; force followed by an - ordinary retry; concurrent successful completions; and aggregate downstream - gating. Extend existing fixtures rather than duplicating analyze's complete - test suite. Exercise actual body reconstruction after partial consumption, - multipart equivalence, cancellation/resource cleanup, and existing bounded - streaming and retry behavior—not just the presence of a callback. Default - tests remain offline and use local/fake HTTP boundaries, never paid services. -- **Documentation:** Update the canonical - [stage guide](../internal/stage-transcribe.md), - [manifest guide](../internal/manifest.md), - [WhisperX contract](../integrations/whisperx.md), - [operations guide](../operations.md), and - [troubleshooting guide](../troubleshooting.md) when implemented. Explain - partial completion, ordinary retry versus force, and the distinction between - request replay limitations and persistent peer protocol failures. - -## Compatibility and Scope Boundaries - -Existing speaker identities, raw transcript formats, and the fixed pipeline -order remain unchanged. Existing manifests remain readable, but must not be -treated as containing per-speaker evidence they never recorded. Recovery relies -on verified recorded evidence, not guessed associations between old transcripts -and current audio. No general legacy-migration framework is part of the feature. - -Automatic adoption of unrecorded outputs from previously failed runs is outside -this feature. Such files may be recoverable after verifying their inputs, but -their presence alone does not prove completion. No recovery/import command, -speaker-selection CLI, historical invocation ledger, new service configuration -discovery, or broader orchestration refactor is included. - -The feature does not diagnose the remote service/proxy's underlying HTTP/2 -problem, guarantee recovery from persistent protocol errors, or change the -policy for continuing unrelated work after failure. - -## Completion Criteria - -For speakers A, B, C, and D: if A and B are checkpointed, C fails, and D has not -completed, ordinary resume retains A and B, retries C, and completes D. No -downstream stage receives an incomplete successful transcription collection. -Force replaces all four, while an ordinary retry can reuse checkpoints from -that forced replacement attempt. - -The HTTP transport can reconstruct the upload after consuming the original -body, without whole-file memory buffering, lost source errors, or stranded -upload resources. A persistent peer error still fails normally after the -configured application attempts, without discarding checkpointed speakers.