diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md index b204926..270a3cc 100644 --- a/docs/roadmap/configkit.md +++ b/docs/roadmap/configkit.md @@ -57,6 +57,10 @@ explicitly coordinated change. ## Provisional Stages +First complete the [shared-boundary and prototype unit](library-boundary.md) from +the coordinated adoption plan. It supplies the target settings model and integration +evidence for the production configuration migration described here. + ### 1. Define the Contract and Coordinate Sequencing Inventory existing configuration fields, invocation options, environment settings, diff --git a/docs/roadmap/library-adoption.md b/docs/roadmap/library-adoption.md index 78896c6..b434e46 100644 --- a/docs/roadmap/library-adoption.md +++ b/docs/roadmap/library-adoption.md @@ -42,6 +42,9 @@ changes, especially changes to module or pipeline configuration. ## Proposed Sequence +The [shared-boundary and prototype roadmap](library-boundary.md) details the next +unit of work, covering stages 1 and 2 below before production migration. + ### 1. Design the Shared Configuration Boundary Define the target settings model for both libraries before implementing either diff --git a/docs/roadmap/library-boundary.md b/docs/roadmap/library-boundary.md new file mode 100644 index 0000000..3a71551 --- /dev/null +++ b/docs/roadmap/library-boundary.md @@ -0,0 +1,211 @@ +# Library Adoption: Shared Boundary and Prototype + +**Last updated:** 2026-09-23 + +**Status:** Detailed proposed roadmap for the next unit of work. Planning only; +no migration implementation has started. Review the proposed choices below before +execution. This document does not authorize publication or changes to upstream +libraries. + +## Objective + +Define and prove the boundary between Audita, ConfigKit, and PromptKit before +replacing Audita's production configuration or LLM runtime. This unit covers the +shared design and representative prototype in stages 1 and 2 of the +[coordinated adoption roadmap](library-adoption.md). The +[ConfigKit roadmap](configkit.md) continues to own the broader configuration scope. + +The deliverable is a concrete target contract supported by an offline executable +prototype, sufficient to implement the ConfigKit migration without subsequently +redesigning its LLM-facing settings. A successful prototype is not completion of +either production migration. + +## Established Direction + +- This is an intentional breaking migration with a migration guide, not legacy + configuration support, aliases, or a permanent compatibility mode. +- ConfigKit owns shared configuration mechanics; PromptKit owns shared prompt and + generation mechanics; Audita retains domain meaning and transcript correction. +- Structural repair is enabled from the outset for proposal and LLM-validator calls. + Positive default repair budgets must be selected, not silently left at zero. +- Transient retries and structural repair are separate. Explicit zero disables + transient retries; Audita must not add a second automatic retry loop. +- One finite concurrency limit per backend protects local capacity and bounds paid + API activity. Calls from both roles share that limit. Fine-grained role budgets + and an additional global concurrency control are unnecessary. +- Keep the sequence: boundary/prototype, ConfigKit migration, PromptKit migration, + then combined major release. No intermediate public release is planned. + +## Scope and Deliverables + +| Deliverable | Required result | +| --- | --- | +| Configuration inventory and migration map | Every existing setting and invocation option has a proposed destination, replacement, or removal, with rationale. | +| Boundary specification | One authoritative source for each setting, explicit precedence/defaults, credential flow, runtime conversion, and failure handling. | +| Representative executable prototype | New-style configuration drives one proposal and one assessment through released libraries and a controlled provider. | +| Focused behavioral evidence | Retry, repair, capacity, credential, observation, and domain-boundary scenarios below are demonstrated. | +| Decision and integration handoff | Concrete choices, measured limitations, affected components, removal plan, and next implementation tasks are recorded. | + +Do not switch the production CLI to the prototype, migrate every module/prompt, +redesign transcript/glossary formats, change correction algorithms, modify retention +or timestamp behavior, or publish a release in this unit. Do not reproduce the +libraries' full test suites. Upstream changes require a separately scoped task if +an actual gap is found. + +## Work Package 1: Inventory and Decide Ownership + +Read the current [configuration](../config.md), [CLI](../cli.md), +[LLM runtime](../internal/llm-runtime.md), [prompts](../internal/prompts.md), +[validators](../internal/validators.md), and [output contracts](../integrations/output-files.md). +Use graph-first discovery to verify the relevant code and tests, especially +configuration application, prompt construction, structured responses, scheduling, +and report assembly. + +Build a migration table grouped by application/domain settings, invocation state, +backend settings, execution profiles, output contracts, and inspection. Include +current environment settings and aliases without treating their existence as a +requirement to preserve them. Identify report fields whose meanings change when +one logical generation contains multiple provider attempts. + +Resolve these design decisions explicitly: + +| Decision | Proposed direction to prove | +| --- | --- | +| File selection and version | Audita selects the main configuration and validates a new schema version; specify discovery and the treatment of old versions. | +| Application versus invocation state | ConfigKit loads persistent settings; Audita parses invocation-only input/output paths and translates explicitly supplied setting flags into supported overrides. | +| Backend identity | One registration represents the service capacity shared by proposal and validation calls. Avoid separate identities for the same service merely because roles differ. | +| Endpoint and capacity ownership | Define them once per backend, with finite limits and an explicit policy for waiting capacity/exhaustion. Do not expose endpoint-only unlimited execution accidentally. | +| Execution profiles | Prefer complete application-owned profiles using PromptKit conventions. Decide their source and selection without duplicating model/settings fields in application configuration. Catalog selection remains a deliberate option. | +| Role selection and inheritance | Specify how proposal and validation choose profiles, including omission and any inheritance. Do not inherit old behavior solely for compatibility. | +| Credentials | Prefer one ConfigKit-owned secret-resolution path passed through PromptKit's supported request credential field. Define backend/profile credential association and avoid competing environment lookups. Validate this against profile requirements. | +| Retry, repair, and deadlines | Assign one owner and explicit defaults for each control. Use positive structural-repair defaults and document combined limits and total-operation scope. | +| Optional library features | Leave ConfigKit imports, configuration profiles, and secret-directory bootstrap out unless a concrete requirement is identified. These are separate from PromptKit execution profiles. | +| Inspection | Define effective configuration and selected-profile inspection without implying that one library's snapshot contains the other's complete state. | + +Choose concrete field names, example values, defaults, units, and omission/null/zero +semantics for the prototype. In particular, decide what validation and inspection +do when configured credentials are unavailable. Classify choices as proposed, +selected, or unresolved; do not present candidates as implemented contracts. + +## Work Package 2: Specify Execution and Reporting Semantics + +Document the flow from selected configuration through backend/profile assembly, +prompt inputs, generation, structural validation/repair, typed decoding, and Audita +domain validation. Identify which values cross the boundary and which component +owns each conversion. Keep library types out of unrelated domain code where a +small existing domain value is sufficient. + +Use the current correction-set and validator-decision schemas as representative +inputs. Reuse domain checks where practical. Select one real proposal/validator +pair; account for prompt helpers, template variables, shared fragments, and schema +references rather than testing only an unrelated hello-world prompt. + +Record the following distinctions: + +- Repair seeks output conforming to its structural contract. Success does not + establish semantic correctness or authorize applying a correction. +- A schema-valid decision set still requires Audita's cardinality and proposal + identity checks. Semantic rejection is not automatically another repair request. +- Exhausted structural repair yields failed content validation; provider failures, + retry exhaustion, cancellation, and capacity rejection have separate outcomes. +- Specify each outcome's intended effect on an Audita section/batch and overall run, + including warnings, rejected work, partial domain results, and terminal failures. + Preserve domain intent without preserving obsolete transport behavior by accident. +- For transient retry budget T and structural repair budget R, one logical call can + dispatch up to `(T + 1) * (R + 1)` provider attempts. For example, T=2 and R=2 + permits nine attempts; these illustrative values do not select production defaults. +- Choose the total-operation deadline scope for a logical proposal or assessment; + distinguish it from per-attempt timeouts and whole-process cancellation. Provider + retry timing may exceed local backoff, so deadline behavior must be explicit. + +Design the Audita observation record and aggregation rules. Include logical-call +and attempt identity, role/backend correlation, queue/generation/retry timing, +repair and retry counts, outcomes, and available usage. Preserve unknown/incomplete +usage rather than inventing zeros or complete costs. Do not add terminal totals to +attempt totals. Collect events safely across concurrent calls, without blocking +callbacks on progress by the same run or persisting raw library events as a stable +Audita schema. Transcript-bearing artifacts remain separately governed by Audita. + +## Work Package 3: Build the Bounded Offline Prototype + +Use ConfigKit v0.3.1 and PromptKit v0.11.0 as reviewed starting points; verify their +published availability and APIs before pinning the prototype. Recheck the combined +Go minimum (the reviewed PromptKit release requires Go 1.25.5). Do not depend on +sibling checkouts, an active Go workspace, or committed local module replacements. + +A proposed home is `examples/library-adoption/` as an explicitly experimental, +standalone Go module, isolating prototype dependencies from the production binary. +Confirm this placement during execution. Its README must give exact offline run +and test commands after dependency acquisition; root `go test ./...` must not be +claimed to cover a nested module. Keep complete proposed configuration and prompt +fixtures with it and clearly label them as prototype-only. + +The prototype should: + +1. Load representative application settings with ConfigKit, using synthetic secret + values supplied by the test environment and an application-owned version rule. +2. Assemble shared backend registrations and proposal/validation profile selections. +3. Supply representative transcript/proposal inputs to PromptKit definitions and + actual response schemas using supported sources. +4. Use the built-in HTTP generation path with a controlled local endpoint or + transport. An injected LLM client alone cannot prove built-in HTTP retry behavior. +5. Run proposal generation, decode and check candidates, run assessment, and perform + the selected Audita decision checks without modifying real transcript files. +6. Collect selected safe observations and show a compact result understandable to a + reviewer, including repaired output and final domain acceptance/rejection. + +No paid calls, real credentials, host environment assumptions, persistent caches, +or general-purpose test framework are needed. Make concurrency and cancellation +checks deterministic through controlled synchronization. Keep delays bounded and +use library-supported controls; do not patch library internals to make tests pass. + +## Focused Verification Matrix + +| Scenario | Evidence required | +| --- | --- | +| Successful assembled path | ConfigKit-loaded selections drive both real prompt roles through PromptKit and Audita checks. | +| Presence, overrides, and inheritance | Representative omission, explicit values, and role selection follow the proposed contract; unknown fields and unsupported versions fail clearly. | +| Credentials | Required missing/empty references fail as intended; controlled calls receive the chosen synthetic credential; inspection and observations do not expose it. | +| Structural repair succeeds | Invalid proposal output and invalid assessment output are repaired in their respective workflows; positive budgets are actually exercised. | +| Repair exhausted | Repeated invalid output produces the selected Audita outcome and accurate accounting, without an unbounded outer retry. | +| Domain rejection | Structurally valid but unacceptable correction or decision is rejected by Audita, demonstrating the semantic boundary. | +| Transient retry and explicit zero | An eligible temporary status recovers; zero makes one attempt. Exhaustion retains a useful classified cause. | +| Combined repair and retry | A repaired generation encountering a transient failure respects both budgets and reports distinct counts. | +| Shared capacity | Concurrent calls from both roles, including retry/repair activity, stay within one selected backend cap; test a representative local cap of four and a finite API cap. | +| Waiting, cancellation, and deadline | Cancellation/deadline stops pending work and retry waits; capacity exhaustion has the intended application outcome and later calls can still proceed. | +| Failed-run observations | Completed work remains observable after later failure; unknown usage, known zero, and aggregate counts are mapped correctly. | + +Tests should protect Audita's integration decisions and important assembled behavior. +Rely on PromptKit's tests for exhaustive HTTP-status, backoff, and scheduler details; +rely on ConfigKit's tests for its parser and presence machinery. Use race testing for +new concurrent observation collection. Keep consumer checks offline and credential-free. + +## Completion Criteria and Handoff + +This unit is complete only when: + +- The settings/ownership table and migration inventory are concrete enough to start + the ConfigKit migration, including file/profile sources, CLI scope, credentials, + defaults, finite backend limits, positive repair budgets, and deadline policy. +- The offline prototype and focused matrix demonstrate the selected boundary with + both libraries together, including both LLM roles and Audita domain checks. +- Exact verification commands, results, limitations, and unresolved upstream gaps + are recorded. Missing evidence is not reported as successful integration. +- No adoption-critical decision remains hidden in prototype code. If a gap prevents + settling the boundary, record it and revise the plan before declaring readiness. +- The existing Audita production behavior remains intact in this unit. Temporary + code is clearly isolated and has a removal/integration destination during the + production migrations; no second supported runtime is introduced. +- A component-level follow-up identifies code to replace, domain code to retain, + tests to update, documentation/migration-guide work, and any temporary bridge + needed while ConfigKit lands ahead of PromptKit. +- Licensing/distribution follow-up for the selected dependency graph is identified + for the later release stage; this prototype does not claim release compliance. +- Roadmap links and examples resolve, appropriate checks pass, and commits contain + only this unit's work. Significant settled architectural choices may be recorded + in an ADR under the [documentation policy](../policy/documentation.md), while + implementation status stays in the roadmaps. + +Retain the overall adoption order unless evidence or the other planned large changes +justifies revising it. Completing this unit enables detailed production migration; +it does not authorize an intermediate release or require unrelated runtime fixes.