eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Documentation cleanup and addition of pre-1.0 roadmap punchlist

Browse Source
This commit is contained in:
Eric Rakestraw
2026-05-12 20:40:35 -05:00
parent 3d45571bb0
commit 20f612215f
6 changed files with 936 additions and 261 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
docs/intra-module-pipeline-audit.md
View File

@@ -1,111 +0,0 @@
# Intra-Module LLM Pipeline Audit
Date: 2026-05-12
## Implementation Status (2026-05-12 Update)
The intra-module pipelining gap identified in this audit has now been addressed:
- section proposal jobs are launched promptly for each module;
- as section proposals become available in deterministic section order, section-local validator work starts without waiting for all section proposals to finish;
- deterministic validators run before LLM-backed validators for each section;
- proposal and validation LLM calls can overlap through the existing composed scheduler path;
- module application remains a single deterministic apply barrier.
## Summary
This audit checks whether Audita currently maximizes available LLM concurrency within each module by overlapping proposal and validation work.
Current state:
- Modules are serial and deterministic.
- Section proposal generation is concurrent and scheduler-limited.
- Validation (deterministic + LLM-backed) starts only after all section proposals complete.
- Proposal and validation LLM work therefore does **not** overlap in time within a module today.
Conclusion:
- Current behavior is correct and deterministic, but it does **not** fully match the target pipelined intra-module behavior.
- A narrow runner refactor is needed to pipeline section-level validation after section proposal completion while preserving apply-once-per-module semantics.
## Current Execution Flow (Per Module)
Implementation anchor: `internal/framework/runner/runner.go`.
1. `chunkWorkingTranscript` is called for the module.
2. `collectSectionProposals(...)` runs section `Module.Propose(...)` calls concurrently.
3. Runner waits for **all** proposal goroutines to finish (`wg.Wait()`).
4. Proposals are flattened deterministically by section order and assigned proposal indexes.
5. Validator chain runs over the full module proposal set (`eligible := enriched`; loop over `module.Validators()`).
6. Approved proposals are applied once via `proposals.ApplyProposals(...)`.
## Audit Answers
1. **At module start, are all section proposal jobs launched/queued promptly, or in smaller batches?**
- Jobs are launched promptly (goroutine per section), but section entry is throttled by worker semaphore in `collectSectionProposals`.
2. **Does worker fan-out submit all sections and let scheduler enforce concurrency, or does fan-out itself block scheduler entry?**
- Fan-out itself blocks scheduler entry: runner’s `sem` gate limits how many section jobs can even call `Module.Propose` (and thus reach scheduler).
3. **When a section proposal returns, do deterministic validators run immediately for that section?**
- No. Deterministic validators run only after all section proposals finish and aggregation completes.
4. **When deterministic validators pass, are LLM validator jobs submitted immediately?**
- No. LLM validator calls occur only during module-level validator pass after full proposal collection.
5. **Does validator chain architecture allow section-level validation independently?**
- Not as currently orchestrated by runner; validators receive module-wide `CandidateProposal` slices.
6. **Does LLM validator batching depend on all module proposals being present?**
- Current invocation pattern does. `LLMBackedValidator.Validate` batches over the full provided candidate set; runner currently supplies full-module candidates.
7. **Would immediate per-section validator submission reduce batching efficiency or change semantics?**
- Likely yes for efficiency: smaller per-section batches can increase LLM calls.
- Semantics can remain equivalent if ordering/cardinality/report mapping is preserved, but batching shape and diagnostic timing will differ.
8. **When are proposal indexes assigned?**
- After concurrent proposal responses return, during deterministic aggregation in `collectSectionProposals`.
9. **Can immediate validator submission preserve deterministic proposal indexes/report order?**
- Yes, if indexes are preallocated/stable by section-order offsets (or equivalent deterministic mapping) before emitting validator work.
10. **Are diagnostics paths stable/deterministic enough if validator calls interleave with proposal calls?**
- Mostly yes: stage names are deterministic (`module:proposal:section-*`, `module:validator:batch-*`).
- But batch indices and artifact emission timing could change unless explicitly stabilized by per-section deterministic indexing.
11. **Do schedulers enforce total/proposal/validation limits correctly when proposal and validation overlap?**
- Scheduler composition supports this (`global` + proposal/validation subcaps), but runner currently does not create overlap, so end-to-end overlap behavior is not exercised by runtime path.
12. **Do existing tests prove proposal and validation jobs can overlap while respecting total concurrency?**
- No direct end-to-end runner test proving overlap of proposal and validation jobs in one module.
13. **Are there tests proving FIFO queue contains both proposal and validation jobs in submission order?**
- No cross-type FIFO test. FIFO is tested at scheduler unit level (`internal/framework/llm/scheduler_test.go`), and composed-cap tests exist in CLI tests, but not mixed proposal+validation submission order in runner.
14. **Smallest safe implementation change to reach target behavior?**
- Refactor runner module execution into a two-lane pipeline:
- keep current concurrent section proposal launch,
- on each section completion, run deterministic validators for that section’s proposals,
- immediately enqueue section LLM-validator work for survivors,
- collect all approved proposals in deterministic global proposal-index order,
- apply once per module exactly as today.
## Minimal Change Plan (No Redesign)
Primary change area:
- `internal/framework/runner/runner.go`
Narrow implementation approach:
1. Split current `collectSectionProposals` into pipeline stages that emit per-section proposal results plus deterministic section ordering metadata.
2. Introduce per-section validator execution helper that preserves existing validator semantics but operates on section-local candidate subsets where safe.
3. Preserve current final aggregation shape (`ModuleResult`, applied/skipped/rejected records) and apply-once-per-module behavior.
4. Preserve deterministic proposal index assignment by precomputing stable per-section index ranges or equivalent deterministic indexing strategy.
5. Keep scheduler interfaces unchanged; continue using composed proposal/validation schedulers from CLI wiring.
Tests to add/update (minimum):
- Runner test proving proposal and validation LLM calls overlap in time while total cap is respected.
- Runner test proving deterministic final transcript/proposal indexes/report ordering under out-of-order section completion.
- Runner/CLI test proving mixed proposal+validation jobs still honor global + subcap limits.
- Optional scheduler integration test proving mixed proposal/validation FIFO submission behavior at runtime boundary.
## Notes
- No runtime behavior was changed in this audit.
- Existing docs contain prior-audit historical notes (`docs/llm-concurrency-audit.md`) that may now be stale relative to current repository state; this file focuses only on current intra-module pipeline behavior.

150
docs/llm-concurrency-audit.md
View File

@@ -1,150 +0,0 @@
# LLM Concurrency Audit
Date: 2026-05-12
## Summary
This document audits the current Audita execution flow for module sequencing, chunk proposal execution, validator execution, and LLM concurrency control.
Current implementation already provides:
- serial module execution over a mutable working transcript,
- per-module chunk recomputation from the current working transcript,
- concurrent per-section proposal execution,
- scheduler wiring for both proposal-generation and LLM-backed validator calls,
- deterministic proposal aggregation and deterministic per-module apply ordering,
- subprocess-safe behavior and deterministic test hooks without requiring live LLM credentials.
At audit time, the main gaps versus the requested target architecture were:
- there was no separate `--proposal-llm-concurrency` flag,
- there was no separate `--total-llm-concurrency` flag,
- global concurrency was represented by existing `--llm-concurrency`,
- scheduler implementation was semaphore-based and did not explicitly guarantee FIFO ordering.
## Implementation Status (2026-05-12 Update)
The targeted concurrency gaps identified in this audit have now been addressed:
- explicit total/proposal/validation LLM concurrency controls are implemented in config/env/CLI,
- legacy `llm-concurrency` settings are preserved as compatibility aliases to total concurrency,
- proposal and validation schedulers are composed with a global total-cap scheduler,
- scheduler default behavior is FIFO with context-aware queued cancellation and reliable permit release,
- runner proposal worker fan-out is aligned with effective proposal concurrency,
- intra-module execution now pipelines section validation so proposal and validation LLM work can overlap within a module while retaining deterministic module-level apply ordering.
## Audit Findings (Questions 1-14)
The findings in this section reflect repository state at audit time (before the refactor). See the implementation-status section for current-state behavior.
1. **Does the current runner execute modules serially?**
- Yes. `Runner.Run` loops through `input.ModuleSpecs` sequentially and updates `working` per module.
2. **Are chunks within a module processed serially or concurrently?**
- Concurrently. `collectSectionProposals` starts one goroutine per section, bounded by `maxWorkers`.
3. **Are chunks recomputed per module from the current working transcript, or computed once before all modules?**
- Recomputed per module from current `working` transcript via `chunkWorkingTranscript` inside the module loop.
4. **Are proposal-generation LLM calls routed through the existing scheduler consistently?**
- Yes for production module paths. Each module `Propose` calls `proposal_generation.GenerateCandidates`, which wraps LLM calls in `req.Scheduler.Run(...)` when a scheduler is present.
5. **Are LLM-backed validator calls routed through the existing scheduler consistently?**
- Yes. `LLMBackedValidator.Validate` wraps each batch call in `req.Scheduler.Run(...)` when provided.
6. **Are proposal-generation and validation LLM calls using the same scheduler, separate schedulers, or no scheduler?**
- By default, same global scheduler instance.
- If validation concurrency is explicitly lower than primary, validation uses a composed scheduler (`global AND validation-subcap`).
- Proposal uses global scheduler only.
7. **Is there currently a single global LLM concurrency limit?**
- Yes. `--llm-concurrency` (primary concurrency) drives the global scheduler capacity.
8. **Is there currently a proposal-specific concurrency limit?**
- No dedicated config/flag. Proposal concurrency is bounded by:
- global scheduler permits (`--llm-concurrency`), and
- runner worker fan-out (`maxWorkers`, currently tied to `PrimaryLLM.Concurrency`).
9. **Is there currently a validation-specific concurrency limit?**
- Yes, optional. `--validation-llm-concurrency` can create an additional validation sub-cap (composed with global). If unset, validation inherits primary/global.
10. **Is the existing scheduler FIFO, semaphore-only, or otherwise unspecified?**
- Semaphore-only (`chan struct{}` permit pool). FIFO queueing is not explicitly implemented/guaranteed.
11. **Are proposals applied per chunk as chunks complete, or collected and applied once per module?**
- Collected across all sections first, validated through the module’s validator chain, then applied once per module.
12. **Are proposal indexes assigned deterministically by module/chunk/segment order, or can completion order affect indexes?**
- Deterministic. Section results are stored by section position and flattened in section order; completion order does not drive index assignment.
13. **Are report entries deterministic if chunks or validators complete out of order?**
- For chunk proposal execution: yes, deterministic output ordering is preserved.
- Validator decisions are recorded in validator-return order; current validator implementations are deterministic for the same inputs, and LLM-backed validator decisions are sorted by proposal index before return.
- Module-level reporting remains deterministic under normal deterministic/fake-client test conditions.
14. **Does the current implementation already have test hooks suitable for deterministic concurrency tests?**
- Yes.
- CLI test hooks: injectable LLM clients/schedulers and subprocess test hooks (`AUDITA_SUBPROCESS_TEST_LLM_MODE`, timeout hook).
- Runner tests already verify concurrent section proposals, deterministic index/order under out-of-order completion, and scheduler invocation behavior.
- LLM validator and proposal-generation packages include scheduler-focused tests.
## Current Flow (Concise)
1. CLI validates inputs, normalizes transcript, and writes diagnostics artifacts.
2. Runner starts with normalized transcript and executes modules sequentially.
3. For each module:
- chunk current `working` transcript,
- run section proposals concurrently (bounded),
- aggregate proposals deterministically,
- execute validator chain,
- apply approved proposals once,
- emit deterministic module result data.
4. Process report and diagnostics are written; subprocess contracts remain stable.
## Gaps vs Desired Target Architecture (Audit-Time Snapshot)
Matches target:
- Modules are serial.
- Each module sees previous module’s output transcript.
- Chunks can run concurrently for proposal generation.
- Proposal and validator LLM calls both use scheduler paths.
- Approved proposals are collected then applied once per module.
- Completion order does not affect final proposal index ordering.
- `go test ./...` does not require real LLM credentials.
- Subprocess-oriented behavior remains intact.
Gaps at audit time:
- missing dedicated `--proposal-llm-concurrency` surface,
- missing dedicated `--total-llm-concurrency` surface (at the time this role was played by `--llm-concurrency`),
- scheduler did not provide explicit FIFO semantics,
- validation remained internally batched and sequential within one validator invocation.
## Minimum Implementation Plan (Completed)
1. **Config/CLI surface**
- Add explicit `total llm concurrency` setting and CLI/env wiring.
- Add explicit `proposal llm concurrency` setting and CLI/env wiring.
- Keep existing validation concurrency as the validator sub-cap.
- Validate: `proposal <= total`, `validation <= total`.
2. **Scheduler construction in CLI runtime**
- Build one global scheduler from total concurrency.
- Build proposal scheduler as composed limiter: `global AND proposal-subcap` when proposal sub-cap is lower than total; otherwise global.
- Keep validation scheduler composition pattern: `global AND validation-subcap` when explicit lower cap exists.
3. **Runner proposal worker limit alignment**
- Decouple section worker fan-out from `PrimaryLLM.Concurrency` and align with effective proposal concurrency cap to avoid worker oversubscription beyond intended proposal limit.
4. **FIFO scheduling policy support**
- Introduce scheduler policy abstraction (default FIFO) in `internal/framework/llm` while preserving `contracts.LLMScheduler` interface shape used by runner/modules/validators.
- Keep current behavior-compatible defaults where policy config is not yet exposed.
5. **Tests**
- Add config/CLI tests for new concurrency constraints and inheritance behavior.
- Add runner/LLM tests proving:
- total cap across proposal + validation combined,
- proposal sub-cap enforcement,
- validation sub-cap enforcement,
- deterministic outputs unchanged under out-of-order completion.
## Notes
- This document is retained as an audit record; see the implementation-status section for current behavior.
- No prompt/module/validator/report schema changes were required to close the identified concurrency gaps.

0
docs/migration-from-python.md → docs/python/migration-from-python.md
View File

0
docs/python-parity.md → docs/python/python-parity.md
View File

0
docs/rewrite-notes.md → docs/python/rewrite-notes.md
View File

936
docs/roadmap.md Normal file
View File

@@ -0,0 +1,936 @@
# Audita Pre-1.0 Roadmap
## Purpose
This roadmap consolidates the remaining architectural, operational, and documentation work planned before the Audita 1.0 release.
Audita already has the core transcript-polishing pipeline in place: production modules are wired, LLM-backed proposal generation and validation are working, validator chains are enforced, diagnostics are emitted, and subprocess behavior has been hardened. The remaining pre-1.0 work should therefore focus on stabilizing the public contract, reducing avoidable dependency and operational risk, improving maintainability around prompts and validators, and making runtime behavior easier to understand after the fact.
This roadmap is intentionally scoped. The goal is not to turn Audita into a fully user-programmable LLM framework before 1.0. The goal is to make the built-in pipeline stable, explainable, testable, and maintainable.
## Release principles
The following principles should guide all pre-1.0 work:
1. Preserve Audita's conservative correction posture. New features should not make the LLM more free-form or less accountable.
2. Keep the Go binary self-contained by default. Built-in prompts, schemas, validators, and output encoders should work without external runtime assets.
3. Prefer registered, versioned extension points over arbitrary user-supplied behavior.
4. Keep subprocess behavior stable: stdout, stderr, exit codes, output files, diagnostics, and reports should be predictable.
5. Treat prompts, validators, schemas, config, and report fields as public or semi-public contracts once 1.0 is released.
6. Make diagnostics good enough that a failed, slow, or surprising run can be understood after the fact.
7. Keep secrets out of config files, logs, reports, and diagnostics.
8. Reduce avoidable dependency weight before 1.0, especially around core LLM infrastructure.
## Target pre-1.0 scope
The agreed pre-1.0 scope consists of the following workstreams:
- Replace the current `instructor-go` structured-output helper with an Audita-owned OpenAI-compatible structured LLM adapter.
- Add versioned configuration file support with a reduced and stabilized CLI surface.
- Document the stable public contract for CLI, subprocess, config, output, reports, diagnostics, and compatibility.
- Add a small registry of supported output schemas.
- Refactor validators into first-class composable components with stable validator keys and built-in chain definitions.
- Move prompt text into embedded Markdown assets with prompt IDs, versions, hashes, and shared prompt-injection hardening.
- Add transcript context support through an explicit description option.
- Add scheduler/module utilization diagnostics.
- Add a correction ledger / review artifact.
- Complete release-hardening tests and documentation updates.
The following should generally be treated as post-1.0 unless they become trivial after the refactors above:
- Arbitrary filesystem prompt overrides.
- User-configurable validator chains.
- Arbitrary user-supplied output schemas.
- Resume/start-at/stop-after execution.
- `--diff`, `--check`, or `--propose-only` review modes.
- Generated transcript descriptions enabled by default.
- Interactive correction review UI.
- Provider-specific model benchmarking harness.
## Recommended implementation order
The work is best handled in seven phases. The ordering is intentional:
1. Replace the structured-output dependency first, because it is core LLM infrastructure and affects schema metadata, diagnostics, and binary/dependency posture.
2. Stabilize configuration and public surfaces before adding more knobs.
3. Define output schema and report contracts before expanding diagnostics.
4. Refactor validators before moving prompt assets, because prompt ownership and validator ownership are closely linked.
5. Move prompts into embedded Markdown files after validator/module identities are stable.
6. Add utilization and correction-ledger diagnostics after module, validator, prompt, and schema IDs are available.
7. Finish with release-hardening documentation and evaluation fixtures.
---
# Phase 0: Replace `instructor-go` with an Audita-owned structured LLM adapter
## Goal
Remove the heavy `instructor-go` dependency and replace it with a small Audita-owned OpenAI-compatible structured-output adapter.
This should happen before 1.0 because structured LLM calls are core runtime infrastructure. Replacing this layer after 1.0 would risk subtle compatibility changes in request construction, schema strictness, retry behavior, error reporting, diagnostics, and provider compatibility.
## Design direction
Audita should retain its own internal structured LLM contract and replace only the implementation behind it.
The rest of the application should continue depending on an Audita-owned interface such as:
- `StructuredLLMClient`
- `CompleteStructured(ctx, req, out)`
The new adapter should be responsible for:
- building OpenAI-compatible chat completion requests;
- attaching `response_format.type = json_schema`;
- attaching a strict JSON Schema response contract;
- sending requests to a configurable OpenAI-compatible endpoint;
- decoding the returned JSON into caller-provided Go structs;
- preserving timeout, retry, cancellation, scheduler, diagnostics, and redaction behavior;
- surfacing provider/model/token metadata where available.
## SDK versus direct HTTP
Two implementation options are reasonable:
1. Use the official OpenAI Go SDK.
2. Use a small direct `net/http` adapter for the OpenAI-compatible request shape Audita needs.
Given Audita's goals, a direct HTTP adapter is attractive because:
- the request shape is small and stable;
- Audita already owns its internal LLM contract;
- several target providers are OpenAI-compatible rather than necessarily OpenAI itself;
- dependency weight and binary size are part of the motivation for the change;
- direct request/response diagnostics are easier to reason about.
The implementation should not expose SDK types beyond the adapter boundary if an SDK is used.
## JSON Schema handling
Do not replace `instructor-go` with another broad schema-generation framework unless there is a clear need.
Audita likely has only a small number of structured response shapes:
- proposal correction set;
- validator decision set;
- optional transcript description summary, if implemented;
- possibly future small metadata responses.
These schemas should be stable Audita contracts. Prefer hand-authored or explicitly defined schemas with IDs, versions, and hashes.
Suggested schema assets:
- `correction_set_v1`
- `validator_decision_set_v1`
- `transcript_description_v1`
Each schema should have:
- schema ID;
- schema version;
- schema hash;
- strict `additionalProperties: false` behavior where appropriate;
- tests showing accepted and rejected example payloads.
## Runtime behavior
The adapter should:
1. Build the OpenAI-compatible request.
2. Include `response_format` with `type: json_schema`.
3. Set `strict: true` where the backend supports it.
4. Receive the response.
5. Extract the assistant message content or equivalent provider response field.
6. Decode JSON into the caller-provided output struct.
7. Run existing Audita-side validation and cardinality checks.
8. Emit diagnostics with secrets redacted.
Provider-level structured output should reduce malformed responses, but Audita should continue treating all model output as untrusted until locally decoded and validated.
## Deliverables
- New Audita-owned OpenAI-compatible structured LLM adapter.
- Removal of `instructor-go` from `go.mod`.
- Preservation of the existing internal `StructuredLLMClient` contract where practical.
- Stable schema registry or schema definitions for Audita structured responses.
- Schema ID/version/hash metadata added to LLM diagnostics.
- Provider/API error redaction.
- Retry, timeout, and cancellation behavior preserved.
- Scheduler integration preserved.
- Binary size and dependency tree comparison before and after the change.
## Tests
- Request body includes `response_format` with `type: json_schema`.
- Request body includes the expected schema name and strictness setting.
- Valid proposal responses decode into existing proposal models.
- Valid validator responses decode into existing validator decision models.
- Malformed JSON fails safely.
- Missing required fields fail safely.
- Unknown extra fields fail according to schema/decoder policy.
- Provider errors are redacted.
- API keys do not appear in errors, reports, or diagnostics.
- Retry behavior remains correct.
- Timeout and context cancellation remain correct.
- Existing fake LLM tests continue passing.
- Normal `go test ./...` does not require live LLM credentials.
## Documentation
Update architecture documentation to explain:
- Audita's internal structured LLM contract;
- the OpenAI-compatible adapter;
- the supported structured-output request shape;
- schema IDs and versions;
- provider compatibility expectations;
- local validation after provider-level structured output.
---
# Phase 1: Stabilize public surfaces and configuration
## Goal
Introduce a versioned configuration file and clarify which settings are stable CLI flags, which settings belong in config, and which settings should remain environment-only.
This phase should happen early because later workstreams need clean config locations for prompt registry settings, output schema selection, validator settings, diagnostics settings, and concurrency tuning.
## Configuration precedence
Use the following precedence order:
1. Built-in defaults.
2. Configuration file.
3. Environment variables for secrets and selected deployment overrides.
4. CLI flags for high-value per-run overrides.
This replaces the current broad “defaults < environment < CLI” model with a cleaner 1.0 contract.
## Default config path
Support a default config path:
- `/etc/audita/config.yml`
Also support:
- `--config <path>`
- optionally `AUDITA_CONFIG`
## Config versioning
Every config file should include:
- `version: 1`
Unknown versions should produce a clear validation error.
## Secrets policy
Do not encourage raw API keys in config files.
Preferred pattern:
- config contains `api_key_env`;
- the actual key is resolved from the environment at runtime;
- diagnostics and effective config output redact secret-bearing values.
## Reduced CLI surface
Keep CLI flags for frequently changed per-run values and subprocess integration:
- `--config`
- transcript positional argument
- `--glossary`
- `--output`
- `--report-json`
- `--modules`
- `--work-dir`
- `--work-dir-retention`
- `--transcript-description`
- `--output-schema`
- `--total-llm-concurrency`
- `--target-sections`
Move lower-level tuning into config-only or config-primary settings:
- normalization knobs;
- min/max section tokens;
- confidence thresholds;
- LLM timeouts and retry counts;
- proposal/validation split concurrency;
- validator batching limits;
- diagnostics retention details;
- default module sequence.
Existing environment variables and CLI flags may remain temporarily as compatibility aliases, but the 1.0 documentation should clearly identify the preferred surface.
## Suggested config shape
Example:
```yaml
version: 1
pipeline:
modules:
- glossary
- homophones
- glossary
- spoken_word
- grammar
llm:
proposal:
base_url: http://localhost:8000/v1
model: nvidia/Nemotron-3-Nano-30B-A3B
api_key_env: AUDITA_LLM_API_KEY
timeout: 120s
max_retries: 2
validation:
base_url: http://localhost:8001/v1
model: nvidia/Gemma-4-31B
api_key_env: AUDITA_VALIDATION_LLM_API_KEY
timeout: 180s
max_retries: 2
concurrency:
total_llm: 4
proposal_llm: 4
validation_llm: 4
chunking:
target_sections: 8
max_section_tokens: 3000
min_section_tokens: 800
normalization:
max_segment_gap: 1.25s
ellipsis_gap: 2s
max_segment_duration: 30s
max_segment_tokens: 120
thresholds:
glossary: 0.70
homophones: 0.75
spoken_word: 0.80
grammar: 0.70
context:
description: ""
output:
schema: bare-segments
diagnostics:
work_dir: /tmp/audita
retention: auto
```
## Suggested commands
Add:
- `audita config validate --config <path>`
- `audita config print-effective --config <path>`
The second command should use the same redaction behavior as run diagnostics.
## Deliverables
- `internal/core/config` support for loading YAML config files.
- Config schema with `version: 1`.
- `--config` flag.
- Optional `AUDITA_CONFIG` environment variable.
- Config validation errors that identify the exact invalid field where practical.
- Redacted effective config output updated to include config-derived values.
- Documentation for precedence and supported fields.
- Compatibility/deprecation notes for legacy env vars and flags.
## Tests
- Defaults-only config behavior.
- Config file loading.
- CLI overrides config.
- Environment secrets resolve through `api_key_env`.
- Unknown config version fails.
- Unknown config fields either fail or warn according to an explicit policy.
- Effective config redacts secrets.
- Existing process tests continue passing.
---
# Phase 2: Define the stable public contract and output schema registry
## Goal
Document and implement the stable boundaries that external callers can rely on for 1.0.
This phase should happen early because Audita is both a user-facing CLI and a subprocess dependency. The public contract should guide the remaining implementation decisions rather than merely documenting them after the fact.
## Public contract document
Add or expand a document such as:
- `docs/public-contract.md`
It should cover:
- CLI stability guarantees;
- config file stability guarantees;
- supported input transcript forms;
- supported glossary form;
- output schema options;
- report schema versioning;
- diagnostics directory behavior;
- stdout/stderr contract;
- exit-code behavior;
- secret redaction guarantees;
- compatibility and deprecation policy;
- what counts as a breaking change after 1.0.
This can complement the current subprocess operations document.
## Output schema registry
Implement a small registry of supported output encoders. Do not support arbitrary user-supplied schemas at runtime.
Suggested initial schemas:
1. `bare-segments`
- Current behavior.
- Top-level JSON array.
- Best for backward compatibility.
2. `audita-v1`
- Object format with schema/version metadata and a `segments` array.
- Preferred stable Audita-native format going forward.
3. `seriatim-intermediate`
- Compatibility format for downstream Seriatim/Narratio workflows, if meaningfully distinct from `audita-v1`.
The command surface should be:
- `--output-schema <name>`
Config should also support:
- `output.schema: <name>`
For 1.0, it is reasonable to keep `bare-segments` as the default to avoid breaking existing consumers, while documenting `audita-v1` as the preferred stable format.
## Report schema versioning
Add explicit report schema metadata, for example:
- report schema name;
- report schema version;
- Audita binary version;
- config version;
- output schema name;
- structured response schema IDs and versions;
- pipeline/module sequence;
- prompt and validator metadata once those phases are complete.
## Deliverables
- Output encoder package or extension to `internal/core/schema`.
- Output schema registry with stable names.
- `--output-schema` and config support.
- `docs/public-contract.md`.
- `docs/output-schemas.md`, if output schema detail becomes too large for the public contract document.
- Tests for each supported output schema.
- Backward-compatibility test for current bare-array output.
## Tests
- Each output schema produces valid JSON.
- Each output schema preserves segment fields correctly.
- Unknown output schema fails clearly.
- Report includes selected output schema.
- Existing stdout/file-output behavior remains unchanged except for selected schema shape.
---
# Phase 3: Refactor validators into first-class composable components
## Goal
Make validators as easy to reason about as modules.
Validators are now central runtime components. They are reused across modules, have deterministic and LLM-backed implementations, produce diagnostics, and affect final correction acceptance. They should therefore have stable identities, registry metadata, and composable chain definitions.
## Package structure
Move from a monolithic `internal/framework/validators` package toward a first-class validator package layout.
One possible structure:
- `internal/validators`
- shared contracts;
- registry;
- chain definitions;
- common models.
- `internal/validators/deterministic`
- confidence threshold;
- original text presence;
- non-empty corrected text;
- no-effect rejection;
- protected terms.
- `internal/validators/llm`
- spoken form plausibility;
- meaning reversal;
- editorial review;
- grammar review;
- spoken-word review.
Alternatively, each validator can live in its own package. The key requirement is that validators have stable keys and are registered in one place.
## Validator keys
Every validator should have a stable key, such as:
- `confidence_threshold`
- `original_text_presence`
- `non_empty_corrected_text`
- `no_effect`
- `protected_terms`
- `spoken_form_plausibility`
- `meaning_reversal_review`
- `editorial_review`
- `grammar_review`
- `spoken_word_review`
These keys should appear consistently in:
- reports;
- diagnostics paths;
- chain definitions;
- tests;
- documentation.
## Validator chains
Each module should define its validator chain compositionally, using validator keys rather than ad hoc wiring.
Example chains:
- `glossary`
- confidence threshold;
- original text presence;
- non-empty corrected text;
- no-effect rejection;
- spoken form plausibility;
- editorial review.
- `homophones`
- confidence threshold;
- original text presence;
- non-empty corrected text;
- no-effect rejection;
- protected terms;
- spoken form plausibility;
- meaning reversal review;
- editorial review.
- `spoken_word`
- confidence threshold;
- original text presence;
- non-empty corrected text;
- no-effect rejection;
- protected terms;
- spoken-word review;
- meaning reversal review.
- `grammar`
- confidence threshold;
- original text presence;
- non-empty corrected text;
- no-effect rejection;
- protected terms;
- grammar review.
These exact chains can be adjusted during implementation, but the chain definition should become explicit and auditable.
## Config boundary
For 1.0, do not expose arbitrary user-configurable validator chains.
Recommended 1.0 boundary:
- built-in chains are fixed and documented;
- thresholds and batching knobs are configurable;
- validator keys and chain membership appear in reports;
- future user-configurable chains remain possible because the registry exists.
## Deliverables
- First-class validator registry.
- Stable validator keys.
- Explicit module validator-chain definitions.
- Updated runner wiring to use chains.
- Updated report fields using stable validator IDs.
- Documentation of built-in validators and validator chains.
- Migration of existing tests to the new package structure.
## Tests
- Every built-in validator is registered.
- Every module references only registered validators.
- Missing/unknown validator keys fail deterministically.
- Existing validator behavior remains unchanged.
- Reports preserve or improve existing validator rejection detail.
- Repeated glossary stages continue to report deterministically.
---
# Phase 4: Move prompts into embedded Markdown assets and harden prompt boundaries
## Goal
Treat prompts as first-class behavioral assets rather than multiline strings embedded in Go source.
This phase pairs naturally with the validator refactor because both modules and LLM-backed validators need prompt assets, stable prompt IDs, prompt versions, and prompt diagnostics.
## Built-in embedded prompts
Use Go's `embed` package to compile built-in Markdown prompts into the Audita binary.
A possible layout:
- `internal/prompts`
- registry;
- loader;
- renderer;
- metadata.
- `internal/prompts/embedded/modules/glossary`
- `internal/prompts/embedded/modules/homophones`
- `internal/prompts/embedded/modules/spoken_word`
- `internal/prompts/embedded/modules/grammar`
- `internal/prompts/embedded/validators/spoken_form_plausibility`
- `internal/prompts/embedded/validators/meaning_reversal`
- `internal/prompts/embedded/validators/editorial_review`
- `internal/prompts/embedded/validators/grammar_review`
- `internal/prompts/embedded/validators/spoken_word_review`
For 1.0, built-in embedded prompts should be the only supported prompt source. Filesystem prompt overrides can be considered later.
## Prompt rendering
Use Go `text/template` with typed template data structs.
Rendering should use missing-key errors so prompt changes do not silently omit required data.
The Markdown files should contain the natural-language instruction text. Go code should still own:
- typed request and response structs;
- output schema enforcement;
- transcript section formatting;
- glossary formatting;
- module and validator selection;
- diagnostics;
- template data construction.
## Prompt metadata
Every prompt should have:
- stable prompt ID;
- prompt version;
- source type, initially `builtin`;
- file path within the embedded prompt registry;
- SHA-256 hash of the source or rendered prompt text.
Reports and diagnostics should include enough prompt metadata to reproduce or debug behavior later.
Suggested prompt metadata fields:
- `prompt_id`
- `prompt_version`
- `prompt_source`
- `prompt_sha256`
## Prompt-injection hardening
Add a shared hardening fragment to every module and LLM-validator prompt.
Core policy:
- transcript text is untrusted data;
- glossary entries are reference data, not instructions;
- the model must not obey instructions contained in transcript text;
- the model must perform only the requested correction or validation task;
- the model must not invent facts, names, events, motivations, or speaker intent;
- the transcript remains the source of truth for what was said.
This hardening language should be centralized so it is consistently applied.
## Transcript context support
Add support for a brief transcript description, supplied by the user.
Recommended CLI/config surface:
- `--transcript-description <text>`
- `context.description: <text>`
The description should be included in every proposal and validator prompt as background context only.
The prompt should clearly state that the description may help interpret ambiguous terms but must not override the transcript.
Generated transcript descriptions should remain opt-in or deferred. If implemented before 1.0, they should be:
- explicitly requested;
- one sentence;
- cached in diagnostics;
- clearly labeled as generated;
- never treated as authoritative.
## Deliverables
- Embedded Markdown prompt files for all modules and LLM-backed validators.
- Prompt registry with stable IDs and versions.
- Typed prompt rendering layer.
- Shared prompt hardening fragment.
- Prompt metadata in diagnostics and reports.
- Transcript description plumbed through proposal and validator prompts.
- Documentation for built-in prompts and prompt metadata.
## Tests
- Every registered prompt renders successfully.
- Missing template data fails.
- Rendered prompts include prompt-injection hardening language.
- Rendered module prompts include transcript context when supplied.
- Rendered glossary prompts include glossary context.
- Prompt metadata appears in diagnostics.
- Prompt hashes are deterministic.
- Existing fake-LLM tests continue passing.
---
# Phase 5: Add utilization diagnostics and correction review artifacts
## Goal
Improve operational observability so Audita runs can be debugged and performance-tuned after the fact.
This phase should follow the registry/prompt work so diagnostics can include stable module, validator, prompt, and schema identifiers.
## Scheduler utilization diagnostics
Add module-level and run-level metrics for LLM scheduling and execution.
Suggested fields:
- total proposal LLM calls;
- total validation LLM calls;
- scheduler queue wait time;
- LLM execution time;
- deterministic validation time;
- total module wall time;
- max observed in-flight LLM calls;
- average observed in-flight LLM calls, if easy to compute;
- total/proposal/validation concurrency limits in effect;
- per-module timing summary;
- per-validator timing summary.
This should be emitted as machine-readable diagnostics and summarized in the process report.
A diagnostics artifact such as `scheduler-utilization.json` or `timing-summary.json` is sufficient.
## Correction ledger
Add a normalized machine-readable correction ledger that records each proposed correction and its final disposition.
This may be a JSON array or JSONL file.
Each record should identify:
- run ID;
- module key;
- module instance;
- section ID or section range;
- proposal index;
- segment ID;
- speaker;
- original text;
- proposed corrected text;
- replacement policy;
- confidence;
- deterministic validator decisions;
- LLM validator decisions;
- final disposition: applied, rejected, skipped, or failed;
- reason codes;
- diagnostics artifact references where available.
This ledger should not replace existing reports. It should provide a flattened review-friendly artifact across all modules.
## Deliverables
- Scheduler/timing metrics collection.
- Run-level timing diagnostics artifact.
- Module-level utilization summaries.
- Correction ledger artifact.
- Report references to the new artifacts.
- Documentation for interpreting utilization and correction ledger fields.
## Tests
- Metrics are emitted on successful runs.
- Metrics are emitted or partially emitted on failed runs where possible.
- Scheduler permit release behavior remains correct.
- Correction ledger includes applied, rejected, and skipped examples.
- Secret redaction still applies.
- Clean successful runs still respect work-dir retention behavior.
---
# Phase 6: Release hardening and 1.0 documentation pass
## Goal
Finish 1.0 by turning the new architecture into a documented, tested, stable release candidate.
## Documentation updates
Update or add:
- `README.md`
- `docs/architecture.md`
- `docs/public-contract.md`
- `docs/configuration.md`
- `docs/output-schemas.md`
- `docs/structured-llm.md`
- `docs/validators.md`
- `docs/prompts.md`
- `docs/subprocess-operations.md`
- `docs/release-checklist.md`
The README should stay concise. Detailed contract and architecture material should live in the docs directory.
## Evaluation fixtures
Add a small curated evaluation set for release confidence.
Each fixture should track:
- expected must-apply corrections;
- expected must-not-apply corrections;
- protected terms that must survive;
- expected module sequence;
- expected broad applied/rejected/skipped counts;
- idempotence expectations for a second pass where practical.
The purpose is not to perfectly score LLM behavior across all providers. The purpose is to catch prompt, validator, schema, and pipeline regressions before 1.0.
## Idempotence check
Add at least one test or documented manual release check where Audita is run twice on the same transcript.
The second run should ideally be close to a no-op. If it is not, the result should be explainable.
## Release checklist
Create a checklist covering:
- structured LLM adapter behavior;
- structured response schema IDs and versions;
- dependency tree and binary size review;
- config validation;
- output schemas;
- subprocess contract;
- report schema;
- diagnostics redaction;
- prompt metadata;
- validator chain metadata;
- scheduler utilization diagnostics;
- correction ledger;
- default full-pipeline run;
- explicit module runs;
- failure diagnostics;
- cancellation/timeout behavior;
- clean stdout/stderr behavior.
## Deliverables
- Updated docs.
- Release checklist.
- Curated evaluation fixtures.
- Final architecture update reflecting the new package layout.
- Final public contract review.
- Optional migration notes from pre-1.0 CLI/env behavior.
## Tests
- `go test ./...`
- CLI integration tests for config and output schemas.
- Subprocess behavior tests.
- Structured LLM adapter tests.
- Structured response schema tests.
- Prompt rendering tests.
- Validator registry tests.
- Report/diagnostics shape tests.
- Redaction tests.
- Fixture/evaluation tests that can run without live LLM credentials, using fake structured responses.
---
# Suggested commit grouping
The phases above can map cleanly to a series of implementation prompts or pull-request commits.
Recommended grouping:
1. Replace `instructor-go` with an Audita-owned structured LLM adapter.
2. Add structured response schema registry and schema metadata diagnostics.
3. Add versioned config file support and config commands.
4. Stabilize CLI/config precedence and update documentation.
5. Add output schema registry and public contract docs.
6. Refactor validators into registry-backed composable chains.
7. Move prompts into embedded Markdown assets with prompt registry metadata.
8. Add shared prompt-injection hardening and transcript description support.
9. Add scheduler utilization diagnostics.
10. Add correction ledger artifact.
11. Complete docs, evaluation fixtures, and release checklist.
The structured LLM adapter and structured response schema registry should be completed before the prompt registry work. The validator and prompt work can be developed in parallel if the interfaces are agreed first, but they should be merged carefully because both affect module construction, reports, diagnostics, and tests.
# Proposed 1.0 completion criteria
Audita is ready for 1.0 when the following are true:
- `instructor-go` has been removed.
- Audita owns its structured LLM adapter behind an internal interface.
- Structured response schemas have stable IDs, versions, and hashes.
- A user can run Audita with a concise command and a versioned config file.
- Secrets are not stored in config files or surfaced in diagnostics.
- The public subprocess contract is documented and tested.
- The output schema is selected from a small supported registry.
- Validators have stable keys and explicit built-in chains.
- Prompts are embedded Markdown assets with stable IDs, versions, and hashes.
- All proposal and validator prompts include prompt-injection hardening.
- Transcript description context can be supplied explicitly.
- Reports include module, validator, prompt, structured response schema, output schema, and timing metadata.
- Scheduler utilization diagnostics make concurrency behavior observable.
- A correction ledger exists for review and debugging.
- Existing full-pipeline behavior remains compatible with current use cases.
- The default run path and explicit module paths are covered by tests.
- Failure diagnostics are retained and useful.
- Clean successful runs remain quiet on stdout/stderr according to the documented contract.
# Deferred post-1.0 opportunities
The following ideas remain attractive but should not block 1.0:
- Filesystem prompt overrides.
- User-configurable validator chains.
- Arbitrary user-supplied output schemas.
- Resume/start-at/stop-after execution.
- `--diff`, `--check`, or `--propose-only` modes.
- Generated transcript descriptions enabled by default.
- Interactive correction review.
- UI or web service wrapper.
- Provider-specific prompt/model benchmarking harness.