Document resilient reports and remove obsolete warning material

This commit is contained in:
2026-09-24 17:55:37 +00:00
parent 2f036b0356
commit a770c77a23
26 changed files with 346 additions and 536 deletions

View File

@@ -39,6 +39,11 @@ Run the included small input with the minimal config:
This makes LLM requests. Explicit setting flags override the selected file and This makes LLM requests. Explicit setting flags override the selected file and
optional overlay; general environment setting overrides are no longer read. optional overlay; general environment setting overrides are no longer read.
By default the command writes `errors.yml`, `recoveries.yml`, and `report.md`
beside `corrected.json`. Use `--reporting=false` to suppress those files, or
`--debug` to retain detailed run artifacts under the configured debug work
directory. Model failures affecting some operations can still yield a completed
transcript with applied corrections; the summary and reports identify them.
See the [v2 migration guide](docs/migrations/v2.md) for old configurations. See the [v2 migration guide](docs/migrations/v2.md) for old configurations.
## Documentation ## Documentation

View File

@@ -41,9 +41,17 @@ validation. The shared setting flags are:
| Normalization (numeric seconds) | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` | | Normalization (numeric seconds) | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` |
| Confidence thresholds | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` | | Confidence thresholds | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` |
| Context | `--transcript-description` | | Context | `--transcript-description` |
| Diagnostics | `--work-dir`, `--work-dir-retention` | | Reporting | `--reporting[=true\|false]`, `--report-dir DIR` |
| Debug | `--debug[=true\|false]`, `--work-dir DIR` |
`process` alone accepts `--glossary` (required), `--output`, and `--report-json`. `process` alone accepts `--glossary` (required) and `--output`.
Reporting defaults to enabled. `--reporting=false` disables all three ordinary
reports; an explicit false overrides a true file or overlay setting. With file
output, reports go beside the transcript and `--report-dir` has no effect. With
stdout output, `--report-dir` selects the report directory, defaulting to the
invocation working directory. `--debug` separately enables retained detailed
artifacts under `--work-dir`. Configuration inspection does not create or probe
these output directories.
Per-role model, base URL, key, timeout, retry, and concurrency flags have been Per-role model, base URL, key, timeout, retry, and concurrency flags have been
removed. Models and generation controls belong to PromptKit profiles, backend removed. Models and generation controls belong to PromptKit profiles, backend
credentials and capacity to configuration, and retry/deadline settings to the credentials and capacity to configuration, and retry/deadline settings to the
@@ -65,13 +73,19 @@ settings YAML is for inspection and is not guaranteed reloadable.
For `process`, `--output` writes the corrected transcript to a file and leaves For `process`, `--output` writes the corrected transcript to a file and leaves
stdout empty on success. Without it, stdout carries transcript JSON only on stdout empty on success. Without it, stdout carries transcript JSON only on
success. `--report-json` writes a separate report file. Stderr carries readable success. Enabled reporting creates `errors.yml`, `recoveries.yml`, and
errors and diagnostic locations. A successful run can contain skipped corrections `report.md` in the selected directory. Successful stderr begins with
or warnings; see the [report contract](integrations/output-files.md). `Completed: N corrections applied; E work units with errors; R recoveries.`
Report and debug locations follow only when created. Model-operation failures
can coexist with a completed run and applied corrections; read the YAML reports
for details. Debug artifacts are created only with `--debug`. See the
[report contract](integrations/output-files.md).
Exit codes are `0` for success, `1` for runtime or output failure, and `2` for Exit codes are `0` for success, `1` for runtime or output failure, and `2` for
usage or configuration input errors. Invalid settings and profiles return `2`; usage or configuration input errors. Invalid settings and profiles return `2`;
broken embedded assets and serialization or write failures return `1`. broken embedded assets, caller cancellation, fatal processing, and required
output failures return `1`. A late report-write failure can follow a published
transcript; exit status and report status remain authoritative.
Configuration commands inspect profiles without contacting a provider. Configuration commands inspect profiles without contacting a provider.
See the [input files](integrations/transcript-glossary-files.md), See the [input files](integrations/transcript-glossary-files.md),

View File

@@ -61,12 +61,18 @@ field provenance separately from ConfigKit's redacted settings YAML.
| `normalization.max_segment_tokens` | `2048` | Positive token estimate. | | `normalization.max_segment_tokens` | `2048` | Positive token estimate. |
| `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `0.8` each | Confidence thresholds in `[0, 1]`. | | `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `0.8` each | Confidence thresholds in `[0, 1]`. |
| `context.description` | Empty | Trimmed prompt context, at most 500 bytes. | | `context.description` | Empty | Trimmed prompt context, at most 500 bytes. |
| `diagnostics.work_dir` | `/tmp/audita` | Nonempty artifact parent directory. | | `reporting.enabled` | `true` | Write the ordinary three-file report bundle when processing. |
| `diagnostics.retention` | `auto` | `auto`, `always`, or `never`; see [operations](operations.md). | | `reporting.stdout_dir` | `.` | Report directory for stdout transcript output; ignored for file output or disabled reporting. |
| `diagnostics.debug` | `false` | Persist detailed debug artifacts when processing. |
| `diagnostics.work_dir` | `/tmp/audita` | Parent of the retained debug run directory; required only when debug is enabled. |
Module keys select correction modules; validator chains remain built into them. Module keys select correction modules; validator chains remain built into them.
The loader preserves module order and repetitions. Normalization time settings The loader preserves module order and repetitions. Normalization time settings
accept numeric seconds only. The operation timeout uses a duration string. accept numeric seconds only. The operation timeout uses a duration string.
`reporting.stdout_dir` must be nonblank only for reporting-enabled stdout runs;
`diagnostics.work_dir` must be nonblank only with debug enabled. These path
requirements are checked when choosing process destinations, not during
configuration inspection.
## Credentials ## Credentials

View File

@@ -22,10 +22,13 @@ cancellation or an earlier deadline also applies. See [configuration](../config.
for defaults and [PromptKit's consumer contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/consumers/pkg-promptkit.md) for defaults and [PromptKit's consumer contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.11.0/docs/consumers/pkg-promptkit.md)
for the pinned protocol and retry details. for the pinned protocol and retry details.
A failed JSON Schema result after repair is classified as malformed generated A model operation that exhausts structural repair, a live-parent local operation
output; proposal and validator callers can turn it into a warning. Provider, timeout, capacity rejection, transport failure, malformed provider response,
transport, malformed HTTP envelope, retry, admission, and deadline failures stay or provider HTTP failure is recoverable for that operation. Proposal generation
runtime failures. Audita decodes into domain types only after PromptKit validates then contributes no candidates from that response; a failed validation batch
the JSON. Errors exposed to users carry safe messages while retaining typed marks its candidates unavailable while other batches and later modules can
causes for internal classification. Diagnostic request/response content is continue. Caller cancellation, broken assets or configuration, and unknown
sensitive and follows [operations guidance](../operations.md#artifact-handling). internal failures remain fatal and cancel peers. Audita decodes into domain
types only after PromptKit validates the JSON. Ordinary reports use safe reason
codes and counts rather than provider messages. Full request/response content
is written only with `--debug`; see [operations](../operations.md#diagnostics-directory).

View File

@@ -1,189 +1,90 @@
# Output Files and Report Formats # Output Files and Reports
This document owns externally consumed corrected-transcript and report formats,
and describes the correction ledger and utilization diagnostics. It documents
current serialized shapes without making every debugging artifact a versioned
compatibility promise. Physical filenames, placement, retention, and permissions
belong to [operations](../operations.md); invocation and output channels belong to
the [CLI](../cli.md#streams-and-exit-status).
## Corrected Transcript ## Corrected Transcript
The [output setting](../config.md#settings) selects the envelope: The corrected transcript uses the selected `output.schema`: `bare-segments`
or `audita-v1`. Its schema and text rules are unchanged by resilience
reporting. See [configuration](../config.md#settings) for selection and
[operations](../operations.md#required-output-files) for placement and writes.
| Encoder | JSON shape | With reporting enabled, Audita produces three ordinary files:
- `errors.yml`: affected work units and run-level failures;
- `recoveries.yml`: successful logical model operations with repair, service
retry, or defective-proposal filtering;
- `report.md`: a concise overview and complete per-chunk matrices.
The YAML documents have Audita-owned schemas `audita-errors` and
`audita-recoveries`, each with integer `version: 1`. Both contain, in order,
`schema`, `version`, `run_id`, `status`, `started_at`, `completed_at`, `summary`,
and an always-present `errors` or `recoveries` sequence. Status is
`in_progress`, `completed`, or `failed`. Times are UTC RFC3339 strings;
`completed_at` is null until finalization. A recoverable model error can appear
with `status: completed`.
## Summary Counts
Both documents carry the same summary, in this order:
| Count | Meaning |
| --- | --- | | --- | --- |
| `bare-segments` | Top-level array of segment objects | | `work_units_total` | Configured module instances × initial chunks. |
| `audita-v1` | Object with `schema: "audita-v1"`, `version: "v1"`, and `segments` containing the segment array | | `work_units_started`, `work_units_completed`, `work_units_interrupted`, `work_units_not_started` | Explicit processing states. |
| `work_units_with_errors` | Distinct units with recoverable operation failures. |
| `run_errors` | Fatal processing or output failures outside that unit count. |
| `corrections_applied` | Approved proposals applied, not replacement occurrences. |
| `llm_operations` | Structured-client invocations, including pre-dispatch failures. |
| `proposal_responses` | Structurally usable proposal responses. |
| `recoveries` | Successful operations with one or more recovery reasons. |
| `proposal_responses_with_filtering` | Proposal responses that filtered at least one defective proposal. |
Both emit these segment fields: Unknown counts before processing are `null`; completed empty counts are `0`.
Retries and repair generations are details of one operation, not additional
operations. Provider token usage, execution timing, and attempt-level evidence
remain in opt-in debug files.
| Field | Type | Meaning | ## Error and Recovery Records
| --- | --- | --- |
| `id` | Integer | Identifier in the normalized transcript |
| `speaker` | String | Speaker label |
| `start`, `end` | Numbers | Start/end times in seconds |
| `text` | String | Text after normalization and accepted corrections |
| `categories` | Array of strings, optional | Segment categories; omitted when empty |
Input IDs and segment boundaries need not survive normalization unchanged; An `errors` record with `scope: work_unit` identifies zero-based `stage_index`
normalization can merge segments and reassign IDs. The output describes the and `chunk_index`, module key/instance, initial segment range, applied-correction
resulting transcript, not a lossless copy of the source JSON. Input acceptance count, and a `failures` sequence. Each failure identifies the operation
rules belong to the [input-file contract](transcript-glossary-files.md). (`proposal`, `validation`, or `validation_input`), optional validator and
zero-based batch index, safe reason/message, sorted proposal indices when
known, and execution counts when available. Multiple failed batches in one
work unit form one error record. `scope: run` instead records phase, safe reason
and message, optional origin, and output path where relevant. It can describe
a fatal error or secondary output-write failure without inventing a failed
model work unit. Raw provider errors and transcript text are absent.
## Process Report Each recovery identifies its work unit and successful operation, with sorted
`reasons`, a safe message, available execution counts, and sorted
`filtered_proposals` details. One recovery can contain `output_repaired`,
`service_recovered`, and `proposals_filtered` together. Filtering covers
defective response proposals; low confidence, protected-term rejection,
ordinary editorial rejection, and an apply-time conflict caused by an earlier
accepted correction are not filtering recoveries. Failed operations are errors,
not recoveries. The initial chunk identity remains stable even if earlier
modules change its text.
The report is a JSON object. `report_metadata` identifies its schema separately ## Markdown Overview
from the transcript output envelope:
| Metadata field | Type / current value | `report.md` includes run ID, status, corrections, work coverage, error and
| --- | --- | recovery totals, input/output paths, and full work-unit error and recovery
| `report_schema_name` | String: `audita-process-report` | matrices. Rows show every initial chunk with 1-based chunk number and initial
| `report_schema_version` | String: `v2` | segment range; columns follow configured module-instance order, including
| `output_schema` | String identifying the selected transcript encoder | repeats. Completed error cells are `0` or `1`; recovery cells are operation
| `config_version` | Optional integer, present when file configuration was loaded | counts. `—` means not started and `n*` means interrupted with n facts recorded.
An empty stage or chunk set says “No work units.” Short reason breakdowns count
records containing each reason; a recovery can have several reasons, so those
counts need not sum to the recovery total. The report refers to both YAML
files. It does not include transcript excerpts or correction lists.
The remaining top-level fields are: ## Debug-Only Files
| Field | Type and meaning | With `--debug`, `correction-ledger.json` contains proposed candidates and
| --- | --- | their application, rejection, or interruption details. `llm-execution.json`
| `phase` | String; currently `default_pipeline` | holds logical calls and PromptKit execution events; `utilization-diagnostics.json`
| `status` | String: `success` or `failed` | groups backend capacity and timing observations while preserving known and
| `operation` | String; currently `process` | complete usage semantics. Interaction request/response artifacts, selected
| `transcript_path`, `glossary_path` | Input path strings | profiles, configuration provenance, and input snapshots are also debug-only.
| `output_path` | Optional transcript destination string | These files are diagnostic evidence, not an alternate ordinary report schema.
| `modules` | Array of configured module keys, including repeats |
| `started_at`, `completed_at` | RFC 3339 timestamps, possibly fractional; completion is an optional field populated by completed process runs |
| `error_phase`, `error_message` | Optional failure classification and human-readable detail |
| `input_segment_count`, `normalized_segment_count` | Optional integer normalization counts |
| `normalization_merges`, `normalization_id_reassignments` | Optional integer transformation counts |
| `normalization_skipped` | Object of optional integer counters: `different_speakers`, `gap_too_large`, `duration_exceeded`, `token_limit_exceeded`; can be empty |
| `chunking` | Optional section summary described below |
| `diagnostics` | Optional artifact-path metadata described below |
| `llm_summary` | Optional logical LLM execution summary, present when observations were initialized |
| `modules_summary` | Optional aggregate module outcome |
| `module_results` | Optional array of module outcomes |
Failures can preserve partial module results. Missing summaries indicate unavailable
information, not necessarily zero work. Early configuration failures occur before
report construction. Report success describes processing results; a later file-write
or cleanup error can still cause a nonzero CLI exit. Use the exit result and report
together as explained in [subprocess guidance](subprocess.md).
Error phases identify where processing failed, for example `transcript_read`,
`glossary_read`, `transcript_schema`, `glossary_schema`, `chunking`, `runner_setup`,
`runner_execution`, `output_schema`, `serialization`, `output_write`, and
`stdout_write`. This is not an exhaustive enumeration or a promise that every
late failure is represented in a successfully written report. Error messages are
human-readable, not structured fields to parse for control flow.
### Chunking and Diagnostics Metadata
`chunking` contains integer fields `chunk_count`, `min_estimated_chunk_tokens`,
`max_estimated_chunk_tokens`, `total_estimated_transcript_tokens`,
`max_section_tokens`, and `min_section_tokens`, plus optional integer
`target_sections`. Token counts are estimates.
`diagnostics` contains optional path strings:
- `directory_path`
- `source_transcript_path`, `parsed_source_transcript_path`, `normalized_transcript_path`
- `normalization_summary_path`, `chunking_summary_path`
- `utilization_summary_path`, `llm_execution_path`, `selected_profiles_path`, `correction_ledger_path`
- `invocation_metadata_path`, `redacted_effective_config_path`, `error_log_path`
The [operations guide](../operations.md#diagnostics-directory) defines the physical
artifacts and path-lifetime caveats.
### Module Outcomes
`modules_summary` contains integer `module_count`, `total_applied_changes`, and
`total_skipped_changes`, plus optional string `failed_module_instance`.
`total_skipped_changes` counts both apply-time skipped changes and validator
rejections. Their detailed records are stored in separate arrays.
Each `module_results` object contains:
| Field | Type and meaning |
| --- | --- |
| `module_key`, `module_instance` | Strings; the instance distinguishes repeated occurrences of a configured module |
| `replacement_policy` | Optional string describing the module's replacement policy |
| `status` | String describing module outcome (`success` or `failed`) |
| `proposal_count` | Integer |
| `started_at`, `completed_at` | Optional timestamps |
| `error_message` | Optional failure detail |
| `warnings` | Optional array of warning records |
| `validator_decisions` | Optional array of decision records |
| `validator_rejected` | Optional array of rejected-correction records |
| `applied_changes`, `skipped_changes` | Optional arrays of apply outcomes |
A validator decision has `validator_name` (string), `proposal_index` (integer),
`approved` (boolean), `reason_code` and `message` (strings), and optional
`diagnostic_artifact_path` (string).
Rejected corrections include `validator_name`, `proposal_index`, `module_key`,
`module_instance`, `target_segment_id`, `original_text`, `corrected_text`,
`reason_code`, and `message`. Applied changes include the same correction identity
and text fields without validator/reason/message fields, plus integer
`replacement_count`. Skipped changes include correction identity/text fields,
`skip_reason`, and `message`.
A warning has string `scope`, `reason_code`, and `message`; optional integer
`section_index` and `batch_index`; and optional string `validator_name` and
`diagnostic_artifact_path`. Warnings, skipped changes, and rejected corrections
can occur without a failed run. Proposal indexes associate outcomes with their
proposal within a module instance; they are not transcript segment IDs.
## Correction Ledger
The ledger is an array of correction entries; it can be `null` when no module
results are available and empty when modules produced no entries. Unlike the
process report, it has no separate schema-version envelope.
Each entry has required `module_key`, `module_instance`, integer `proposal_index`,
and `disposition`. The current dispositions are:
| Disposition | Meaning |
| --- | --- |
| `applied` | The proposed change was applied |
| `skipped` | The proposal reached application but was skipped |
| `rejected` | A validator rejected the proposal |
| `failed` | Module-level failure record, not necessarily an individual proposal |
Optional fields are `run_id`, integer `segment_id`, `original_text`,
`proposed_corrected_text`, `applied_corrected_text`, `replacement_policy`,
`disposition_reason_code`, and `disposition_message`. Optional validator arrays
are `deterministic_validator_decisions` and `llm_validator_decisions`; each record
contains string `validator_key`, boolean `approved`, string `reason_code`, and
optional string `message`.
Module failure entries can have the zero-value proposal index and no correction
text or segment. Do not interpret them as evidence that proposal zero failed.
Empty optional values are omitted. The ledger records disposition, not a replay
or resume protocol.
## Utilization Diagnostics
`llm_summary` contains integer `logical_calls`, `provider_attempts`,
`transient_retries`, and `structural_repairs`. Its `usage` contains `prompt`,
`completion`, `total`, `cached`, and `cache_write`, each with integer `value`
and boolean `known` and `complete`. A call counts once even if preparation fails.
If no call dispatches, usage is known-zero and complete. Missing `llm_summary`
means observations were unavailable.
The unversioned utilization JSON contains `backends`, keyed by backend ID. Each
entry has configured `concurrency` and `queue_capacity`, the same execution
summary fields, and integer `capacity_wait_ms`, `generation_ms`,
`retry_delay_ms`, and `logical_wall_ms`. It also has `run_wall_time_ms` and
optional `modules` and `validators` arrays. Module records contain the module
identity, logical proposal/validation call counts, and `module_wall_time_ms`.
Validator records contain module/validator identity, `llm_backed`, `calls`, and
`elapsed_ms`. Timings can overlap; they do not decompose total wall time.
`llm-execution.json` has `calls` and `events` arrays joined by `call_id`. Calls
include logical attribution, final classification, summary, elapsed milliseconds,
and available prompt/profile/model metadata. Events contain safe execution
metadata, including attempt and terminal records, with millisecond durations.
Preparation failures have a call and no fabricated events. This artifact has no
provider messages, endpoints, credentials, or transcript content. Other request
and response artifacts can contain redacted prompt and transcript content.

View File

@@ -8,8 +8,10 @@ The [CLI reference](../cli.md) owns command syntax, streams, and exit semantics;
Use an argument array with the transcript path and required glossary argument, Use an argument array with the transcript path and required glossary argument,
following the [process examples](../cli.md#commands-and-help). For machine workflows, following the [process examples](../cli.md#commands-and-help). For machine workflows,
select explicit transcript and report destinations rather than depending on the select an explicit transcript file destination and a configuration path. Reports
working directory or run-directory retention. Supplying a configuration path also then go beside the transcript. For stdout output, select `--report-dir` or use
the invocation working directory. Debug is independent and off by default.
Supplying a configuration path also
makes file selection predictable; only explicitly referenced credential variables makes file selection predictable; only explicitly referenced credential variables
are read from the environment. are read from the environment.
@@ -24,15 +26,20 @@ Interpret the exit result according to the
[CLI output and exit contract](../cli.md#streams-and-exit-status). [CLI output and exit contract](../cli.md#streams-and-exit-status).
Consume transcript/report data through the Consume transcript/report data through the
[output-file contract](output-files.md), and treat stderr as human-readable detail. [output-file contract](output-files.md), and treat stderr as human-readable detail.
The report supplements the exit result: it may be absent after early failures, The three reports may be absent after early configuration or destination
and output may have been written before a later reporting or cleanup failure. failures. After initialization, failed reports can describe invalid input or
partial processing. Exit `0` can include recoverable model errors and applied
corrections; inspect `errors.yml` and `recoveries.yml` for those outcomes. A
transcript may have been written before a later required report failure, so
the file alone does not prove completion.
If processing fails, preserve available evidence and follow If processing fails, preserve available evidence and follow
[operations recovery](../operations.md#failure-inspection-and-recovery). [operations recovery](../operations.md#failure-inspection-and-recovery).
For reports retained outside the run directory, account for the Ordinary reports are independent of optional retained debug evidence. Parent
[artifact lifecycle](../operations.md#retention-behavior) before dereferencing cancellation reaches the child through its context; Audita joins workers and
reported paths. Neither the existence of an output file nor a diagnostics path attempts failed-status report finalization before exiting. Uncatchable
alone establishes successful completion. termination can leave `in_progress` reports. Neither the existence of an output
file nor a debug directory alone establishes successful completion.
## Related Contracts ## Related Contracts

View File

@@ -1,91 +1,42 @@
# Diagnostics and Report Assembly # Diagnostics and Reporting Internals
## Scope The [output-file contract](../integrations/output-files.md) owns public
schemas and the [operations guide](../operations.md) owns physical placement.
This document owns implementation flow and collaborators. Physical artifacts and `internal/framework/outcomes` contains processing-owned operation references,
retention belong to [operations](../operations.md). Serialized reports, ledger counts, failure reasons, filtering details, and work states. It imports no
records, and utilization shapes belong to the reporting, CLI, or provider package. The runner records those facts per initial
[output-file contract](../integrations/output-files.md). section and module instance. `internal/framework/processreport` maps them once
into `internal/core/reporting.Snapshot`, including nullable counts, grouped
work-unit errors, separate run errors, and one recovery per successful logical
operation. Reports order stages, sections, validators, and batches by semantic
identity, never concurrent completion time. The core reporting package owns
Audita's versioned YAML DTOs, fixed Markdown rendering, and brief CLI
completion formatter. YAML redaction walks serialized string nodes so numeric
and boolean values retain their types. Ordinary reports omit raw causes,
transcript text, complete decisions, and token/timing payloads.
## Run Directory and Artifact Writers `internal/core/diagnostics.OutputSession` owns active destination checks,
transcript preflight, report initialization, and per-file final replacement.
CLI composes it after configuration/profile inspection and before reading
inputs or dispatching model work. It does not create ordinary report parents.
On a final write error it attempts other enabled reports and one failed-status
pass; a published transcript is never erased by a later report failure.
`internal/core/diagnostics` creates run directories, writes snapshots and metadata, Debug is independent. Only `diagnostics.debug` creates a unique retained child
and evaluates retention. CLI orchestration supplies the selected configuration, under `diagnostics.work_dir`. CLI passes empty diagnostics paths and nil writers
paths, run outcome, and correction-presence information. Artifact names are when debug is off. With debug on, source/normalized snapshots, invocation,
centralized in diagnostics; their operator-facing inventory is in effective settings, origins, profiles, initial chunk summary, correction ledger,
[operations](../operations.md#diagnostics-directory). indexed execution/utilization, and model interactions are written. Runner
candidate capture includes proposals later rejected or interrupted. Debug
Snapshot writes are attempted at the relevant processing phases. Several diagnostic write failures produce safe stderr messages and do not abort processing.
writes are best-effort, so report metadata can identify a location without proving Every debug writer uses all configured backend secrets for redaction, including
that a file was written. The CLI distinguishes primary processing errors from unselected backend credentials and source snapshots. Debug can still contain
secondary diagnostic/report write errors. sensitive transcript and prompt content.
## Process Report Mapping
`internal/framework/processreport.Build` maps invocation facts, normalization and
chunking summaries, and runner output into `reporting.ProcessReport`. Core reporting
owns the Go data structures and JSON writer. The builder attaches diagnostic
metadata when a run-directory path is available.
Module mapping copies outcomes from runner results, collects summary counts, and
preserves available partial module results on failure. The CLI failure path does
not pass normalization or chunking summaries to report construction even when
some earlier processing completed. Consumer interpretation belongs to the
[process-report contract](../integrations/output-files.md#process-report).
## Correction Ledger and Utilization
`BuildCorrectionLedger` flattens module changes, rejections, and failures into
ledger entries. It associates validator records by proposal index within each
module, separates execution classes through validator metadata, and sorts entries
by module instance, proposal index, and disposition.
Runner instrumentation and PromptKit observations collect utilization; CLI
orchestration writes the backend summary and safe call/event artifact when run
output is available. Report assembly maps the same observations to `llm_summary`
without independently measuring calls. A selected backend remains attributable
when preparation fails before dispatch. Domain module/validator timing stays
separate from provider timing, and those durations can overlap.
For the serialized forms see [ledger](../integrations/output-files.md#correction-ledger)
and [utilization](../integrations/output-files.md#utilization-diagnostics).
## CLI Coordination and Retention
CLI orchestration writes caller-selected reports and run-local artifacts before
asking diagnostics to apply retention. `HasSkippedCorrections` derives the
retention input from runner outcomes. Keep the exact lifecycle definition in
[operations](../operations.md#retention-behavior), not in another internal table.
On a processing failure, secondary report-write errors do not replace the original
error. On the success path, an external report-write failure returns before the
run-local report write; retention errors occur later. This ordering explains why
artifacts and process exit status must be assessed together.
## Redaction
Effective configuration uses ConfigKit redaction. The CLI supplies all resolved
backend credentials, including unselected backends, to report and diagnostic
writers. `core/redaction` sanitizes the serialized representation rather than
mutating processing results. JSON string values and keys are redacted after
parsing with number preservation; YAML redaction preserves scalar types.
Serialization failure returns an error without writing the original object.
Interaction artifacts, correction ledgers, execution/utilization diagnostics,
selected profiles, both report copies, and surfaced LLM errors share this
string-redaction implementation. Longer overlapping values take precedence,
and replacement text is not processed again within a redaction pass.
Execution reports index events by call ID. Utilization groups calls directly by
backend and actual module instance, keeping module result order and retaining
usage knownness/completeness. Section and batch indices remain separate from
module identity; section-qualified interaction filenames avoid collisions.
These boundaries implement the [architecture requirement](../policy/architecture.md#errors-and-secret-handling).
Operational handling of transcript-bearing artifacts belongs to
[operations](../operations.md#artifact-handling).
## Key Tests ## Key Tests
- `internal/core/redaction/redaction_test.go` Useful tests: `internal/core/reporting/*_test.go`,
- `internal/core/diagnostics/*_test.go` `internal/core/diagnostics/*_test.go`,
- `internal/framework/processreport/*_test.go` `internal/framework/processreport/*_test.go`, and
- `internal/core/reporting/report_test.go` `cmd/audita/main_integration_test.go`.
- `cmd/audita/main_integration_test.go`

View File

@@ -6,7 +6,7 @@
prompt and schema assets, registers application backends, and inspects the prompt and schema assets, registers application backends, and inspects the
selected proposal and validation profiles offline. A role-bound `llm.RoleClient` selected proposal and validation profiles offline. A role-bound `llm.RoleClient`
implements `structuredoutput.Client`; proposal and validator contracts alias that implements `structuredoutput.Client`; proposal and validator contracts alias that
interface during migration. Both roles share engine-local backend capacity. interface. Both roles share engine-local backend capacity.
For each logical call, the adapter starts an operation context and a diagnostic For each logical call, the adapter starts an operation context and a diagnostic
record before preparation. It supplies the reserved hardening text, the selected record before preparation. It supplies the reserved hardening text, the selected
@@ -15,12 +15,13 @@ contract. `PrepareExecution` produces the exact messages and hashes retained in
an Audita snapshot. The adapter checks the prepared backend and endpoint before an Audita snapshot. The adapter checks the prepared backend and endpoint before
`RunPrepared`. It records available final content, usage, model, and profile `RunPrepared`. It records available final content, usage, model, and profile
metadata and decodes only a validated result. Call hooks expose logical metadata and decodes only a validated result. Call hooks expose logical
start/finish records and PromptKit execution events for the report collector. start/finish records and PromptKit execution events for debug utilization.
The [integration contract](../integrations/openai-compatible-llm.md) describes The [integration contract](../integrations/openai-compatible-llm.md) describes
wire ownership and failure behavior. `structuredoutput.Failure` has a safe wire ownership and failure behavior. `structuredoutput.Failure` has a safe
message and a typed cause; only exhausted structural repair is a malformed message, fixed reason code, typed cause, and recoverable/fatal/canceled class.
output failure. Secrets from all configured backends, including unselected Counts distinguish provider attempts, transient retries, and structural
repairs, including on errors. Secrets from all configured backends, including unselected
ones, are available to the diagnostics redaction boundary. ones, are available to the diagnostics redaction boundary.
## Orchestration ## Orchestration
@@ -29,10 +30,10 @@ CLI processing supplies two role clients from the same engine to the runner.
PromptKit's per-backend admission applies to both roles and every generation, PromptKit's per-backend admission applies to both roles and every generation,
retry, and repair. The runner still processes module instances in order, retry, and repair. The runner still processes module instances in order,
collects concurrent section work deterministically, and cancels remaining collects concurrent section work deterministically, and cancels remaining
section work on a runtime failure. `llm.DiagnosticsWriter` remains the protected section work only on fatal failures. `llm.DiagnosticsWriter` remains the protected
request/response artifact writer. `llm.ObservationCollector` captures metadata request/response artifact writer. `llm.ObservationCollector` captures metadata
events under a short mutex and freezes them after runner work joins. Its logical events under a short mutex and freezes them after runner work joins. Its logical
records include preparation failures; summary usage comes from terminal events records include preparation failures; summary usage comes from terminal events
once per call and retains knownness and completeness for each field. Runner wall once per call and retains knownness and completeness for each field. Runner wall
and validator timings remain separate from provider attempts. The report and and validator timings remain separate from provider attempts. Ordinary reports
artifact mapping is completed by the following roadmap work. derive compact runner outcomes; full observations are debug-only.

View File

@@ -49,12 +49,13 @@ policy. Shared context encoding lives in `internal/framework/promptcontext`.
Built-in chains are resolved in `internal/validators` per module key. Built-in chains are resolved in `internal/validators` per module key.
Module packages call the built-in chain resolver at construction. Module packages call the built-in chain resolver at construction.
## Failure and Warning Behavior ## Failure and Outcome Behavior
Module construction failures propagate to orchestration; execution failures can Module construction failures are fatal. Recoverable proposal model failures
preserve partial module results. Exhausted structural repair becomes a warning record an unsuccessful operation and yield no candidates for that section;
and section-level rejection; provider and protocol failures propagate. Consumer-visible outcomes are defined by other sections and module instances continue. Fatal adapter or application
the [report contract](../integrations/output-files.md#module-outcomes). failures cancel peers and retain partial observations. Consumer-visible
outcomes are defined by the [report contract](../integrations/output-files.md).
## Changing Modules ## Changing Modules

View File

@@ -47,7 +47,8 @@ Supporting areas:
## Current Scope ## Current Scope
Runtime state is held in memory plus per-run diagnostics artifacts. Audita does Runtime state is held in memory; optional debug artifacts are retained only
when requested. Audita does
not currently implement a plugin framework, generic workflow engine, not currently implement a plugin framework, generic workflow engine,
resume/checkpoint/manifest system, remote storage, or multi-process service mode. resume/checkpoint/manifest system, remote storage, or multi-process service mode.
Its role-bound PromptKit adapter implements the structured LLM client contract. Its role-bound PromptKit adapter implements the structured LLM client contract.
@@ -55,7 +56,9 @@ These are current implementation boundaries, not bans on future work.
## Main Execution Path (`audita process`) ## Main Execution Path (`audita process`)
The CLI prepares input and configuration, invokes the runner, and writes results. The CLI inspects configuration and profiles, preflights and initializes active
outputs, optionally creates debug evidence, invokes the runner, and finalizes
transcript and reports.
The [pipeline](pipeline.md) owns detailed state transitions; the The [pipeline](pipeline.md) owns detailed state transitions; the
[operations guide](../operations.md) owns the operator workflow and physical [operations guide](../operations.md) owns the operator workflow and physical
artifact lifecycle. artifact lifecycle.

View File

@@ -1,66 +1,44 @@
# Audita Internal Pipeline # Audita Internal Pipeline
## Scope The [input contract](../integrations/transcript-glossary-files.md) owns input
formats; [configuration](../config.md) owns settings; the
[output contract](../integrations/output-files.md) owns serialized results.
This document describes processing collaborators and state transitions. External 1. CLI filesystem adapters read transcript and glossary files. Schema parsing
input definitions belong to the [input-file contract](../integrations/transcript-glossary-files.md), produces source objects; transcript segments enter canonical normalization.
settings to [configuration](../config.md), and serialized results to the 2. Normalization sorts and merges segments using configured time and token
[output-file contract](../integrations/output-files.md). limits, reassigning IDs where needed.
3. The runner partitions the normalized transcript exactly once before its
module loop. Initial section identity, segment membership, and token
estimates stay fixed while text evolves.
4. Module instances execute in configured order. Sections may propose
concurrently, but proposal indices follow section order and response order.
5. Deterministic validators precede LLM-backed validators. Every submitted
candidate receives one decision per validator. Recoverable failed batches
mark their candidates unavailable; successful survivors continue through
all required validators.
6. Only fully approved proposals reach application. Corrections apply in
proposal-index order within their owning section. Existing replacement
policies and apply-time safety checks resolve stale conflicts.
7. CLI encodes the final transcript and publishes active files through the
output session. `framework/processreport` builds one snapshot for YAML,
Markdown, and CLI totals. Debug evidence is captured separately when enabled.
## Inputs `runner.Run` returns the final working transcript, initial section inventory,
explicit `not_started`/`interrupted`/`completed` work units, compact model
operation outcomes, and per-module results. Candidate payloads are retained
in the returned output only when debug capture is requested. A work unit can
complete with model errors and applied corrections. Recoverable model failures
never trigger shared cancellation. Fatal setup, internal, application, or
parent-cancellation errors cancel peers, join workers, and preserve earlier
committed module text for partial reports; no completed transcript is published.
Pipeline inputs are: Required transcript/report write failures are fatal. Optional debug writes
- effective config (`internal/core/config`); report safely to stderr without changing decisions or exit status. See
- transcript JSON (`internal/core/schema`); [operations](../operations.md) for file placement and failure handling, and
- glossary YAML (`internal/core/schema`). [architecture policy](../policy/architecture.md#correction-and-concurrency-invariants)
for invariants.
## Pipeline Phases Key tests: `internal/framework/runner/*_test.go`,
`internal/framework/validators/*_test.go`, and
1. **Input loading:** CLI filesystem adapters read inputs; schema parsing produces `cmd/audita/main_integration_test.go`.
source transcript and glossary objects. Source segments are converted to the
canonical transcript shape before normalization.
2. **Normalization:** the normalizer sorts and merges segments using its configured
time/token rules, reassigns IDs, and produces summary counters.
3. **Chunking:** the chunker partitions the normalized transcript and returns
sections and token estimates, or fails when constraints are impossible.
4. **Proposal generation:** the runner executes configured module instances in
sequence. Sections can generate proposals concurrently; results are collected
for deterministic indexing.
5. **Validation:** deterministic validators run before LLM-backed validators.
LLM validation is batched by prompt budget; candidate decisions feed application.
6. **Application:** `internal/framework/proposals` clones transcript state and
applies approved proposals in index order through replacement-policy checks.
7. **Serialization and reporting:** the CLI encodes the final transcript, maps
runner output into reports, writes artifacts, and delegates retention to
diagnostics. See [report assembly](diagnostics-reporting.md).
## Runner Outputs
`runner.Run` returns:
- final transcript;
- per-module results.
CLI/reporting then map these results into process reports and diagnostics
artifacts. PromptKit execution observations are collected at the role-client
boundary for the new report summary.
## Failure Behavior
CLI orchestration labels failures at the phase boundary. Runner execution failures
can return completed or partial module results, which report assembly preserves.
[Report fields](../integrations/output-files.md#process-report) define how consumers
see these outcomes; [CLI semantics](../cli.md#streams-and-exit-status)
define streams and exits. Diagnostic write failures do not always abort processing.
## Ordering and State
The runner mutates its working transcript between module instances, after collecting
section results and validating proposals. Proposal indexes follow section order;
reports are derived from actual run results. Global requirements are owned by the
[architecture policy](../policy/architecture.md#correction-and-concurrency-invariants).
## Key Tests
- `internal/framework/runner/runner_test.go`
- `internal/framework/proposal_generation/*_test.go`
- `cmd/audita/main_integration_test.go`

View File

@@ -42,7 +42,7 @@ Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chai
For each module section: For each module section:
1. run deterministic validators; 1. run deterministic validators;
2. run LLM-backed validators; 2. run LLM-backed validators;
3. record decisions and warnings; 3. record decisions and compact operation outcomes;
4. carry only approved proposals forward. 4. carry only approved proposals forward.
Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator. Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator.
@@ -55,15 +55,16 @@ LLM validators:
- select an embedded prompt ID and pass payload/description variables to the - select an embedded prompt ID and pass payload/description variables to the
role-bound structured client with the validator-decision schema. role-bound structured client with the validator-decision schema.
Oversized single proposals are rejected with `validator_input_too_large`. Oversized single inputs are rejected with a consolidated
Exhausted structural repair becomes a warning and rejected batch decisions. `validation_input_too_large` operation, without a provider call. Recoverable
Provider, malformed HTTP envelope, deadline, and admission failures propagate model failures make only their batch candidates unavailable while other
as runtime errors. Schema-valid decisions still undergo exact index/cardinality batches continue. Missing, duplicate, or unknown decision indices invalidate
checks and domain acceptance without a semantic retry. that batch as a whole. Fatal errors stop the chain without returning unchecked
candidates as approved. Domain acceptance has no semantic retry.
## Decision and Rejection Reporting ## Decision and Rejection Reporting
Runner collects decisions, rejected candidates, and warning records. Ledger Runner collects decisions, rejected candidates, and operation outcomes. Ledger
assembly groups decisions by deterministic versus LLM-backed execution metadata. assembly groups decisions by deterministic versus LLM-backed execution metadata.
The serialized names and fields belong to the The serialized names and fields belong to the
[report and ledger contract](../integrations/output-files.md); implementation mapping [report and ledger contract](../integrations/output-files.md); implementation mapping

View File

@@ -19,8 +19,11 @@ Settings resolve from defaults, main file, optional `--overlay <name>` in
environment overlays no longer apply. Unknown fields and version-1 files fail; environment overlays no longer apply. Unknown fields and version-1 files fail;
there is no automatic converter. there is no automatic converter.
Keep transcript as the positional path and `--glossary`, `--output`, and Keep transcript as the positional path and `--glossary` and `--output` as
`--report-json` as invocation paths. Keep `pipeline.modules` / `--modules` and invocation paths. The former `--report-json` output has been replaced by the
default `errors.yml`, `recoveries.yml`, and `report.md` bundle; use
`--reporting=false` to disable it or `--report-dir` for stdout output. Keep
`pipeline.modules` / `--modules` and
`output.schema` / `--output-schema`. An explicit `pipeline.modules: []` or `output.schema` / `--output-schema`. An explicit `pipeline.modules: []` or
`--modules ''` runs no correction modules; omission uses the default sequence. `--modules ''` runs no correction modules; omission uses the default sequence.
Ordered repetitions still matter. Ordered repetitions still matter.
@@ -73,14 +76,16 @@ The following domain settings and explicit flags retain their meaning:
| `normalization.max_segment_gap`, `.ellipsis_gap`, `.max_segment_duration`, `.max_segment_tokens` | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` | | `normalization.max_segment_gap`, `.ellipsis_gap`, `.max_segment_duration`, `.max_segment_tokens` | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` |
| `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` | | `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` |
| `context.description` | `--transcript-description` | | `context.description` | `--transcript-description` |
| `diagnostics.work_dir`, `.retention` | `--work-dir`, `--work-dir-retention` | | `reporting.enabled`, `.stdout_dir` | `--reporting[=true\|false]`, `--report-dir` |
| `diagnostics.debug`, `.work_dir` | `--debug[=true\|false]`, `--work-dir` |
Normalization gaps and durations are **numeric seconds**, so replace `4s`, Normalization gaps and durations are **numeric seconds**, so replace `4s`,
`3.5s`, and `60s` with `4.0`, `3.5`, and `60.0`. The operation timeout remains a `3.5s`, and `60s` with `4.0`, `3.5`, and `60.0`. The operation timeout remains a
Go duration string. `chunking.target_sections` may be omitted or null for Go duration string. `chunking.target_sections` may be omitted or null for
automatic chunking; zero is invalid. Null is not a general synonym for omission: automatic chunking; zero is invalid. Null is not a general synonym for omission:
nonoptional scalar settings reject it. Empty `context.description` clears nonoptional scalar settings reject it. Empty `context.description` clears
context. Retention behavior is unchanged. context. Legacy `diagnostics.retention` and `--work-dir-retention` are removed.
Debug is off by default and its directory is retained whenever enabled.
Remove old implicit variables `AUDITA_MODULES`, `AUDITA_MODEL`, Remove old implicit variables `AUDITA_MODULES`, `AUDITA_MODEL`,
`AUDITA_VALIDATION_MODEL`, `AUDITA_BASE_URL`, `AUDITA_VALIDATION_BASE_URL`, `AUDITA_VALIDATION_MODEL`, `AUDITA_BASE_URL`, `AUDITA_VALIDATION_BASE_URL`,
@@ -109,14 +114,14 @@ calling a provider. `--overlay testing` selects the same overlay for validation,
inspection, and processing. The old effective-config JSON format is replaced by inspection, and processing. The old effective-config JSON format is replaced by
YAML; settings YAML alone does not show profile-owned model controls. YAML; settings YAML alone does not show profile-owned model controls.
The [process report](../integrations/output-files.md#process-report) now uses The [ordinary output contract](../integrations/output-files.md) now uses
schema `v2` and includes an `llm_summary` when execution observations are version-1 `audita-errors` and `audita-recoveries` YAML documents plus a fixed
available. Logical calls are distinct from provider attempts; usage fields carry Markdown overview. Compact outcomes distinguish logical operations from
known and complete flags. Utilization is grouped by backend capacity rather than provider attempts. With debug enabled, utilization remains grouped by backend
old scheduler and active-concurrency fields. Retained runs write capacity and usage fields retain known/complete flags. Debug runs write
`effective-config.redacted.yaml`, `selected-profiles.yaml`, `effective-config.redacted.yaml`, `selected-profiles.yaml`,
`config-origins.yaml`, `utilization-diagnostics.json`, and `config-origins.yaml`, `utilization-diagnostics.json`, and
`llm-execution.json`. Request and response artifacts remain redacted debugging `llm-execution.json`. Request and response artifacts remain redacted debugging
material. See [operations](../operations.md#diagnostics-directory) for retention material. See [operations](../operations.md#diagnostics-directory) for placement
and paths, [configuration](../config.md) for all settings, and the and paths, [configuration](../config.md) for all settings, and the
[CLI reference](../cli.md) for current flags. [CLI reference](../cli.md) for current flags.

View File

@@ -1,113 +1,73 @@
# Audita Operations # Audita Operations
This document owns runtime workflow, physical artifact placement, retention, and This guide owns physical output placement and failure handling. The
recovery. Use the [CLI reference](cli.md) for invocation and stream/exit semantics, [CLI reference](cli.md) owns flags and streams, [configuration](config.md) owns
[configuration](config.md) for settings, and [output-file contracts](integrations/output-files.md) settings, and the [output contract](integrations/output-files.md) owns schemas.
for logical formats.
## Normal Workflow ## Normal Workflow
1. Choose a configuration and validate it using the 1. Validate a version-2 configuration and its selected profiles with
[configuration commands](cli.md#commands-and-help). `audita config validate --config config.yml`.
2. Prepare input files conforming to the [input contract](integrations/transcript-glossary-files.md). 2. Prepare a transcript and glossary following the [input contract](integrations/transcript-glossary-files.md).
3. Run the [processing command](cli.md#commands-and-help), choosing explicit output 3. Run `audita process input.json --config config.yml --glossary glossary.yaml --output corrected.json`.
and report destinations when those files must survive run-directory cleanup. 4. Check the exit status, stderr summary, corrected transcript, and reports.
4. Inspect the corrected transcript and process report. Review rejected or skipped A completed run can contain recoverable model errors and still apply
corrections when the report indicates them; successful execution does not mean corrections that passed every required validator.
every proposed correction was applied.
Processing creates a run directory, loads and normalizes the inputs, chunks the ## Required Output Files
transcript, executes the correction pipeline, writes results, and applies
retention. Early configuration failures happen before run-directory creation.
Failures after initialization attempt to preserve diagnostic evidence; individual
artifact writes can fail, so a run directory need not contain every listed file.
## Output Files Reporting is enabled by default. For file transcript output, `errors.yml`,
`recoveries.yml`, and `report.md` go in the transcript output's parent directory.
For stdout transcript output, they go in `reporting.stdout_dir` (default `.`),
overridden by `--report-dir`. `--reporting=false` suppresses all three reports
and removes their path requirements. It does not suppress the stderr summary or
fatal errors. Stdout contains only transcript JSON; file output leaves it empty.
The caller chooses transcript and optional external report destinations using the Existing report directories and transcript-output parents must be writable.
[CLI](cli.md#streams-and-exit-status). These are separate from the per-run Audita checks active destinations against input, configuration, and selected
diagnostic directory unless the caller deliberately places them inside it. profile files before model dispatch. It preflights file transcript output
Choose existing, writable parent directories. Output/report files can be overwritten without truncating an existing file. With reporting enabled, all three reports
by subsequent runs using the same destinations. are initialized as `in_progress` before input processing. Output is encoded
fully before publication. Each file is replaced through a temporary file in
A later report or cleanup failure can occur after transcript output was written. its destination directory: transcript first, then `errors.yml`,
Inspect the process result and available report before treating the files as a `recoveries.yml`, and `report.md`. This is per-file replacement, not a
completed run. Retention of the diagnostic directory does not archive caller-selected multi-file transaction. A late report error may leave the transcript published;
output files elsewhere. the process exits nonzero and attempts failed-status reports. On fatal processing
or cancellation, no completed transcript is published, but partial reports are
finalized where possible. An uncatchable termination can leave `in_progress`.
## Diagnostics Directory ## Diagnostics Directory
The [diagnostics configuration](config.md#settings) `--debug` enables detailed evidence independently of reporting. One directory
selects the parent work directory. Runs create children named like named `run-<id>` is created under `diagnostics.work_dir` (default `/tmp/audita`)
`run-<unix-nanoseconds>`. The runtime creates the work directory if needed. and retained on success or failure. Without debug, no directory or payload
Report paths may be relative when the supplied work directory is relative; files are created. The parent is created when needed only for debug. Failure to
resolve them using the invocation's working directory. write optional debug evidence prints a redacted stderr diagnostic but does not
change correction decisions or the exit status.
| Artifact in the run directory | Purpose | Debug evidence includes invocation and effective settings, configuration
| --- | --- | origins, selected profiles, redacted source and normalized snapshots,
| `source-transcript.json` | Original transcript input | normalization and initial chunk summaries, all captured proposed candidates
| `source-transcript-parsed.json` | Parsed source representation | and validator decisions in `correction-ledger.json`, execution/utilization
| `normalized-transcript.json` | Transcript after normalization | records, and model interaction files. Individual artifact writes can fail; a
| `normalization-summary.json` | Normalization counters | directory's existence does not prove every file was written. These files can
| `chunking-summary.json` | Section/token summary | contain transcript text and prompt/response payloads. Configured backend
| `invocation.json` | Invocation metadata | credentials are redacted, but the transcript itself is not anonymized. Protect
| `effective-config.redacted.yaml` | Redacted ConfigKit configuration snapshot | and retain the directory according to the sensitivity of the input.
| `config-origins.yaml` | Safe configuration origin and resolution metadata |
| `selected-profiles.yaml` | Safe selected role profiles and model controls |
| `report.json` | Run's [process report](integrations/output-files.md#process-report) |
| `correction-ledger.json` | [Correction dispositions](integrations/output-files.md#correction-ledger) |
| `utilization-diagnostics.json` | [Backend execution and domain timing observations](integrations/output-files.md#utilization-diagnostics) |
| `llm-execution.json` | Safe logical-call and execution event records |
| `error.log` | Failure details when written |
LLM request, response, and error diagnostics are also written within the run
directory. Their nested naming and payloads are debugging details. Prefer paths
provided by the report and its decision/warning records over reconstructing those
paths. The report describes intended artifact locations, not proof that every
write succeeded or that the files still exist after retention.
## Retention Behavior
[Configuration](config.md#settings) owns accepted
retention values and the default. Current lifecycle behavior is:
| Run outcome / selection | Directory outcome |
| --- | --- |
| Failed run | Retained regardless of selection |
| Successful run, `always` | Retained |
| Successful run, `auto` | Retained if corrections were skipped or rejected; otherwise removed |
| Successful run, `never` | Currently retained, the same outcome as `always` |
The name `never` does not currently mean deletion. Warnings alone do not trigger
retention under `auto` unless they also produce skipped or rejected corrections.
An external report outside the run directory survives automatic cleanup, but its
artifact paths can then point to removed files.
## Artifact Handling
Run artifacts can contain full transcripts, glossary-related context, and LLM
prompts/responses. Configured-key redaction does not anonymize those contents.
Choose a work directory whose ownership and permissions fit the transcript data,
and retain or share artifacts intentionally. Credential-supply mechanisms are
owned by [configuration](config.md#credentials).
For a debugging run, select retained diagnostics using the
[retention flag](cli.md#setting-overrides). Copy needed artifacts before
removing completed run directories manually. Avoid deleting a directory while
its run is active. Audita does not provide a separate archival/backup or remote
artifact storage facility.
## Failure Inspection and Recovery ## Failure Inspection and Recovery
1. Read stderr and the exit classification defined by the [CLI](cli.md#streams-and-exit-status). Exit `0` means processing and every enabled required output completed. Exit
2. Locate the external report, or the run directory identified by the error when `1` means cancellation, fatal processing, or required-output failure. Exit `2`
available. If a late error omits the path, inspect the selected work directory. identifies usage or configuration/destination selection errors. Recoverable
3. Read the available report and error log. Use model-operation failures appear in `errors.yml` as one record per affected
[report status and error fields](integrations/output-files.md#process-report) work unit; they do not automatically make the run fail. `recoveries.yml`
to identify the failed phase, then inspect referenced artifacts. records successful operations that needed retry, repair, or proposal filtering.
4. Correct the concrete input, configuration, endpoint, or filesystem issue and `report.md` shows every initial chunk and configured module instance, including
rerun the command with diagnostics retained if further investigation is needed. interrupted and unattempted work. Ordinary reports omit transcript excerpts and
full decision ledgers; enable debug when those are needed.
Each invocation starts a new run. There is no implemented resume/checkpoint API; Read stderr and report status first. If a required output failed after a
recovery is a rerun, not continuation from an artifact. Failures during early transcript was published, do not treat the file alone as proof of a completed
configuration or directory creation may leave no report or diagnostic files. run. Fix the input, service, or destination problem and rerun. Audita has no
resume/checkpoint API.

View File

@@ -2,7 +2,7 @@
**Base Policy Version:** 1.0 **Base Policy Version:** 1.0
**Last Updated**: 2026-09-23 **Last Updated**: 2026-09-24
## Purpose and System Shape ## Purpose and System Shape
@@ -48,7 +48,9 @@ hidden global state.
- Keep LLM concurrency bounded and context-aware through PromptKit's engine-local - Keep LLM concurrency bounded and context-aware through PromptKit's engine-local
per-backend admission. Proposal and validation calls selecting the same backend per-backend admission. Proposal and validation calls selecting the same backend
share that capacity, including retries and repairs. Preserve deterministic share that capacity, including retries and repairs. Preserve deterministic
collection order and cancellation of remaining work on runtime failure. collection order. Recoverable model-operation failures stay local to their
proposal response or validation batch; fatal invariant/application failure
and parent cancellation stop peers and join them.
- Keep module and validator identities, reason codes, and execution-class - Keep module and validator identities, reason codes, and execution-class
distinctions stable where exposed as contracts. distinctions stable where exposed as contracts.
@@ -61,12 +63,14 @@ describe how these requirements are implemented.
Keep transcript mutation and output handling deterministic around external Keep transcript mutation and output handling deterministic around external
calls. Reports and correction ledgers are derived from processing results. calls. Reports and correction ledgers are derived from processing results.
Preserve the separation between transient processing state and filesystem Preserve the separation between transient processing state and filesystem
adapters; physical artifact placement and retention belong to adapters; physical report placement and optional debug retention belong to
[operations](../operations.md). [operations](../operations.md).
Public CLI, configuration, output-schema, diagnostics, report, prompt, module, Public CLI, configuration, output-schema, diagnostics, report, prompt, module,
and validator contracts should remain stable unless a change is deliberate and and validator contracts should remain stable unless a change is deliberate and
tested. Prefer additive, compatible report changes. Current implementation tested. The resilience work intentionally replaced the earlier JSON process
report with versioned errors/recoveries YAML and a fixed Markdown overview;
future changes should treat those current contracts deliberately. Current implementation
limits are documented in the [internal overview](../internal/overview.md#current-scope); limits are documented in the [internal overview](../internal/overview.md#current-scope);
they are not additional prohibitions on future development. they are not additional prohibitions on future development.

View File

@@ -3,7 +3,7 @@
**Last updated:** 2026-09-24 **Last updated:** 2026-09-24
**Status:** Production implementation complete. The selected configuration and **Status:** Production implementation complete. The selected configuration and
overlay model is implemented; [implementation evidence](implementation.md) overlay model is implemented; [implementation evidence](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
records verification. No release has been published. records verification. No release has been published.
## Purpose and Direction ## Purpose and Direction
@@ -137,7 +137,7 @@ ConfigKit and PromptKit form one intentional breaking refactor. Intermediate
commits need not build or run. The target contains no temporary old-runtime adapter, commits need not build or run. The target contains no temporary old-runtime adapter,
compatibility shim, processing gate, dual loader, or automatic legacy conversion. compatibility shim, processing gate, dual loader, or automatic legacy conversion.
The [combined feature roadmap](library-adoption.md) owns execution/reporting intent; The [combined feature roadmap](library-adoption.md) owns execution/reporting intent;
[implementation.md](implementation.md) alone owns stage order and implementation tasks. [implementation.md](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) alone owns stage order and implementation tasks.
The final application loads all retained domain settings, assembles both selected The final application loads all retained domain settings, assembles both selected
profiles, and processes through PromptKit. Controlled HTTP evidence must connect profiles, and processes through PromptKit. Controlled HTTP evidence must connect

View File

@@ -782,6 +782,17 @@ has no obsolete report/retention path keeping debug enabled implicitly.
## Stage 13 — Update Canonical Guidance, Examples, and Remove Obsolete Material ## Stage 13 — Update Canonical Guidance, Examples, and Remove Obsolete Material
**Completed 2026-09-24.** Canonical CLI, configuration, operations, integration,
internal, architecture, README, example, and migration guidance now describes
three ordinary reports, independent debug selection, partial outcomes, fixed
matrices, and finalization/exit behavior. The obsolete retention follow-up is
resolved while timestamp policy remains open. Historical library-adoption
links point to the prior plan's completion evidence. The unreferenced warning
package and retention-only helper/tests were removed. Maintained config and
profile inspection examples passed offline, relative file and heading links
resolved, and `go test ./internal/framework/processreport
./internal/core/config ./internal/cli` plus `git diff --check` passed.
**Work:** Update `docs/cli.md`, `config.md`, `operations.md`, integration output/subprocess/ **Work:** Update `docs/cli.md`, `config.md`, `operations.md`, integration output/subprocess/
LLM contracts, internal pipeline/validators/LLM/diagnostics/overview guidance, README, LLM contracts, internal pipeline/validators/LLM/diagnostics/overview guidance, README,
and maintained examples/migration guidance. Architecture policy must distinguish and maintained examples/migration guidance. Architecture policy must distinguish

View File

@@ -3,8 +3,9 @@
**Last updated:** 2026-09-24 **Last updated:** 2026-09-24
**Status:** Production implementation complete. ConfigKit v0.4.0 and PromptKit **Status:** Production implementation complete. ConfigKit v0.4.0 and PromptKit
v0.11.0 are adopted; [implementation evidence](implementation.md) records the v0.11.0 are adopted; [historical implementation evidence](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) records the
completed checks. Release compliance and publication remain separate work. completed checks. The current `implementation.md` is a separate resilience
roadmap. Release compliance and publication remain separate work.
## Goal and Ownership ## Goal and Ownership

View File

@@ -312,12 +312,12 @@ claims of completion for the prototype.
The new profile/credential/capacity and deadline choices are explicit above; no The new profile/credential/capacity and deadline choices are explicit above; no
unresolved upstream requirement prevents starting the ConfigKit implementation unresolved upstream requirement prevents starting the ConfigKit implementation
plan. Production report field names and final runtime wiring are selected in the plan. Production report field names and final runtime wiring are selected in the
[implementation plan](implementation.md), following the outcome and observation [implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md), following the outcome and observation
contract here. The coordinated refactor targets both libraries directly; intermediate commits do not require a working legacy runtime. contract here. The coordinated refactor targets both libraries directly; intermediate commits do not require a working legacy runtime.
### Production Implementation Reference ### Production Implementation Reference
The [implementation plan](implementation.md) owns ordered production work, The [implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md) owns ordered production work,
component changes, tests, and acceptance criteria. The [combined roadmap](library-adoption.md) component changes, tests, and acceptance criteria. The [combined roadmap](library-adoption.md)
and [configuration roadmap](configkit.md) own the final target. This evidence does and [configuration roadmap](configkit.md) own the final target. This evidence does
not require an interim runtime bridge or separately runnable configuration migration. not require an interim runtime bridge or separately runnable configuration migration.
@@ -388,5 +388,5 @@ The tests use a controlled built-in HTTP transport and synthetic credentials;
there were no live provider calls. The prototype still covers one representative there were no live provider calls. The prototype still covers one representative
proposal/assessment pair, not all production modules, prompts, CLI paths, or proposal/assessment pair, not all production modules, prompts, CLI paths, or
diagnostic artifacts. Root `go test ./...` does not run this nested module. No diagnostic artifacts. Root `go test ./...` does not run this nested module. No
upstream API gap appeared in this extension. The [production implementation plan](implementation.md) upstream API gap appeared in this extension. The [production implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
keeps implementation and release pending. keeps implementation and release pending.

View File

@@ -12,7 +12,7 @@ changes were made in that historical unit.
## Next Unit: Testing Overlays and Production Migration Plan ## Next Unit: Testing Overlays and Production Migration Plan
**Status:** Complete. The [production implementation plan](implementation.md) **Status:** Complete. The [production implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md)
owns subsequent work; the [overlay findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24) owns subsequent work; the [overlay findings](library-boundary-findings.md#selected-overlay-extension-2026-09-24)
record this unit's checks. The original work packages below remain record this unit's checks. The original work packages below remain
historical scope for the completed first unit. historical scope for the completed first unit.

View File

@@ -2,7 +2,7 @@
**Reviewed:** 2026-09-24, source commit `7cefb3f`. **Reviewed:** 2026-09-24, source commit `7cefb3f`.
**Scope:** The twelve implementation commits after `1ad2119`, compared with the **Scope:** The twelve implementation commits after `1ad2119`, compared with the
[implementation plan](implementation.md), [combined target](library-adoption.md), [implementation plan](https://gitea.maximumdirect.net/eric/audita/src/commit/cb405f6/docs/roadmap/implementation.md), [combined target](library-adoption.md),
and [configuration target](configkit.md). and [configuration target](configkit.md).
**Status:** R1–R5 addressed and verified on 2026-09-24. The evidence in the **Status:** R1–R5 addressed and verified on 2026-09-24. The evidence in the
original findings below describes source commit `7cefb3f`; line references are original findings below describes source commit `7cefb3f`; line references are

View File

@@ -2,13 +2,13 @@
**Last updated:** 2026-09-24 **Last updated:** 2026-09-24
The documentation migration identified these existing behaviors for separate The documentation migration identified existing behaviors for separate product
product consideration. Implementation status and any selected direction are noted consideration. The resilience implementation resolved the diagnostics-retention
below; this document does not claim that runtime changes have landed. question; the input-timestamp question remains open.
- Decide whether the `never` retention selection should delete successful-run - Resolved: legacy `auto`/`always`/`never` retention settings were removed.
diagnostics; current behavior is documented in Debug is explicitly enabled with `diagnostics.debug` or `--debug`, creates no
[operations](../operations.md#retention-behavior). directory when disabled, and retains its directory on every outcome when enabled.
- Resolved: [configuration](../config.md) now treats explicit zero as disabling - Resolved: [configuration](../config.md) now treats explicit zero as disabling
transient retries or structural repair independently; PromptKit performs both. transient retries or structural repair independently; PromptKit performs both.
- Decide whether missing or null input timestamps should be rejected; current - Decide whether missing or null input timestamps should be rejected; current

View File

@@ -32,6 +32,9 @@ thresholds:
grammar: 0.8 grammar: 0.8
context: context:
description: "General context for domain vocabulary and speaker names." description: "General context for domain vocabulary and speaker names."
reporting:
enabled: true
stdout_dir: .
diagnostics: diagnostics:
debug: false
work_dir: /tmp/audita work_dir: /tmp/audita
retention: auto

View File

@@ -171,18 +171,6 @@ func BuildCorrectionLedger(input CorrectionLedgerInput) []CorrectionLedgerEntry
return entries return entries
} }
func HasSkippedCorrections(runOutput *runner.RunOutput) bool {
if runOutput == nil {
return false
}
for _, mr := range runOutput.ModuleResults {
if len(mr.SkippedChanges) > 0 || len(mr.ValidatorRejected) > 0 {
return true
}
}
return false
}
func filterLedgerDecisions(in []runner.ValidatorDecisionRecord, wantLLM bool) []LedgerValidatorDecisionRecord { func filterLedgerDecisions(in []runner.ValidatorDecisionRecord, wantLLM bool) []LedgerValidatorDecisionRecord {
if len(in) == 0 { if len(in) == 0 {
return nil return nil

View File

@@ -112,21 +112,6 @@ func TestBuildCorrectionLedgerPreservesDispositionPolicy(t *testing.T) {
} }
} }
func TestHasSkippedCorrectionsIncludesApplicationSkipsAndValidatorRejections(t *testing.T) {
if HasSkippedCorrections(nil) {
t.Fatal("nil output should not have skipped corrections")
}
if HasSkippedCorrections(&runner.RunOutput{ModuleResults: []runner.ModuleResult{{AppliedChanges: []proposals.AppliedChange{{ProposalIndex: 1}}}}}) {
t.Fatal("applied-only output should not have skipped corrections")
}
if !HasSkippedCorrections(&runner.RunOutput{ModuleResults: []runner.ModuleResult{{SkippedChanges: []proposals.SkippedChange{{ProposalIndex: 1}}}}}) {
t.Fatal("application skips should count as skipped corrections")
}
if !HasSkippedCorrections(&runner.RunOutput{ModuleResults: []runner.ModuleResult{{ValidatorRejected: []runner.ValidatorRejectedChange{{ProposalIndex: 1}}}}}) {
t.Fatal("validator rejections should count as skipped corrections")
}
}
func TestCorrectionLedgerIncludesUnappliedCandidateAfterFatalValidation(t *testing.T) { func TestCorrectionLedgerIncludesUnappliedCandidateAfterFatalValidation(t *testing.T) {
output := &runner.RunOutput{ output := &runner.RunOutput{
Candidates: []proposals.EnrichedCorrectionProposal{{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 7, OriginalText: "old", CorrectedText: "new"}, ProposalMetadata: proposals.ProposalMetadata{ModuleKey: "grammar", ModuleInstance: "grammar", ProposalIndex: 2}}}, Candidates: []proposals.EnrichedCorrectionProposal{{CorrectionProposal: proposals.CorrectionProposal{TargetSegmentID: 7, OriginalText: "old", CorrectedText: "new"}, ProposalMetadata: proposals.ProposalMetadata{ModuleKey: "grammar", ModuleInstance: "grammar", ProposalIndex: 2}}},

View File

@@ -1,18 +0,0 @@
package warnings
type Scope string
const (
ScopeProposalGeneration Scope = "proposal_generation"
ScopeValidator Scope = "validator"
)
type StageWarning struct {
Scope Scope `json:"scope"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
SectionIndex *int `json:"section_index,omitempty"`
ValidatorName string `json:"validator_name,omitempty"`
BatchIndex *int `json:"batch_index,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
}