Document resilient reports and remove obsolete warning material
This commit is contained in:
@@ -1,91 +1,42 @@
|
||||
# Diagnostics and Report Assembly
|
||||
# Diagnostics and Reporting Internals
|
||||
|
||||
## Scope
|
||||
The [output-file contract](../integrations/output-files.md) owns public
|
||||
schemas and the [operations guide](../operations.md) owns physical placement.
|
||||
|
||||
This document owns implementation flow and collaborators. Physical artifacts and
|
||||
retention belong to [operations](../operations.md). Serialized reports, ledger
|
||||
records, and utilization shapes belong to the
|
||||
[output-file contract](../integrations/output-files.md).
|
||||
`internal/framework/outcomes` contains processing-owned operation references,
|
||||
counts, failure reasons, filtering details, and work states. It imports no
|
||||
reporting, CLI, or provider package. The runner records those facts per initial
|
||||
section and module instance. `internal/framework/processreport` maps them once
|
||||
into `internal/core/reporting.Snapshot`, including nullable counts, grouped
|
||||
work-unit errors, separate run errors, and one recovery per successful logical
|
||||
operation. Reports order stages, sections, validators, and batches by semantic
|
||||
identity, never concurrent completion time. The core reporting package owns
|
||||
Audita's versioned YAML DTOs, fixed Markdown rendering, and brief CLI
|
||||
completion formatter. YAML redaction walks serialized string nodes so numeric
|
||||
and boolean values retain their types. Ordinary reports omit raw causes,
|
||||
transcript text, complete decisions, and token/timing payloads.
|
||||
|
||||
## Run Directory and Artifact Writers
|
||||
`internal/core/diagnostics.OutputSession` owns active destination checks,
|
||||
transcript preflight, report initialization, and per-file final replacement.
|
||||
CLI composes it after configuration/profile inspection and before reading
|
||||
inputs or dispatching model work. It does not create ordinary report parents.
|
||||
On a final write error it attempts other enabled reports and one failed-status
|
||||
pass; a published transcript is never erased by a later report failure.
|
||||
|
||||
`internal/core/diagnostics` creates run directories, writes snapshots and metadata,
|
||||
and evaluates retention. CLI orchestration supplies the selected configuration,
|
||||
paths, run outcome, and correction-presence information. Artifact names are
|
||||
centralized in diagnostics; their operator-facing inventory is in
|
||||
[operations](../operations.md#diagnostics-directory).
|
||||
|
||||
Snapshot writes are attempted at the relevant processing phases. Several diagnostic
|
||||
writes are best-effort, so report metadata can identify a location without proving
|
||||
that a file was written. The CLI distinguishes primary processing errors from
|
||||
secondary diagnostic/report write errors.
|
||||
|
||||
## Process Report Mapping
|
||||
|
||||
`internal/framework/processreport.Build` maps invocation facts, normalization and
|
||||
chunking summaries, and runner output into `reporting.ProcessReport`. Core reporting
|
||||
owns the Go data structures and JSON writer. The builder attaches diagnostic
|
||||
metadata when a run-directory path is available.
|
||||
|
||||
Module mapping copies outcomes from runner results, collects summary counts, and
|
||||
preserves available partial module results on failure. The CLI failure path does
|
||||
not pass normalization or chunking summaries to report construction even when
|
||||
some earlier processing completed. Consumer interpretation belongs to the
|
||||
[process-report contract](../integrations/output-files.md#process-report).
|
||||
|
||||
## Correction Ledger and Utilization
|
||||
|
||||
`BuildCorrectionLedger` flattens module changes, rejections, and failures into
|
||||
ledger entries. It associates validator records by proposal index within each
|
||||
module, separates execution classes through validator metadata, and sorts entries
|
||||
by module instance, proposal index, and disposition.
|
||||
|
||||
Runner instrumentation and PromptKit observations collect utilization; CLI
|
||||
orchestration writes the backend summary and safe call/event artifact when run
|
||||
output is available. Report assembly maps the same observations to `llm_summary`
|
||||
without independently measuring calls. A selected backend remains attributable
|
||||
when preparation fails before dispatch. Domain module/validator timing stays
|
||||
separate from provider timing, and those durations can overlap.
|
||||
For the serialized forms see [ledger](../integrations/output-files.md#correction-ledger)
|
||||
and [utilization](../integrations/output-files.md#utilization-diagnostics).
|
||||
|
||||
## CLI Coordination and Retention
|
||||
|
||||
CLI orchestration writes caller-selected reports and run-local artifacts before
|
||||
asking diagnostics to apply retention. `HasSkippedCorrections` derives the
|
||||
retention input from runner outcomes. Keep the exact lifecycle definition in
|
||||
[operations](../operations.md#retention-behavior), not in another internal table.
|
||||
|
||||
On a processing failure, secondary report-write errors do not replace the original
|
||||
error. On the success path, an external report-write failure returns before the
|
||||
run-local report write; retention errors occur later. This ordering explains why
|
||||
artifacts and process exit status must be assessed together.
|
||||
|
||||
## Redaction
|
||||
|
||||
Effective configuration uses ConfigKit redaction. The CLI supplies all resolved
|
||||
backend credentials, including unselected backends, to report and diagnostic
|
||||
writers. `core/redaction` sanitizes the serialized representation rather than
|
||||
mutating processing results. JSON string values and keys are redacted after
|
||||
parsing with number preservation; YAML redaction preserves scalar types.
|
||||
Serialization failure returns an error without writing the original object.
|
||||
Interaction artifacts, correction ledgers, execution/utilization diagnostics,
|
||||
selected profiles, both report copies, and surfaced LLM errors share this
|
||||
string-redaction implementation. Longer overlapping values take precedence,
|
||||
and replacement text is not processed again within a redaction pass.
|
||||
|
||||
Execution reports index events by call ID. Utilization groups calls directly by
|
||||
backend and actual module instance, keeping module result order and retaining
|
||||
usage knownness/completeness. Section and batch indices remain separate from
|
||||
module identity; section-qualified interaction filenames avoid collisions.
|
||||
These boundaries implement the [architecture requirement](../policy/architecture.md#errors-and-secret-handling).
|
||||
Operational handling of transcript-bearing artifacts belongs to
|
||||
[operations](../operations.md#artifact-handling).
|
||||
Debug is independent. Only `diagnostics.debug` creates a unique retained child
|
||||
under `diagnostics.work_dir`. CLI passes empty diagnostics paths and nil writers
|
||||
when debug is off. With debug on, source/normalized snapshots, invocation,
|
||||
effective settings, origins, profiles, initial chunk summary, correction ledger,
|
||||
indexed execution/utilization, and model interactions are written. Runner
|
||||
candidate capture includes proposals later rejected or interrupted. Debug
|
||||
write failures produce safe stderr messages and do not abort processing.
|
||||
Every debug writer uses all configured backend secrets for redaction, including
|
||||
unselected backend credentials and source snapshots. Debug can still contain
|
||||
sensitive transcript and prompt content.
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/core/redaction/redaction_test.go`
|
||||
- `internal/core/diagnostics/*_test.go`
|
||||
- `internal/framework/processreport/*_test.go`
|
||||
- `internal/core/reporting/report_test.go`
|
||||
- `cmd/audita/main_integration_test.go`
|
||||
Useful tests: `internal/core/reporting/*_test.go`,
|
||||
`internal/core/diagnostics/*_test.go`,
|
||||
`internal/framework/processreport/*_test.go`, and
|
||||
`cmd/audita/main_integration_test.go`.
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
prompt and schema assets, registers application backends, and inspects the
|
||||
selected proposal and validation profiles offline. A role-bound `llm.RoleClient`
|
||||
implements `structuredoutput.Client`; proposal and validator contracts alias that
|
||||
interface during migration. Both roles share engine-local backend capacity.
|
||||
interface. Both roles share engine-local backend capacity.
|
||||
|
||||
For each logical call, the adapter starts an operation context and a diagnostic
|
||||
record before preparation. It supplies the reserved hardening text, the selected
|
||||
@@ -15,12 +15,13 @@ contract. `PrepareExecution` produces the exact messages and hashes retained in
|
||||
an Audita snapshot. The adapter checks the prepared backend and endpoint before
|
||||
`RunPrepared`. It records available final content, usage, model, and profile
|
||||
metadata and decodes only a validated result. Call hooks expose logical
|
||||
start/finish records and PromptKit execution events for the report collector.
|
||||
start/finish records and PromptKit execution events for debug utilization.
|
||||
|
||||
The [integration contract](../integrations/openai-compatible-llm.md) describes
|
||||
wire ownership and failure behavior. `structuredoutput.Failure` has a safe
|
||||
message and a typed cause; only exhausted structural repair is a malformed
|
||||
output failure. Secrets from all configured backends, including unselected
|
||||
message, fixed reason code, typed cause, and recoverable/fatal/canceled class.
|
||||
Counts distinguish provider attempts, transient retries, and structural
|
||||
repairs, including on errors. Secrets from all configured backends, including unselected
|
||||
ones, are available to the diagnostics redaction boundary.
|
||||
|
||||
## Orchestration
|
||||
@@ -29,10 +30,10 @@ CLI processing supplies two role clients from the same engine to the runner.
|
||||
PromptKit's per-backend admission applies to both roles and every generation,
|
||||
retry, and repair. The runner still processes module instances in order,
|
||||
collects concurrent section work deterministically, and cancels remaining
|
||||
section work on a runtime failure. `llm.DiagnosticsWriter` remains the protected
|
||||
section work only on fatal failures. `llm.DiagnosticsWriter` remains the protected
|
||||
request/response artifact writer. `llm.ObservationCollector` captures metadata
|
||||
events under a short mutex and freezes them after runner work joins. Its logical
|
||||
records include preparation failures; summary usage comes from terminal events
|
||||
once per call and retains knownness and completeness for each field. Runner wall
|
||||
and validator timings remain separate from provider attempts. The report and
|
||||
artifact mapping is completed by the following roadmap work.
|
||||
and validator timings remain separate from provider attempts. Ordinary reports
|
||||
derive compact runner outcomes; full observations are debug-only.
|
||||
|
||||
@@ -49,12 +49,13 @@ policy. Shared context encoding lives in `internal/framework/promptcontext`.
|
||||
Built-in chains are resolved in `internal/validators` per module key.
|
||||
Module packages call the built-in chain resolver at construction.
|
||||
|
||||
## Failure and Warning Behavior
|
||||
## Failure and Outcome Behavior
|
||||
|
||||
Module construction failures propagate to orchestration; execution failures can
|
||||
preserve partial module results. Exhausted structural repair becomes a warning
|
||||
and section-level rejection; provider and protocol failures propagate. Consumer-visible outcomes are defined by
|
||||
the [report contract](../integrations/output-files.md#module-outcomes).
|
||||
Module construction failures are fatal. Recoverable proposal model failures
|
||||
record an unsuccessful operation and yield no candidates for that section;
|
||||
other sections and module instances continue. Fatal adapter or application
|
||||
failures cancel peers and retain partial observations. Consumer-visible
|
||||
outcomes are defined by the [report contract](../integrations/output-files.md).
|
||||
|
||||
## Changing Modules
|
||||
|
||||
|
||||
@@ -47,7 +47,8 @@ Supporting areas:
|
||||
|
||||
## Current Scope
|
||||
|
||||
Runtime state is held in memory plus per-run diagnostics artifacts. Audita does
|
||||
Runtime state is held in memory; optional debug artifacts are retained only
|
||||
when requested. Audita does
|
||||
not currently implement a plugin framework, generic workflow engine,
|
||||
resume/checkpoint/manifest system, remote storage, or multi-process service mode.
|
||||
Its role-bound PromptKit adapter implements the structured LLM client contract.
|
||||
@@ -55,7 +56,9 @@ These are current implementation boundaries, not bans on future work.
|
||||
|
||||
## Main Execution Path (`audita process`)
|
||||
|
||||
The CLI prepares input and configuration, invokes the runner, and writes results.
|
||||
The CLI inspects configuration and profiles, preflights and initializes active
|
||||
outputs, optionally creates debug evidence, invokes the runner, and finalizes
|
||||
transcript and reports.
|
||||
The [pipeline](pipeline.md) owns detailed state transitions; the
|
||||
[operations guide](../operations.md) owns the operator workflow and physical
|
||||
artifact lifecycle.
|
||||
|
||||
@@ -1,66 +1,44 @@
|
||||
# Audita Internal Pipeline
|
||||
|
||||
## Scope
|
||||
The [input contract](../integrations/transcript-glossary-files.md) owns input
|
||||
formats; [configuration](../config.md) owns settings; the
|
||||
[output contract](../integrations/output-files.md) owns serialized results.
|
||||
|
||||
This document describes processing collaborators and state transitions. External
|
||||
input definitions belong to the [input-file contract](../integrations/transcript-glossary-files.md),
|
||||
settings to [configuration](../config.md), and serialized results to the
|
||||
[output-file contract](../integrations/output-files.md).
|
||||
1. CLI filesystem adapters read transcript and glossary files. Schema parsing
|
||||
produces source objects; transcript segments enter canonical normalization.
|
||||
2. Normalization sorts and merges segments using configured time and token
|
||||
limits, reassigning IDs where needed.
|
||||
3. The runner partitions the normalized transcript exactly once before its
|
||||
module loop. Initial section identity, segment membership, and token
|
||||
estimates stay fixed while text evolves.
|
||||
4. Module instances execute in configured order. Sections may propose
|
||||
concurrently, but proposal indices follow section order and response order.
|
||||
5. Deterministic validators precede LLM-backed validators. Every submitted
|
||||
candidate receives one decision per validator. Recoverable failed batches
|
||||
mark their candidates unavailable; successful survivors continue through
|
||||
all required validators.
|
||||
6. Only fully approved proposals reach application. Corrections apply in
|
||||
proposal-index order within their owning section. Existing replacement
|
||||
policies and apply-time safety checks resolve stale conflicts.
|
||||
7. CLI encodes the final transcript and publishes active files through the
|
||||
output session. `framework/processreport` builds one snapshot for YAML,
|
||||
Markdown, and CLI totals. Debug evidence is captured separately when enabled.
|
||||
|
||||
## Inputs
|
||||
`runner.Run` returns the final working transcript, initial section inventory,
|
||||
explicit `not_started`/`interrupted`/`completed` work units, compact model
|
||||
operation outcomes, and per-module results. Candidate payloads are retained
|
||||
in the returned output only when debug capture is requested. A work unit can
|
||||
complete with model errors and applied corrections. Recoverable model failures
|
||||
never trigger shared cancellation. Fatal setup, internal, application, or
|
||||
parent-cancellation errors cancel peers, join workers, and preserve earlier
|
||||
committed module text for partial reports; no completed transcript is published.
|
||||
|
||||
Pipeline inputs are:
|
||||
- effective config (`internal/core/config`);
|
||||
- transcript JSON (`internal/core/schema`);
|
||||
- glossary YAML (`internal/core/schema`).
|
||||
Required transcript/report write failures are fatal. Optional debug writes
|
||||
report safely to stderr without changing decisions or exit status. See
|
||||
[operations](../operations.md) for file placement and failure handling, and
|
||||
[architecture policy](../policy/architecture.md#correction-and-concurrency-invariants)
|
||||
for invariants.
|
||||
|
||||
## Pipeline Phases
|
||||
|
||||
1. **Input loading:** CLI filesystem adapters read inputs; schema parsing produces
|
||||
source transcript and glossary objects. Source segments are converted to the
|
||||
canonical transcript shape before normalization.
|
||||
2. **Normalization:** the normalizer sorts and merges segments using its configured
|
||||
time/token rules, reassigns IDs, and produces summary counters.
|
||||
3. **Chunking:** the chunker partitions the normalized transcript and returns
|
||||
sections and token estimates, or fails when constraints are impossible.
|
||||
4. **Proposal generation:** the runner executes configured module instances in
|
||||
sequence. Sections can generate proposals concurrently; results are collected
|
||||
for deterministic indexing.
|
||||
5. **Validation:** deterministic validators run before LLM-backed validators.
|
||||
LLM validation is batched by prompt budget; candidate decisions feed application.
|
||||
6. **Application:** `internal/framework/proposals` clones transcript state and
|
||||
applies approved proposals in index order through replacement-policy checks.
|
||||
7. **Serialization and reporting:** the CLI encodes the final transcript, maps
|
||||
runner output into reports, writes artifacts, and delegates retention to
|
||||
diagnostics. See [report assembly](diagnostics-reporting.md).
|
||||
|
||||
## Runner Outputs
|
||||
|
||||
`runner.Run` returns:
|
||||
- final transcript;
|
||||
- per-module results.
|
||||
|
||||
CLI/reporting then map these results into process reports and diagnostics
|
||||
artifacts. PromptKit execution observations are collected at the role-client
|
||||
boundary for the new report summary.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
CLI orchestration labels failures at the phase boundary. Runner execution failures
|
||||
can return completed or partial module results, which report assembly preserves.
|
||||
[Report fields](../integrations/output-files.md#process-report) define how consumers
|
||||
see these outcomes; [CLI semantics](../cli.md#streams-and-exit-status)
|
||||
define streams and exits. Diagnostic write failures do not always abort processing.
|
||||
|
||||
## Ordering and State
|
||||
|
||||
The runner mutates its working transcript between module instances, after collecting
|
||||
section results and validating proposals. Proposal indexes follow section order;
|
||||
reports are derived from actual run results. Global requirements are owned by the
|
||||
[architecture policy](../policy/architecture.md#correction-and-concurrency-invariants).
|
||||
|
||||
## Key Tests
|
||||
|
||||
- `internal/framework/runner/runner_test.go`
|
||||
- `internal/framework/proposal_generation/*_test.go`
|
||||
- `cmd/audita/main_integration_test.go`
|
||||
Key tests: `internal/framework/runner/*_test.go`,
|
||||
`internal/framework/validators/*_test.go`, and
|
||||
`cmd/audita/main_integration_test.go`.
|
||||
|
||||
@@ -42,7 +42,7 @@ Glossary, homophones, spoken_word, and grammar each resolve a fixed ordered chai
|
||||
For each module section:
|
||||
1. run deterministic validators;
|
||||
2. run LLM-backed validators;
|
||||
3. record decisions and warnings;
|
||||
3. record decisions and compact operation outcomes;
|
||||
4. carry only approved proposals forward.
|
||||
|
||||
Decision cardinality is enforced: each candidate proposal must receive exactly one decision per validator.
|
||||
@@ -55,15 +55,16 @@ LLM validators:
|
||||
- select an embedded prompt ID and pass payload/description variables to the
|
||||
role-bound structured client with the validator-decision schema.
|
||||
|
||||
Oversized single proposals are rejected with `validator_input_too_large`.
|
||||
Exhausted structural repair becomes a warning and rejected batch decisions.
|
||||
Provider, malformed HTTP envelope, deadline, and admission failures propagate
|
||||
as runtime errors. Schema-valid decisions still undergo exact index/cardinality
|
||||
checks and domain acceptance without a semantic retry.
|
||||
Oversized single inputs are rejected with a consolidated
|
||||
`validation_input_too_large` operation, without a provider call. Recoverable
|
||||
model failures make only their batch candidates unavailable while other
|
||||
batches continue. Missing, duplicate, or unknown decision indices invalidate
|
||||
that batch as a whole. Fatal errors stop the chain without returning unchecked
|
||||
candidates as approved. Domain acceptance has no semantic retry.
|
||||
|
||||
## Decision and Rejection Reporting
|
||||
|
||||
Runner collects decisions, rejected candidates, and warning records. Ledger
|
||||
Runner collects decisions, rejected candidates, and operation outcomes. Ledger
|
||||
assembly groups decisions by deterministic versus LLM-backed execution metadata.
|
||||
The serialized names and fields belong to the
|
||||
[report and ledger contract](../integrations/output-files.md); implementation mapping
|
||||
|
||||
Reference in New Issue
Block a user