Refresh documentation and policies and simplify releases

docs/integrations/output-files.md

@@ -0,0 +1,186 @@
+# Output Files and Report Formats
+
+This document owns externally consumed corrected-transcript and report formats,
+and describes the correction ledger and utilization diagnostics. It documents
+current serialized shapes without making every debugging artifact a versioned
+compatibility promise. Physical filenames, placement, retention, and permissions
+belong to [operations](../operations.md); invocation and output channels belong to
+the [CLI](../cli.md#process-output-and-exit-behavior).
+
+## Corrected Transcript
+
+The [output setting](../config.md#pipeline-and-output) selects the envelope:
+
+| Encoder | JSON shape |
+| --- | --- |
+| `bare-segments` | Top-level array of segment objects |
+| `audita-v1` | Object with `schema: "audita-v1"`, `version: "v1"`, and `segments` containing the segment array |
+
+Both emit these segment fields:
+
+| Field | Type | Meaning |
+| --- | --- | --- |
+| `id` | Integer | Identifier in the normalized transcript |
+| `speaker` | String | Speaker label |
+| `start`, `end` | Numbers | Start/end times in seconds |
+| `text` | String | Text after normalization and accepted corrections |
+| `categories` | Array of strings, optional | Segment categories; omitted when empty |
+
+Input IDs and segment boundaries need not survive normalization unchanged;
+normalization can merge segments and reassign IDs. The output describes the
+resulting transcript, not a lossless copy of the source JSON. Input acceptance
+rules belong to the [input-file contract](transcript-glossary-files.md).
+
+## Process Report
+
+The report is a JSON object. `report_metadata` identifies its schema separately
+from the transcript output envelope:
+
+| Metadata field | Type / current value |
+| --- | --- |
+| `report_schema_name` | String: `audita-process-report` |
+| `report_schema_version` | String: `v1` |
+| `output_schema` | String identifying the selected transcript encoder |
+| `config_version` | Optional integer, present when file configuration was loaded |
+
+The remaining top-level fields are:
+
+| Field | Type and meaning |
+| --- | --- |
+| `phase` | String; currently `default_pipeline` |
+| `status` | String: `success` or `failed` |
+| `operation` | String; currently `process` |
+| `transcript_path`, `glossary_path` | Input path strings |
+| `output_path` | Optional transcript destination string |
+| `modules` | Array of configured module keys, including repeats |
+| `started_at`, `completed_at` | RFC 3339 timestamps, possibly fractional; completion is an optional field populated by completed process runs |
+| `error_phase`, `error_message` | Optional failure classification and human-readable detail |
+| `input_segment_count`, `normalized_segment_count` | Optional integer normalization counts |
+| `normalization_merges`, `normalization_id_reassignments` | Optional integer transformation counts |
+| `normalization_skipped` | Object of optional integer counters: `different_speakers`, `gap_too_large`, `duration_exceeded`, `token_limit_exceeded`; can be empty |
+| `chunking` | Optional section summary described below |
+| `diagnostics` | Optional artifact-path metadata described below |
+| `modules_summary` | Optional aggregate module outcome |
+| `module_results` | Optional array of module outcomes |
+
+Failures can preserve partial module results. Missing summaries indicate unavailable
+information, not necessarily zero work. Early configuration failures occur before
+report construction. Report success describes processing results; a later file-write
+or cleanup error can still cause a nonzero CLI exit. Use the exit result and report
+together as explained in [subprocess guidance](subprocess.md).
+
+Error phases identify where processing failed, for example `transcript_read`,
+`glossary_read`, `transcript_schema`, `glossary_schema`, `chunking`, `runner_setup`,
+`runner_execution`, `output_schema`, `serialization`, `output_write`, and
+`stdout_write`. This is not an exhaustive enumeration or a promise that every
+late failure is represented in a successfully written report. Error messages are
+human-readable, not structured fields to parse for control flow.
+
+### Chunking and Diagnostics Metadata
+
+`chunking` contains integer fields `chunk_count`, `min_estimated_chunk_tokens`,
+`max_estimated_chunk_tokens`, `total_estimated_transcript_tokens`,
+`max_section_tokens`, and `min_section_tokens`, plus optional integer
+`target_sections`. Token counts are estimates.
+
+`diagnostics` contains optional path strings:
+
+- `directory_path`
+- `source_transcript_path`, `parsed_source_transcript_path`, `normalized_transcript_path`
+- `normalization_summary_path`, `chunking_summary_path`
+- `utilization_summary_path`, `correction_ledger_path`
+- `invocation_metadata_path`, `redacted_effective_config_path`, `error_log_path`
+
+The [operations guide](../operations.md#diagnostics-directory) defines the physical
+artifacts and path-lifetime caveats.
+
+### Module Outcomes
+
+`modules_summary` contains integer `module_count`, `total_applied_changes`, and
+`total_skipped_changes`, plus optional string `failed_module_instance`.
+`total_skipped_changes` counts both apply-time skipped changes and validator
+rejections. Their detailed records are stored in separate arrays.
+
+Each `module_results` object contains:
+
+| Field | Type and meaning |
+| --- | --- |
+| `module_key`, `module_instance` | Strings; the instance distinguishes repeated occurrences of a configured module |
+| `replacement_policy` | Optional string describing the module's replacement policy |
+| `status` | String describing module outcome (`success` or `failed`) |
+| `proposal_count` | Integer |
+| `started_at`, `completed_at` | Optional timestamps |
+| `error_message` | Optional failure detail |
+| `warnings` | Optional array of warning records |
+| `validator_decisions` | Optional array of decision records |
+| `validator_rejected` | Optional array of rejected-correction records |
+| `applied_changes`, `skipped_changes` | Optional arrays of apply outcomes |
+
+A validator decision has `validator_name` (string), `proposal_index` (integer),
+`approved` (boolean), `reason_code` and `message` (strings), and optional
+`diagnostic_artifact_path` (string).
+
+Rejected corrections include `validator_name`, `proposal_index`, `module_key`,
+`module_instance`, `target_segment_id`, `original_text`, `corrected_text`,
+`reason_code`, and `message`. Applied changes include the same correction identity
+and text fields without validator/reason/message fields, plus integer
+`replacement_count`. Skipped changes include correction identity/text fields,
+`skip_reason`, and `message`.
+
+A warning has string `scope`, `reason_code`, and `message`; optional integer
+`section_index` and `batch_index`; and optional string `validator_name` and
+`diagnostic_artifact_path`. Warnings, skipped changes, and rejected corrections
+can occur without a failed run. Proposal indexes associate outcomes with their
+proposal within a module instance; they are not transcript segment IDs.
+
+## Correction Ledger
+
+The ledger is an array of correction entries; it can be `null` when no module
+results are available and empty when modules produced no entries. Unlike the
+process report, it has no separate schema-version envelope.
+
+Each entry has required `module_key`, `module_instance`, integer `proposal_index`,
+and `disposition`. The current dispositions are:
+
+| Disposition | Meaning |
+| --- | --- |
+| `applied` | The proposed change was applied |
+| `skipped` | The proposal reached application but was skipped |
+| `rejected` | A validator rejected the proposal |
+| `failed` | Module-level failure record, not necessarily an individual proposal |
+
+Optional fields are `run_id`, integer `segment_id`, `original_text`,
+`proposed_corrected_text`, `applied_corrected_text`, `replacement_policy`,
+`disposition_reason_code`, and `disposition_message`. Optional validator arrays
+are `deterministic_validator_decisions` and `llm_validator_decisions`; each record
+contains string `validator_key`, boolean `approved`, string `reason_code`, and
+optional string `message`.
+
+Module failure entries can have the zero-value proposal index and no correction
+text or segment. Do not interpret them as evidence that proposal zero failed.
+Empty optional values are omitted. The ledger records disposition, not a replay
+or resume protocol.
+
+## Utilization Diagnostics
+
+This unversioned JSON object describes observations from the runtime, not service
+limits or performance guarantees:
+
+- `effective_concurrency`: integer `total_llm`, `proposal_llm`, `validation_llm`.
+- `run_timing`: millisecond integers `run_wall_time_ms`, `scheduler_queue_wait_ms`,
+  `llm_execution_time_ms`, `deterministic_validation_time_ms`; integer
+  `max_in_flight_llm_calls` and `average_in_flight_llm_calls`.
+- `llm_calls`: integer `total_proposal_calls` and `total_validation_calls`.
+- Optional `modules`: records with `module_key`, `module_instance`, integer
+  `proposal_llm_calls` and `validation_llm_calls`, and millisecond integer
+  `scheduler_queue_wait_ms`, `llm_execution_time_ms`,
+  `deterministic_validation_time_ms`, and `module_wall_time_ms`.
+- Optional `validators`: records with `module_key`, `module_instance`,
+  `validator_key`, boolean `llm_backed`, integer `calls`, and millisecond integer
+  `elapsed_ms`.
+
+Aggregated execution/wait durations can overlap in concurrent work; they need
+not sum to wall time. The observed average concurrency is serialized as an integer.
+Other source snapshots, prompt payloads, and diagnostic files are debugging
+material; their placement is documented in operations without defining a separate
+public schema for every file.