diff --git a/README.md b/README.md index 50e14fe..8e69d52 100644 --- a/README.md +++ b/README.md @@ -45,9 +45,9 @@ YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast. ## Transcript Tiers - `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge -- `transcripts/processed.json`: full polished transcript from Audita polish +- `transcripts/processed.json`: full raw Audita-polished transcript output - `transcripts/normalized.json`: Seriatim-normalized transcript from the normalize stage -- `transcripts/trimmed.json`: gameplay-only polished transcript from trim stage +- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage ## Normalize Configuration @@ -65,6 +65,17 @@ Allowed `normalize.output_schema` values: `normalize.output_path` is treated as session-workdir-relative when not absolute. +Normalize stage behavior summary: + +- normalize runs after `polish` and before `trim` +- normalize resolves `transcripts/processed.json` +- normalize runs Seriatim `normalize` to produce `transcripts/normalized.json` +- normalize diagnostics are written to: + - `artifacts/seriatim.normalize.report.json` (when enabled) + - `logs/seriatim.normalize.stdout.log` + - `logs/seriatim.normalize.stderr.log` + - `config/seriatim.normalize.generated.yml` + ## Trim Configuration `pipeline.trim` is optional. If omitted, no trim config is loaded. If `trim.enabled` is omitted, it defaults to `false`. @@ -192,6 +203,7 @@ For the initial implementation, only `session_recap` generation is supported. Analyze-stage session recap behavior: +- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript` - session recap should use gameplay-only transcript input (`source: trimmed_transcript`) - Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json` - Narratio resolves `normalized_transcript` from normalize manifest output (`transcript_normalized`) or fallback `transcripts/normalized.json` diff --git a/architecture.md b/architecture.md index d968af7..8f191bc 100644 --- a/architecture.md +++ b/architecture.md @@ -175,13 +175,32 @@ trim: Narratio currently produces and uses four transcript tiers: - `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge -- `transcripts/processed.json`: full polished transcript from Audita polish (includes pre/post-game content) +- `transcripts/processed.json`: full raw Audita-polished transcript output (includes pre/post-game content) - `transcripts/normalized.json`: normalized transcript generated by Seriatim normalize -- `transcripts/trimmed.json`: gameplay-only polished transcript from trim stage +- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage Trim reads `transcripts/normalized.json`, validates bounds IDs against that same transcript ID space, and writes `transcripts/trimmed.json`. -## 7. Trim Stage (Current Implementation) +## 7. Normalize Stage (Current Implementation) + +Normalize stage behavior: + +- stage order position: after `polish` and before `trim` +- discovers processed transcript from manifest polish outputs (`transcript_processed`) when present, else `work//transcripts/processed.json` +- validates processed transcript JSON shape (`segments` array required) +- runs Seriatim `normalize` to produce normalized transcript +- validates normalized transcript JSON shape (`segments` array required) +- validates normalize report JSON when enabled + +Expected normalize outputs and diagnostics: + +- `transcripts/normalized.json` +- `artifacts/seriatim.normalize.report.json` (when normalize report is enabled) +- `logs/seriatim.normalize.stdout.log` +- `logs/seriatim.normalize.stderr.log` +- `config/seriatim.normalize.generated.yml` + +## 8. Trim Stage (Current Implementation) Trim stage behavior: @@ -216,7 +235,7 @@ Expected trim outputs and diagnostics: Render-debug files are diagnostics. They are recorded in stage metadata/log/config refs and are not treated as canonical stage output artifact refs. -## 8. Analyze Stage (Current Implementation) +## 9. Analyze Stage (Current Implementation) The current real analyze implementation supports only `scriptorium.artifacts.session_recap`. @@ -225,6 +244,7 @@ Behavior: - if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata - if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata - if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly +- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript` - `session_recap` should use `trimmed_transcript` input (`transcripts/trimmed.json`) for in-universe recap generation - `trimmed_transcript` input is resolved from manifest (`trim` output kind `transcript_trimmed`) when available, otherwise fallback path `work//transcripts/trimmed.json` - `normalized_transcript` input is resolved from manifest (`normalize` output kind `transcript_normalized`) when available, otherwise fallback path `work//transcripts/normalized.json` @@ -244,7 +264,7 @@ Behavior: - missing `trimmed_transcript` input for configured `trimmed_transcript` source fails clearly with guidance to run trim stage first - manifest records output refs, logs, generated config paths, and non-secret provenance metadata -## 9. Session Recap Paths +## 10. Session Recap Paths Current expected paths for `session_recap`: @@ -257,14 +277,14 @@ Current expected paths for `session_recap`: - render stderr log: `logs/scriptorium.session_recap.render.stderr.log` - render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml` -## 10. Security and Privacy +## 11. Security and Privacy - do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata - if API-key integration is configured, pass env var names only (never raw key values) - avoid logging transcript content or rendered prompt content by default - treat generated artifacts and logs as potentially sensitive session material -## 11. Operational Caveat (Pre-Stale-Detection) +## 12. Operational Caveat (Pre-Stale-Detection) Checksum-based stale detection is not implemented yet. @@ -276,7 +296,7 @@ Examples: - trim prompt/profile changes require rerunning at least `normalize`, `trim`, and `analyze` - session recap prompt/profile/input-source changes require rerunning `analyze` -## 12. Roadmap +## 13. Roadmap Planned next steps: diff --git a/examples/pipeline.minimal.yml b/examples/pipeline.minimal.yml index 6a05867..4e5b963 100644 --- a/examples/pipeline.minimal.yml +++ b/examples/pipeline.minimal.yml @@ -81,6 +81,10 @@ scriptorium: # render_debug: true inputs: transcript: + # Available transcript sources: + # - trimmed_transcript (recommended for session_recap) + # - normalized_transcript (recommended for future full-session analysis) + # - processed_transcript (raw Audita-polished output) source: "trimmed_transcript" required: true previous_recap: