Document transcript trim workflow
This commit is contained in:
67
README.md
67
README.md
@@ -9,15 +9,13 @@ Implemented now:
|
||||
- strict config loading/validation (`pipeline.yml` and `session.yml`)
|
||||
- local workspace/session layout, locking, and manifest persistence
|
||||
- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`)
|
||||
- real `prepare`, `transcribe`, `merge`, and `polish` stages
|
||||
- real `prepare`, `transcribe`, `merge`, `polish`, `trim`, and `analyze` stages
|
||||
- real WhisperX, Seriatim, and Audita adapters
|
||||
- real Scriptorium subprocess adapter
|
||||
- real `analyze` stage for `session_recap`
|
||||
- optional Scriptorium render diagnostics (`render_debug`)
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- real `trim` behavior
|
||||
- real `archive` behavior
|
||||
- real `notify` behavior
|
||||
- additional analyze artifacts beyond `session_recap`
|
||||
@@ -32,7 +30,24 @@ Narratio expects two YAML files:
|
||||
|
||||
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||||
|
||||
## Trim Configuration (Scaffold)
|
||||
## Canonical Stage Order
|
||||
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `trim`
|
||||
6. `analyze`
|
||||
7. `archive`
|
||||
8. `notify`
|
||||
|
||||
## Transcript Tiers
|
||||
|
||||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||||
- `transcripts/processed.json`: full polished transcript from Audita polish
|
||||
- `transcripts/trimmed.json`: gameplay-only polished transcript from trim stage
|
||||
|
||||
## Trim Configuration
|
||||
|
||||
`pipeline.trim` is optional. If omitted, no trim config is loaded. If `trim.enabled` is omitted, it defaults to `false`.
|
||||
|
||||
@@ -48,7 +63,7 @@ When `trim.enabled: true`:
|
||||
|
||||
Trim paths are treated as session-workdir-relative when not absolute.
|
||||
|
||||
Example trim scaffold config:
|
||||
Example trim config:
|
||||
|
||||
```yaml
|
||||
trim:
|
||||
@@ -66,6 +81,30 @@ trim:
|
||||
report: false
|
||||
```
|
||||
|
||||
Trim behavior summary:
|
||||
|
||||
- trim discovers and validates `transcripts/processed.json`
|
||||
- trim uses Scriptorium bounds (`dnd_session.bounds` by example config) to produce `artifacts/session_bounds.json`
|
||||
- bounds IDs are validated against the same processed transcript ID space that Seriatim trim will consume
|
||||
- trim converts bounds to Seriatim keep selector (for example `10-868`) and runs Seriatim trim
|
||||
- if trim is disabled, Narratio copies processed transcript to trimmed transcript and records `trim_action=copy_disabled`
|
||||
|
||||
Trim outputs and diagnostics:
|
||||
|
||||
- `artifacts/session_bounds.json`
|
||||
- `transcripts/trimmed.json`
|
||||
- `logs/scriptorium.bounds.stdout.log`
|
||||
- `logs/scriptorium.bounds.stderr.log`
|
||||
- `config/scriptorium.bounds.generated.yml`
|
||||
- `logs/seriatim.trim.stdout.log`
|
||||
- `logs/seriatim.trim.stderr.log`
|
||||
- `config/seriatim.trim.generated.yml`
|
||||
- optional bounds render-debug outputs:
|
||||
- `artifacts/session_bounds.render.json`
|
||||
- `logs/scriptorium.bounds.render.stdout.log`
|
||||
- `logs/scriptorium.bounds.render.stderr.log`
|
||||
- `config/scriptorium.bounds.render.generated.yml`
|
||||
|
||||
## Scriptorium Configuration
|
||||
|
||||
`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.
|
||||
@@ -133,8 +172,10 @@ For the initial implementation, only `session_recap` generation is supported.
|
||||
|
||||
Analyze-stage session recap behavior:
|
||||
|
||||
- defaults to trimmed transcript input (`transcripts/trimmed.json`) when configured with `source: trimmed_transcript`
|
||||
- still supports full polished transcript input (`transcripts/processed.json`) when configured with `source: processed_transcript`
|
||||
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
|
||||
- Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json`
|
||||
- missing trimmed transcript fails clearly and advises running trim stage first
|
||||
- full polished transcript input (`source: processed_transcript`) remains supported for future table/meta-analysis artifacts
|
||||
- optionally includes `previous_recap` when configured and resolvable
|
||||
- omits optional previous recap when unavailable
|
||||
- fails if required inputs are missing
|
||||
@@ -182,6 +223,18 @@ Run analyze only:
|
||||
go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze
|
||||
```
|
||||
|
||||
## Operational Note
|
||||
|
||||
Checksum-based stale detection is not implemented yet.
|
||||
|
||||
If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- glossary/autocorrect/speaker-context changes: rerun at least `merge`, `polish`, `trim`, and `analyze`
|
||||
- trim bounds prompt/profile/config changes: rerun at least `trim` and `analyze`
|
||||
- session recap prompt/profile/input-source changes: rerun `analyze`
|
||||
|
||||
## Roadmap
|
||||
|
||||
Near-term roadmap:
|
||||
|
||||
@@ -135,7 +135,7 @@ For each artifact definition:
|
||||
|
||||
Prompt IDs and profile IDs are configuration values, not hardcoded stage logic.
|
||||
|
||||
Trim scaffold config shape:
|
||||
Trim config shape:
|
||||
|
||||
```yaml
|
||||
trim:
|
||||
@@ -153,7 +153,50 @@ trim:
|
||||
report: false
|
||||
```
|
||||
|
||||
## 6. Analyze Stage (Current Implementation)
|
||||
## 6. Transcript Tiers
|
||||
|
||||
Narratio currently produces and uses three transcript tiers:
|
||||
|
||||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||||
- `transcripts/processed.json`: full polished transcript from Audita polish (includes pre/post-game content)
|
||||
- `transcripts/trimmed.json`: gameplay-only polished transcript from trim stage
|
||||
|
||||
Trim reads `transcripts/processed.json`, validates bounds IDs against that same transcript ID space, and writes `transcripts/trimmed.json`.
|
||||
|
||||
## 7. Trim Stage (Current Implementation)
|
||||
|
||||
Trim stage behavior:
|
||||
|
||||
- stage order position: after `polish` and before `analyze`
|
||||
- discovers processed transcript from manifest polish outputs (`transcript_processed`) when present, else `work/<session_id>/transcripts/processed.json`
|
||||
- validates processed transcript JSON shape (`segments` array required)
|
||||
- when `trim.enabled: false` (or trim config omitted), deterministically copies processed transcript to `transcripts/trimmed.json` and records `trim_action=copy_disabled`
|
||||
- when `trim.enabled: true`:
|
||||
- runs Scriptorium bounds prompt using configured `trim.bounds.prompt_id`
|
||||
- writes bounds output to configured path (typically `artifacts/session_bounds.json`)
|
||||
- parses and validates bounds output against the same processed transcript being trimmed
|
||||
- converts bounds range to Seriatim keep selector (for example `10-868`)
|
||||
- runs Seriatim `trim` to produce `transcripts/trimmed.json`
|
||||
- supports no-trim bounds actions (`none`/`copy`) by copying processed transcript unchanged
|
||||
- validates trimmed transcript JSON shape (`segments` array required)
|
||||
|
||||
Expected trim outputs and diagnostics:
|
||||
|
||||
- `artifacts/session_bounds.json`
|
||||
- `transcripts/trimmed.json`
|
||||
- `logs/scriptorium.bounds.stdout.log`
|
||||
- `logs/scriptorium.bounds.stderr.log`
|
||||
- `config/scriptorium.bounds.generated.yml`
|
||||
- `logs/seriatim.trim.stdout.log`
|
||||
- `logs/seriatim.trim.stderr.log`
|
||||
- `config/seriatim.trim.generated.yml`
|
||||
- optional bounds render-debug outputs when enabled:
|
||||
- `artifacts/session_bounds.render.json`
|
||||
- `logs/scriptorium.bounds.render.stdout.log`
|
||||
- `logs/scriptorium.bounds.render.stderr.log`
|
||||
- `config/scriptorium.bounds.render.generated.yml`
|
||||
|
||||
## 8. Analyze Stage (Current Implementation)
|
||||
|
||||
The current real analyze implementation supports only `scriptorium.artifacts.session_recap`.
|
||||
|
||||
@@ -162,8 +205,10 @@ Behavior:
|
||||
- if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata
|
||||
- if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata
|
||||
- if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly
|
||||
- `session_recap` should use `trimmed_transcript` input (`transcripts/trimmed.json`) for in-universe recap generation
|
||||
- `trimmed_transcript` input is resolved from manifest (`trim` output kind `transcript_trimmed`) when available, otherwise fallback path `work/<session_id>/transcripts/trimmed.json`
|
||||
- `processed_transcript` input is resolved from manifest (`polish` output kind `transcript_processed`) when available, otherwise fallback path `work/<session_id>/transcripts/processed.json`
|
||||
- `processed_transcript` remains available for future table/meta-analysis artifacts
|
||||
- transcript inputs are validated as JSON with top-level `segments` array
|
||||
- configured inputs are resolved by source
|
||||
- optional `previous_recap` is omitted when unavailable
|
||||
@@ -174,9 +219,10 @@ Behavior:
|
||||
- render output is validated as JSON
|
||||
- production call uses Scriptorium adapter `RunArtifact`
|
||||
- successful run output must exist and be non-empty
|
||||
- missing `trimmed_transcript` input for configured `trimmed_transcript` source fails clearly with guidance to run trim stage first
|
||||
- manifest records output refs, logs, generated config paths, and non-secret provenance metadata
|
||||
|
||||
## 7. Session Recap Paths
|
||||
## 9. Session Recap Paths
|
||||
|
||||
Current expected paths for `session_recap`:
|
||||
|
||||
@@ -189,14 +235,26 @@ Current expected paths for `session_recap`:
|
||||
- render stderr log: `logs/scriptorium.session_recap.render.stderr.log`
|
||||
- render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml`
|
||||
|
||||
## 8. Security and Privacy
|
||||
## 10. Security and Privacy
|
||||
|
||||
- do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata
|
||||
- if API-key integration is configured, pass env var names only (never raw key values)
|
||||
- avoid logging transcript content or rendered prompt content by default
|
||||
- treat generated artifacts and logs as potentially sensitive session material
|
||||
|
||||
## 9. Roadmap
|
||||
## 11. Operational Caveat (Pre-Stale-Detection)
|
||||
|
||||
Checksum-based stale detection is not implemented yet.
|
||||
|
||||
If prepared inputs or prompt/runtime configuration change (for example glossary files, prompt IDs, profile IDs, or relevant pipeline settings), rerun the appropriate prior stages to refresh downstream artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- glossary or autocorrect changes usually require rerunning at least `merge`, `polish`, `trim`, and `analyze`
|
||||
- trim prompt/profile changes require rerunning at least `trim` and `analyze`
|
||||
- session recap prompt/profile/input-source changes require rerunning `analyze`
|
||||
|
||||
## 12. Roadmap
|
||||
|
||||
Planned next steps:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user