Plan post-transcript artifact regeneration

docs/roadmap/post-transcript-artifact-workflow.md

@@ -0,0 +1,460 @@
+# Post-Transcript Artifact Development Workflow
+
+## Status
+
+Accepted target state; not yet implemented.
+
+## Goal
+
+Make repeated development of extraction and analysis artifacts fast, explicit,
+and safe after a session's transcripts are complete. Narratio should expose a
+clear transcript/post-transcript boundary, allow operators to run a bounded
+part of its canonical pipeline, provide one transparent convenience alias for
+the common forced-regeneration workflow, and reuse analysis artifacts whose
+observable inputs have not changed.
+
+The feature must preserve Narratio's intentionally simple orchestration model:
+one fixed stage sequence, explicit stages, manifest-authoritative state, and no
+configurable workflow graph.
+
+## User Intent
+
+Transcript production is comparatively infrequent after a session reaches a
+good final transcript. Development of Notarius extraction and Scriptorium
+artifacts continues much longer and commonly requires repeated execution.
+
+The normal development workflow should therefore:
+
+- treat completed transcript work as read-only unless the operator explicitly
+  selects transcript stages;
+- regenerate extraction and analysis without attempting earlier stages;
+- make forced scope visible in the command itself;
+- support focused Scriptorium artifact selection; and
+- avoid rerunning unrelated analysis artifacts when their meaningful inputs
+  and dependencies are unchanged.
+
+Persistent transcript seals and run-to-run comparison tools are intentionally
+deferred. Bounded execution provides the immediate mutation boundary without
+adding another kind of durable lock.
+
+## Canonical Pipeline And Phase Boundary
+
+Move `render` before `extract` so all transcript representations are complete
+before post-transcript artifact generation begins. The canonical order becomes:
+
+1. `prepare`
+2. `transcribe`
+3. `merge`
+4. `polish`
+5. `normalize`
+6. `trim`
+7. `render`
+8. `extract`
+9. `analyze`
+10. `publish`
+11. `notify`
+
+This creates two useful conceptual regions without making phases configurable:
+
+- transcript production: `prepare` through `render`; and
+- post-transcript processing and delivery: `extract` through `notify`.
+
+`render` and `extract` are independent sibling consumers of completed
+transcript data. Render uses the canonical final and final-trimmed transcripts
+to create Markdown representations. Extract uses the canonical final-trimmed
+transcript and prepared references to create Notarius artifacts. Neither stage
+consumes the other's output.
+
+Narratio's render path is deterministic. A render failure will stop a full run
+before extraction under the new order, and that behavior is accepted: a full
+run should complete transcript representations before starting post-transcript
+work. Recovery remains available through continuation and bounded or
+single-stage execution.
+
+## Execution Order And Invalidation
+
+Execution order and invalidation dependencies must have separate explicit
+owners. The canonical stage registry defines when stages are attempted. A
+central, fixed, configuration-independent invalidation relation defines which
+recorded results may no longer be trustworthy after a stage outcome changes.
+
+The relation is conservative across supported configurations. For example,
+`analyze` remains dependent on `render` because a configured artifact may
+consume rendered Markdown, even if one particular pipeline does not.
+
+The required transitive invalidation sets, returned in canonical execution
+order, are:
+
+| Changed stage | Succeeded stages eligible to become stale |
+| --- | --- |
+| `prepare` | `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
+| `transcribe` | `merge`, `polish`, `normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
+| `merge` | `polish`, `normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
+| `polish` | `normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
+| `normalize` | `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
+| `trim` | `render`, `extract`, `analyze`, `publish`, `notify` |
+| `render` | `analyze`, `publish`, `notify` |
+| `extract` | `analyze`, `publish`, `notify` |
+| `analyze` | `publish`, `notify` |
+| `publish` | `notify` |
+| `notify` | none |
+
+In particular, render and extract must not invalidate one another. A change to
+either still invalidates analysis and delivery, while a change to trim
+invalidates both branches and their consumers.
+
+The relation applies to every existing invalidation trigger, including forced
+replacement, a non-resumable success, failure, and a changed effective outcome.
+Only succeeded dependent stage records become stale under the existing status
+rules. Failed and incomplete records retain their meaning, and an identical
+repeated self-skip does not cause perpetual reruns.
+
+The application owner must validate the fixed relation against the canonical
+stage inventory so a stage addition, removal, rename, duplication, or missing
+classification cannot silently produce incorrect invalidation behavior. This
+relation is not configurable and is not an alternate execution planner.
+
+## Bounded Canonical Execution
+
+Extend `run` with inclusive canonical bounds:
+
+```text
+narratio run <session_id> [--from <stage>] [--through <stage>] [--force]
+```
+
+Examples:
+
+```bash
+narratio run SESSION --from extract --through analyze --force
+narratio run SESSION --from render --through render --force
+narratio run SESSION --from analyze --through analyze
+```
+
+The bounds have these settled semantics:
+
+- they select one contiguous slice of the fixed canonical stage sequence;
+- `--from` defaults to the first stage and `--through` defaults to the last
+  stage when omitted;
+- both stage names must exist, and `--from` must not occur after `--through`;
+- with neither option, `run` retains its current full-pipeline behavior;
+- `--force` applies only to stages inside the selected range;
+- stages before and after the range are not executed or resume-validated;
+- excluded upstream records and artifacts may be resolved and validated as
+  stage inputs, but Narratio must not repair or regenerate them implicitly;
+- missing, stale, unsafe, or otherwise unusable prerequisites produce an
+  actionable error rather than widening the requested range;
+- invalidation caused by an executed stage may mark dependent stages outside
+  the range stale, but those stages are not executed; and
+- stage failure retains the existing stop-on-failure behavior.
+
+`session plan` must accept the same bounds, force scope, and artifact selection
+needed to preview the corresponding `run` without executing stages. Plan and
+run must use one selection implementation so their range validation and
+run/skip decisions cannot drift.
+
+`--artifacts` retains its existing meaning for `analyze` and `publish` when
+those stages are inside the selected range. Supplying artifact selection for a
+range containing neither consumer is an error rather than a silent no-op.
+Repeated artifact-selection flags retain their existing normalization and
+deduplication behavior.
+
+Production composition should follow the bounded plan. Selecting a range must
+not require an adapter used only by an excluded stage, while a selected stage
+continues to require and validate its own collaborators.
+
+## `regenerate-artifacts` Convenience Alias
+
+Add this top-level command:
+
+```text
+narratio regenerate-artifacts <session_id> [--artifacts <name[,name...]>]
+```
+
+It is exactly a convenience alias for:
+
+```text
+narratio run <session_id> --force --from extract --through analyze [--artifacts <name[,name...]>]
+```
+
+The alias has no independent orchestration semantics, prerequisites, force
+rules, or execution path. Its implementation must rewrite or construct the
+equivalent `run` arguments before invoking the shared run parser and handler.
+All common session/configuration arguments and repeatable `--artifacts` values
+pass through to `run` unchanged.
+
+The shared parser owns validation, planning, execution, errors, and summaries.
+Alias help must state the exact equivalence. It is acceptable and desirable for
+runtime errors and summaries to identify the canonical `run` operation. The
+alias must not gain private flags or behavior; a future capability belongs on
+`run` first.
+
+Because `--force`, `--from`, and `--through` define the alias, callers must not
+override them. The shared command parsing layer should reject duplicate
+singleton options rather than use ordering to choose a winner. That rule should
+apply consistently to bounded `run` itself, not only to the alias.
+
+Without `--artifacts`, the alias force-runs extraction and all configured
+analysis artifacts. With `--artifacts`, extraction still produces its complete
+configured Notarius bundle, while forced analysis targets only the selected
+Scriptorium artifacts and any prerequisites required to build them. It never
+runs publish or notify. Changed results may correctly leave those later stages
+stale.
+
+The existing `analyze` command remains the convenience path for forcing
+analysis without rerunning Notarius.
+
+## Incremental Analysis Artifacts
+
+### Analyze-Owned State
+
+`analyze` currently has one aggregate stage result. Extend its manifest-owned
+state so each configured Scriptorium artifact has an explicit current result
+identity. Keep this model specific to analysis artifacts; do not introduce
+dynamic pipeline stages or a generic subtask framework without another proven
+consumer.
+
+Each current artifact result must identify at least:
+
+- the normalized configured artifact key;
+- a versioned input fingerprint;
+- the output source ID, contract, confined canonical path, checksum, and size;
+- the producing Narratio run identity and useful non-secret Scriptorium
+  provenance; and
+- enough status or disposition information to distinguish current, stale,
+  missing, failed, and intentionally unselected work.
+
+The session manifest remains the authority for current availability. An
+incidental output file is not current merely because it exists. Invocation
+manifests continue to record what one run attempted and produced.
+
+### Artifact Fingerprints
+
+Define one deterministic, versioned fingerprint per configured artifact using
+all Narratio-observable inputs that can change its result:
+
+- its normalized Scriptorium artifact configuration;
+- its ordered input names, source IDs, required/optional policy, and resolved
+  input content identities;
+- transcript, prepared-input, previous-session, extraction-lane, and other
+  configured artifact contracts and content checksums;
+- the current content identities of configured artifact dependencies;
+- result-affecting Scriptorium adapter configuration visible to Narratio; and
+- an explicit fingerprint contract version.
+
+Fingerprint ordering must be deterministic. Identity must not change solely
+because a workspace moved, an absolute path changed, or an otherwise identical
+producer used a new run ID. In particular, a forced Notarius invocation that
+produces byte-identical lanes must not make unrelated analysis artifacts stale
+solely because the extraction run identity changed.
+
+Narratio cannot observe arbitrary files, prompts, modules, executable contents,
+or transitive configuration loaded privately by Scriptorium. Documentation must
+state that changing an unobserved external input requires explicit force. Do
+not claim perfect content-addressed reuse beyond Narratio's declared inputs.
+
+### Freshness And Selection
+
+Before skipping a succeeded `analyze` stage, an analyze-specific resume
+validator must reconcile the requested artifact set against current
+configuration, dependencies, input fingerprints, output records, confined
+regular files, and stored output checksums.
+
+The execution rules are:
+
+- an ordinary run executes only requested artifacts that are missing, stale,
+  invalid, or no longer resumable;
+- forcing analyze rebuilds all requested targets even when their fingerprints
+  are current;
+- `--artifacts` identifies explicit targets, not the complete set of records
+  that may remain current;
+- a selected target's configured prerequisites are processed in deterministic
+  dependency order, reusing them when current and rebuilding them when stale;
+- forcing a target does not force an otherwise current prerequisite unless it
+  was also explicitly selected;
+- valid unselected artifact records and outputs survive a partial rerun;
+- artifacts removed or renamed in current configuration cease to be advertised
+  as current;
+- an artifact whose dependency or resolved input changes becomes stale unless
+  the new semantic content identity is unchanged; and
+- stale, missing, failed, or unverified artifacts are unavailable to downstream
+  catalog and publish resolution even if an older file remains on disk.
+
+If a rebuilt artifact changes, configured dependents that were not part of the
+invocation are not silently rebuilt. They become stale and will be rebuilt by a
+later run that selects them. If the rebuilt output is content-identical and the
+dependent fingerprint remains equal, the dependent may remain current.
+
+A partial invocation succeeds when its explicit targets and required
+prerequisites succeed. The aggregate stage record may therefore describe a
+successful partial invocation while other configured artifacts are stale. The
+resume validator, not aggregate status alone, must ensure a later full run does
+not skip unresolved artifact work.
+
+### Replacement And Failure Safety
+
+Artifact replacement must preserve unrelated current results while ensuring a
+failed target is not presented as freshly generated. Run-local output must be
+validated before canonical materialization and manifest promotion, consistent
+with existing stage safety policy.
+
+On partial failure:
+
+- successfully completed and validated targets may be recorded in the
+  invocation history according to existing runner transaction boundaries;
+- the failed target and any result whose current identity depends on it must
+  not be advertised as current;
+- unrelated previously validated artifacts must not be erased merely because
+  they share the `analyze` stage; and
+- publish and later stage state must remain conservatively stale or failed.
+
+The implementation must define one clear manifest transition boundary and must
+not synthesize current output records from directory contents.
+
+### Legacy Analyze Results
+
+Existing manifests may contain only an aggregate analyze success and outputs,
+without versioned per-artifact fingerprints. They remain readable, but Narratio
+must not invent trustworthy fingerprints for work whose exact inputs were not
+recorded.
+
+On first incremental evaluation, legacy analysis artifacts are non-resumable.
+A full analysis selection rebuilds the effective configured set. A partial
+selection may rebuild its targets and prerequisites, but unselected legacy
+outputs remain stale and unavailable until regenerated. Old files and
+invocation manifests may remain for inspection under existing retention rules.
+No wholesale manifest rewrite or version-based transcript invalidation is
+required.
+
+## Resume And Existing Pipeline Manifests
+
+The render/extract order change itself requires no manifest migration because
+stage records use stable names. Under the new sequence:
+
+- succeeded render and extract records remain eligible for ordinary reuse and
+  their stage-specific validation;
+- stale, failed, interrupted, and absent records execute in the new order; and
+- neither result is discarded merely because its relative position changed.
+
+Bounded execution does not rewrite excluded stage records. Compatibility logic
+must remain name- and evidence-based; do not invalidate historical transcript
+work solely because it was produced by an earlier Narratio version.
+
+The incremental-analysis model may add backward-compatible manifest fields or
+versioned metadata. Readers must accept older manifests, while new writers must
+emit one canonical representation and must not maintain parallel legacy and new
+analysis state indefinitely.
+
+## Required Test Coverage
+
+Allocate tests to the behavior owner and avoid duplicating every case at every
+layer:
+
+- planner and command tests prove the new canonical order, inclusive range
+  selection, omitted-bound defaults, invalid/reversed bounds, duplicate-option
+  rejection, artifact-range validation, and unchanged unbounded behavior;
+- alias tests prove exact argument equivalence and pass-through without
+  re-testing the runner through an independent path;
+- invalidation tests prove the complete fixed relation, inventory validation,
+  deterministic ordering, unknown-stage handling, and render/extract
+  independence;
+- runner tests prove force is confined to the selected range, excluded stages
+  are not resume-validated or composed unnecessarily, outside dependents may be
+  staled but not executed, and unusable prerequisites do not widen the range;
+- continuation tests cover representative manifests produced under the old
+  render/extract order;
+- analyze tests prove fingerprint stability and sensitivity, dependency
+  ordering, selected force, stale-prerequisite rebuilding, valid unselected
+  preservation, changed-dependent invalidation, identical-content reuse,
+  removed configuration, output tampering, partial failure, and legacy-result
+  handling; and
+- catalog and publish tests prove that only current per-artifact results are
+  exposed downstream.
+
+Tests must assert observable plans, manifest transitions, adapter requests, and
+artifact availability rather than private map layout. The default suite remains
+offline, deterministic, and independent of installed Seriatim, Notarius, or
+Scriptorium binaries.
+
+## Documentation And Policy Changes
+
+When implementation lands, update all current-behavior owners in the same
+change:
+
+- revise the architecture policy so canonical order owns execution,
+  dependency-aware invalidation owns affected work, and bounded runs remain
+  contiguous selections rather than arbitrary workflows;
+- document the new order, range flags, exact alias expansion, force scope,
+  plan behavior, incremental reuse, and legacy-analysis transition in the CLI
+  and operations references;
+- update the internal overview and the focused render, extract, analyze,
+  manifest, artifact, adapter, and publish documents where their implemented
+  contracts change;
+- update troubleshooting guidance for incomplete bounded prerequisites, stale
+  analysis artifacts, fingerprint limitations, and forced recovery; and
+- remove every current claim that extract runs between trim and render or that
+  all invalidation is a canonical-order suffix.
+
+Documentation should continue to describe one explicit pipeline, not a DAG
+scheduler. Volatile command syntax and manifest details remain in their
+canonical owners rather than being copied across orientation documents.
+
+## Compatibility And Operational Effects
+
+- Existing unbounded `run`, `run-stage`, `analyze`, and `publish` commands keep
+  their current meanings except for the accepted render/extract order change
+  and more precise analysis reuse.
+- `--from` and `--through` are additive CLI options; configuration does not gain
+  stage-order or range fields.
+- `regenerate-artifacts` adds no semantics beyond its documented `run` alias.
+- Bounded forced runs cannot mutate transcript stages outside their range.
+- Full runs attempt deterministic render before invoking Notarius.
+- Forcing render no longer regenerates an otherwise valid Notarius bundle, and
+  forcing extraction no longer regenerates Markdown.
+- Artifact-level validation adds filesystem hashing and fingerprint work before
+  some analyze skips, trading modest local inspection cost for fewer
+  Scriptorium invocations.
+- The first analysis evaluation after upgrade may require regeneration because
+  legacy aggregate results do not contain sufficient freshness evidence.
+
+## Out Of Scope
+
+- Persistent transcript seals, transcript freeze state, or another lock type.
+- Run-history listing, run-to-run artifact comparison, or draft promotion.
+- Parallel execution of render and extract or of analysis artifacts.
+- Non-contiguous stage selection.
+- User-configurable stage order or invalidation dependencies.
+- A generic DAG, phase, job, workflow, or manifest-subtask framework.
+- New Seriatim, Notarius, or Scriptorium CLI capabilities.
+- Changes to Notarius lane contracts or Scriptorium output schemas.
+- Automatic observation of arbitrary transitive Scriptorium files or executable
+  contents.
+- New render retry, caching, resumability, or fingerprint behavior.
+- Automatic publish or notification as part of `regenerate-artifacts`.
+
+## Target End State
+
+Narratio has one comprehensible pipeline in which transcript production ends at
+render and post-transcript generation begins at extract. Operators can run any
+contiguous canonical range without accidentally executing stages outside it,
+and force applies only within the requested range.
+
+The common development command:
+
+```bash
+narratio regenerate-artifacts SESSION
+```
+
+is transparently identical to a forced bounded run from extract through
+analyze. It preserves transcript state, regenerates the complete Notarius
+bundle, rebuilds the requested Scriptorium targets, and leaves delivery as a
+separate explicit action.
+
+Analysis artifacts have independent, manifest-authoritative freshness within
+the fixed `analyze` stage. Narratio reuses valid unselected work, rebuilds stale
+dependencies and selected targets deterministically, withholds stale outputs
+from downstream consumers, and recognizes content-identical upstream results
+without tying reuse to ephemeral run paths or IDs.
+
+Together, canonical ordering, dependency-aware invalidation, bounded execution,
+the transparent alias, and artifact-level analysis reuse provide an ergonomic
+development loop without turning Narratio into a general workflow engine.