narratio/docs/roadmap/post-transcript-artifact-workflow.md

# Post-Transcript Artifact Development Workflow

## Status

Accepted target state; not yet implemented.

## Goal

Make repeated development of extraction and analysis artifacts fast, explicit,
and safe after a session's transcripts are complete. Narratio should expose a
clear transcript/post-transcript boundary, allow operators to run a bounded
part of its canonical pipeline, provide one transparent convenience alias for
the common forced-regeneration workflow, and reuse analysis artifacts whose
observable inputs have not changed.

The feature must preserve Narratio's intentionally simple orchestration model:
one fixed stage sequence, explicit stages, manifest-authoritative state, and no
configurable workflow graph.

## User Intent

Transcript production is comparatively infrequent after a session reaches a
good final transcript. Development of Notarius extraction and Scriptorium
artifacts continues much longer and commonly requires repeated execution.

The normal development workflow should therefore:

- treat completed transcript work as read-only unless the operator explicitly
  selects transcript stages;
- regenerate extraction and analysis without attempting earlier stages;
- make forced scope visible in the command itself;
- support focused Scriptorium artifact selection; and
- avoid rerunning unrelated analysis artifacts when their meaningful inputs
  and dependencies are unchanged.

Persistent transcript seals and run-to-run comparison tools are intentionally
deferred. Bounded execution provides the immediate mutation boundary without
adding another kind of durable lock.

## Canonical Pipeline And Phase Boundary

Move `render` before `extract` so all transcript representations are complete
before post-transcript artifact generation begins. The canonical order becomes:

1. `prepare`
2. `transcribe`
3. `merge`
4. `polish`
5. `normalize`
6. `trim`
7. `render`
8. `extract`
9. `analyze`
10. `publish`
11. `notify`

This creates two useful conceptual regions without making phases configurable:

- transcript production: `prepare` through `render`; and
- post-transcript processing and delivery: `extract` through `notify`.

`render` and `extract` are independent sibling consumers of completed
transcript data. Render uses the canonical final and final-trimmed transcripts
to create Markdown representations. Extract uses the canonical final-trimmed
transcript and prepared references to create Notarius artifacts. Neither stage
consumes the other's output.

Narratio's render path is deterministic. A render failure will stop a full run
before extraction under the new order, and that behavior is accepted: a full
run should complete transcript representations before starting post-transcript
work. Recovery remains available through continuation and bounded or
single-stage execution.

## Execution Order And Invalidation

Execution order and invalidation dependencies must have separate explicit
owners. The canonical stage registry defines when stages are attempted. A
central, fixed, configuration-independent invalidation relation defines which
recorded results may no longer be trustworthy after a stage outcome changes.

The relation is conservative across supported configurations. For example,
`analyze` remains dependent on `render` because a configured artifact may
consume rendered Markdown, even if one particular pipeline does not.

The required transitive invalidation sets, returned in canonical execution
order, are:

| Changed stage | Succeeded stages eligible to become stale |
| --- | --- |
| `prepare` | `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
| `transcribe` | `merge`, `polish`, `normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
| `merge` | `polish`, `normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
| `polish` | `normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
| `normalize` | `trim`, `render`, `extract`, `analyze`, `publish`, `notify` |
| `trim` | `render`, `extract`, `analyze`, `publish`, `notify` |
| `render` | `analyze`, `publish`, `notify` |
| `extract` | `analyze`, `publish`, `notify` |
| `analyze` | `publish`, `notify` |
| `publish` | `notify` |
| `notify` | none |

In particular, render and extract must not invalidate one another. A change to
either still invalidates analysis and delivery, while a change to trim
invalidates both branches and their consumers.

The relation applies to every existing invalidation trigger, including forced
replacement, a non-resumable success, failure, and a changed effective outcome.
Only succeeded dependent stage records become stale under the existing status
rules. Failed and incomplete records retain their meaning, and an identical
repeated self-skip does not cause perpetual reruns.

The application owner must validate the fixed relation against the canonical
stage inventory so a stage addition, removal, rename, duplication, or missing
classification cannot silently produce incorrect invalidation behavior. This
relation is not configurable and is not an alternate execution planner.

## Bounded Canonical Execution

Extend `run` with inclusive canonical bounds:

```text
narratio run <session_id> [--from <stage>] [--through <stage>] [--force]
```

Examples:

```bash
narratio run SESSION --from extract --through analyze --force
narratio run SESSION --from render --through render --force
narratio run SESSION --from analyze --through analyze
```

The bounds have these settled semantics:

- they select one contiguous slice of the fixed canonical stage sequence;
- `--from` defaults to the first stage and `--through` defaults to the last
  stage when omitted;
- both stage names must exist, and `--from` must not occur after `--through`;
- with neither option, `run` retains its current full-pipeline behavior;
- `--force` applies only to stages inside the selected range;
- stages before and after the range are not executed or resume-validated;
- excluded upstream records and artifacts may be resolved and validated as
  stage inputs, but Narratio must not repair or regenerate them implicitly;
- missing, stale, unsafe, or otherwise unusable prerequisites produce an
  actionable error rather than widening the requested range;
- invalidation caused by an executed stage may mark dependent stages outside
  the range stale, but those stages are not executed; and
- stage failure retains the existing stop-on-failure behavior.

`session plan` must accept the same bounds, force scope, and artifact selection
needed to preview the corresponding `run` without executing stages. Plan and
run must use one selection implementation so their range validation and
run/skip decisions cannot drift.

`--artifacts` retains its existing meaning for `analyze` and `publish` when
those stages are inside the selected range. Supplying artifact selection for a
range containing neither consumer is an error rather than a silent no-op.
Repeated artifact-selection flags retain their existing normalization and
deduplication behavior.

Production composition should follow the bounded plan. Selecting a range must
not require an adapter used only by an excluded stage, while a selected stage
continues to require and validate its own collaborators.

## `regenerate-artifacts` Convenience Alias

Add this top-level command:

```text
narratio regenerate-artifacts <session_id> [--artifacts <name[,name...]>]
```

It is exactly a convenience alias for:

```text
narratio run <session_id> --force --from extract --through analyze [--artifacts <name[,name...]>]
```

The alias has no independent orchestration semantics, prerequisites, force
rules, or execution path. Its implementation must rewrite or construct the
equivalent `run` arguments before invoking the shared run parser and handler.
All common session/configuration arguments and repeatable `--artifacts` values
pass through to `run` unchanged.

The shared parser owns validation, planning, execution, errors, and summaries.
Alias help must state the exact equivalence. It is acceptable and desirable for
runtime errors and summaries to identify the canonical `run` operation. The
alias must not gain private flags or behavior; a future capability belongs on
`run` first.

Because `--force`, `--from`, and `--through` define the alias, callers must not
override them. The shared command parsing layer should reject duplicate
singleton options rather than use ordering to choose a winner. That rule should
apply consistently to bounded `run` itself, not only to the alias.

Without `--artifacts`, the alias force-runs extraction and all enabled
configured analysis artifacts, matching the existing default analysis
selection. With `--artifacts`, extraction still produces its complete
configured Notarius bundle, while forced analysis targets only the selected
Scriptorium artifacts and any prerequisites required to build them. An
explicitly selected disabled artifact remains a valid target under the existing
selection rules. The alias never runs publish or notify. Changed results may
correctly leave those later stages stale.

The existing `analyze` command remains the convenience path for forcing
analysis without rerunning Notarius.

## Incremental Analysis Artifacts

### Analyze-Owned State

`analyze` currently has one aggregate stage result. Extend its manifest-owned
state so each configured Scriptorium artifact has an explicit current result
identity. Keep this model specific to analysis artifacts; do not introduce
dynamic pipeline stages or a generic subtask framework without another proven
consumer.

Each current artifact result must identify at least:

- the normalized configured artifact key;
- a versioned input fingerprint;
- the output source ID, contract, confined canonical path, checksum, and size;
- the producing Narratio run identity and useful non-secret Scriptorium
  provenance; and
- enough status or disposition information to distinguish current, stale,
  missing, failed, and intentionally unselected work.

The session manifest remains the authority for current availability. An
incidental output file is not current merely because it exists. Invocation
manifests continue to record what one run attempted and produced.

### Artifact Fingerprints

Define one deterministic, versioned fingerprint per configured artifact using
all Narratio-observable inputs that can change its result:

- its normalized Scriptorium artifact configuration;
- its ordered input names, source IDs, required/optional policy, and resolved
  input content identities;
- transcript, prepared-input, previous-session, extraction-lane, and other
  configured artifact contracts and content checksums;
- the current content identities of configured artifact dependencies;
- result-affecting Scriptorium adapter configuration visible to Narratio; and
- an explicit fingerprint contract version.

Fingerprint ordering must be deterministic. Identity must not change solely
because a workspace moved, an absolute path changed, or an otherwise identical
producer used a new run ID. In particular, a forced Notarius invocation that
produces byte-identical lanes must not make unrelated analysis artifacts stale
solely because the extraction run identity changed.

Narratio cannot observe arbitrary files, prompts, modules, executable contents,
or transitive configuration loaded privately by Scriptorium. Documentation must
state that changing an unobserved external input requires explicit force. Do
not claim perfect content-addressed reuse beyond Narratio's declared inputs.

### Freshness And Selection

Before skipping a succeeded `analyze` stage, an analyze-specific resume
validator must reconcile the requested artifact set against current
configuration, dependencies, input fingerprints, output records, confined
regular files, and stored output checksums.

The execution rules are:

- an ordinary run executes only requested artifacts that are missing, stale,
  invalid, or no longer resumable;
- forcing analyze rebuilds all requested targets even when their fingerprints
  are current;
- `--artifacts` identifies explicit targets, not the complete set of records
  that may remain current;
- a selected target's configured prerequisites are processed in deterministic
  dependency order, reusing them when current and rebuilding them when stale;
- forcing a target does not force an otherwise current prerequisite unless it
  was also explicitly selected;
- valid unselected artifact records and outputs survive a partial rerun;
- artifacts removed or renamed in current configuration cease to be advertised
  as current;
- an artifact whose dependency or resolved input changes becomes stale unless
  the new semantic content identity is unchanged; and
- stale, missing, failed, or unverified artifacts are unavailable to downstream
  catalog and publish resolution even if an older file remains on disk.

If a rebuilt artifact changes, configured dependents that were not part of the
invocation are not silently rebuilt. They become stale and will be rebuilt by a
later run that selects them. If the rebuilt output is content-identical and the
dependent fingerprint remains equal, the dependent may remain current.

A partial invocation succeeds when its explicit targets and required
prerequisites succeed. The aggregate stage record may therefore describe a
successful partial invocation while other configured artifacts are stale. The
resume validator, not aggregate status alone, must ensure a later full run does
not skip unresolved artifact work.

### Replacement And Failure Safety

Artifact replacement must preserve unrelated current results while ensuring a
failed target is not presented as freshly generated. Run-local output must be
validated before canonical materialization and manifest promotion, consistent
with existing stage safety policy.

On partial failure:

- successfully completed and validated targets may be recorded in the
  invocation history according to existing runner transaction boundaries;
- the failed target and any result whose current identity depends on it must
  not be advertised as current;
- unrelated previously validated artifacts must not be erased merely because
  they share the `analyze` stage; and
- publish and later stage state must remain conservatively stale or failed.

The implementation must define one clear manifest transition boundary and must
not synthesize current output records from directory contents.

### Legacy Analyze Results

Existing manifests may contain only an aggregate analyze success and outputs,
without versioned per-artifact fingerprints. They remain readable, but Narratio
must not invent trustworthy fingerprints for work whose exact inputs were not
recorded.

On first incremental evaluation, legacy analysis artifacts are non-resumable.
A full analysis selection rebuilds the effective configured set. A partial
selection may rebuild its targets and prerequisites, but unselected legacy
outputs remain stale and unavailable until regenerated. Old files and
invocation manifests may remain for inspection under existing retention rules.
No wholesale manifest rewrite or version-based transcript invalidation is
required.

## Resume And Existing Pipeline Manifests

The render/extract order change itself requires no manifest migration because
stage records use stable names. Under the new sequence:

- succeeded render and extract records remain eligible for ordinary reuse and
  their stage-specific validation;
- stale, failed, interrupted, and absent records execute in the new order; and
- neither result is discarded merely because its relative position changed.

Bounded execution does not rewrite excluded stage records. Compatibility logic
must remain name- and evidence-based; do not invalidate historical transcript
work solely because it was produced by an earlier Narratio version.

The incremental-analysis model may add backward-compatible manifest fields or
versioned metadata. Readers must accept older manifests, while new writers must
emit one canonical representation and must not maintain parallel legacy and new
analysis state indefinitely.

## Compatibility And Operational Effects

- Existing unbounded `run`, `run-stage`, `analyze`, and `publish` commands keep
  their current meanings except for the accepted render/extract order change
  and more precise analysis reuse.
- `--from` and `--through` are additive CLI options; configuration does not gain
  stage-order or range fields.
- `regenerate-artifacts` adds no semantics beyond its documented `run` alias.
- Bounded forced runs cannot mutate transcript stages outside their range.
- Full runs attempt deterministic render before invoking Notarius.
- Forcing render no longer regenerates an otherwise valid Notarius bundle, and
  forcing extraction no longer regenerates Markdown.
- Artifact-level validation adds filesystem hashing and fingerprint work before
  some analyze skips, trading modest local inspection cost for fewer
  Scriptorium invocations.
- The first analysis evaluation after upgrade may require regeneration because
  legacy aggregate results do not contain sufficient freshness evidence.

## Out Of Scope

- Persistent transcript seals, transcript freeze state, or another lock type.
- Run-history listing, run-to-run artifact comparison, or draft promotion.
- Parallel execution of render and extract or of analysis artifacts.
- Non-contiguous stage selection.
- User-configurable stage order or invalidation dependencies.
- A generic DAG, phase, job, workflow, or manifest-subtask framework.
- New Seriatim, Notarius, or Scriptorium CLI capabilities.
- Changes to Notarius lane contracts or Scriptorium output schemas.
- Automatic observation of arbitrary transitive Scriptorium files or executable
  contents.
- New render retry, caching, resumability, or fingerprint behavior.
- Automatic publish or notification as part of `regenerate-artifacts`.

## Target End State

Narratio has one comprehensible pipeline in which transcript production ends at
render and post-transcript generation begins at extract. Operators can run any
contiguous canonical range without accidentally executing stages outside it,
and force applies only within the requested range.

The common development command:

```bash
narratio regenerate-artifacts SESSION
```

is transparently identical to a forced bounded run from extract through
analyze. It preserves transcript state, regenerates the complete Notarius
bundle, rebuilds the requested Scriptorium targets, and leaves delivery as a
separate explicit action.

Analysis artifacts have independent, manifest-authoritative freshness within
the fixed `analyze` stage. Narratio reuses valid unselected work, rebuilds stale
dependencies and selected targets deterministically, withholds stale outputs
from downstream consumers, and recognizes content-identical upstream results
without tying reuse to ephemeral run paths or IDs.

Together, canonical ordering, dependency-aware invalidation, bounded execution,
the transparent alias, and artifact-level analysis reuse provide an ergonomic
development loop without turning Narratio into a general workflow engine.