790 lines
40 KiB
Markdown
790 lines
40 KiB
Markdown
# Post-Transcript Artifact Workflow Implementation Plan
|
||
|
||
## Purpose And Status
|
||
|
||
This document is the executable implementation plan for
|
||
[`post-transcript-artifact-workflow.md`](post-transcript-artifact-workflow.md).
|
||
That roadmap owns the accepted user intent, compatibility policy, and target
|
||
state. This plan translates it into bounded stages suitable for one
|
||
`gpt-5.6-terra` implementation prompt apiece.
|
||
|
||
All stages below are pending and must be implemented in numeric order. A later
|
||
stage may rely on every earlier stage having been completed, tested, documented
|
||
where directed, and committed. Do not use this plan to redesign the accepted
|
||
feature. If existing code makes a specified name awkward, preserve the stated
|
||
behavior and ownership boundaries rather than introducing a second workflow
|
||
model.
|
||
|
||
## Settled Implementation Decisions
|
||
|
||
The following choices make the plan decision-complete:
|
||
|
||
- The canonical order is `prepare`, `transcribe`, `merge`, `polish`,
|
||
`normalize`, `trim`, `render`, `extract`, `analyze`, `publish`, `notify`.
|
||
- Execution order remains a flat, fixed registry. Invalidation uses a separate
|
||
fixed direct-dependency table whose transitive closure is returned in
|
||
canonical order. It is not configurable and is not an execution DAG.
|
||
- `run` and `session plan` share one inclusive contiguous-range selector.
|
||
Omitted endpoints select the beginning or end of the canonical registry.
|
||
- A bounded `run` requires every excluded prefix stage to have a terminal
|
||
manifest status of `succeeded` or `skipped`. The first absent, pending,
|
||
running, failed, stale, or interrupted prerequisite is an actionable error.
|
||
Excluded prefix stages are not resume-validated or repaired; selected stages
|
||
still validate the concrete inputs they consume. Stages after `--through`
|
||
are not prerequisites and receive no analogous check.
|
||
- `--force` affects only selected pipeline stages. Invalidation may mark a
|
||
dependent outside the range stale, but cannot execute it.
|
||
- `--artifacts` is valid only when the selected range contains `analyze` or
|
||
`publish`. Its existing repeatable, comma-separated normalization and
|
||
deduplication semantics remain unchanged.
|
||
- `regenerate-artifacts` performs argument expansion and calls the canonical
|
||
`run` handler. It owns no planner, runner, prerequisite, or force behavior.
|
||
Duplicate `--force`, `--from`, or `--through` options are rejected by the
|
||
shared bounded-run parser, including `--name=value` spellings.
|
||
- Incremental state is specific to configured Scriptorium artifacts inside the
|
||
fixed `analyze` stage. Do not create dynamic stages, generic jobs, or a
|
||
general subtask framework.
|
||
- The session manifest owns current artifact availability. A configured output
|
||
file found on disk without current manifest evidence is unavailable.
|
||
- Analysis fingerprints contain only Narratio-observable semantic inputs. They
|
||
exclude run IDs, absolute workspace paths, timeouts, and executable contents.
|
||
They also cannot observe arbitrary files or transitive configuration loaded
|
||
privately by Scriptorium; those changes require explicit force.
|
||
- Partial analysis preserves unrelated current records. On an error, the
|
||
runner accepts a restricted analyze-state result alongside the error so it
|
||
can record completed, failed, and newly stale artifact state before marking
|
||
the aggregate stage failed. It must never promote unvalidated files.
|
||
- Legacy aggregate-only analysis output remains readable but is not current
|
||
evidence. Only regenerated artifacts enter the new per-artifact state.
|
||
|
||
## Instructions For Every Stage
|
||
|
||
Before changing code in each stage:
|
||
|
||
1. Read `docs/development.md`, its task-specific references, all documents in
|
||
`docs/policy/`, this plan, and the relevant portions of the feature roadmap.
|
||
2. Inspect the current implementation before editing. Prefer the codebase
|
||
knowledge graph for code discovery, then read the exact owning files.
|
||
3. Confirm the worktree state and preserve unrelated user changes.
|
||
|
||
During each stage:
|
||
|
||
- Keep changes within that stage's scope and the accepted roadmap.
|
||
- Follow the architecture policy's dependency direction, filesystem safety,
|
||
manifest authority, and application-owned lifecycle rules.
|
||
- Follow the testing policy: place behavior at its narrowest public owner, use
|
||
fakes at owned boundaries, keep the default suite offline and deterministic,
|
||
and avoid duplicating the same assertion across layers.
|
||
- Update a canonical current-behavior document in the same stage when that
|
||
stage changes already usable behavior. Do not describe unimplemented later
|
||
stages as current.
|
||
- Preserve backward-compatible manifest reads and emit only the new canonical
|
||
representation after its writer is introduced.
|
||
- Run focused tests during development, then `go test ./...` before completing
|
||
the stage. Run the repository's required formatting, static checks, and
|
||
documentation checks identified by `docs/development.md`.
|
||
|
||
At the end of each stage, leave a cohesive change that can be reviewed and
|
||
committed independently. Do not begin a later stage while an earlier stage has
|
||
failing tests or incomplete acceptance criteria.
|
||
|
||
## Stage 1 — Canonical Order And Dependency-Aware Invalidation
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Move `render` before `extract` and replace suffix-based invalidation with the
|
||
fixed dependency relation required by the feature roadmap.
|
||
|
||
### Required Work
|
||
|
||
1. Change the single canonical stage registry so `render` immediately follows
|
||
`trim` and `extract` immediately follows `render`. Do not add a second stage
|
||
list in a command or test helper.
|
||
2. In the application orchestration owner, define a fixed table of direct
|
||
invalidation edges:
|
||
- `prepare -> transcribe`
|
||
- `transcribe -> merge`
|
||
- `merge -> polish`
|
||
- `polish -> normalize`
|
||
- `normalize -> trim`
|
||
- `trim -> render, extract`
|
||
- `render -> analyze`
|
||
- `extract -> analyze`
|
||
- `analyze -> publish`
|
||
- `publish -> notify`
|
||
- `notify ->` none
|
||
3. Compute transitive dependents from that table and return them in canonical
|
||
order. Unknown stages must return an error; callers must not silently treat
|
||
them as having no dependents.
|
||
4. Validate the relation against the canonical registry when it is constructed
|
||
or first used. Reject duplicate registry names, unknown edge endpoints,
|
||
cycles, missing stage classifications, and registry/table drift.
|
||
5. Route every existing invalidation trigger through this owner: force,
|
||
non-resumable success, failure, and changed effective outcome. Retain the
|
||
existing rule that only succeeded dependent records become stale and retain
|
||
the existing no-op treatment for an identical repeated self-skip.
|
||
6. Remove assumptions that invalidation is the canonical suffix. In
|
||
particular, `render` and `extract` must never invalidate one another, while
|
||
either invalidates `analyze`, `publish`, and `notify`.
|
||
7. Update the architecture policy, internal overview, and focused render,
|
||
extract, and manifest documentation for the implemented order and separate
|
||
invalidation owner.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Registry/planner tests assert the complete new order.
|
||
- Table-driven invalidation tests assert every transitive set in the feature
|
||
roadmap, canonical output ordering, render/extract independence, unknown
|
||
stages, cycle rejection, and inventory drift rejection.
|
||
- Representative runner tests prove each invalidation trigger uses the new
|
||
relation and does not execute newly stale stages implicitly.
|
||
- Existing manifests with stable stage names remain readable and resumable;
|
||
no version-based invalidation or manifest migration is introduced.
|
||
|
||
## Stage 2 — Shared Bounded Plan Selection
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Give the application one validated representation of a contiguous canonical
|
||
stage range, shared by execution and plan preview.
|
||
|
||
### Required Work
|
||
|
||
1. Extend the application planner with a bounded-plan constructor accepting
|
||
optional `from` and `through` names. The returned plan contains an inclusive
|
||
contiguous slice of the canonical registry.
|
||
2. Default an omitted `from` to the first stage and an omitted `through` to the
|
||
last stage. Preserve the current full plan when both are omitted.
|
||
3. Reject unknown endpoints and a `from` endpoint occurring after `through`.
|
||
Errors must name the bad value or reversed pair and list or point to valid
|
||
canonical stage names.
|
||
4. Put range membership and endpoint information on one application-level
|
||
value so `run`, `session plan`, prerequisite validation, and adapter
|
||
composition do not independently recalculate bounds.
|
||
5. Keep `run-stage` and the dedicated `analyze` and `publish` conveniences on
|
||
their existing single-stage paths. Do not reinterpret them through new
|
||
range flags.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Planner tests cover every endpoint, one-stage ranges, omitted starts, omitted
|
||
ends, a full omitted range, unknown names, reversed ranges, and stable order.
|
||
- Tests assert that selecting a range cannot create a non-contiguous plan or
|
||
mutate the canonical registry.
|
||
- No CLI behavior changes in this stage; the new planner API is ready for both
|
||
`run` and `session plan` to consume next.
|
||
|
||
## Stage 3 — Bounded `run` And `session plan` Command Contracts
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Expose the shared range through both commands with one parsing and structural
|
||
validation contract.
|
||
|
||
### Required Work
|
||
|
||
1. Add repeat-safe singleton option parsing for `--from`, `--through`, and
|
||
`--force` to the shared bounded-run command layer. Reject duplicates whether
|
||
written as separate arguments or `--name=value`; never use argument order to
|
||
choose a winner.
|
||
2. Preserve the existing session-oriented argument conventions and common
|
||
configuration flags. Preserve repeatable/comma-separated `--artifacts`
|
||
normalization and deduplication.
|
||
3. Add `--from` and `--through` to `run`, and pass the resulting bounded plan to
|
||
the existing runner rather than filtering stages after planning.
|
||
4. Add the same bounds, force flag, and artifact selection to `session plan`.
|
||
Both commands must call the Stage 2 selector and surface identical range
|
||
validation errors.
|
||
5. Reject `--artifacts` when the range contains neither `analyze` nor `publish`.
|
||
Accept it when either consumer is present, including a single-stage range.
|
||
6. Update command help, usage errors, `docs/cli.md`, and `docs/operations.md` for
|
||
the inclusive/defaulting behavior. Do not document the alias until Stage 5.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Command tests cover valid ranges, defaults, invalid/reversed ranges,
|
||
singleton duplicates in both syntaxes, and artifact/range compatibility.
|
||
- Unbounded `run` and unbounded `session plan` retain their prior behavior
|
||
except for the canonical order implemented in Stage 1.
|
||
- Parsing tests assert structural equivalence between the run and plan range
|
||
values, without duplicating all planner cases at the command layer.
|
||
|
||
## Stage 4 — Bounded Runner Prerequisites And Composition
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Make bounded execution honor its mutation boundary while still failing safely
|
||
when excluded upstream work cannot support the selected stages.
|
||
|
||
### Required Work
|
||
|
||
1. Before starting a bounded run, inspect the session manifest for every
|
||
canonical stage before `--from`. Accept only `succeeded` or `skipped` as a
|
||
terminal prerequisite status. Treat an absent record and every other status
|
||
as unusable.
|
||
2. Report the first unusable prefix stage in canonical order, its status (or
|
||
absence), the selected start, and an actionable suggestion to widen the
|
||
range or recover that stage explicitly. Do not mutate the manifest, create a
|
||
run record, or invoke an adapter before this check succeeds.
|
||
3. Do not call resume validators for excluded prefix stages. Concrete selected
|
||
stages continue to resolve and validate their own manifest-authoritative
|
||
inputs, so an unsafe or missing artifact still fails at its owning boundary.
|
||
4. Do not inspect stages after `--through` as prerequisites. They may become
|
||
stale through Stage 1 invalidation, but must not be scheduled.
|
||
5. Ensure force decisions are calculated only for stages in the bounded plan.
|
||
A forced selected stage may stale dependencies outside the range but cannot
|
||
execute them.
|
||
6. Make production composition plan-aware. Initialize and validate only the
|
||
external adapters, object storage, remote locks, and other collaborators
|
||
needed by selected stages or by shared session lifecycle requirements.
|
||
Preserve each selected stage's current fail-fast configuration validation.
|
||
7. Keep post-publish cleanup conditional on publish actually executing.
|
||
8. Update operational and troubleshooting documentation for prerequisite
|
||
failures, recovery, excluded-stage behavior, and plan-aware composition.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Runner tests cover every unacceptable prefix status, accepted skipped
|
||
prerequisites, no prefix for a first-stage run, and no check after the end.
|
||
- Tests prove excluded stages are neither executed nor resume-validated and
|
||
that a failed prerequisite check performs no persistent run mutation.
|
||
- Adapter-composition tests prove a bounded render-only run does not require
|
||
Notarius or Scriptorium, an extract-only run requires Notarius but not
|
||
Scriptorium, and analyze requires only its actual collaborators.
|
||
- Tests prove force/invalidation can stale an out-of-range dependent without
|
||
executing it and that stop-on-selected-stage-failure remains unchanged.
|
||
|
||
## Stage 5 — Exact `regenerate-artifacts` Alias
|
||
|
||
**Status: Completed**
|
||
|
||
### Goal
|
||
|
||
Add the transparent convenience command without creating another orchestration
|
||
path.
|
||
|
||
### Required Work
|
||
|
||
1. Register the top-level `regenerate-artifacts` command.
|
||
2. Implement it only by constructing the equivalent canonical arguments and
|
||
invoking the shared `run` parser/handler:
|
||
|
||
```text
|
||
run <session_id> --force --from extract --through analyze [caller options]
|
||
```
|
||
|
||
Preserve common session/configuration arguments and all repeatable
|
||
`--artifacts` values.
|
||
3. Do not add an alias-specific loader, plan, runner, prerequisite check,
|
||
summary, artifact rule, or force rule. Canonical run diagnostics may call
|
||
the operation `run`.
|
||
4. Let the shared duplicate-singleton validation from Stage 3 reject a caller's
|
||
`--force`, `--from`, or `--through`, since the alias already supplies them.
|
||
5. Give the alias concise help that states the exact expansion, that extraction
|
||
always runs, that selected analysis artifacts and required prerequisites are
|
||
rebuilt, and that publish/notify never run.
|
||
6. Add the alias to `docs/cli.md` and the development workflow in
|
||
`docs/operations.md`.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- A narrow command test captures the forwarded arguments or resulting shared
|
||
command request and proves exact equivalence, including option pass-through.
|
||
- Tests prove alias help does not execute, duplicate owned options fail through
|
||
the shared parser, and unknown/private alias options are not accepted.
|
||
- Do not duplicate runner integration cases under the alias name; its only
|
||
behavior is expansion.
|
||
|
||
## Stage 6 — Versioned Analyze-Artifact Manifest State
|
||
|
||
### Goal
|
||
|
||
Introduce a backward-compatible, analyze-specific session-manifest model that
|
||
can represent independently current artifacts without treating them as stages.
|
||
|
||
### Required Work
|
||
|
||
1. Add an explicit analysis-state contract version to the `analyze` stage
|
||
record. Absence means legacy aggregate-only state; a present supported
|
||
version distinguishes a valid empty set from legacy data.
|
||
2. Add a map keyed by normalized configured artifact key. Each record must
|
||
contain:
|
||
- key and disposition/status (`current`, `stale`, `missing`, `failed`, or
|
||
`unselected`);
|
||
- fingerprint contract version and fingerprint when evaluated;
|
||
- normalized configured dependency keys;
|
||
- the existing output artifact record when current, including source ID,
|
||
contract, canonical confined path, and checksum, plus a separate
|
||
analyze-record output-size field (do not broaden every artifact schema
|
||
solely to carry this analyze-specific evidence);
|
||
- producer run ID and update time;
|
||
- a bounded non-secret error for failed work; and
|
||
- useful non-secret Scriptorium provenance, logs, and generated config paths
|
||
already allowed by manifest policy.
|
||
3. Add the corresponding analyze-artifact collection to invocation-stage
|
||
records. A session record describes the reconciled current set; an
|
||
invocation record describes only work evaluated or attempted by that run.
|
||
4. Keep the schema explicitly owned by analyze. Do not add dynamic stage names
|
||
or generic pipeline-subtask abstractions.
|
||
5. Extend validation and canonical serialization for the new fields. Validate
|
||
normalized unique keys, supported versions, status-specific required and
|
||
forbidden fields, output identity, checksum/size, and non-secret bounded
|
||
metadata. Maintain deterministic JSON output.
|
||
6. Readers must accept old manifests with no new fields. Writers must not
|
||
fabricate fingerprints from legacy aggregate outputs or emit parallel
|
||
legacy state as current evidence.
|
||
7. Preserve per-artifact analysis state when ordinary stage lifecycle helpers
|
||
clear aggregate result details during running, failure, or skip. Other stage
|
||
records keep their existing behavior.
|
||
8. Update `docs/internal/manifest.md` with the new authority, legacy meaning,
|
||
status model, and session-versus-invocation distinction.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Manifest round-trip tests cover every status and deterministic map output.
|
||
- Validation tests cover malformed keys, unsupported versions, impossible
|
||
status/field combinations, incomplete current output, and bad checksum/size.
|
||
- Legacy fixtures remain readable and are explicitly identified as lacking
|
||
current artifact evidence.
|
||
- Lifecycle tests prove unrelated current records survive transitions while
|
||
aggregate output/log/config/metadata fields retain their prior clearing
|
||
semantics.
|
||
|
||
## Stage 7 — Runner Projection And Partial-Error State Boundary
|
||
|
||
### Goal
|
||
|
||
Give analyze one safe way to promote reconciled session state and invocation
|
||
history through the existing application-owned runner transaction.
|
||
|
||
### Required Work
|
||
|
||
1. Extend the stage result contract with an optional analyze-specific state
|
||
projection containing:
|
||
- the complete reconciled session analysis state; and
|
||
- the invocation subset attempted or evaluated in the current run.
|
||
Keep ordinary stage results unchanged.
|
||
2. On successful analyze completion, have the runner validate and apply the
|
||
session projection to the session manifest and the invocation projection to
|
||
the run manifest. Rebuild aggregate session `Outputs` deterministically from
|
||
current per-artifact records only. Keep invocation `Outputs` limited to
|
||
artifacts actually produced by that invocation.
|
||
3. Add a restricted result-plus-error path for analyze. If a stage returns an
|
||
error with an analyze-state projection, validate and persist only that state
|
||
before marking the aggregate stage failed. Ignore/reject ordinary success
|
||
outputs, success disposition, or unrelated stage projections alongside an
|
||
error.
|
||
4. Apply state only after the stage has validated run-local outputs and safely
|
||
materialized the records it marks current. The runner must never derive
|
||
records by scanning output directories.
|
||
5. On projection validation or persistence failure, fail conservatively and do
|
||
not advertise newly attempted artifacts as current. Preserve the last
|
||
durable unrelated current records.
|
||
6. Add `Force` to the stage environment as application-owned invocation
|
||
context, set it from the selected stage decision, and leave unselected
|
||
stages unaffected. This will distinguish explicit targets from ordinary
|
||
stale prerequisite rebuilding in later stages.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Runner tests prove separate session and invocation projections on success.
|
||
- Error-path tests prove completed artifacts can be durably represented, the
|
||
failed target is not current, unrelated prior current records survive, and
|
||
aggregate analyze/publish state remains conservative.
|
||
- Tests reject state projection from non-analyze stages and malformed or
|
||
contradictory result-plus-error payloads.
|
||
- Existing stages and ordinary error behavior remain unchanged.
|
||
|
||
## Stage 8 — Manifest-Authoritative Configured Artifact Evidence
|
||
|
||
### Goal
|
||
|
||
Make configured analysis outputs available to analyze and publish only through
|
||
validated current manifest evidence.
|
||
|
||
### Required Work
|
||
|
||
1. Add an analysis-evidence inspector and catalog hydrator analogous to the
|
||
extraction-evidence owner, but specific to configured Scriptorium artifacts.
|
||
2. For a `current` record, verify the supported state/fingerprint version,
|
||
configured key and source ID, contract, configured canonical path, confined
|
||
no-follow regular file, stored size, and stored checksum. Return a typed
|
||
current/non-current result with an actionable reason.
|
||
3. Treat stale, missing, failed, unselected, legacy, removed, malformed,
|
||
unsafe, missing, or checksum-mismatched evidence as unavailable. Do not
|
||
silently rewrite status during read-only catalog hydration.
|
||
4. Remove current-session configured-artifact fallbacks that call `stat` and
|
||
mark canonical files available merely because they exist. Hydrate analyze,
|
||
publish, operator display, and artifact resolution from the evidence owner.
|
||
5. Retain existing behavior for prior-session inputs or other source kinds only
|
||
where their canonical owner already has an explicit compatibility policy;
|
||
do not broaden filesystem fallback.
|
||
6. When current configuration removes or renames a key, omit its old record
|
||
from the current catalog even if the manifest retains history until the next
|
||
analyze reconciliation.
|
||
7. Update `docs/internal/artifacts.md` and `docs/internal/stage-publish.md` for
|
||
manifest-authoritative configured results.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Evidence tests cover valid current output, each non-current status, legacy
|
||
absence, config/path/source mismatch, symlink/non-regular files, missing
|
||
files, size mismatch, and checksum mismatch.
|
||
- Analyze and publish catalog tests prove an incidental file is unavailable and
|
||
a validated current record is available.
|
||
- Publish tests prove stale configured artifacts cannot be selected or uploaded
|
||
while unrelated current artifacts remain publishable.
|
||
|
||
## Stage 9 — Deterministic Analyze Input Identity
|
||
|
||
### Goal
|
||
|
||
Resolve every Narratio-visible analyze input into a stable semantic identity
|
||
that the fingerprint engine can consume without depending on workspace paths or
|
||
producer runs.
|
||
|
||
### Required Work
|
||
|
||
1. Define an ordered input-identity record containing the configured input
|
||
name, source ID, required/optional policy, presence/absence, contract,
|
||
content checksum, and size. Include a stable session-relative or source-based
|
||
logical identity where needed; never include an absolute path.
|
||
2. Resolve transcript, prepared-input, extraction-lane, previous-session, and
|
||
configured-artifact sources through their existing artifact resolvers and
|
||
catalogs. Configured-artifact dependencies must pass Stage 8 evidence.
|
||
3. Hash validated regular files using the repository's streaming file-safety
|
||
primitives and central size limits. Reuse an already validated manifest
|
||
checksum when its owning evidence contract proves it represents the same
|
||
bytes; do not read whole artifacts into memory.
|
||
4. Represent an absent optional input explicitly so appearance/disappearance
|
||
changes identity. A missing required input remains an error.
|
||
5. Normalize input ordering independently of Go map iteration while preserving
|
||
any configured order whose semantics are observable to Scriptorium.
|
||
6. Keep identity resolution read-only. It may inspect files and manifests but
|
||
cannot materialize output, update status, invoke adapters, or create run
|
||
records.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Table-driven tests cover every supported source kind, optional absence,
|
||
missing required input, unsafe path/type, checksum reuse, and changed bytes.
|
||
- Identical bytes at relocated workspace roots produce identical semantic
|
||
identities; changed bytes or contracts produce different identities.
|
||
- Configured dependencies cannot resolve from incidental or stale files.
|
||
- Ordering remains deterministic across randomized map insertion.
|
||
|
||
## Stage 10 — Versioned Analysis Fingerprints And Reconciliation
|
||
|
||
### Goal
|
||
|
||
Classify configured artifacts as current or requiring work from one deterministic
|
||
fingerprint contract shared by resume validation, planning, and execution.
|
||
|
||
### Required Work
|
||
|
||
1. Define a centrally named fingerprint contract version and a canonical
|
||
serialization used only as hash input. Do not use ad hoc string concatenation
|
||
or Go map serialization.
|
||
2. Include all Narratio-observable result-affecting fields:
|
||
- normalized artifact key and normalized Scriptorium artifact configuration;
|
||
- deterministic dependency keys;
|
||
- effective prompt/profile identifiers, render-debug behavior, normalized
|
||
output identity, ordered input declarations, and sorted effective vars;
|
||
- Stage 9 resolved input identities and current configured-dependency output
|
||
identities; and
|
||
- result-affecting global Scriptorium configuration that Narratio directly
|
||
passes or interprets.
|
||
3. Exclude timeout/retry settings, absolute binary/config/output/workspace
|
||
paths, run IDs, timestamps, log/config output locations, and executable or
|
||
arbitrary transitive file contents. Document that explicit force is required
|
||
when an unobserved Scriptorium-private input changes.
|
||
4. Compute dependency artifact fingerprints in deterministic topological order
|
||
and reject unknown dependencies and cycles through the existing configuration
|
||
validation owner.
|
||
5. Build a read-only reconciliation function that compares current config,
|
||
stored record/version/fingerprint, Stage 8 output evidence, and newly resolved
|
||
identities. Return a typed reason for `current`, `stale`, `missing`, `failed`,
|
||
`legacy`, `removed`, or otherwise non-resumable state.
|
||
6. Fingerprint byte-identical input content identically even when it came from
|
||
a forced upstream run with a different producer run ID.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Golden or table-driven tests prove deterministic fingerprints across map
|
||
order, workspace relocation, and producer run IDs.
|
||
- Sensitivity tests change each included semantic field independently and
|
||
assert a new fingerprint. Exclusion tests cover timeout, timestamp, run ID,
|
||
and absolute-root-only changes.
|
||
- Reconciliation tests cover current, tampered output, changed dependency,
|
||
optional input transition, legacy record, removed config, version mismatch,
|
||
and byte-identical upstream replacement.
|
||
- `docs/internal/stage-analyze.md` records the implemented fingerprint boundary
|
||
and its explicit limitations.
|
||
|
||
## Stage 11 — Incremental Analysis Work Planning
|
||
|
||
### Goal
|
||
|
||
Turn reconciliation results, explicit selection, dependencies, and force into
|
||
a deterministic artifact execution plan without invoking Scriptorium.
|
||
|
||
### Required Work
|
||
|
||
1. Define explicit targets as the normalized `--artifacts` selection when
|
||
present, otherwise all enabled configured artifacts. Preserve existing
|
||
unknown/ambiguous-selection validation.
|
||
2. Compute the transitive configured prerequisite closure of explicit targets
|
||
and order the closure deterministically and topologically.
|
||
3. Reuse a prerequisite when Stage 10 classifies it current. Schedule a stale,
|
||
missing, failed, legacy, or invalid prerequisite before its dependent.
|
||
4. Force only explicit targets. A prerequisite is forced only when it is also
|
||
an explicit target; otherwise a current prerequisite is reused.
|
||
5. Preserve the existing selection rule: default selection includes enabled
|
||
artifacts, while an explicitly named disabled artifact is a valid target.
|
||
A disabled configured prerequisite may be reused when current or executed
|
||
when it is in a selected target's required closure. Never execute an
|
||
unrelated disabled artifact.
|
||
6. Retain valid unselected records. Classify artifacts removed or renamed from
|
||
current configuration as unavailable in the reconciled session projection.
|
||
7. Produce a typed plan containing explicit targets, prerequisite-only work,
|
||
reused current artifacts, invalidated/removed records, reasons, and the
|
||
deterministic execution order. Keep the function read-only and free of
|
||
adapter calls.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Planning tests cover full/default selection, partial selection, dependency
|
||
closure, nested dependencies, current prerequisite reuse, stale prerequisite
|
||
rebuilding, target-only force, disabled prerequisites, unknown keys, cycles,
|
||
removed configuration, and deterministic order.
|
||
- Tests prove valid unrelated records survive the projected result and legacy
|
||
unselected outputs do not become current.
|
||
- The planner exposes enough typed information for `session plan` and analyze
|
||
execution to share decisions rather than recomputing them differently.
|
||
|
||
## Stage 12 — Incremental Analyze Execution And Promotion
|
||
|
||
### Goal
|
||
|
||
Execute the Stage 11 work plan on the successful path, safely promote validated
|
||
results, and preserve or invalidate records according to actual output identity.
|
||
|
||
### Required Work
|
||
|
||
1. Refactor analyze to execute only scheduled artifacts in deterministic order.
|
||
Reused current artifacts must be exposed to later scheduled dependents
|
||
through the runtime catalog without invoking Scriptorium.
|
||
2. For each scheduled artifact, keep output run-local until the adapter result
|
||
and output pass existing safety, contract, size, and checksum validation.
|
||
Materialize canonically only through the established file-operation owner.
|
||
3. Record the new fingerprint, full output evidence, producer run ID, bounded
|
||
non-secret provenance, logs, and generated configs in the Stage 7 projection.
|
||
4. After replacement, compare semantic output identity. If bytes/contract are
|
||
unchanged, allow an unselected dependent whose recomputed fingerprint is
|
||
equal to remain current. If identity changes, mark every unselected
|
||
configured dependent stale without executing it.
|
||
5. Preserve valid unrelated current records and outputs during partial runs.
|
||
Reconstruct aggregate session outputs from all current records; report only
|
||
newly produced outputs in the invocation manifest.
|
||
6. A partial invocation succeeds when every explicit target and required
|
||
prerequisite succeeds, even if unrelated configured records remain stale.
|
||
7. Never synthesize a current record from an existing canonical output.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Analyze tests cover current reuse, partial rerun preservation, nested stale
|
||
prerequisite rebuilding, selected force, dependency ordering, changed-output
|
||
dependent invalidation, and identical-output dependent preservation.
|
||
- Legacy tests prove full selection rebuilds the effective set, while partial
|
||
selection promotes only targets/prerequisites and leaves legacy unselected
|
||
outputs unavailable.
|
||
- Adapter fakes remain offline and assertions focus on requests, resulting
|
||
files, manifest records, and catalogs rather than private implementation maps.
|
||
|
||
## Stage 13 — Incremental Analyze Failure Safety
|
||
|
||
### Goal
|
||
|
||
Complete the incremental executor with conservative, durable behavior for
|
||
partial adapter, validation, materialization, and persistence failures.
|
||
|
||
### Required Work
|
||
|
||
1. At each artifact boundary, retain enough reconciled state to return the
|
||
restricted Stage 7 projection if later work fails. Do not report a completion
|
||
until its run-local output has been validated and canonically materialized.
|
||
2. On failure, mark the active target `failed` with a bounded non-secret error.
|
||
Mark any artifact whose current identity depends on the unavailable result
|
||
stale, including unselected dependents, without executing them.
|
||
3. Preserve unrelated previous current records. Preserve earlier completions
|
||
from this invocation only when they crossed the defined materialization and
|
||
validation boundary; include them in invocation history.
|
||
4. Return the restricted state projection alongside the original error so the
|
||
runner can persist artifact state before marking aggregate analyze failed.
|
||
Dependency-aware invalidation must keep publish and notify conservative.
|
||
5. If persistence of the partial projection itself fails, surface that failure
|
||
with the original context, retain the last durable manifest, and treat any
|
||
newly materialized file as incidental rather than current evidence.
|
||
6. Never mark a failed or unverified artifact current merely because old bytes
|
||
remain at its canonical path. Never reconstruct partial success by scanning
|
||
output directories after an error.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Tests cover first, middle, and last artifact adapter failures; unsafe or
|
||
invalid adapter output; canonical materialization failure; and session/run
|
||
manifest persistence failure.
|
||
- Each case proves the failed target is unavailable, dependent state is stale,
|
||
unrelated current work survives, and only durably completed work appears in
|
||
invocation history.
|
||
- Tests prove old canonical bytes and incidental newly materialized bytes do
|
||
not override the durable manifest authority.
|
||
- Error wrapping remains actionable and non-secret, and the original adapter or
|
||
filesystem cause remains discoverable.
|
||
|
||
## Stage 14 — Analyze Resume Validation And Plan/Run Parity
|
||
|
||
### Goal
|
||
|
||
Make aggregate analyze skipping and `session plan` reflect artifact-level
|
||
freshness using the same read-only decision engine as execution.
|
||
|
||
### Required Work
|
||
|
||
1. Implement an analyze-specific resume validator that runs Stages 8–11
|
||
reconciliation for the requested artifact set. It is resumable only when all
|
||
explicit targets and required prerequisites are current and no selected work
|
||
is scheduled.
|
||
2. A succeeded aggregate analyze record with stale unrelated artifacts may
|
||
still skip for a partial selection that does not require them. A later full
|
||
selection must not skip them.
|
||
3. If coarse stage invalidation marked analyze stale but recomputation proves
|
||
selected fingerprints and outputs unchanged, allow analyze execution to
|
||
perform zero Scriptorium calls and restore the correct successful aggregate
|
||
state through the ordinary runner boundary.
|
||
4. Extend `session plan` to invoke the same stage resume validators and analyze
|
||
work planner as `run`, using a cloned/in-memory manifest transition model to
|
||
account for earlier selected stages and their invalidation. It must not
|
||
persist a session manifest, create an invocation directory, materialize a
|
||
file, or call an external adapter.
|
||
5. Plan output must distinguish pipeline-stage run/skip decisions and, for
|
||
analyze, explicit targets, prerequisite-only rebuilds, and current reuse with
|
||
concise reasons. Do not promise output identities that require execution.
|
||
6. Preserve the rule that excluded stages are never resume-validated. The
|
||
Stage 4 prefix status check remains structural and read-only.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Resume tests cover full and partial selections, force, current/stale/legacy
|
||
mixtures, current prerequisites, changed inputs, output tampering, and a
|
||
stale aggregate record with semantically current selected artifacts.
|
||
- Paired plan/run tests feed the same fixture and assert identical selected
|
||
stage and analyze-artifact decisions before execution.
|
||
- Side-effect tests prove planning performs no manifest write, run-directory
|
||
creation, canonical materialization, or adapter invocation.
|
||
- Existing extract resume validation remains functional in bounded and
|
||
unbounded plans.
|
||
|
||
## Stage 15 — Assembled Workflow And Compatibility Coverage
|
||
|
||
### Goal
|
||
|
||
Verify the complete feature through production composition and representative
|
||
historical state without adding another behavior path.
|
||
|
||
### Required Work
|
||
|
||
1. Add assembled application tests for:
|
||
- an ordinary full run in the new canonical order;
|
||
- `run --from extract --through analyze --force` with and without artifact
|
||
selection;
|
||
- the exact alias reaching the same bounded-run request;
|
||
- render-only and extract-only forced runs demonstrating sibling
|
||
invalidation independence;
|
||
- a later publish consuming only current configured artifacts; and
|
||
- stop-on-failure with dependent stages outside the range left stale but
|
||
unexecuted.
|
||
2. Add compatibility fixtures or focused tests for manifests created under the
|
||
old render/extract order and aggregate-only analyze state. Confirm no
|
||
transcript stage is invalidated solely by version or old relative order.
|
||
3. Exercise a complete legacy transition: partial regeneration, unavailable
|
||
unselected legacy output, later full regeneration, then successful publish
|
||
from current per-artifact records.
|
||
4. Verify production composition does not eagerly require excluded adapters
|
||
and that generated run manifests list only the bounded requested stages and
|
||
actual invocation artifacts.
|
||
5. Remove obsolete suffix-invalidation, aggregate-analysis-replacement, and
|
||
filesystem-discovery compatibility code that no current path requires. Keep
|
||
any unavoidable compatibility shim narrow, clearly commented with its
|
||
removal condition, and covered by a legacy test.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- Assembled tests use controlled fakes and real application wiring; they remain
|
||
offline and deterministic.
|
||
- The full suite passes under the standard test command and repository race or
|
||
platform checks required by policy.
|
||
- Coverage is allocated to behavior owners; this stage adds only integration
|
||
assertions that smaller tests cannot prove.
|
||
|
||
## Stage 16 — Canonical Documentation And Quality Closure
|
||
|
||
### Goal
|
||
|
||
Bring every current-behavior owner into agreement and perform the final
|
||
repository-wide policy and regression check.
|
||
|
||
### Required Work
|
||
|
||
1. Audit current documentation for the old extract/render order, suffix-only
|
||
invalidation, unbounded-only run behavior, aggregate analysis replacement,
|
||
and filesystem-discovered configured artifacts.
|
||
2. Finalize:
|
||
- `docs/cli.md` for range syntax, duplicate singleton rejection, artifact
|
||
selection, exact alias expansion, and examples;
|
||
- `docs/operations.md` for the post-transcript development loop, force scope,
|
||
plan preview, explicit publish, and recovery;
|
||
- `docs/troubleshooting.md` for bounded prerequisite failures, stale or
|
||
tampered analysis evidence, legacy regeneration, fingerprint limitations,
|
||
and explicit force for unobserved Scriptorium inputs;
|
||
- `docs/internal/overview.md`, `stage-render.md`, `stage-extract.md`,
|
||
`stage-analyze.md`, `stage-publish.md`, `artifacts.md`, `manifest.md`, and
|
||
`adapters.md` for their implemented contracts; and
|
||
- `docs/policy/architecture.md` for flat canonical execution plus separate
|
||
dependency-aware invalidation and contiguous bounded runs.
|
||
3. Keep volatile syntax and schema details in their canonical owners. Other
|
||
documents should link rather than duplicate large flag or field inventories.
|
||
4. Confirm all examples use the new order and that none describe the fixed
|
||
dependency relation as a configurable DAG or claim perfect observation of
|
||
Scriptorium-private inputs.
|
||
5. Run formatting, documentation checks, static analysis, `go test ./...`, and
|
||
any race/platform checks required by repository policy. Review the final diff
|
||
for accidental generated files, secrets, broad refactors, stale compatibility
|
||
branches, and policy violations.
|
||
6. Update this plan's status only after all prior acceptance criteria pass. Do
|
||
not delete the accepted feature roadmap until the project's normal roadmap
|
||
closeout process authorizes it.
|
||
|
||
### Tests And Exit Criteria
|
||
|
||
- All maintained documentation agrees with implemented behavior and passes the
|
||
documentation policy's discoverability and ownership requirements.
|
||
- All repository-required checks pass from a clean checkout without installed
|
||
external adapter binaries.
|
||
- The worktree contains only intentional feature changes, no unresolved TODOs
|
||
standing in for this plan, and no known divergence from the feature roadmap.
|
||
|
||
## Open Questions
|
||
|
||
None. The accepted feature roadmap and the settled decisions above are
|
||
sufficient to implement every stage without a further product or architecture
|
||
choice.
|