Update documentation for the new analyze stage and artifact registry
This commit is contained in:
@@ -4,13 +4,13 @@
|
||||
Developers and LLM coding agents changing Narratio internals.
|
||||
|
||||
## Scope
|
||||
Implementation-accurate contracts for workspace/state, stages, and external adapter boundaries.
|
||||
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, and adapter boundaries.
|
||||
|
||||
## Component Docs
|
||||
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
|
||||
- `storage.md`: remote storage backend contracts and object-store invariants.
|
||||
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
|
||||
- `artifacts.md`: supported artifact IDs, transcript tiers, and source-resolution behavior.
|
||||
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
|
||||
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
|
||||
- `stage-prepare.md`: input materialization and provenance capture.
|
||||
- `stage-transcribe.md`: WhisperX transcript generation.
|
||||
@@ -18,7 +18,7 @@ Implementation-accurate contracts for workspace/state, stages, and external adap
|
||||
- `stage-polish.md`: Audita transcript polishing.
|
||||
- `stage-normalize.md`: post-polish normalization.
|
||||
- `stage-trim.md`: bounds-driven transcript trimming.
|
||||
- `stage-analyze.md`: Scriptorium session recap generation.
|
||||
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
|
||||
- `stage-archive.md`: archive upload and current-pointer publish contract.
|
||||
|
||||
## External Integration Notes
|
||||
|
||||
@@ -1,39 +1,40 @@
|
||||
# Internal: Artifacts
|
||||
|
||||
## Purpose
|
||||
Describe supported session artifact IDs, transcript tiers, and artifact resolution/provenance behavior used by stage logic and Scriptorium input configuration.
|
||||
Define Narratio's artifact identity and resolution model for built-in transcript/bounds artifacts and runtime-configured analyze artifacts.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Artifact source identifiers from stage config/runtime (for example `pipeline.scriptorium.artifacts.*.inputs.*.source`).
|
||||
- Session paths and optional session manifest stage outputs.
|
||||
- artifact sources from config/runtime (`pipeline.scriptorium.artifacts.*.inputs.*.source`)
|
||||
- session paths and optional session manifest stage outputs
|
||||
- runtime artifact catalog state for configured artifact sources
|
||||
|
||||
Outputs:
|
||||
- Resolved local artifact path + provenance (`ResolvedSessionArtifact`).
|
||||
- Validation errors for unsupported or unreadable artifact sources.
|
||||
- resolved local artifact path and provenance (`ResolvedSessionArtifact`)
|
||||
- runtime catalog entries for planned/executable/available artifacts
|
||||
- validation errors for unsupported, missing, or invalid artifact sources
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Canonical artifact ID registry and metadata (`internal/artifacts/artifact_resolver.go`).
|
||||
- Alias normalization for legacy source names.
|
||||
- Resolution order and artifact content validation.
|
||||
- built-in artifact registry and content validation rules
|
||||
- runtime artifact catalog for configured artifact source IDs
|
||||
- source resolution behavior for built-in and configured artifact sources
|
||||
|
||||
Does not own:
|
||||
- Artifact generation (stages produce files).
|
||||
- Manifest transition policy.
|
||||
- Remote archive publishing behavior.
|
||||
- artifact generation (stages produce files)
|
||||
- manifest transition policy
|
||||
- archive promotion behavior
|
||||
|
||||
## Config fields used
|
||||
Artifact source usage is driven by:
|
||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
||||
- `pipeline.scriptorium.artifacts.<name>.output_path`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
||||
- Optional source-specific fields for previous artifact input (`artifact`, `path`, `required`).
|
||||
|
||||
## External adapters used
|
||||
- No external service adapters.
|
||||
- Resolver relies on local filesystem checks + session manifest state.
|
||||
- none
|
||||
|
||||
## State and manifest behavior
|
||||
Supported canonical IDs and current mappings:
|
||||
Built-in registry entries:
|
||||
|
||||
| Artifact ID | Canonical file | Producer stage | Output kind |
|
||||
| --- | --- | --- | --- |
|
||||
@@ -42,39 +43,45 @@ Supported canonical IDs and current mappings:
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
|
||||
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `session_recap` |
|
||||
|
||||
Resolution order:
|
||||
1. Session manifest producer-stage outputs (if readable/valid).
|
||||
2. Canonical session path fallback.
|
||||
Runtime catalog entries include built-ins and configured `narratio.artifact.<name>` sources.
|
||||
|
||||
Provenance fields:
|
||||
- `ProducerStage`
|
||||
- `OutputKind`
|
||||
- `ProducerRunID` (when resolved from manifest output)
|
||||
- `Provenance` (`manifest.<stage>.outputs` or `fallback.canonical_path`)
|
||||
Catalog states:
|
||||
- `planned`: source is registered and known for this run
|
||||
- `executable`: configured artifact is selected for analyze execution
|
||||
- `available`: artifact has a usable file path (generated this run or reused from disk)
|
||||
|
||||
Content validation by artifact type:
|
||||
- Transcript artifacts: JSON with top-level `segments` array.
|
||||
- `narratio.bounds.session`: valid JSON.
|
||||
- `narratio.artifact.session_recap`: non-empty text.
|
||||
Resolution behavior:
|
||||
- built-in sources resolve via manifest producer outputs first, then canonical fallback path
|
||||
- configured `narratio.artifact.<name>` sources resolve through runtime catalog availability
|
||||
- configured source lookup requires catalog context
|
||||
|
||||
Configured artifact provenance values:
|
||||
- `generated.current_analyze_run`
|
||||
- `filesystem.disabled_artifact_output`
|
||||
|
||||
Content validation:
|
||||
- transcript built-ins: JSON with top-level `segments` array
|
||||
- bounds built-in: valid JSON
|
||||
- configured artifacts: non-empty text file
|
||||
|
||||
## Skip and resume behavior
|
||||
- Resolver has no direct skip/resume logic.
|
||||
- Resolver output influences stage behavior (for example analyze input resolution and required-input failures).
|
||||
- resolver and catalog have no direct skip/resume decisions
|
||||
- stage/runner skip-resume behavior consumes catalog/resolver results
|
||||
|
||||
## Failure behavior
|
||||
- Unsupported or empty artifact source -> normalization error.
|
||||
- Known source not found/readable -> `ErrSessionArtifactNotFound` wrapped error.
|
||||
- Found but invalid content -> validation error.
|
||||
- unsupported source -> source validation error
|
||||
- known source unavailable -> `ErrSessionArtifactNotFound`
|
||||
- configured source without catalog -> resolution error
|
||||
- resolved file with invalid content -> validation error
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/artifacts/resolve_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/config/scriptorium_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Artifact IDs are canonical interface values for stage/config integration.
|
||||
- Alias support is compatibility behavior layered on top of canonical IDs.
|
||||
- Manifest producer outputs are preferred over canonical fallback when both exist.
|
||||
- built-in IDs are static and registry-backed
|
||||
- configured artifact IDs are runtime-derived (`narratio.artifact.<name>`) and catalog-backed
|
||||
- built-in/source resolution remains deterministic and validation-gated
|
||||
|
||||
@@ -55,6 +55,7 @@ Relationship during execution:
|
||||
- Runner updates both manifests for every stage transition.
|
||||
- Session manifest is the durable pipeline-progress ledger.
|
||||
- Run manifest is invocation history and audit record.
|
||||
- Analyze stage outputs are persisted as `kind=scriptorium_artifact` with `source_id=narratio.artifact.<name>` for configured artifact identity.
|
||||
|
||||
## Skip and resume behavior
|
||||
- Resume and skip decisions are based on session-manifest stage statuses.
|
||||
|
||||
@@ -1,25 +1,31 @@
|
||||
# Stage: analyze
|
||||
|
||||
## Purpose
|
||||
Generate the session recap artifact using configured Scriptorium artifact settings.
|
||||
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- transcript inputs as requested by selected artifact config (processed/normalized/trimmed/current recap, depending on `pipeline.scriptorium.artifacts.session_recap.inputs`)
|
||||
- configured artifact definitions from `pipeline.scriptorium.artifacts`
|
||||
- selected artifact filter from runtime (`--artifacts`) when provided
|
||||
- resolved artifact input sources declared per artifact (`inputs.*.source`)
|
||||
- optional previous-session file inputs (`previous_session_artifact`)
|
||||
|
||||
Outputs:
|
||||
- `artifacts/session_recap.md`
|
||||
- one promoted output file per executed configured artifact at that artifact's configured `output_path`
|
||||
- stage metadata containing generated artifact entries and reused disabled-artifact entries
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Selecting supported analyze artifact (`session_recap` only)
|
||||
- Resolving transcript/reference inputs and vars
|
||||
- Optional render-debug execution before run
|
||||
- Main Scriptorium run and output promotion
|
||||
- runtime artifact catalog construction for analyze execution
|
||||
- selected-artifact planning and dependency ordering
|
||||
- per-artifact input resolution, var resolution, timeout/render-debug resolution
|
||||
- Scriptorium run/render invocation for each selected artifact
|
||||
- run-local output generation and canonical promotion
|
||||
|
||||
Does not own:
|
||||
- Transcript processing pipeline stages
|
||||
- Archive publish/pointer behavior
|
||||
- transcript generation/processing stages
|
||||
- archive promotion policy
|
||||
- per-artifact resume semantics
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
@@ -29,8 +35,9 @@ Does not own:
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
- `pipeline.scriptorium.render_debug`
|
||||
- `pipeline.scriptorium.artifacts.session_recap.*`
|
||||
- `pipeline.scriptorium.artifacts.<name>.*`
|
||||
- `enabled`
|
||||
- `depends_on`
|
||||
- `prompt_id`
|
||||
- `profile_id`
|
||||
- `timeout`
|
||||
@@ -41,29 +48,37 @@ Does not own:
|
||||
|
||||
## External Adapters Used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` (debug diagnostics)
|
||||
- `RunArtifact` (actual recap generation)
|
||||
- optional `RenderArtifact` (render debug)
|
||||
- `RunArtifact` (artifact generation)
|
||||
|
||||
## State and Manifest Behavior
|
||||
- If `pipeline.scriptorium` is nil, stage returns success metadata with `skipped=true`.
|
||||
- If no enabled artifacts exist, stage returns success metadata with `skipped=true`.
|
||||
- If enabled artifacts exist but any artifact other than `session_recap` is enabled, stage fails.
|
||||
- Uses run-local output/log/config/reports paths when run layout is enabled.
|
||||
- Promotes canonical recap output and records adapter metadata.
|
||||
- If `pipeline.scriptorium` is absent, stage returns success metadata with `skipped=true`.
|
||||
- If no artifacts are configured, stage returns success metadata with `skipped=true`.
|
||||
- If zero artifacts are executable after `enabled` + `--artifacts` filtering, stage returns success metadata with `skipped=true`.
|
||||
- Builds runtime catalog with built-ins and configured artifacts.
|
||||
- Non-executable configured artifacts are marked available only when their configured output file exists and is valid on disk.
|
||||
- Executes selected configured artifacts in topological order with deterministic tie-breaking.
|
||||
- For each generated artifact, records metadata fields including `name`, `source_id`, `output_kind`, `path`, `prompt_id`, `profile_id`, and `provenance`.
|
||||
- Reused disabled artifacts are recorded separately in `reused_artifacts` with provenance `filesystem.disabled_artifact_output`.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
- Stage-local "skipped" metadata is distinct from runner-level stage status skip.
|
||||
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
|
||||
- Analyze remains stage-scoped for resume/skip; there is no per-artifact resume state.
|
||||
- `--artifacts` filters which configured artifacts are executable when analyze runs; it does not imply `--force`.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing required resolved inputs, invalid transcript inputs, render/run adapter failures, or validation-failed run results.
|
||||
- Fails on invalid dependency ordering, unavailable required configured inputs, invalid built-in input prerequisites, render/run adapter failures, validation-failed adapter results, or missing/empty outputs.
|
||||
- Required configured dependency missing from catalog availability fails clearly before invocation.
|
||||
- Optional missing inputs are omitted.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Analyze implementation supports only `artifacts.session_recap` as executable artifact.
|
||||
- Optional inputs may be omitted; required inputs must resolve.
|
||||
- Successful output must exist and be non-empty before promotion.
|
||||
- Configured artifacts are identified by `narratio.artifact.<name>` source IDs.
|
||||
- Artifact-to-artifact references rely on explicit `depends_on` declarations validated in config.
|
||||
- Generated analyze outputs are treated uniformly as Scriptorium artifacts.
|
||||
- Successful outputs must exist and be non-empty before promotion.
|
||||
|
||||
Reference in New Issue
Block a user