diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..c011aac --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,577 @@ +# Notarius Extraction Implementation Plan + +## Status And Audience + +Proposed and not started. + +This plan is written for a GPT-5.6 Terra coding agent. Implement the stages in +strict numerical order. Do not skip ahead, combine stages merely to reduce the +number of prompts, or describe unfinished behavior as current functionality. + +The target state and acceptance criteria are owned by +[notarius-extract-stage.md](notarius-extract-stage.md). This document owns the +implementation sequence. + +## Progress + +| Stage | Status | +| --- | --- | +| Stage 1 | Complete | +| Stage 2 | Not started | +| Stage 3 | Not started | +| Stage 4 | Not started | +| Stage 5 | Not started | +| Stage 6 | Not started | +| Stage 7 | Not started | +| Stage 8 | Not started | +| Stage 9 | Not started | + +After completing and validating a stage, update only that stage's row to +`Complete` and record any material deviation in the relevant stage section. +Do not mark a stage complete while required tests or exit criteria remain. + +## Working Rules + +Before Stage 1, read: + +- `docs/development.md` and its task-specific references; +- `docs/roadmap/notarius-extract-stage.md` completely; +- `../notarius/docs/consumers/dnd-pipeline.md` and its linked subprocess, + receipt, JSON-output, and lane-contract documentation; and +- the focused Narratio documents and tests named by the current stage. + +For every stage: + +1. Inspect the current implementation before editing. Prefer the codebase + knowledge graph for code discovery as required by `AGENTS.md`. +2. Preserve unrelated worktree changes. +3. Implement the complete stage scope, including focused tests. Do not leave + TODO implementations, compatibility shims without owners, or knowingly + dead code for a later stage. +4. Use real internal collaborators in tests where fast and deterministic; fake + only subprocess, remote, clock, randomness, or other external boundaries. +5. Keep tests offline and independent of real Notarius, PromptKit, LLM + providers, credentials, and mutable services. +6. Run the focused package tests listed for the stage. Fix failures before + proceeding. +7. Run `go test ./...` after every stage. Also run `go vet ./...` and + `go build ./cmd/narratio` when the stage changes shared contracts, + composition, CLI behavior, or documentation examples. +8. Keep proposed behavior in `docs/roadmap/` until Stage 9. Intermediate code + is development work and must not cause current-behavior documentation to + claim that the end-to-end feature is complete. + +Intermediate stages must compile and pass the repository test suite. The work +is not release-ready until Stage 9 is complete. + +## Stage 1: Establish Shared Stage-Outcome And Artifact-Provenance Contracts + +Introduce the reusable internal contracts needed by extraction without adding +Notarius configuration or a registered stage. + +Implementation: + +1. Under `internal/artifactmodel`, add optional neutral models for: + - contract metadata: `media_type`, `schema_id`, `schema_version`, and + optional `module_key`; + - external provenance: `system`, `run_id`, `pipeline_id`, and + `artifact_id`. +2. Extend `artifacts.Ref` with: + - optional explicit `SourceID`; + - optional contract metadata; and + - optional external provenance. +3. Extend `manifest.ArtifactRecord` with the same nested optional metadata. + Use backward-compatible `omitempty` JSON fields. Old manifests must load + and round-trip without fabricated metadata. +4. Refactor `internal/app.mapResultOutputs` to prefer `Ref.SourceID` when set, + copy both metadata structures, and retain existing output-kind and analyze + inference only as fallback behavior. +5. Add an explicit `StageResult` disposition with succeeded and skipped + values. Preserve zero-value success so existing stages do not need + mechanical edits. +6. Add a stable skip-reason field. Reject or fail safely if a skipped result + contains outputs. +7. Teach `executeStages` to persist a self-skipped stage as skipped in both + session and run manifests, apply bounded result metadata and diagnostics, + clear any older session-stage outputs, and continue to later stages. + Distinguish this from the runner's existing decision to skip an already + succeeded stage. +8. Keep successful and failed behavior unchanged for existing stages. + +Tests: + +- `internal/artifactmodel`: JSON behavior for complete and omitted metadata. +- `internal/manifest`: old manifest compatibility and self-skip clearing of + earlier outputs. +- `internal/app`: explicit source ID precedence, fallback inference, + provenance copying, self-skip persistence, continuation after self-skip, + and rejection of skipped results containing outputs. + +Exit criteria: + +- Existing stage outputs and manifests remain compatible. +- A synthetic stage can self-skip without being recorded as succeeded. +- A later invocation will reconsider a self-skipped stage. +- `go test ./...`, `go vet ./...`, and `go build ./cmd/narratio` pass. + +## Stage 2: Add Notarius Configuration And Extraction Source Policy + +Add strict configuration and stable source identity, but do not invoke +Notarius or register `extract` yet. + +Implementation: + +1. Add an optional `Notarius *NotariusConfig` field to `PipelineConfig`. +2. Define: + - `enabled` with false default; + - `binary` with `notarius` default; + - `config_path`; + - `pipeline_id`; + - `timeout` with `3h` default; + - optional `working_directory`; and + - `outputs map[string]NotariusOutputConfig`. +3. Define each output with required `lane_id`, `media_type`, `schema_id`, and + `schema_version`, plus optional `module_key`. +4. Apply defaults centrally. When enabled, resolve `config_path` to an + absolute path using the same configuration-source semantics as comparable + adapter paths. Default `working_directory` to the resolved config file's + directory, and resolve an explicitly configured working directory to an + absolute path. +5. Validate enabled configuration before stage execution: + - non-empty binary, config path, pipeline ID, and output map; + - positive duration; + - non-empty output contract fields; + - output keys accepted by the existing configured-artifact key policy; + - unique normalized extraction source IDs; + - unique lane IDs; + - no collision with built-in, configured Scriptorium, or previous-session + source identities. +6. Add `SourceKindExtraction` and helpers for the exact + `narratio.extraction.` family in `internal/artifactpolicy` and + `internal/artifacts`. Do not accept arbitrary unconfigured extraction + sources at configuration validation boundaries. +7. Extend Scriptorium input and publish source validation to accept only + extraction keys declared by the same effective pipeline configuration. +8. Do not expose a Notarius session-ID setting, lane-selection flag, prompt, + profile, model, retry, or reference configuration. + +Tests: + +- strict YAML decoding and unknown-field rejection; +- omitted and disabled behavior; +- defaults and path resolution from pipeline-file location; +- enabled required fields and positive timeout; +- invalid/safe keys, normalized collisions, duplicate lanes, incomplete + contracts, and source-family collisions; +- valid and unknown extraction sources in Scriptorium inputs and publish + rules; and +- existing configurations and maintained examples continue to load. + +Exit criteria: + +- A complete Notarius configuration loads and normalizes deterministically. +- Invalid downstream extraction references fail during config validation. +- No runtime code invokes Notarius yet. +- `go test ./internal/config ./internal/artifactpolicy ./internal/artifacts` + and `go test ./...` pass. + +## Stage 3: Add Safe Immutable Directory Promotion + +Implement the filesystem primitive used to promote a validated external bundle +without exposing a partial or mixed durable directory. + +Implementation: + +1. Add a narrowly named directory-promotion operation to `internal/fileops`. + It accepts an existing source directory and a new destination directory. +2. Require a non-empty source and destination. The destination must not exist. +3. Create a temporary sibling of the destination so final installation uses a + same-filesystem rename. +4. Recursively walk with `Lstat` semantics: + - permit directories and regular files only; + - reject symlinks, devices, sockets, named pipes, and other special files; + - never follow a symlink; + - use deterministic lexical traversal; + - preserve relative layout; and + - use bounded safe permissions rather than copying unsafe mode bits. +5. Copy and sync each regular file, sync the completed temporary tree where + practical, and rename the temporary directory to the unique destination. +6. On any failure, remove only the operation's validated temporary sibling. + Never remove or overwrite the source, destination, workspace root, or a + broad caller-provided directory. +7. Return an error if a concurrent actor creates the destination before final + installation. + +Tests using `t.TempDir()`: + +- nested regular-tree success and byte-for-byte preservation; +- deterministic layout; +- destination already exists; +- source is not a directory; +- symlink to a file, symlink to a directory, and escaping symlink rejection; +- representative special-file rejection where portable; +- copy/install failure cleanup with no partial destination; and +- source preservation after success and failure. + +Exit criteria: + +- A promoted directory appears only as a complete unique tree. +- The operation cannot follow symlinks or replace an existing destination. +- `go test ./internal/fileops` and `go test ./...` pass. + +## Stage 4: Implement The Notarius Adapter + +Create the external subprocess boundary independently of stage policy. + +Implementation: + +1. Add `internal/adapters/notarius` with: + - a `Runner` interface; + - transport-neutral request and result models; + - receipt, index, lane-descriptor, and pipeline-wide-descriptor models; + - a production subprocess runner; and + - a small configurable fake for stage tests. +2. Build exactly: + + ```text + notarius run --config --input + --output-dir --json + ``` + + Use separate argument elements, absolute supplied paths, the configured + working directory, inherited environment, and Narratio's shared subprocess + timeout/cancellation behavior. Do not pass `--session-id`. +3. Write stdout directly to the configured receipt path and stderr to the + configured log path. Do not combine streams. +4. On nonzero exit, cancellation, or timeout, return the subprocess error and + do not parse stdout. +5. After exit zero, read receipt bytes with an explicit upper bound, decode + `notarius.run-result.v1`, tolerate unknown fields, and validate required + fields and matching pipeline ID. +6. Require an absolute `output_directory` and confine it beneath the absolute + output root from the request. Confine the relative `index_file` beneath the + returned directory using Narratio's root-scoped path helpers. Reject + absolute logical paths, traversal, symlinks at consumed paths, and + non-regular files. +7. Decode the supported production JSON index tolerantly. Validate required + management paths and lane descriptors, reject duplicate lane IDs, and + confine every lane and pipeline-wide descriptor file beneath the bundle + root. +8. Load `rejected.json` and `warnings.json` with explicit size bounds and + tolerant decoding. Return only their generic summaries, including lane ID + and reason code where present; do not read lane payload bodies or interpret + D&D fields. Keep free-form external messages bounded for immediate errors + and out of manifest metadata. +9. Return descriptors and paths without deciding which lanes are required or + whether their configured contracts match. +10. Do not implement automatic `notarius config validate` execution. + +Tests: + +- exact argument order and omission of `--session-id`; +- working directory, environment inheritance, stream separation, cancellation, + timeout, and nonzero exit; +- stdout ignored after failure; +- receipt size limit, malformed JSON, unsupported version, missing fields, + pipeline mismatch, and tolerated unknown fields; +- output-directory escape from the requested output root; +- index malformed/unsupported shapes, duplicate lanes, missing management + paths, and tolerated optional fields; +- malformed or oversized rejection/warning documents and tolerated unknown + summary fields; +- absolute logical paths, lexical traversal, root-prefix confusion, symlink + paths, and descriptor escape attempts; and +- valid lane plus chunk-map/evidence-context discovery. + +Use a test helper subprocess or injected shared runner. Do not require a real +Notarius binary. + +Exit criteria: + +- The adapter fully enforces the generic Notarius subprocess and discovery + boundary without D&D policy. +- `go test ./internal/adapters/notarius ./internal/adapters/subprocess` and + `go test ./...` pass. + +## Stage 5: Implement Extract-Stage Execution And Bundle Materialization + +Implement the stage behind direct package tests, but do not yet add it to the +canonical `stage.All()` registry. + +Implementation: + +1. Add `Notarius notarius.Runner` to `stage.Env` and default production + composition in `internal/app/runner.go` when enabled extraction is selected. +2. Add canonical path helpers for: + - the run-local extract directory; + - receipt and stderr paths; + - the run-local Notarius output root; and + - immutable `artifacts/notarius/` destinations. +3. Implement `extractStage` with name `extract` and declarations matching its + final-trimmed input and structured outputs. +4. When configuration is absent or disabled, return the explicit skipped + result with `notarius_disabled` and no outputs. +5. Resolve `narratio.transcript.final_trimmed` through the manifest-aware + resolver. Do not fall back to guessed filenames outside that resolver. +6. Invoke the adapter once with resolved absolute paths. +7. Apply required-output policy: + - find each configured output by exact lane ID; + - require exactly one descriptor; + - compare media type, schema ID/version, and optional module key; + - treat rejected or absent configured lanes as failure; + - tolerate unconfigured lanes; + - require each selected file to be regular, non-empty, and syntactically + valid JSON; and + - compute a staging checksum for later copy verification. +8. Validate the complete bundle tree for promotion, then use Stage 3's helper + to copy it to the unique immutable destination. +9. Re-resolve the promoted index and configured descriptor paths rather than + retaining staging paths. Compute authoritative SHA-256 checksums from the + promoted files and fail if promoted bytes differ from the validated source. +10. Return one `artifacts.Ref` per configured lane with explicit extraction + source ID, checksum, contract metadata, and Notarius external provenance. +11. Return the promoted `index.json` as a `notarius_index` output without a + selectable source ID. +12. Return bounded stage metadata containing durable bundle root, receipt and + diagnostic paths, receipt counts/status, rejection/warning summaries, the + producing Narratio run ID, and a deterministic fingerprint of the resolved + binary, config path, pipeline ID, timeout, working directory, and sorted + output-contract map. +13. Do not delete run-local diagnostics or failed bundles. + +Tests: + +- disabled self-skip; +- missing/invalid final-trimmed input before adapter invocation; +- exact adapter request construction; +- required, missing, rejected, duplicate, incompatible, and unconfigured + lanes; +- non-empty and JSON-syntax validation without D&D decoding; +- immutable promotion and promoted-path re-resolution; +- checksums, explicit source IDs, nested metadata, and external provenance; +- adapter and promotion failures do not return successful outputs; and +- complete bundle files are preserved while unknown files are not registered + as sources. + +Exit criteria: + +- Direct extract-stage execution produces a complete manifest-ready result + using only a fake Notarius runner. +- The stage is not yet in normal CLI plans. +- `go test ./internal/stage ./internal/artifacts ./internal/fileops` and + `go test ./...` pass. + +## Stage 6: Integrate Canonical Ordering, CLI Lifecycle, Force, And Resume + +Make `extract` a real pipeline stage and complete its orchestration semantics. + +Implementation: + +1. Register `extractStage` in `stage.All()` immediately after `trim` and before + `render`. +2. Update all hard-coded or user-visible stage inventories, stage-name + validation, CLI help, planning, prerequisite checks, and test fixtures. +3. Confirm full and single-stage commands can select `extract`. Do not add a + lane-selection flag and do not broaden `--artifacts` beyond Scriptorium + artifact selection. +4. Add an optional stage resume-validation contract. Use a result containing: + - resumable boolean; and + - bounded non-resumable reason. + Reserve an ordinary error for cases where a safe decision cannot be made. +5. Before skipping a succeeded stage, `executeStages` invokes the optional + validator. A non-resumable result marks the session stage stale with its + reason and schedules it to run. A validation error stops execution without + mutating the succeeded record. +6. Implement extraction resume validation against: + - enabled state; + - stored invocation/config-contract fingerprint; + - succeeded extract record; + - immutable bundle and index paths; + - exact configured source set; + - regular confined lane files; + - checksums; + - contracts; and + - Notarius external provenance. + Require the producing Narratio run ID recorded in extract metadata, output + records, and the immutable bundle path to agree. Do not compare it with the + session manifest's top-level `run_id`, which identifies the latest + invocation and may legitimately differ after resume. +7. Do not inspect or hash Notarius's transitive reference/profile closure. +8. Verify canonical downstream invalidation: + - force through `trim` stales extract and later succeeded stages; + - force `extract` stales render and later succeeded stages; + - force `render` does not stale extract. +9. Verify existing session manifests without an extract record normalize and + execute the new stage rather than failing migration. + +Tests: + +- exact canonical order and CLI stage inventory; +- full-plan and `run-stage extract` behavior; +- disabled skip followed by later enablement; +- successful resume skip; +- missing, tampered, incompatible, or config-mismatched outputs trigger stale + and rerun; +- unsafe validation errors stop without corrupting prior success; +- force and downstream invalidation in each direction named above; and +- old manifest compatibility. + +Exit criteria: + +- Normal execution includes `extract` in the agreed position. +- Disabled and resume behavior cannot make obsolete outputs current. +- `go test ./internal/stage ./internal/app ./internal/manifest`, + `go test ./...`, `go vet ./...`, and `go build ./cmd/narratio` pass. + +## Stage 7: Integrate Extraction Sources With The Runtime Catalog And Analyze + +Make succeeded extraction lanes selectable by configured Scriptorium +artifacts. + +Implementation: + +1. Extend `ArtifactCatalog` with extraction definitions registered from the + effective Notarius output map. +2. Add one shared manifest-backed hydration operation for extraction sources. + It accepts session paths, manifest, and configured extraction definitions + and marks entries available only after verifying the roadmap's full + successful-stage, source, path, checksum, contract, and provenance rules. +3. Add a distinct extraction provenance value such as + `manifest.current_extract_run`. +4. Extend `ResolveSessionArtifactWithCatalog` to resolve configured extraction + sources through catalog availability. It must never scan the Notarius + artifact directory or accept an incidental file. +5. Refactor catalog construction enough that extraction registration and + hydration are reusable by analyze and publish. Do not create a generic + workflow abstraction or move Scriptorium execution policy into artifacts. +6. Update analyze catalog creation to include effective extraction + definitions and current manifest state before planning Scriptorium inputs. +7. Preserve required/optional input behavior. Required unavailable extraction + inputs fail with actionable extract configuration/rerun guidance; optional + inputs are omitted normally. +8. Do not automatically add extraction inputs to any configured artifact. + +Tests: + +- registration, lookup, deterministic ordering, and source-family collision; +- successful current-manifest hydration; +- missing stage, skipped, failed, stale, interrupted, missing-source, + internally inconsistent producer identity, incompatible-contract, + missing-file, tampered-checksum, and unsafe-path rejection; +- required and optional analyze input behavior; +- selected Scriptorium artifact execution receives only explicitly configured + extraction inputs; and +- no object-store calls or directory scanning occur during analyze + extraction resolution. + +Exit criteria: + +- A fake successful extract run can feed selected JSON files into a fake + Scriptorium analysis run. +- No failed or incidental bundle can become available through the catalog. +- `go test ./internal/artifacts ./internal/stage ./internal/config` and + `go test ./...` pass. + +## Stage 8: Integrate Extraction Sources With Publish And Operator Inspection + +Complete downstream artifact handling without automatically publishing the +bundle. + +Implementation: + +1. Use Stage 7's shared extraction catalog registration and hydration in + publish planning and source resolution. +2. Permit explicit publish rules whose source is a configured extraction + source. Preserve existing destination policy, lock behavior, required + output policy, selected-Scriptorium behavior, and remote commit order. +3. Do not publish the complete Notarius bundle unless individual explicit + rules name selectable sources. `notarius_index` remains non-selectable in + the initial feature. +4. Ensure the published session manifest retains extraction artifact contract + and external provenance metadata through ordinary serialization. +5. Extend existing artifact/status/inspection output only where necessary so + configured extraction sources report planned, available, unavailable, and + published states consistently. Do not expose payload contents. +6. Confirm restore safely round-trips explicitly published extraction files + through existing manifest and confined-path behavior; add code only if a + real incompatibility is found. + +Tests: + +- successful explicit lane publication; +- required missing or invalid extraction source failure; +- optional/unselected behavior remains consistent; +- publish lock and commit-marker order remain unchanged; +- `--artifacts` does not partially select Notarius lanes; +- manifest metadata survives publish serialization and restore loading; and +- operator inspection never reads or prints extraction payload bodies. + +Exit criteria: + +- Explicit extraction publish rules work through the shared catalog. +- No implicit whole-bundle publication is introduced. +- `go test ./internal/stage ./internal/app ./internal/artifacts` and + `go test ./...` pass. + +## Stage 9: Add Maintained Examples, Current Documentation, And Final Validation + +Finish the public and maintainer contract only after the implementation is +complete. + +Implementation: + +1. Read Notarius's current published lane contracts and add exact media type, + schema ID, schema version, and module-key constraints for all ten lanes to + a maintained Narratio complete example. Do not guess or copy stale roadmap + placeholders. +2. Add a small maintained Scriptorium example that consumes a purpose-specific + subset of extraction sources rather than all ten. +3. Add `docs/integrations/notarius.md` as Narratio's external-consumer + contract. Link to Notarius's canonical documents instead of duplicating + complete schemas. +4. Add `docs/internal/stage-extract.md` for implemented stage mechanics, + lifecycle, resume validation, failure behavior, and focused tests. +5. Update the canonical owners listed in the roadmap's Documentation + Deliverables section. Keep configuration fields/defaults only in + `docs/config.md`, commands only in `docs/cli.md`, physical layout and force + procedures only in `docs/operations.md`, and implementation mechanics only + in internal documents. +6. Update every implemented stage inventory to show `extract` between `trim` + and `render`. +7. Add troubleshooting guidance for missing Notarius, nonzero exit, + receipt/index incompatibility, required-lane rejection, resume + invalidation, and the requirement to force extraction after changing + Notarius's transitive configuration inputs. +8. Review the roadmap status. If every acceptance criterion is implemented, + mark it complete or move completed planning material according to repository + convention without deleting useful decision context prematurely. +9. Check all changed links, examples, commands, field names, defaults, schemas, + and paths against the implementation. + +Validation: + +- focused example/configuration validation tests; +- `go test ./...`; +- `go vet ./...`; +- `go build ./cmd/narratio`; +- inspect `git diff --check`; +- confirm ordinary tests use no live Notarius, PromptKit, LLM provider, + credentials, network service, or mutable external state; and +- confirm no secrets or private campaign content were added. + +Exit criteria: + +- Every feature-roadmap acceptance criterion is satisfied. +- Maintained examples load under strict configuration validation. +- Current-behavior documentation accurately describes the shipped feature + without duplicating Notarius-owned schema definitions. +- The repository-wide validation suite passes. + +## Open Questions + +None. The target architecture, defaults, lifecycle behavior, compatibility +boundary, storage layout, source identity, downstream selection model, and +implementation order are decision-complete for implementation. If repository +inspection exposes a contradiction with an implemented invariant, stop at the +affected stage and document the concrete conflict rather than silently +changing this plan's target behavior. diff --git a/docs/roadmap/notarius-extract-stage.md b/docs/roadmap/notarius-extract-stage.md index dba0d3c..7a536af 100644 --- a/docs/roadmap/notarius-extract-stage.md +++ b/docs/roadmap/notarius-extract-stage.md @@ -2,266 +2,497 @@ ## Status -Proposed. +Proposed. Implementation has not started. + +The ordered implementation plan is maintained in +[implementation.md](implementation.md). Until that plan is complete, this +document is the only Narratio documentation that describes the proposed +`extract` stage, Notarius configuration, or `narratio.extraction.*` sources. ## Purpose Add a first-class Narratio `extract` stage that runs Notarius against the -session's final trimmed transcript, validates and collects the resulting -structured D&D artifacts, and registers those artifacts for later use by the -`analyze` and `publish` stages. +session's final trimmed transcript and makes validated structured artifacts +available to later analysis and publish work. -This feature should integrate Notarius through Narratio's existing stage, -adapter, manifest, workspace, and artifact-catalog boundaries. It must not turn -Narratio into a generic workflow engine or a second configuration language for -Notarius pipelines. +Notarius remains responsible for its D&D extraction pipeline, prompts, +references, LLM profiles, retries, validation, normalization, and published +schemas. Narratio owns invocation, required-output policy, safe bundle +ingestion, artifact identity, manifest state, resume, and downstream +availability. + +The integration must preserve Narratio's explicit stage model. It must not +turn Narratio into a generic workflow engine or reproduce Notarius's +configuration language. ## User Outcome -An operator can enable one configured Notarius pipeline for a Narratio -campaign. During a normal run, Narratio will: +An operator can enable one Notarius pipeline for a Narratio campaign. During a +normal run, Narratio will: -1. finish producing the session transcript tiers; -2. invoke Notarius once with the final trimmed Seriatim JSON transcript; -3. collect and validate the configured structured artifact lanes; -4. record their exact files and provenance in the Narratio manifest; and -5. make those artifacts selectable as inputs to Scriptorium artifacts in the - later `analyze` stage. +1. produce the final trimmed Seriatim JSON transcript; +2. invoke Notarius once with that transcript; +3. discover the exact run bundle through Notarius's machine-readable receipt; +4. validate every output contract Narratio is configured to require; +5. promote the complete validated bundle into immutable session storage; +6. record exact artifact paths, checksums, contracts, and external provenance; +7. make configured lanes available as `narratio.extraction.` sources; and +8. allow individual Scriptorium artifacts and publish rules to select those + sources explicitly. -The maintained D&D example should demonstrate all ten lanes emitted by -Notarius's complete `dnd-session` pipeline. +The maintained complete D&D example will require all ten lanes published by +Notarius's `dnd-session` pipeline. Ordinary deployments may configure a +narrower required set. -## Target Stage Architecture +## Chosen Architecture + +### First-Class Stage + +Extraction is an independently observable and resumable pipeline stage. It is +not part of `analyze`, a Scriptorium artifact producer, or an implicit external +preprocessing requirement. + +This boundary is required because Notarius work is expensive, produces +multiple durable outputs, has its own compatibility and diagnostic contracts, +and may be consumed by both `analyze` and `publish`. ### Canonical Order The canonical stage order becomes: ```text -prepare -> transcribe -> merge -> polish -> normalize -> trim -> render - -> extract -> analyze -> publish -> notify +prepare -> transcribe -> merge -> polish -> normalize -> trim + -> extract -> render -> analyze -> publish -> notify ``` -`extract` is deliberately after all transcript-producing stages and before -analysis. Its source document is the manifest-resolved -`narratio.transcript.final_trimmed` artifact, normally -`transcripts/final.trimmed.json`. It does not consume rendered Markdown. +`extract` consumes `narratio.transcript.final_trimmed`, produced by `trim`. It +does not consume rendered Markdown. -Adding the stage must update full-plan construction, explicit stage selection, -downstream invalidation, prerequisite checks, resume behavior, run manifests, -CLI stage validation and help, and every canonical-stage inventory. Forcing an -upstream transcript stage must stale a previously successful `extract` stage -and its downstream stages. Forcing `extract` must stale `analyze`, `publish`, -and `notify` according to existing rules. +Narratio currently invalidates succeeded stages by canonical downstream order. +Placing `extract` before `render` means forcing extraction may rerun the less +expensive deterministic render stage, while forcing render does not rerun the +more expensive Notarius pipeline. This is preferable to placing extraction +after render and does not require dependency-aware scheduling or a DAG. + +Adding the stage must update every canonical-stage inventory, full-plan and +single-stage selection, prerequisite validation, downstream invalidation, +resume behavior, run manifests, CLI validation and help, and focused tests. ### Stage Boundary -The stage owns Narratio policy and state transitions: +The stage owns Narratio policy: -- resolve the final trimmed transcript through the runtime artifact catalog; -- build a Narratio-level Notarius request from validated configuration and - run-local paths; -- call a narrow Notarius adapter; -- apply the configured required-output policy; -- materialize the validated bundle into its canonical session location; -- return manifest-ready artifact references and bounded metadata; and -- fail without marking the stage successful when any required contract or - materialization step fails. +- resolve the final trimmed transcript through the manifest-aware artifact + resolver; +- allocate run-local receipt, log, and output-root paths; +- build a transport-neutral Notarius request from resolved configuration; +- call the configured Notarius adapter once; +- enforce Narratio's configured required-output policy; +- validate selected payloads as regular, non-empty, syntactically valid JSON; +- promote the validated bundle to immutable session artifact storage; +- return explicit source IDs, checksums, contracts, and external provenance; + and +- report an explicit skipped outcome when Notarius is disabled. -The stage must not construct subprocess arguments, infer Notarius output -filenames, parse provider logs, or decode individual D&D payload bodies. +The stage must not construct subprocess arguments, guess Notarius filenames, +parse interactive output, decode D&D payload structures, or reproduce +Notarius pipeline configuration. ### Adapter Boundary -Add a dedicated Notarius adapter package with a small interface, production -subprocess implementation, and test fake. Its request should contain only the -resolved Notarius binary, configuration path, pipeline ID, transcript path, -output root, working directory, timeout, and process-log destinations needed -for one run. +Add `internal/adapters/notarius` with a narrow runner interface, production +subprocess implementation, and small fake. + +The request contains only: + +- resolved executable path or name; +- absolute Notarius configuration path; +- pipeline ID; +- absolute transcript path; +- absolute output root; +- working directory; +- timeout; and +- stdout receipt and stderr log destinations. The adapter owns: -- optional `notarius config validate` preflight for the configured pipeline; - exact `notarius run ... --json` argument construction; - stdout and stderr separation; -- context cancellation and timeout propagation through Narratio's shared - subprocess boundary; +- context cancellation and timeout through Narratio's shared subprocess + boundary; +- environment inheritance; - exit-status handling; -- decoding the `notarius.run-result.v1` success receipt; -- receipt and index path-confinement checks; -- decoding `index.json` and resolving descriptor paths safely beneath the - reported output directory; and -- returning a transport-neutral result containing the bundle location, - receipt summary, lane descriptors, pipeline-wide descriptors, warnings and - rejection locations, and diagnostic log paths. +- bounded receipt loading after exit status zero; +- tolerant decoding of supported `notarius.run-result.v1` documents; +- validation of required receipt fields; +- confinement of the receipt's absolute `output_directory` beneath the + absolute output root Narratio supplied for this invocation; +- confinement of `index_file` beneath the receipt's absolute + `output_directory`; +- tolerant decoding of the supported `index.json` contract; +- confinement of every index descriptor path beneath the bundle root; and +- bounded tolerant decoding of `rejected.json` and `warnings.json` into + transport-neutral summaries without reading lane payload bodies; summaries + persisted by Narratio contain structured scope, lane, and reason fields but + not unbounded free-form external messages; and +- returning transport-neutral receipt, descriptor, diagnostic, and bundle + information. -Only exit status zero permits receipt decoding. Receipt, index, or descriptor -paths that are absolute where a logical relative path is required, or that -escape their owning root, are integration failures. Unknown fields in a -supported receipt or index schema should be tolerated. Unsupported schema -versions and incompatible descriptor metadata should fail clearly. +Only exit status zero permits receipt decoding. Unsupported schema versions, +malformed documents, missing required fields, absolute logical paths, path +escapes, symlinks at consumed paths, and incompatible structural metadata are +integration failures. -The adapter must not write Narratio manifests, choose required lanes, decide -analysis inputs, or contain D&D domain logic. +The production runner will not execute `notarius config validate` before every +session. Notarius run-time validation remains authoritative, while operators +may use the separate validation command as deployment preflight. A second +automatic subprocess can be added later only if operational evidence warrants +it. + +The adapter does not write Narratio manifests, decide required lanes, choose +analysis inputs, or interpret D&D payloads. + +## External Contract Baseline + +The integration consumes the contracts documented by Notarius in +`../notarius/docs/consumers/dnd-pipeline.md` and its linked canonical +integration documents. + +The initial compatibility baseline is: + +- `notarius run --config ... --input ... --output-dir ... --json`; +- successful receipt schema `notarius.run-result.v1`; +- an absolute receipt `output_directory`; +- logical `index_file` discovery beneath that directory; +- the production JSON `index.json` descriptor model; and +- the exact media type and schema identity configured for each required lane. + +Compatibility is decided from these published contracts, not by parsing +`notarius --version`. Unknown fields in supported receipt and index versions +are tolerated. Unsupported versions or incompatible required descriptors fail +before any artifact becomes current in Narratio. ## Configuration Contract -Add a strict optional `pipeline.notarius` configuration section. Omission or -`enabled: false` keeps the current workflow usable and causes `extract` to -self-skip without outputs. +Add a strict optional `pipeline.notarius` section: -The section should provide: - -- `enabled`: explicit opt-in; -- `binary`: Notarius executable, defaulting to `notarius`; -- `config_path`: required when enabled; -- `pipeline_id`: required when enabled; -- `timeout`: a positive stage timeout with a documented default; -- `working_directory`: optional explicit subprocess working directory, - defaulting to the directory containing `config_path`; and -- an `outputs` map defining the Notarius lane artifacts Narratio promises to - collect. - -Each output-map key is a stable Narratio extraction key. Each value must define: - -- the exact Notarius `lane_id`; -- the expected `media_type`; -- the expected `schema_id`; -- the expected `schema_version`; and -- optionally an expected `module_key` when the operator needs to constrain the - producing module as part of compatibility. - -Narratio derives the downstream source ID -`narratio.extraction.` from the map key. Keys and lane IDs must be -non-empty, unique after normalization, path-safe under the existing artifact -policy, and collision-free with built-in and configured artifact identities. -Every configured output is required: a successful Notarius process that omits -one, rejects it, or reports incompatible descriptor metadata fails the -`extract` stage. - -This explicit map keeps Narratio's consumer contract stable when a Notarius -lane ID or schema changes and avoids hard-coding the current D&D family into a -generic adapter. It also replaces a separate `required_lanes` list, which would -duplicate configuration. - -Narratio should not reproduce Notarius lane selection, references, LLM -profiles, model settings, retries, concurrency, or prompt configuration. Those -remain in the referenced Notarius configuration. Narratio should not expose a -runtime lane-selection flag for `extract`; one stage invocation runs the -configured Notarius pipeline as a unit. - -All configured paths should become absolute during Narratio configuration -resolution. The deterministic default working directory allows a Notarius -profile path relative to that directory, but operator documentation should -still recommend absolute deployment paths where practical. Notarius reference -paths continue to follow Notarius's own configuration-relative rules. - -## Output And Artifact Model - -### Canonical Bundle - -Run Notarius against a run-local output root. After all configured descriptors -are validated, materialize the contents of the exact run-specific Notarius -bundle into a fixed canonical session directory: - -```text -artifacts/notarius/ +```yaml +notarius: + enabled: true + binary: notarius + config_path: /absolute/path/to/notarius.yml + pipeline_id: dnd-session + timeout: 3h + working_directory: /absolute/path/to/deployment + outputs: + npc_registry: + lane_id: npc-registry + media_type: application/json + schema_id: + schema_version: ``` -Preserve its relative layout, including `index.json`, `manifest.json`, -`rejected.json`, `warnings.json`, `lanes/`, and any indexed `chunk-map.json` or -`evidence-context.json`. Materialize the complete directory as one narrow, -transactional replacement so a failed or interrupted rerun cannot mix files -from different Notarius runs. +Fields: -The raw subprocess receipt and stderr log belong in the run-local `extract` -report and log directories. The raw receipt identifies the original run-local -Notarius bundle and must not be rewritten to pretend that the canonical copy -was its original `output_directory`. Narratio's manifest is the durable ledger -for the canonical materialized paths. +- `enabled` is an explicit opt-in and defaults to false; +- `binary` defaults to `notarius`; +- `config_path` is required when enabled; +- `pipeline_id` is required when enabled; +- `timeout` defaults to `3h` and must be positive; +- `working_directory` is optional and defaults to the directory containing + `config_path`; and +- `outputs` maps stable Narratio extraction keys to required Notarius lane + contracts. -### Registered Artifact Sources +All configured paths become absolute during configuration resolution. Notarius +reference paths continue to follow Notarius's configuration-relative rules, +while PromptKit profile paths remain relative to the chosen process working +directory where Notarius permits that behavior. -For each configured output, locate the lane through the canonical copy of -`index.json` and record a manifest artifact with: +Each output entry contains: -- source ID `narratio.extraction.`; -- canonical lane-file path discovered from the index; +- exact `lane_id`; +- exact `media_type`; +- exact `schema_id`; +- exact `schema_version`; and +- optional `module_key`. + +The map key produces `narratio.extraction.`. Keys use Narratio's existing +path-safe configured-artifact key grammar. Keys, normalized source IDs, and +lane IDs must be non-empty and unique. Extraction source IDs must not collide +with built-ins or configured Scriptorium sources. + +Every configured output is required for extraction-stage success. Operators +who need only a subset configure only that subset. Narratio does not add a +second `required_lanes` list or a runtime lane-selection flag. + +Narratio does not configure Notarius lane topology, references, prompts, LLM +profiles, model settings, concurrency, retry behavior, or session IDs. One +stage execution runs the configured Notarius pipeline as a unit. Narratio does +not pass Notarius's `--session-id` override. + +## Stage Outcomes And Resume + +### Explicit Self-Skip + +Extend the stage-result contract with an explicit disposition whose zero value +remains successful for backward compatibility. A stage may return: + +- succeeded; or +- skipped with a stable reason. + +When Notarius is absent or disabled, `extract` returns skipped with reason +`notarius_disabled`, no outputs, and bounded metadata. The runner records a +skipped session-stage and run-stage outcome rather than a successful empty +stage. Skipped stages are reconsidered on later invocations, so subsequently +enabling Notarius causes extraction to run without requiring force. + +A genuine self-skip clears any older outputs for that stage before persisting +the new skipped state. Downstream consumers cannot resolve artifacts retained +from an earlier extraction after the stage is disabled. + +### Resume Validation + +Add a small optional resume-validation interface implemented by `extract`. +Before skipping an already-succeeded extract stage, the runner asks it whether +the recorded result remains resumable. + +The resume validator confirms: + +- Notarius is still enabled; +- recorded configuration identity matches the current config path, pipeline + ID, and output-contract map; +- the immutable bundle and canonical index still exist; +- every configured source is present in the succeeded extract record; +- lane files remain confined regular files; +- stored checksums still match; and +- stored descriptor contracts and external provenance remain compatible. + +An ordinary contract mismatch returns a non-resumable decision with a bounded +reason. The runner marks the stage stale and executes it. An environmental +error that prevents making a safe decision returns an error and stops the run. + +Narratio does not recursively interpret Notarius configuration, PromptKit +profiles, or reference files. Changes to those external inputs are therefore +not automatically detectable. Operator documentation must require +`--force extract` after changing them. + +The configuration fingerprint covers the resolved binary, config path, +pipeline ID, timeout, working directory, and the deterministically sorted +output-contract map. It identifies Narratio's invocation contract, not the +transitive content of files owned by Notarius. + +### Force And Invalidation + +- Forcing `trim` or an earlier stage stales succeeded `extract` and all later + stages. +- Forcing `extract` stales succeeded `render`, `analyze`, `publish`, and + `notify` under the existing canonical-order rule. +- Forcing `render` does not stale `extract` because extraction precedes it. +- A failed, skipped, stale, or interrupted extract stage never supplies current + extraction sources. + +## Bundle Storage And Commit + +### Run-Local Execution + +Notarius runs against a run-local output root beneath the Narratio run's +`extract` directory. Receipt bytes and stderr remain run-local diagnostics. +Failed and malformed bundles remain outside durable artifact storage for +inspection and never become current merely because files exist. + +### Immutable Durable Bundles + +After complete validation, promote the exact Notarius bundle into: + +```text +artifacts/notarius// +``` + +The destination is unique and must not already exist. Promotion uses a sibling +temporary directory on the same filesystem, recursively copies only regular +files and directories, rejects symlinks and special files, preserves relative +layout, and renames the completed temporary tree into place. + +The promoted tree preserves `index.json`, `manifest.json`, `rejected.json`, +`warnings.json`, `lanes/`, and any emitted pipeline-wide artifacts such as +`chunk-map.json` and `evidence-context.json`. Unknown regular files may be +preserved because the complete bundle is provenance, but no unknown file is +registered as a stable Narratio source. + +There is no mutable filesystem `current` directory or symlink. The atomically +saved Narratio session manifest selects the current successful bundle. Older +successful bundles remain immutable until explicit cleanup policy removes +them. + +The raw Notarius receipt is retained without rewriting its original +`output_directory`. Narratio's manifest records promoted durable paths. + +## Artifact And Manifest Model + +### Explicit Source Identity + +Extend `artifacts.Ref` with an optional explicit `SourceID`. The application +runner prefers that value and retains existing stage-specific inference only +as a backward-compatible fallback. `extract` must not require another +stage-name special case in output mapping. + +Add neutral optional artifact metadata models under `internal/artifactmodel`: + +- contract metadata: media type, schema ID, schema version, and optional + module key; and +- external provenance: system, external run ID, pipeline ID, and external + artifact ID. + +Both `artifacts.Ref` and `manifest.ArtifactRecord` carry these nested models. +Existing manifests remain readable because the fields are optional and use +`omitempty` encoding. + +For a Notarius lane, external provenance uses: + +- system `notarius`; +- receipt run ID; +- receipt pipeline ID; and +- lane ID as the external artifact ID. + +### Registered Sources + +For each configured output, `extract` records one artifact with: + +- source ID `narratio.extraction.`; +- durable lane path discovered through the promoted canonical index; - producer stage and Narratio run ID; -- checksum; -- Notarius lane ID; and -- descriptor media type, schema identity/version, and module key when present. +- SHA-256 checksum; +- configured and observed contract metadata; and +- Notarius external provenance. -If the current manifest model cannot carry descriptor compatibility metadata, -extend its artifact metadata in a backward-tolerant way rather than encoding -that information in filenames or source IDs. +The canonical promoted `index.json` is also a stage output with kind +`notarius_index`, but it is not a selectable extraction source. Extract-stage +metadata records the durable bundle root, receipt path, Notarius validation +status, counts, warnings/rejections paths and summaries, the producing Narratio +run ID, and the normalized configured-contract fingerprint used by resume +validation. -Also record the canonical Notarius index as a stage output or stage metadata so -operators can discover the complete bundle, including non-lane artifacts. The -configured lane sources are the stable interface for analysis; the index and -bundle remain the provenance and inspection interface. +The producing Narratio run ID belongs to the successful extract result. It is +not compared with the session manifest's top-level `run_id`, which advances on +later invocations even when extraction is validly resumed. Resume and catalog +checks instead require the extract outputs, immutable bundle path, and stored +extract-stage producer identity to agree with one another. -## Analysis And Publish Integration +## Artifact Catalog And Resolution -Extend the runtime artifact catalog and configured Scriptorium input validation -so an enabled analysis artifact can declare, for example: +Add extraction as a first-class artifact-policy and runtime-catalog family: + +- source kind `extraction`; +- canonical prefix `narratio.extraction.`; +- registration from `pipeline.notarius.outputs`; and +- manifest-backed availability from the current successful `extract` record. + +Extraction availability is never inferred by scanning +`artifacts/notarius/`. A source is available only when: + +- it is declared in current configuration; +- the session manifest records `extract` as succeeded and not stale; +- the exact matching source output is present; +- its durable path is confined and valid; +- its checksum matches; and +- its recorded contract and external provenance are compatible. + +The catalog should expose one shared registration and hydration path used by +both `analyze` and `publish`. Avoid parallel extract-specific resolution logic +inside each stage. + +## Analyze Integration + +An enabled Scriptorium artifact may select an extraction source through the +existing input contract: ```yaml inputs: - npc_registry: + npcs: source: narratio.extraction.npc_registry + required: true ``` -Resolution must remain manifest-first and verify that the recorded artifact -was produced by a successful current `extract` stage. A required extraction -source that is unavailable must fail analysis with guidance to configure or -rerun `extract`; an optional source may be omitted according to the existing -Scriptorium input contract. +Required missing extraction inputs fail with guidance to enable/configure or +rerun `extract`. Optional missing inputs follow the existing Scriptorium input +contract. -Publish source resolution should accept configured -`narratio.extraction.` sources through the same artifact catalog so -operators may publish selected structured artifacts without manually copying -paths. The existing `--artifacts` flag remains scoped to Scriptorium artifact -selection and must not partially execute the Notarius pipeline. +Narratio never injects every extraction output into every analysis. Each +Scriptorium artifact chooses the smallest useful set. This limits context +size, cost, and the risk of treating derived claims as transcript authority. +Analysis prompts should continue to treat the transcript as authoritative and +Notarius artifacts as structured, cited, derived evidence. -No current-session analysis artifact should consume an incidental file from a -failed, stale, skipped, or superseded extraction run. +The existing `--artifacts` selection remains scoped to Scriptorium artifacts. +It does not select Notarius lanes or partially run the Notarius pipeline. -## Failure, Skip, Resume, And Diagnostics +## Publish Integration -- Missing or invalid enabled Notarius configuration fails configuration - validation before stage execution where statically discoverable. -- A disabled or absent Notarius configuration makes `extract` skip with clear - stage metadata and no new outputs. -- A missing or invalid final trimmed transcript fails `extract` before starting - Notarius. -- Preflight failure, nonzero Notarius exit, cancellation, timeout, malformed or - unsupported receipt/index data, unsafe paths, incompatible descriptors, - rejected required outputs, or missing configured lanes fails the entire - stage. -- Process success does not override Narratio's required-output policy. -- A failed run retains bounded run-local receipt bytes, stderr, and the - unpublished Notarius bundle for diagnosis, subject to Narratio's existing - sensitive-data and cleanup policies. -- The canonical bundle and manifest artifacts are updated only after complete - validation and materialization. -- Resume skips a succeeded, non-stale `extract` stage only when its - manifest-recorded canonical index and configured lane outputs still validate. -- Force and staleness behavior follows the ordinary stage contract; it must not - depend on merely finding `artifacts/notarius/` on disk. +Publish source validation and resolution accept configured +`narratio.extraction.` sources through the shared runtime artifact +catalog. Operators may publish individual structured lanes without manually +copying files. -Transcripts, Notarius outputs, evidence context, manifests, receipts, and logs -are private campaign material. Subprocess arguments and manifest metadata must -not contain secrets. Credentials remain in the environment or in mechanisms -owned by Notarius and PromptKit. +The Notarius bundle is not automatically published wholesale. Bundle files or +lanes are published only through explicit configured publish rules. Existing +publish locking, destination safety, commit ordering, and required/unselected +artifact behavior remain unchanged. -## Maintained D&D Example +## Failure And Diagnostic Semantics -Add or update a Narratio example that enables Notarius's complete -`dnd-session` pipeline and maps these ten required lanes to stable extraction -keys: +The stage fails before invoking Notarius when enabled configuration or the +final trimmed transcript is invalid. -| Output key | Notarius lane ID | +The stage fails after invocation for: + +- cancellation or timeout; +- nonzero process exit; +- malformed, oversized, or unsupported receipt data; +- malformed or unsupported index data; +- unsafe receipt, descriptor, or filesystem paths; +- symlinks or special files in the promoted bundle; +- mismatched receipt pipeline identity; +- missing configured lanes; +- duplicate lane descriptors; +- rejected required lanes; +- incompatible media type, schema identity/version, or module key; +- empty or syntactically invalid required JSON payloads; +- checksum or promotion failure; or +- manifest persistence failure. + +Process success alone does not establish consumer success. Notarius may exit +zero while omitting or rejecting a lane, and Narratio's configured required +set remains authoritative. + +Failure retains bounded receipt and stderr diagnostics plus the run-local +bundle where available. Durable artifact storage and extraction source records +are updated only after complete validation and promotion. A promoted unique +bundle whose later manifest save fails is unreferenced and may be reclaimed by +explicit cleanup; it is never inferred as current. + +## Security And Privacy + +Transcripts, Notarius lanes, evidence context, manifests, receipts, warnings, +rejections, debug data, and logs are private campaign material. + +- Secrets do not appear in command arguments, generated configuration, + artifact metadata, logs, examples, or documentation. +- Credentials continue to enter through Notarius and PromptKit's documented + environment or secret mechanisms. +- Receipt and index paths are untrusted external input until confined. +- Recursive promotion never follows symlinks or copies special files. +- Diagnostics remain bounded and do not echo payload bodies. +- Automatic cleanup follows Narratio's existing post-publish gates and path + safety rules; it does not silently remove immutable extraction bundles + outside an explicit covered policy. + +## Maintained Complete D&D Example + +Add a maintained example that enables Notarius's complete `dnd-session` +pipeline and maps these required lanes: + +| Extraction key | Notarius lane ID | | --- | --- | | `item_registry` | `item-registry` | | `npc_registry` | `npc-registry` | @@ -274,106 +505,100 @@ keys: | `location_occurrences` | `location-occurrences` | | `enemy_events` | `enemy-events` | -The example must include each lane's current media type and schema identity -from Notarius's published contracts. It should also demonstrate at least one -Scriptorium analysis artifact consuming one or more -`narratio.extraction.*` sources. The example must use placeholders and relative -paths suitable for the example tree, contain no credentials, and pass the -repository's configuration validation tests. +The example obtains exact media types and schema identities from Notarius's +published contracts at implementation time. It demonstrates at least one +Scriptorium artifact consuming a small, purpose-specific subset of extraction +sources. It uses only placeholders and repository-relative example paths, +contains no credentials, and passes maintained example validation. -## Compatibility Policy +## Documentation Deliverables -The initial integration baseline is the public subprocess contract available -in Notarius v0.3.0: +When implementation lands, update current-behavior documentation in the same +change: -- successful JSON receipt schema `notarius.run-result.v1`; -- production JSON bundle discovery through `index.json`; and -- the schema IDs and versions explicitly configured for required lanes. - -Runtime compatibility should be decided from those published contracts, not -from textual parsing of `notarius --version`. New optional receipt or index -fields must not break Narratio. An unsupported receipt version or lane schema -must fail before the artifact is registered for analysis. - -## Documentation Deliverables When Implemented - -Update current-behavior documentation in the same change that implements the -feature: - -- add `docs/integrations/notarius.md` for the external CLI, receipt, bundle, - and adapter contract, linking to Notarius's canonical documentation; -- add `docs/internal/stage-extract.md` for stage inputs, outputs, collaborators, - state transitions, failures, and focused tests; -- update `docs/internal/adapters.md`, `docs/internal/artifacts.md`, - `docs/internal/manifest.md`, and the internal stage inventory; -- update `docs/policy/architecture.md` to list Notarius among isolated external - systems and preserve the adapter/stage boundary; +- add `docs/integrations/notarius.md` for the consumed CLI, receipt, index, and + compatibility contract, linking to Notarius's canonical documentation; +- add `docs/internal/stage-extract.md` for stage flow, collaborators, state, + failures, resume validation, and focused tests; +- update `docs/internal/overview.md`, `docs/internal/adapters.md`, + `docs/internal/artifacts.md`, `docs/internal/manifest.md`, and + `docs/internal/workspace.md` within their canonical scopes; +- update `docs/policy/architecture.md` for the Notarius boundary and explicit + skipped/resume-validation contracts; - update `docs/config.md`, `docs/cli.md`, `docs/operations.md`, - `docs/troubleshooting.md`, `README.md`, and maintained examples only to the - extent their canonical scopes require; and -- update `docs/development.md` only to the extent its canonical contributor - routing scope requires. + `docs/troubleshooting.md`, `README.md`, and maintained examples only within + their canonical scopes; and +- update `docs/development.md` only if its contributor routing changes. -Outside this roadmap, do not describe `extract`, Notarius configuration, or -`narratio.extraction.*` sources as implemented until the code exists. +Outside this roadmap, do not describe the feature as implemented until its +code and documentation are complete. -## Testing And Validation Expectations +## Testing Expectations -Implementation should provide focused tests for: +Tests should protect contracts and meaningful risks rather than private helper +structure. The implementation plan assigns detailed ownership, with coverage +for: -- strict configuration decoding, defaults, required fields, path resolution, - output-map validation, normalized-key collisions, and example loading; -- exact stage order, selection, downstream staleness, resume, force, and - prerequisite behavior; -- adapter command construction, deterministic working directory, environment - inheritance, stdout/stderr separation, cancellation, timeout, and nonzero - exits; -- supported and unsupported receipt versions, unknown optional fields, - malformed receipts, index decoding, and path escapes at every boundary; -- descriptor lookup by lane ID rather than filename, expected metadata checks, - missing/rejected configured lanes, and tolerated unconfigured lanes; -- run-local execution, transactional canonical-bundle replacement, checksums, - failed-run preservation, and manifest recording; -- artifact-catalog resolution from `narratio.extraction.*` into analysis and - publish, including required, optional, missing, stale, and skipped cases; and -- end-to-end stage execution with a fake Notarius adapter, without live LLM or - external subprocess requirements in the ordinary test suite. +- strict configuration, defaults, normalization, cross-source validation, and + maintained examples; +- stage result dispositions and clearing of self-skipped outputs; +- backward-compatible artifact metadata serialization; +- exact adapter arguments, streams, cancellation, timeout, exit behavior, + receipt/index compatibility, and every path-confinement boundary; +- recursive promotion safety, atomic visibility, cleanup on failure, symlink + rejection, and immutable destination behavior; +- required-lane policy, descriptor compatibility, JSON syntax, checksums, and + provenance; +- canonical order, single-stage selection, force, staleness, resume + validation, and enable-after-skip behavior; +- manifest-backed extraction catalog resolution for required, optional, + missing, stale, skipped, incompatible, and tampered artifacts; +- analyze and publish integration without automatic lane injection; and +- representative assembled execution with a fake Notarius runner and no live + LLM, credentials, or external subprocess in the ordinary test suite. -Run the repository-wide Go tests, vet, build, and maintained example validation -after focused tests pass. +Repository-wide tests, vet, build, and maintained-example validation are +required after focused tests pass. ## Acceptance Criteria -- `extract` is a first-class transactional stage between `render` and - `analyze` everywhere Narratio models stage order or state. -- Narratio invokes Notarius only through a narrow, tested adapter. -- The stage consumes the manifest-resolved final trimmed Seriatim transcript. -- The Notarius configuration remains owned by Notarius; Narratio configures - only invocation and its downstream consumer contract. -- Every configured output is discovered through the receipt and `index.json`, - contract-checked, materialized transactionally, and recorded with a stable - `narratio.extraction.*` source ID. -- The complete D&D example maps all ten current lanes and passes strict config - validation. -- Analysis can consume extraction sources through the existing artifact input - model, and publish can select them through the artifact catalog. -- Failed, partial, rejected, unsafe, stale, or incompatible output never becomes - a current analysis input. -- Resume and force behavior remains manifest-driven. -- Documentation accurately describes the implemented stage, adapter, - configuration, operations, and compatibility boundary without duplicating - Notarius's canonical schemas. +- `extract` is a first-class stage between `trim` and `render` everywhere + Narratio models stage order and lifecycle. +- Narratio invokes Notarius only through a narrow tested adapter. +- Extraction consumes the manifest-resolved final trimmed Seriatim JSON. +- Disabled extraction is recorded as skipped and runs normally if later + enabled. +- Successful resume requires valid manifest-recorded immutable outputs rather + than filesystem presence alone. +- The complete bundle is promoted to a unique immutable directory without + following symlinks or exposing a partial destination. +- Every configured lane is discovered by lane ID, contract-checked, checksummed, + and recorded with explicit source identity and external provenance. +- Analysis and publish resolve extraction sources only from a current + successful extract manifest record. +- Failed, skipped, stale, partial, rejected, unsafe, incompatible, or tampered + output never becomes a current input. +- The complete D&D example maps all ten current lanes and demonstrates curated + analysis inputs. +- Tests remain deterministic, offline, and independent of real Notarius, + PromptKit, LLM providers, and credentials. +- Current-behavior documentation is updated only as implementation becomes + complete. ## Non-Goals -- Reimplementing Notarius extraction, prompts, schemas, references, retries, - profiles, or lane orchestration in Narratio. -- Allowing one Narratio run to invoke arbitrary extractor programs or multiple - Notarius pipelines. -- Making `extract` a configurable DAG or folding it into the Scriptorium - `analyze` stage. -- Partially selecting Notarius lanes through Narratio's `--artifacts` flag. -- Decoding D&D payload bodies in the generic Notarius adapter. -- Supporting previous-session extraction artifacts in the initial feature. +- Reimplementing Notarius extraction, configuration, schemas, prompts, + references, retries, profiles, validation, or lane orchestration. +- Supporting arbitrary extractor programs or multiple Notarius pipelines in + one Narratio run. +- Turning canonical stage execution into a DAG or generic workflow engine. +- Folding extraction into `analyze` or Scriptorium. +- Automatically injecting all structured outputs into every prompt. +- Selecting Notarius lanes through Narratio's `--artifacts` flag. +- Decoding D&D payload bodies in the generic adapter or stage. +- Automatically detecting changes throughout Notarius's referenced config, + PromptKit profile, and campaign-reference closure. +- Supporting previous-session extraction sources in the initial feature. +- Automatically publishing the complete Notarius bundle. - Requiring live Notarius, PromptKit, an LLM provider, or external services in - the ordinary unit test suite. + the ordinary test suite. diff --git a/internal/app/runner.go b/internal/app/runner.go index bfe7bba..d61e995 100644 --- a/internal/app/runner.go +++ b/internal/app/runner.go @@ -195,6 +195,9 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "running", "path", manifestPath) result, err := s.Run(ctx, stageEnv, m) + if err == nil { + err = validateStageResult(result) + } if err != nil { failedAt := nowUTC() m.MarkStageFailed(s.Name(), failedAt, err.Error()) @@ -209,6 +212,25 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage env.Logger.Info("stage failed", "stage", s.Name(), "error", err) return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err) } + if result != nil && result.Disposition == stage.StageDispositionSkipped { + skipped = append(skipped, s.Name()) + skippedAt := nowUTC() + m.MarkStageSkipped(s.Name(), skippedAt, result.SkipReason) + clearStageResultDetails(m.Stages[s.Name()]) + applyStageResultToManifest(m, s.Name(), result) + if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil { + return nil, fmt.Errorf("save manifest after self-skip %q: %w", s.Name(), err) + } + runManifest.MarkStageSkipped(s.Name(), skippedAt, result.SkipReason) + applyStageResultToRunManifest(runManifest, s.Name(), result) + syncRunManifestIdentityFromSession(m, runManifest) + if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil { + return nil, fmt.Errorf("save run manifest after self-skip %q: %w", s.Name(), err) + } + env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "skipped", "path", manifestPath) + env.Logger.Info("stage skipped", "stage", s.Name(), "reason", result.SkipReason) + continue + } outputs := mapResultOutputs(s.Name(), result, runID) succeededAt := nowUTC() @@ -407,26 +429,69 @@ func mapResultOutputs(stageName string, result *stage.StageResult, runID string) localPath = ref.RelativePath } kind := ref.Kind - sourceID := "" - if stageName == "analyze" { - sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind) - kind = "scriptorium_artifact" - } else { - sourceID = sourceIDForOutputKind(kind) + sourceID := strings.TrimSpace(ref.SourceID) + if sourceID == "" { + if stageName == "analyze" { + sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind) + kind = "scriptorium_artifact" + } else { + sourceID = sourceIDForOutputKind(kind) + } } out = append(out, manifest.ArtifactRecord{ - Kind: kind, - SourceID: sourceID, - LocalPath: localPath, - ProducerRunID: runID, - RemoteKey: ref.RemoteKey, - Checksum: ref.Checksum, + Kind: kind, + SourceID: sourceID, + LocalPath: localPath, + Contract: cloneContractMetadata(ref.Contract), + ExternalProvenance: cloneExternalProvenance(ref.ExternalProvenance), + ProducerRunID: runID, + RemoteKey: ref.RemoteKey, + Checksum: ref.Checksum, }) } return out } +func validateStageResult(result *stage.StageResult) error { + if result == nil { + return nil + } + switch result.Disposition { + case stage.StageDispositionSucceeded: + if strings.TrimSpace(result.SkipReason) != "" { + return fmt.Errorf("successful result contains a skip reason") + } + return nil + case stage.StageDispositionSkipped: + if strings.TrimSpace(result.SkipReason) == "" { + return fmt.Errorf("skipped result requires a skip reason") + } + if len(result.Outputs) != 0 { + return fmt.Errorf("skipped result contains %d output(s)", len(result.Outputs)) + } + return nil + default: + return fmt.Errorf("unsupported stage result disposition %q", result.Disposition) + } +} + +func cloneContractMetadata(value *artifactmodel.ContractMetadata) *artifactmodel.ContractMetadata { + if value == nil { + return nil + } + cloned := *value + return &cloned +} + +func cloneExternalProvenance(value *artifactmodel.ExternalProvenance) *artifactmodel.ExternalProvenance { + if value == nil { + return nil + } + cloned := *value + return &cloned +} + func sourceIDForOutputKind(kind string) string { trimmed := strings.TrimSpace(kind) if trimmed == "" { @@ -462,6 +527,15 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result * } } +func clearStageResultDetails(sr *manifest.StageRecord) { + if sr == nil { + return + } + sr.Logs = nil + sr.GeneratedConfigs = nil + sr.Metadata = nil +} + func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest, runID string) (bool, error) { if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil { return false, nil diff --git a/internal/app/runner_test.go b/internal/app/runner_test.go index 4f6dacb..98773c6 100644 --- a/internal/app/runner_test.go +++ b/internal/app/runner_test.go @@ -16,6 +16,7 @@ import ( "gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim" "gitea.maximumdirect.net/eric/narratio/internal/adapters/storage" "gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx" + "gitea.maximumdirect.net/eric/narratio/internal/artifactmodel" "gitea.maximumdirect.net/eric/narratio/internal/artifacts" "gitea.maximumdirect.net/eric/narratio/internal/config" "gitea.maximumdirect.net/eric/narratio/internal/manifest" @@ -38,6 +39,25 @@ type countingStage struct { runs *int } +type resultStage struct { + name string + result *stage.StageResult + runs *int + order *[]string +} + +func (s resultStage) Name() string { return s.name } +func (s resultStage) Declares() stage.IODecl { return stage.IODecl{} } +func (s resultStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) { + if s.runs != nil { + *s.runs = *s.runs + 1 + } + if s.order != nil { + *s.order = append(*s.order, s.name) + } + return s.result, nil +} + func (s countingStage) Name() string { return s.name } func (s countingStage) Declares() stage.IODecl { return stage.IODecl{} } func (s countingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) { @@ -199,6 +219,61 @@ func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T) } } +func TestMapResultOutputsPrefersExplicitSourceAndCopiesMetadata(t *testing.T) { + contract := &artifactmodel.ContractMetadata{ + MediaType: "application/json", + SchemaID: "notarius.dnd.npc_registry", + SchemaVersion: "v1", + } + provenance := &artifactmodel.ExternalProvenance{ + System: "notarius", + RunID: "external-run", + PipelineID: "dnd-session", + ArtifactID: "npc-registry", + } + result := &stage.StageResult{Outputs: []artifacts.Ref{{ + Kind: "structured_data", + SourceID: "narratio.example.npcs", + RelativePath: "artifacts/npcs.json", + Contract: contract, + ExternalProvenance: provenance, + }}} + + got := mapResultOutputs("analyze", result, "narratio-run") + if len(got) != 1 { + t.Fatalf("outputs len = %d, want 1", len(got)) + } + if got[0].SourceID != "narratio.example.npcs" { + t.Fatalf("source_id = %q, want explicit source", got[0].SourceID) + } + if got[0].Kind != "structured_data" { + t.Fatalf("kind = %q, want explicit output kind preserved", got[0].Kind) + } + if got[0].Contract == nil || *got[0].Contract != *contract { + t.Fatalf("contract = %#v, want %#v", got[0].Contract, contract) + } + if got[0].ExternalProvenance == nil || *got[0].ExternalProvenance != *provenance { + t.Fatalf("external provenance = %#v, want %#v", got[0].ExternalProvenance, provenance) + } + if got[0].Contract == contract || got[0].ExternalProvenance == provenance { + t.Fatal("mapped metadata should not alias the stage result") + } +} + +func TestMapResultOutputsRetainsFallbackInference(t *testing.T) { + transcript := mapResultOutputs("trim", &stage.StageResult{Outputs: []artifacts.Ref{{ + Kind: artifactmodel.TranscriptOutputKindFinalTrimmed, + }}}, "run-id") + if len(transcript) != 1 || transcript[0].SourceID != artifacts.ArtifactTranscriptFinalTrimmed { + t.Fatalf("transcript fallback = %#v, want final-trimmed source", transcript) + } + + analyze := mapResultOutputs("analyze", &stage.StageResult{Outputs: []artifacts.Ref{{Kind: "session_recap"}}}, "run-id") + if len(analyze) != 1 || analyze[0].SourceID != "narratio.artifact.session_recap" || analyze[0].Kind != "scriptorium_artifact" { + t.Fatalf("analyze fallback = %#v, want configured artifact inference", analyze) + } +} + func TestNeedsObjectStoreForRunPrepareWithPreviousRequirements(t *testing.T) { tests := []struct { name string @@ -738,6 +813,126 @@ func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) { } } +func TestExecuteStagesPersistsSelfSkipAndContinues(t *testing.T) { + cfg := testConfig(t) + store := &manifest.LocalStore{} + manifestPath := manifestPathFor(cfg) + seed := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC)) + seed.MarkStageSucceeded("optional", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{{ + Kind: "old_output", + SourceID: "narratio.example.old", + LocalPath: "artifacts/old.json", + }}) + seed.Stages["optional"].Logs = []string{"old.log"} + seed.Stages["optional"].GeneratedConfigs = []string{"old.yml"} + seed.Stages["optional"].Metadata = map[string]any{"old": true} + if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil { + t.Fatalf("MkdirAll() error = %v", err) + } + if err := store.Save(context.Background(), manifestPath, seed); err != nil { + t.Fatalf("Save() seed manifest error = %v", err) + } + + order := []string{} + optionalRuns := 0 + stages := []stage.Stage{ + resultStage{ + name: "optional", + runs: &optionalRuns, + order: &order, + result: &stage.StageResult{ + Disposition: stage.StageDispositionSkipped, + SkipReason: "integration_disabled", + Logs: []string{"runs/current/optional.log"}, + GeneratedConfigs: []string{"runs/current/optional.yml"}, + Metadata: map[string]any{"enabled": false}, + }, + }, + resultStage{name: "later", order: &order, result: &stage.StageResult{}}, + } + summary, err := executeStages(context.Background(), cfg, stages, RunOptions{Force: true}) + if err != nil { + t.Fatalf("executeStages() error = %v", err) + } + if strings.Join(order, ",") != "optional,later" { + t.Fatalf("execution order = %v, want optional then later", order) + } + if len(summary.Executed) != 2 || len(summary.Skipped) != 1 || summary.Skipped[0] != "optional" { + t.Fatalf("summary = %#v, want optional executed and self-skipped before later", summary) + } + + sessionManifest, err := store.Load(context.Background(), summary.ManifestPath) + if err != nil { + t.Fatalf("Load() session manifest error = %v", err) + } + selfSkipped := sessionManifest.Stages["optional"] + if selfSkipped == nil || selfSkipped.Status != manifest.StatusSkipped { + t.Fatalf("optional stage = %#v, want skipped", selfSkipped) + } + if len(selfSkipped.Outputs) != 0 { + t.Fatalf("optional outputs = %#v, want old outputs cleared", selfSkipped.Outputs) + } + if selfSkipped.Error == nil || selfSkipped.Error.Message != "integration_disabled" { + t.Fatalf("optional skip reason = %#v, want integration_disabled", selfSkipped.Error) + } + if len(selfSkipped.Logs) != 1 || selfSkipped.Logs[0] != "runs/current/optional.log" || + len(selfSkipped.GeneratedConfigs) != 1 || selfSkipped.GeneratedConfigs[0] != "runs/current/optional.yml" || + selfSkipped.Metadata["enabled"] != false || selfSkipped.Metadata["old"] != nil { + t.Fatalf("optional result details = %#v, want current bounded diagnostics and metadata", selfSkipped) + } + if later := sessionManifest.Stages["later"]; later == nil || later.Status != manifest.StatusSucceeded { + t.Fatalf("later stage = %#v, want succeeded", later) + } + + runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath) + if err != nil { + t.Fatalf("LoadRun() error = %v", err) + } + runStage := runManifest.Stages["optional"] + if runStage == nil || runStage.Action != manifest.RunStageActionRun || runStage.Status != manifest.StatusSkipped { + t.Fatalf("run optional stage = %#v, want run action with skipped status", runStage) + } + if len(runStage.Logs) != 1 || runStage.Metadata["enabled"] != false { + t.Fatalf("run optional stage details = %#v, want result diagnostics and metadata", runStage) + } + + _, err = executeStages(context.Background(), cfg, []stage.Stage{stages[0]}, RunOptions{}) + if err != nil { + t.Fatalf("second executeStages() error = %v", err) + } + if optionalRuns != 2 { + t.Fatalf("optional runs = %d, want self-skipped stage reconsidered", optionalRuns) + } +} + +func TestExecuteStagesRejectsSkippedResultWithOutputs(t *testing.T) { + cfg := testConfig(t) + invalid := resultStage{name: "optional", result: &stage.StageResult{ + Disposition: stage.StageDispositionSkipped, + SkipReason: "integration_disabled", + Outputs: []artifacts.Ref{{Kind: "unexpected"}}, + }} + + summary, err := executeStages(context.Background(), cfg, []stage.Stage{invalid}, RunOptions{}) + if err == nil { + t.Fatal("executeStages() error = nil, want invalid skipped result failure") + } + if summary != nil { + t.Fatalf("summary = %#v, want nil", summary) + } + if !strings.Contains(err.Error(), "skipped result contains 1 output") { + t.Fatalf("error = %q, want skipped-output validation", err) + } + + loaded, loadErr := (&manifest.LocalStore{}).Load(context.Background(), manifestPathFor(cfg)) + if loadErr != nil { + t.Fatalf("Load() session manifest error = %v", loadErr) + } + if got := loaded.Stages["optional"]; got == nil || got.Status != manifest.StatusFailed { + t.Fatalf("optional stage = %#v, want failed", got) + } +} + func TestExecuteStagesRunLocalArtifactsAndCanonicalSync(t *testing.T) { cfg := testConfig(t) stages := []stage.Stage{ diff --git a/internal/artifactmodel/metadata.go b/internal/artifactmodel/metadata.go new file mode 100644 index 0000000..5b24ac5 --- /dev/null +++ b/internal/artifactmodel/metadata.go @@ -0,0 +1,17 @@ +package artifactmodel + +// ContractMetadata identifies the data contract implemented by an artifact. +type ContractMetadata struct { + MediaType string `json:"media_type"` + SchemaID string `json:"schema_id"` + SchemaVersion string `json:"schema_version"` + ModuleKey string `json:"module_key,omitempty"` +} + +// ExternalProvenance identifies an artifact produced by an external system. +type ExternalProvenance struct { + System string `json:"system"` + RunID string `json:"run_id"` + PipelineID string `json:"pipeline_id"` + ArtifactID string `json:"artifact_id"` +} diff --git a/internal/artifactmodel/metadata_test.go b/internal/artifactmodel/metadata_test.go new file mode 100644 index 0000000..5be0880 --- /dev/null +++ b/internal/artifactmodel/metadata_test.go @@ -0,0 +1,56 @@ +package artifactmodel + +import ( + "encoding/json" + "strings" + "testing" +) + +func TestArtifactMetadataJSON(t *testing.T) { + type envelope struct { + Contract *ContractMetadata `json:"contract,omitempty"` + ExternalProvenance *ExternalProvenance `json:"external_provenance,omitempty"` + } + + complete, err := json.Marshal(envelope{ + Contract: &ContractMetadata{ + MediaType: "application/json", + SchemaID: "notarius.dnd.npc_registry", + SchemaVersion: "v1", + ModuleKey: "dnd/npc-registry", + }, + ExternalProvenance: &ExternalProvenance{ + System: "notarius", + RunID: "run-123", + PipelineID: "dnd-session", + ArtifactID: "npc-registry", + }, + }) + if err != nil { + t.Fatalf("Marshal() complete metadata error = %v", err) + } + wantComplete := `{"contract":{"media_type":"application/json","schema_id":"notarius.dnd.npc_registry","schema_version":"v1","module_key":"dnd/npc-registry"},"external_provenance":{"system":"notarius","run_id":"run-123","pipeline_id":"dnd-session","artifact_id":"npc-registry"}}` + if string(complete) != wantComplete { + t.Fatalf("complete metadata JSON = %s, want %s", complete, wantComplete) + } + + omitted, err := json.Marshal(envelope{}) + if err != nil { + t.Fatalf("Marshal() omitted metadata error = %v", err) + } + if string(omitted) != `{}` { + t.Fatalf("omitted metadata JSON = %s, want {}", omitted) + } + + withoutModule, err := json.Marshal(envelope{Contract: &ContractMetadata{ + MediaType: "application/json", + SchemaID: "notarius.dnd.npc_registry", + SchemaVersion: "v1", + }}) + if err != nil { + t.Fatalf("Marshal() contract without module key error = %v", err) + } + if strings.Contains(string(withoutModule), "module_key") { + t.Fatalf("contract JSON unexpectedly contains omitted module_key: %s", withoutModule) + } +} diff --git a/internal/artifacts/store.go b/internal/artifacts/store.go index 55d988c..e2806b1 100644 --- a/internal/artifacts/store.go +++ b/internal/artifacts/store.go @@ -1,16 +1,23 @@ package artifacts -import "os" +import ( + "os" + + "gitea.maximumdirect.net/eric/narratio/internal/artifactmodel" +) // Ref identifies a pipeline artifact and its local/remote coordinates. type Ref struct { - Kind string - Category string - SessionID string - RelativePath string - AbsolutePath string - RemoteKey string - Checksum string + Kind string + SourceID string + Category string + SessionID string + RelativePath string + AbsolutePath string + RemoteKey string + Checksum string + Contract *artifactmodel.ContractMetadata + ExternalProvenance *artifactmodel.ExternalProvenance } // Store is the local artifact/workdir abstraction used by orchestration code. diff --git a/internal/manifest/manifest.go b/internal/manifest/manifest.go index 65a2307..dde661e 100644 --- a/internal/manifest/manifest.go +++ b/internal/manifest/manifest.go @@ -3,6 +3,8 @@ package manifest import ( "strings" "time" + + "gitea.maximumdirect.net/eric/narratio/internal/artifactmodel" ) // ErrorRecord captures structured error metadata at run or stage scope. @@ -28,9 +30,11 @@ type InputRecord struct { // ArtifactRecord captures one produced artifact and optional remote metadata. type ArtifactRecord struct { - Kind string `json:"kind"` - SourceID string `json:"source_id,omitempty"` - LocalPath string `json:"local_path"` + Kind string `json:"kind"` + SourceID string `json:"source_id,omitempty"` + LocalPath string `json:"local_path"` + Contract *artifactmodel.ContractMetadata `json:"contract,omitempty"` + ExternalProvenance *artifactmodel.ExternalProvenance `json:"external_provenance,omitempty"` // ProducerRunID identifies the run that produced this durable artifact. ProducerRunID string `json:"producer_run_id,omitempty"` RemoteKey string `json:"remote_key,omitempty"` @@ -120,6 +124,7 @@ func (m *Manifest) MarkStageSkipped(name string, at time.Time, reason string) { s.Status = StatusSkipped s.CompletedAt = timePtr(at) s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)} + s.Outputs = nil s.UpdatedAt = at m.UpdatedAt = at } diff --git a/internal/manifest/manifest_test.go b/internal/manifest/manifest_test.go index e39b6a3..cafa9ef 100644 --- a/internal/manifest/manifest_test.go +++ b/internal/manifest/manifest_test.go @@ -110,3 +110,23 @@ func TestMarkStageSucceededClearsError(t *testing.T) { t.Fatalf("error = %#v, want nil on success", stage.Error) } } + +func TestMarkStageSkippedClearsEarlierOutputs(t *testing.T) { + m := New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC)) + m.MarkStageSucceeded("extract", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), []ArtifactRecord{ + {Kind: "structured_data", SourceID: "narratio.example.characters", LocalPath: "artifacts/characters.json"}, + }) + + m.MarkStageSkipped("extract", time.Date(2026, 5, 3, 10, 2, 0, 0, time.UTC), "integration_disabled") + + stage := m.Stages["extract"] + if stage == nil { + t.Fatal("missing stage record") + } + if stage.Status != StatusSkipped { + t.Fatalf("status = %q, want %q", stage.Status, StatusSkipped) + } + if len(stage.Outputs) != 0 { + t.Fatalf("outputs = %#v, want cleared", stage.Outputs) + } +} diff --git a/internal/manifest/run_manifest.go b/internal/manifest/run_manifest.go index 63f08e3..8ca66b9 100644 --- a/internal/manifest/run_manifest.go +++ b/internal/manifest/run_manifest.go @@ -119,6 +119,7 @@ func (m *RunManifest) MarkStageSkipped(name string, at time.Time, reason string) s.Status = StatusSkipped s.CompletedAt = timePtr(at) s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)} + s.Outputs = nil s.UpdatedAt = at m.UpdatedAt = at } diff --git a/internal/manifest/store_test.go b/internal/manifest/store_test.go index 74b36ec..f7f79ac 100644 --- a/internal/manifest/store_test.go +++ b/internal/manifest/store_test.go @@ -8,6 +8,8 @@ import ( "strings" "testing" "time" + + "gitea.maximumdirect.net/eric/narratio/internal/artifactmodel" ) func TestLocalStoreCreateSaveLoadRoundTrip(t *testing.T) { @@ -64,6 +66,76 @@ func TestLocalStoreCreateSaveLoadRoundTrip(t *testing.T) { } } +func TestLocalStoreArtifactMetadataCompatibility(t *testing.T) { + store := &LocalStore{} + ctx := context.Background() + dir := t.TempDir() + oldPath := filepath.Join(dir, "old-manifest.json") + oldJSON := `{ + "session_id": "2026-05-03", + "created_at": "2026-05-03T10:00:00Z", + "updated_at": "2026-05-03T10:01:00Z", + "stages": { + "analyze": { + "name": "analyze", + "status": "succeeded", + "created_at": "2026-05-03T10:00:00Z", + "updated_at": "2026-05-03T10:01:00Z", + "outputs": [{"kind":"scriptorium_artifact","local_path":"artifacts/recap.md"}] + } + } +}` + if err := os.WriteFile(oldPath, []byte(oldJSON), 0o644); err != nil { + t.Fatalf("WriteFile() old manifest error = %v", err) + } + + loaded, err := store.Load(ctx, oldPath) + if err != nil { + t.Fatalf("Load() old manifest error = %v", err) + } + oldOutput := loaded.Stages["analyze"].Outputs[0] + if oldOutput.Contract != nil || oldOutput.ExternalProvenance != nil { + t.Fatalf("old output metadata = %#v, %#v; want nil", oldOutput.Contract, oldOutput.ExternalProvenance) + } + if err := store.Save(ctx, oldPath, loaded); err != nil { + t.Fatalf("Save() old manifest error = %v", err) + } + roundTripped, err := os.ReadFile(oldPath) + if err != nil { + t.Fatalf("ReadFile() round-tripped old manifest error = %v", err) + } + if strings.Contains(string(roundTripped), `"contract"`) || strings.Contains(string(roundTripped), `"external_provenance"`) { + t.Fatalf("old manifest gained fabricated metadata:\n%s", roundTripped) + } + + loaded.Stages["analyze"].Outputs[0].Contract = &artifactmodel.ContractMetadata{ + MediaType: "application/json", + SchemaID: "example.recap", + SchemaVersion: "v1", + } + loaded.Stages["analyze"].Outputs[0].ExternalProvenance = &artifactmodel.ExternalProvenance{ + System: "example", + RunID: "external-run", + PipelineID: "pipeline", + ArtifactID: "recap", + } + metadataPath := filepath.Join(dir, "metadata-manifest.json") + if err := store.Save(ctx, metadataPath, loaded); err != nil { + t.Fatalf("Save() metadata manifest error = %v", err) + } + withMetadata, err := store.Load(ctx, metadataPath) + if err != nil { + t.Fatalf("Load() metadata manifest error = %v", err) + } + got := withMetadata.Stages["analyze"].Outputs[0] + if got.Contract == nil || got.Contract.SchemaID != "example.recap" { + t.Fatalf("contract = %#v, want persisted contract", got.Contract) + } + if got.ExternalProvenance == nil || got.ExternalProvenance.RunID != "external-run" { + t.Fatalf("external provenance = %#v, want persisted provenance", got.ExternalProvenance) + } +} + func TestLocalStoreSaveUpdatesTimestamp(t *testing.T) { store := &LocalStore{} ctx := context.Background() diff --git a/internal/stage/stage.go b/internal/stage/stage.go index 59d251f..d63c0d0 100644 --- a/internal/stage/stage.go +++ b/internal/stage/stage.go @@ -44,8 +44,19 @@ type Stage interface { Run(ctx context.Context, env *Env, m *manifest.Manifest) (*StageResult, error) } +// StageDisposition describes the outcome of a stage that returned without an error. +type StageDisposition string + +const ( + // StageDispositionSucceeded is the zero value so existing stages remain successful. + StageDispositionSucceeded StageDisposition = "" + StageDispositionSkipped StageDisposition = "skipped" +) + // StageResult is the declared output of a stage execution. type StageResult struct { + Disposition StageDisposition + SkipReason string Outputs []artifacts.Ref Logs []string GeneratedConfigs []string