eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity

Add artifact provenance and stage skip outcomes

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-09 23:20:32 +00:00
parent df58595d1e
commit 951383226c
12 changed files with 1565 additions and 305 deletions
Download Patch File Download Diff File Expand all files Collapse all files

577
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,577 @@
# Notarius Extraction Implementation Plan
## Status And Audience
Proposed and not started.
This plan is written for a GPT-5.6 Terra coding agent. Implement the stages in
strict numerical order. Do not skip ahead, combine stages merely to reduce the
number of prompts, or describe unfinished behavior as current functionality.
The target state and acceptance criteria are owned by
[notarius-extract-stage.md](notarius-extract-stage.md). This document owns the
implementation sequence.
## Progress
| Stage | Status |
| --- | --- |
| Stage 1 | Complete |
| Stage 2 | Not started |
| Stage 3 | Not started |
| Stage 4 | Not started |
| Stage 5 | Not started |
| Stage 6 | Not started |
| Stage 7 | Not started |
| Stage 8 | Not started |
| Stage 9 | Not started |
After completing and validating a stage, update only that stage's row to
`Complete` and record any material deviation in the relevant stage section.
Do not mark a stage complete while required tests or exit criteria remain.
## Working Rules
Before Stage 1, read:
- `docs/development.md` and its task-specific references;
- `docs/roadmap/notarius-extract-stage.md` completely;
- `../notarius/docs/consumers/dnd-pipeline.md` and its linked subprocess,
receipt, JSON-output, and lane-contract documentation; and
- the focused Narratio documents and tests named by the current stage.
For every stage:
1. Inspect the current implementation before editing. Prefer the codebase
knowledge graph for code discovery as required by `AGENTS.md`.
2. Preserve unrelated worktree changes.
3. Implement the complete stage scope, including focused tests. Do not leave
TODO implementations, compatibility shims without owners, or knowingly
dead code for a later stage.
4. Use real internal collaborators in tests where fast and deterministic; fake
only subprocess, remote, clock, randomness, or other external boundaries.
5. Keep tests offline and independent of real Notarius, PromptKit, LLM
providers, credentials, and mutable services.
6. Run the focused package tests listed for the stage. Fix failures before
proceeding.
7. Run `go test ./...` after every stage. Also run `go vet ./...` and
`go build ./cmd/narratio` when the stage changes shared contracts,
composition, CLI behavior, or documentation examples.
8. Keep proposed behavior in `docs/roadmap/` until Stage 9. Intermediate code
is development work and must not cause current-behavior documentation to
claim that the end-to-end feature is complete.
Intermediate stages must compile and pass the repository test suite. The work
is not release-ready until Stage 9 is complete.
## Stage 1: Establish Shared Stage-Outcome And Artifact-Provenance Contracts
Introduce the reusable internal contracts needed by extraction without adding
Notarius configuration or a registered stage.
Implementation:
1. Under `internal/artifactmodel`, add optional neutral models for:
- contract metadata: `media_type`, `schema_id`, `schema_version`, and
optional `module_key`;
- external provenance: `system`, `run_id`, `pipeline_id`, and
`artifact_id`.
2. Extend `artifacts.Ref` with:
- optional explicit `SourceID`;
- optional contract metadata; and
- optional external provenance.
3. Extend `manifest.ArtifactRecord` with the same nested optional metadata.
Use backward-compatible `omitempty` JSON fields. Old manifests must load
and round-trip without fabricated metadata.
4. Refactor `internal/app.mapResultOutputs` to prefer `Ref.SourceID` when set,
copy both metadata structures, and retain existing output-kind and analyze
inference only as fallback behavior.
5. Add an explicit `StageResult` disposition with succeeded and skipped
values. Preserve zero-value success so existing stages do not need
mechanical edits.
6. Add a stable skip-reason field. Reject or fail safely if a skipped result
contains outputs.
7. Teach `executeStages` to persist a self-skipped stage as skipped in both
session and run manifests, apply bounded result metadata and diagnostics,
clear any older session-stage outputs, and continue to later stages.
Distinguish this from the runner's existing decision to skip an already
succeeded stage.
8. Keep successful and failed behavior unchanged for existing stages.
Tests:
- `internal/artifactmodel`: JSON behavior for complete and omitted metadata.
- `internal/manifest`: old manifest compatibility and self-skip clearing of
earlier outputs.
- `internal/app`: explicit source ID precedence, fallback inference,
provenance copying, self-skip persistence, continuation after self-skip,
and rejection of skipped results containing outputs.
Exit criteria:
- Existing stage outputs and manifests remain compatible.
- A synthetic stage can self-skip without being recorded as succeeded.
- A later invocation will reconsider a self-skipped stage.
- `go test ./...`, `go vet ./...`, and `go build ./cmd/narratio` pass.
## Stage 2: Add Notarius Configuration And Extraction Source Policy
Add strict configuration and stable source identity, but do not invoke
Notarius or register `extract` yet.
Implementation:
1. Add an optional `Notarius *NotariusConfig` field to `PipelineConfig`.
2. Define:
- `enabled` with false default;
- `binary` with `notarius` default;
- `config_path`;
- `pipeline_id`;
- `timeout` with `3h` default;
- optional `working_directory`; and
- `outputs map[string]NotariusOutputConfig`.
3. Define each output with required `lane_id`, `media_type`, `schema_id`, and
`schema_version`, plus optional `module_key`.
4. Apply defaults centrally. When enabled, resolve `config_path` to an
absolute path using the same configuration-source semantics as comparable
adapter paths. Default `working_directory` to the resolved config file's
directory, and resolve an explicitly configured working directory to an
absolute path.
5. Validate enabled configuration before stage execution:
- non-empty binary, config path, pipeline ID, and output map;
- positive duration;
- non-empty output contract fields;
- output keys accepted by the existing configured-artifact key policy;
- unique normalized extraction source IDs;
- unique lane IDs;
- no collision with built-in, configured Scriptorium, or previous-session
source identities.
6. Add `SourceKindExtraction` and helpers for the exact
`narratio.extraction.<key>` family in `internal/artifactpolicy` and
`internal/artifacts`. Do not accept arbitrary unconfigured extraction
sources at configuration validation boundaries.
7. Extend Scriptorium input and publish source validation to accept only
extraction keys declared by the same effective pipeline configuration.
8. Do not expose a Notarius session-ID setting, lane-selection flag, prompt,
profile, model, retry, or reference configuration.
Tests:
- strict YAML decoding and unknown-field rejection;
- omitted and disabled behavior;
- defaults and path resolution from pipeline-file location;
- enabled required fields and positive timeout;
- invalid/safe keys, normalized collisions, duplicate lanes, incomplete
contracts, and source-family collisions;
- valid and unknown extraction sources in Scriptorium inputs and publish
rules; and
- existing configurations and maintained examples continue to load.
Exit criteria:
- A complete Notarius configuration loads and normalizes deterministically.
- Invalid downstream extraction references fail during config validation.
- No runtime code invokes Notarius yet.
- `go test ./internal/config ./internal/artifactpolicy ./internal/artifacts`
and `go test ./...` pass.
## Stage 3: Add Safe Immutable Directory Promotion
Implement the filesystem primitive used to promote a validated external bundle
without exposing a partial or mixed durable directory.
Implementation:
1. Add a narrowly named directory-promotion operation to `internal/fileops`.
It accepts an existing source directory and a new destination directory.
2. Require a non-empty source and destination. The destination must not exist.
3. Create a temporary sibling of the destination so final installation uses a
same-filesystem rename.
4. Recursively walk with `Lstat` semantics:
- permit directories and regular files only;
- reject symlinks, devices, sockets, named pipes, and other special files;
- never follow a symlink;
- use deterministic lexical traversal;
- preserve relative layout; and
- use bounded safe permissions rather than copying unsafe mode bits.
5. Copy and sync each regular file, sync the completed temporary tree where
practical, and rename the temporary directory to the unique destination.
6. On any failure, remove only the operation's validated temporary sibling.
Never remove or overwrite the source, destination, workspace root, or a
broad caller-provided directory.
7. Return an error if a concurrent actor creates the destination before final
installation.
Tests using `t.TempDir()`:
- nested regular-tree success and byte-for-byte preservation;
- deterministic layout;
- destination already exists;
- source is not a directory;
- symlink to a file, symlink to a directory, and escaping symlink rejection;
- representative special-file rejection where portable;
- copy/install failure cleanup with no partial destination; and
- source preservation after success and failure.
Exit criteria:
- A promoted directory appears only as a complete unique tree.
- The operation cannot follow symlinks or replace an existing destination.
- `go test ./internal/fileops` and `go test ./...` pass.
## Stage 4: Implement The Notarius Adapter
Create the external subprocess boundary independently of stage policy.
Implementation:
1. Add `internal/adapters/notarius` with:
- a `Runner` interface;
- transport-neutral request and result models;
- receipt, index, lane-descriptor, and pipeline-wide-descriptor models;
- a production subprocess runner; and
- a small configurable fake for stage tests.
2. Build exactly:
```text
notarius run <pipeline-id> --config <path> --input <path>
--output-dir <path> --json
```
Use separate argument elements, absolute supplied paths, the configured
working directory, inherited environment, and Narratio's shared subprocess
timeout/cancellation behavior. Do not pass `--session-id`.
3. Write stdout directly to the configured receipt path and stderr to the
configured log path. Do not combine streams.
4. On nonzero exit, cancellation, or timeout, return the subprocess error and
do not parse stdout.
5. After exit zero, read receipt bytes with an explicit upper bound, decode
`notarius.run-result.v1`, tolerate unknown fields, and validate required
fields and matching pipeline ID.
6. Require an absolute `output_directory` and confine it beneath the absolute
output root from the request. Confine the relative `index_file` beneath the
returned directory using Narratio's root-scoped path helpers. Reject
absolute logical paths, traversal, symlinks at consumed paths, and
non-regular files.
7. Decode the supported production JSON index tolerantly. Validate required
management paths and lane descriptors, reject duplicate lane IDs, and
confine every lane and pipeline-wide descriptor file beneath the bundle
root.
8. Load `rejected.json` and `warnings.json` with explicit size bounds and
tolerant decoding. Return only their generic summaries, including lane ID
and reason code where present; do not read lane payload bodies or interpret
D&D fields. Keep free-form external messages bounded for immediate errors
and out of manifest metadata.
9. Return descriptors and paths without deciding which lanes are required or
whether their configured contracts match.
10. Do not implement automatic `notarius config validate` execution.
Tests:
- exact argument order and omission of `--session-id`;
- working directory, environment inheritance, stream separation, cancellation,
timeout, and nonzero exit;
- stdout ignored after failure;
- receipt size limit, malformed JSON, unsupported version, missing fields,
pipeline mismatch, and tolerated unknown fields;
- output-directory escape from the requested output root;
- index malformed/unsupported shapes, duplicate lanes, missing management
paths, and tolerated optional fields;
- malformed or oversized rejection/warning documents and tolerated unknown
summary fields;
- absolute logical paths, lexical traversal, root-prefix confusion, symlink
paths, and descriptor escape attempts; and
- valid lane plus chunk-map/evidence-context discovery.
Use a test helper subprocess or injected shared runner. Do not require a real
Notarius binary.
Exit criteria:
- The adapter fully enforces the generic Notarius subprocess and discovery
boundary without D&D policy.
- `go test ./internal/adapters/notarius ./internal/adapters/subprocess` and
`go test ./...` pass.
## Stage 5: Implement Extract-Stage Execution And Bundle Materialization
Implement the stage behind direct package tests, but do not yet add it to the
canonical `stage.All()` registry.
Implementation:
1. Add `Notarius notarius.Runner` to `stage.Env` and default production
composition in `internal/app/runner.go` when enabled extraction is selected.
2. Add canonical path helpers for:
- the run-local extract directory;
- receipt and stderr paths;
- the run-local Notarius output root; and
- immutable `artifacts/notarius/<narratio-run-id>` destinations.
3. Implement `extractStage` with name `extract` and declarations matching its
final-trimmed input and structured outputs.
4. When configuration is absent or disabled, return the explicit skipped
result with `notarius_disabled` and no outputs.
5. Resolve `narratio.transcript.final_trimmed` through the manifest-aware
resolver. Do not fall back to guessed filenames outside that resolver.
6. Invoke the adapter once with resolved absolute paths.
7. Apply required-output policy:
- find each configured output by exact lane ID;
- require exactly one descriptor;
- compare media type, schema ID/version, and optional module key;
- treat rejected or absent configured lanes as failure;
- tolerate unconfigured lanes;
- require each selected file to be regular, non-empty, and syntactically
valid JSON; and
- compute a staging checksum for later copy verification.
8. Validate the complete bundle tree for promotion, then use Stage 3's helper
to copy it to the unique immutable destination.
9. Re-resolve the promoted index and configured descriptor paths rather than
retaining staging paths. Compute authoritative SHA-256 checksums from the
promoted files and fail if promoted bytes differ from the validated source.
10. Return one `artifacts.Ref` per configured lane with explicit extraction
source ID, checksum, contract metadata, and Notarius external provenance.
11. Return the promoted `index.json` as a `notarius_index` output without a
selectable source ID.
12. Return bounded stage metadata containing durable bundle root, receipt and
diagnostic paths, receipt counts/status, rejection/warning summaries, the
producing Narratio run ID, and a deterministic fingerprint of the resolved
binary, config path, pipeline ID, timeout, working directory, and sorted
output-contract map.
13. Do not delete run-local diagnostics or failed bundles.
Tests:
- disabled self-skip;
- missing/invalid final-trimmed input before adapter invocation;
- exact adapter request construction;
- required, missing, rejected, duplicate, incompatible, and unconfigured
lanes;
- non-empty and JSON-syntax validation without D&D decoding;
- immutable promotion and promoted-path re-resolution;
- checksums, explicit source IDs, nested metadata, and external provenance;
- adapter and promotion failures do not return successful outputs; and
- complete bundle files are preserved while unknown files are not registered
as sources.
Exit criteria:
- Direct extract-stage execution produces a complete manifest-ready result
using only a fake Notarius runner.
- The stage is not yet in normal CLI plans.
- `go test ./internal/stage ./internal/artifacts ./internal/fileops` and
`go test ./...` pass.
## Stage 6: Integrate Canonical Ordering, CLI Lifecycle, Force, And Resume
Make `extract` a real pipeline stage and complete its orchestration semantics.
Implementation:
1. Register `extractStage` in `stage.All()` immediately after `trim` and before
`render`.
2. Update all hard-coded or user-visible stage inventories, stage-name
validation, CLI help, planning, prerequisite checks, and test fixtures.
3. Confirm full and single-stage commands can select `extract`. Do not add a
lane-selection flag and do not broaden `--artifacts` beyond Scriptorium
artifact selection.
4. Add an optional stage resume-validation contract. Use a result containing:
- resumable boolean; and
- bounded non-resumable reason.
Reserve an ordinary error for cases where a safe decision cannot be made.
5. Before skipping a succeeded stage, `executeStages` invokes the optional
validator. A non-resumable result marks the session stage stale with its
reason and schedules it to run. A validation error stops execution without
mutating the succeeded record.
6. Implement extraction resume validation against:
- enabled state;
- stored invocation/config-contract fingerprint;
- succeeded extract record;
- immutable bundle and index paths;
- exact configured source set;
- regular confined lane files;
- checksums;
- contracts; and
- Notarius external provenance.
Require the producing Narratio run ID recorded in extract metadata, output
records, and the immutable bundle path to agree. Do not compare it with the
session manifest's top-level `run_id`, which identifies the latest
invocation and may legitimately differ after resume.
7. Do not inspect or hash Notarius's transitive reference/profile closure.
8. Verify canonical downstream invalidation:
- force through `trim` stales extract and later succeeded stages;
- force `extract` stales render and later succeeded stages;
- force `render` does not stale extract.
9. Verify existing session manifests without an extract record normalize and
execute the new stage rather than failing migration.
Tests:
- exact canonical order and CLI stage inventory;
- full-plan and `run-stage extract` behavior;
- disabled skip followed by later enablement;
- successful resume skip;
- missing, tampered, incompatible, or config-mismatched outputs trigger stale
and rerun;
- unsafe validation errors stop without corrupting prior success;
- force and downstream invalidation in each direction named above; and
- old manifest compatibility.
Exit criteria:
- Normal execution includes `extract` in the agreed position.
- Disabled and resume behavior cannot make obsolete outputs current.
- `go test ./internal/stage ./internal/app ./internal/manifest`,
`go test ./...`, `go vet ./...`, and `go build ./cmd/narratio` pass.
## Stage 7: Integrate Extraction Sources With The Runtime Catalog And Analyze
Make succeeded extraction lanes selectable by configured Scriptorium
artifacts.
Implementation:
1. Extend `ArtifactCatalog` with extraction definitions registered from the
effective Notarius output map.
2. Add one shared manifest-backed hydration operation for extraction sources.
It accepts session paths, manifest, and configured extraction definitions
and marks entries available only after verifying the roadmap's full
successful-stage, source, path, checksum, contract, and provenance rules.
3. Add a distinct extraction provenance value such as
`manifest.current_extract_run`.
4. Extend `ResolveSessionArtifactWithCatalog` to resolve configured extraction
sources through catalog availability. It must never scan the Notarius
artifact directory or accept an incidental file.
5. Refactor catalog construction enough that extraction registration and
hydration are reusable by analyze and publish. Do not create a generic
workflow abstraction or move Scriptorium execution policy into artifacts.
6. Update analyze catalog creation to include effective extraction
definitions and current manifest state before planning Scriptorium inputs.
7. Preserve required/optional input behavior. Required unavailable extraction
inputs fail with actionable extract configuration/rerun guidance; optional
inputs are omitted normally.
8. Do not automatically add extraction inputs to any configured artifact.
Tests:
- registration, lookup, deterministic ordering, and source-family collision;
- successful current-manifest hydration;
- missing stage, skipped, failed, stale, interrupted, missing-source,
internally inconsistent producer identity, incompatible-contract,
missing-file, tampered-checksum, and unsafe-path rejection;
- required and optional analyze input behavior;
- selected Scriptorium artifact execution receives only explicitly configured
extraction inputs; and
- no object-store calls or directory scanning occur during analyze
extraction resolution.
Exit criteria:
- A fake successful extract run can feed selected JSON files into a fake
Scriptorium analysis run.
- No failed or incidental bundle can become available through the catalog.
- `go test ./internal/artifacts ./internal/stage ./internal/config` and
`go test ./...` pass.
## Stage 8: Integrate Extraction Sources With Publish And Operator Inspection
Complete downstream artifact handling without automatically publishing the
bundle.
Implementation:
1. Use Stage 7's shared extraction catalog registration and hydration in
publish planning and source resolution.
2. Permit explicit publish rules whose source is a configured extraction
source. Preserve existing destination policy, lock behavior, required
output policy, selected-Scriptorium behavior, and remote commit order.
3. Do not publish the complete Notarius bundle unless individual explicit
rules name selectable sources. `notarius_index` remains non-selectable in
the initial feature.
4. Ensure the published session manifest retains extraction artifact contract
and external provenance metadata through ordinary serialization.
5. Extend existing artifact/status/inspection output only where necessary so
configured extraction sources report planned, available, unavailable, and
published states consistently. Do not expose payload contents.
6. Confirm restore safely round-trips explicitly published extraction files
through existing manifest and confined-path behavior; add code only if a
real incompatibility is found.
Tests:
- successful explicit lane publication;
- required missing or invalid extraction source failure;
- optional/unselected behavior remains consistent;
- publish lock and commit-marker order remain unchanged;
- `--artifacts` does not partially select Notarius lanes;
- manifest metadata survives publish serialization and restore loading; and
- operator inspection never reads or prints extraction payload bodies.
Exit criteria:
- Explicit extraction publish rules work through the shared catalog.
- No implicit whole-bundle publication is introduced.
- `go test ./internal/stage ./internal/app ./internal/artifacts` and
`go test ./...` pass.
## Stage 9: Add Maintained Examples, Current Documentation, And Final Validation
Finish the public and maintainer contract only after the implementation is
complete.
Implementation:
1. Read Notarius's current published lane contracts and add exact media type,
schema ID, schema version, and module-key constraints for all ten lanes to
a maintained Narratio complete example. Do not guess or copy stale roadmap
placeholders.
2. Add a small maintained Scriptorium example that consumes a purpose-specific
subset of extraction sources rather than all ten.
3. Add `docs/integrations/notarius.md` as Narratio's external-consumer
contract. Link to Notarius's canonical documents instead of duplicating
complete schemas.
4. Add `docs/internal/stage-extract.md` for implemented stage mechanics,
lifecycle, resume validation, failure behavior, and focused tests.
5. Update the canonical owners listed in the roadmap's Documentation
Deliverables section. Keep configuration fields/defaults only in
`docs/config.md`, commands only in `docs/cli.md`, physical layout and force
procedures only in `docs/operations.md`, and implementation mechanics only
in internal documents.
6. Update every implemented stage inventory to show `extract` between `trim`
and `render`.
7. Add troubleshooting guidance for missing Notarius, nonzero exit,
receipt/index incompatibility, required-lane rejection, resume
invalidation, and the requirement to force extraction after changing
Notarius's transitive configuration inputs.
8. Review the roadmap status. If every acceptance criterion is implemented,
mark it complete or move completed planning material according to repository
convention without deleting useful decision context prematurely.
9. Check all changed links, examples, commands, field names, defaults, schemas,
and paths against the implementation.
Validation:
- focused example/configuration validation tests;
- `go test ./...`;
- `go vet ./...`;
- `go build ./cmd/narratio`;
- inspect `git diff --check`;
- confirm ordinary tests use no live Notarius, PromptKit, LLM provider,
credentials, network service, or mutable external state; and
- confirm no secrets or private campaign content were added.
Exit criteria:
- Every feature-roadmap acceptance criterion is satisfied.
- Maintained examples load under strict configuration validation.
- Current-behavior documentation accurately describes the shipped feature
without duplicating Notarius-owned schema definitions.
- The repository-wide validation suite passes.
## Open Questions
None. The target architecture, defaults, lifecycle behavior, compatibility
boundary, storage layout, source identity, downstream selection model, and
implementation order are decision-complete for implementation. If repository
inspection exposes a contradiction with an implemented invariant, stop at the
affected stage and document the concrete conflict rather than silently
changing this plan's target behavior.

789
docs/roadmap/notarius-extract-stage.md
View File

@@ -2,266 +2,497 @@
## Status
Proposed.
Proposed. Implementation has not started.
The ordered implementation plan is maintained in
[implementation.md](implementation.md). Until that plan is complete, this
document is the only Narratio documentation that describes the proposed
`extract` stage, Notarius configuration, or `narratio.extraction.*` sources.
## Purpose
Add a first-class Narratio `extract` stage that runs Notarius against the
session's final trimmed transcript, validates and collects the resulting
structured D&D artifacts, and registers those artifacts for later use by the
`analyze` and `publish` stages.
session's final trimmed transcript and makes validated structured artifacts
available to later analysis and publish work.
This feature should integrate Notarius through Narratio's existing stage,
adapter, manifest, workspace, and artifact-catalog boundaries. It must not turn
Narratio into a generic workflow engine or a second configuration language for
Notarius pipelines.
Notarius remains responsible for its D&D extraction pipeline, prompts,
references, LLM profiles, retries, validation, normalization, and published
schemas. Narratio owns invocation, required-output policy, safe bundle
ingestion, artifact identity, manifest state, resume, and downstream
availability.
The integration must preserve Narratio's explicit stage model. It must not
turn Narratio into a generic workflow engine or reproduce Notarius's
configuration language.
## User Outcome
An operator can enable one configured Notarius pipeline for a Narratio
campaign. During a normal run, Narratio will:
An operator can enable one Notarius pipeline for a Narratio campaign. During a
normal run, Narratio will:
1. finish producing the session transcript tiers;
2. invoke Notarius once with the final trimmed Seriatim JSON transcript;
3. collect and validate the configured structured artifact lanes;
4. record their exact files and provenance in the Narratio manifest; and
5. make those artifacts selectable as inputs to Scriptorium artifacts in the
later `analyze` stage.
1. produce the final trimmed Seriatim JSON transcript;
2. invoke Notarius once with that transcript;
3. discover the exact run bundle through Notarius's machine-readable receipt;
4. validate every output contract Narratio is configured to require;
5. promote the complete validated bundle into immutable session storage;
6. record exact artifact paths, checksums, contracts, and external provenance;
7. make configured lanes available as `narratio.extraction.<key>` sources; and
8. allow individual Scriptorium artifacts and publish rules to select those
sources explicitly.
The maintained D&D example should demonstrate all ten lanes emitted by
Notarius's complete `dnd-session` pipeline.
The maintained complete D&D example will require all ten lanes published by
Notarius's `dnd-session` pipeline. Ordinary deployments may configure a
narrower required set.
## Target Stage Architecture
## Chosen Architecture
### First-Class Stage
Extraction is an independently observable and resumable pipeline stage. It is
not part of `analyze`, a Scriptorium artifact producer, or an implicit external
preprocessing requirement.
This boundary is required because Notarius work is expensive, produces
multiple durable outputs, has its own compatibility and diagnostic contracts,
and may be consumed by both `analyze` and `publish`.
### Canonical Order
The canonical stage order becomes:
```text
prepare -> transcribe -> merge -> polish -> normalize -> trim -> render
-> extract -> analyze -> publish -> notify
prepare -> transcribe -> merge -> polish -> normalize -> trim
-> extract -> render -> analyze -> publish -> notify
```
`extract` is deliberately after all transcript-producing stages and before
analysis. Its source document is the manifest-resolved
`narratio.transcript.final_trimmed` artifact, normally
`transcripts/final.trimmed.json`. It does not consume rendered Markdown.
`extract` consumes `narratio.transcript.final_trimmed`, produced by `trim`. It
does not consume rendered Markdown.
Adding the stage must update full-plan construction, explicit stage selection,
downstream invalidation, prerequisite checks, resume behavior, run manifests,
CLI stage validation and help, and every canonical-stage inventory. Forcing an
upstream transcript stage must stale a previously successful `extract` stage
and its downstream stages. Forcing `extract` must stale `analyze`, `publish`,
and `notify` according to existing rules.
Narratio currently invalidates succeeded stages by canonical downstream order.
Placing `extract` before `render` means forcing extraction may rerun the less
expensive deterministic render stage, while forcing render does not rerun the
more expensive Notarius pipeline. This is preferable to placing extraction
after render and does not require dependency-aware scheduling or a DAG.
Adding the stage must update every canonical-stage inventory, full-plan and
single-stage selection, prerequisite validation, downstream invalidation,
resume behavior, run manifests, CLI validation and help, and focused tests.
### Stage Boundary
The stage owns Narratio policy and state transitions:
The stage owns Narratio policy:
- resolve the final trimmed transcript through the runtime artifact catalog;
- build a Narratio-level Notarius request from validated configuration and
run-local paths;
- call a narrow Notarius adapter;
- apply the configured required-output policy;
- materialize the validated bundle into its canonical session location;
- return manifest-ready artifact references and bounded metadata; and
- fail without marking the stage successful when any required contract or
materialization step fails.
- resolve the final trimmed transcript through the manifest-aware artifact
resolver;
- allocate run-local receipt, log, and output-root paths;
- build a transport-neutral Notarius request from resolved configuration;
- call the configured Notarius adapter once;
- enforce Narratio's configured required-output policy;
- validate selected payloads as regular, non-empty, syntactically valid JSON;
- promote the validated bundle to immutable session artifact storage;
- return explicit source IDs, checksums, contracts, and external provenance;
and
- report an explicit skipped outcome when Notarius is disabled.
The stage must not construct subprocess arguments, infer Notarius output
filenames, parse provider logs, or decode individual D&D payload bodies.
The stage must not construct subprocess arguments, guess Notarius filenames,
parse interactive output, decode D&D payload structures, or reproduce
Notarius pipeline configuration.
### Adapter Boundary
Add a dedicated Notarius adapter package with a small interface, production
subprocess implementation, and test fake. Its request should contain only the
resolved Notarius binary, configuration path, pipeline ID, transcript path,
output root, working directory, timeout, and process-log destinations needed
for one run.
Add `internal/adapters/notarius` with a narrow runner interface, production
subprocess implementation, and small fake.
The request contains only:
- resolved executable path or name;
- absolute Notarius configuration path;
- pipeline ID;
- absolute transcript path;
- absolute output root;
- working directory;
- timeout; and
- stdout receipt and stderr log destinations.
The adapter owns:
- optional `notarius config validate` preflight for the configured pipeline;
- exact `notarius run ... --json` argument construction;
- stdout and stderr separation;
- context cancellation and timeout propagation through Narratio's shared
subprocess boundary;
- context cancellation and timeout through Narratio's shared subprocess
boundary;
- environment inheritance;
- exit-status handling;
- decoding the `notarius.run-result.v1` success receipt;
- receipt and index path-confinement checks;
- decoding `index.json` and resolving descriptor paths safely beneath the
reported output directory; and
- returning a transport-neutral result containing the bundle location,
receipt summary, lane descriptors, pipeline-wide descriptors, warnings and
rejection locations, and diagnostic log paths.
- bounded receipt loading after exit status zero;
- tolerant decoding of supported `notarius.run-result.v1` documents;
- validation of required receipt fields;
- confinement of the receipt's absolute `output_directory` beneath the
absolute output root Narratio supplied for this invocation;
- confinement of `index_file` beneath the receipt's absolute
`output_directory`;
- tolerant decoding of the supported `index.json` contract;
- confinement of every index descriptor path beneath the bundle root; and
- bounded tolerant decoding of `rejected.json` and `warnings.json` into
transport-neutral summaries without reading lane payload bodies; summaries
persisted by Narratio contain structured scope, lane, and reason fields but
not unbounded free-form external messages; and
- returning transport-neutral receipt, descriptor, diagnostic, and bundle
information.
Only exit status zero permits receipt decoding. Receipt, index, or descriptor
paths that are absolute where a logical relative path is required, or that
escape their owning root, are integration failures. Unknown fields in a
supported receipt or index schema should be tolerated. Unsupported schema
versions and incompatible descriptor metadata should fail clearly.
Only exit status zero permits receipt decoding. Unsupported schema versions,
malformed documents, missing required fields, absolute logical paths, path
escapes, symlinks at consumed paths, and incompatible structural metadata are
integration failures.
The adapter must not write Narratio manifests, choose required lanes, decide
analysis inputs, or contain D&D domain logic.
The production runner will not execute `notarius config validate` before every
session. Notarius run-time validation remains authoritative, while operators
may use the separate validation command as deployment preflight. A second
automatic subprocess can be added later only if operational evidence warrants
it.
The adapter does not write Narratio manifests, decide required lanes, choose
analysis inputs, or interpret D&D payloads.
## External Contract Baseline
The integration consumes the contracts documented by Notarius in
`../notarius/docs/consumers/dnd-pipeline.md` and its linked canonical
integration documents.
The initial compatibility baseline is:
- `notarius run <pipeline-id> --config ... --input ... --output-dir ... --json`;
- successful receipt schema `notarius.run-result.v1`;
- an absolute receipt `output_directory`;
- logical `index_file` discovery beneath that directory;
- the production JSON `index.json` descriptor model; and
- the exact media type and schema identity configured for each required lane.
Compatibility is decided from these published contracts, not by parsing
`notarius --version`. Unknown fields in supported receipt and index versions
are tolerated. Unsupported versions or incompatible required descriptors fail
before any artifact becomes current in Narratio.
## Configuration Contract
Add a strict optional `pipeline.notarius` configuration section. Omission or
`enabled: false` keeps the current workflow usable and causes `extract` to
self-skip without outputs.
Add a strict optional `pipeline.notarius` section:
The section should provide:
- `enabled`: explicit opt-in;
- `binary`: Notarius executable, defaulting to `notarius`;
- `config_path`: required when enabled;
- `pipeline_id`: required when enabled;
- `timeout`: a positive stage timeout with a documented default;
- `working_directory`: optional explicit subprocess working directory,
defaulting to the directory containing `config_path`; and
- an `outputs` map defining the Notarius lane artifacts Narratio promises to
collect.
Each output-map key is a stable Narratio extraction key. Each value must define:
- the exact Notarius `lane_id`;
- the expected `media_type`;
- the expected `schema_id`;
- the expected `schema_version`; and
- optionally an expected `module_key` when the operator needs to constrain the
producing module as part of compatibility.
Narratio derives the downstream source ID
`narratio.extraction.<output-key>` from the map key. Keys and lane IDs must be
non-empty, unique after normalization, path-safe under the existing artifact
policy, and collision-free with built-in and configured artifact identities.
Every configured output is required: a successful Notarius process that omits
one, rejects it, or reports incompatible descriptor metadata fails the
`extract` stage.
This explicit map keeps Narratio's consumer contract stable when a Notarius
lane ID or schema changes and avoids hard-coding the current D&D family into a
generic adapter. It also replaces a separate `required_lanes` list, which would
duplicate configuration.
Narratio should not reproduce Notarius lane selection, references, LLM
profiles, model settings, retries, concurrency, or prompt configuration. Those
remain in the referenced Notarius configuration. Narratio should not expose a
runtime lane-selection flag for `extract`; one stage invocation runs the
configured Notarius pipeline as a unit.
All configured paths should become absolute during Narratio configuration
resolution. The deterministic default working directory allows a Notarius
profile path relative to that directory, but operator documentation should
still recommend absolute deployment paths where practical. Notarius reference
paths continue to follow Notarius's own configuration-relative rules.
## Output And Artifact Model
### Canonical Bundle
Run Notarius against a run-local output root. After all configured descriptors
are validated, materialize the contents of the exact run-specific Notarius
bundle into a fixed canonical session directory:
```text
artifacts/notarius/
```yaml
notarius:
enabled: true
binary: notarius
config_path: /absolute/path/to/notarius.yml
pipeline_id: dnd-session
timeout: 3h
working_directory: /absolute/path/to/deployment
outputs:
npc_registry:
lane_id: npc-registry
media_type: application/json
schema_id: <published-schema-id>
schema_version: <published-schema-version>
```
Preserve its relative layout, including `index.json`, `manifest.json`,
`rejected.json`, `warnings.json`, `lanes/`, and any indexed `chunk-map.json` or
`evidence-context.json`. Materialize the complete directory as one narrow,
transactional replacement so a failed or interrupted rerun cannot mix files
from different Notarius runs.
Fields:
The raw subprocess receipt and stderr log belong in the run-local `extract`
report and log directories. The raw receipt identifies the original run-local
Notarius bundle and must not be rewritten to pretend that the canonical copy
was its original `output_directory`. Narratio's manifest is the durable ledger
for the canonical materialized paths.
- `enabled` is an explicit opt-in and defaults to false;
- `binary` defaults to `notarius`;
- `config_path` is required when enabled;
- `pipeline_id` is required when enabled;
- `timeout` defaults to `3h` and must be positive;
- `working_directory` is optional and defaults to the directory containing
`config_path`; and
- `outputs` maps stable Narratio extraction keys to required Notarius lane
contracts.
### Registered Artifact Sources
All configured paths become absolute during configuration resolution. Notarius
reference paths continue to follow Notarius's configuration-relative rules,
while PromptKit profile paths remain relative to the chosen process working
directory where Notarius permits that behavior.
For each configured output, locate the lane through the canonical copy of
`index.json` and record a manifest artifact with:
Each output entry contains:
- source ID `narratio.extraction.<output-key>`;
- canonical lane-file path discovered from the index;
- exact `lane_id`;
- exact `media_type`;
- exact `schema_id`;
- exact `schema_version`; and
- optional `module_key`.
The map key produces `narratio.extraction.<key>`. Keys use Narratio's existing
path-safe configured-artifact key grammar. Keys, normalized source IDs, and
lane IDs must be non-empty and unique. Extraction source IDs must not collide
with built-ins or configured Scriptorium sources.
Every configured output is required for extraction-stage success. Operators
who need only a subset configure only that subset. Narratio does not add a
second `required_lanes` list or a runtime lane-selection flag.
Narratio does not configure Notarius lane topology, references, prompts, LLM
profiles, model settings, concurrency, retry behavior, or session IDs. One
stage execution runs the configured Notarius pipeline as a unit. Narratio does
not pass Notarius's `--session-id` override.
## Stage Outcomes And Resume
### Explicit Self-Skip
Extend the stage-result contract with an explicit disposition whose zero value
remains successful for backward compatibility. A stage may return:
- succeeded; or
- skipped with a stable reason.
When Notarius is absent or disabled, `extract` returns skipped with reason
`notarius_disabled`, no outputs, and bounded metadata. The runner records a
skipped session-stage and run-stage outcome rather than a successful empty
stage. Skipped stages are reconsidered on later invocations, so subsequently
enabling Notarius causes extraction to run without requiring force.
A genuine self-skip clears any older outputs for that stage before persisting
the new skipped state. Downstream consumers cannot resolve artifacts retained
from an earlier extraction after the stage is disabled.
### Resume Validation
Add a small optional resume-validation interface implemented by `extract`.
Before skipping an already-succeeded extract stage, the runner asks it whether
the recorded result remains resumable.
The resume validator confirms:
- Notarius is still enabled;
- recorded configuration identity matches the current config path, pipeline
ID, and output-contract map;
- the immutable bundle and canonical index still exist;
- every configured source is present in the succeeded extract record;
- lane files remain confined regular files;
- stored checksums still match; and
- stored descriptor contracts and external provenance remain compatible.
An ordinary contract mismatch returns a non-resumable decision with a bounded
reason. The runner marks the stage stale and executes it. An environmental
error that prevents making a safe decision returns an error and stops the run.
Narratio does not recursively interpret Notarius configuration, PromptKit
profiles, or reference files. Changes to those external inputs are therefore
not automatically detectable. Operator documentation must require
`--force extract` after changing them.
The configuration fingerprint covers the resolved binary, config path,
pipeline ID, timeout, working directory, and the deterministically sorted
output-contract map. It identifies Narratio's invocation contract, not the
transitive content of files owned by Notarius.
### Force And Invalidation
- Forcing `trim` or an earlier stage stales succeeded `extract` and all later
stages.
- Forcing `extract` stales succeeded `render`, `analyze`, `publish`, and
`notify` under the existing canonical-order rule.
- Forcing `render` does not stale `extract` because extraction precedes it.
- A failed, skipped, stale, or interrupted extract stage never supplies current
extraction sources.
## Bundle Storage And Commit
### Run-Local Execution
Notarius runs against a run-local output root beneath the Narratio run's
`extract` directory. Receipt bytes and stderr remain run-local diagnostics.
Failed and malformed bundles remain outside durable artifact storage for
inspection and never become current merely because files exist.
### Immutable Durable Bundles
After complete validation, promote the exact Notarius bundle into:
```text
artifacts/notarius/<narratio-run-id>/
```
The destination is unique and must not already exist. Promotion uses a sibling
temporary directory on the same filesystem, recursively copies only regular
files and directories, rejects symlinks and special files, preserves relative
layout, and renames the completed temporary tree into place.
The promoted tree preserves `index.json`, `manifest.json`, `rejected.json`,
`warnings.json`, `lanes/`, and any emitted pipeline-wide artifacts such as
`chunk-map.json` and `evidence-context.json`. Unknown regular files may be
preserved because the complete bundle is provenance, but no unknown file is
registered as a stable Narratio source.
There is no mutable filesystem `current` directory or symlink. The atomically
saved Narratio session manifest selects the current successful bundle. Older
successful bundles remain immutable until explicit cleanup policy removes
them.
The raw Notarius receipt is retained without rewriting its original
`output_directory`. Narratio's manifest records promoted durable paths.
## Artifact And Manifest Model
### Explicit Source Identity
Extend `artifacts.Ref` with an optional explicit `SourceID`. The application
runner prefers that value and retains existing stage-specific inference only
as a backward-compatible fallback. `extract` must not require another
stage-name special case in output mapping.
Add neutral optional artifact metadata models under `internal/artifactmodel`:
- contract metadata: media type, schema ID, schema version, and optional
module key; and
- external provenance: system, external run ID, pipeline ID, and external
artifact ID.
Both `artifacts.Ref` and `manifest.ArtifactRecord` carry these nested models.
Existing manifests remain readable because the fields are optional and use
`omitempty` encoding.
For a Notarius lane, external provenance uses:
- system `notarius`;
- receipt run ID;
- receipt pipeline ID; and
- lane ID as the external artifact ID.
### Registered Sources
For each configured output, `extract` records one artifact with:
- source ID `narratio.extraction.<key>`;
- durable lane path discovered through the promoted canonical index;
- producer stage and Narratio run ID;
- checksum;
- Notarius lane ID; and
- descriptor media type, schema identity/version, and module key when present.
- SHA-256 checksum;
- configured and observed contract metadata; and
- Notarius external provenance.
If the current manifest model cannot carry descriptor compatibility metadata,
extend its artifact metadata in a backward-tolerant way rather than encoding
that information in filenames or source IDs.
The canonical promoted `index.json` is also a stage output with kind
`notarius_index`, but it is not a selectable extraction source. Extract-stage
metadata records the durable bundle root, receipt path, Notarius validation
status, counts, warnings/rejections paths and summaries, the producing Narratio
run ID, and the normalized configured-contract fingerprint used by resume
validation.
Also record the canonical Notarius index as a stage output or stage metadata so
operators can discover the complete bundle, including non-lane artifacts. The
configured lane sources are the stable interface for analysis; the index and
bundle remain the provenance and inspection interface.
The producing Narratio run ID belongs to the successful extract result. It is
not compared with the session manifest's top-level `run_id`, which advances on
later invocations even when extraction is validly resumed. Resume and catalog
checks instead require the extract outputs, immutable bundle path, and stored
extract-stage producer identity to agree with one another.
## Analysis And Publish Integration
## Artifact Catalog And Resolution
Extend the runtime artifact catalog and configured Scriptorium input validation
so an enabled analysis artifact can declare, for example:
Add extraction as a first-class artifact-policy and runtime-catalog family:
- source kind `extraction`;
- canonical prefix `narratio.extraction.`;
- registration from `pipeline.notarius.outputs`; and
- manifest-backed availability from the current successful `extract` record.
Extraction availability is never inferred by scanning
`artifacts/notarius/`. A source is available only when:
- it is declared in current configuration;
- the session manifest records `extract` as succeeded and not stale;
- the exact matching source output is present;
- its durable path is confined and valid;
- its checksum matches; and
- its recorded contract and external provenance are compatible.
The catalog should expose one shared registration and hydration path used by
both `analyze` and `publish`. Avoid parallel extract-specific resolution logic
inside each stage.
## Analyze Integration
An enabled Scriptorium artifact may select an extraction source through the
existing input contract:
```yaml
inputs:
npc_registry:
npcs:
source: narratio.extraction.npc_registry
required: true
```
Resolution must remain manifest-first and verify that the recorded artifact
was produced by a successful current `extract` stage. A required extraction
source that is unavailable must fail analysis with guidance to configure or
rerun `extract`; an optional source may be omitted according to the existing
Scriptorium input contract.
Required missing extraction inputs fail with guidance to enable/configure or
rerun `extract`. Optional missing inputs follow the existing Scriptorium input
contract.
Publish source resolution should accept configured
`narratio.extraction.<output-key>` sources through the same artifact catalog so
operators may publish selected structured artifacts without manually copying
paths. The existing `--artifacts` flag remains scoped to Scriptorium artifact
selection and must not partially execute the Notarius pipeline.
Narratio never injects every extraction output into every analysis. Each
Scriptorium artifact chooses the smallest useful set. This limits context
size, cost, and the risk of treating derived claims as transcript authority.
Analysis prompts should continue to treat the transcript as authoritative and
Notarius artifacts as structured, cited, derived evidence.
No current-session analysis artifact should consume an incidental file from a
failed, stale, skipped, or superseded extraction run.
The existing `--artifacts` selection remains scoped to Scriptorium artifacts.
It does not select Notarius lanes or partially run the Notarius pipeline.
## Failure, Skip, Resume, And Diagnostics
## Publish Integration
- Missing or invalid enabled Notarius configuration fails configuration
validation before stage execution where statically discoverable.
- A disabled or absent Notarius configuration makes `extract` skip with clear
stage metadata and no new outputs.
- A missing or invalid final trimmed transcript fails `extract` before starting
Notarius.
- Preflight failure, nonzero Notarius exit, cancellation, timeout, malformed or
unsupported receipt/index data, unsafe paths, incompatible descriptors,
rejected required outputs, or missing configured lanes fails the entire
stage.
- Process success does not override Narratio's required-output policy.
- A failed run retains bounded run-local receipt bytes, stderr, and the
unpublished Notarius bundle for diagnosis, subject to Narratio's existing
sensitive-data and cleanup policies.
- The canonical bundle and manifest artifacts are updated only after complete
validation and materialization.
- Resume skips a succeeded, non-stale `extract` stage only when its
manifest-recorded canonical index and configured lane outputs still validate.
- Force and staleness behavior follows the ordinary stage contract; it must not
depend on merely finding `artifacts/notarius/` on disk.
Publish source validation and resolution accept configured
`narratio.extraction.<key>` sources through the shared runtime artifact
catalog. Operators may publish individual structured lanes without manually
copying files.
Transcripts, Notarius outputs, evidence context, manifests, receipts, and logs
are private campaign material. Subprocess arguments and manifest metadata must
not contain secrets. Credentials remain in the environment or in mechanisms
owned by Notarius and PromptKit.
The Notarius bundle is not automatically published wholesale. Bundle files or
lanes are published only through explicit configured publish rules. Existing
publish locking, destination safety, commit ordering, and required/unselected
artifact behavior remain unchanged.
## Maintained D&D Example
## Failure And Diagnostic Semantics
Add or update a Narratio example that enables Notarius's complete
`dnd-session` pipeline and maps these ten required lanes to stable extraction
keys:
The stage fails before invoking Notarius when enabled configuration or the
final trimmed transcript is invalid.
| Output key | Notarius lane ID |
The stage fails after invocation for:
- cancellation or timeout;
- nonzero process exit;
- malformed, oversized, or unsupported receipt data;
- malformed or unsupported index data;
- unsafe receipt, descriptor, or filesystem paths;
- symlinks or special files in the promoted bundle;
- mismatched receipt pipeline identity;
- missing configured lanes;
- duplicate lane descriptors;
- rejected required lanes;
- incompatible media type, schema identity/version, or module key;
- empty or syntactically invalid required JSON payloads;
- checksum or promotion failure; or
- manifest persistence failure.
Process success alone does not establish consumer success. Notarius may exit
zero while omitting or rejecting a lane, and Narratio's configured required
set remains authoritative.
Failure retains bounded receipt and stderr diagnostics plus the run-local
bundle where available. Durable artifact storage and extraction source records
are updated only after complete validation and promotion. A promoted unique
bundle whose later manifest save fails is unreferenced and may be reclaimed by
explicit cleanup; it is never inferred as current.
## Security And Privacy
Transcripts, Notarius lanes, evidence context, manifests, receipts, warnings,
rejections, debug data, and logs are private campaign material.
- Secrets do not appear in command arguments, generated configuration,
artifact metadata, logs, examples, or documentation.
- Credentials continue to enter through Notarius and PromptKit's documented
environment or secret mechanisms.
- Receipt and index paths are untrusted external input until confined.
- Recursive promotion never follows symlinks or copies special files.
- Diagnostics remain bounded and do not echo payload bodies.
- Automatic cleanup follows Narratio's existing post-publish gates and path
safety rules; it does not silently remove immutable extraction bundles
outside an explicit covered policy.
## Maintained Complete D&D Example
Add a maintained example that enables Notarius's complete `dnd-session`
pipeline and maps these required lanes:
| Extraction key | Notarius lane ID |
| --- | --- |
| `item_registry` | `item-registry` |
| `npc_registry` | `npc-registry` |
@@ -274,106 +505,100 @@ keys:
| `location_occurrences` | `location-occurrences` |
| `enemy_events` | `enemy-events` |
The example must include each lane's current media type and schema identity
from Notarius's published contracts. It should also demonstrate at least one
Scriptorium analysis artifact consuming one or more
`narratio.extraction.*` sources. The example must use placeholders and relative
paths suitable for the example tree, contain no credentials, and pass the
repository's configuration validation tests.
The example obtains exact media types and schema identities from Notarius's
published contracts at implementation time. It demonstrates at least one
Scriptorium artifact consuming a small, purpose-specific subset of extraction
sources. It uses only placeholders and repository-relative example paths,
contains no credentials, and passes maintained example validation.
## Compatibility Policy
## Documentation Deliverables
The initial integration baseline is the public subprocess contract available
in Notarius v0.3.0:
When implementation lands, update current-behavior documentation in the same
change:
- successful JSON receipt schema `notarius.run-result.v1`;
- production JSON bundle discovery through `index.json`; and
- the schema IDs and versions explicitly configured for required lanes.
Runtime compatibility should be decided from those published contracts, not
from textual parsing of `notarius --version`. New optional receipt or index
fields must not break Narratio. An unsupported receipt version or lane schema
must fail before the artifact is registered for analysis.
## Documentation Deliverables When Implemented
Update current-behavior documentation in the same change that implements the
feature:
- add `docs/integrations/notarius.md` for the external CLI, receipt, bundle,
and adapter contract, linking to Notarius's canonical documentation;
- add `docs/internal/stage-extract.md` for stage inputs, outputs, collaborators,
state transitions, failures, and focused tests;
- update `docs/internal/adapters.md`, `docs/internal/artifacts.md`,
`docs/internal/manifest.md`, and the internal stage inventory;
- update `docs/policy/architecture.md` to list Notarius among isolated external
systems and preserve the adapter/stage boundary;
- add `docs/integrations/notarius.md` for the consumed CLI, receipt, index, and
compatibility contract, linking to Notarius's canonical documentation;
- add `docs/internal/stage-extract.md` for stage flow, collaborators, state,
failures, resume validation, and focused tests;
- update `docs/internal/overview.md`, `docs/internal/adapters.md`,
`docs/internal/artifacts.md`, `docs/internal/manifest.md`, and
`docs/internal/workspace.md` within their canonical scopes;
- update `docs/policy/architecture.md` for the Notarius boundary and explicit
skipped/resume-validation contracts;
- update `docs/config.md`, `docs/cli.md`, `docs/operations.md`,
`docs/troubleshooting.md`, `README.md`, and maintained examples only to the
extent their canonical scopes require; and
- update `docs/development.md` only to the extent its canonical contributor
routing scope requires.
`docs/troubleshooting.md`, `README.md`, and maintained examples only within
their canonical scopes; and
- update `docs/development.md` only if its contributor routing changes.
Outside this roadmap, do not describe `extract`, Notarius configuration, or
`narratio.extraction.*` sources as implemented until the code exists.
Outside this roadmap, do not describe the feature as implemented until its
code and documentation are complete.
## Testing And Validation Expectations
## Testing Expectations
Implementation should provide focused tests for:
Tests should protect contracts and meaningful risks rather than private helper
structure. The implementation plan assigns detailed ownership, with coverage
for:
- strict configuration decoding, defaults, required fields, path resolution,
output-map validation, normalized-key collisions, and example loading;
- exact stage order, selection, downstream staleness, resume, force, and
prerequisite behavior;
- adapter command construction, deterministic working directory, environment
inheritance, stdout/stderr separation, cancellation, timeout, and nonzero
exits;
- supported and unsupported receipt versions, unknown optional fields,
malformed receipts, index decoding, and path escapes at every boundary;
- descriptor lookup by lane ID rather than filename, expected metadata checks,
missing/rejected configured lanes, and tolerated unconfigured lanes;
- run-local execution, transactional canonical-bundle replacement, checksums,
failed-run preservation, and manifest recording;
- artifact-catalog resolution from `narratio.extraction.*` into analysis and
publish, including required, optional, missing, stale, and skipped cases; and
- end-to-end stage execution with a fake Notarius adapter, without live LLM or
external subprocess requirements in the ordinary test suite.
- strict configuration, defaults, normalization, cross-source validation, and
maintained examples;
- stage result dispositions and clearing of self-skipped outputs;
- backward-compatible artifact metadata serialization;
- exact adapter arguments, streams, cancellation, timeout, exit behavior,
receipt/index compatibility, and every path-confinement boundary;
- recursive promotion safety, atomic visibility, cleanup on failure, symlink
rejection, and immutable destination behavior;
- required-lane policy, descriptor compatibility, JSON syntax, checksums, and
provenance;
- canonical order, single-stage selection, force, staleness, resume
validation, and enable-after-skip behavior;
- manifest-backed extraction catalog resolution for required, optional,
missing, stale, skipped, incompatible, and tampered artifacts;
- analyze and publish integration without automatic lane injection; and
- representative assembled execution with a fake Notarius runner and no live
LLM, credentials, or external subprocess in the ordinary test suite.
Run the repository-wide Go tests, vet, build, and maintained example validation
after focused tests pass.
Repository-wide tests, vet, build, and maintained-example validation are
required after focused tests pass.
## Acceptance Criteria
- `extract` is a first-class transactional stage between `render` and
`analyze` everywhere Narratio models stage order or state.
- Narratio invokes Notarius only through a narrow, tested adapter.
- The stage consumes the manifest-resolved final trimmed Seriatim transcript.
- The Notarius configuration remains owned by Notarius; Narratio configures
only invocation and its downstream consumer contract.
- Every configured output is discovered through the receipt and `index.json`,
contract-checked, materialized transactionally, and recorded with a stable
`narratio.extraction.*` source ID.
- The complete D&D example maps all ten current lanes and passes strict config
validation.
- Analysis can consume extraction sources through the existing artifact input
model, and publish can select them through the artifact catalog.
- Failed, partial, rejected, unsafe, stale, or incompatible output never becomes
a current analysis input.
- Resume and force behavior remains manifest-driven.
- Documentation accurately describes the implemented stage, adapter,
configuration, operations, and compatibility boundary without duplicating
Notarius's canonical schemas.
- `extract` is a first-class stage between `trim` and `render` everywhere
Narratio models stage order and lifecycle.
- Narratio invokes Notarius only through a narrow tested adapter.
- Extraction consumes the manifest-resolved final trimmed Seriatim JSON.
- Disabled extraction is recorded as skipped and runs normally if later
enabled.
- Successful resume requires valid manifest-recorded immutable outputs rather
than filesystem presence alone.
- The complete bundle is promoted to a unique immutable directory without
following symlinks or exposing a partial destination.
- Every configured lane is discovered by lane ID, contract-checked, checksummed,
and recorded with explicit source identity and external provenance.
- Analysis and publish resolve extraction sources only from a current
successful extract manifest record.
- Failed, skipped, stale, partial, rejected, unsafe, incompatible, or tampered
output never becomes a current input.
- The complete D&D example maps all ten current lanes and demonstrates curated
analysis inputs.
- Tests remain deterministic, offline, and independent of real Notarius,
PromptKit, LLM providers, and credentials.
- Current-behavior documentation is updated only as implementation becomes
complete.
## Non-Goals
- Reimplementing Notarius extraction, prompts, schemas, references, retries,
profiles, or lane orchestration in Narratio.
- Allowing one Narratio run to invoke arbitrary extractor programs or multiple
Notarius pipelines.
- Making `extract` a configurable DAG or folding it into the Scriptorium
`analyze` stage.
- Partially selecting Notarius lanes through Narratio's `--artifacts` flag.
- Decoding D&D payload bodies in the generic Notarius adapter.
- Supporting previous-session extraction artifacts in the initial feature.
- Reimplementing Notarius extraction, configuration, schemas, prompts,
references, retries, profiles, validation, or lane orchestration.
- Supporting arbitrary extractor programs or multiple Notarius pipelines in
one Narratio run.
- Turning canonical stage execution into a DAG or generic workflow engine.
- Folding extraction into `analyze` or Scriptorium.
- Automatically injecting all structured outputs into every prompt.
- Selecting Notarius lanes through Narratio's `--artifacts` flag.
- Decoding D&D payload bodies in the generic adapter or stage.
- Automatically detecting changes throughout Notarius's referenced config,
PromptKit profile, and campaign-reference closure.
- Supporting previous-session extraction sources in the initial feature.
- Automatically publishing the complete Notarius bundle.
- Requiring live Notarius, PromptKit, an LLM provider, or external services in
the ordinary unit test suite.
the ordinary test suite.

98
internal/app/runner.go
View File

@@ -195,6 +195,9 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "running", "path", manifestPath)
result, err := s.Run(ctx, stageEnv, m)
if err == nil {
err = validateStageResult(result)
}
if err != nil {
failedAt := nowUTC()
m.MarkStageFailed(s.Name(), failedAt, err.Error())
@@ -209,6 +212,25 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
}
if result != nil && result.Disposition == stage.StageDispositionSkipped {
skipped = append(skipped, s.Name())
skippedAt := nowUTC()
m.MarkStageSkipped(s.Name(), skippedAt, result.SkipReason)
clearStageResultDetails(m.Stages[s.Name()])
applyStageResultToManifest(m, s.Name(), result)
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
return nil, fmt.Errorf("save manifest after self-skip %q: %w", s.Name(), err)
}
runManifest.MarkStageSkipped(s.Name(), skippedAt, result.SkipReason)
applyStageResultToRunManifest(runManifest, s.Name(), result)
syncRunManifestIdentityFromSession(m, runManifest)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save run manifest after self-skip %q: %w", s.Name(), err)
}
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "skipped", "path", manifestPath)
env.Logger.Info("stage skipped", "stage", s.Name(), "reason", result.SkipReason)
continue
}
outputs := mapResultOutputs(s.Name(), result, runID)
succeededAt := nowUTC()
@@ -407,26 +429,69 @@ func mapResultOutputs(stageName string, result *stage.StageResult, runID string)
localPath = ref.RelativePath
}
kind := ref.Kind
sourceID := ""
if stageName == "analyze" {
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
kind = "scriptorium_artifact"
} else {
sourceID = sourceIDForOutputKind(kind)
sourceID := strings.TrimSpace(ref.SourceID)
if sourceID == "" {
if stageName == "analyze" {
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
kind = "scriptorium_artifact"
} else {
sourceID = sourceIDForOutputKind(kind)
}
}
out = append(out, manifest.ArtifactRecord{
Kind: kind,
SourceID: sourceID,
LocalPath: localPath,
ProducerRunID: runID,
RemoteKey: ref.RemoteKey,
Checksum: ref.Checksum,
Kind: kind,
SourceID: sourceID,
LocalPath: localPath,
Contract: cloneContractMetadata(ref.Contract),
ExternalProvenance: cloneExternalProvenance(ref.ExternalProvenance),
ProducerRunID: runID,
RemoteKey: ref.RemoteKey,
Checksum: ref.Checksum,
})
}
return out
}
func validateStageResult(result *stage.StageResult) error {
if result == nil {
return nil
}
switch result.Disposition {
case stage.StageDispositionSucceeded:
if strings.TrimSpace(result.SkipReason) != "" {
return fmt.Errorf("successful result contains a skip reason")
}
return nil
case stage.StageDispositionSkipped:
if strings.TrimSpace(result.SkipReason) == "" {
return fmt.Errorf("skipped result requires a skip reason")
}
if len(result.Outputs) != 0 {
return fmt.Errorf("skipped result contains %d output(s)", len(result.Outputs))
}
return nil
default:
return fmt.Errorf("unsupported stage result disposition %q", result.Disposition)
}
}
func cloneContractMetadata(value *artifactmodel.ContractMetadata) *artifactmodel.ContractMetadata {
if value == nil {
return nil
}
cloned := *value
return &cloned
}
func cloneExternalProvenance(value *artifactmodel.ExternalProvenance) *artifactmodel.ExternalProvenance {
if value == nil {
return nil
}
cloned := *value
return &cloned
}
func sourceIDForOutputKind(kind string) string {
trimmed := strings.TrimSpace(kind)
if trimmed == "" {
@@ -462,6 +527,15 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *
}
}
func clearStageResultDetails(sr *manifest.StageRecord) {
if sr == nil {
return
}
sr.Logs = nil
sr.GeneratedConfigs = nil
sr.Metadata = nil
}
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest, runID string) (bool, error) {
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil {
return false, nil

195
internal/app/runner_test.go
View File

@@ -16,6 +16,7 @@ import (
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
@@ -38,6 +39,25 @@ type countingStage struct {
runs *int
}
type resultStage struct {
name string
result *stage.StageResult
runs *int
order *[]string
}
func (s resultStage) Name() string { return s.name }
func (s resultStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s resultStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
if s.runs != nil {
*s.runs = *s.runs + 1
}
if s.order != nil {
*s.order = append(*s.order, s.name)
}
return s.result, nil
}
func (s countingStage) Name() string { return s.name }
func (s countingStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s countingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
@@ -199,6 +219,61 @@ func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T)
}
}
func TestMapResultOutputsPrefersExplicitSourceAndCopiesMetadata(t *testing.T) {
contract := &artifactmodel.ContractMetadata{
MediaType: "application/json",
SchemaID: "notarius.dnd.npc_registry",
SchemaVersion: "v1",
}
provenance := &artifactmodel.ExternalProvenance{
System: "notarius",
RunID: "external-run",
PipelineID: "dnd-session",
ArtifactID: "npc-registry",
}
result := &stage.StageResult{Outputs: []artifacts.Ref{{
Kind: "structured_data",
SourceID: "narratio.example.npcs",
RelativePath: "artifacts/npcs.json",
Contract: contract,
ExternalProvenance: provenance,
}}}
got := mapResultOutputs("analyze", result, "narratio-run")
if len(got) != 1 {
t.Fatalf("outputs len = %d, want 1", len(got))
}
if got[0].SourceID != "narratio.example.npcs" {
t.Fatalf("source_id = %q, want explicit source", got[0].SourceID)
}
if got[0].Kind != "structured_data" {
t.Fatalf("kind = %q, want explicit output kind preserved", got[0].Kind)
}
if got[0].Contract == nil || *got[0].Contract != *contract {
t.Fatalf("contract = %#v, want %#v", got[0].Contract, contract)
}
if got[0].ExternalProvenance == nil || *got[0].ExternalProvenance != *provenance {
t.Fatalf("external provenance = %#v, want %#v", got[0].ExternalProvenance, provenance)
}
if got[0].Contract == contract || got[0].ExternalProvenance == provenance {
t.Fatal("mapped metadata should not alias the stage result")
}
}
func TestMapResultOutputsRetainsFallbackInference(t *testing.T) {
transcript := mapResultOutputs("trim", &stage.StageResult{Outputs: []artifacts.Ref{{
Kind: artifactmodel.TranscriptOutputKindFinalTrimmed,
}}}, "run-id")
if len(transcript) != 1 || transcript[0].SourceID != artifacts.ArtifactTranscriptFinalTrimmed {
t.Fatalf("transcript fallback = %#v, want final-trimmed source", transcript)
}
analyze := mapResultOutputs("analyze", &stage.StageResult{Outputs: []artifacts.Ref{{Kind: "session_recap"}}}, "run-id")
if len(analyze) != 1 || analyze[0].SourceID != "narratio.artifact.session_recap" || analyze[0].Kind != "scriptorium_artifact" {
t.Fatalf("analyze fallback = %#v, want configured artifact inference", analyze)
}
}
func TestNeedsObjectStoreForRunPrepareWithPreviousRequirements(t *testing.T) {
tests := []struct {
name string
@@ -738,6 +813,126 @@ func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
}
}
func TestExecuteStagesPersistsSelfSkipAndContinues(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
manifestPath := manifestPathFor(cfg)
seed := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
seed.MarkStageSucceeded("optional", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{{
Kind: "old_output",
SourceID: "narratio.example.old",
LocalPath: "artifacts/old.json",
}})
seed.Stages["optional"].Logs = []string{"old.log"}
seed.Stages["optional"].GeneratedConfigs = []string{"old.yml"}
seed.Stages["optional"].Metadata = map[string]any{"old": true}
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
t.Fatalf("Save() seed manifest error = %v", err)
}
order := []string{}
optionalRuns := 0
stages := []stage.Stage{
resultStage{
name: "optional",
runs: &optionalRuns,
order: &order,
result: &stage.StageResult{
Disposition: stage.StageDispositionSkipped,
SkipReason: "integration_disabled",
Logs: []string{"runs/current/optional.log"},
GeneratedConfigs: []string{"runs/current/optional.yml"},
Metadata: map[string]any{"enabled": false},
},
},
resultStage{name: "later", order: &order, result: &stage.StageResult{}},
}
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{Force: true})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if strings.Join(order, ",") != "optional,later" {
t.Fatalf("execution order = %v, want optional then later", order)
}
if len(summary.Executed) != 2 || len(summary.Skipped) != 1 || summary.Skipped[0] != "optional" {
t.Fatalf("summary = %#v, want optional executed and self-skipped before later", summary)
}
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
if err != nil {
t.Fatalf("Load() session manifest error = %v", err)
}
selfSkipped := sessionManifest.Stages["optional"]
if selfSkipped == nil || selfSkipped.Status != manifest.StatusSkipped {
t.Fatalf("optional stage = %#v, want skipped", selfSkipped)
}
if len(selfSkipped.Outputs) != 0 {
t.Fatalf("optional outputs = %#v, want old outputs cleared", selfSkipped.Outputs)
}
if selfSkipped.Error == nil || selfSkipped.Error.Message != "integration_disabled" {
t.Fatalf("optional skip reason = %#v, want integration_disabled", selfSkipped.Error)
}
if len(selfSkipped.Logs) != 1 || selfSkipped.Logs[0] != "runs/current/optional.log" ||
len(selfSkipped.GeneratedConfigs) != 1 || selfSkipped.GeneratedConfigs[0] != "runs/current/optional.yml" ||
selfSkipped.Metadata["enabled"] != false || selfSkipped.Metadata["old"] != nil {
t.Fatalf("optional result details = %#v, want current bounded diagnostics and metadata", selfSkipped)
}
if later := sessionManifest.Stages["later"]; later == nil || later.Status != manifest.StatusSucceeded {
t.Fatalf("later stage = %#v, want succeeded", later)
}
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
if err != nil {
t.Fatalf("LoadRun() error = %v", err)
}
runStage := runManifest.Stages["optional"]
if runStage == nil || runStage.Action != manifest.RunStageActionRun || runStage.Status != manifest.StatusSkipped {
t.Fatalf("run optional stage = %#v, want run action with skipped status", runStage)
}
if len(runStage.Logs) != 1 || runStage.Metadata["enabled"] != false {
t.Fatalf("run optional stage details = %#v, want result diagnostics and metadata", runStage)
}
_, err = executeStages(context.Background(), cfg, []stage.Stage{stages[0]}, RunOptions{})
if err != nil {
t.Fatalf("second executeStages() error = %v", err)
}
if optionalRuns != 2 {
t.Fatalf("optional runs = %d, want self-skipped stage reconsidered", optionalRuns)
}
}
func TestExecuteStagesRejectsSkippedResultWithOutputs(t *testing.T) {
cfg := testConfig(t)
invalid := resultStage{name: "optional", result: &stage.StageResult{
Disposition: stage.StageDispositionSkipped,
SkipReason: "integration_disabled",
Outputs: []artifacts.Ref{{Kind: "unexpected"}},
}}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{invalid}, RunOptions{})
if err == nil {
t.Fatal("executeStages() error = nil, want invalid skipped result failure")
}
if summary != nil {
t.Fatalf("summary = %#v, want nil", summary)
}
if !strings.Contains(err.Error(), "skipped result contains 1 output") {
t.Fatalf("error = %q, want skipped-output validation", err)
}
loaded, loadErr := (&manifest.LocalStore{}).Load(context.Background(), manifestPathFor(cfg))
if loadErr != nil {
t.Fatalf("Load() session manifest error = %v", loadErr)
}
if got := loaded.Stages["optional"]; got == nil || got.Status != manifest.StatusFailed {
t.Fatalf("optional stage = %#v, want failed", got)
}
}
func TestExecuteStagesRunLocalArtifactsAndCanonicalSync(t *testing.T) {
cfg := testConfig(t)
stages := []stage.Stage{

17
internal/artifactmodel/metadata.go Normal file
View File

@@ -0,0 +1,17 @@
package artifactmodel
// ContractMetadata identifies the data contract implemented by an artifact.
type ContractMetadata struct {
MediaType string `json:"media_type"`
SchemaID string `json:"schema_id"`
SchemaVersion string `json:"schema_version"`
ModuleKey string `json:"module_key,omitempty"`
}
// ExternalProvenance identifies an artifact produced by an external system.
type ExternalProvenance struct {
System string `json:"system"`
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
ArtifactID string `json:"artifact_id"`
}

56
internal/artifactmodel/metadata_test.go Normal file
View File

@@ -0,0 +1,56 @@
package artifactmodel
import (
"encoding/json"
"strings"
"testing"
)
func TestArtifactMetadataJSON(t *testing.T) {
type envelope struct {
Contract *ContractMetadata `json:"contract,omitempty"`
ExternalProvenance *ExternalProvenance `json:"external_provenance,omitempty"`
}
complete, err := json.Marshal(envelope{
Contract: &ContractMetadata{
MediaType: "application/json",
SchemaID: "notarius.dnd.npc_registry",
SchemaVersion: "v1",
ModuleKey: "dnd/npc-registry",
},
ExternalProvenance: &ExternalProvenance{
System: "notarius",
RunID: "run-123",
PipelineID: "dnd-session",
ArtifactID: "npc-registry",
},
})
if err != nil {
t.Fatalf("Marshal() complete metadata error = %v", err)
}
wantComplete := `{"contract":{"media_type":"application/json","schema_id":"notarius.dnd.npc_registry","schema_version":"v1","module_key":"dnd/npc-registry"},"external_provenance":{"system":"notarius","run_id":"run-123","pipeline_id":"dnd-session","artifact_id":"npc-registry"}}`
if string(complete) != wantComplete {
t.Fatalf("complete metadata JSON = %s, want %s", complete, wantComplete)
}
omitted, err := json.Marshal(envelope{})
if err != nil {
t.Fatalf("Marshal() omitted metadata error = %v", err)
}
if string(omitted) != `{}` {
t.Fatalf("omitted metadata JSON = %s, want {}", omitted)
}
withoutModule, err := json.Marshal(envelope{Contract: &ContractMetadata{
MediaType: "application/json",
SchemaID: "notarius.dnd.npc_registry",
SchemaVersion: "v1",
}})
if err != nil {
t.Fatalf("Marshal() contract without module key error = %v", err)
}
if strings.Contains(string(withoutModule), "module_key") {
t.Fatalf("contract JSON unexpectedly contains omitted module_key: %s", withoutModule)
}
}

23
internal/artifacts/store.go
View File

@@ -1,16 +1,23 @@
package artifacts
import "os"
import (
"os"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
)
// Ref identifies a pipeline artifact and its local/remote coordinates.
type Ref struct {
Kind string
Category string
SessionID string
RelativePath string
AbsolutePath string
RemoteKey string
Checksum string
Kind string
SourceID string
Category string
SessionID string
RelativePath string
AbsolutePath string
RemoteKey string
Checksum string
Contract *artifactmodel.ContractMetadata
ExternalProvenance *artifactmodel.ExternalProvenance
}
// Store is the local artifact/workdir abstraction used by orchestration code.

11
internal/manifest/manifest.go
View File

@@ -3,6 +3,8 @@ package manifest
import (
"strings"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
)
// ErrorRecord captures structured error metadata at run or stage scope.
@@ -28,9 +30,11 @@ type InputRecord struct {
// ArtifactRecord captures one produced artifact and optional remote metadata.
type ArtifactRecord struct {
Kind string `json:"kind"`
SourceID string `json:"source_id,omitempty"`
LocalPath string `json:"local_path"`
Kind string `json:"kind"`
SourceID string `json:"source_id,omitempty"`
LocalPath string `json:"local_path"`
Contract *artifactmodel.ContractMetadata `json:"contract,omitempty"`
ExternalProvenance *artifactmodel.ExternalProvenance `json:"external_provenance,omitempty"`
// ProducerRunID identifies the run that produced this durable artifact.
ProducerRunID string `json:"producer_run_id,omitempty"`
RemoteKey string `json:"remote_key,omitempty"`
@@ -120,6 +124,7 @@ func (m *Manifest) MarkStageSkipped(name string, at time.Time, reason string) {
s.Status = StatusSkipped
s.CompletedAt = timePtr(at)
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)}
s.Outputs = nil
s.UpdatedAt = at
m.UpdatedAt = at
}

20
internal/manifest/manifest_test.go
View File

@@ -110,3 +110,23 @@ func TestMarkStageSucceededClearsError(t *testing.T) {
t.Fatalf("error = %#v, want nil on success", stage.Error)
}
}
func TestMarkStageSkippedClearsEarlierOutputs(t *testing.T) {
m := New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
m.MarkStageSucceeded("extract", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), []ArtifactRecord{
{Kind: "structured_data", SourceID: "narratio.example.characters", LocalPath: "artifacts/characters.json"},
})
m.MarkStageSkipped("extract", time.Date(2026, 5, 3, 10, 2, 0, 0, time.UTC), "integration_disabled")
stage := m.Stages["extract"]
if stage == nil {
t.Fatal("missing stage record")
}
if stage.Status != StatusSkipped {
t.Fatalf("status = %q, want %q", stage.Status, StatusSkipped)
}
if len(stage.Outputs) != 0 {
t.Fatalf("outputs = %#v, want cleared", stage.Outputs)
}
}

1
internal/manifest/run_manifest.go
View File

@@ -119,6 +119,7 @@ func (m *RunManifest) MarkStageSkipped(name string, at time.Time, reason string)
s.Status = StatusSkipped
s.CompletedAt = timePtr(at)
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)}
s.Outputs = nil
s.UpdatedAt = at
m.UpdatedAt = at
}

72
internal/manifest/store_test.go
View File

@@ -8,6 +8,8 @@ import (
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
)
func TestLocalStoreCreateSaveLoadRoundTrip(t *testing.T) {
@@ -64,6 +66,76 @@ func TestLocalStoreCreateSaveLoadRoundTrip(t *testing.T) {
}
}
func TestLocalStoreArtifactMetadataCompatibility(t *testing.T) {
store := &LocalStore{}
ctx := context.Background()
dir := t.TempDir()
oldPath := filepath.Join(dir, "old-manifest.json")
oldJSON := `{
"session_id": "2026-05-03",
"created_at": "2026-05-03T10:00:00Z",
"updated_at": "2026-05-03T10:01:00Z",
"stages": {
"analyze": {
"name": "analyze",
"status": "succeeded",
"created_at": "2026-05-03T10:00:00Z",
"updated_at": "2026-05-03T10:01:00Z",
"outputs": [{"kind":"scriptorium_artifact","local_path":"artifacts/recap.md"}]
}
}
}`
if err := os.WriteFile(oldPath, []byte(oldJSON), 0o644); err != nil {
t.Fatalf("WriteFile() old manifest error = %v", err)
}
loaded, err := store.Load(ctx, oldPath)
if err != nil {
t.Fatalf("Load() old manifest error = %v", err)
}
oldOutput := loaded.Stages["analyze"].Outputs[0]
if oldOutput.Contract != nil || oldOutput.ExternalProvenance != nil {
t.Fatalf("old output metadata = %#v, %#v; want nil", oldOutput.Contract, oldOutput.ExternalProvenance)
}
if err := store.Save(ctx, oldPath, loaded); err != nil {
t.Fatalf("Save() old manifest error = %v", err)
}
roundTripped, err := os.ReadFile(oldPath)
if err != nil {
t.Fatalf("ReadFile() round-tripped old manifest error = %v", err)
}
if strings.Contains(string(roundTripped), `"contract"`) || strings.Contains(string(roundTripped), `"external_provenance"`) {
t.Fatalf("old manifest gained fabricated metadata:\n%s", roundTripped)
}
loaded.Stages["analyze"].Outputs[0].Contract = &artifactmodel.ContractMetadata{
MediaType: "application/json",
SchemaID: "example.recap",
SchemaVersion: "v1",
}
loaded.Stages["analyze"].Outputs[0].ExternalProvenance = &artifactmodel.ExternalProvenance{
System: "example",
RunID: "external-run",
PipelineID: "pipeline",
ArtifactID: "recap",
}
metadataPath := filepath.Join(dir, "metadata-manifest.json")
if err := store.Save(ctx, metadataPath, loaded); err != nil {
t.Fatalf("Save() metadata manifest error = %v", err)
}
withMetadata, err := store.Load(ctx, metadataPath)
if err != nil {
t.Fatalf("Load() metadata manifest error = %v", err)
}
got := withMetadata.Stages["analyze"].Outputs[0]
if got.Contract == nil || got.Contract.SchemaID != "example.recap" {
t.Fatalf("contract = %#v, want persisted contract", got.Contract)
}
if got.ExternalProvenance == nil || got.ExternalProvenance.RunID != "external-run" {
t.Fatalf("external provenance = %#v, want persisted provenance", got.ExternalProvenance)
}
}
func TestLocalStoreSaveUpdatesTimestamp(t *testing.T) {
store := &LocalStore{}
ctx := context.Background()

11
internal/stage/stage.go
View File

@@ -44,8 +44,19 @@ type Stage interface {
Run(ctx context.Context, env *Env, m *manifest.Manifest) (*StageResult, error)
}
// StageDisposition describes the outcome of a stage that returned without an error.
type StageDisposition string
const (
// StageDispositionSucceeded is the zero value so existing stages remain successful.
StageDispositionSucceeded StageDisposition = ""
StageDispositionSkipped StageDisposition = "skipped"
)
// StageResult is the declared output of a stage execution.
type StageResult struct {
Disposition StageDisposition
SkipReason string
Outputs []artifacts.Ref
Logs []string
GeneratedConfigs []string