Added a roadmap for new work to support configurable artifacts defined at runtime
This commit is contained in:
502
docs/roadmap/runtime-artifacts.md
Normal file
502
docs/roadmap/runtime-artifacts.md
Normal file
@@ -0,0 +1,502 @@
|
||||
# Roadmap: Runtime-Defined Scriptorium Artifacts
|
||||
|
||||
## Status
|
||||
|
||||
Proposed implementation roadmap.
|
||||
|
||||
## Purpose
|
||||
|
||||
Narratio currently treats artifact generation as a narrow `analyze` stage that supports a hard-coded `session_recap` artifact. This roadmap describes how to generalize artifact generation so operators can define multiple Scriptorium-backed output artifacts at runtime through `pipeline.yml`.
|
||||
|
||||
The goal is to let Narratio continue acting as an orchestrator while making session artifacts configurable, composable, resumable, and visible through a unified artifact model.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
Operators should be able to define artifacts such as session recaps, player handouts, NPC summaries, quest logs, entity maps, or other campaign-specific outputs without changing Narratio code.
|
||||
|
||||
A configured artifact should be declared under `pipeline.scriptorium.artifacts.<name>` and should define, at minimum:
|
||||
|
||||
- whether it is enabled;
|
||||
- which Scriptorium prompt to run;
|
||||
- where the output should be written;
|
||||
- which Narratio artifacts should be passed as Scriptorium inputs;
|
||||
- which static vars should be passed to Scriptorium.
|
||||
|
||||
Configured artifacts should become canonical runtime artifact IDs using this form:
|
||||
|
||||
```text
|
||||
narratio.artifact.<artifact_name>
|
||||
```
|
||||
|
||||
For example, an artifact declared as:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
```
|
||||
|
||||
should be registered as:
|
||||
|
||||
```text
|
||||
narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
Other configured artifacts should then be able to use it as an input:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd_session.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
```
|
||||
|
||||
## Scope
|
||||
|
||||
This roadmap covers:
|
||||
|
||||
- introducing a runtime artifact catalog;
|
||||
- generalizing configured Scriptorium artifact execution;
|
||||
- supporting `narratio.artifact.<name>` source IDs;
|
||||
- adding explicit artifact dependencies;
|
||||
- recording dynamic artifacts in stage metadata and the session manifest;
|
||||
- preserving compatibility for the existing `session_recap` behavior;
|
||||
- updating tests and documentation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This roadmap does not attempt to turn Narratio into a general workflow engine.
|
||||
|
||||
Specifically, this feature should not add:
|
||||
|
||||
- arbitrary shell-command artifacts;
|
||||
- multi-stage user-defined workflows;
|
||||
- conditional branching;
|
||||
- loops;
|
||||
- remote artifact discovery beyond existing archive/session behavior;
|
||||
- semantic understanding of each configured artifact type.
|
||||
|
||||
Narratio should continue to orchestrate a fixed pipeline. The configurable part is the set of Scriptorium artifact invocations performed during the `analyze` stage.
|
||||
|
||||
## Current State
|
||||
|
||||
Narratio already has several relevant pieces in place:
|
||||
|
||||
- `pipeline.scriptorium.artifacts` is modeled as a map of artifact definitions.
|
||||
- The Scriptorium adapter already accepts generic run/render requests.
|
||||
- The artifact resolver already understands canonical artifact source IDs.
|
||||
- The `analyze` stage already resolves inputs, optionally runs render-debug, invokes Scriptorium, verifies output, and records metadata.
|
||||
|
||||
The main limitation is that `analyze` currently treats `session_recap` as the only executable artifact and rejects other enabled artifact definitions.
|
||||
|
||||
## Target Architecture
|
||||
|
||||
### Runtime Artifact Catalog
|
||||
|
||||
Introduce a per-run artifact catalog that tracks both built-in and configured artifacts.
|
||||
|
||||
The catalog should include:
|
||||
|
||||
1. Built-in artifacts produced by fixed pipeline stages.
|
||||
2. Configured Scriptorium artifacts declared under `pipeline.scriptorium.artifacts`.
|
||||
3. Availability/provenance state for artifacts that have been produced or resolved from the manifest.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```text
|
||||
ArtifactCatalog
|
||||
├── built-in artifacts
|
||||
│ ├── narratio.transcript.merged
|
||||
│ ├── narratio.transcript.polished
|
||||
│ ├── narratio.transcript.full
|
||||
│ ├── narratio.transcript.trimmed
|
||||
│ └── narratio.bounds.session
|
||||
│
|
||||
└── configured artifacts
|
||||
├── narratio.artifact.session_recap
|
||||
├── narratio.artifact.player_handout
|
||||
└── narratio.artifact.npc_summary
|
||||
```
|
||||
|
||||
The catalog should distinguish between planned and available artifacts:
|
||||
|
||||
- A planned artifact is validly declared and may be produced during the current run.
|
||||
- An available artifact has been produced successfully in the current run or resolved from prior successful manifest state.
|
||||
|
||||
### Artifact Source IDs
|
||||
|
||||
Configured artifact keys should map directly to canonical source IDs:
|
||||
|
||||
```text
|
||||
pipeline.scriptorium.artifacts.<name>
|
||||
→ narratio.artifact.<name>
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
pipeline.scriptorium.artifacts.session_recap
|
||||
→ narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
The existing hard-coded `narratio.artifact.session_recap` source should become a normal configured-artifact source, while retaining compatibility behavior where needed.
|
||||
|
||||
### Configured Artifact Dependencies
|
||||
|
||||
Add optional `depends_on` support to configured artifacts.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd_session.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
```
|
||||
|
||||
`depends_on` values should refer to configured artifact keys, not full source IDs.
|
||||
|
||||
Use topological sorting to determine execution order. Fail validation on:
|
||||
|
||||
- dependency references to missing or disabled artifacts;
|
||||
- self-dependencies;
|
||||
- dependency cycles.
|
||||
|
||||
If two artifacts are independent, execute them in deterministic sorted-name order.
|
||||
|
||||
### Input Resolution
|
||||
|
||||
Input resolution should use the artifact catalog and existing artifact resolver behavior.
|
||||
|
||||
For each configured artifact input:
|
||||
|
||||
- built-in sources should resolve through the existing resolver;
|
||||
- `previous_session_artifact` should preserve existing behavior;
|
||||
- `narratio.artifact.<name>` should resolve only if the named configured artifact is available;
|
||||
- optional missing inputs should be omitted;
|
||||
- required missing inputs should fail the artifact run.
|
||||
|
||||
Runtime-configured artifacts should not be considered available merely because their output path exists on disk. They should be available only when:
|
||||
|
||||
1. they were produced successfully earlier in the current `analyze` execution; or
|
||||
2. they are recorded as successful outputs in prior manifest state being used for resume; or
|
||||
3. Narratio intentionally supports a documented canonical fallback for that artifact.
|
||||
|
||||
For the initial implementation, prefer options 1 and 2 only.
|
||||
|
||||
### Analyze Stage Generalization
|
||||
|
||||
The `analyze` stage should become the generic Scriptorium artifact stage.
|
||||
|
||||
Its high-level flow should be:
|
||||
|
||||
1. Load configured Scriptorium artifacts.
|
||||
2. Filter to enabled artifacts.
|
||||
3. If no artifacts are enabled, return success metadata with `skipped=true`.
|
||||
4. Build the runtime artifact catalog.
|
||||
5. Validate configured artifact names, source IDs, paths, dependencies, and required fields.
|
||||
6. Sort enabled artifacts by dependency order.
|
||||
7. For each artifact:
|
||||
- resolve configured inputs;
|
||||
- build the Scriptorium run request;
|
||||
- optionally run Scriptorium render-debug;
|
||||
- run Scriptorium;
|
||||
- fail on validation-failed result;
|
||||
- verify the output exists and is non-empty;
|
||||
- record artifact metadata;
|
||||
- register `narratio.artifact.<name>` as available in the catalog.
|
||||
8. Return aggregate stage metadata containing all generated artifacts.
|
||||
|
||||
The Scriptorium adapter should remain generic. It should not decide which artifacts run, how dependencies work, or how artifacts are registered.
|
||||
|
||||
### Manifest and Metadata
|
||||
|
||||
The analyze stage should record all generated configured artifacts in manifest/stage metadata.
|
||||
|
||||
Recommended metadata shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"skipped": false,
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "session_recap",
|
||||
"source_id": "narratio.artifact.session_recap",
|
||||
"output_kind": "scriptorium_artifact",
|
||||
"path": "artifacts/session_recap.md",
|
||||
"prompt_id": "dnd_session.session_recap",
|
||||
"profile_id": "local-gemma-31b",
|
||||
"provenance": "manifest.analyze.outputs"
|
||||
},
|
||||
{
|
||||
"name": "player_handout",
|
||||
"source_id": "narratio.artifact.player_handout",
|
||||
"output_kind": "scriptorium_artifact",
|
||||
"path": "artifacts/player_handout.md",
|
||||
"prompt_id": "dnd_session.player_handout",
|
||||
"profile_id": "local-gemma-31b",
|
||||
"provenance": "manifest.analyze.outputs"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
For backward compatibility, `session_recap` may continue to emit any legacy output kind or metadata expected by existing tests and archive behavior.
|
||||
|
||||
### Resume Behavior
|
||||
|
||||
The initial implementation can keep stage-level resume behavior.
|
||||
|
||||
That means:
|
||||
|
||||
- if `analyze` has already succeeded and is not forced, the runner can skip it as before;
|
||||
- if `analyze` is forced, all enabled configured artifacts should be regenerated;
|
||||
- if one artifact fails, the stage fails;
|
||||
- a later rerun can re-execute the analyze stage as a whole.
|
||||
|
||||
Per-artifact resume can be considered later, but it is not necessary for the first version.
|
||||
|
||||
### Archive Behavior
|
||||
|
||||
Do not automatically archive every generated artifact.
|
||||
|
||||
Artifact generation and archive promotion should remain separate concerns. Operators should continue to use `archive.promote_artifacts` to decide which generated files should be promoted or uploaded.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
archive:
|
||||
promote_artifacts:
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
- from: artifacts/player_handout.md
|
||||
to: artifacts/player_handout.md
|
||||
required: false
|
||||
```
|
||||
|
||||
A later enhancement may add opt-in automatic promotion of generated artifacts, but explicit promotion should remain the default.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Config Model and Validation
|
||||
|
||||
Add or update the configured artifact model to include:
|
||||
|
||||
- `enabled`;
|
||||
- `depends_on`;
|
||||
- `prompt_id`;
|
||||
- `profile_id`;
|
||||
- `output_path`;
|
||||
- `timeout`;
|
||||
- `render_debug`;
|
||||
- `inputs`;
|
||||
- `vars`.
|
||||
|
||||
Validation rules:
|
||||
|
||||
- artifact names must match a conservative identifier pattern such as `^[a-z][a-z0-9_]*$`;
|
||||
- enabled artifacts require `prompt_id` and `output_path`;
|
||||
- enabled artifact output paths must be run-relative and must not escape the run workspace;
|
||||
- dependency references must point to enabled configured artifacts;
|
||||
- dependencies must not contain cycles;
|
||||
- `narratio.artifact.<name>` input sources must refer to known configured artifacts;
|
||||
- input names and var names must remain compatible with the Scriptorium adapter's validation rules;
|
||||
- disabled artifacts should not be executable or dependency targets.
|
||||
|
||||
Tests:
|
||||
|
||||
- valid single configured artifact;
|
||||
- valid multiple independent artifacts;
|
||||
- valid artifact-to-artifact dependency;
|
||||
- invalid artifact name;
|
||||
- missing required fields;
|
||||
- dependency on missing artifact;
|
||||
- dependency on disabled artifact;
|
||||
- cycle detection;
|
||||
- typo in `narratio.artifact.<name>` source;
|
||||
- unknown YAML fields still fail strict decode.
|
||||
|
||||
### Phase 2: Runtime Artifact Catalog
|
||||
|
||||
Introduce an internal artifact catalog abstraction.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- register built-in artifact definitions;
|
||||
- register configured artifact definitions;
|
||||
- map configured artifact keys to `narratio.artifact.<name>` IDs;
|
||||
- track planned versus available artifacts;
|
||||
- expose lookup by canonical source ID;
|
||||
- record provenance when an artifact becomes available.
|
||||
|
||||
Keep the catalog narrow. It should not execute anything and should not know about Scriptorium prompts.
|
||||
|
||||
Tests:
|
||||
|
||||
- built-in source lookup;
|
||||
- configured source registration;
|
||||
- duplicate/conflicting source handling;
|
||||
- planned but unavailable artifact lookup;
|
||||
- registering an artifact as available after generation;
|
||||
- resolving a configured artifact from prior manifest metadata.
|
||||
|
||||
### Phase 3: Resolver Integration
|
||||
|
||||
Update artifact resolution so configured artifact IDs are resolved through the runtime catalog.
|
||||
|
||||
Resolution behavior:
|
||||
|
||||
- built-in sources continue using existing manifest-preferred, canonical-fallback behavior;
|
||||
- configured artifact sources resolve from catalog availability/provenance;
|
||||
- missing optional configured artifact inputs are omitted;
|
||||
- missing required configured artifact inputs fail clearly.
|
||||
|
||||
Tests:
|
||||
|
||||
- configured artifact consumes a built-in transcript source;
|
||||
- configured artifact consumes another configured artifact produced earlier in the same analyze run;
|
||||
- configured artifact consumes another configured artifact from prior manifest state;
|
||||
- required missing configured artifact fails;
|
||||
- optional missing configured artifact is omitted.
|
||||
|
||||
### Phase 4: Analyze Stage Generalization
|
||||
|
||||
Refactor `analyze` to execute all enabled configured artifacts.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- preserve the existing skip behavior when Scriptorium config is absent or no artifacts are enabled;
|
||||
- remove the hard-coded rejection of non-`session_recap` artifacts;
|
||||
- compute deterministic dependency order before execution;
|
||||
- execute artifacts one at a time in dependency order;
|
||||
- keep render-debug behavior at global and artifact levels;
|
||||
- keep Scriptorium adapter invocation logic generic;
|
||||
- after each successful run, register the artifact as available in the catalog;
|
||||
- aggregate metadata across all artifacts.
|
||||
|
||||
Tests:
|
||||
|
||||
- no Scriptorium config skips;
|
||||
- empty artifact map skips;
|
||||
- disabled artifacts do not run;
|
||||
- one enabled artifact runs;
|
||||
- multiple independent artifacts run in deterministic order;
|
||||
- dependent artifact receives prior artifact as input;
|
||||
- render-debug works for configured artifacts;
|
||||
- Scriptorium validation failure fails the stage;
|
||||
- missing required input fails the stage;
|
||||
- successful outputs are non-empty and recorded.
|
||||
|
||||
### Phase 5: Manifest Compatibility and Output Kinds
|
||||
|
||||
Update manifest/stage output recording to support dynamic configured artifacts.
|
||||
|
||||
Recommended behavior:
|
||||
|
||||
- every configured artifact gets `source_id: narratio.artifact.<name>`;
|
||||
- every configured artifact gets a generic output kind such as `scriptorium_artifact`;
|
||||
- `session_recap` may also retain legacy metadata/output kind for compatibility;
|
||||
- manifest provenance should be sufficient for later resolution during resume.
|
||||
|
||||
Tests:
|
||||
|
||||
- manifest records one configured artifact;
|
||||
- manifest records multiple configured artifacts;
|
||||
- `session_recap` remains compatible with existing expectations;
|
||||
- configured artifact can be resolved from manifest metadata on later run/resume.
|
||||
|
||||
### Phase 6: Documentation and Examples
|
||||
|
||||
Update documentation after the implementation is complete.
|
||||
|
||||
Recommended documentation changes:
|
||||
|
||||
- update `docs/config.md` with the generalized artifact configuration model;
|
||||
- update `docs/internal/artifacts.md` to describe the runtime artifact catalog;
|
||||
- update `docs/stages/analyze.md` to describe generic Scriptorium artifact generation;
|
||||
- update Scriptorium integration docs only if the adapter contract changes;
|
||||
- update full annotated pipeline examples;
|
||||
- add at least one example with multiple artifacts and one dependency.
|
||||
|
||||
Documentation should make clear that:
|
||||
|
||||
- configured artifact source IDs use `narratio.artifact.<name>`;
|
||||
- `depends_on` uses artifact keys, not full source IDs;
|
||||
- archive promotion remains explicit;
|
||||
- per-artifact resume is not part of the initial implementation.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
Existing configurations using `session_recap` should continue working.
|
||||
|
||||
Recommended migration path:
|
||||
|
||||
1. Treat `pipeline.scriptorium.artifacts.session_recap` as a normal configured artifact.
|
||||
2. Keep `narratio.artifact.session_recap` as a supported source ID.
|
||||
3. Preserve existing default archive promotion for `artifacts/session_recap.md` where applicable.
|
||||
4. Preserve existing tests for session recap behavior while adding new generic artifact tests.
|
||||
5. Remove or update documentation that says only `session_recap` is supported.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
Before implementation, decide the following:
|
||||
|
||||
1. Should configured artifact output paths be required to live under `artifacts/`?
|
||||
2. Should disabled artifacts be valid references for `narratio.artifact.<name>` sources, or should validation fail immediately?
|
||||
3. Should `depends_on` be required whenever an artifact input references another configured artifact, or should Narratio infer dependencies from input source IDs?
|
||||
4. Should the first implementation support configured artifact resolution from prior manifest state, or only from artifacts produced earlier in the same analyze execution?
|
||||
5. Should `session_recap` keep a legacy output kind forever, or only through a compatibility window?
|
||||
|
||||
Recommended answers:
|
||||
|
||||
1. Prefer requiring configured artifact outputs under `artifacts/` unless there is a strong reason not to.
|
||||
2. Fail references to disabled artifacts.
|
||||
3. Require `depends_on` for clarity, and validate that it matches artifact input references.
|
||||
4. Support prior manifest resolution if the existing manifest model makes this straightforward; otherwise defer.
|
||||
5. Keep legacy `session_recap` compatibility until the next major release boundary.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The feature should be considered complete when:
|
||||
|
||||
- operators can define more than one enabled Scriptorium artifact in `pipeline.yml`;
|
||||
- Narratio runs all enabled artifacts in deterministic dependency order;
|
||||
- configured artifacts are addressable as `narratio.artifact.<name>`;
|
||||
- one configured artifact can consume another configured artifact as an input;
|
||||
- missing required inputs fail clearly;
|
||||
- optional missing inputs are omitted;
|
||||
- render-debug behavior works for all configured artifacts;
|
||||
- generated artifacts are recorded in manifest/stage metadata;
|
||||
- existing `session_recap` behavior remains compatible;
|
||||
- archive promotion remains explicit;
|
||||
- tests cover config validation, dependency sorting, resolver behavior, analyze execution, and manifest metadata.
|
||||
|
||||
Reference in New Issue
Block a user