eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 15 Wiki Activity
Files
2356688cb93607170f4ba935c343966a8a86afba
narratio/docs/development/workspace.md

745 lines
25 KiB
Markdown
Raw Blame History

# Narratio Workspace, Run History, and Artifact Resolution Architecture
## 1. Purpose
This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.
Narratio is an idempotent session orchestrator. The command:
```bash
narratio run --session-id 2026-05-07
```
means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state."
This distinction drives the architecture:
* A **session** is the durable domain object and idempotency boundary.
* A **run** is an execution attempt that may update the session's durable state.
* Durable outputs live at the session level.
* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`.
* Successful stage outputs are promoted from run-local locations into canonical session-level locations.
* The session manifest records current durable state.
* Run manifests record execution history and debugging/provenance details.
This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.
## 2. Core Concepts
### 2.1 Session
A session is the stable unit of work identified by `campaign_id` and `session_id`.
Examples:
```text
campaign_id = dilfs
session_id = 2026-05-07
```
The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.
### 2.2 Run
A run is a particular execution attempt identified by a generated `run_id`, for example:
```text
20260517T174748Z-abcd1234
```
A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command.
Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.
### 2.3 Durable Output
A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.
Examples:
```text
transcripts/merged.json
transcripts/processed.json
transcripts/normalized.json
transcripts/trimmed.json
artifacts/session_recap.md
```
Durable outputs live directly under the session directory, not under a particular run directory.
### 2.4 Run-Local Output
A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.
Run-local outputs, logs, generated configs, reports, and scratch files should remain under:
```text
runs/{run_id}/{stage}/...
```
## 3. Local Workspace Layout
The canonical local workspace layout is:
```text
{workspace.root}/work/{campaign_id}/{session_id}/
manifest.json
current/
manifest.json
run_id.txt
inputs/
transcripts/
artifacts/
reports/
logs/
config/
runs/
{run_id}/
manifest.json
prepare/
transcribe/
merge/
polish/
normalize/
trim/
analyze/
archive/
notify/
```
Not every directory must exist at all times. Directories should be created idempotently when needed.
### 3.1 Session Root
The session root is:
```text
{workspace.root}/work/{campaign_id}/{session_id}/
```
The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.
The only files that should live directly in the session root are core session-state files, primarily:
```text
manifest.json
```
Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.
### 3.2 Session-Level Canonical Directories
The following directories contain current durable session state:
```text
inputs/
transcripts/
artifacts/
reports/
logs/
config/
current/
```
Recommended meanings:
| Directory | Purpose |
| -------------- | ----------------------------------------------------------------------------- |
| `inputs/` | Materialized or copied input files used by the current durable session state. |
| `transcripts/` | Canonical transcript tiers. |
| `artifacts/` | User-facing and machine-readable generated artifacts. |
| `reports/` | Canonical stage reports worth preserving at the session level. |
| `logs/` | Optional session-level logs or promoted/latest logs. |
| `config/` | Optional session-level generated config snapshots or promoted/latest configs. |
| `current/` | Current published session pointers, mirroring the archive backend. |
Canonical durable outputs should use stable paths under these directories.
### 3.3 Run History Directory
Run history lives under:
```text
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/
```
Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.
Example full run:
```text
runs/20260517T174748Z-abcd1234/
manifest.json
prepare/
transcribe/
merge/
polish/
normalize/
trim/
analyze/
archive/
notify/
```
Example sparse forced analyze run:
```text
runs/20260518T030000Z-efgh5678/
manifest.json
analyze/
```
Example sparse polish-through-analyze rerun:
```text
runs/20260518T041500Z-a1b2c3d4/
manifest.json
polish/
normalize/
trim/
analyze/
```
Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.
### 3.4 Stage Run-Local Directories
Each stage receives a run-local directory:
```text
runs/{run_id}/{stage}/
```
Within that stage directory, the stage may use subdirectories such as:
```text
outputs/
logs/
reports/
config/
scratch/
```
For example:
```text
runs/{run_id}/polish/
outputs/transcripts/processed.json
reports/audita.polish.report.json
logs/stdout.log
logs/stderr.log
config/audita.polish.generated.yml
scratch/
```
The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.
## 4. Promotion Model
Narratio uses stage-level promotion with immediate promotion after successful validation.
The stage lifecycle is:
1. Resolve required inputs from the current session state and/or run-local context.
2. Create the run-local stage directory.
3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`.
4. Validate run-local outputs.
5. Promote durable outputs into session-level canonical paths.
6. Update the session manifest.
7. Update the run manifest.
Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.
Example:
```text
runs/{run_id}/polish/outputs/transcripts/processed.json
```
is promoted to:
```text
transcripts/processed.json
```
Promotion should be safe and deterministic:
* Validate before promotion.
* Write promoted files atomically where possible.
* Never leave partially written durable outputs.
* Record the producing `run_id` in the session manifest.
* Preserve run-local files for debugging unless retention policy deletes them.
## 5. Promotion Policy: Option A
Narratio uses immediate stage-level promotion.
If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.
Example:
```bash
narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze
```
If `polish` succeeds and `normalize` fails:
* `transcripts/processed.json` may be updated from the new run.
* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state.
* Downstream outputs may now be stale relative to the newly promoted polished transcript.
This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.
## 6. Stale and Invalidation Semantics
Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.
When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.
Example stage order:
```text
prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify
```
If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:
```text
normalize
trim
analyze
archive
notify
```
A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.
Minimum manifest state model:
```text
pending
running
succeeded
failed
skipped
stale
```
If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.
## 7. Manifest Responsibilities
Narratio should distinguish between session manifests and run manifests.
The same underlying Go types may be reused where practical, but the concepts should remain separate.
### 7.1 Session Manifest
Path:
```text
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
```
The session manifest answers:
```text
What is the current durable state of this session?
```
It should record:
* campaign ID
* session ID
* current or latest run ID
* current stage states
* canonical durable output refs
* artifact IDs and paths
* producing run ID for each current stage output
* relevant input/config checksums when available
* stale/invalidated stage information
* archive/current publication metadata
A session's durable state may be a composite of multiple runs.
For example:
```text
transcripts/merged.json produced by run A
transcripts/processed.json produced by run B
transcripts/normalized.json produced by run B
transcripts/trimmed.json produced by run B
artifacts/session_recap.md produced by run C
```
This is valid and expected.
### 7.2 Run Manifest
Path:
```text
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json
```
The run manifest answers:
```text
What happened during this specific execution attempt?
```
It should record:
* run ID
* campaign ID
* session ID
* command mode and selected stages
* force flags or stage selection flags
* stages considered during this run
* stages executed during this run
* stages skipped during this run and reasons
* run-local output paths
* promoted output paths
* logs
* reports
* generated configs
* timings
* errors
* non-secret subprocess invocation metadata
Run manifests are primarily for debugging, auditability, and archive history.
## 8. Idempotency and Resume Behavior
The idempotency boundary is the session, not the run.
By default:
```bash
narratio run --session-id 2026-05-07
```
should consult the session manifest and skip stages that are already succeeded and not stale.
If all stages are already complete, the command should execute zero stages and report that the session is already complete.
Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.
Examples:
```bash
narratio run --session-id 2026-05-07 --force
```
Creates a new run and attempts to re-execute the selected/default stage set.
```bash
narratio run-stage --session-id 2026-05-07 analyze --force
```
Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful.
```bash
narratio resume --session-id 2026-05-07
```
Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run.
## 9. Artifact Resolution Contract
Narratio should provide a first-class artifact registry and resolver.
The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.
Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.
### 9.1 Canonical Artifact IDs
Preferred artifact IDs should be namespaced:
```text
narratio.transcript.merged
narratio.transcript.polished
narratio.transcript.full
narratio.transcript.trimmed
narratio.bounds.session
narratio.artifact.session_recap
```
Recommended initial registry:
| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning |
| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- |
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. |
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. |
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. |
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. |
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. |
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. |
### 9.2 Backward-Compatible Aliases
Existing source names should remain supported:
| Legacy Source | Preferred Artifact ID |
| ----------------------- | ------------------------------ |
| `processed_transcript` | `narratio.transcript.polished` |
| `normalized_transcript` | `narratio.transcript.full` |
| `trimmed_transcript` | `narratio.transcript.trimmed` |
These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.
### 9.3 Resolver Behavior
Artifact resolution should follow this order:
1. Normalize aliases to canonical artifact IDs.
2. Look for a current output reference in the session manifest.
3. Fall back to the canonical session-level path.
4. If the artifact is required, fail clearly if missing.
5. If the artifact is optional and missing, omit it from the downstream invocation.
6. Validate the artifact using the expected content validator.
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.
Example conceptual result:
```json
{
"id": "narratio.transcript.trimmed",
"path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
"producer_stage": "trim",
"producer_run_id": "20260517T174748Z-abcd1234",
"output_kind": "transcript_trimmed",
"content_type": "application/json"
}
```
### 9.4 Artifact Validation
Transcript artifacts must be valid JSON with a top-level `segments` array.
Markdown/text artifacts must exist and be non-empty when required.
Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.
Validation should happen before a resolved artifact is passed to another stage or external subprocess.
## 10. Analyze Stage Implications
The analyze stage should consume artifacts through the artifact resolver.
Preferred Scriptorium config shape:
```yaml
scriptorium:
artifacts:
session_recap:
enabled: true
prompt_id: "dnd.session_recap"
output_path: "artifacts/session_recap.md"
inputs:
transcript:
source: "narratio.transcript.trimmed"
required: true
```
Additional artifacts can choose different transcript tiers:
```yaml
scriptorium:
artifacts:
table_summary:
enabled: true
prompt_id: "dnd.table_summary"
output_path: "artifacts/table_summary.md"
inputs:
transcript:
source: "narratio.transcript.full"
required: true
```
For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.
Rules:
* Artifact inputs resolve from current session-level durable state.
* Outputs are first written run-locally.
* Successful analyze outputs are promoted to session-level `artifacts/` paths.
* Manifest output refs record the producing run ID.
* Optional inputs are omitted when unavailable.
* Required missing inputs fail before invoking Scriptorium.
## 11. Archive Alignment
Local workspace semantics should mirror archive semantics.
Local session-level durable paths:
```text
work/{campaign}/{session}/transcripts/trimmed.json
work/{campaign}/{session}/artifacts/session_recap.md
work/{campaign}/{session}/current/manifest.json
work/{campaign}/{session}/current/run_id.txt
work/{campaign}/{session}/runs/{run_id}/...
```
should map naturally to remote archive paths:
```text
{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...
```
The archive stage should publish run records and promoted current artifacts consistently with the local model.
`current/run_id.txt` remains the effective commit marker for the archived current session state.
## 12. Path Helper Requirements
All code should use centralized path helpers for workspace paths.
Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.
Recommended helper surface:
```text
SessionRoot(campaignID, sessionID)
SessionManifestPath(campaignID, sessionID)
SessionCurrentDir(campaignID, sessionID)
SessionTranscriptsDir(campaignID, sessionID)
SessionArtifactsDir(campaignID, sessionID)
SessionReportsDir(campaignID, sessionID)
SessionLogsDir(campaignID, sessionID)
SessionConfigDir(campaignID, sessionID)
RunsDir(campaignID, sessionID)
RunRoot(campaignID, sessionID, runID)
RunManifestPath(campaignID, sessionID, runID)
RunStageDir(campaignID, sessionID, runID, stage)
RunStageOutputsDir(campaignID, sessionID, runID, stage)
RunStageLogsDir(campaignID, sessionID, runID, stage)
RunStageReportsDir(campaignID, sessionID, runID, stage)
RunStageConfigDir(campaignID, sessionID, runID, stage)
CanonicalArtifactPath(campaignID, sessionID, artifactID)
```
Path helpers should enforce safe relative paths for configured output paths:
* reject absolute paths unless explicitly allowed for a particular config field
* reject `..` traversal
* normalize separators
* preserve deterministic output paths
## 13. Directory Creation Policy
Directory creation should be centralized and idempotent.
Recommended policy:
* `prepare` ensures the baseline session directory structure exists.
* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
* `run-stage` should not depend on a prior `prepare` invocation merely to create folders.
* Missing directories should be created with appropriate permissions.
* Directory creation should not imply stage success.
This provides consistent layout while keeping direct stage execution robust.
## 14. Cleanup and Retention
Cleanup must preserve the distinction between durable session state and run history.
Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.
Potential retention policies:
```text
keep_all_runs
keep_failed_runs
keep_last_n_runs
delete_run_after_success
```
For v1.0, conservative retention is preferred:
* Do not delete durable session-level outputs unless explicitly requested.
* Do not delete failed run directories by default.
* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
* Local development audio inputs must never be deleted by workspace cleanup.
## 15. Canonical-Only Layout Policy
Narratio now supports only the canonical campaign-aware layout:
```text
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/...
```
Legacy session-only layout compatibility is intentionally not implemented.
If legacy workspace data exists, operators should recreate or manually migrate that data outside Narratio before running v1.0 commands.
## 16. Documentation Updates Required
The following documentation should be updated to reflect this architecture:
* `README.md`
* `docs/architecture.md`
* a dedicated workspace/run-history document, such as this file
* S3/archive documentation
* analyze/artifact configuration documentation
* example pipeline files
Documentation should consistently use the following terms:
| Term | Meaning |
| ---------------- | ---------------------------------------------------------------------- |
| Session | Durable domain object and idempotency boundary. |
| Run | Execution attempt that may update session state. |
| Durable output | Canonical current session-level output. |
| Run-local output | Output produced inside a specific run directory before promotion. |
| Promotion | Validated copy/rename from run-local output to durable session output. |
| Session manifest | Current durable state of the session. |
| Run manifest | Execution record for a particular run. |
| Artifact ID | Symbolic source name resolved by the artifact registry. |
## 17. Architectural Invariants
The following invariants should hold after implementation:
1. `session_id` remains the idempotency boundary for normal operator commands.
2. `run_id` identifies an execution attempt, not the primary durable workspace.
3. Session-level canonical artifacts are the default inputs for downstream stages.
4. Run-local outputs are promoted only after validation.
5. A session's current durable state may be composed of outputs from multiple runs.
6. Sparse run directories are valid and expected.
7. The session manifest records current stage/artifact state and producer run IDs.
8. The run manifest records what happened during one invocation.
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
10. Local workspace semantics mirror S3 archive semantics.
11. Directory creation is centralized and idempotent.
12. Stage code uses path helpers rather than ad hoc path construction.
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.
## 18. Implementation Guidance
A practical implementation sequence is:
1. Add this architecture document.
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
3. Introduce session manifest versus run manifest concepts.
4. Route stage outputs through run-local directories.
5. Add promotion helpers with validation and atomic writes.
6. Update existing stages to promote durable outputs to session-level canonical paths.
7. Add artifact registry and resolver.
8. Update analyze to use artifact IDs and aliases.
9. Add simple downstream stale invalidation for forced upstream reruns.
10. Align archive/local path behavior and documentation.
11. Update examples and README.
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.
This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.
Reference in New Issue View Git Blame Copy Permalink