narratio/docs/development/workspace.md

Narratio Workspace, Run History, and Artifact Resolution Architecture

1. Purpose

This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.

Narratio is an idempotent session orchestrator. The command:

narratio run --session-id 2026-05-07

means "bring the identified session to its desired completed state." It does not mean "always create an entirely new independent output tree and ignore prior session state."

This distinction drives the architecture:

• A session is the durable domain object and idempotency boundary.
• A run is an execution attempt that may update the session's durable state.
• Durable outputs live at the session level.
• Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under runs/{run_id}/.
• Successful stage outputs are promoted from run-local locations into canonical session-level locations.
• The session manifest records current durable state.
• Run manifests record execution history and debugging/provenance details.

This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.

2. Core Concepts

2.1 Session

A session is the stable unit of work identified by campaign_id and session_id.

Examples:

campaign_id = dilfs
session_id  = 2026-05-07

The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.

2.2 Run

A run is a particular execution attempt identified by a generated run_id, for example:

20260517T174748Z-abcd1234

A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes --force, run-stage, or a stage-limited command.

Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.

2.3 Durable Output

A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.

Examples:

transcripts/merged.json
transcripts/processed.json
transcripts/normalized.json
transcripts/trimmed.json
artifacts/session_recap.md

Durable outputs live directly under the session directory, not under a particular run directory.

2.4 Run-Local Output

A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.

Run-local outputs, logs, generated configs, reports, and scratch files should remain under:

runs/{run_id}/{stage}/...

3. Local Workspace Layout

The canonical local workspace layout is:

{workspace.root}/work/{campaign_id}/{session_id}/
  manifest.json
  current/
    manifest.json
    run_id.txt
  inputs/
  transcripts/
  artifacts/
  reports/
  logs/
  config/
  runs/
    {run_id}/
      manifest.json
      prepare/
      transcribe/
      merge/
      polish/
      normalize/
      trim/
      analyze/
      archive/
      notify/

Not every directory must exist at all times. Directories should be created idempotently when needed.

3.1 Session Root

The session root is:

{workspace.root}/work/{campaign_id}/{session_id}/

The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.

The only files that should live directly in the session root are core session-state files, primarily:

manifest.json

Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.

3.2 Session-Level Canonical Directories

The following directories contain current durable session state:

inputs/
transcripts/
artifacts/
reports/
logs/
config/
current/

Recommended meanings:

Directory	Purpose
inputs/	Materialized or copied input files used by the current durable session state.
transcripts/	Canonical transcript tiers.
artifacts/	User-facing and machine-readable generated artifacts.
reports/	Canonical stage reports worth preserving at the session level.
logs/	Optional session-level logs or promoted/latest logs.
config/	Optional session-level generated config snapshots or promoted/latest configs.
current/	Current published session pointers, mirroring the archive backend.

Canonical durable outputs should use stable paths under these directories.

3.3 Run History Directory

Run history lives under:

{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/

Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.

Example full run:

runs/20260517T174748Z-abcd1234/
  manifest.json
  prepare/
  transcribe/
  merge/
  polish/
  normalize/
  trim/
  analyze/
  archive/
  notify/

Example sparse forced analyze run:

runs/20260518T030000Z-efgh5678/
  manifest.json
  analyze/

Example sparse polish-through-analyze rerun:

runs/20260518T041500Z-a1b2c3d4/
  manifest.json
  polish/
  normalize/
  trim/
  analyze/

Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.

3.4 Stage Run-Local Directories

Each stage receives a run-local directory:

runs/{run_id}/{stage}/

Within that stage directory, the stage may use subdirectories such as:

outputs/
logs/
reports/
config/
scratch/

For example:

runs/{run_id}/polish/
  outputs/transcripts/processed.json
  reports/audita.polish.report.json
  logs/stdout.log
  logs/stderr.log
  config/audita.polish.generated.yml
  scratch/

The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.

4. Promotion Model

Narratio uses stage-level promotion with immediate promotion after successful validation.

The stage lifecycle is:

1. Resolve required inputs from the current session state and/or run-local context.
2. Create the run-local stage directory.
3. Execute the stage, writing outputs under runs/{run_id}/{stage}/....
4. Validate run-local outputs.
5. Promote durable outputs into session-level canonical paths.
6. Update the session manifest.
7. Update the run manifest.

Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.

Example:

runs/{run_id}/polish/outputs/transcripts/processed.json

is promoted to:

transcripts/processed.json

Promotion should be safe and deterministic:

• Validate before promotion.
• Write promoted files atomically where possible.
• Never leave partially written durable outputs.
• Record the producing run_id in the session manifest.
• Preserve run-local files for debugging unless retention policy deletes them.

5. Promotion Policy: Option A

Narratio uses immediate stage-level promotion.

If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.

Example:

narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze

If polish succeeds and normalize fails:

• transcripts/processed.json may be updated from the new run.
• normalize, trim, and analyze should not be marked succeeded for the new input state.
• Downstream outputs may now be stale relative to the newly promoted polished transcript.

This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.

6. Stale and Invalidation Semantics

Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.

When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.

Example stage order:

prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify

If polish is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:

normalize
trim
analyze
archive
notify

A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.

Minimum manifest state model:

pending
running
succeeded
failed
skipped
stale

If adding a new stale state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.

7. Manifest Responsibilities

Narratio should distinguish between session manifests and run manifests.

The same underlying Go types may be reused where practical, but the concepts should remain separate.

7.1 Session Manifest

Path:

{workspace.root}/work/{campaign_id}/{session_id}/manifest.json

The session manifest answers:

What is the current durable state of this session?

It should record:

• campaign ID
• session ID
• current or latest run ID
• current stage states
• canonical durable output refs
• artifact IDs and paths
• producing run ID for each current stage output
• relevant input/config checksums when available
• stale/invalidated stage information
• archive/current publication metadata

A session's durable state may be a composite of multiple runs.

For example:

transcripts/merged.json      produced by run A
transcripts/processed.json   produced by run B
transcripts/normalized.json  produced by run B
transcripts/trimmed.json     produced by run B
artifacts/session_recap.md   produced by run C

This is valid and expected.

7.2 Run Manifest

Path:

{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json

The run manifest answers:

What happened during this specific execution attempt?

It should record:

• run ID
• campaign ID
• session ID
• command mode and selected stages
• force flags or stage selection flags
• stages considered during this run
• stages executed during this run
• stages skipped during this run and reasons
• run-local output paths
• promoted output paths
• logs
• reports
• generated configs
• timings
• errors
• non-secret subprocess invocation metadata

Run manifests are primarily for debugging, auditability, and archive history.

8. Idempotency and Resume Behavior

The idempotency boundary is the session, not the run.

By default:

narratio run --session-id 2026-05-07

should consult the session manifest and skip stages that are already succeeded and not stale.

If all stages are already complete, the command should execute zero stages and report that the session is already complete.

Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.

Examples:

narratio run --session-id 2026-05-07 --force

Creates a new run and attempts to re-execute the selected/default stage set.

narratio run-stage --session-id 2026-05-07 analyze --force

Creates a sparse run that executes only analyze, then promotes updated analysis artifacts if successful.

narratio resume --session-id 2026-05-07

Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same run_id unless the implementation explicitly supports resuming an interrupted active run.

9. Artifact Resolution Contract

Narratio should provide a first-class artifact registry and resolver.

The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.

Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.

9.1 Canonical Artifact IDs

Preferred artifact IDs should be namespaced:

narratio.transcript.merged
narratio.transcript.polished
narratio.transcript.full
narratio.transcript.trimmed
narratio.bounds.session
narratio.artifact.session_recap

Recommended initial registry:

Artifact ID	Canonical Path	Producer Stage	Output Kind	Meaning
narratio.transcript.merged	transcripts/merged.json	merge	transcript_merged	Deterministic Seriatim merge.
narratio.transcript.polished	transcripts/processed.json	polish	transcript_processed	Full Audita-polished transcript.
narratio.transcript.full	transcripts/normalized.json	normalize	transcript_normalized	Preferred full normalized transcript.
narratio.transcript.trimmed	transcripts/trimmed.json	trim	transcript_trimmed	Gameplay-only transcript.
narratio.bounds.session	artifacts/session_bounds.json	trim	session_bounds	Trim bounds selected for the session.
narratio.artifact.session_recap	artifacts/session_recap.md	analyze	artifact_session_recap	Generated session recap.

9.2 Backward-Compatible Aliases

Existing source names should remain supported:

Legacy Source	Preferred Artifact ID
processed_transcript	narratio.transcript.polished
normalized_transcript	narratio.transcript.full
trimmed_transcript	narratio.transcript.trimmed

These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.

9.3 Resolver Behavior

Artifact resolution should follow this order:

1. Normalize aliases to canonical artifact IDs.
2. Look for a current output reference in the session manifest.
3. Fall back to the canonical session-level path.
4. If the artifact is required, fail clearly if missing.
5. If the artifact is optional and missing, omit it from the downstream invocation.
6. Validate the artifact using the expected content validator.
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.

Example conceptual result:

{
  "id": "narratio.transcript.trimmed",
  "path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
  "producer_stage": "trim",
  "producer_run_id": "20260517T174748Z-abcd1234",
  "output_kind": "transcript_trimmed",
  "content_type": "application/json"
}

9.4 Artifact Validation

Transcript artifacts must be valid JSON with a top-level segments array.

Markdown/text artifacts must exist and be non-empty when required.

Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.

Validation should happen before a resolved artifact is passed to another stage or external subprocess.

10. Analyze Stage Implications

The analyze stage should consume artifacts through the artifact resolver.

Preferred Scriptorium config shape:

scriptorium:
  artifacts:
    session_recap:
      enabled: true
      prompt_id: "dnd.session_recap"
      output_path: "artifacts/session_recap.md"
      inputs:
        transcript:
          source: "narratio.transcript.trimmed"
          required: true

Additional artifacts can choose different transcript tiers:

scriptorium:
  artifacts:
    table_summary:
      enabled: true
      prompt_id: "dnd.table_summary"
      output_path: "artifacts/table_summary.md"
      inputs:
        transcript:
          source: "narratio.transcript.full"
          required: true

For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.

Rules:

• Artifact inputs resolve from current session-level durable state.
• Outputs are first written run-locally.
• Successful analyze outputs are promoted to session-level artifacts/ paths.
• Manifest output refs record the producing run ID.
• Optional inputs are omitted when unavailable.
• Required missing inputs fail before invoking Scriptorium.

11. Archive Alignment

Local workspace semantics should mirror archive semantics.

Local session-level durable paths:

work/{campaign}/{session}/transcripts/trimmed.json
work/{campaign}/{session}/artifacts/session_recap.md
work/{campaign}/{session}/current/manifest.json
work/{campaign}/{session}/current/run_id.txt
work/{campaign}/{session}/runs/{run_id}/...

should map naturally to remote archive paths:

{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...

The archive stage should publish run records and promoted current artifacts consistently with the local model.

current/run_id.txt remains the effective commit marker for the archived current session state.

12. Path Helper Requirements

All code should use centralized path helpers for workspace paths.

Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.

Recommended helper surface:

SessionRoot(campaignID, sessionID)
SessionManifestPath(campaignID, sessionID)
SessionCurrentDir(campaignID, sessionID)
SessionTranscriptsDir(campaignID, sessionID)
SessionArtifactsDir(campaignID, sessionID)
SessionReportsDir(campaignID, sessionID)
SessionLogsDir(campaignID, sessionID)
SessionConfigDir(campaignID, sessionID)
RunsDir(campaignID, sessionID)
RunRoot(campaignID, sessionID, runID)
RunManifestPath(campaignID, sessionID, runID)
RunStageDir(campaignID, sessionID, runID, stage)
RunStageOutputsDir(campaignID, sessionID, runID, stage)
RunStageLogsDir(campaignID, sessionID, runID, stage)
RunStageReportsDir(campaignID, sessionID, runID, stage)
RunStageConfigDir(campaignID, sessionID, runID, stage)
CanonicalArtifactPath(campaignID, sessionID, artifactID)

Path helpers should enforce safe relative paths for configured output paths:

• reject absolute paths unless explicitly allowed for a particular config field
• reject .. traversal
• normalize separators
• preserve deterministic output paths

13. Directory Creation Policy

Directory creation should be centralized and idempotent.

Recommended policy:

• prepare ensures the baseline session directory structure exists.
• Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
• run-stage should not depend on a prior prepare invocation merely to create folders.
• Missing directories should be created with appropriate permissions.
• Directory creation should not imply stage success.

This provides consistent layout while keeping direct stage execution robust.

14. Cleanup and Retention

Cleanup must preserve the distinction between durable session state and run history.

Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.

Potential retention policies:

keep_all_runs
keep_failed_runs
keep_last_n_runs
delete_run_after_success

For v1.0, conservative retention is preferred:

• Do not delete durable session-level outputs unless explicitly requested.
• Do not delete failed run directories by default.
• If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
• Local development audio inputs must never be deleted by workspace cleanup.

15. Migration From Existing Layout

Existing installations may currently use a simpler path such as:

{workspace.root}/work/{session_id}/manifest.json

The v1.0 layout introduces campaign-aware session roots:

{workspace.root}/work/{campaign_id}/{session_id}/manifest.json

Migration options:

1. Best-effort automatic discovery of legacy session manifests.
2. A one-time migration command.
3. Clear release notes requiring users to move or regenerate workspace state.

For v1.0, it is acceptable to require explicit migration if the user base is small and the archive contains the authoritative durable outputs. However, the application should fail clearly when it detects an ambiguous legacy layout rather than silently creating duplicate state.

16. Documentation Updates Required

The following documentation should be updated to reflect this architecture:

• README.md
• docs/architecture.md
• a dedicated workspace/run-history document, such as this file
• S3/archive documentation
• analyze/artifact configuration documentation
• example pipeline files

Documentation should consistently use the following terms:

Term	Meaning
Session	Durable domain object and idempotency boundary.
Run	Execution attempt that may update session state.
Durable output	Canonical current session-level output.
Run-local output	Output produced inside a specific run directory before promotion.
Promotion	Validated copy/rename from run-local output to durable session output.
Session manifest	Current durable state of the session.
Run manifest	Execution record for a particular run.
Artifact ID	Symbolic source name resolved by the artifact registry.

17. Architectural Invariants

The following invariants should hold after implementation:

1. session_id remains the idempotency boundary for normal operator commands.
2. run_id identifies an execution attempt, not the primary durable workspace.
3. Session-level canonical artifacts are the default inputs for downstream stages.
4. Run-local outputs are promoted only after validation.
5. A session's current durable state may be composed of outputs from multiple runs.
6. Sparse run directories are valid and expected.
7. The session manifest records current stage/artifact state and producer run IDs.
8. The run manifest records what happened during one invocation.
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
10. Local workspace semantics mirror S3 archive semantics.
11. Directory creation is centralized and idempotent.
12. Stage code uses path helpers rather than ad hoc path construction.
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.

18. Implementation Guidance

A practical implementation sequence is:

1. Add this architecture document.
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
3. Introduce session manifest versus run manifest concepts.
4. Route stage outputs through run-local directories.
5. Add promotion helpers with validation and atomic writes.
6. Update existing stages to promote durable outputs to session-level canonical paths.
7. Add artifact registry and resolver.
8. Update analyze to use artifact IDs and aliases.
9. Add simple downstream stale invalidation for forced upstream reruns.
10. Align archive/local path behavior and documentation.
11. Update examples and README.
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.

This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.