Rewrite internal documentation for current stage and state contracts
This commit is contained in:
@@ -1,78 +1,57 @@
|
||||
# Workspace internals
|
||||
# Internal: Workspace
|
||||
|
||||
## Purpose
|
||||
Define the local durable and run-local workspace model used by stages, manifests, resume, and publish.
|
||||
Define local session layout, run-local stage layout, and cleanup guardrails.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
- generated `run_id`
|
||||
## Canonical Session Layout
|
||||
Session root:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||
|
||||
Outputs:
|
||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
||||
- Canonical durable session directories and run-local stage trees
|
||||
Core directories/files:
|
||||
- `inputs/`
|
||||
- `audio/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/`
|
||||
- `logs/`
|
||||
- `config/`
|
||||
- `current/`
|
||||
- `runs/`
|
||||
- `previous/`
|
||||
- `manifest.json`
|
||||
- `.lock`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`, `previous/`)
|
||||
- `previous/manifest.json` and `previous/artifacts/**` are reserved for previous-session cache state materialized by `prepare` or `restore`
|
||||
- Run-local stage sandbox layout under `runs/{run_id}/{stage}/`
|
||||
- Session lock acquisition/release (`.lock`)
|
||||
`previous/` reserved files:
|
||||
- `previous/manifest.json`
|
||||
- `previous/artifacts/**`
|
||||
|
||||
Does not own:
|
||||
- Stage business logic
|
||||
- Remote publish semantics (documented in `stage-publish.md`)
|
||||
- CLI argument parsing
|
||||
## Run-Local Stage Layout
|
||||
When run context is available, stages use:
|
||||
- `runs/{run_id}/{stage}/outputs/`
|
||||
- `runs/{run_id}/{stage}/logs/`
|
||||
- `runs/{run_id}/{stage}/reports/`
|
||||
- `runs/{run_id}/{stage}/config/`
|
||||
- `runs/{run_id}/{stage}/scratch/`
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.workspace.cleanup_after_publish`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.spool.delete_audio_after_publish`
|
||||
- `pipeline.cache.root`
|
||||
- `pipeline.cache.s3_audio`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
Run-local outputs are materialized back into canonical session paths before stage success.
|
||||
`previous/**` writes are never redirected to run-local output paths.
|
||||
|
||||
## External Adapters Used
|
||||
None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout.
|
||||
## Locking
|
||||
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Session state is persisted in the session manifest (`manifest.Manifest`).
|
||||
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
|
||||
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and then materialized to canonical session paths after stage success.
|
||||
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
|
||||
- For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object.
|
||||
- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current publish state rather than treating current-session stored `previous/**` as authoritative.
|
||||
- Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`.
|
||||
- `narratio clean <id>` removes the session work root and session spool root.
|
||||
- `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`.
|
||||
- `narratio clean --clear-cache` is the explicit opt-in for deleting matching S3 audio cache entries.
|
||||
## Cleanup Semantics
|
||||
Automatic post-publish cleanup (`runPostArchiveCleanup`):
|
||||
- only runs when publish actually executed and succeeded;
|
||||
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
||||
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
|
||||
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest.
|
||||
- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`.
|
||||
- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs.
|
||||
Manual clean command:
|
||||
- `clean <session_id>` removes session work and spool subtree.
|
||||
- `clean --all` removes all workspace work and spool children.
|
||||
- durable cache is preserved unless `--clear-cache` is requested.
|
||||
|
||||
## Failure Behavior
|
||||
- Failures preserve manifests and run-local files for inspection.
|
||||
- Lock conflicts fail fast via `ErrLockConflict`.
|
||||
- Cleanup can fail post-publish; failure is recorded in publish stage metadata and returned by the run.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/artifacts/local_test.go`
|
||||
- `internal/stage/run_local_test.go`
|
||||
- `internal/app/run_control_test.go`
|
||||
- `internal/app/resume_run_stage_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
|
||||
- Run roots are always nested: `runs/{run_id}` under the session root.
|
||||
- Run-local output materialization must end in canonical session paths.
|
||||
- `previous/**` is session-durable state and must not be treated as run-local output scratch state.
|
||||
- Automatic post-publish cleanup only targets run-scoped directories and must never delete configured root directories.
|
||||
- Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets.
|
||||
## Invariants
|
||||
- campaign-aware session root is mandatory.
|
||||
- manifest-driven stage state is durable across runs.
|
||||
- cleanup guardrails prevent destructive root/out-of-scope deletion.
|
||||
|
||||
Reference in New Issue
Block a user