Integrate cleanup into pipeline execution

This commit is contained in:
2026-09-05 15:27:08 +00:00
parent 8d9eb5cc40
commit adc60aa588
25 changed files with 405 additions and 1013 deletions

View File

@@ -26,7 +26,8 @@ The pipeline runner follows one explicit invocation lifecycle: inspect bounded
prerequisites without mutation, prepare the workspace layout, acquire one
session lock, load or create the authoritative session manifest, compose only
the collaborators needed by the selected range, execute each selected stage in
order, complete any required post-publish cleanup, and then return the summary.
order, and then return the summary. Cleanup, when selected, is the ordinary
final stage rather than runner finalization work.
The lock remains held from manifest initialization through finalization, and a
failed stage or required manifest save prevents later stages from starting.
@@ -35,7 +36,7 @@ failed stage or required manifest save prevents later stages from starting.
| Area | Implemented owners | Responsibility |
| --- | --- | --- |
| Executable | `cmd/narratio` | Process entry, standard stream wiring, argument handoff, and exit status. |
| Application orchestration | `internal/app` | Command dispatch, configuration selection, secret-file environment loading, production composition, session locking, planning, execution, restore, cleanup gates, and user-facing reporting. |
| Application orchestration | `internal/app` | Command dispatch, configuration selection, secret-file environment loading, production composition, session locking, planning, execution, restore, and user-facing reporting. |
| Configuration | [`internal/config`](configuration.md) | Presence-aware root/import/profile composition, canonical party and family expansion, strict YAML loading, defaults, normalization, session templating, and validation. |
| Pipeline stages | `internal/stage` | Canonical stage registry, shared stage contract, execution dependencies, and implemented stage behavior. |
| External boundaries | `internal/adapters`, `internal/audio` | WhisperX HTTP, downstream subprocesses, notification, object storage, and S3 audio materialization behind Narratio contracts. |
@@ -78,6 +79,7 @@ The implemented canonical order is:
9. [`analyze`](stage-analyze.md)
10. [`publish`](stage-publish.md)
11. `notify` (no-op)
12. [`cleanup`](stage-cleanup.md)
`notify` currently has no persisted pipeline outputs and uses the explicit
`noop` notification mode. Its versioned semantic evidence records that delivery
@@ -91,7 +93,8 @@ registry owns the flat execution sequence. The application orchestration owner
uses a fixed, validated dependency relation to find transitive dependents in
canonical order. In particular, `render` and `extract` are sibling consumers of
trimmed transcript state: neither invalidates the other, while either can stale
`analyze`, `publish`, and `notify`.
`analyze`, `publish`, `notify`, and `cleanup`. Cleanup is a dependent of notify,
so any earlier replacement can stale a prior cleanup result.
## Focused Documentation
@@ -118,6 +121,7 @@ trimmed transcript state: neither invalidates the other, while either can stale
- [`extract`](stage-extract.md)
- [`analyze`](stage-analyze.md)
- [`publish`](stage-publish.md)
- [`cleanup`](stage-cleanup.md)
Use this map to find an owner, then read its focused documentation and tests
before changing behavior.

View File

@@ -0,0 +1,54 @@
# Stage: cleanup
## Purpose
Remove explicitly configured local data for the run selected by a successful
publish commit. Cleanup is the twelfth and final canonical stage, after
`notify`.
## Authorization And Targets
Cleanup is disabled unless at least one of these policies is true:
- `pipeline.spool.delete_audio_after_publish`
- `pipeline.workspace.cleanup_after_publish`
An enabled cleanup requires the manifest's publish stage to be succeeded with
`uploaded=true`, a canonical `published_run_id`, and the exact remote commit and
current commit-pointer keys for the manifest campaign/session prefix. The
configured campaign and session must match the manifest. Cleanup derives its
only possible targets through the canonical artifact path helpers:
- the published run's session spool-audio directory; and
- the published run's workspace run directory.
It does not scan parent directories, infer the newest run, or read object
storage. Every deletion uses the root-confined file operation, which rejects a
root itself, traversal, symlink escape, and out-of-root targets.
## Execution And Resume
Disabled cleanup records an explicit skipped result. Enabled cleanup removes
targets in deterministic policy order and treats an already-missing target as
success. A successful record contains the published run ID and completed policy
names.
A previous cleanup success is reusable only while its semantic configuration
matches and every selected target remains absent. A recreated target makes the
stage non-resumable. A failure is recorded against cleanup; publish remains
succeeded and remote current state is unchanged. On a later normal full run,
valid upstream stages are skipped and cleanup is retried without republishing.
`narratio publish` selects publish only and never invokes cleanup. A bounded run
through `notify` or any earlier endpoint also excludes it. Operators may run
`narratio run-stage cleanup <session_id>` directly; missing, failed, malformed,
or identity-mismatched publication evidence fails before deletion.
## Implementation And Tests
- Planning and evidence validation: `internal/stage/cleanup_plan.go`
- Execution and resume: `internal/stage/cleanup.go`
- Root-confined deletion: `internal/fileops/cleanup.go`
- Focused tests: `internal/stage/cleanup_plan_test.go`,
`internal/stage/cleanup_test.go`, and
`internal/app/cleanup_orchestration_test.go`

View File

@@ -109,8 +109,8 @@ Includes counts/lists for:
- independent publish locks are not overridden by `--force`; every initially
unlocked relevant source is reread immediately before current-state
selection.
- post-commit local cleanup is authorized by the committed publish metadata and
is durably recorded by the application lifecycle before any local deletion.
- successful uploaded publication records the exact run ID, remote commit key,
and current-pointer key consumed by the later cleanup stage.
## Resume Semantics
@@ -123,9 +123,12 @@ and run identifiers are excluded. Independent remote locks remain mutable state
and are read directly by publish at planning and commit boundaries; semantic
evidence does not replace those safety checks.
The commit boundary and cleanup gate are normative architecture invariants; see
The commit boundary and cleanup authorization are normative architecture invariants; see
[Architecture](../policy/architecture.md#publish-commit-boundary).
Publish never deletes local state. The standalone `publish` command selects
only this stage; a full run selects notify and cleanup afterward.
## Related Contracts And Tests
- [Configuration](../config.md#publish-configuration-summary) owns output
@@ -141,4 +144,4 @@ The commit boundary and cleanup gate are normative architecture invariants; see
`internal/stage/publish_test.go`,
`internal/stage/semantic_contracts_delivery.go`,
`internal/app/operator_helpers_test.go`, and
`internal/app/post_publish_cleanup_test.go`
`internal/app/cleanup_orchestration_test.go`

View File

@@ -47,18 +47,18 @@ lock only while their context remains active, and report a release failure.
## Cleanup Semantics
Automatic post-publish cleanup:
The canonical cleanup stage:
- is created only after a successful publish commit with complete publish
metadata, then is persisted before any deletion;
- requires `uploaded=true`, a remote commit key, and a current commit-pointer
key in publish metadata;
- runs after notify only when selected by the invocation range;
- requires a succeeded publish record with `uploaded=true`, the published run
ID, remote commit key, and current commit-pointer key;
- consumes the resolved cleanup policy described in
[Configuration](../config.md);
- refuses unsafe deletes (root delete, traversal, and out-of-root targets);
- retries any recorded incomplete target on later invocations even when no
publish work is selected. Missing targets are a successful, idempotent
cleanup result only after the completion evidence is saved.
- treats missing targets as successful, idempotent deletion; and
- uses its ordinary failed/succeeded manifest stage record for retry. A later
full run can skip reusable upstream stages and retry cleanup without
republishing.
Manual cleanup uses the same root-confined deletion mechanism. Invocation
syntax and exact deletion scope belong in [CLI](../cli.md#clean) and
@@ -82,9 +82,11 @@ syntax and exact deletion scope belong in [CLI](../cli.md#clean) and
- Immutable bundle installation: `internal/stage/extract.go`
- Workspace modes: `internal/fileops/modes.go`
- Cleanup confinement: `internal/fileops/cleanup.go`,
`internal/app/cleanup_targets.go`, `internal/app/post_publish_cleanup.go`
`internal/stage/cleanup.go`, `internal/stage/cleanup_plan.go`, and
`internal/app/cleanup_targets.go`
- Tests: `internal/artifacts/paths_model_test.go`,
`internal/artifacts/local_test.go`, `internal/stage/run_local_test.go`,
`internal/fileops/modes_posix_test.go`,
`internal/fileops/cleanup_test.go`, `internal/app/cleanup_targets_test.go`,
`internal/app/post_publish_cleanup_test.go`
`internal/stage/cleanup_test.go`, and
`internal/app/cleanup_orchestration_test.go`