Compare commits
2 Commits
bc2ade38d9
...
dffb432537
| Author | SHA1 | Date | |
|---|---|---|---|
| dffb432537 | |||
| 2dd38c7913 |
@@ -1,629 +0,0 @@
|
||||
# Roadmap: Previous-Session Artifacts
|
||||
|
||||
## Status
|
||||
|
||||
Completed.
|
||||
|
||||
This roadmap describes the implementation strategy for first-class previous-session artifact support in Narratio. It belongs under `docs/roadmap/previous.md` until the feature is implemented. After implementation, current behavior should be documented in the appropriate user-facing and internal documentation files, and this roadmap should be removed or marked complete according to the documentation policy.
|
||||
|
||||
## Summary
|
||||
|
||||
Narratio should support using artifacts from a previous session as inputs to artifacts generated for the current session.
|
||||
|
||||
The primary use case is session recap continuity: a current session recap should be able to consume the previous session recap. The design should support arbitrary previous-session artifacts from the start, not just `session_recap`.
|
||||
|
||||
The canonical source syntax should be:
|
||||
|
||||
```yaml
|
||||
source: narratio.previous_session.artifact.<artifact_name>
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
```
|
||||
|
||||
Previous-session artifacts are materialized during the `prepare` stage into a current-session top-level `previous/` directory. Downstream stages consume only the local `previous/` copies. The S3 backend is authoritative for previous-session state.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### 1. Add `previous_session_id` to `session.yml`
|
||||
|
||||
Add an optional top-level session key:
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
previous_session_id: "{{ previous_session_id }}"
|
||||
campaign: sample-campaign
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `previous_session_id` is optional.
|
||||
- If present, it identifies the previous session within the same campaign.
|
||||
- It must not equal `session_id`.
|
||||
- It should use the same validation rules as `session_id`.
|
||||
- It may be supplied through session templating.
|
||||
- Add a CLI/template value such as `--previous-session-id <id>` if required by the existing session templating implementation.
|
||||
- If a template placeholder for `previous_session_id` is present and no value is supplied, loading should fail with a clear unresolved-template error.
|
||||
|
||||
### 2. Use canonical previous-session artifact source IDs
|
||||
|
||||
Support this source pattern in Scriptorium artifact input definitions:
|
||||
|
||||
```text
|
||||
narratio.previous_session.artifact.<artifact_name>
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
```yaml
|
||||
inputs:
|
||||
previous_recap:
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: false
|
||||
|
||||
previous_quest_log:
|
||||
source: narratio.previous_session.artifact.quest_log
|
||||
required: true
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `<artifact_name>` must be a valid configured artifact key.
|
||||
- Use the same artifact key validation rules as current-session runtime artifacts.
|
||||
- Do not special-case `session_recap`.
|
||||
- Do not limit implementation to a fixed list of previous artifacts.
|
||||
|
||||
### 3. Add a top-level `previous/` workspace directory
|
||||
|
||||
Extend the session workspace layout with:
|
||||
|
||||
```text
|
||||
previous/
|
||||
manifest.json
|
||||
artifacts/
|
||||
<artifact outputs copied from the previous session>
|
||||
```
|
||||
|
||||
The `previous/` directory is current-session state. It is a prepared input cache, not a full mirror of the previous session workspace.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```text
|
||||
work/<campaign>/<current_session_id>/
|
||||
previous/
|
||||
manifest.json
|
||||
artifacts/
|
||||
session_recap.md
|
||||
quest_log.json
|
||||
```
|
||||
|
||||
The current session should not read directly from the previous session's local workspace during ordinary operation.
|
||||
|
||||
### 4. S3 is authoritative for previous-session state
|
||||
|
||||
For this initial implementation, previous-session artifacts should be downloaded from the configured S3 backend.
|
||||
|
||||
Do not compare local and remote copies.
|
||||
|
||||
Do not prefer local previous-session workspace state.
|
||||
|
||||
Do not implement a `--local` override in this roadmap. That can be considered later.
|
||||
|
||||
The simplified stage behavior is:
|
||||
|
||||
1. If the local manifest indicates `prepare` already succeeded and `--force` is not supplied, the runner skips `prepare`. No previous-session download occurs.
|
||||
2. If `prepare` has not succeeded, `prepare` runs and downloads referenced previous-session artifacts from S3.
|
||||
3. If `prepare` previously succeeded but `--force` is supplied, `prepare` runs again and overwrites local `previous/` state from S3.
|
||||
|
||||
### 5. Materialize only referenced previous-session artifacts
|
||||
|
||||
During `prepare`, scan the current resolved pipeline/session configuration for Scriptorium artifact inputs whose source matches:
|
||||
|
||||
```text
|
||||
narratio.previous_session.artifact.<artifact_name>
|
||||
```
|
||||
|
||||
Only those referenced previous-session artifacts need to be downloaded.
|
||||
|
||||
Do not blindly download every previous-session artifact.
|
||||
|
||||
If no previous-session artifact sources are referenced, `prepare` should not require `previous_session_id` and should not touch `previous/`.
|
||||
|
||||
### 6. Preserve stage boundaries
|
||||
|
||||
`prepare` owns previous-session artifact materialization because these files are inputs to later stages.
|
||||
|
||||
`analyze` should not talk to S3.
|
||||
|
||||
The Scriptorium adapter should not know about previous sessions.
|
||||
|
||||
The storage adapter should not infer campaign, session, run, or root-prefix semantics. Callers should continue to provide explicit bucket-relative keys.
|
||||
|
||||
### 7. Archive and restore `previous/`
|
||||
|
||||
After implementation:
|
||||
|
||||
- `archive` should include `previous/` as durable current-session prepared input state.
|
||||
- `restore` should restore `previous/` along with the rest of the durable session state it already restores.
|
||||
- `previous/` should not be treated as current-session generated artifacts.
|
||||
- `previous/` entries should be recorded as inputs/provenance, not as outputs produced by the current session.
|
||||
|
||||
## Target User Workflow
|
||||
|
||||
A typical session config:
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
previous_session_id: "{{ previous_session_id }}"
|
||||
campaign: sample-campaign
|
||||
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
```
|
||||
|
||||
A typical artifact config:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: false
|
||||
```
|
||||
|
||||
Typical command:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-11 --previous-session-id 2026-04-04
|
||||
```
|
||||
|
||||
Expected behavior:
|
||||
|
||||
1. `prepare` sees a referenced previous-session artifact: `session_recap`.
|
||||
2. `prepare` downloads the previous session's `narratio.artifact.session_recap` from S3.
|
||||
3. `prepare` writes it under the current session workspace, for example `previous/artifacts/session_recap.md`.
|
||||
4. `prepare` records provenance in the current session manifest.
|
||||
5. `analyze` resolves `narratio.previous_session.artifact.session_recap` from the local `previous/` directory.
|
||||
6. Scriptorium receives the previous recap as a normal input file.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Session config and templating
|
||||
|
||||
Update session configuration structs to include:
|
||||
|
||||
```yaml
|
||||
previous_session_id: ""
|
||||
```
|
||||
|
||||
Implementation steps:
|
||||
|
||||
1. Add `PreviousSessionID` or equivalent to the session config type.
|
||||
2. Add validation:
|
||||
- optional;
|
||||
- same format constraints as `session_id`;
|
||||
- must not equal `session_id`.
|
||||
3. Extend session templating support to include:
|
||||
- `{{previous_session_id}}`
|
||||
- `{{ previous_session_id }}`
|
||||
4. Add a CLI flag if required by current templating flow:
|
||||
- `--previous-session-id <id>`
|
||||
5. Ensure unresolved `previous_session_id` placeholders fail clearly.
|
||||
6. Update config tests for:
|
||||
- no previous session;
|
||||
- valid previous session;
|
||||
- previous session equal to current session;
|
||||
- unresolved placeholder;
|
||||
- CLI/template rendering.
|
||||
|
||||
Do not add future workflow flags in this phase.
|
||||
|
||||
### Phase 2: Workspace path helpers
|
||||
|
||||
Add centralized path helpers for current-session previous-state paths.
|
||||
|
||||
Suggested helpers:
|
||||
|
||||
```text
|
||||
SessionPreviousDir()
|
||||
SessionPreviousManifestPath()
|
||||
SessionPreviousArtifactsDir()
|
||||
SessionPreviousArtifactPath(name or relative output path)
|
||||
```
|
||||
|
||||
The exact names should match existing path-helper style.
|
||||
|
||||
Rules:
|
||||
|
||||
- Do not construct `previous/` paths through scattered string concatenation.
|
||||
- Keep paths session-relative where possible.
|
||||
- Ensure workspace layout creation includes `previous/` only when appropriate, or creates it idempotently with the rest of the layout if simpler.
|
||||
|
||||
Tests:
|
||||
|
||||
- path helper tests;
|
||||
- workspace layout tests;
|
||||
- ensure cleanup logic does not accidentally delete configured roots;
|
||||
- ensure `previous/` is treated as session-durable state, not run-local state.
|
||||
|
||||
### Phase 3: Previous-session source parsing
|
||||
|
||||
Add parsing/recognition for:
|
||||
|
||||
```text
|
||||
narratio.previous_session.artifact.<artifact_name>
|
||||
```
|
||||
|
||||
Implementation steps:
|
||||
|
||||
1. Add constants/helpers in the artifact/source parsing layer.
|
||||
2. Validate artifact names using the same rules as current runtime artifact keys.
|
||||
3. Add helpers such as:
|
||||
- `IsPreviousSessionArtifactSource(source string) bool`
|
||||
- `PreviousSessionArtifactName(source string) (string, bool)`
|
||||
4. Ensure config validation accepts this source pattern.
|
||||
5. Ensure invalid sources fail clearly.
|
||||
|
||||
Tests:
|
||||
|
||||
- valid previous-session artifact source;
|
||||
- invalid/missing artifact name;
|
||||
- invalid artifact key characters;
|
||||
- ordinary current-session sources still validate;
|
||||
- unknown sources still fail.
|
||||
|
||||
### Phase 4: Scan configured artifacts for previous-session inputs
|
||||
|
||||
Add a helper that inspects resolved Scriptorium artifact definitions and returns the set of referenced previous-session artifact names.
|
||||
|
||||
Rules:
|
||||
|
||||
- Scan enabled artifacts according to current artifact-enable semantics.
|
||||
- Include all inputs whose source matches `narratio.previous_session.artifact.<name>`.
|
||||
- Deduplicate artifact names.
|
||||
- Sort results deterministically.
|
||||
- Preserve required/optional information per reference.
|
||||
- If the same previous artifact is referenced both required and optional, treat it as required.
|
||||
|
||||
Suggested output model:
|
||||
|
||||
```go
|
||||
type PreviousArtifactRequirement struct {
|
||||
Name string
|
||||
Required bool
|
||||
Sources []string // optional diagnostics
|
||||
}
|
||||
```
|
||||
|
||||
Tests:
|
||||
|
||||
- no artifacts;
|
||||
- no previous inputs;
|
||||
- one optional previous input;
|
||||
- one required previous input;
|
||||
- duplicate references;
|
||||
- required plus optional reference to the same artifact;
|
||||
- deterministic ordering.
|
||||
|
||||
### Phase 5: Resolve previous-session archive keys
|
||||
|
||||
Implement a narrow service/helper used by `prepare` to resolve previous-session artifact files from S3.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
1. Locate the previous session's current remote state.
|
||||
2. Download the previous session manifest, or the minimum remote metadata needed to resolve artifact source IDs.
|
||||
3. Resolve `narratio.artifact.<name>` inside the previous session's artifact catalog/manifest.
|
||||
4. Download the resolved artifact into the current session's `previous/` directory.
|
||||
5. Download/store the previous session manifest as `previous/manifest.json`.
|
||||
6. Return provenance records for manifest input recording.
|
||||
|
||||
Important archive invariant:
|
||||
|
||||
- Remote current state must be based on the committed archive marker.
|
||||
- Do not treat incomplete archive uploads as current state.
|
||||
- Use the existing archive/current remote layout and commit-marker rules.
|
||||
- `current/run_id.txt` is the final remote commit marker and should be respected when locating current remote state.
|
||||
|
||||
Do not put prefix semantics into the storage adapter. Compute explicit bucket-relative keys in app/stage/archive helper code, then call the storage adapter.
|
||||
|
||||
Required vs optional behavior:
|
||||
|
||||
- Required previous artifact missing from S3/current manifest: fail `prepare`.
|
||||
- Optional previous artifact missing from S3/current manifest: continue without materializing that input.
|
||||
- Previous session missing entirely:
|
||||
- fail if any referenced previous artifact is required;
|
||||
- continue if all referenced previous artifacts are optional.
|
||||
- If previous_session_id is unset:
|
||||
- fail if any referenced previous artifact is required;
|
||||
- continue and omit all previous-session inputs if all are optional.
|
||||
|
||||
Validation behavior:
|
||||
|
||||
- Downloaded artifacts should pass the same validation rules as current-session artifacts where practical.
|
||||
- Generic Scriptorium artifacts should at least be non-empty.
|
||||
- Invalid required artifact: fail.
|
||||
- Invalid optional artifact: prefer fail if the object exists but is invalid, because invalid archived data is usually an operator problem rather than absence.
|
||||
|
||||
Tests:
|
||||
|
||||
- downloads previous manifest;
|
||||
- downloads required previous artifact;
|
||||
- skips missing optional previous artifact;
|
||||
- fails missing required previous artifact;
|
||||
- fails required previous artifact when previous_session_id is unset;
|
||||
- optional previous artifact with no previous_session_id does not fail;
|
||||
- respects current remote commit marker;
|
||||
- does not use local previous-session workspace state;
|
||||
- uses storage adapter with explicit keys.
|
||||
|
||||
### Phase 6: Integrate with `prepare`
|
||||
|
||||
Extend the `prepare` stage:
|
||||
|
||||
1. Run existing input materialization as before.
|
||||
2. Detect previous-session artifact requirements.
|
||||
3. If requirements exist, hydrate `previous/` from S3 according to the rules above.
|
||||
4. Record hydrated previous artifacts in `manifest.Inputs`.
|
||||
5. Preserve existing prepare outputs and provenance behavior.
|
||||
|
||||
Overwrite behavior:
|
||||
|
||||
- If `prepare` runs, it owns `previous/`.
|
||||
- Before hydrating, clear the managed `previous/` directory, or clear the managed previous artifact paths.
|
||||
- Prefer clearing the whole `previous/` directory if no other feature writes there.
|
||||
- Under `--force`, this naturally overwrites local `previous/` state.
|
||||
- Do not compare local and remote copies.
|
||||
|
||||
Skip behavior:
|
||||
|
||||
- Do not add stage-local skip logic.
|
||||
- Runner-level skip remains authoritative.
|
||||
- If the manifest says `prepare` succeeded and `--force` is not supplied, `prepare` does not run and no S3 downloads occur.
|
||||
- If users add a new previous-session input after `prepare` already succeeded, they must rerun prepare with `--force`.
|
||||
|
||||
Error message requirement:
|
||||
|
||||
If an analyze-stage input cannot be resolved because `previous/` is missing or stale, the error should tell the operator to run:
|
||||
|
||||
```bash
|
||||
narratio run-stage --force prepare
|
||||
```
|
||||
|
||||
or the appropriate existing CLI command shape.
|
||||
|
||||
Tests:
|
||||
|
||||
- ordinary prepare without previous_session_id remains unchanged;
|
||||
- prepare with optional previous artifact and no previous_session_id succeeds;
|
||||
- prepare with required previous artifact and no previous_session_id fails;
|
||||
- prepare with required previous artifact downloads to `previous/`;
|
||||
- force prepare overwrites `previous/`;
|
||||
- prepare records manifest inputs for previous artifacts;
|
||||
- prepare skip behavior remains controlled by runner tests;
|
||||
- no regression in S3 audio prepare behavior.
|
||||
|
||||
### Phase 7: Artifact resolver support
|
||||
|
||||
Update artifact resolution so Scriptorium inputs can resolve:
|
||||
|
||||
```text
|
||||
narratio.previous_session.artifact.<artifact_name>
|
||||
```
|
||||
|
||||
from the current session's `previous/` directory.
|
||||
|
||||
Rules:
|
||||
|
||||
- The resolver should not call S3.
|
||||
- The resolver should not read the previous session's local workspace.
|
||||
- The resolver should map the previous-session source ID to the local prepared copy under `previous/`.
|
||||
- Resolution should use manifest input records when available.
|
||||
- Fallback to the local `previous/` path may be allowed if consistent with existing resolver behavior, but manifest provenance should be preferred.
|
||||
- Missing required input should fail with a clear prepare-oriented message.
|
||||
- Optional missing input should be omitted.
|
||||
|
||||
Suggested provenance:
|
||||
|
||||
```text
|
||||
previous_session.manifest.outputs
|
||||
previous_session.archive.current
|
||||
current_session.previous_cache
|
||||
```
|
||||
|
||||
Use names that fit the existing manifest/resolver vocabulary.
|
||||
|
||||
Tests:
|
||||
|
||||
- resolves prepared previous artifact through manifest input record;
|
||||
- resolves or fails appropriately when only filesystem copy exists, depending on chosen fallback policy;
|
||||
- missing optional previous artifact is omitted;
|
||||
- missing required previous artifact fails clearly;
|
||||
- current-session `narratio.artifact.<name>` behavior is unchanged.
|
||||
|
||||
### Phase 8: Analyze-stage integration
|
||||
|
||||
The analyze stage should require little or no special previous-session logic if the resolver is designed correctly.
|
||||
|
||||
Confirm:
|
||||
|
||||
- Scriptorium input resolution accepts previous-session source IDs.
|
||||
- The Scriptorium adapter receives a normal local input path.
|
||||
- Render-debug and run modes behave the same as for ordinary inputs.
|
||||
- Stage metadata includes useful input provenance if current structures support it.
|
||||
|
||||
Tests:
|
||||
|
||||
- configured artifact receives previous recap input;
|
||||
- optional previous recap omitted when not prepared;
|
||||
- required previous recap fails when not prepared;
|
||||
- render-debug path works with previous-session inputs;
|
||||
- no S3 calls occur from analyze.
|
||||
|
||||
### Phase 9: Archive `previous/`
|
||||
|
||||
Update archive behavior so the current session's durable `previous/` directory is uploaded/preserved.
|
||||
|
||||
Rules:
|
||||
|
||||
- `previous/` is current-session input/provenance state.
|
||||
- It is not a generated current-session artifact.
|
||||
- Archive it with the current session's durable state, alongside other durable session files according to the current archive layout.
|
||||
- Preserve existing archive commit ordering.
|
||||
- Do not make `previous/` upload the final commit marker.
|
||||
- Do not treat missing `previous/` as an error when no previous-session inputs were prepared.
|
||||
|
||||
Tests:
|
||||
|
||||
- archive includes `previous/manifest.json` and prepared previous artifacts when present;
|
||||
- archive omits or tolerates absent `previous/` when unused;
|
||||
- archive commit ordering remains valid;
|
||||
- cleanup behavior does not delete durable `previous/` before archive commit.
|
||||
|
||||
### Phase 10: Restore `previous/`
|
||||
|
||||
Update `narratio restore` so it restores the current session's archived `previous/` directory.
|
||||
|
||||
Rules:
|
||||
|
||||
- Restore `previous/` as durable current-session state.
|
||||
- Do not infer or restore the previous session workspace merely because `previous_session_id` exists.
|
||||
- Do not redownload previous-session artifacts from the previous session archive during restore; restore the current session's archived `previous/` cache.
|
||||
- Respect existing restore flags and overwrite behavior.
|
||||
|
||||
Tests:
|
||||
|
||||
- restore downloads `previous/` when present;
|
||||
- restore succeeds when `previous/` is absent;
|
||||
- restore with force overwrites local `previous/` according to existing restore semantics;
|
||||
- restored current session can run analyze using `previous/` without needing previous session archive access.
|
||||
|
||||
### Phase 11: Documentation updates after implementation
|
||||
|
||||
After implementation, update current-behavior docs. Do not document implemented behavior only in this roadmap.
|
||||
|
||||
Likely files:
|
||||
|
||||
- `docs/config.md`
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/stage-prepare.md`
|
||||
- `docs/internal/artifacts.md`
|
||||
- `docs/internal/storage.md` only if storage contracts change
|
||||
- `docs/internal/workspace.md`
|
||||
- relevant maintained examples under `examples/`
|
||||
|
||||
Docs should explain:
|
||||
|
||||
- `session.previous_session_id`;
|
||||
- `--previous-session-id` if added;
|
||||
- `narratio.previous_session.artifact.<artifact_name>`;
|
||||
- `previous/` workspace directory;
|
||||
- S3-authoritative previous-session behavior;
|
||||
- required vs optional previous-session artifact handling;
|
||||
- when to run `narratio run-stage --force prepare`.
|
||||
|
||||
Do not document future `--local` support as current behavior.
|
||||
|
||||
## Future Work
|
||||
|
||||
These items are explicitly out of scope for the initial implementation.
|
||||
|
||||
### `--local` previous-session source
|
||||
|
||||
A future flag may allow `prepare` to copy previous-session artifacts from a local previous session workspace instead of S3.
|
||||
|
||||
Possible future command shape:
|
||||
|
||||
```bash
|
||||
narratio run-stage --force prepare --local
|
||||
```
|
||||
|
||||
or a more specific flag such as:
|
||||
|
||||
```bash
|
||||
narratio run-stage --force prepare --previous-source local
|
||||
```
|
||||
|
||||
Do not implement this now.
|
||||
|
||||
### `narratio run --restore`
|
||||
|
||||
A future flag may run `narratio restore` before starting the regular pipeline:
|
||||
|
||||
```bash
|
||||
narratio run --restore --session-id 2026-04-11
|
||||
```
|
||||
|
||||
Do not implement this as part of previous-session artifact support unless it already exists and only requires documentation.
|
||||
|
||||
### Multi-previous-session support
|
||||
|
||||
A future design may support more than one previous/reference session.
|
||||
|
||||
Do not implement this now.
|
||||
|
||||
### Previous-session artifact version pinning
|
||||
|
||||
A future design may pin previous-session artifact inputs to a specific previous run ID or checksum.
|
||||
|
||||
Do not implement this now.
|
||||
|
||||
## Test Plan Summary
|
||||
|
||||
Run at least:
|
||||
|
||||
```bash
|
||||
go test ./internal/config -v
|
||||
go test ./internal/artifacts -v
|
||||
go test ./internal/stage -run Prepare -v
|
||||
go test ./internal/stage -run Analyze -v
|
||||
go test ./internal/app -run TestExecute -v
|
||||
go test ./...
|
||||
```
|
||||
|
||||
Add focused tests for:
|
||||
|
||||
- session config and templating;
|
||||
- previous-session source parsing;
|
||||
- previous artifact requirement scanning;
|
||||
- S3-backed previous artifact download;
|
||||
- prepare skip/force semantics;
|
||||
- manifest input provenance;
|
||||
- resolver behavior;
|
||||
- analyze integration;
|
||||
- archive/restore `previous/` persistence;
|
||||
- examples load/validate.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The feature is complete when:
|
||||
|
||||
1. `session.yml` supports optional `previous_session_id`.
|
||||
2. Session templating supports `previous_session_id`.
|
||||
3. Scriptorium artifact inputs accept `narratio.previous_session.artifact.<artifact_name>`.
|
||||
4. `prepare` scans enabled Scriptorium artifacts for previous-session artifact inputs.
|
||||
5. `prepare` downloads referenced previous-session artifacts from S3 into `previous/`.
|
||||
6. Required/optional behavior is correct.
|
||||
7. `prepare` uses stage-level skip/force behavior and does not compare local and remote copies.
|
||||
8. `analyze` resolves previous-session artifacts only from local prepared `previous/` state.
|
||||
9. `archive` persists `previous/`.
|
||||
10. `restore` restores `previous/`.
|
||||
11. Tests cover config, prepare, resolver, analyze, archive, and restore behavior.
|
||||
12. Current-behavior docs and maintained examples are updated after implementation.
|
||||
13. No storage adapter implementation infers campaign/session/root-prefix semantics.
|
||||
14. No previous-session S3 logic is added to `analyze` or the Scriptorium adapter.
|
||||
318
docs/roadmap/remote.md
Normal file
318
docs/roadmap/remote.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# Roadmap: Campaign State, Remote Sessions, and Archive Locks
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap describes planned, unimplemented work for three related feature areas:
|
||||
|
||||
1. `campaign.yml` configuration for stable campaign-level inputs.
|
||||
2. Remote `session.yml` loading from the existing S3 object-store backend.
|
||||
3. Logical archive locks that prevent selected top-level transcript/artifact promotions from overwriting curated archive state while still preserving run-local outputs.
|
||||
|
||||
Treat this document as an implementation plan, not as current behavior. Keep planned behavior under `docs/roadmap/` until each phase is implemented and canonical docs are updated.
|
||||
|
||||
## Current Code Facts
|
||||
|
||||
The current codebase already settles several design choices:
|
||||
|
||||
- CLI commands use short noun flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--force`, and `--artifacts`.
|
||||
- `run-stage` uses flags before the positional stage name, for example:
|
||||
|
||||
narratio run-stage --session ./session.yml prepare
|
||||
|
||||
- `session.yml` is represented by `config.SessionConfig` and currently owns `session_id`, `previous_session_id`, `campaign`, `date`, `title`, and `inputs`.
|
||||
- Strict YAML decoding is already implemented with `yaml.Decoder.KnownFields(true)`.
|
||||
- Local session discovery is already ordered as `./session.yml`, `/usr/local/etc/narratio/session.yml`, then `/etc/narratio/session.yml`.
|
||||
- The canonical S3 session prefix is already:
|
||||
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session_id}/
|
||||
|
||||
- Archive promotion is already source-based through `archive.promote_artifacts[].source`, with destination derivation and validation in `internal/config`.
|
||||
- Storage adapters receive bucket-relative keys and do not infer campaign, session, run, or root-prefix semantics.
|
||||
|
||||
## Guardrails
|
||||
|
||||
Keep Narratio explicit and stage-driven. Do not introduce a generic workflow engine, broad config language, or stage behavior that reaches through adapter boundaries.
|
||||
|
||||
Implementation must preserve these constraints:
|
||||
|
||||
- Keep storage details behind `internal/adapters/storage`.
|
||||
- Compute session, campaign, archive, and remote config keys in app/artifact/path helpers, not inside storage implementations.
|
||||
- Use centralized path helpers in `internal/artifacts` or the established local path model.
|
||||
- Preserve manifest-driven resume and stage status semantics.
|
||||
- Keep strict YAML decoding for `pipeline.yml`, `campaign.yml`, and `session.yml`.
|
||||
- Keep raw secrets out of configs, manifests, logs, generated configs, archive metadata, and roadmap examples.
|
||||
- Update canonical user-facing docs only after behavior is implemented.
|
||||
|
||||
## Phase 1: Add `campaign.yml`
|
||||
|
||||
Add campaign-level configuration for stable campaign identity and stable input files. Do not add remote campaign loading in this phase.
|
||||
|
||||
### CLI and Discovery
|
||||
|
||||
Add `--campaign <path>` to `run`, `plan`, `resume`, `run-stage`, and `restore`.
|
||||
|
||||
Examples:
|
||||
|
||||
narratio run --campaign ./campaign.yml --session ./session.yml
|
||||
narratio plan --campaign ./campaign.yml --session ./session.yml
|
||||
narratio resume --campaign ./campaign.yml --session ./session.yml
|
||||
narratio run-stage --campaign ./campaign.yml --session ./session.yml prepare
|
||||
narratio restore --campaign ./campaign.yml --session ./session.yml
|
||||
|
||||
Campaign config discovery order:
|
||||
|
||||
1. explicit `--campaign <path>`;
|
||||
2. `./campaign.yml`;
|
||||
3. `/usr/local/etc/narratio/campaign.yml`;
|
||||
4. `/etc/narratio/campaign.yml`.
|
||||
|
||||
Implement this in the same style as `resolvePipelineConfigPath` and `resolveSessionConfigPath`. Add default path constants and a search-path variable in `internal/config/defaults.go`.
|
||||
|
||||
### Config Shape
|
||||
|
||||
Initial `campaign.yml` fields:
|
||||
|
||||
campaign: icewind-dale
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
|
||||
Do not add speculative campaign artifact defaults, prompt defaults, or title conventions in the first implementation.
|
||||
|
||||
### Merge Behavior
|
||||
|
||||
Add `CampaignConfig` and keep the final stage-facing config explicit.
|
||||
|
||||
Required behavior:
|
||||
|
||||
- `pipeline.yml` remains host/runtime configuration.
|
||||
- `campaign.yml` supplies campaign identity and stable input file defaults.
|
||||
- `session.yml` remains the source for `session_id`, `previous_session_id`, `date`, `title`, and audio input.
|
||||
- Campaign-level `speakers_file`, `autocorrect_file`, and `glossary_file` fill missing session-level stable input fields.
|
||||
- Session-level stable input fields override campaign-level stable input fields.
|
||||
- If both `campaign.yml` and `session.yml` specify `campaign`, the values must match.
|
||||
- The resolved session must satisfy the existing session validation rules before stages run.
|
||||
- Unknown fields in `campaign.yml` fail strict decode.
|
||||
|
||||
Path resolution must preserve source-file locality:
|
||||
|
||||
- campaign-provided stable input paths resolve relative to `campaign.yml`;
|
||||
- session-provided stable input overrides resolve relative to `session.yml`;
|
||||
- absolute paths keep existing behavior.
|
||||
|
||||
Track enough provenance in the resolved config or prepare inputs so `prepare` can copy the correct source files without guessing which file supplied each path.
|
||||
|
||||
### Prepare Behavior
|
||||
|
||||
Update `prepare` to materialize the resolved campaign/session inputs into canonical session input paths:
|
||||
|
||||
inputs/campaign.yml
|
||||
inputs/session.yml
|
||||
inputs/pipeline.resolved.yml
|
||||
inputs/speakers.yml
|
||||
inputs/autocorrect.yml
|
||||
inputs/glossary.yml
|
||||
|
||||
Continue recording deterministic `manifest.Inputs` records with checksums. If a prepared input came from `campaign.yml`, record source/provenance using the existing manifest input fields where practical; add narrow metadata only if the existing fields cannot describe it.
|
||||
|
||||
## Phase 2: Load Remote `session.yml`
|
||||
|
||||
Support running with no local session file when a remote session file exists under the canonical session prefix.
|
||||
|
||||
### Preconditions
|
||||
|
||||
Build this phase after `campaign.yml`, because campaign identity is required to compute the remote session key. Do not infer campaign identity from object-store listing.
|
||||
|
||||
### Loading Precedence
|
||||
|
||||
Session loading order:
|
||||
|
||||
1. If `--session <path>` is supplied, load that local file.
|
||||
2. If `--session` is omitted, use existing local discovery: `./session.yml`, `/usr/local/etc/narratio/session.yml`, `/etc/narratio/session.yml`.
|
||||
3. If no local session file is found, `--session-id` is present, storage is configured, and campaign identity is resolved, load remote `session.yml`.
|
||||
4. If no local or remote session can be loaded, fail with a message that lists the local search paths and the remote key that was attempted when applicable.
|
||||
|
||||
Do not make remote loading mask local discovery. Existing local discovery remains the local fallback before remote is attempted. Once remote loading is attempted, a missing remote object, storage init error, or malformed remote YAML fails clearly because no local session was available.
|
||||
|
||||
### Remote Key Layout
|
||||
|
||||
Use the existing canonical S3 layout:
|
||||
|
||||
session prefix: {root_prefix}/campaigns/{campaign}/sessions/{session_id}/
|
||||
session file: {session_prefix}/session.yml
|
||||
audio prefix: {session_prefix}/{session.inputs.audio_s3.prefix}
|
||||
|
||||
Add a centralized helper near `internal/artifacts/s3_keys.go`:
|
||||
|
||||
S3SessionConfigKey(sessionPrefix string) string
|
||||
|
||||
The helper should return `{session_prefix}/session.yml` using the same key normalization style as `S3CurrentManifestKey`, `S3CurrentRunPointerKey`, and `S3PromotedArtifactKey`.
|
||||
|
||||
### Decode, Template, and Provenance
|
||||
|
||||
Remote `session.yml` uses the same template variables and mismatch checks as local sessions:
|
||||
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- `{{previous_session_id}}`
|
||||
- `{{ previous_session_id }}`
|
||||
|
||||
Decode remote session YAML with strict known-field validation. Reuse the current session template/render/decode path by adding a byte/string-based loader rather than duplicating YAML decode logic.
|
||||
|
||||
When `prepare` materializes a remote session into `inputs/session.yml`, record that it came from S3. Preserve useful non-secret provenance when available:
|
||||
|
||||
- bucket;
|
||||
- key;
|
||||
- ETag;
|
||||
- size;
|
||||
- local checksum;
|
||||
- downloaded temp/materialized path.
|
||||
|
||||
## Phase 3: Add Logical Archive Locks
|
||||
|
||||
Add source-based archive locks under `pipeline.archive.locks`. The current promotion system is already source-based, so the first implementation must not support destination-based locks.
|
||||
|
||||
### Config Shape
|
||||
|
||||
Add lock entries:
|
||||
|
||||
archive:
|
||||
locks:
|
||||
- source: narratio.transcript.polished
|
||||
reason: Human-reviewed transcript; do not overwrite automatically.
|
||||
- source: narratio.artifact.session_recap
|
||||
reason: Final recap was manually edited.
|
||||
|
||||
Validation rules:
|
||||
|
||||
- `source` is required.
|
||||
- `source` must be a built-in source ID or configured `narratio.artifact.<key>` accepted by the same source validation used for `promote_artifacts`.
|
||||
- `reason` is optional and non-secret.
|
||||
- duplicate lock sources fail validation.
|
||||
- lock entries do not support `dest` in the first implementation; unknown fields already fail strict decode.
|
||||
|
||||
### Archive Behavior
|
||||
|
||||
Archive must continue uploading complete run-local outputs under `runs/{run_id}/`.
|
||||
|
||||
Promotion behavior:
|
||||
|
||||
1. Resolve promotion source and destination using existing source-based promotion logic.
|
||||
2. If the promotion source is unlocked, upload the top-level promoted object normally.
|
||||
3. If the promotion source is locked, skip only the top-level promotion overwrite.
|
||||
4. Treat locked required promotions as intentional successful skips by default.
|
||||
5. Continue archive commit when all run-local uploads and all non-locked required promotions succeed.
|
||||
6. Upload `current/manifest.json` and `current/run_id.txt` in the existing order, with `current/run_id.txt` last.
|
||||
|
||||
Ordinary `--force` must not override locks. Do not implement a lock-break override in this phase.
|
||||
|
||||
### Metadata
|
||||
|
||||
Record locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions.
|
||||
|
||||
Include:
|
||||
|
||||
- source ID;
|
||||
- destination relative path and remote key;
|
||||
- reason;
|
||||
- local resolved path;
|
||||
- resolved provenance;
|
||||
- whether the original promotion rule was required.
|
||||
|
||||
Keep existing metadata such as `promoted_paths`, `skipped_optional_promotions`, `current_manifest_key`, `current_run_id_key`, and `current_pointer_written`.
|
||||
|
||||
## Phase 4: Future Operator Helpers
|
||||
|
||||
These commands are future work only. Do not implement them with the first campaign, remote-session, or lock changes.
|
||||
|
||||
Potential helper shapes:
|
||||
|
||||
narratio session validate --session-id 2026-06-07
|
||||
narratio session init --session-id 2026-06-07 --title "The Black Cabin"
|
||||
narratio status --session-id 2026-06-07
|
||||
narratio locks --session-id 2026-06-07
|
||||
narratio lock narratio.artifact.session_recap --session-id 2026-06-07
|
||||
narratio unlock narratio.artifact.session_recap --session-id 2026-06-07
|
||||
|
||||
Potential behavior:
|
||||
|
||||
- validate remote session config;
|
||||
- check audio object availability;
|
||||
- inspect committed remote current state;
|
||||
- list promoted transcripts/artifacts;
|
||||
- list archive lock status;
|
||||
- initialize a remote session skeleton;
|
||||
- publish or sync campaign assets.
|
||||
|
||||
## Implementation Sequence
|
||||
|
||||
Use small, reviewable commits.
|
||||
|
||||
1. Campaign config types and discovery:
|
||||
add `CampaignConfig`, strict loading, defaults/search paths, `--campaign` flags, and config/app tests.
|
||||
2. Campaign/session merge:
|
||||
implement resolved stable input merge, path provenance, validation, and prepare materialization.
|
||||
3. Campaign docs after implementation:
|
||||
update canonical docs and examples only for implemented behavior.
|
||||
4. Remote session key and loader:
|
||||
add `S3SessionConfigKey`, byte/string session loading, remote download through `ObjectStore`, and app-level precedence tests.
|
||||
5. Remote session prepare provenance:
|
||||
materialize downloaded session config and record S3 provenance.
|
||||
6. Remote session docs after implementation:
|
||||
update canonical docs and examples only after behavior exists.
|
||||
7. Archive lock config:
|
||||
add lock config structs, strict decode coverage, source validation, and duplicate detection.
|
||||
8. Archive lock enforcement:
|
||||
skip locked top-level promotions, preserve run-local uploads, record lock metadata, and protect commit ordering.
|
||||
9. Final sweep:
|
||||
run focused tests, then `go test ./...`; verify planned behavior remains only in roadmap docs until implemented.
|
||||
|
||||
## Test Plan
|
||||
|
||||
Add focused coverage in these packages:
|
||||
|
||||
- `internal/config`: campaign load, strict decode, discovery constants, merge validation, campaign/session mismatch, lock validation, duplicate lock rejection.
|
||||
- `internal/app`: `--campaign` parsing on `run`, `plan`, `resume`, `run-stage`, and `restore`; campaign discovery; explicit `--session` precedence; local discovery before remote; remote session fallback when local discovery misses.
|
||||
- `internal/artifacts`: `S3SessionConfigKey`; canonical session prefix compatibility; source ID validation for lock sources.
|
||||
- `internal/stage/prepare`: campaign/session stable input materialization; campaign-relative and session-relative path resolution; remote session provenance in `manifest.Inputs`.
|
||||
- `internal/stage/archive`: locked required promotion succeeds as skipped; unlocked promotion uploads; run-local outputs upload when top-level promotion is locked; `--force` does not break locks; `current/run_id.txt` remains the last upload.
|
||||
- `internal/adapters/storage`: fake object key normalization and remote session download expectations.
|
||||
|
||||
Run at least:
|
||||
|
||||
go test ./internal/config -v
|
||||
go test ./internal/app -run TestExecute -v
|
||||
go test ./internal/artifacts -v
|
||||
go test ./internal/stage -run 'Prepare|Archive' -v
|
||||
go test ./internal/adapters/storage -v
|
||||
go test ./...
|
||||
|
||||
Use fake storage for remote-session and archive-lock behavior. Ordinary tests must not require live S3.
|
||||
|
||||
## Documentation Updates After Implementation
|
||||
|
||||
After each phase is implemented, update only docs for behavior that exists.
|
||||
|
||||
Likely files:
|
||||
|
||||
- `docs/config.md`
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/stage-prepare.md`
|
||||
- `docs/internal/stage-archive.md`
|
||||
- `docs/internal/storage.md`
|
||||
- `docs/internal/artifacts.md`
|
||||
- relevant examples under `examples/`
|
||||
|
||||
Do not document remote campaign loading, helper commands, or lock override flags as current behavior until implemented.
|
||||
|
||||
## Remaining Open Decisions
|
||||
|
||||
The codebase resolves the campaign flag name, campaign discovery order, remote session layout, local-vs-remote session precedence, source-based archive lock model, and locked required promotion policy.
|
||||
|
||||
Remaining decisions:
|
||||
|
||||
1. Whether `campaign.yml` should eventually be loadable from S3. Do not implement remote campaign loading in the first phase.
|
||||
2. Whether a future explicit lock override command or flag is needed. Do not make ordinary `--force` break locks.
|
||||
3. Whether future helper commands should be top-level commands or subcommands. Keep them out of the first implementation.
|
||||
Reference in New Issue
Block a user