diff --git a/docs/cli.md b/docs/cli.md index 0dd4c5e..4dd1436 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -6,7 +6,9 @@ narratio run --session-id 2026-04-04 ``` -This command uses default discovery for `pipeline.yml`, `campaign.yml`, and local `session.yml`. If local session discovery misses and S3 storage is configured, `--session-id` can load remote `session.yml` from the canonical session prefix. +This command uses default system discovery for `pipeline.yml`, `campaign.yml`, and local `session.yml`. If local session discovery misses and S3 storage is configured, `--session-id` can load remote `session.yml` from the canonical session prefix. + +Default discovery checks system config locations only. Pass `--config`, `--campaign`, and `--session` to use files from the current working directory. ## Command Overview @@ -109,7 +111,7 @@ Success output: - `narratio run: session ; executed= skipped=; manifest=` Common failure cases: -- missing default config/campaign/session paths when flags omitted. +- missing system default config/campaign/session paths when flags omitted. - missing local session plus missing/unavailable remote `session.yml`. - invalid template/rendered session mismatch. - unknown/invalid `--artifacts` value. diff --git a/docs/config.md b/docs/config.md index 9e87ce0..3cf37a3 100644 --- a/docs/config.md +++ b/docs/config.md @@ -28,7 +28,15 @@ Behavior: ## 2. Config file discovery -Pipeline config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`: +These commands use the same config discovery behavior: + +- `narratio run` +- `narratio plan` +- `narratio resume` +- `narratio run-stage` +- `narratio restore` + +Pipeline config lookup: - if `--config ` is provided, that path is used. - if omitted, Narratio searches in order: @@ -36,30 +44,29 @@ Pipeline config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`: 2. `/etc/narratio/pipeline.yml` - first existing file wins. -Campaign config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`: +Campaign config lookup: - if `--campaign ` is provided, that path is used. - if omitted, Narratio searches in order: - 1. `./campaign.yml` - 2. `/usr/local/etc/narratio/campaign.yml` - 3. `/etc/narratio/campaign.yml` + 1. `/usr/local/etc/narratio/campaign.yml` + 2. `/etc/narratio/campaign.yml` - first existing file wins. -## 3. Session file discovery and templating - -Session config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`: +Session config lookup: - if `--session ` is provided, that path is used. - if `--session` is omitted, Narratio searches locally in order: - 1. `./session.yml` - 2. `/usr/local/etc/narratio/session.yml` - 3. `/etc/narratio/session.yml` + 1. `/usr/local/etc/narratio/session.yml` + 2. `/etc/narratio/session.yml` - first existing local file wins. - if no local session file is found, `--session-id ` is present, storage is configured, and campaign identity is resolved, Narratio loads remote `session.yml` from: - `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml` - local discovery always runs before remote fallback. +- local files in the current working directory are used only when passed explicitly, for example `--config ./pipeline.yml --campaign ./campaign.yml --session ./session.yml`. -Template behavior: +## 3. Session templating + +Template behavior for local and remote `session.yml`: - supported placeholders: - `{{session_id}}` @@ -72,7 +79,9 @@ Template behavior: - if rendered `session_id` mismatches `--session-id`, load fails. - if rendered `previous_session_id` mismatches `--previous-session-id`, load fails. -## 4. Minimal pipeline config +## 4. Minimal config set + +### `pipeline.yml` ```yaml whisperx: @@ -85,9 +94,7 @@ Why this is sufficient: - `workspace.root` defaults to `/var/lib/narratio`. - optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive. -## 5. Minimal session template - -`campaign.yml`: +### `campaign.yml` ```yaml campaign: sample-campaign @@ -97,7 +104,12 @@ inputs: glossary_file: ./glossary.yml ``` -`session.yml`: +Why this is sufficient: + +- `campaign` supplies the stable campaign identity. +- stable input files are required and resolve relative to `campaign.yml` when copied during `prepare`. + +### `session.yml` ```yaml session_id: "{{ session_id }}" @@ -105,7 +117,14 @@ inputs: audio_dir: ./audio ``` -Usage: +Why this is sufficient: + +- `session_id` is required and can be rendered from `--session-id`. +- `campaign` can be omitted because it is supplied by `campaign.yml`. +- stable input paths can be omitted because `campaign.yml` supplies defaults. +- local `audio_dir` resolves relative to `session.yml`. + +Minimal local-file usage: ```bash narratio run --config /path/to/pipeline.yml --campaign ./campaign.yml --session ./session.yml --session-id 2026-05-03 @@ -124,7 +143,9 @@ inputs: narratio run --config /path/to/pipeline.yml --campaign ./campaign.yml --session ./session.yml --session-id 2026-05-03 --previous-session-id 2026-04-26 ``` -## 6. Production-oriented config +## 5. Production-oriented config set + +### `pipeline.yml` ```yaml workspace: @@ -176,6 +197,42 @@ scriptorium: required: false ``` +### `campaign.yml` + +```yaml +campaign: forsaken +inputs: + speakers_file: /srv/narratio/campaigns/forsaken/speakers.yml + autocorrect_file: /srv/narratio/campaigns/forsaken/autocorrect.yml + glossary_file: /srv/narratio/campaigns/forsaken/glossary.yml +``` + +### Local `session.yml` + +```yaml +session_id: "{{ session_id }}" +previous_session_id: "{{ previous_session_id }}" +date: 2026-05-03 +title: The Black Cabin +inputs: + audio_s3: + prefix: audio/ +``` + +### S3-first session config + +For S3-first operation, upload the same `session.yml` content to: + +```text +{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml +``` + +Then run with explicit or discovered pipeline/campaign config and no `--session`: + +```bash +narratio run --config /usr/local/etc/narratio/pipeline.yml --campaign /usr/local/etc/narratio/campaign.yml --session-id 2026-05-03 --previous-session-id 2026-04-26 +``` + Operational notes: - archive promotion is explicit and source-based via `archive.promote_artifacts`. @@ -184,7 +241,7 @@ Operational notes: - Narratio does not auto-promote all generated analyze artifacts. - `restore` reads the same config/campaign/session inputs and restore scope is bounded by committed archive current state. -## 7. Full pipeline reference +## 6. Full pipeline reference | Path | Type | Required | Default | | --- | --- | --- | --- | @@ -336,7 +393,7 @@ Restore-related implications: - restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs). - restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`). -## 8. Full campaign reference +## 7. Full campaign reference | Path | Type | Required | Default | | --- | --- | --- | --- | @@ -347,7 +404,7 @@ Restore-related implications: Campaign input paths may be absolute or relative. Relative paths resolve from the directory containing `campaign.yml`. -## 9. Full session reference +## 8. Full session reference | Path | Type | Required | Default | | --- | --- | --- | --- | @@ -378,7 +435,7 @@ Previous-session rule: - if `session.previous_session_id` is set, it must not equal `session.session_id`. - canonical previous-session sources (`narratio.previous_session.artifact.`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts. -## 10. Secrets +## 9. Secrets Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`. @@ -397,7 +454,7 @@ Guidance: - do not put secret values directly in YAML. - configure env var names in config and provide values via env/secrets files. -## 11. Examples +## 10. Examples Maintained examples: diff --git a/docs/operations.md b/docs/operations.md index add0860..238713b 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -6,7 +6,7 @@ For field-level configuration, see [docs/config.md](./config.md). For full comma ## Normal workflow (S3-first path) -1. Upload `session.yml` to the configured session prefix, or keep a local `session.yml` available. +1. Upload `session.yml` to the configured session prefix, or pass a local `session.yml` explicitly. 2. Upload session `.flac` files to object storage under the configured session audio prefix. 3. Run Narratio: @@ -19,7 +19,7 @@ narratio run --session-id 2026-04-04 - use `manifest=` with `status` for inspection. Notes: -- default config/campaign/session discovery applies unless `--config`, `--campaign`, and `--session` are passed. +- default config/campaign/session discovery checks system config locations unless `--config`, `--campaign`, and `--session` are passed. - when local `session.yml` discovery misses, `--session-id` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. - S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access. diff --git a/docs/roadmap/remote.md b/docs/roadmap/remote.md index f0b87d1..3a916c1 100644 --- a/docs/roadmap/remote.md +++ b/docs/roadmap/remote.md @@ -17,11 +17,11 @@ The current codebase already settles several design choices: - CLI commands use short noun flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--force`, and `--artifacts`. - `run-stage` uses flags before the positional stage name, for example: - narratio run-stage --session ./session.yml prepare + narratio run-stage --config ./pipeline.yml --campaign ./campaign.yml --session ./session.yml prepare - `session.yml` is represented by `config.SessionConfig` and currently owns `session_id`, `previous_session_id`, `campaign`, `date`, `title`, and `inputs`. - Strict YAML decoding is already implemented with `yaml.Decoder.KnownFields(true)`. -- Local session discovery is already ordered as `./session.yml`, `/usr/local/etc/narratio/session.yml`, then `/etc/narratio/session.yml`. +- Default local config discovery uses system paths under `/usr/local/etc/narratio/` and `/etc/narratio/`; working-directory files are used only when passed explicitly. - The canonical S3 session prefix is already: {root_prefix}/campaigns/{campaign}/sessions/{session_id}/ @@ -62,9 +62,8 @@ Examples: Campaign config discovery order: 1. explicit `--campaign `; -2. `./campaign.yml`; -3. `/usr/local/etc/narratio/campaign.yml`; -4. `/etc/narratio/campaign.yml`. +2. `/usr/local/etc/narratio/campaign.yml`; +3. `/etc/narratio/campaign.yml`. Implement this in the same style as `resolvePipelineConfigPath` and `resolveSessionConfigPath`. Add default path constants and a search-path variable in `internal/config/defaults.go`. @@ -129,7 +128,7 @@ Build this phase after `campaign.yml`, because campaign identity is required to Session loading order: 1. If `--session ` is supplied, load that local file. -2. If `--session` is omitted, use existing local discovery: `./session.yml`, `/usr/local/etc/narratio/session.yml`, `/etc/narratio/session.yml`. +2. If `--session` is omitted, use existing local discovery: `/usr/local/etc/narratio/session.yml`, `/etc/narratio/session.yml`. 3. If no local session file is found, `--session-id` is present, storage is configured, and campaign identity is resolved, load remote `session.yml`. 4. If no local or remote session can be loaded, fail with a message that lists the local search paths and the remote key that was attempted when applicable. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index a69f578..604d3c8 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -9,19 +9,19 @@ Symptom: - `run`, `plan`, `resume`, `run-stage`, or `restore` fails with config/session not found. Likely Cause: -- `pipeline.yml` or `session.yml` is missing from discovery paths. -- wrong working directory when relying on `./session.yml`. +- `pipeline.yml`, `campaign.yml`, or `session.yml` is missing from system discovery paths. +- a local working-directory config file was not passed explicitly. Diagnostics: ```bash -pwd -ls -l ./session.yml ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml +ls -l /usr/local/etc/narratio/campaign.yml /etc/narratio/campaign.yml +ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml ``` Safe Fix: -- pass explicit `--config` and `--session`. +- pass explicit `--config`, `--campaign`, and `--session`. - or place files in documented discovery paths. Links: @@ -40,7 +40,7 @@ Likely Cause: Diagnostics: ```bash -narratio plan --session ./session.yml --session-id 2026-04-04 +narratio plan --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session ./session.yml --session-id 2026-04-04 ``` Safe Fix: @@ -62,7 +62,7 @@ Likely Cause: Diagnostics: ```bash -narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 +narratio plan --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 ``` Safe Fix: @@ -84,7 +84,7 @@ Likely Cause: Diagnostics: ```bash -narratio run --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout +narratio run --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout ``` Safe Fix: @@ -106,7 +106,7 @@ Likely Cause: Diagnostics: ```bash -narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts session_recap polish +narratio run-stage --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts session_recap polish ``` Safe Fix: @@ -129,7 +129,7 @@ Likely Cause: Diagnostics: ```bash -narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 +narratio plan --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 ``` Safe Fix: @@ -154,7 +154,7 @@ Diagnostics: ```bash narratio status --manifest /path/to/manifest.json -narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout analyze +narratio run-stage --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout analyze ``` Safe Fix: @@ -227,7 +227,7 @@ Likely Cause: Diagnostics: ```bash -narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run +narratio restore --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run ``` Safe Fix: @@ -250,7 +250,7 @@ Likely Cause: Diagnostics: ```bash -narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run +narratio restore --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run ``` Safe Fix: @@ -272,7 +272,7 @@ Likely Cause: Diagnostics: ```bash -narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run +narratio restore --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run ``` Safe Fix: @@ -342,7 +342,7 @@ Likely Cause: Diagnostics: ```bash -narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 prepare +narratio run-stage --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 prepare ``` Safe Fix: @@ -366,7 +366,7 @@ Diagnostics: ```bash narratio status --manifest /path/to/manifest.json -narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 archive +narratio run-stage --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 archive ``` Safe Fix: diff --git a/internal/app/campaign_config_path_test.go b/internal/app/campaign_config_path_test.go index 50cca1d..fca8458 100644 --- a/internal/app/campaign_config_path_test.go +++ b/internal/app/campaign_config_path_test.go @@ -36,7 +36,7 @@ func TestResolveCampaignConfigPathUsesFirstExistingDefault(t *testing.T) { } func TestResolveCampaignConfigPathErrorIncludesSearchedPaths(t *testing.T) { - _, err := resolveCampaignConfigPathWithCandidates("", []string{"./campaign.yml", "/usr/local/etc/narratio/campaign.yml", "/etc/narratio/campaign.yml"}) + _, err := resolveCampaignConfigPathWithCandidates("", []string{"/usr/local/etc/narratio/campaign.yml", "/etc/narratio/campaign.yml"}) if err == nil { t.Fatal("expected error, got nil") } diff --git a/internal/app/remote_session_test.go b/internal/app/remote_session_test.go index 1a7087e..fcca606 100644 --- a/internal/app/remote_session_test.go +++ b/internal/app/remote_session_test.go @@ -4,7 +4,6 @@ import ( "bytes" "context" "errors" - "os" "path/filepath" "strings" "testing" @@ -66,16 +65,7 @@ func TestExecuteLocalSessionDiscoveryPrecedenceSkipsRemote(t *testing.T) { pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot) fake := &storage.FakeBackend{} var storeInitCalls int - restoreAppConfigTestGlobals(t, fake, &storeInitCalls, config.DefaultSessionConfigSearchPaths) - - originalWD, err := os.Getwd() - if err != nil { - t.Fatalf("Getwd(): %v", err) - } - if err := os.Chdir(filepath.Dir(sessionPath)); err != nil { - t.Fatalf("Chdir(%q): %v", filepath.Dir(sessionPath), err) - } - t.Cleanup(func() { _ = os.Chdir(originalWD) }) + restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath}) var stdout bytes.Buffer var stderr bytes.Buffer diff --git a/internal/app/session_cli_test.go b/internal/app/session_cli_test.go index b512e69..70f5a6a 100644 --- a/internal/app/session_cli_test.go +++ b/internal/app/session_cli_test.go @@ -4,14 +4,15 @@ import ( "bytes" "context" "os" - "path/filepath" "strings" "testing" + + "gitea.maximumdirect.net/eric/narratio/internal/config" ) func TestPlanUsesDiscoveredSessionTemplateWithSessionIDs(t *testing.T) { workspaceRoot := t.TempDir() - pipelinePath, _, sessionPath := writeValidConfigFiles(t, workspaceRoot) + pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot) sessionTemplate := `session_id: "{{ session_id }}" previous_session_id: "{{ previous_session_id }}" @@ -26,19 +27,14 @@ inputs: t.Fatalf("write session template: %v", err) } - cwd := filepath.Dir(sessionPath) - originalWD, err := os.Getwd() - if err != nil { - t.Fatalf("Getwd(): %v", err) - } - if err := os.Chdir(cwd); err != nil { - t.Fatalf("Chdir(%q): %v", cwd, err) - } - t.Cleanup(func() { _ = os.Chdir(originalWD) }) + origSessionDefaults := append([]string(nil), config.DefaultSessionConfigSearchPaths...) + config.DefaultSessionConfigSearchPaths = []string{sessionPath} + t.Cleanup(func() { config.DefaultSessionConfigSearchPaths = origSessionDefaults }) var out bytes.Buffer if err := Plan(context.Background(), []string{ "--config", pipelinePath, + "--campaign", campaignPath, "--session-id", "2026-04-04", "--previous-session-id", "2026-03-28", }, &out); err != nil { @@ -111,7 +107,7 @@ func TestRunStageAcceptsSessionIDFlagAndParsesStageName(t *testing.T) { } func TestResolveSessionConfigPathErrorIncludesSearchedPaths(t *testing.T) { - _, err := resolveSessionConfigPathWithCandidates("", []string{"./session.yml", "/usr/local/etc/narratio/session.yml", "/etc/narratio/session.yml"}) + _, err := resolveSessionConfigPathWithCandidates("", []string{"/usr/local/etc/narratio/session.yml", "/etc/narratio/session.yml"}) if err == nil { t.Fatal("expected error, got nil") } diff --git a/internal/config/campaign_config_test.go b/internal/config/campaign_config_test.go index 1dfaac4..bbab106 100644 --- a/internal/config/campaign_config_test.go +++ b/internal/config/campaign_config_test.go @@ -9,7 +9,6 @@ import ( func TestCampaignConfigDefaultSearchOrder(t *testing.T) { want := []string{ - "./campaign.yml", "/usr/local/etc/narratio/campaign.yml", "/etc/narratio/campaign.yml", } diff --git a/internal/config/defaults.go b/internal/config/defaults.go index 86b413e..ab6b48b 100644 --- a/internal/config/defaults.go +++ b/internal/config/defaults.go @@ -1,14 +1,12 @@ package config -// Default filesystem locations for pipeline configuration lookup when --config -// is omitted. Order is highest to lowest precedence. +// Default filesystem locations for config lookup when config path flags are +// omitted. Order is highest to lowest precedence. const ( DefaultPipelineConfigPathUsrLocal = "/usr/local/etc/narratio/pipeline.yml" DefaultPipelineConfigPathEtc = "/etc/narratio/pipeline.yml" - DefaultCampaignConfigPathLocal = "./campaign.yml" DefaultCampaignConfigPathUsrLocal = "/usr/local/etc/narratio/campaign.yml" DefaultCampaignConfigPathEtc = "/etc/narratio/campaign.yml" - DefaultSessionConfigPathLocal = "./session.yml" DefaultSessionConfigPathUsrLocal = "/usr/local/etc/narratio/session.yml" DefaultSessionConfigPathEtc = "/etc/narratio/session.yml" DefaultS3AccessKeyIDEnv = "OBJECT_STORAGE_KEY_ID" @@ -97,7 +95,6 @@ var DefaultPipelineConfigSearchPaths = []string{ // Keep this in a variable so future defaults can be extended without changing // call sites. var DefaultCampaignConfigSearchPaths = []string{ - DefaultCampaignConfigPathLocal, DefaultCampaignConfigPathUsrLocal, DefaultCampaignConfigPathEtc, } @@ -108,7 +105,6 @@ var DefaultCampaignConfigSearchPaths = []string{ // Keep this in a variable so future defaults can be extended without changing // call sites. var DefaultSessionConfigSearchPaths = []string{ - DefaultSessionConfigPathLocal, DefaultSessionConfigPathUsrLocal, DefaultSessionConfigPathEtc, }