Rewrite docs for the publish stage contract and current behavior
This commit is contained in:
@@ -1,17 +1,17 @@
|
||||
# Troubleshooting
|
||||
|
||||
## Purpose
|
||||
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
|
||||
Canonical operator troubleshooting guide for recurring Narratio failures.
|
||||
|
||||
## Config file discovery failure
|
||||
## Config discovery failure
|
||||
|
||||
Symptom:
|
||||
- `run`, `resume`, `run-stage`, `session plan`, or `session restore` fails with config/session not found.
|
||||
- command fails because `pipeline.yml`, `campaign.yml`, or `session.yml` was not found.
|
||||
|
||||
Likely Cause:
|
||||
- `pipeline.yml` or `session.yml` is missing from system discovery paths.
|
||||
- the selected campaign ID does not exist under `pipeline.campaigns.root`.
|
||||
- a local working-directory config file was not passed explicitly.
|
||||
Likely cause:
|
||||
- missing files in discovery paths.
|
||||
- missing/incorrect campaign selection.
|
||||
- local file exists but was not passed explicitly.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
@@ -20,160 +20,73 @@ ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
|
||||
ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- pass explicit `--config`, `--campaign <id>`, `--campaign-file <path>`, and `--session` as appropriate.
|
||||
- or place files in documented discovery paths and set `pipeline.campaigns.default_campaign_id`.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/cli.md](./cli.md)
|
||||
Safe fix:
|
||||
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
|
||||
|
||||
## Templated session file rejected
|
||||
|
||||
Symptom:
|
||||
- load fails with a message that `session.yml must be concrete`.
|
||||
- load fails because `session.yml` must be concrete.
|
||||
|
||||
Likely Cause:
|
||||
- a template authoring file such as `session.template.yml` was passed to `--session` or uploaded as remote `session.yml`.
|
||||
- `session.yml` still contains `{{ ... }}` placeholders.
|
||||
Likely cause:
|
||||
- template placeholders (`{{ ... }}`) still present in loaded session config.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session ./session.yml
|
||||
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- generate concrete YAML with `narratio session init`.
|
||||
- pass the generated concrete `session.yml` to downstream commands or upload it through `session init --remote`.
|
||||
Safe fix:
|
||||
- generate concrete session YAML via `narratio session init`.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## Strict YAML decode or validation failure
|
||||
## Strict decode or validation failure
|
||||
|
||||
Symptom:
|
||||
- config load fails with unknown field or validation error.
|
||||
- unknown field or invalid value error during config load.
|
||||
|
||||
Likely Cause:
|
||||
- typo/stale field name.
|
||||
- missing required fields or invalid constraints.
|
||||
Likely cause:
|
||||
- typo, stale field name, or invalid value.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
|
||||
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- align fields/values to canonical config reference and examples.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [examples/](../examples/)
|
||||
Safe fix:
|
||||
- align config with [docs/config.md](./config.md) and maintained examples.
|
||||
|
||||
## `--artifacts` selection failure
|
||||
|
||||
Symptom:
|
||||
- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection.
|
||||
- command fails on unknown/invalid selected artifact key.
|
||||
|
||||
Likely Cause:
|
||||
- `--artifacts` contains blank names or unknown artifact keys.
|
||||
- `pipeline.scriptorium.artifacts` missing while using `--artifacts`.
|
||||
Likely cause:
|
||||
- artifact key not defined in `pipeline.scriptorium.artifacts`.
|
||||
- empty token in `--artifacts` input.
|
||||
|
||||
Diagnostics:
|
||||
Safe fix:
|
||||
- use only configured artifact keys.
|
||||
|
||||
```bash
|
||||
narratio run 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use configured artifact keys only.
|
||||
- ensure `pipeline.scriptorium.artifacts` is defined.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## `run-stage --artifacts` on unsupported stage
|
||||
## `run-stage --artifacts` unsupported stage
|
||||
|
||||
Symptom:
|
||||
- `run-stage` fails because `--artifacts` is only supported for `analyze` and `archive`.
|
||||
- `run-stage` rejects `--artifacts` for the selected stage.
|
||||
|
||||
Likely Cause:
|
||||
- `--artifacts` was used with a stage other than `analyze` or `archive`.
|
||||
Likely cause:
|
||||
- `--artifacts` used with a stage other than `analyze` or `publish`.
|
||||
|
||||
Diagnostics:
|
||||
Safe fix:
|
||||
- use `--artifacts` only with `run-stage analyze ...` or `run-stage publish ...`.
|
||||
|
||||
```bash
|
||||
narratio run-stage polish 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts session_recap
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use `--artifacts` only with `run-stage analyze ...` or `run-stage archive ...`.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
|
||||
## Configured artifact dependency/input validation failure
|
||||
## Previous-session input unavailable
|
||||
|
||||
Symptom:
|
||||
- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path.
|
||||
- analyze fails on required previous-session artifact input.
|
||||
|
||||
Likely Cause:
|
||||
- `narratio.artifact.<name>` source missing matching `depends_on` key.
|
||||
- dependency references unknown artifact key.
|
||||
- dependency self-reference or enabled dependency cycle.
|
||||
- artifact output path missing/invalid/outside `artifacts/` root.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys.
|
||||
- ensure referenced artifacts exist and define valid `output_path` values.
|
||||
- keep output paths relative and under `artifacts/`.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-analyze.md](./internal/stage-analyze.md)
|
||||
|
||||
## Required configured artifact input unavailable at analyze time
|
||||
|
||||
Symptom:
|
||||
- analyze fails because configured input source is unavailable.
|
||||
|
||||
Likely Cause:
|
||||
- required upstream configured artifact was not selected/executed this run.
|
||||
- non-executable dependency output file is missing or invalid on disk.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session status 2026-04-04
|
||||
narratio run-stage analyze 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- run analyze with needed artifacts selected.
|
||||
- or ensure dependency output file exists at configured path and is valid.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## Manifest/status path failure
|
||||
|
||||
Symptom:
|
||||
- `session status` fails because config/session state is missing, unreadable, or invalid.
|
||||
|
||||
Likely Cause:
|
||||
- wrong session ID.
|
||||
- wrong config/campaign/session file selected.
|
||||
- manifest removed after cleanup.
|
||||
Likely cause:
|
||||
- `previous/**` cache not hydrated for this session.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
@@ -181,137 +94,75 @@ Diagnostics:
|
||||
narratio session status 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use the same session ID and config files that will be used for `run`, `resume`, or `run-stage`.
|
||||
Safe fix:
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
```bash
|
||||
narratio run-stage prepare 2026-04-04 --force
|
||||
```
|
||||
|
||||
Or rehydrate from remote current state:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04
|
||||
```
|
||||
|
||||
## Session lock conflict (`.lock`)
|
||||
|
||||
Symptom:
|
||||
- `run`, `resume`, `run-stage`, or `session restore` fails with lock conflict for session workdir.
|
||||
- command fails with lock conflict.
|
||||
|
||||
Likely Cause:
|
||||
- another Narratio process is running same session.
|
||||
- stale lock from interrupted prior run.
|
||||
Likely cause:
|
||||
- another process is running for the same session.
|
||||
- stale lock file from interrupted command.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
cat {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
ps aux | grep narratio
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- wait for active process to finish.
|
||||
- if no process is active, remove only stale session `.lock` file.
|
||||
Safe fix:
|
||||
- wait for active process; remove stale lock only if no process is active.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/workspace.md](./internal/workspace.md)
|
||||
|
||||
## Restore remote current pointer or manifest missing
|
||||
## Restore current pointer/manifest missing
|
||||
|
||||
Symptom:
|
||||
- `session restore` fails with remote current pointer or current manifest errors.
|
||||
- restore fails reading remote current state.
|
||||
|
||||
Likely Cause:
|
||||
- `current/run_id.txt` was never published.
|
||||
- `current/manifest.json` is missing for the session prefix.
|
||||
- archive commit did not complete.
|
||||
Likely cause:
|
||||
- publish commit did not complete.
|
||||
- `current/run_id.txt` or `current/manifest.json` is missing.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
|
||||
narratio session restore 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- verify archive stage succeeded for the target session.
|
||||
- rerun/archive from a healthy source workspace so current pointers are published.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
||||
|
||||
## Restore manifest identity mismatch
|
||||
|
||||
Symptom:
|
||||
- `session restore` fails because remote manifest session or campaign does not match requested values.
|
||||
|
||||
Likely Cause:
|
||||
- wrong positional session ID or wrong session config selected.
|
||||
- archive prefix points to a different campaign/session.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use the correct session config and positional session ID.
|
||||
- verify campaign/session identity in local config before restore.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
Safe fix:
|
||||
- republish from a healthy local session state.
|
||||
|
||||
## Restore conflict without `--force`
|
||||
|
||||
Symptom:
|
||||
- `session restore` fails with `restore conflict` and conflict counts.
|
||||
- restore reports conflict and exits.
|
||||
|
||||
Likely Cause:
|
||||
- local durable file differs from remote file for one or more planned restore paths.
|
||||
Likely cause:
|
||||
- local durable file differs from remote restore source.
|
||||
|
||||
Diagnostics:
|
||||
Safe fix:
|
||||
- inspect with `--dry-run`.
|
||||
- rerun with `--force` only when remote should overwrite local.
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- review planned conflicts.
|
||||
- rerun with `--force` only when remote state should overwrite local state.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Restore report expectations
|
||||
## Secrets or credentials failure
|
||||
|
||||
Symptom:
|
||||
- operator expects restore report file but does not find one.
|
||||
- startup fails loading secrets dir, or storage/tool auth fails at runtime.
|
||||
|
||||
Likely Cause:
|
||||
- restore was executed in `--dry-run` mode.
|
||||
- restore failed before report persistence path (for example lock acquisition failure).
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- run non-dry-run restore for durable report output.
|
||||
- resolve lock or early preflight failures and retry.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Secrets env-dir or credential-env failure
|
||||
|
||||
Symptom:
|
||||
- startup fails loading secrets directory, or stage fails due to missing credential env vars.
|
||||
|
||||
Likely Cause:
|
||||
- invalid `pipeline.secrets.env_dir` path/permissions.
|
||||
- required credential env var unset/empty.
|
||||
Likely cause:
|
||||
- invalid `pipeline.secrets.env_dir`.
|
||||
- missing credential env vars.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
@@ -320,60 +171,53 @@ ls -la /path/to/secrets_dir
|
||||
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- fix secrets directory and credential env vars.
|
||||
- keep secret values out of YAML.
|
||||
Safe fix:
|
||||
- fix path/permissions/env vars; keep secret values out of YAML.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## S3-audio prepare failure
|
||||
## S3 audio prepare failure
|
||||
|
||||
Symptom:
|
||||
- `prepare` fails in S3 mode (listing/downloading/no audio/backend error).
|
||||
- prepare fails in S3 mode (list/download/no files/backend error).
|
||||
|
||||
Likely Cause:
|
||||
- wrong `session.inputs.audio_s3.prefix`.
|
||||
- no `.flac` files at resolved prefix.
|
||||
- invalid/missing object-store credentials or backend config.
|
||||
- mixed local+S3 audio input config.
|
||||
Likely cause:
|
||||
- bad `session.inputs.audio_s3.prefix`.
|
||||
- no `.flac` objects at prefix.
|
||||
- bad storage credentials/config.
|
||||
- mixed local+S3 audio config.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run-stage prepare 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
|
||||
narratio run-stage prepare 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- configure exactly one audio source mode.
|
||||
- verify `.flac` files and storage access.
|
||||
Safe fix:
|
||||
- configure exactly one audio mode and verify storage access.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Archive promotion/current-pointer failure
|
||||
## Publish output or current-pointer failure
|
||||
|
||||
Symptom:
|
||||
- archive fails on required promotion source missing or pointer write failure.
|
||||
- publish fails on required output source missing, upload error, or commit-marker write failure.
|
||||
|
||||
Likely Cause:
|
||||
- required promoted file absent (including analyze outputs not generated for this run).
|
||||
- storage upload failed before `current/run_id.txt` commit marker write.
|
||||
Likely cause:
|
||||
- required source file not produced.
|
||||
- storage upload failed before `current/run_id.txt` write.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session status 2026-04-04
|
||||
narratio run-stage archive 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
|
||||
narratio run-stage publish 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- rerun or resume upstream stages to generate required files.
|
||||
- adjust promotion `source`/`dest` rules to match artifacts that must exist.
|
||||
- retry after storage issue is resolved.
|
||||
Safe fix:
|
||||
- rerun upstream stages to regenerate required outputs.
|
||||
- adjust `pipeline.publish.outputs` source/dest rules.
|
||||
- retry after storage issue is fixed.
|
||||
|
||||
## Helpful Links
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/stage-publish.md](./internal/stage-publish.md)
|
||||
|
||||
Reference in New Issue
Block a user