# Troubleshooting ## Purpose Canonical operator troubleshooting guide for recurring implemented Narratio failures. ## Config file discovery failure Symptom: - `run`, `plan`, `resume`, or `run-stage` fails saying config/session file was not found. Likely Cause: - `pipeline.yml` or `session.yml` is missing from default search paths. - Wrong working directory when relying on `./session.yml`. Diagnostics: ```bash pwd ls -l ./session.yml ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml ``` Safe Fix: - Pass explicit paths with `--config` and `--session`. - Or place files in documented discovery paths. Links: - [docs/config.md](./config.md) - [docs/cli.md](./cli.md) ## Session template rendering failure Symptom: - Load fails with unresolved template placeholder or `session_id` mismatch. Likely Cause: - `session.yml` contains `{{session_id}}`/`{{ session_id }}` but `--session-id` was omitted. - Provided `--session-id` does not match rendered `session_id`. Diagnostics: ```bash narratio plan --session ./session.yml --session-id 2026-04-04 ``` Safe Fix: - Always pass `--session-id` when using template placeholders. - Ensure rendered `session_id` equals intended run session id. Links: - [docs/config.md](./config.md) ## Strict YAML decode or validation failure Symptom: - Config load fails with unknown field, missing required field, invalid duration, or invalid cross-field constraint. Likely Cause: - YAML key typo or stale field name. - Required fields missing. - Invalid value format (for example duration/URL/env var name). Diagnostics: ```bash narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 ``` Safe Fix: - Correct fields/values to match canonical reference and examples. - Validate against `docs/examples/` shapes. Links: - [docs/config.md](./config.md) - [docs/examples/](./examples/) ## Manifest/status path failure Symptom: - `status` fails because manifest path is missing, unreadable, or invalid. Likely Cause: - Wrong manifest path. - Manifest removed after cleanup. - Trying to run `status` without `--manifest`. Diagnostics: ```bash narratio status --manifest /path/to/manifest.json ls -l /path/to/manifest.json ``` Safe Fix: - Use manifest path printed by `run`, `resume`, or `run-stage` output. - Re-run with correct session/config if inspecting a different session. Links: - [docs/cli.md](./cli.md) - [docs/operations.md](./operations.md) ## Session lock conflict (`.lock`) Symptom: - Run fails with lock conflict indicating session workdir is already locked. Likely Cause: - Another Narratio process is actively running the same session. - Prior run exited unexpectedly and left a stale lock file. Diagnostics: ```bash ls -l {workspace.root}/work/{campaign}/{session_id}/.lock cat {workspace.root}/work/{campaign}/{session_id}/.lock ps aux | grep narratio ``` Safe Fix: - If another run is active, wait for it to finish. - If no process is active and lock is stale, remove only that session `.lock` file and retry. Links: - [docs/operations.md](./operations.md) - [docs/internal/workspace.md](./internal/workspace.md) ## Secrets env-dir or credential env failure Symptom: - Startup fails loading secrets directory, or a stage fails because required credential env var is missing. Likely Cause: - `pipeline.secrets.env_dir` path is wrong/unreadable. - Credential env var referenced in config is unset or empty. Diagnostics: ```bash ls -la /path/to/secrets_dir env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM' ``` Safe Fix: - Fix `pipeline.secrets.env_dir` path/permissions. - Ensure required env vars are set to non-empty values. - Keep secrets out of YAML; use env references only. Links: - [docs/config.md](./config.md) - [docs/operations.md](./operations.md) ## S3-audio prepare failure Symptom: - `prepare` fails in S3 mode (no audio found, list/download failure, backend missing, path conflict). Likely Cause: - Wrong `session.inputs.audio_s3.prefix`. - No `.flac` files at expected prefix. - Missing or invalid S3 backend credentials/config. - Conflicting audio-source settings (`audio_s3` plus local audio fields). Diagnostics: ```bash narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 prepare ``` Safe Fix: - Ensure `audio_s3` is the only audio source configured for that session. - Confirm `.flac` objects exist under the resolved session audio prefix. - Fix S3 storage configuration and credentials. Links: - [docs/config.md](./config.md) - [docs/operations.md](./operations.md) - [docs/internal/stage-prepare.md](./internal/stage-prepare.md) ## Archive prerequisite or promotion/current-pointer failure Symptom: - `archive` fails due to prerequisite stage status, missing required promotion source, or pointer write failure. Likely Cause: - One or more prerequisite stages are not `succeeded`. - Required promoted artifact does not exist. - Remote upload failure before `current/run_id.txt` write. Diagnostics: ```bash narratio status --manifest /path/to/manifest.json narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 archive ``` Safe Fix: - Resume or rerun failed upstream stage(s). - Ensure required promoted artifact paths exist locally before archive. - Retry archive after storage/connectivity issue is resolved. Links: - [docs/operations.md](./operations.md) - [docs/config.md](./config.md) - [docs/internal/stage-archive.md](./internal/stage-archive.md) ## `run-stage` invalid stage name or invalid flags Symptom: - `run-stage` fails with unknown stage or invalid flag/argument usage. Likely Cause: - Stage name typo. - Missing positional stage argument. - Unsupported/incorrect flag syntax. Diagnostics: ```bash narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 normalize ``` Safe Fix: - Use only supported stage names. - Provide exactly one positional stage argument. - Align flags to documented command reference. Links: - [docs/cli.md](./cli.md) - [docs/operations.md](./operations.md)