230 lines
6.0 KiB
Markdown
230 lines
6.0 KiB
Markdown
# Troubleshooting
|
|
|
|
## Purpose
|
|
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
|
|
|
|
## Config file discovery failure
|
|
|
|
Symptom:
|
|
- `run`, `plan`, `resume`, or `run-stage` fails saying config/session file was not found.
|
|
|
|
Likely Cause:
|
|
- `pipeline.yml` or `session.yml` is missing from default search paths.
|
|
- Wrong working directory when relying on `./session.yml`.
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
pwd
|
|
ls -l ./session.yml
|
|
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
|
|
```
|
|
|
|
Safe Fix:
|
|
- Pass explicit paths with `--config` and `--session`.
|
|
- Or place files in documented discovery paths.
|
|
|
|
Links:
|
|
- [docs/config.md](./config.md)
|
|
- [docs/cli.md](./cli.md)
|
|
|
|
## Session template rendering failure
|
|
|
|
Symptom:
|
|
- Load fails with unresolved template placeholder or `session_id` mismatch.
|
|
|
|
Likely Cause:
|
|
- `session.yml` contains `{{session_id}}`/`{{ session_id }}` but `--session-id` was omitted.
|
|
- Provided `--session-id` does not match rendered `session_id`.
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
narratio plan --session ./session.yml --session-id 2026-04-04
|
|
```
|
|
|
|
Safe Fix:
|
|
- Always pass `--session-id` when using template placeholders.
|
|
- Ensure rendered `session_id` equals intended run session id.
|
|
|
|
Links:
|
|
- [docs/config.md](./config.md)
|
|
|
|
## Strict YAML decode or validation failure
|
|
|
|
Symptom:
|
|
- Config load fails with unknown field, missing required field, invalid duration, or invalid cross-field constraint.
|
|
|
|
Likely Cause:
|
|
- YAML key typo or stale field name.
|
|
- Required fields missing.
|
|
- Invalid value format (for example duration/URL/env var name).
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04
|
|
```
|
|
|
|
Safe Fix:
|
|
- Correct fields/values to match canonical reference and examples.
|
|
- Validate against `docs/examples/` shapes.
|
|
|
|
Links:
|
|
- [docs/config.md](./config.md)
|
|
- [docs/examples/](./examples/)
|
|
|
|
## Manifest/status path failure
|
|
|
|
Symptom:
|
|
- `status` fails because manifest path is missing, unreadable, or invalid.
|
|
|
|
Likely Cause:
|
|
- Wrong manifest path.
|
|
- Manifest removed after cleanup.
|
|
- Trying to run `status` without `--manifest`.
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
narratio status --manifest /path/to/manifest.json
|
|
ls -l /path/to/manifest.json
|
|
```
|
|
|
|
Safe Fix:
|
|
- Use manifest path printed by `run`, `resume`, or `run-stage` output.
|
|
- Re-run with correct session/config if inspecting a different session.
|
|
|
|
Links:
|
|
- [docs/cli.md](./cli.md)
|
|
- [docs/operations.md](./operations.md)
|
|
|
|
## Session lock conflict (`.lock`)
|
|
|
|
Symptom:
|
|
- Run fails with lock conflict indicating session workdir is already locked.
|
|
|
|
Likely Cause:
|
|
- Another Narratio process is actively running the same session.
|
|
- Prior run exited unexpectedly and left a stale lock file.
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
|
|
cat {workspace.root}/work/{campaign}/{session_id}/.lock
|
|
ps aux | grep narratio
|
|
```
|
|
|
|
Safe Fix:
|
|
- If another run is active, wait for it to finish.
|
|
- If no process is active and lock is stale, remove only that session `.lock` file and retry.
|
|
|
|
Links:
|
|
- [docs/operations.md](./operations.md)
|
|
- [docs/internal/workspace.md](./internal/workspace.md)
|
|
|
|
## Secrets env-dir or credential env failure
|
|
|
|
Symptom:
|
|
- Startup fails loading secrets directory, or a stage fails because required credential env var is missing.
|
|
|
|
Likely Cause:
|
|
- `pipeline.secrets.env_dir` path is wrong/unreadable.
|
|
- Credential env var referenced in config is unset or empty.
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
ls -la /path/to/secrets_dir
|
|
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
|
|
```
|
|
|
|
Safe Fix:
|
|
- Fix `pipeline.secrets.env_dir` path/permissions.
|
|
- Ensure required env vars are set to non-empty values.
|
|
- Keep secrets out of YAML; use env references only.
|
|
|
|
Links:
|
|
- [docs/config.md](./config.md)
|
|
- [docs/operations.md](./operations.md)
|
|
|
|
## S3-audio prepare failure
|
|
|
|
Symptom:
|
|
- `prepare` fails in S3 mode (no audio found, list/download failure, backend missing, path conflict).
|
|
|
|
Likely Cause:
|
|
- Wrong `session.inputs.audio_s3.prefix`.
|
|
- No `.flac` files at expected prefix.
|
|
- Missing or invalid S3 backend credentials/config.
|
|
- Conflicting audio-source settings (`audio_s3` plus local audio fields).
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 prepare
|
|
```
|
|
|
|
Safe Fix:
|
|
- Ensure `audio_s3` is the only audio source configured for that session.
|
|
- Confirm `.flac` objects exist under the resolved session audio prefix.
|
|
- Fix S3 storage configuration and credentials.
|
|
|
|
Links:
|
|
- [docs/config.md](./config.md)
|
|
- [docs/operations.md](./operations.md)
|
|
- [docs/internal/stage-prepare.md](./internal/stage-prepare.md)
|
|
|
|
## Archive prerequisite or promotion/current-pointer failure
|
|
|
|
Symptom:
|
|
- `archive` fails due to prerequisite stage status, missing required promotion source, or pointer write failure.
|
|
|
|
Likely Cause:
|
|
- One or more prerequisite stages are not `succeeded`.
|
|
- Required promoted artifact does not exist.
|
|
- Remote upload failure before `current/run_id.txt` write.
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
narratio status --manifest /path/to/manifest.json
|
|
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 archive
|
|
```
|
|
|
|
Safe Fix:
|
|
- Resume or rerun failed upstream stage(s).
|
|
- Ensure required promoted artifact paths exist locally before archive.
|
|
- Retry archive after storage/connectivity issue is resolved.
|
|
|
|
Links:
|
|
- [docs/operations.md](./operations.md)
|
|
- [docs/config.md](./config.md)
|
|
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
|
|
|
## `run-stage` invalid stage name or invalid flags
|
|
|
|
Symptom:
|
|
- `run-stage` fails with unknown stage or invalid flag/argument usage.
|
|
|
|
Likely Cause:
|
|
- Stage name typo.
|
|
- Missing positional stage argument.
|
|
- Unsupported/incorrect flag syntax.
|
|
|
|
Diagnostics:
|
|
|
|
```bash
|
|
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 normalize
|
|
```
|
|
|
|
Safe Fix:
|
|
- Use only supported stage names.
|
|
- Provide exactly one positional stage argument.
|
|
- Align flags to documented command reference.
|
|
|
|
Links:
|
|
- [docs/cli.md](./cli.md)
|
|
- [docs/operations.md](./operations.md)
|