5.0 KiB
5.0 KiB
Troubleshooting
Purpose
Canonical operator troubleshooting guide for recurring Narratio failures.
Config discovery failure
Symptom:
- command fails because
pipeline.yml,campaign.yml, orsession.ymlwas not found.
Likely cause:
- missing files in discovery paths.
- missing/incorrect campaign selection.
- local file exists but was not passed explicitly.
Diagnostics:
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml
Safe fix:
- pass explicit
--config,--campaignor--campaign-file, and--session.
Templated session file rejected
Symptom:
- load fails because
session.ymlmust be concrete.
Likely cause:
- template placeholders (
{{ ... }}) still present in loaded session config.
Diagnostics:
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
Safe fix:
- generate concrete session YAML via
narratio session init.
Strict decode or validation failure
Symptom:
- unknown field or invalid value error during config load.
Likely cause:
- typo, stale field name, or invalid value.
Diagnostics:
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
Safe fix:
- align config with docs/config.md and maintained examples.
--artifacts selection failure
Symptom:
- command fails on unknown/invalid selected artifact key.
Likely cause:
- artifact key not defined in
pipeline.scriptorium.artifacts. - empty token in
--artifactsinput.
Safe fix:
- use only configured artifact keys.
run-stage --artifacts unsupported stage
Symptom:
run-stagerejects--artifactsfor the selected stage.
Likely cause:
--artifactsused with a stage other thananalyzeorpublish.
Safe fix:
- use
--artifactsonly withrun-stage analyze ...orrun-stage publish ....
Previous-session input unavailable
Symptom:
- analyze fails on required previous-session artifact input.
Likely cause:
previous/**cache not hydrated for this session.
Diagnostics:
narratio session status 2026-04-04
Safe fix:
narratio run-stage prepare 2026-04-04 --force
Or rehydrate from remote current state:
narratio session restore 2026-04-04
Session lock conflict (.lock)
Symptom:
- command fails with lock conflict.
Likely cause:
- another process is running for the same session.
- stale lock file from interrupted command.
Diagnostics:
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
ps aux | grep narratio
Safe fix:
- wait for active process; remove stale lock only if no process is active.
Restore current pointer/manifest missing
Symptom:
- restore fails reading remote current state.
Likely cause:
- publish commit did not complete.
current/run_id.txtorcurrent/manifest.jsonis missing.
Diagnostics:
narratio session restore 2026-04-04 --dry-run
Safe fix:
- republish from a healthy local session state.
Restore conflict without --force
Symptom:
- restore reports conflict and exits.
Likely cause:
- local durable file differs from remote restore source.
Safe fix:
- inspect with
--dry-run. - rerun with
--forceonly when remote should overwrite local.
Secrets or credentials failure
Symptom:
- startup fails loading secrets dir, or storage/tool auth fails at runtime.
Likely cause:
- invalid
pipeline.secrets.env_dir. - missing credential env vars.
Diagnostics:
ls -la /path/to/secrets_dir
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
Safe fix:
- fix path/permissions/env vars; keep secret values out of YAML.
S3 audio prepare failure
Symptom:
- prepare fails in S3 mode (list/download/no files/backend error).
Likely cause:
- bad
session.inputs.audio_s3.prefix. - no
.flacobjects at prefix. - bad storage credentials/config.
- mixed local+S3 audio config.
Diagnostics:
narratio run-stage prepare 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
Safe fix:
- configure exactly one audio mode and verify storage access.
Publish output or current-pointer failure
Symptom:
- publish fails on required output source missing, upload error, or commit-marker write failure.
Likely cause:
- required source file not produced.
- storage upload failed before
current/run_id.txtwrite.
Diagnostics:
narratio session status 2026-04-04
narratio run-stage publish 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
Safe fix:
- rerun upstream stages to regenerate required outputs.
- adjust
pipeline.publish.outputssource/dest rules. - retry after storage issue is fixed.