Files
narratio/docs/troubleshooting.md

29 KiB

Troubleshooting

Operational diagnosis guide for common Narratio failures.

Config file not found

Symptom:

  • command fails to resolve pipeline.yml, campaign.yml, or session.yml.

Likely causes:

  • missing files in default search paths;
  • wrong campaign selection;
  • omitted explicit flags.

Diagnostics:

narratio session plan 2026-04-04

Safe fix:

  • pass explicit --config, --campaign or --campaign-file, and --session.

Relevant reference: Configuration discovery.

Session template placeholders rejected

Symptom:

  • load error says session file must be concrete or contains {{ ... }} placeholders.

Likely cause:

  • using template content as runtime session config.

Diagnostics:

narratio session validate 2026-04-04 --session /path/session.yml

Safe fix:

  • generate concrete session YAML with narratio session init.

Relevant reference: Operations: Session Initialization.

Strict decode or schema validation failure

Symptom:

  • unknown field / invalid value error during config load.

Likely cause:

  • stale field name, typo, invalid enum, invalid duration/path format, or an unsupported local manifest schema/state marker.

Diagnostics:

narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml

Safe fix:

  • align config with Configuration and the maintained examples.
  • for a manifest schema error, move the unsupported local state aside and regenerate it; restore a verified current committed snapshot or republish healthy regenerated state when appropriate. Do not add version or stage markers by hand.
  • if an older manifest is rejected for its retired cleanup-obligation field, finish outstanding cleanup with the pre-upgrade binary. Otherwise verify the remote commit, move aside or regenerate the incompatible manifest, and use the cleanup stage. Do not treat manual JSON editing as the normal migration path.

Relevant reference: Configuration.

Party schema version rejected

Symptom:

  • configuration loading reports that party.schema_version is required or unsupported.

Likely cause:

  • the campaign party.yml is an unversioned legacy document, or names a schema other than narratio.party.v1.

Safe fix:

  • replace the file with one exact, single-document narratio.party.v1 party; do not add a version marker to an otherwise incompatible legacy structure. The versioned party is the only configured roster authority.

Relevant reference: Canonical Party Input.

Unexpected imported or overlay value

Symptom:

  • an effective configuration value differs from the root file, or a duplicate ownership/configuration error is hard to locate.

Diagnostics:

narratio config sources --config /path/pipeline.yml --overlay testing

Add --campaign or --campaign-file when the pipeline has party-driven artifact families. The output identifies each effective logical field's root, import, overlay, default, campaign, party, or family source without printing the field value or credential contents.

Safe fix:

  • move a duplicated base field so it has one owner;
  • correct the selected overlay; or
  • correct the campaign party/family declaration that owns generated values.

To review what would actually change before switching overlays, run config diff with the same pipeline and campaign selectors. It compares normalized effective values rather than YAML formatting or source-file layout.

Relevant reference: Configuration inspection.

Audio mode conflict

Symptom:

  • validation fails on session audio configuration.

Likely cause:

  • configured both local and S3 session audio inputs.

Diagnostics:

narratio session validate 2026-04-04

Safe fix:

  • use local mode (audio_dir or audio_files) or S3 mode (audio_s3.prefix), not both.

Relevant reference: Session configuration.

--artifacts selection error

Symptom:

  • unknown artifact key or invalid --artifacts usage.

Likely causes:

  • key not defined in pipeline.scriptorium.artifacts;
  • empty list entry (for example trailing comma);
  • run-stage used with non-analyze/publish target.

Diagnostics:

narratio session artifacts 2026-04-04

Safe fix:

  • provide only configured keys and use --artifacts with supported commands/stages.

Relevant reference: CLI artifact selection.

Bounded run prerequisite is unusable

Symptom:

  • run or session plan reports that a prerequisite stage is absent or has a pending, running, failed, stale, or interrupted status before the selected start.

Likely cause:

  • --from excludes upstream work that has not reached the terminal succeeded or skipped state in the session manifest.

Diagnostics:

narratio session status 2026-04-04
narratio session plan 2026-04-04 --from render --through analyze

Safe fix:

  • widen the bounded range to include the first reported stage, or recover that stage explicitly with run-stage before retrying. The failed check does not create a run-local directory or modify the manifest. Narratio does not resume-validate excluded prefix stages, and stages after --through are not prerequisites.

If prerequisite statuses are terminal but a selected stage reports a missing, unsafe, or checksum-inconsistent artifact, repair the artifact at the stage that owns it; do not edit the manifest to bypass the selected stage's concrete input validation.

Relevant reference: Operations: Stage Execution and Continuation Behavior.

Transcription retries speakers that previously finished

Symptom:

  • an ordinary transcribe retry calls WhisperX for a speaker that appeared to finish earlier; or
  • an older successful session retranscribes all speakers once after upgrade.

Likely causes:

  • the earlier response or canonical file was not followed by a successful speaker manifest checkpoint;
  • the prepared audio bytes/path, WhisperX URL, or language changed;
  • the canonical transcript is missing, unsafe, non-regular, invalid JSON, or no longer matches its recorded size/checksum;
  • the previous invocation used --force, which durably cleared pre-force speaker authority; or
  • the manifest has only legacy aggregate transcribe state and cannot bind old outputs to their source audio.

Safe fix:

  • inspect the reported speaker and filesystem error, repair the prepared input or canonical-output path when needed, then retry transcribe normally;
  • allow a legacy aggregate-only record to regenerate once; and
  • do not edit the manifest or copy an old JSON file into place as a substitute for a successful checkpoint.
narratio run-stage transcribe 2026-04-04

Valid earlier speaker checkpoints remain reusable even while aggregate transcribe status is failed or interrupted. Merge remains gated until transcribe reaches aggregate success with the complete current set.

Relevant reference: Operations: Transcription continuation and replacement.

WhisperX model changed but transcription was reused

Symptom:

  • WhisperX model weights, prompts, or private server configuration changed at the same configured URL, but Narratio still considers speaker transcripts current.

Likely cause:

  • Narratio can fingerprint its configured URL and language, not private state loaded behind the service boundary.

Safe fix:

  • explicitly force transcribe to establish a durable reset, then use ordinary retry if the replacement attempt is interrupted.
narratio run-stage transcribe 2026-04-04 --force

Transport replay and application retries do not provide an exactly-once remote execution guarantee. Investigate server-side failures or duplicate work using WhisperX diagnostics; do not infer completion from a server log without the local speaker checkpoint.

Relevant references: WhisperX integration and Transcribe internals.

Transcription manifest checkpoint failed

Symptom:

  • transcribe reports that a speaker completed but the session manifest could not be saved.

Likely causes:

  • the workspace is unwritable or out of space;
  • the manifest path or parent directory is unsafe or the wrong type; or
  • a transient filesystem error interrupted atomic replacement.

Safe fix:

  • correct the reported filesystem or permission problem and retry transcribe normally. Narratio reconciles from the last readable collection and reuses only successfully checkpointed speakers.

A canonical raw transcript left by the failed checkpoint is incidental and does not become reusable authority. Preserve the manifest and diagnostics if the failure repeats; do not advertise the file by editing manifest evidence.

Relevant references: Manifest persistence and Transcribe internals.

Notarius credential reference unavailable

Symptom:

  • configuration loading reports secret.unset, secret.empty, or an invalid name for pipeline.notarius.llm_api_key, including during inspection or a run that would not execute extraction.

Likely causes:

  • the named environment variable is absent or empty;
  • the selected secrets directory lacks a file with that exact name;
  • a process environment variable exists but is empty, so bootstrap correctly leaves it in place; or
  • the Narratio reference name does not match api_key_env in the selected Notarius/PromptKit profile.

Safe fix:

  • set the exact nonempty environment variable or provide an equivalently named file through the conventional or explicit secrets directory;
  • remove or set the field to null when that deployment needs no credential; and
  • use config show, config sources, and config diff to inspect the public reference name and ownership without exposing its value.

Narratio resolves selected credential references during ordinary pipeline loading, before stage selection or resume. Credential changes alone do not invalidate a completed extraction, but the current reference must still resolve for the command to proceed.

Relevant references: Secrets and Notarius credential.

Notarius executable missing

Symptom:

  • extraction fails while resolving or starting the Notarius executable.

Likely causes:

  • pipeline.notarius.binary is not installed, executable, or on PATH;
  • a configured executable path is wrong.

Safe fix:

  • install a compatible Notarius release or correct the binary setting, then rerun extraction.

Relevant references: Notarius configuration and Notarius integration.

Notarius exits nonzero

Symptom:

  • extraction reports a Notarius exit error instead of a receipt.

Diagnostics:

  • inspect runs/{run_id}/extract/notarius.stderr.log; stdout is reserved for the receipt and is not merged with diagnostics.

Safe fix:

  • correct the reported Notarius pipeline, input, provider, or configuration failure and rerun extraction. Do not edit a staged output bundle into place.

After a failed replacement, an older immutable bundle may still exist even though the current session manifest has no successful extraction payload. This is expected audit state, not a signal to relink the old bundle manually.

Relevant reference: Operations: Extraction Workflow.

Prepared Notarius reference missing or inconsistent

Symptom:

  • extraction or resume validation reports that a configured reference source is unavailable, unsafe, empty, or checksum-inconsistent and recommends prepare --force.

Likely causes:

  • prepare has not run since the campaign/session stable input changed;
  • the configured source file is missing;
  • a prepared inputs/ file or its manifest record was modified independently;
  • a spell-catalog binding exists without an effective spell_catalog_file.

Diagnostics:

narratio session status 2026-04-04
narratio session validate 2026-04-04

Safe fix:

  • correct the campaign/session input path, then refresh canonical prepared evidence before extraction:
narratio run-stage prepare 2026-04-04 --force

Do not point Notarius directly at the original source path or edit the manifest checksum. Relevant references: Notarius reference configuration and Operations: Extraction Workflow.

Notarius reference snapshot cache conflict

Symptom:

  • extraction reports that a path beneath cache/notarius/references/<sha256>/ is a symlink, directory, or another non-regular or unsafe entry.

Safe fix:

  • stop commands for the affected session and inspect the exact reported path;
  • investigate unexpected links or special files as possible tampering; and
  • remove only the reported conflicting leaf or digest directory after confirming it belongs to the reconstructible reference cache.

A missing snapshot or corrupt regular snapshot is reconstructed automatically from currently verified prepared input. Do not recursively delete the session root, edit the manifest checksum, or remove cache entries while extraction is active.

Relevant reference: Operations: Extraction Workflow.

Notarius resume executes work instead of reusing it

Symptom:

  • Narratio executes extract, and Notarius makes provider calls despite pipeline.notarius.resume: true.

Likely causes:

  • checkpoint recording is disabled in the Notarius configuration;
  • Notarius is using a nonpersistent or different checkpoint root;
  • transcript/reference bytes, reference paths, pipeline/profile content, local-backend endpoint, selected lanes, or another checkpoint identity input changed; or
  • Narratio extraction was forced, which deliberately omits --resume.

Safe fix:

  • validate the deployed Notarius configuration and configure explicit persistent chunk-plan and checkpoint roots;
  • inspect Notarius checkpoint decision reason codes for a legitimate cold miss; and
  • retry without force only when reuse is intended.

Remember that Narratio skipping a completed extract stage is separate from Notarius reusing work inside an extraction that does execute. Narratio does not read the Notarius configuration to infer checkpoint enablement.

Relevant references: Notarius cache controls and Notarius integration.

Notarius reference selector or generated-handoff collision

Symptom:

  • Notarius exits nonzero with an undeclared reference-slot, incompatible media, or external/generated reference collision error.

Likely causes:

  • a selector does not identify a slot declared by the selected Notarius target;
  • a prepared file does not satisfy that slot's Notarius media contract; or
  • a CLI binding attempts to replace a same-run generated D&D handoff.

Safe fix:

  • compare external bindings with the selected Notarius pipeline's canonical consumer documentation;
  • keep only campaign-owned external slots on the CLI; and
  • leave registry, scene, combat, and occurrence handoffs to Notarius pipeline composition.

Narratio validates selector structure and prepared evidence, while Notarius owns slot declarations, media compatibility, and generated-handoff conflicts. Relevant reference: Notarius integration.

Notarius receipt or index incompatible

Symptom:

  • extraction rejects the receipt schema, pipeline identity, bundle/index path, lane descriptor, or payload path even though Notarius exited successfully.

Likely causes:

  • Narratio and Notarius versions disagree on their consumer contract;
  • the configured pipeline or lane constraints are stale;
  • output paths escape the bundle or traverse symlinks.

Safe fix:

  • compare installed Notarius output with the canonical Notarius contracts, including receipt index_file: index.json and index management names manifest.json, rejected.json, warnings.json, and diagnostics.json; align pipeline.notarius constraints and rerun. Do not bypass confinement or schema checks.

Relevant reference: Notarius integration.

Required Notarius lane rejected or missing

Symptom:

  • extraction fails because a configured lane is rejected, missing, duplicated, or incompatible, including after a zero exit.

Safe fix:

  • inspect the Notarius diagnostic log and bundle rejection/warning information;
  • correct the Notarius module or the exact declared lane contract;
  • remove an output declaration only if downstream consumers genuinely no longer require that source, then rerun extraction.

Every configured output is required. Narratio does not promote a partial result.

Extraction resume invalidated

Symptom:

  • a previously successful extraction runs again during ordinary continuation.

Likely causes:

  • the pipeline ID or configured output contracts changed;
  • a configured prepared reference selector, source, path, checksum, or byte size changed;
  • the direct trimmed transcript identity or bytes changed; or
  • the durable bundle, index, lane set, provenance, regular-file status, or checksum no longer validates.

Safe fix:

  • allow the automatic rerun after verifying the current configuration. Treat an unsafe path or bundle symlink error as filesystem corruption or tampering and investigate it rather than replacing files manually.

Executable, config path, timeout, working directory, and Notarius credential changes are operational and do not by themselves invalidate reusable extraction evidence. A newly selected credential must nevertheless resolve during command configuration loading.

Notarius transitive configuration changed

Symptom:

  • Notarius profiles, prompts, modules, imported files, or references changed, but Narratio still considers the previous extraction resumable.

Safe fix:

narratio run-stage extract 2026-04-04 --force

Narratio fingerprints its invocation contract and prepared Narratio reference identities, not the contents of other transitive Notarius inputs. Always force extraction after changing those external inputs; downstream successful stages are then marked stale normally.

Relevant reference: Operations: Extraction Workflow.

Analysis artifact evidence is not current

Symptom:

  • ordinary continuation or session plan schedules one or more configured artifacts even though a canonical output file exists; or
  • publish reports a configured artifact source unavailable.

Likely causes:

  • the per-artifact record is stale, missing, failed, unselected, or malformed;
  • a configured prompt/profile, dependency, input identity, output path, or effective variable changed; or
  • the recorded output is missing, unsafe, empty, or has a size/checksum that no longer matches its manifest evidence.

Diagnostics:

narratio session status 2026-04-04
narratio session artifacts 2026-04-04
narratio session plan 2026-04-04 --from analyze --through analyze

Safe fix:

  • investigate unexpected path or checksum changes as possible tampering;
  • otherwise let the selected analyze work rerun, or explicitly regenerate only the affected targets; and
  • never edit the fingerprint/checksum in the manifest or copy an old file into the canonical path as a substitute for current evidence.
narratio analyze 2026-04-04 --artifacts session_recap

Relevant references: Operations: Artifact Selection and Artifact Internals.

Analysis manifest checkpoint failed

Symptom:

  • analyze reports that it could not save the session manifest after completing an artifact.

Likely causes:

  • the workspace is unwritable or out of space;
  • the manifest path or its parent directory has unsafe permissions or type; or
  • a transient local filesystem error interrupted atomic manifest replacement.

Safe fix:

  • correct the reported filesystem or permission problem;
  • inspect session artifacts to identify the last durable current records; and
  • rerun analyze normally. Narratio reuses earlier successful checkpoints and retries the failed or non-current artifact and its required dependents.

Do not treat a newly created canonical artifact file as current when its checkpoint failed, and do not edit the manifest to advertise it. If the error recurs, preserve the manifest and run-local diagnostics before repairing the workspace.

Relevant references: Operations: Stage Execution and Continuation Behavior and Manifest Internals.

Built-in transcript or bounds source is unavailable

Symptom:

  • analyze, extract, render, or publish says a built-in source is unavailable even though a file exists at its usual session path.

Likely cause:

  • the producing stage has not succeeded, its output lacks the exact registered source, run ID, or checksum, its evidence is duplicated or stale, or the durable bytes no longer match the manifest checksum.

Safe fix:

  • do not copy or rename the file to make it discoverable. Rerun the producing stage to regenerate both the bytes and current manifest evidence, then retry the consumer.

An older manifest schema is rejected

Symptom:

  • Narratio rejects a local session manifest because schema_version is missing or unsupported; or
  • restore rejects a committed remote session manifest for the same reason.

Likely causes:

  • the manifest predates the supported canonical state contract. Narratio rejects it at the schema boundary rather than interpreting aggregate output lists or incidental files as current authority; or
  • an older strict-decoding Narratio binary is reading a current manifest that contains additive transcribe speaker state it does not recognize.

Safe fix:

  • for unsupported local state, move the old manifest aside and regenerate, or restore a verified current committed snapshot when one is authoritative;
  • for unsupported committed remote state, regenerate healthy local state as needed and republish the session; and
  • for a downgrade rejection, return to the current binary or restore a verified pre-upgrade backup. Do not delete only the speaker fields by hand.
narratio run 2026-04-04
narratio publish 2026-04-04

After current records exist, inspect them and publish explicitly. Do not copy old files into canonical paths merely to influence selection; availability is manifest-owned.

Relevant references: Operations: Stage Execution and Continuation Behavior and Manifest Internals.

Scriptorium private input changed without a rerun

Symptom:

  • a prompt, profile, imported configuration file, executable, or other input loaded privately by Scriptorium changed, but Narratio still considers an artifact current.

Likely cause:

  • analysis fingerprints cover Narratio-observable semantic identities, not executable contents or arbitrary files and transitive configuration that Scriptorium loads behind its configured paths and identifiers.

Safe fix:

  • explicitly force the affected target after changing an unobserved private input. Force applies to explicit targets; current prerequisites remain reusable unless selected themselves.
narratio analyze 2026-04-04 --artifacts session_recap

Relevant reference: Analyze Internals.

Previous-session artifact input missing

Symptom:

  • prepare/analyze fails due to missing required previous-session artifact cache input.

Likely causes:

  • missing session.previous_session_id;
  • the selected previous-session commit does not declare the required canonical source as a published_output; or
  • the declared published object is unavailable or fails identity validation.

Diagnostics:

narratio session validate 2026-04-04
narratio session status 2026-04-04

Safe fix:

narratio session restore 2026-04-04

or rerun prepare after correcting session config:

narratio run-stage prepare 2026-04-04 --force

Relevant reference: Operations: Restore Workflow.

Session lock conflict (.lock)

Symptom:

  • command fails acquiring session lock.

Likely causes:

  • another process is running for the same session;
  • a process still holds the operating-system lock while it is shutting down.

Diagnostics:

ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
ps aux | grep narratio

Safe fix:

  • wait for active process completion;
  • retry after an interrupted holder has exited; the kernel releases its lock even though the .lock metadata file remains for inspection.

Relevant reference: Operations: Local State Layout.

Restore conflict without --force

Symptom:

  • restore fails with conflict count.

Likely cause:

  • local durable files differ from remote restore sources.
  • a restore target or one of its ancestors is a symlink or another non-regular filesystem entry.

Diagnostics:

narratio session restore 2026-04-04 --dry-run

Safe fix:

  • review conflicts;
  • rerun with --force only when remote state should overwrite an eligible regular local file; repair a symlink or other non-regular conflict first.

If the filesystem changed after the plan was printed, rerun restore after repairing the destination; installation performs the same safety check again.

Relevant reference: Operations: Restore Workflow.

Restore current-state discovery failure

Symptom:

  • restore cannot find current pointer or current manifest.

Likely causes:

  • no committed publish current state;
  • storage credentials or connectivity failure.

Diagnostics:

narratio session status 2026-04-04
narratio session restore 2026-04-04 --dry-run

Safe fix:

  • resolve storage/auth issue;
  • republish from healthy local state if current pointer is missing.

Relevant reference: Operations: Publish Workflow.

Publish output failure

Symptom:

  • publish fails on missing required source, upload error, or commit write.

Likely causes:

  • required source file not produced;
  • lock/state expectations mismatch;
  • remote storage failure.

Diagnostics:

narratio session artifacts 2026-04-04 --remote
narratio session status 2026-04-04
narratio run-stage publish 2026-04-04 --force

Safe fix:

  • regenerate missing sources by rerunning prerequisite stages;
  • correct publish source/destination rules;
  • retry after storage failure is resolved.

Relevant reference: Publish configuration.

Render markdown source missing

Symptom:

  • analyze or publish fails because narratio.transcript.final_markdown or narratio.transcript.final_trimmed_markdown is unavailable.

Likely causes:

  • render stage was not executed after transcript changes;
  • render stage failed before producing canonical markdown outputs.

Diagnostics:

narratio session status 2026-04-04

Safe fix:

  • rerun render and then retry downstream stage(s):
narratio run-stage render 2026-04-04 --force
narratio run-stage analyze 2026-04-04 --force

Relevant reference: Operations: Stage Execution.

Secrets or storage credential failure

Symptom:

  • object-store command fails at initialization/auth.

Likely causes:

  • missing, unreadable, or invalid explicit --secrets-dir;
  • invalid entries in the conventional secrets/ directory beside the root pipeline;
  • missing credential environment variables;
  • invalid S3 endpoint/bucket settings.

Diagnostics:

ls -la /path/to/secrets_dir
env | sed 's/=.*//' | grep -E 'OBJECT_STORAGE|AWS|AUDITA|NOTARIUS|SCRIPTORIUM'

Safe fix:

  • correct --secrets-dir, secret-file type/name, and operator-managed permissions;
  • provide required env vars;
  • keep secret values out of YAML.

Relevant reference: Secrets.

S3 audio prepare failure

Symptom:

  • prepare fails listing/downloading session S3 audio.

Likely causes:

  • incorrect session.inputs.audio_s3.prefix;
  • no matching .flac objects;
  • storage connectivity or permissions failure.

Diagnostics:

narratio session validate 2026-04-04

Safe fix:

  • verify prefix contents and storage access;
  • keep session audio mode consistent.

Relevant reference: Operations.

References