Documentation update for the restore subcommand

This commit is contained in:
2026-05-19 22:32:55 -05:00
parent 648001a8fe
commit 5bc8e8683f
7 changed files with 333 additions and 30 deletions

View File

@@ -6,7 +6,7 @@
narratio run --session-id 2026-04-04 narratio run --session-id 2026-04-04
``` ```
This command uses default config discovery for `pipeline.yml` and `session.yml`; both files must be discoverable unless you pass explicit `--config` and `--session` paths. This command uses default discovery for `pipeline.yml` and `session.yml`; both files must be discoverable unless you pass explicit `--config` and `--session` paths.
## Command Overview ## Command Overview
@@ -17,6 +17,7 @@ Implemented commands:
- `resume`: continue from first non-succeeded stage unless forced. - `resume`: continue from first non-succeeded stage unless forced.
- `status`: read and print stage statuses from an existing manifest. - `status`: read and print stage statuses from an existing manifest.
- `run-stage`: execute exactly one stage. - `run-stage`: execute exactly one stage.
- `restore`: restore durable local session state from the committed remote archive state.
Unknown commands print usage and exit non-zero. Unknown commands print usage and exit non-zero.
@@ -68,6 +69,15 @@ Valid stage names:
- `archive` - `archive`
- `notify` - `notify`
### `restore`
- `--config <path>`
- `--session <path>`
- `--session-id <value>`
- `--dry-run`: plan restore actions without writing local files.
- `--force`: overwrite local conflicting files with remote archive files.
- `--include-audio`: include durable archived `audio/**` files in restore scope.
### `status` ### `status`
- `--manifest <path>`: required manifest path. - `--manifest <path>`: required manifest path.
@@ -178,6 +188,39 @@ Common failure cases:
- unknown stage name. - unknown stage name.
- using `--artifacts` with any non-`analyze` stage. - using `--artifacts` with any non-`analyze` stage.
### `restore`
Purpose:
- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, and optional `audio/**`) from the committed remote archive current state.
Syntax:
```bash
narratio restore [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--dry-run] [--force] [--include-audio]
```
Success output (dry-run):
- `Restore plan for <campaign>/<session_id>`
- `Remote run: <run_id>`
- `Would download: <n>`
- `Would skip unchanged: <n>`
- `Conflicts: <n>`
Success output (non-dry-run):
- `Restored session archive for <campaign>/<session_id>`
- `Remote run: <run_id>`
- `Downloaded: <n>`
- `Skipped unchanged: <n>`
- `Conflicts: <n>`
Common failure cases:
- storage backend is not configured.
- remote `current/run_id.txt` missing/empty.
- remote `current/manifest.json` missing or invalid.
- remote manifest session/campaign mismatch.
- local conflicts without `--force`.
- session lock conflict.
## Common Workflows ## Common Workflows
Default-discovery run: Default-discovery run:
@@ -204,6 +247,19 @@ Run only analyze stage with selected artifacts:
narratio run-stage --session-id 2026-04-04 --artifacts player_handout analyze narratio run-stage --session-id 2026-04-04 --artifacts player_handout analyze
``` ```
Preview restore actions without writes:
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
Restore and then force analyze:
```bash
narratio restore --session-id 2026-04-04
narratio run-stage --session-id 2026-04-04 --force analyze
```
## Diagnostic / Recovery Commands ## Diagnostic / Recovery Commands
Inspect stage status: Inspect stage status:
@@ -219,4 +275,4 @@ Get manifest path from previous output:
- `--artifacts` filters which configured artifacts are executable when analyze runs. - `--artifacts` filters which configured artifacts are executable when analyze runs.
- `--artifacts` does not imply `--force`. - `--artifacts` does not imply `--force`.
- If analyze is already `succeeded` and `--force` is not set, runner-level skip still applies. - if analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.

View File

@@ -13,6 +13,7 @@ These commands load and validate both files before running:
- `narratio plan` - `narratio plan`
- `narratio resume` - `narratio resume`
- `narratio run-stage` - `narratio run-stage`
- `narratio restore`
Behavior: Behavior:
@@ -23,28 +24,28 @@ Behavior:
## 2. Config file discovery ## 2. Config file discovery
Pipeline config lookup for `run`, `plan`, `resume`, and `run-stage`: Pipeline config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
- If `--config <path>` is provided, that path is used. - if `--config <path>` is provided, that path is used.
- If omitted, Narratio searches in order: - if omitted, Narratio searches in order:
1. `/usr/local/etc/narratio/pipeline.yml` 1. `/usr/local/etc/narratio/pipeline.yml`
2. `/etc/narratio/pipeline.yml` 2. `/etc/narratio/pipeline.yml`
- First existing file wins. - first existing file wins.
## 3. Session file discovery and templating ## 3. Session file discovery and templating
Session config lookup for `run`, `plan`, `resume`, and `run-stage`: Session config lookup for `run`, `plan`, `resume`, `run-stage`, and `restore`:
- If `--session <path>` is provided, that path is used. - if `--session <path>` is provided, that path is used.
- If omitted, Narratio searches in order: - if omitted, Narratio searches in order:
1. `./session.yml` 1. `./session.yml`
2. `/usr/local/etc/narratio/session.yml` 2. `/usr/local/etc/narratio/session.yml`
3. `/etc/narratio/session.yml` 3. `/etc/narratio/session.yml`
- First existing file wins. - first existing file wins.
Template behavior: Template behavior:
- Supported placeholders: - supported placeholders:
- `{{session_id}}` - `{{session_id}}`
- `{{ session_id }}` - `{{ session_id }}`
- `--session-id <value>` supplies the placeholder value. - `--session-id <value>` supplies the placeholder value.
@@ -133,6 +134,7 @@ Operational notes:
- archive promotion is explicit and source-based via `archive.promote_artifacts`. - archive promotion is explicit and source-based via `archive.promote_artifacts`.
- `source` is required; `dest` is optional and derived when omitted. - `source` is required; `dest` is optional and derived when omitted.
- Narratio does not auto-promote all generated analyze artifacts. - Narratio does not auto-promote all generated analyze artifacts.
- `restore` reads the same config/session inputs and restore scope is bounded by committed archive current state.
## 7. Full pipeline reference ## 7. Full pipeline reference
@@ -267,6 +269,11 @@ Archive promotion destination rules:
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`; - configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
- derivation failure is a config validation error. - derivation failure is a config validation error.
Restore-related implications:
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
- restore scope considers only committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, optional `audio/**`).
## 8. Full session reference ## 8. Full session reference
| Path | Type | Required | Default | | Path | Type | Required | Default |

View File

@@ -4,7 +4,7 @@
Developers and LLM coding agents changing Narratio internals. Developers and LLM coding agents changing Narratio internals.
## Scope ## Scope
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, and adapter boundaries. Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, adapter boundaries, and restore command behavior.
## Component Docs ## Component Docs
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership. - `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
@@ -12,6 +12,7 @@ Implementation-accurate contracts for workspace/state, manifests, stages, artifa
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics. - `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior. - `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants. - `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
- `command-restore.md`: restore command discovery/planning/execution/reporting contract.
- `stage-prepare.md`: input materialization and provenance capture. - `stage-prepare.md`: input materialization and provenance capture.
- `stage-transcribe.md`: WhisperX transcript generation. - `stage-transcribe.md`: WhisperX transcript generation.
- `stage-merge.md`: Seriatim normalization + merge. - `stage-merge.md`: Seriatim normalization + merge.

View File

@@ -0,0 +1,86 @@
# Internal: Command Restore
## Purpose
Define the implemented `narratio restore` command contract: committed remote-state discovery, deterministic planning, safe file installation, conflict policy, and restore reporting.
## Inputs and outputs
Inputs:
- CLI flags: `--config`, `--session`, `--session-id`, `--dry-run`, `--force`, `--include-audio`.
- Resolved/validated `pipeline.yml` and `session.yml`.
- Configured remote object store.
- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
Outputs:
- Dry-run summary to stdout (plan + counts).
- Non-dry-run completion summary to stdout.
- Local durable session files restored under canonical session root.
- Non-dry-run restore report at `reports/restore-latest.json`.
## Boundaries
Owns:
- Restore command flag parsing and command wiring.
- Remote current-state discovery and identity validation.
- Restore plan construction and conflict classification.
- Restore execution for planned downloads.
- Restore report model and persistence.
Does not own:
- Stage execution orchestration (`run`, `resume`, `run-stage`).
- Archive publish behavior (owned by archive stage).
- Storage transport implementation details (owned by storage adapters).
## Config fields used
- Config/session discovery and templating fields consumed by all commands.
- `pipeline.workspace.root` (local restore target root).
- `pipeline.storage.*` (remote backend + archive identity derivation).
- `pipeline.storage.s3.*` identity components used by archive prefix helpers.
- `session.session_id`
- `session.campaign`
## External adapters used
- `storage.ObjectStore` for `Exists`, `List`, `Download`.
- `artifacts.Store` (`LocalStore`) for layout and session lock management.
- `manifest.LocalStore` for manifest decode/validation and identity checks.
## State and manifest behavior
- Restore is not a pipeline run and does not create a run manifest.
- Restore uses committed remote current state only:
- `current/run_id.txt` must exist and be non-empty.
- `current/manifest.json` must decode and match requested session/campaign.
- Non-dry-run writes restore files to canonical session paths.
- Manifest install behavior:
- validated before replacement.
- installed last among download actions.
- existing local manifest is preserved if restored manifest validation/install fails.
- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
## Skip and resume behavior
- Restore does not participate in stage skip/resume decisions.
- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
- Dry-run is read-only and returns plan output only.
## Failure behavior
- Fails when storage backend is unavailable or archive identity cannot be resolved.
- Fails when remote current pointer/manifest is missing or invalid.
- Fails when remote manifest identity mismatches requested campaign/session.
- Fails on local conflicts unless `--force` is set.
- Fails fast on session lock acquisition conflict for non-dry-run execution.
- On execution failure, previously installed files remain; no rollback is performed.
## Tests to inspect before changing
- `internal/app/restore_test.go`
- `internal/app/restore_discovery_test.go`
- `internal/app/restore_plan_test.go`
- `internal/app/restore_execution_test.go`
- `internal/app/restore_workflow_test.go`
- `internal/artifacts/archive_identity_test.go`
## Architectural invariants
- Restore relies on centralized archive identity/key helpers (`internal/artifacts`) rather than ad hoc key building.
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
- Local path mapping is traversal-safe and constrained to session root.
- Restore scope is deterministic and path-classified:
- include `manifest.json`, `transcripts/**`, `artifacts/**`
- include `audio/**` only with `--include-audio`
- exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
- Command remains standalone; no implicit `run --restore` behavior.

View File

@@ -6,7 +6,7 @@ For field-level configuration, see [docs/config.md](./config.md). For full comma
## Normal workflow (S3-first path) ## Normal workflow (S3-first path)
1. Upload session `.flac` files to object storage under the session audio prefix. 1. Upload session `.flac` files to object storage under the configured session audio prefix.
2. Run Narratio: 2. Run Narratio:
```bash ```bash
@@ -21,6 +21,37 @@ Notes:
- default config/session discovery applies unless `--config` and `--session` are passed. - default config/session discovery applies unless `--config` and `--session` are passed.
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access. - S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
## Restore workflow
Use restore when local durable session state is missing or stale and archive current state is authoritative.
Dry-run (no local writes):
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
Execution:
```bash
narratio restore --session-id 2026-04-04
```
Post-restore analyze rerun pattern:
```bash
narratio run-stage --session-id 2026-04-04 --force analyze
```
Restore source-of-truth:
- remote commit marker: `current/run_id.txt`
- remote current manifest: `current/manifest.json`
Restore default scope:
- includes `manifest.json`, `transcripts/**`, `artifacts/**`
- includes `audio/**` only with `--include-audio`
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
## Local filesystem layout and state artifacts ## Local filesystem layout and state artifacts
Session root: Session root:
@@ -29,7 +60,7 @@ Session root:
Primary state: Primary state:
- `manifest.json`: session-level stage state. - `manifest.json`: session-level stage state.
- `runs/{run_id}/manifest.json`: invocation-level state. - `runs/{run_id}/manifest.json`: invocation-level state.
- `.lock`: session lock while a run is active. - `.lock`: session lock while a modifying command is active.
Canonical session directories: Canonical session directories:
- `inputs/` - `inputs/`
@@ -48,6 +79,7 @@ Run-local stage directories:
Behavior: Behavior:
- directory creation is idempotent. - directory creation is idempotent.
- stage outputs are generally generated run-local first, then promoted to canonical paths on success. - stage outputs are generally generated run-local first, then promoted to canonical paths on success.
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.
## Analyze artifact execution lifecycle ## Analyze artifact execution lifecycle
@@ -91,7 +123,7 @@ Archive promotion is explicit and source-based:
- missing optional promotion sources are skipped. - missing optional promotion sources are skipped.
- invalid resolved artifacts fail archive stage. - invalid resolved artifacts fail archive stage.
## Resume, retry, and safe rerun behavior ## Resume, retry, restore, and safe rerun behavior
Default skip: Default skip:
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set. - `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
@@ -100,6 +132,11 @@ Resume:
- `resume` starts at first non-succeeded stage. - `resume` starts at first non-succeeded stage.
- `resume --force` runs full stage order. - `resume --force` runs full stage order.
Restore conflict policy:
- restore classifies local differences as conflicts.
- without `--force`, restore fails when conflicts exist.
- with `--force`, conflicting local files are overwritten by remote archive files.
Forced reruns: Forced reruns:
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`. - force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
@@ -125,13 +162,18 @@ No cleanup for failed/incomplete/unarchived/archive-skipped runs.
## Failure and recovery playbooks ## Failure and recovery playbooks
After failure, Narratio keeps: After run failure, Narratio keeps:
- session manifest - session manifest
- run manifest - run manifest
- run-local artifacts/logs/config/reports - run-local artifacts/logs/config/reports
Failed or incomplete runs remain local-only. Failed or incomplete runs remain local-only.
After restore failure:
- already-installed restore files remain in place.
- restore does not roll back prior successful installs.
- existing local manifest is preserved if restored manifest validation/install fails.
Recommended recovery: Recommended recovery:
1. inspect state: 1. inspect state:
@@ -140,8 +182,27 @@ Recommended recovery:
narratio status --manifest <manifest-path> narratio status --manifest <manifest-path>
``` ```
2. fix root cause (config/input/credentials/service availability). 2. for restore-specific checks, run:
3. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
```bash
narratio restore --session-id 2026-04-04 --dry-run
```
3. fix root cause (config/input/credentials/storage/service availability).
4. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
## Restore report
Non-dry-run restore writes a durable report at:
- `reports/restore-latest.json`
Report content includes:
- identity (`campaign`, `session_id`, `run_id`)
- mode flags (`dry_run`, `force`, `include_audio`)
- plan counts and execution counts
- per-action status
Dry-run does not write restore report files.
## Operational caveats ## Operational caveats
@@ -149,3 +210,4 @@ narratio status --manifest <manifest-path>
- local and S3 audio input modes are mutually exclusive. - local and S3 audio input modes are mutually exclusive.
- archive publish requires upstream stages through `analyze` to be `succeeded`. - archive publish requires upstream stages through `analyze` to be `succeeded`.
- required promotion rules can fail when selected analyze artifacts did not generate a required file path. - required promotion rules can fail when selected analyze artifacts did not generate a required file path.
- restore requires configured remote object storage and committed remote current state.

View File

@@ -2,7 +2,7 @@
## Status ## Status
Planned. This document is implementation guidance for 5.3-Codex. Implemented through Step 8. This document remains as roadmap and design history for the restore feature, and as the home for future restore-related ideas (for example `run --restore`).
## Summary ## Summary
@@ -546,7 +546,7 @@ Do not document future `narratio run --restore` behavior outside `docs/roadmap/`
## Implementation phases ## Implementation phases
### Phase 1: Audit existing archive and path helpers ### Phase 1: Audit existing archive and path helpers (completed)
Before coding behavior, inspect: Before coding behavior, inspect:
@@ -574,7 +574,7 @@ Deliverable:
- small code comments or internal helper selection; - small code comments or internal helper selection;
- no large behavior change yet unless required by tests. - no large behavior change yet unless required by tests.
### Phase 2: Add CLI surface and command wiring ### Phase 2: Add CLI surface and command wiring (completed)
Add `narratio restore` command parsing. Add `narratio restore` command parsing.
@@ -599,7 +599,7 @@ Deliverable:
- command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired; - command returns a clear “not yet implemented” or calls an empty planner if phased commits are desired;
- CLI tests pass. - CLI tests pass.
### Phase 3: Implement remote current-state discovery ### Phase 3: Implement remote current-state discovery (completed)
Add restore code that: Add restore code that:
@@ -615,7 +615,7 @@ Deliverable:
- fake-storage tests for current-state discovery; - fake-storage tests for current-state discovery;
- no local file writes beyond temporary files. - no local file writes beyond temporary files.
### Phase 4: Implement restore planning ### Phase 4: Implement restore planning (completed)
Build deterministic restore plans for default scope and `--include-audio`. Build deterministic restore plans for default scope and `--include-audio`.
@@ -627,7 +627,7 @@ Deliverable:
- dry-run output works; - dry-run output works;
- no real file overwrite yet except temp comparisons as needed. - no real file overwrite yet except temp comparisons as needed.
### Phase 5: Implement restore execution ### Phase 5: Implement restore execution (completed)
Execute the plan safely: Execute the plan safely:
@@ -644,7 +644,7 @@ Deliverable:
- restore works end-to-end against fake storage; - restore works end-to-end against fake storage;
- failures are clear and do not corrupt existing local manifest. - failures are clear and do not corrupt existing local manifest.
### Phase 6: Add restore report and operator summary ### Phase 6: Add restore report and operator summary (completed)
Add concise stdout summary and optional JSON restore report if consistent with project diagnostics. Add concise stdout summary and optional JSON restore report if consistent with project diagnostics.
@@ -654,7 +654,7 @@ Deliverable:
- durable diagnostic report if implemented; - durable diagnostic report if implemented;
- no content leakage. - no content leakage.
### Phase 7: Workflow integration test ### Phase 7: Workflow integration test (completed)
Add a test for restoring a previous session and then forcing `analyze`. Add a test for restoring a previous session and then forcing `analyze`.
@@ -664,7 +664,7 @@ Deliverable:
- no upstream stages rerun; - no upstream stages rerun;
- no reliance on live subprocesses or S3. - no reliance on live subprocesses or S3.
### Phase 8: Documentation update ### Phase 8: Documentation update (completed)
Once implemented, update current-behavior docs and internal command docs. Once implemented, update current-behavior docs and internal command docs.

View File

@@ -6,7 +6,7 @@ Canonical operator troubleshooting guide for recurring implemented Narratio fail
## Config file discovery failure ## Config file discovery failure
Symptom: Symptom:
- `run`, `plan`, `resume`, or `run-stage` fails with config/session not found. - `run`, `plan`, `resume`, `run-stage`, or `restore` fails with config/session not found.
Likely Cause: Likely Cause:
- `pipeline.yml` or `session.yml` is missing from discovery paths. - `pipeline.yml` or `session.yml` is missing from discovery paths.
@@ -192,7 +192,7 @@ Links:
## Session lock conflict (`.lock`) ## Session lock conflict (`.lock`)
Symptom: Symptom:
- run fails with lock conflict for session workdir. - `run`, `resume`, `run-stage`, or `restore` fails with lock conflict for session workdir.
Likely Cause: Likely Cause:
- another Narratio process is running same session. - another Narratio process is running same session.
@@ -207,13 +207,104 @@ ps aux | grep narratio
``` ```
Safe Fix: Safe Fix:
- wait for active run to finish. - wait for active process to finish.
- if no process is active, remove only stale session `.lock` file. - if no process is active, remove only stale session `.lock` file.
Links: Links:
- [docs/operations.md](./operations.md) - [docs/operations.md](./operations.md)
- [docs/internal/workspace.md](./internal/workspace.md) - [docs/internal/workspace.md](./internal/workspace.md)
## Restore remote current pointer or manifest missing
Symptom:
- `restore` fails with remote current pointer or current manifest errors.
Likely Cause:
- `current/run_id.txt` was never published.
- `current/manifest.json` is missing for the session prefix.
- archive commit did not complete.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- verify archive stage succeeded for the target session.
- rerun/archive from a healthy source workspace so current pointers are published.
Links:
- [docs/operations.md](./operations.md)
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
## Restore manifest identity mismatch
Symptom:
- `restore` fails because remote manifest session or campaign does not match requested values.
Likely Cause:
- wrong `--session-id` or wrong session config selected.
- archive prefix points to a different campaign/session.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- use the correct session config and `--session-id`.
- verify campaign/session identity in local config before restore.
Links:
- [docs/config.md](./config.md)
- [docs/operations.md](./operations.md)
## Restore conflict without `--force`
Symptom:
- `restore` fails with `restore conflict` and conflict counts.
Likely Cause:
- local durable file differs from remote file for one or more planned restore paths.
Diagnostics:
```bash
narratio restore --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
```
Safe Fix:
- review planned conflicts.
- rerun with `--force` only when remote state should overwrite local state.
Links:
- [docs/cli.md](./cli.md)
- [docs/operations.md](./operations.md)
## Restore report expectations
Symptom:
- operator expects restore report file but does not find one.
Likely Cause:
- restore was executed in `--dry-run` mode.
- restore failed before report persistence path (for example lock acquisition failure).
Diagnostics:
```bash
ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json
```
Safe Fix:
- run non-dry-run restore for durable report output.
- resolve lock or early preflight failures and retry.
Links:
- [docs/operations.md](./operations.md)
## Secrets env-dir or credential-env failure ## Secrets env-dir or credential-env failure
Symptom: Symptom: