notarius/docs/operations.md

# Operations

This is the canonical guide to operating Notarius filesystem state. Command
syntax is in the [CLI reference](cli.md); field definitions and precedence are
in [Configuration](config.md).

## State Model

Notarius uses three independent filesystem surfaces:

- output is durable user data;
- cache is reconstructible chunk-plan and checkpoint state; and
- debug is explicitly requested inspection data.

Choose separate roots and access controls for each surface. A normal run writes
durable output, may use the chunk-plan cache, and records checkpoints when
`cache.checkpoints.enabled` is true. It does not create debug state unless its
invocation includes `--debug`.

## Output

Durable logical files are written under:

```text
<output-root>/<run-id>/
```

The CLI generates one run ID in the form
`run-<started-at-unix-nanoseconds>-<32-lowercase-hex-characters>` and uses it
for output, manifests, and any requested debug bundle. It validates every
logical output name before exclusively creating the run directory. If that
directory already exists, the invocation fails without changing it.

Each output file is written atomically. A later file-write failure leaves the
newly allocated partial run directory in place for inspection; Notarius never
automatically removes output. The
[JSON output contract](integrations/json-output.md) owns the logical file
names, schemas, and media types inside a run directory.

Remove an output run directory only after its consumer data is no longer
needed. This is data deletion, not cache cleanup.

## Ordered D&D Workflow

The maintained [NPC-grounded configuration](../examples/dnd-npc-grounded.config.yml)
contains one pipeline with two ordered steps. The first step extracts and
normalizes NPCs. Only after that lane reaches an accepted terminal result does
the second step begin; its generated NPC reference is supplied in memory to
spell extraction, combat extraction, and combat normalization.

```sh
go run ./cmd/notarius run dnd-npc-grounded \
  --config examples/dnd-npc-grounded.config.yml \
  --input examples/seriatim-minimal-transcript.json \
  --output-dir ./npc-grounded-output
```

The NPC artifact grounds canonical names and aliases, not spell or combat
evidence. Current-transcript source ranges remain the only event evidence. The
manifest records generated-reference identity and bounded producer provenance;
it does not record generated payload content, and no generated content is
exposed through a filesystem path. The same producer artifact may fan out to
compatible consumers, while a missing or rejected producer prevents the later
step from starting.

Standalone module configurations continue to support external NPC files when a
workflow intentionally crosses a process or session boundary. Those files are
validated against the consumer slot and must be protected as sensitive
campaign data. They are not part of the maintained ordered handoff workflow.

## Chunk-Plan Cache

Chunk plans are stored at:

```text
<chunk-plan-root>/<source-sha256-hex>/plan.json
```

`auto` reuses a complete valid plan or regenerates missing or invalid state.
`refresh` regenerates and atomically replaces a plan after chunk validation.
`bypass` performs no plan-cache I/O and does not resolve or create the root.
Plan selection is source-addressed and independent of checkpoint and debug
roots.

When its directory is empty in configuration, the root is
`<os.UserCacheDir>/notarius/chunk-plans`. A configured directory is the exact
root; no suffix is appended. Directories and files created by the store use
`0700` and `0600` permissions on supported Unix systems. The configured root
is a trust boundary: do not share it among mutually untrusted users.

Remove an exact digest directory or the configured root only when accepting the
cost of recomputing plans and any chunk-stage work. Cache publication is atomic;
there is no history, locking, garbage collection, or rollback facility.

For a Linux service account, provision a dedicated restrictive root such as:

```yaml
cache:
  chunk_plans:
    directory: /var/cache/notarius/chunk-plans
```

## Checkpoint Cache

Checkpoint recording is controlled by `cache.checkpoints.enabled`, which
defaults to `false`. When enabled, every run records running, succeeded, and
failed transitions and reusable validator-approved results. Successful,
rejected, and failed runs may therefore all leave checkpoint state. The
`--resume` flag additionally loads compatible completed work before executing
missing or incompatible stages. Without `--resume`, a recording-enabled run
never loads checkpoints. Using `--resume` while recording is disabled is an
error.

Checkpoints use the selected root and the existing identity hierarchy:

```text
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/...
```

The final identity digest includes stable semantic fingerprints explicitly
contributed by prepared modules and validators. Adding or changing one of
these fingerprints intentionally causes a cold cache miss; old checkpoint
directories are left in place and are never migrated or deleted automatically.

An empty configured directory selects
`<os.UserCacheDir>/notarius/checkpoints`. The root is exact when configured.
Created directories and files use `0700` and `0600` permissions on supported
Unix systems.

Checkpoint payloads can contain source text, intermediate artifacts, metadata,
warnings, and content digests. Treat them as sensitive derived application
data. Compatible files from a former checkpoint root remain reusable when
`cache.checkpoints.directory` names that exact existing root. They are not
moved, migrated, or deleted automatically. The frozen serialized identifier
`workspace_schema_version` remains part of checkpoint compatibility; it is not
a configuration setting.

For a Linux service account, independently provision:

```yaml
cache:
  checkpoints:
    enabled: true
    directory: /var/cache/notarius/checkpoints
```

Remove an exact checkpoint identity directory or the configured root only when
recomputation is acceptable.

### Resume And Selective Recompute

`--resume` loads compatible accepted work only when checkpoint recording is
enabled. A normal resumed run may reuse source, extract, merge, and normalize
checkpoints independently and may recompute a stage after a cache miss.
Generated references add a dependency fingerprint
for the producer's artifact kind, schema identity, media type, canonical
content digest, and size. If that fingerprint changes or the producer is
missing, dependent checkpoints are invalidated; unrelated work remains eligible
for reuse.

`--recompute-step <step-id>` requires both `--resume` and
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
dependents to execute, while compatible predecessors and unrelated lanes remain
reusable. The ID may be an explicit configured step or `default` for an
implicit single-step pipeline. It cannot be combined with `--only`, and it does
not change the persistent identity of otherwise identical checkpoints.
Decisions are bounded and categorized as `reused`, `executed`,
`forced_recompute`, or `dependency_invalidated`.

For an unselected producer required by a recomputed step, Notarius loads the
accepted normalized artifact directly. Valid normalize state is sufficient even
when that producer's extract or merge checkpoint is missing or corrupt. The
normalize manifest must be successful and match workspace schema v3, the exact
current invocation identity, step, lane, and normalizer; its payload digest and
canonical codec representation must also validate. A forced producer bypasses
this lookup and executes.

If a required predecessor's accepted normalized artifact is missing, rejected,
corrupt, non-canonical, or incompatible, the run fails before the dependent
step starts. It does not fall back to rerunning that predecessor. The failure
manifest retains completed upstream outcomes and dependency context but not
generated reference content. For diagnosis, first check the producer step and
lane in the manifest, then inspect checkpoint decision categories and reason
codes. Rerun the producer explicitly rather than copying an artifact into the
checkpoint root.

The decision that caused a required-predecessor failure is retained before the
run returns, and the CLI error identifies its step, lane, and reason code.

Checkpoint reason codes are stable diagnostic identifiers:

| Reason code | Operator meaning |
| --- | --- |
| `loading_disabled` | This invocation did not enable checkpoint loading. |
| `checkpoint_missing` | The requested checkpoint file does not exist. |
| `checkpoint_path_invalid` | The requested checkpoint location failed confinement validation. |
| `checkpoint_read_failed` | An existing checkpoint could not be read. |
| `checkpoint_decode_failed` | Checkpoint JSON could not be decoded. |
| `workspace_schema_incompatible` | The stored workspace schema is not supported by this build. |
| `identity_mismatch` | The stored invocation identity differs from the current invocation. |
| `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch` | Stored scope does not match the requested pipeline scope. |
| `status_not_reusable` | The stored operation did not finish in a reusable status. |
| `dependency_mismatch` | Stored dependencies differ; the category is `dependency_invalidated`. |
| `artifact_payload_invalid` | Stored artifact payload structure or encoding is invalid. |
| `artifact_digest_mismatch` | Stored artifact bytes do not match their recorded digest. |
| `artifact_codec_incompatible` | Stored artifact identity is incomplete or incompatible with the codec contract. |
| `artifact_not_canonical` | The codec can decode the artifact, but its bytes are not canonical. |
| `checkpoint_reused` | The stored checkpoint passed validation and was reused. |
| `accepted_artifact_reused` | A required producer's accepted normalized artifact was canonically validated and hydrated. |
| `recompute_step` | Selective recomputation forced execution of this lane. |

Decision detail is bounded explanatory text derived from the stable reason code,
not caller-supplied prose or a data-recovery channel. It never contains
checkpoint paths, artifact or reference content, source content, credentials,
or environment values.

## Debug Bundles

Only `notarius run --debug` enables debug collection. The selected root contains
one retained bundle per invocation:

```text
<debug-root>/<run-id>/
  summary/
  trace/
```

`summary/` contains redacted invocation, effective-configuration, resolved
pipeline and reference provenance, checkpoint and chunk-plan decisions, run
manifest, warnings, report, and any available error text. It excludes raw
source, references, annotations, prompts, model responses, credentials, and
malformed cache bytes.

`trace/` contains application-owned execution detail, including source and
stage material, plans, chunks, validator attempts, prompts, model responses,
timing, and serialized artifacts. It may retain application data omitted from
output. Credentials, credential-shaped values, sensitive metadata, unrelated
environment values, and unrelated filesystem content are not captured.

Bundles inherit the sensitivity of the application data they capture. Their
additional risk comes from copying and aggregating that data, so restrict
access, avoid shared roots between untrusted users, and define retention outside
Notarius. Created bundle directories use `0700` and files use `0600` on
supported Unix systems.

Notarius never automatically deletes a requested bundle. If allocation
succeeds, its path is reported on success and failure. A requested summary or
trace write failure makes the command fail, preserving whatever bundle data was
already written for inspection. Every allocated bundle makes one best-effort
attempt to record a terminal `run-report.json`.

## Failures And Warnings

Failures before debug allocation are reported on stderr without a bundle.
Failures after allocation report the bundle path on stderr and make independent
attempts to write a failure `run-report.json` and `error.log`. The report retains
the paths and pipeline outcome fields known at the failure point. If either
terminal write fails, the original command error remains first on stderr,
followed by the persistence error and bundle path. An output-write failure
leaves the allocated bundle in place. A successful run with warnings exits `0`,
reports a warning count on stderr, and records warnings in durable output and
any requested debug summary.

## Cleanup

Use exact paths for manual cleanup. Examples:

```sh
rm -rf ./notarius-output/run-1721300000000000000-0123456789abcdef0123456789abcdef
rm -rf /var/cache/notarius/chunk-plans/0123abcd
rm -rf /var/cache/notarius/checkpoints/pipeline/input-0123/pipeline-4567/identity-89ab
rm -rf ./notarius-debug/run-1721300000000000000-0123456789abcdef0123456789abcdef
```

Avoid broad recursive cleanup against a parent root unless it is an explicit
operator policy. Output deletion is permanent user-data loss. Cache deletion is
recoverable but can repeat expensive work. Debug deletion removes troubleshooting
evidence and any retained application-data copy.

## Operational Limits

Provider retries and timeouts are handled by Scriptorium according to the
selected execution profile. Pipeline module retry settings are defined in
[Configuration](config.md#module-bindings). Extract worker concurrency and
actual provider-call concurrency are separate limits; their fields and
validation are defined in [Configuration](config.md#concurrency). Notarius
writes local files only; remote storage and archive management are outside the
implemented CLI.