207 lines
10 KiB
Markdown
207 lines
10 KiB
Markdown
# Operations
|
|
|
|
This is the canonical guide for operating Notarius runtime state. The
|
|
[CLI reference](cli.md) owns command syntax and exit statuses, while
|
|
[Configuration](config.md) owns fields, defaults, and precedence. Maintainers
|
|
who need implementation mechanics should read [Run State Internals](internal/state.md).
|
|
|
|
## State Surfaces
|
|
|
|
Each run can use independent roots with different retention and access-control
|
|
needs.
|
|
|
|
| Surface | Purpose | Created when | Retention |
|
|
| --- | --- | --- | --- |
|
|
| Output | Durable user-facing result bundle | A pipeline completes and returns logical output files | Keep until consumers no longer need it. |
|
|
| Chunk-plan cache | Reconstructible source-addressed plan | The configured cache mode permits cache I/O | Keep while reuse is useful. |
|
|
| Checkpoint cache | Reconstructible execution and recovery state | Checkpoint recording is enabled | Keep only while recovery or reuse is useful. |
|
|
| Debug bundle | Explicit diagnostic record | A run requests debug collection | Keep only under an intentional sensitive-data retention policy. |
|
|
|
|
Output, cache, and debug roots are never merged or cleaned automatically. Use
|
|
separate locations and permissions for operators or services that must not
|
|
share application data.
|
|
|
|
## Roots And Permissions
|
|
|
|
The configured output and debug directories are exact roots. An empty cache
|
|
directory selects a per-user root:
|
|
|
|
~~~
|
|
<os.UserCacheDir>/notarius/chunk-plans
|
|
<os.UserCacheDir>/notarius/checkpoints
|
|
~~~
|
|
|
|
The field definitions and configuration examples are in [Configuration](config.md).
|
|
On supported Unix systems, output directories and files are created with
|
|
requested modes **0755** and **0644**. Chunk-plan, checkpoint, and debug
|
|
directories and files use **0700** and **0600**. The operating system's umask
|
|
may impose stricter output modes. Cache and debug roots may contain sensitive
|
|
source-derived data, so provision them for one trusted account or service.
|
|
|
|
## Run Lifecycle
|
|
|
|
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
|
and resolves configuration before module preparation and source parsing. It
|
|
then performs any permitted cache lookup, executes the pipeline, and publishes
|
|
logical output files only after a successful runner result.
|
|
|
|
On success, the command reports the output bundle path. A warning-bearing run
|
|
still succeeds and reports its warning count on standard error. Errors and
|
|
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
|
|
|
|
## Output Bundles
|
|
|
|
Each successful run receives a generated safe run identifier and writes beneath:
|
|
|
|
~~~
|
|
<output-root>/<run-id>/
|
|
~~~
|
|
|
|
The [JSON output contract](integrations/json-output.md) owns the logical files
|
|
and their schemas. Before creating the run directory, Notarius validates every
|
|
logical output path. It refuses an existing run directory without changing it.
|
|
Files are written atomically; if a later write fails, the newly created partial
|
|
run directory remains for inspection and is never removed automatically.
|
|
|
|
Treat an output bundle as durable user data. Do not use cache-cleanup policy to
|
|
remove it. An optional accepted chunk map is also durable output and can carry
|
|
source- or model-derived annotations; its content and compatibility contract
|
|
are defined in [Accepted Chunk Map](integrations/chunk-map.md).
|
|
|
|
## Chunk-Plan Cache
|
|
|
|
Chunk plans live beneath the selected chunk-plan root:
|
|
|
|
~~~
|
|
<chunk-plan-root>/<source-sha256-hex>/plan.json
|
|
~~~
|
|
|
|
One validated canonical plan is active for each source digest. The plan stores
|
|
boundaries and provenance, not a second copy of the entire source. This
|
|
source-addressed policy is recorded in [ADR-0005](adr/0005-cache-canonical-chunk-plans-by-source.md).
|
|
|
|
The configured cache mode controls one invocation:
|
|
|
|
- **auto** looks for a valid active plan. Missing or invalid state causes a new
|
|
plan to be generated; an accepted new plan is atomically published.
|
|
- **refresh** skips lookup, generates a plan with the configured chunker, and
|
|
atomically replaces the active plan after it is accepted.
|
|
- **bypass** performs no chunk-plan cache I/O. It does not resolve or create a
|
|
chunk-plan root.
|
|
|
|
A reused plan is still materialized and validated against the current source.
|
|
If a prior plan no longer gives acceptable results, use a refresh run rather
|
|
than editing cache files. Deleting a plan is recoverable but can repeat costly
|
|
chunking work.
|
|
|
|
## Checkpoint Recording, Resume, And Recompute
|
|
|
|
Checkpoint recording is an explicit configuration choice and is disabled by
|
|
default. When enabled, each run records stage transitions and the state needed
|
|
for compatible recovery. A run records checkpoints even when it does not ask
|
|
to reuse them. Checkpoint payloads can contain source-derived and intermediate
|
|
application data, so treat the entire root as sensitive.
|
|
|
|
Checkpoint loading is separate: [**--resume**](cli.md#run) asks a run to reuse
|
|
compatible recorded work. A resume request fails when checkpoint recording is
|
|
disabled. Without **--resume**, a recording-enabled run executes normally and
|
|
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
|
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
|
provenance, and prepared-component fingerprints. A changed identity produces a
|
|
cold miss; Notarius does not migrate, rewrite, or delete older checkpoint
|
|
directories.
|
|
|
|
Checkpoint state is confined below an identity-specific path:
|
|
|
|
~~~
|
|
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest-prefix>/<pipeline-digest-prefix>/<identity-digest-prefix>/
|
|
~~~
|
|
|
|
### Selective Recompute
|
|
|
|
[**--recompute-step**](cli.md#run) requires both **--resume** and enabled
|
|
checkpoint recording. It forces the selected ordered step and every lane that
|
|
depends on it through generated artifact references. Unrelated lanes remain
|
|
eligible for reuse.
|
|
|
|
For an earlier producer required by a forced consumer, Notarius requires a
|
|
compatible accepted normalized artifact. It validates that artifact before
|
|
hydrating it and does not silently rerun the producer. If that state is
|
|
missing, rejected, corrupt, non-canonical, or incompatible, the run stops
|
|
before its dependent starts. Rerun the required producer deliberately instead
|
|
of copying or editing checkpoint files.
|
|
|
|
## Checkpoint Decisions And Recovery
|
|
|
|
Checkpoint events classify work as **executed**, **reused**,
|
|
**forced_recompute**, or **dependency_invalidated**. Their stable reason codes
|
|
are written to run diagnostics and provenance. Use the code, not a copied
|
|
error message, to decide what to repair.
|
|
|
|
| Reason code | Recovery meaning |
|
|
| --- | --- |
|
|
| **loading_disabled** | This invocation did not permit checkpoint loading. |
|
|
| **checkpoint_missing**, **checkpoint_path_invalid**, **checkpoint_read_failed**, **checkpoint_decode_failed** | The stored checkpoint could not be located or read safely; normal resume work can execute again. |
|
|
| **workspace_schema_incompatible**, **identity_mismatch**, **stage_mismatch**, **step_mismatch**, **lane_mismatch**, **module_mismatch** | Stored state belongs to a different compatible scope or identity; allow a fresh run to create new state. |
|
|
| **status_not_reusable** | The recorded operation did not end in reusable state. |
|
|
| **dependency_mismatch** | A dependency changed; dependent work is invalidated rather than reused. |
|
|
| **artifact_payload_invalid**, **artifact_digest_mismatch**, **artifact_codec_incompatible**, **artifact_not_canonical** | A stored artifact cannot safely be hydrated; rerun the producer instead of modifying the cache. |
|
|
| **checkpoint_reused** | A normal checkpoint passed compatibility checks. |
|
|
| **accepted_artifact_reused** | A required predecessor's accepted normalized artifact was safely hydrated. |
|
|
| **recompute_step** | Selective recomputation deliberately forced this work. |
|
|
|
|
Reason detail is bounded code-owned text. It is diagnostic information, not a
|
|
path-discovery or data-recovery mechanism, and does not contain checkpoint,
|
|
source, reference, credential, or environment content.
|
|
|
|
## Debug Bundles
|
|
|
|
Only a [debug-enabled run](cli.md#run) creates a bundle:
|
|
|
|
~~~
|
|
<debug-root>/<run-id>/
|
|
summary/
|
|
trace/
|
|
~~~
|
|
|
|
The summary contains redacted invocation and resolution information plus run,
|
|
warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
|
|
contains allowlisted application diagnostic records and can include source or
|
|
derived application data. Neither surface is a cache input. Do not treat a
|
|
debug bundle as safe to share merely because its configuration summary is
|
|
redacted.
|
|
|
|
Notarius never creates debug state without an explicit request and never
|
|
automatically deletes a requested bundle. If allocation succeeds, the command
|
|
reports its path on both success and later failure. A summary, trace, or
|
|
terminal-report persistence failure fails the command while preserving any
|
|
already-written diagnostic data for inspection.
|
|
|
|
## Cleanup
|
|
|
|
Cleanup is manual and destructive. First inspect the exact leaf directory,
|
|
then remove only that leaf; do not use a glob or a parent root as the target.
|
|
|
|
~~~
|
|
rm -rf -- /srv/notarius/output/run-1721300000000000000-0123456789abcdef0123456789abcdef
|
|
rm -rf -- /srv/notarius/chunk-plans/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
|
|
rm -rf -- /srv/notarius/checkpoints/example/seriatim-0123456789abcdef/0123456789abcdef/0123456789abcdef
|
|
rm -rf -- /srv/notarius/debug/run-1721300000000000000-0123456789abcdef0123456789abcdef
|
|
~~~
|
|
|
|
Deleting output permanently removes user data. Deleting chunk plans or
|
|
checkpoints is recoverable but may repeat expensive provider or pipeline work.
|
|
Deleting a debug bundle removes troubleshooting evidence and a retained copy of
|
|
application data. Notarius has no cache garbage collector, rollback operation,
|
|
or automatic cleanup command.
|
|
|
|
## Operational Limits
|
|
|
|
Provider retries and timeouts are supplied by the selected Scriptorium profile.
|
|
Module retry settings and concurrency limits are configuration contracts; see
|
|
[module bindings](config.md#module-bindings-and-validators) and
|
|
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
|
limits and actual provider-call limits are independent. Notarius writes local
|
|
filesystem state only; remote storage, archival, and retention automation are
|
|
outside the implemented CLI.
|