notarius/docs/operations.md

# Operations

This is the canonical guide for operating Notarius runtime state. The
[CLI reference](cli.md) owns command syntax and exit statuses, while
[Configuration](config.md) owns fields, defaults, and precedence. Maintainers
who need implementation mechanics should read [Run State Internals](internal/state.md).

## Source Deployment

Linux is the supported deployment platform. Install a pinned source release
with the Go version declared in `go.mod` (currently Go 1.25.5):

~~~sh
GOWORK=off go install \
  gitea.maximumdirect.net/eric/notarius/cmd/notarius@vMAJOR.MINOR.PATCH
~~~

Pin the exact tag in deployment automation rather than following a branch.
Use [`notarius --version`](cli.md#command-summary) as a diagnostic after
installation; its syntax and semantics are owned by the [CLI reference](cli.md).
The maintainer publication process, including tag guards and verification,
belongs to [Source Releases](release.md). macOS builds are best-effort for
development, and Windows is unsupported.

## State Surfaces

Each run can use independent roots with different retention and access-control
needs.

| Surface | Purpose | Created when | Retention |
| --- | --- | --- | --- |
| Output | Durable user-facing result bundle | A pipeline completes and returns logical output files | Keep until consumers no longer need it. |
| Chunk-plan cache | Reconstructible source-addressed plan | The configured cache mode permits cache I/O | Keep while reuse is useful. |
| Checkpoint cache | Reconstructible execution and recovery state | Checkpoint recording is enabled | Keep only while recovery or reuse is useful. |
| Debug bundle | Explicit diagnostic record | A run requests debug collection | Keep only under an intentional sensitive-data retention policy. |

Output, cache, and debug roots are never merged or cleaned automatically. Use
separate locations and permissions for operators or services that must not
share application data.

## Roots And Permissions

The configured output and debug directories are exact roots. An empty cache
directory selects a per-user root:

~~~
<os.UserCacheDir>/notarius/chunk-plans
<os.UserCacheDir>/notarius/checkpoints
~~~

The field definitions and configuration examples are in [Configuration](config.md).
On supported Unix systems, output directories and files are created with
requested modes **0755** and **0644**; debug directories and files use
**0700** and **0600**. The operating system's umask may impose stricter output
or debug modes. Chunk-plan and checkpoint caches instead request maximum modes
of **0777** for new directories and **0666** for new files, including atomic
temporary files. The kernel applies the service's umask and applicable default
ACLs when those entries are created.

## Cache Provisioning

Cache locations select where Notarius writes, not who can access them. A
private service can use a restrictive umask:

~~~sh
umask 0077
notarius run example --config /etc/notarius/config.yml --input /srv/input.txt
~~~

A trusted-group deployment can provision a setgid cache root and use a
group-oriented umask (or an equivalent default ACL):

~~~sh
install -d -m 2770 -g notarius-operators /srv/notarius/chunk-plans /srv/notarius/checkpoints
umask 0007
notarius run example --config /etc/notarius/config.yml --input /srv/input.txt
~~~

Setgid and default ACLs control inherited group ownership and access policy;
Notarius does not copy a parent directory's literal mode bits. Cache state is
reconstructible but can contain source-derived data, so operators own its
access and retention policy. Existing cache inodes retain their prior modes
after an upgrade. Recreate a cache beneath correctly provisioned parents, or
stop writers and perform a deliberate administrative migration when a
whole-tree change is required.

Cache publication remains atomic and stored records remain validated. Those
guarantees do not add cross-process locking or transactional coordination for
simultaneous writers to one checkpoint identity.

Output bundles can also contain source content when JSON output enables
evidence publication. Apply an appropriate umask and output-root access policy
before enabling that option; the requested output modes alone may not be
suitable for transcript-bearing bundles.

## PromptKit Profile Deployment

Profile deployment has four distinct layers:

| Layer | Owner | Operational role |
| --- | --- | --- |
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |

The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
embedded fallback makes that ID usable without an operator file. Production,
development, and local deployments can each install a different complete
definition for the same ID, retaining the pipeline while choosing their own
model, backend, timeout, or reasoning policy. An operator definition wins over
the fallback; it is not merged with it. The configuration field and full
precedence rules are owned by [Configuration](config.md#promptkit-profiles).

Use a profile source owned by the service account, keep it readable only by
the intended operator, and supply provider credentials through the service
environment—not in the Notarius configuration or profile YAML. The maintained
[LLM profile](../examples/llm-profiles/dnd-extraction.yml) is secret-free and
can be copied as a format starting point. Validate a deployment without a
provider call or credentials:

~~~sh
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
~~~

An unset optional `api_key_env` reaches the provider without authorization and
may receive a 401 or 403 response.

Configured PromptKit profile paths resolve from the file that supplies their
surviving declaration. The complete example's
`./llm-profiles/dnd-extraction.yml` path is therefore relative to that example.
Use absolute paths such as `/etc/notarius/profiles/dnd-extraction.yml` for
services and containers when that is clearer for the deployment.

## Run Lifecycle

Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
and resolves configuration before module preparation and source parsing. It
then performs any permitted cache lookup, executes the pipeline, and publishes
logical output files only after a successful runner result.

On success, the command reports the output bundle path. A run with actionable
process warnings still succeeds and reports warning-group and occurrence counts
on standard error; advisory and observation findings do not produce a warning
line. Errors and
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).

## Validation Retries And Terminal Outcomes

When debug recording is enabled, per-call response artifacts include the
terminal execution summary for retry, repair, timing, and presence-aware usage
diagnosis. Normal output, receipts, and warnings do not aggregate or publish
this diagnostic metadata.

Each producer binding has one outer **retries** budget. It covers complete
producer attempts for operational failures, invalid structured output,
normalizer fallback retry, and semantic correction. It is independent from
PromptKit's structural-repair calls inside one completion, its configured
`promptkit.transient_retries` built-in HTTP attempts, and an LLM-backed
validator's own retry budget. The transient setting defaults to two additional
attempts for eligible provider statuses; set it explicitly to zero when an
operator needs one HTTP attempt per generation. A semantic correction rebuilds
the ordinary producer request and supplies only the latest rejected model
response plus aggregated validator guidance; it is not a conversation replay.

These budgets multiply only by their declared scopes. With two outer producer
retries, two transient retries, and one structural repair, a single producer
can issue up to 18 HTTP attempts: three producer attempts, each with an initial
generation and one repair generation, each allowing three HTTP attempts. This
is an illustration, not a total-deadline or exactly-once guarantee. Every
built-in HTTP attempt has its own generation timeout; caller cancellation
remains the overall authority and interrupts Retry-After waiting. The shared
Notarius scheduler keeps its permit through those waits, so retry budgets can
increase latency and provider cost. Observed usage can remain incomplete when
providers omit accounting fields.

After the applicable budgets are exhausted, the resolved
[`validation_policy`](config.md#pipelines) determines the result. Structural
failure and semantic rejection normally fail the run; an explicit
`reject_output` records a rejection and allows unrelated work to finish. A
validator execution failure normally uses `warn_continue`, which keeps an
otherwise accepted result in the current run with `incomplete` validation
provenance. It emits one bounded warning for every validator whose execution
budget was exhausted. A corrected result that later passes validation does not
retain diagnostics from abandoned attempts.

Treat a successful process exit as a completed run, not as proof that every
candidate was fully validated. Inspect the receipt's `validation_status`,
`validation_summaries`, rejection count, and warning group and occurrence
counts when an orchestrator requires complete validation. The durable fields
and their meanings are owned by the
[run-result receipt](integrations/run-result.md) and
[published JSON output contract](integrations/json-output.md).

## Output Bundles

Each successful run receives a generated safe run identifier and writes beneath:

~~~
<output-root>/<run-id>/
~~~

The [JSON output contract](integrations/json-output.md) owns the logical files
and their schemas. Before creating the run directory, Notarius validates every
logical output path. It refuses an existing run directory without changing it.
Files are written atomically; if a later write fails, the newly created partial
run directory remains for inspection and is never removed automatically.

Treat an output bundle as durable user data. Do not use cache-cleanup policy to
remove it. An optional accepted chunk map is also durable output and can carry
source- or model-derived annotations; its content and compatibility contract
are defined in [Accepted Chunk Map](integrations/chunk-map.md). An optional
[evidence context](integrations/evidence-context.md) contains source-unit text
and metadata. It is not a cache or debug artifact: retain it with the output
bundle only for as long as consumers need it, and apply source-content access
controls to the entire bundle. Its selected source-unit excerpt may include
every source unit once when coverage is broad or its configured window is
large, so do not assume a byte or token reduction or reduced sensitivity.

## Chunk-Plan Cache

Chunk plans live beneath the selected chunk-plan root:

~~~
<chunk-plan-root>/<source-sha256-hex>/plan.json
~~~

One validated canonical plan is active for each source digest. The plan stores
boundaries and provenance, not a second copy of the entire source. This
source-addressed policy is recorded in [ADR-0005](adr/0005-cache-canonical-chunk-plans-by-source.md).

The configured cache mode controls one invocation:

- **auto** looks for a valid active plan. Missing or invalid state causes a new
  plan to be generated; an accepted new plan is atomically published.
- **refresh** skips lookup, generates a plan with the configured chunker, and
  atomically replaces the active plan after it is accepted.
- **bypass** performs no chunk-plan cache I/O. It does not resolve or create a
  chunk-plan root.

A reused plan is still materialized and validated against the current source.
If a prior plan no longer gives acceptable results, use a refresh run rather
than editing cache files. Deleting a plan is recoverable but can repeat costly
chunking work. A plan accepted only under incomplete validation is not
published, and a rejected cache hit falls through to ordinary generation rather
than becoming a correction candidate.

## Checkpoint Recording, Resume, And Recompute

Checkpoint recording is an explicit configuration choice and is disabled by
default. When enabled, each run records stage transitions and the state needed
for compatible recovery. A run records checkpoints even when it does not ask
to reuse them. Checkpoint payloads can contain source-derived and intermediate
application data, so treat the entire root as sensitive.

Checkpoint loading is separate: [**--resume**](cli.md#run) asks a run to reuse
compatible recorded work. A resume request fails when checkpoint recording is
disabled. Without **--resume**, a recording-enabled run executes normally and
does not load checkpoint state. Compatibility includes the resolved pipeline,
input, selected lanes, runtime overrides, reference provenance, LLM-profile
provenance, the effective PromptKit profile-source fingerprint, and
prepared-component fingerprints. When a local PromptKit backend is configured,
compatibility also includes a non-secret fingerprint of its endpoint. Changing
profile content or the local endpoint causes a cold miss; changing only the
local concurrency limit does not. A changed identity produces a cold miss;
Notarius does not migrate, rewrite, or delete older checkpoint directories.
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
runtime identities, so checkpoints created under one state are not reused by
either of the others.

Only accepted, completely validated chunk, extract, merge, and normalize
results are checkpointed for reuse. Rejected, structurally invalid, and
validation-incomplete producer results remain non-reusable, even when a
`warn_continue` result advanced during its original run. A resumed invocation
therefore reruns that producer rather than treating degraded state as accepted.

Checkpoint state is confined below an identity-specific path:

~~~
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest-prefix>/<pipeline-digest-prefix>/<identity-digest-prefix>/
~~~

### Selective Recompute

[**--recompute-step**](cli.md#run) requires both **--resume** and enabled
checkpoint recording. It forces the selected ordered step and every lane that
depends on it through generated artifact references. Unrelated lanes remain
eligible for reuse.

For an earlier producer required by a forced consumer, Notarius requires a
compatible accepted normalized artifact. It validates that artifact before
hydrating it and does not silently rerun the producer. If that state is
missing, rejected, corrupt, non-canonical, or incompatible, the run stops
before its dependent starts. Rerun the required producer deliberately instead
of copying or editing checkpoint files.

## Checkpoint Decisions And Recovery

Checkpoint events classify work as **executed**, **reused**,
**forced_recompute**, or **dependency_invalidated**. Their stable reason codes
are written to run diagnostics and provenance. Use the code, not a copied
error message, to decide what to repair.

| Reason code | Recovery meaning |
| --- | --- |
| **loading_disabled** | This invocation did not permit checkpoint loading. |
| **checkpoint_missing**, **checkpoint_path_invalid**, **checkpoint_read_failed**, **checkpoint_decode_failed** | The stored checkpoint could not be located or read safely; normal resume work can execute again. |
| **workspace_schema_incompatible**, **identity_mismatch**, **stage_mismatch**, **step_mismatch**, **lane_mismatch**, **module_mismatch** | Stored state belongs to a different compatible scope or identity; allow a fresh run to create new state. |
| **status_not_reusable** | The recorded operation did not end in reusable state. |
| **dependency_mismatch** | A dependency changed; dependent work is invalidated rather than reused. |
| **artifact_payload_invalid**, **artifact_digest_mismatch**, **artifact_codec_incompatible**, **artifact_not_canonical** | A stored artifact cannot safely be hydrated; rerun the producer instead of modifying the cache. |
| **checkpoint_reused** | A normal checkpoint passed compatibility checks. |
| **accepted_artifact_reused** | A required predecessor's accepted normalized artifact was safely hydrated. |
| **recompute_step** | Selective recomputation deliberately forced this work. |

Reason detail is bounded code-owned text. It is diagnostic information, not a
path-discovery or data-recovery mechanism, and does not contain checkpoint,
source, reference, credential, or environment content.

## Debug Bundles

Only a [debug-enabled run](cli.md#run) creates a bundle:

~~~
<debug-root>/<run-id>/
  summary/
  trace/
~~~

The summary contains redacted invocation and resolution information, including
`effective-config.yml` and the separately redacted resolved pipeline, plus run,
final grouped diagnostic, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
terminal records contain bounded attempt kinds, validator outcomes, policy,
decision, PromptKit repair count, and usage; they do not contain assistant
responses or complete correction messages. The trace contains allowlisted
application diagnostic records and can include source, model, and correction
content. Neither surface is a cache input. Do not treat a debug bundle as safe
to share merely because its configuration summary is redacted. Invocation
metadata omits reasoning effort when it is inherited, records the replacement
value when one is supplied, and records an empty value when inherited reasoning
was explicitly cleared. It records `config_overlay` only when a non-base
configuration overlay was selected.

Notarius never creates debug state without an explicit request and never
automatically deletes a requested bundle. If allocation succeeds, the command
reports its path on both success and later failure. A summary, trace, or
terminal-report persistence failure fails the command while preserving any
already-written diagnostic data for inspection.

## Cleanup

Cleanup is manual and destructive. First inspect the exact leaf directory,
then remove only that leaf; do not use a glob or a parent root as the target.

~~~
rm -rf -- /srv/notarius/output/run-1721300000000000000-0123456789abcdef0123456789abcdef
rm -rf -- /srv/notarius/chunk-plans/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
rm -rf -- /srv/notarius/checkpoints/example/seriatim-0123456789abcdef/0123456789abcdef/0123456789abcdef
rm -rf -- /srv/notarius/debug/run-1721300000000000000-0123456789abcdef0123456789abcdef
~~~

Deleting output permanently removes user data. Deleting chunk plans or
checkpoints is recoverable but may repeat expensive provider or pipeline work.
Deleting a debug bundle removes troubleshooting evidence and a retained copy of
application data. Notarius has no cache garbage collector, rollback operation,
or automatic cleanup command.

## Operational Limits

Provider execution settings and the generation timeout come from the selected
PromptKit profile. The invocation-only **--reasoning-effort** and
**--clear-reasoning-effort** controls may replace or clear that profile setting
for all LLM-backed calls in one run without changing the profile. PromptKit
v0.11.0 also retries eligible transient HTTP failures with its built-in default
of two additional attempts per generation. This retry work is separate from
structural output repair, which happens within one structured-completion call. Its
effective `structured_output_repair_attempts` limit is resolved from the
selected binding, then the pipeline, then the prompt declaration; see
[module bindings](config.md#module-bindings-and-validators). This is distinct
from Notarius binding **retries**, which rerun the complete module operation
and validation chain and do not consume or replenish the structural-repair
limit. The maintained production prompts declare one repair attempt, paid only
after a structural failure. One structured completion with repair budget **R**
contains at most **R + 1** serial generations. Each generation can now contain
more than one provider call when PromptKit retries an eligible transient
failure. LLM-backed validators have their own corresponding invocation and
repair budgets. These are bounds on attempted work, not a promise that every
call reaches a provider.

Timeouts are layered. Caller cancellation is the outer authority. A positive
effective generation timeout adds an inner request deadline to each built-in
generation attempt, while zero disables only that generation deadline. The HTTP
client timeout remains a transport-wide cap. Notarius does not add another
timeout around PromptKit. Retries and repairs are serial within the same caller
context, so their worst-case latency and cost can exceed the structural-repair
bound above; provision run deadlines and provider budgets accordingly.
Credentials remain optional unless the selected
PromptKit profile requires one, in which case preparation fails before a
provider call when its configured credential is unavailable.
The pinned upstream boundary and profile-format links are in
[PromptKit Integration](integrations/pkg-promptkit.md).

Concurrency has two independent layers. Notarius **total_llm** defaults to 16
and is the application-wide provider-call limit shared by all backends,
modules, retries, and validators. PromptKit may impose a narrower admission
limit for the selected backend. The effective active-generation bound is the
intersection of the Notarius limit, any PromptKit backend limit, and work made
available by the pipeline. Built-in OpenRouter profiles use PromptKit's
upstream backend limit; endpoint-only profiles have no PromptKit backend limit
and remain bounded by Notarius. For the configured local backend, a zero
**concurrency_limit** leaves only the Notarius scheduler as a call limit. A
positive value makes the effective active local-generation bound the smaller
of **total_llm** and that local limit, so a local limit of four permits no more
than four active local generations.

The Notarius scheduler admits one logical structured completion and holds that
permit while PromptKit performs its serial corrective calls. PromptKit applies
its selected-backend admission to each provider call; Notarius does not
reacquire a permit or add another scheduler for a repair.

For a positive local limit, PromptKit owns its default waiting capacity and
admission behavior. When a PromptKit backend has admitted all active and queued
work, a new call fails as capacity exhaustion before generation. The adapter
maps that failure to Notarius's existing provider-neutral capacity error and
does not retry it. The calling stage's configured retry policy applies
normally, and the run fails if those attempts are exhausted. Caller
cancellation remains authoritative. Configuration contracts are documented
under [PromptKit profiles](config.md#promptkit-profiles) and
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
limits and actual provider-call limits are independent. Notarius writes local
filesystem state only; remote storage, archival, and retention automation are
outside the implemented CLI.

Every run has an effective prompt session used for provider routing and run
provenance. The generated default is stable for the same input module and raw
input bytes; use [**--session-id**](cli.md#run) only when intentionally grouping
different invocations. Both generated and explicit values can be visible to
providers, manifests, checkpoints, and requested debug bundles. Do not put
credentials or other secrets in an explicit session identifier; command-line
values are not a credential mechanism.