diff --git a/docs/internal/state.md b/docs/internal/state.md index 1dd5fd0..4e01327 100644 --- a/docs/internal/state.md +++ b/docs/internal/state.md @@ -2,7 +2,8 @@ This document describes the implementation collaborators behind output, cache, and debug state. User-visible fields belong in [Configuration](../config.md), -and layouts and lifecycle belong in [Operations](../operations.md). +and physical layout, retention, recovery, reason codes, and cleanup belong in +[Operations](../operations.md). ## Composition @@ -12,6 +13,12 @@ constructs cache collaborators, writes logical output files, and reports paths. Pipeline modules receive interfaces and request data, never output, cache, or debug roots. +The CLI creates no chunk-plan store in bypass mode. It creates a checkpoint +recorder only when recording is enabled and a checkpoint loader only for a +resume invocation. It allocates debug state only after a safe run identity has +been generated and only when debug capture was requested. These choices keep +the three state families independently composable. + ## Output And Cache The pipeline runner returns logical output files. After validating every @@ -68,12 +75,19 @@ names only the step, lane, and stable reason code. Decision detail is selected from code-owned descriptions by reason code and then UTF-8 normalized and bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and codes remain intact through pipeline events and become strings only in manifest -and debug-summary JSON. [Operations](../operations.md#resume-and-selective-recompute) -is the canonical operator-facing reason-code reference. +and debug-summary JSON. +[Operations](../operations.md#checkpoint-recording-resume-and-recompute) is the +canonical operator-facing reason-code reference. `internal/core/fileio` provides confined atomic file writes used by state collaborators. The chunk-plan store retains its stronger entry validation. +The CLI constructs selective-recomputation policy from resolved generated +artifact dependencies. It forces the selected step and transitive consumers, +while marking unforced producers as required reusable inputs. The runner owns +the actual hydration and rejection decisions; the [Operations guide](../operations.md#checkpoint-recording-resume-and-recompute) +owns the operator workflow and stable reason-code meanings. + ## Debug Bundles `internal/core/debugbundle` allocates an explicitly requested per-run bundle @@ -98,12 +112,27 @@ operation writes the success report, or makes one attempt each to write the failure report and error log. Terminal persistence failures are reported separately and never replace the command's primary error. +## Invariants To Preserve + +- Modules receive state collaborators and request data, never physical roots. +- Output logical paths are validated before a run directory is allocated, and + files are atomically written within that directory. +- Chunk-plan publication occurs only for accepted plans; bypass does not + construct or touch a plan store. +- Checkpoint recording and checkpoint loading remain separate collaborators. +- Debug state is opt-in, is not cache input, and terminal reporting does not + obscure the command's primary failure. + ## Tests To Inspect - `internal/cli/run_contract_test.go`: command-owned state allocation, terminalization, and output/report boundaries. +- `internal/cli/cache_contract_test.go`: cache-mode precedence, root selection, + and resume collaborator construction. - `internal/cli/state_hardening_test.go`: independent roots, reuse, failures, permissions, cleanup, and redaction. +- `internal/cli/recompute_policy_test.go`: forced dependents and required + reusable predecessors for selective recomputation. - `internal/cli/recompute_execution_contract_test.go`: selective recomputation, filesystem recovery, deterministic decisions, and failed predecessor state. - `internal/cli/production_contract_test.go`: production composition and diff --git a/docs/operations.md b/docs/operations.md index 03f4076..7b501e8 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,322 +1,206 @@ # Operations -This is the canonical guide to operating Notarius filesystem state. Command -syntax is in the [CLI reference](cli.md); field definitions and precedence are -in [Configuration](config.md). +This is the canonical guide for operating Notarius runtime state. The +[CLI reference](cli.md) owns command syntax and exit statuses, while +[Configuration](config.md) owns fields, defaults, and precedence. Maintainers +who need implementation mechanics should read [Run State Internals](internal/state.md). -## State Model +## State Surfaces -Notarius uses three independent filesystem surfaces: +Each run can use independent roots with different retention and access-control +needs. -- output is durable user data; -- cache is reconstructible chunk-plan and checkpoint state; and -- debug is explicitly requested inspection data. +| Surface | Purpose | Created when | Retention | +| --- | --- | --- | --- | +| Output | Durable user-facing result bundle | A pipeline completes and returns logical output files | Keep until consumers no longer need it. | +| Chunk-plan cache | Reconstructible source-addressed plan | The configured cache mode permits cache I/O | Keep while reuse is useful. | +| Checkpoint cache | Reconstructible execution and recovery state | Checkpoint recording is enabled | Keep only while recovery or reuse is useful. | +| Debug bundle | Explicit diagnostic record | A run requests debug collection | Keep only under an intentional sensitive-data retention policy. | -Choose separate roots and access controls for each surface. A normal run writes -durable output, may use the chunk-plan cache, and records checkpoints when -`cache.checkpoints.enabled` is true. It does not create debug state unless its -invocation includes `--debug`. +Output, cache, and debug roots are never merged or cleaned automatically. Use +separate locations and permissions for operators or services that must not +share application data. -## Output +## Roots And Permissions -Durable logical files are written under: +The configured output and debug directories are exact roots. An empty cache +directory selects a per-user root: -```text +~~~ +/notarius/chunk-plans +/notarius/checkpoints +~~~ + +The field definitions and configuration examples are in [Configuration](config.md). +On supported Unix systems, output directories and files are created with +requested modes **0755** and **0644**. Chunk-plan, checkpoint, and debug +directories and files use **0700** and **0600**. The operating system's umask +may impose stricter output modes. Cache and debug roots may contain sensitive +source-derived data, so provision them for one trusted account or service. + +## Run Lifecycle + +Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads +and resolves configuration before module preparation and source parsing. It +then performs any permitted cache lookup, executes the pipeline, and publishes +logical output files only after a successful runner result. + +On success, the command reports the output bundle path. A warning-bearing run +still succeeds and reports its warning count on standard error. Errors and +their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses). + +## Output Bundles + +Each successful run receives a generated safe run identifier and writes beneath: + +~~~ // -``` +~~~ -The CLI generates one run ID in the form -`run--<32-lowercase-hex-characters>` and uses it -for output, manifests, and any requested debug bundle. It validates every -logical output name before exclusively creating the run directory. If that -directory already exists, the invocation fails without changing it. +The [JSON output contract](integrations/json-output.md) owns the logical files +and their schemas. Before creating the run directory, Notarius validates every +logical output path. It refuses an existing run directory without changing it. +Files are written atomically; if a later write fails, the newly created partial +run directory remains for inspection and is never removed automatically. -Each output file is written atomically. A later file-write failure leaves the -newly allocated partial run directory in place for inspection; Notarius never -automatically removes output. The -[JSON output contract](integrations/json-output.md) owns the logical file -names, schemas, and media types inside a run directory. - -An enabled JSON `include_chunk_map` option adds an accepted chunk map to durable -output. Its annotations may contain source- or model-derived data, so retain -and protect it like lane output. The map is opt-in and does not alter existing -bundles; its payload exclusions are defined in the -[Accepted Chunk Map contract](integrations/chunk-map.md). - -Remove an output run directory only after its consumer data is no longer -needed. This is data deletion, not cache cleanup. - -## Ordered D&D Workflow - -The maintained [complete D&D configuration](../examples/dnd-complete.config.yml) -contains one pipeline with two ordered steps. The first step extracts and -normalizes independent item events, NPCs, and scene descriptions. Only after -the NPC and scene-description lanes reach accepted terminal results does the -second step begin; its generated NPC reference is -supplied in memory to spell, combat-turn, and NPC-interaction extraction and -the applicable normalizers, while its generated scene-description reference is -supplied to combat-turn extraction. The item-event lane has no generated -reference dependency and retains only current-transcript evidence. - -```sh -go run ./cmd/notarius run dnd-session \ - --config examples/dnd-complete.config.yml \ - --input examples/seriatim-minimal-transcript.json \ - --output-dir ./npc-grounded-output -``` - -The NPC artifact grounds canonical names through a names-only prompt projection, not spell or combat -evidence. Current-transcript source ranges remain the only event evidence. The -manifest records generated-reference identity and bounded producer provenance; -it does not record generated payload content, and no generated content is -exposed through a filesystem path. The same producer artifact may fan out to -compatible consumers, while a missing or rejected producer prevents the later -step from starting. - -Combat-turn extraction calls its LLM only when a scene record exactly matches -the accepted chunk and has kind `combat`. Exact `narrative`, `recap`, and -`meta` records produce an accepted empty combat-turn result without an LLM call. -Missing or mismatched scene coverage also produces an accepted empty result and -a bounded warning. Scene descriptions remain control context rather than combat -evidence or prompt material; the complete behavior is defined by the -[D&D combat-turn artifact contract](integrations/dnd-combat-turn-artifacts.md). - -Standalone module configurations continue to support external NPC files when a -workflow intentionally crosses a process or session boundary. Those files are -validated against the consumer slot and must be protected as sensitive -campaign data. They are not part of the maintained ordered handoff workflow. - -### NPC Semantic Normalization - -Before the first step can release its accepted NPC artifact across the ordered -generated-reference barrier, `dnd/npcs` performs one document-level semantic -normalization call for each configured normalize attempt when eligible distinct -names remain. The normalize binding's `retries` setting controls additional -attempts. If an invalid or unsafe identity proposal exhausts that budget, the -run safely accepts the deterministic base result and any independently -validated, model-proposed partial consolidation, with a bounded warning; -ordinary validation still applies before the artifact can cross the barrier. - -An accepted normalized NPC checkpoint can be reused on `--resume` just like -other accepted normalize work. A changed normalization prompt, response schema, -or policy identity produces a cold cache miss, so the current reconciliation is -recomputed rather than silently reusing incompatible state. +Treat an output bundle as durable user data. Do not use cache-cleanup policy to +remove it. An optional accepted chunk map is also durable output and can carry +source- or model-derived annotations; its content and compatibility contract +are defined in [Accepted Chunk Map](integrations/chunk-map.md). ## Chunk-Plan Cache -Chunk plans are stored at: +Chunk plans live beneath the selected chunk-plan root: -```text +~~~ //plan.json -``` +~~~ -`auto` reuses a complete valid plan or regenerates missing or invalid state. -`refresh` regenerates and atomically replaces a plan after chunk validation. -`bypass` performs no plan-cache I/O and does not resolve or create the root. -Plan selection is source-addressed and independent of checkpoint and debug -roots. +One validated canonical plan is active for each source digest. The plan stores +boundaries and provenance, not a second copy of the entire source. This +source-addressed policy is recorded in [ADR-0005](adr/0005-cache-canonical-chunk-plans-by-source.md). -When its directory is empty in configuration, the root is -`/notarius/chunk-plans`. A configured directory is the exact -root; no suffix is appended. Directories and files created by the store use -`0700` and `0600` permissions on supported Unix systems. The configured root -is a trust boundary: do not share it among mutually untrusted users. +The configured cache mode controls one invocation: -Remove an exact digest directory or the configured root only when accepting the -cost of recomputing plans and any chunk-stage work. Cache publication is atomic; -there is no history, locking, garbage collection, or rollback facility. +- **auto** looks for a valid active plan. Missing or invalid state causes a new + plan to be generated; an accepted new plan is atomically published. +- **refresh** skips lookup, generates a plan with the configured chunker, and + atomically replaces the active plan after it is accepted. +- **bypass** performs no chunk-plan cache I/O. It does not resolve or create a + chunk-plan root. -For a Linux service account, provision a dedicated restrictive root such as: +A reused plan is still materialized and validated against the current source. +If a prior plan no longer gives acceptable results, use a refresh run rather +than editing cache files. Deleting a plan is recoverable but can repeat costly +chunking work. -```yaml -cache: - chunk_plans: - directory: /var/cache/notarius/chunk-plans -``` +## Checkpoint Recording, Resume, And Recompute -## Checkpoint Cache +Checkpoint recording is an explicit configuration choice and is disabled by +default. When enabled, each run records stage transitions and the state needed +for compatible recovery. A run records checkpoints even when it does not ask +to reuse them. Checkpoint payloads can contain source-derived and intermediate +application data, so treat the entire root as sensitive. -Checkpoint recording is controlled by `cache.checkpoints.enabled`, which -defaults to `false`. When enabled, every run records running, succeeded, and -failed transitions and reusable validator-approved results. Successful, -rejected, and failed runs may therefore all leave checkpoint state. The -`--resume` flag additionally loads compatible completed work before executing -missing or incompatible stages. Without `--resume`, a recording-enabled run -never loads checkpoints. Using `--resume` while recording is disabled is an -error. +Checkpoint loading is separate: [**--resume**](cli.md#run) asks a run to reuse +compatible recorded work. A resume request fails when checkpoint recording is +disabled. Without **--resume**, a recording-enabled run executes normally and +does not load checkpoint state. Compatibility includes the resolved pipeline, +input, selected lanes, runtime overrides, reference provenance, LLM-profile +provenance, and prepared-component fingerprints. A changed identity produces a +cold miss; Notarius does not migrate, rewrite, or delete older checkpoint +directories. -Checkpoints use the selected root and the existing identity hierarchy: +Checkpoint state is confined below an identity-specific path: -```text -//-///... -``` +~~~ +//-/// +~~~ -The final identity digest includes stable semantic fingerprints explicitly -contributed by prepared modules and validators. Adding or changing one of -these fingerprints intentionally causes a cold cache miss; old checkpoint -directories are left in place and are never migrated or deleted automatically. +### Selective Recompute -An empty configured directory selects -`/notarius/checkpoints`. The root is exact when configured. -Created directories and files use `0700` and `0600` permissions on supported -Unix systems. +[**--recompute-step**](cli.md#run) requires both **--resume** and enabled +checkpoint recording. It forces the selected ordered step and every lane that +depends on it through generated artifact references. Unrelated lanes remain +eligible for reuse. -Checkpoint payloads can contain source text, intermediate artifacts, metadata, -warnings, and content digests. Treat them as sensitive derived application -data. Compatible files from a former checkpoint root remain reusable when -`cache.checkpoints.directory` names that exact existing root. They are not -moved, migrated, or deleted automatically. The frozen serialized identifier -`workspace_schema_version` remains part of checkpoint compatibility; it is not -a configuration setting. +For an earlier producer required by a forced consumer, Notarius requires a +compatible accepted normalized artifact. It validates that artifact before +hydrating it and does not silently rerun the producer. If that state is +missing, rejected, corrupt, non-canonical, or incompatible, the run stops +before its dependent starts. Rerun the required producer deliberately instead +of copying or editing checkpoint files. -For a Linux service account, independently provision: +## Checkpoint Decisions And Recovery -```yaml -cache: - checkpoints: - enabled: true - directory: /var/cache/notarius/checkpoints -``` +Checkpoint events classify work as **executed**, **reused**, +**forced_recompute**, or **dependency_invalidated**. Their stable reason codes +are written to run diagnostics and provenance. Use the code, not a copied +error message, to decide what to repair. -Remove an exact checkpoint identity directory or the configured root only when -recomputation is acceptable. - -### Resume And Selective Recompute - -`--resume` loads compatible accepted work only when checkpoint recording is -enabled. A normal resumed run may reuse source, extract, merge, and normalize -checkpoints independently and may recompute a stage after a cache miss. -Generated references add a dependency fingerprint -for the producer's artifact kind, schema identity, media type, canonical -content digest, and size. If that fingerprint changes or the producer is -missing, dependent checkpoints are invalidated; unrelated work remains eligible -for reuse. - -`--recompute-step ` requires both `--resume` and -`cache.checkpoints.enabled: true`. It forces the named step and all transitive -dependents to execute, while compatible predecessors and unrelated lanes remain -reusable. The ID may be an explicit configured step or `default` for an -implicit single-step pipeline. It cannot be combined with `--only`, and it does -not change the persistent identity of otherwise identical checkpoints. -Decisions are bounded and categorized as `reused`, `executed`, -`forced_recompute`, or `dependency_invalidated`. - -For an unselected producer required by a recomputed step, Notarius loads the -accepted normalized artifact directly. Valid normalize state is sufficient even -when that producer's extract or merge checkpoint is missing or corrupt. The -normalize manifest must be successful and match workspace schema v3, the exact -current invocation identity, step, lane, and normalizer; its payload digest and -canonical codec representation must also validate. A forced producer bypasses -this lookup and executes. - -If a required predecessor's accepted normalized artifact is missing, rejected, -corrupt, non-canonical, or incompatible, the run fails before the dependent -step starts. It does not fall back to rerunning that predecessor. The failure -manifest retains completed upstream outcomes and dependency context but not -generated reference content. For diagnosis, first check the producer step and -lane in the manifest, then inspect checkpoint decision categories and reason -codes. Rerun the producer explicitly rather than copying an artifact into the -checkpoint root. - -The decision that caused a required-predecessor failure is retained before the -run returns, and the CLI error identifies its step, lane, and reason code. - -Checkpoint reason codes are stable diagnostic identifiers: - -| Reason code | Operator meaning | +| Reason code | Recovery meaning | | --- | --- | -| `loading_disabled` | This invocation did not enable checkpoint loading. | -| `checkpoint_missing` | The requested checkpoint file does not exist. | -| `checkpoint_path_invalid` | The requested checkpoint location failed confinement validation. | -| `checkpoint_read_failed` | An existing checkpoint could not be read. | -| `checkpoint_decode_failed` | Checkpoint JSON could not be decoded. | -| `workspace_schema_incompatible` | The stored workspace schema is not supported by this build. | -| `identity_mismatch` | The stored invocation identity differs from the current invocation. | -| `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch` | Stored scope does not match the requested pipeline scope. | -| `status_not_reusable` | The stored operation did not finish in a reusable status. | -| `dependency_mismatch` | Stored dependencies differ; the category is `dependency_invalidated`. | -| `artifact_payload_invalid` | Stored artifact payload structure or encoding is invalid. | -| `artifact_digest_mismatch` | Stored artifact bytes do not match their recorded digest. | -| `artifact_codec_incompatible` | Stored artifact identity is incomplete or incompatible with the codec contract. | -| `artifact_not_canonical` | The codec can decode the artifact, but its bytes are not canonical. | -| `checkpoint_reused` | The stored checkpoint passed validation and was reused. | -| `accepted_artifact_reused` | A required producer's accepted normalized artifact was canonically validated and hydrated. | -| `recompute_step` | Selective recomputation forced execution of this lane. | +| **loading_disabled** | This invocation did not permit checkpoint loading. | +| **checkpoint_missing**, **checkpoint_path_invalid**, **checkpoint_read_failed**, **checkpoint_decode_failed** | The stored checkpoint could not be located or read safely; normal resume work can execute again. | +| **workspace_schema_incompatible**, **identity_mismatch**, **stage_mismatch**, **step_mismatch**, **lane_mismatch**, **module_mismatch** | Stored state belongs to a different compatible scope or identity; allow a fresh run to create new state. | +| **status_not_reusable** | The recorded operation did not end in reusable state. | +| **dependency_mismatch** | A dependency changed; dependent work is invalidated rather than reused. | +| **artifact_payload_invalid**, **artifact_digest_mismatch**, **artifact_codec_incompatible**, **artifact_not_canonical** | A stored artifact cannot safely be hydrated; rerun the producer instead of modifying the cache. | +| **checkpoint_reused** | A normal checkpoint passed compatibility checks. | +| **accepted_artifact_reused** | A required predecessor's accepted normalized artifact was safely hydrated. | +| **recompute_step** | Selective recomputation deliberately forced this work. | -Decision detail is bounded explanatory text derived from the stable reason code, -not caller-supplied prose or a data-recovery channel. It never contains -checkpoint paths, artifact or reference content, source content, credentials, -or environment values. +Reason detail is bounded code-owned text. It is diagnostic information, not a +path-discovery or data-recovery mechanism, and does not contain checkpoint, +source, reference, credential, or environment content. ## Debug Bundles -Only `notarius run --debug` enables debug collection. The selected root contains -one retained bundle per invocation: +Only a [debug-enabled run](cli.md#run) creates a bundle: -```text +~~~ // summary/ trace/ -``` +~~~ -`summary/` contains redacted invocation, effective-configuration, resolved -pipeline and reference provenance, checkpoint and chunk-plan decisions, run -manifest, warnings, report, and any available error text. It excludes raw -source, references, annotations, prompts, model responses, credentials, and -malformed cache bytes. +The summary contains redacted invocation and resolution information plus run, +warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace +contains allowlisted application diagnostic records and can include source or +derived application data. Neither surface is a cache input. Do not treat a +debug bundle as safe to share merely because its configuration summary is +redacted. -`trace/` contains application-owned execution detail, including source and -stage material, plans, chunks, validator attempts, prompts, model responses, -timing, and serialized artifacts. It may retain application data omitted from -output. Credentials, credential-shaped values, sensitive metadata, unrelated -environment values, and unrelated filesystem content are not captured. - -Bundles inherit the sensitivity of the application data they capture. Their -additional risk comes from copying and aggregating that data, so restrict -access, avoid shared roots between untrusted users, and define retention outside -Notarius. Created bundle directories use `0700` and files use `0600` on -supported Unix systems. - -Notarius never automatically deletes a requested bundle. If allocation -succeeds, its path is reported on success and failure. A requested summary or -trace write failure makes the command fail, preserving whatever bundle data was -already written for inspection. Every allocated bundle makes one best-effort -attempt to record a terminal `run-report.json`. - -## Failures And Warnings - -Failures before debug allocation are reported on stderr without a bundle. -Failures after allocation report the bundle path on stderr and make independent -attempts to write a failure `run-report.json` and `error.log`. The report retains -the paths and pipeline outcome fields known at the failure point. If either -terminal write fails, the original command error remains first on stderr, -followed by the persistence error and bundle path. An output-write failure -leaves the allocated bundle in place. A successful run with warnings exits `0`, -reports a warning count on stderr, and records warnings in durable output and -any requested debug summary. +Notarius never creates debug state without an explicit request and never +automatically deletes a requested bundle. If allocation succeeds, the command +reports its path on both success and later failure. A summary, trace, or +terminal-report persistence failure fails the command while preserving any +already-written diagnostic data for inspection. ## Cleanup -Use exact paths for manual cleanup. Examples: +Cleanup is manual and destructive. First inspect the exact leaf directory, +then remove only that leaf; do not use a glob or a parent root as the target. -```sh -rm -rf ./notarius-output/run-1721300000000000000-0123456789abcdef0123456789abcdef -rm -rf /var/cache/notarius/chunk-plans/0123abcd -rm -rf /var/cache/notarius/checkpoints/pipeline/input-0123/pipeline-4567/identity-89ab -rm -rf ./notarius-debug/run-1721300000000000000-0123456789abcdef0123456789abcdef -``` +~~~ +rm -rf -- /srv/notarius/output/run-1721300000000000000-0123456789abcdef0123456789abcdef +rm -rf -- /srv/notarius/chunk-plans/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef +rm -rf -- /srv/notarius/checkpoints/example/seriatim-0123456789abcdef/0123456789abcdef/0123456789abcdef +rm -rf -- /srv/notarius/debug/run-1721300000000000000-0123456789abcdef0123456789abcdef +~~~ -Avoid broad recursive cleanup against a parent root unless it is an explicit -operator policy. Output deletion is permanent user-data loss. Cache deletion is -recoverable but can repeat expensive work. Debug deletion removes troubleshooting -evidence and any retained application-data copy. +Deleting output permanently removes user data. Deleting chunk plans or +checkpoints is recoverable but may repeat expensive provider or pipeline work. +Deleting a debug bundle removes troubleshooting evidence and a retained copy of +application data. Notarius has no cache garbage collector, rollback operation, +or automatic cleanup command. ## Operational Limits -Provider retries and timeouts are handled by Scriptorium according to the -selected execution profile. Pipeline module retry settings are defined in -[Configuration](config.md#module-bindings). Extract worker concurrency and -actual provider-call concurrency are separate limits; their fields and -validation are defined in [Configuration](config.md#concurrency). Notarius -writes local files only; remote storage and archive management are outside the -implemented CLI. +Provider retries and timeouts are supplied by the selected Scriptorium profile. +Module retry settings and concurrency limits are configuration contracts; see +[module bindings](config.md#module-bindings-and-validators) and +[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker +limits and actual provider-call limits are independent. Notarius writes local +filesystem state only; remote storage, archival, and retention automation are +outside the implemented CLI.