diff --git a/docs/roadmap/deployment-managed-cache-permissions.md b/docs/roadmap/deployment-managed-cache-permissions.md new file mode 100644 index 00000000..aaf2d486 --- /dev/null +++ b/docs/roadmap/deployment-managed-cache-permissions.md @@ -0,0 +1,236 @@ +# Deployment-Managed Cache Permissions + +Status: proposed + +## Summary + +Stop treating cache access permissions as application policy. Notarius should +create chunk-plan and checkpoint state through conventional Unix filesystem +semantics and leave the effective owner, group, and access policy to the +operator's provisioned directories, process umask, setgid inheritance, and +default ACLs. + +Notarius continues to own safe paths, symlink rejection, atomic publication, +cache validation, and recovery. It does not decide whether a configured cache +is private to one account, shared by a trusted group, or readable under some +other deployment policy. + +## Motivation And Current Problem + +The current cache implementations enforce a policy that was stronger than the +application requires: + +- the chunk-plan store creates and repeatedly converges cache directories to + `0700` and atomic files to `0600`; and +- the checkpoint recorder supplies `0700` directory and `0600` file modes to + the shared confined writer, which explicitly applies the file mode during + atomic publication. + +This prevents an administrator from using ordinary Unix provisioning to share +one cache between a service account and trusted operator accounts. Adding a +second Notarius-specific `private` or `group` configuration policy would solve +that immediate symptom, but it would preserve the underlying ownership error: +Notarius would still decide access policy instead of respecting the deployment. + +Routine Notarius operation does not require an application-enforced private +cache. Cache records can contain source-derived data, so operators must be told +what they are storing, but sensitivity does not imply that Notarius should +override a deliberately configured filesystem policy. API credentials remain +outside cache payloads and continue to follow their existing configuration and +redaction boundaries. + +## Target Policy + +The target contract is: + +> Notarius creates cache entries with conventional maximum creation modes and +> allows the operating system to determine effective access through the +> process umask, inherited group ownership, and default ACLs. + +Concretely: + +- newly created cache directories request `0777`; +- newly created cache files, including atomic temporary files, request `0666`; +- the kernel applies the invoking process's umask and any applicable default + ACL; +- setgid directories provisioned by the operator control inherited group + ownership and, on supported Unix systems, setgid propagation to descendants; +- Notarius does not call `chmod`, `chown`, or `chgrp` to impose or repair a + cache access policy; and +- Notarius does not reject a cache merely because its effective modes are more + or less permissive than a Notarius-selected default. + +This is policy inheritance, not literal copying of a parent directory's mode +bits. Directories and non-executable data files appropriately have different +maximum modes. A parent mode alone also does not determine child modes: an +operator who needs a particular result must configure the service umask and, +where appropriate, a setgid parent or default ACL. + +For example, a service started with umask `0077` ordinarily creates `0700` +directories and `0600` files. A trusted-group deployment can provision a +setgid cache root and use umask `0007`, or an equivalent default ACL, to obtain +`0770` directories and `0660` files. These are deployment choices rather than +configuration values interpreted by Notarius. + +## Public Contract + +No new Notarius configuration field, environment variable, or CLI flag is +introduced. In particular, do not add the previously considered +`cache.access: private|group` setting. + +Existing cache enablement, locations, chunk-plan modes, resume controls, +identity, and compatibility rules remain unchanged. The behavioral change is +that effective permissions on newly created cache entries are controlled by +the deployment environment rather than forcibly converged by Notarius. + +The policy applies uniformly to both cache families: + +- canonical chunk plans; and +- invocation checkpoints. + +Output and debug storage are outside this feature's scope. Their current +contracts must not change incidentally through a shared helper refactor. A +separate review may later consider whether the same deployment-owned policy is +appropriate for those surfaces. + +## Filesystem And Atomic-Publication Requirements + +Permission deference must cover every cache write path, not only final files: + +- configured cache-family roots created by Notarius; +- chunk-plan source-digest directories; +- checkpoint identity and stage directories; +- atomic temporary files; and +- final files published by rename. + +Atomic publication must not bypass the deployment policy. Go's +`os.CreateTemp` creates a file with `0600`, and applying `chmod(0666)` afterward +would bypass the process umask. Cache writers therefore need an atomic +temporary-file creation path that chooses a collision-resistant name and opens +it with `O_CREATE|O_EXCL` and requested mode `0666`, allowing the kernel to +apply umask and default ACL policy at creation time. + +Replacement writes publish a newly created temporary inode. Its effective +permissions and ACL are therefore those selected for new entries by the +current parent and process policy; Notarius does not copy mode bits or ACLs +from the replaced file. Operators who require stable replacement permissions +must express that policy through a stable service umask and directory default +ACL. + +Opening an existing cache for writing must not alter its root or descendant +modes. Read-only lookup of a missing cache must remain side-effect free. + +## Preserved Safety And Correctness Boundaries + +Deferring access policy does not weaken Notarius's filesystem correctness +rules. The implementation must preserve: + +- confined resolution beneath the selected cache root; +- rejection of symlinked or unexpected cache entries; +- validation of source digests, checkpoint identities, schemas, and recorded + provenance; +- atomic replacement, existing synchronization behavior, cleanup of abandoned + temporary files, and complete-record visibility; +- existing missing, malformed, incompatible, and operational-error recovery + classifications; and +- separation of output, cache, and debug roots. + +An administrator's decision to permit multiple accounts to reach one cache +does not create a new Notarius guarantee for simultaneous writers. Preserve +the existing atomicity and validation behavior, but do not add locking or claim +transactional coordination across concurrent processes as part of this work. +Concurrent writes to the same multi-file checkpoint identity remain subject to +the currently documented cache behavior; the implementation documentation +must state any existing limitation accurately. + +## Architectural Boundaries + +The target removes application-owned cache modes without creating a general +permission-policy framework. Chunk-plan and checkpoint storage may use +different filesystem APIs internally, but both must implement the same +deployment-owned result. Pipeline modules and the runner remain unaware of +physical roots and filesystem access policy. + +Any shared writer API must distinguish an application-enforced exact mode from +an umask-filtered maximum creation mode. The cache change must not silently +alter output or debug behavior merely because checkpoints currently share +filesystem helpers with those surfaces. + +## Existing Cache Migration + +Stopping permission convergence does not change existing inode modes. A cache +tree created by an older Notarius version may therefore remain `0700`/`0600` +and inaccessible to another account after upgrade. + +Operators who want a different policy must either: + +- remove the reconstructible cache and allow it to be recreated beneath a + correctly provisioned parent and service environment; or +- stop writers and deliberately update ownership, modes, and ACLs using normal + system-administration tools. + +Notarius must not perform a recursive startup migration. The same rule applies +when an operator wants to make a previously shared cache private: change the +deployment policy and deliberately migrate or recreate existing state when an +immediate whole-tree change is required. + +## Documentation And Decision Records + +The final documentation set includes an accepted ADR titled along the lines of +"Defer cache access policy to the deployment." The ADR records the durable +ownership boundary so later maintenance does not reintroduce +application-enforced modes solely because cache data can contain source-derived +content. + +The new ADR need not supersede ADR-0005 or ADR-0007: those decisions govern +canonical chunk-plan reuse and checkpoint recording/reuse, while leaving exact +storage permissions to operational contracts. It should reference those ADRs +and the output/cache/debug separation established by ADR-0006 and refined by +ADR-0007. + +In the final current-behavior documentation, `docs/operations.md` owns cache +provisioning, umask, setgid, default ACL, migration, sensitivity, and +concurrent-use guidance. Architecture owns the deployment/application boundary; +the configuration reference clarifies that selecting a cache path does not +select its permissions; and the internal state guide explains the implemented +write mechanics. The release that first contains the feature summarizes the +operational change without becoming its canonical reference. + +Current-behavior documentation must not describe this target behavior before +the implementation lands. + +## Verification Expectations + +Behavioral tests protect kernel-selected permissions for both cache families, +non-mutation of existing roots, atomic replacement, and representative +assembled CLI wiring. Existing confinement, symlink, validation, identity, +reuse, and recovery coverage remains authoritative. Umask tests are POSIX-only, +serialized, and restore process-global state. The default suite does not create +privileged accounts or groups; setgid and default-ACL examples may be verified +in a purpose-built Linux deployment environment. + +## Acceptance Criteria + +- No cache access-mode field or application-specific private/group policy is + added. +- Both chunk-plan and checkpoint writers request `0777` for new directories + and `0666` for new files, including atomic temporary files. +- Effective permissions reflect the process umask and applicable default ACLs + rather than a later Notarius chmod. +- Existing cache roots and descendants are not chmod-converged during reads or + writes. +- A private deployment can obtain private cache state through its filesystem + provisioning and umask without a Notarius-specific setting. +- A trusted-group deployment can obtain shared cache state through setgid, + umask, and/or default ACL policy without a Notarius-specific setting. +- Existing cache trees are not recursively migrated. +- Chunk-plan and checkpoint enablement, lookup, recording, resume, identity, + validation, confinement, atomicity, and recovery semantics remain unchanged. +- Output and debug permission behavior does not change as an incidental effect. +- Notarius performs no user/group management, ownership mutation, ACL + management, privilege escalation, or cross-process locking. +- The ADR and canonical operations, architecture, configuration, and internal + documentation accurately describe the implemented boundary; the next + release note summarizes the change when that release is prepared. +- Focused tests and repository-wide `go test ./...`, `go vet ./...`, and + `go build ./cmd/notarius` checks pass. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 91f666eb..958f8bc7 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -150,6 +150,16 @@ address Go dependency cycles, registrar ownership, stable public module keys, and supersession of the affected ADR-0004 decision. Conceptual artifact-family ownership does not by itself require this move. +## Near-Term Operations + +### Deployment-Managed Cache Permissions + +- Implement the proposed + [deployment-managed cache permissions](deployment-managed-cache-permissions.md) + contract so Notarius respects the access policy established by the cache + root, process umask, setgid inheritance, and default ACLs instead of imposing + application-owned private or group modes. + ## Blue-Sky Platform And Operations These ideas are intentionally less specified. Promote one into an earlier diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 00000000..309cd2b2 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,347 @@ +# Deployment-Managed Cache Permissions Implementation Plan + +## Purpose + +Implement the target contract in +[Deployment-Managed Cache Permissions](deployment-managed-cache-permissions.md): +Notarius must preserve safe, confined, atomic cache I/O while allowing normal +Unix creation semantics to determine who can access newly created chunk-plan +and checkpoint state. + +This document is the execution sequence. The feature roadmap is authoritative +for user intent, policy, scope, and the target end state. Each numbered stage +below is sized for one `gpt-5.6-terra` implementation prompt and must be +completed in order. + +## Cross-Stage Requirements + +Every stage must preserve these decisions: + +- Do not add a `cache.access` field, environment variable, CLI flag, or an + application-level private/group policy. +- New cache directories request mode `0777`; new cache files request mode + `0666`. The kernel applies umask and default ACL policy. +- Do not chmod, chown, chgrp, or recursively migrate existing cache entries to + enforce access policy. +- Keep chunk-plan and checkpoint confinement, symlink rejection, identity, + validation, atomic replacement, cleanup, and recovery behavior intact. +- Do not change output or debug permissions as an incidental consequence of a + shared-helper change. +- Do not add cross-process locking or promise transactional concurrent writes + to one checkpoint identity. +- Follow `docs/policy/testing.md`: protect the observable permission boundary + and meaningful filesystem risks without snapshotting helper structure or + duplicating all cases at every layer. +- Keep the repository buildable and the tests relevant to the completed stage + passing. Do not retire either roadmap document during implementation; they + remain review evidence until the completed feature is accepted. + +## Stage 1: Record The Decision And Add An Umask-Respecting Atomic Writer + +### Goal + +Record the durable ownership boundary and add the narrow filesystem primitive +needed by checkpoint storage, without changing any production caller yet. + +### Required Changes + +1. Add `docs/adr/0017-defer-cache-access-policy-to-deployment.md` using the + repository's Nygard ADR format, with status `Accepted` and the implementation + date. The decision must: + - assign cache access policy to deployment provisioning, process umask, + setgid inheritance, and default ACLs; + - distinguish access policy from Notarius-owned confinement, validation, + atomicity, and recovery; + - reject both forced private modes and a Notarius-specific private/group + setting; + - state that source-derived cache contents may be sensitive and that the + operator owns access and retention; + - state that API credentials must remain outside cache payloads under the + existing secret boundary; + - explain that existing cache modes are not migrated automatically; + - reference ADR-0005, ADR-0006, and ADR-0007 without superseding them; and + - record application-enforced `0700`/`0600`, configurable permission modes, + and recursive permission convergence as rejected alternatives. +2. Extend `internal/core/fileio` with exactly two cache-oriented entry points: + - `WriteBytesWithProcessPermissions(root, name string, data []byte) error`; + - `WriteJSONWithProcessPermissions(root, name string, payload any) error`. +3. The new byte writer must reuse `SafePath` and symlink-component rejection, + call `os.MkdirAll` with `0777`, and atomically replace the target. The JSON + wrapper must retain the existing indented JSON plus trailing-newline format. +4. Implement the new atomic path independently from the existing exact-mode + `WriteBytes` path: + - generate a 16-byte cryptographically random suffix; + - attempt at most 32 unique sibling names of the form + `..tmp-`; + - open with `os.O_WRONLY|os.O_CREATE|os.O_EXCL` and mode `0666` so the kernel + applies umask and default ACLs at creation; + - retry only name collisions; + - write, close, rename, and remove an unpublished temporary file on every + failure path; and + - do not call `chmod` or inspect/change the process umask in production code. +5. Leave `WriteBytes`, `WriteJSON`, and their existing callers unchanged so + output and debug behavior cannot drift in this stage. + +### Tests + +- Add supported-Unix (`linux || darwin`) behavioral tests using + `syscall.Umask`. Each test must restore the prior umask with `defer` and must + not use `t.Parallel`. +- Prove a restrictive umask (`0077`) produces `0700` nested directories and a + `0600` final file. +- Prove a group-oriented umask (`0007`) produces `0770` nested directories and + a `0660` final file. +- Pre-create the selected root with a distinctive mode, write beneath it, and + prove the root mode is unchanged. +- Replace an existing file created under a different mode and prove the new + inode follows the current process policy and contains the complete new + content. +- Prove no temporary sibling remains after success. Retain all existing + confinement, symlink, and exact-mode writer tests. + +### Validation + +Run: + +```sh +gofmt -w internal/core/fileio/*.go +go test ./internal/core/fileio +go test -race ./internal/core/fileio +git diff --check +``` + +### Stage Acceptance + +The ADR records the agreed boundary; the new writer demonstrates both target +umasks without chmod; existing file-writing behavior has not changed; and the +stage is small enough for one implementation prompt. + +## Stage 2: Convert Canonical Chunk-Plan Storage + +### Goal + +Make every chunk-plan creation and replacement path follow deployment-managed +permissions while preserving the rooted filesystem and cache-recovery +protocol. + +### Required Changes + +1. In `internal/framework/chunkplan`: + - request `0777` when creating the configured root; + - remove the chmod of an opened existing root; + - request `0777` when creating a source-digest directory; + - remove the chmod branch for an opened existing digest directory; + - retain the before/open/after identity checks that defend against entry + replacement races; + - change the existing rooted `O_CREATE|O_EXCL` temporary-file open mode from + `0600` to `0666`; and + - remove the subsequent `temp.Chmod(0600)` call while preserving write, + sync, close, rename, cleanup, and interruption hooks. +2. Do not replace `os.Root` operations with ordinary path operations, weaken + unexpected-entry handling, or create a missing root during read-only load. +3. Rewrite `TestFilesystemStorePermissions` into a behavioral Unix test that + controls and restores umask. Consolidate permission cases rather than + retaining a stale forced-private assertion alongside the new contract. + +### Tests + +- Under umask `0077`, prove the root, digest directory, and published plan are + `0700`, `0700`, and `0600`. +- Under umask `0007`, prove they are `0770`, `0770`, and `0660`. +- Because the published plan is the renamed temporary inode, its final mode + proves the temporary-file creation policy; use the existing pre-rename hook + only if a separate assertion is needed to prevent a plausible regression. +- Pre-create both the root and digest directory with deliberate modes, save a + plan, and prove neither directory is modified. +- Retain the existing missing-root side-effect, symlink, entry-replacement, + interruption, atomic-replacement, record-validation, and recovery tests. + +### Validation + +Run: + +```sh +gofmt -w internal/framework/chunkplan/*.go +go test ./internal/framework/chunkplan +go test -race ./internal/framework/chunkplan +git diff --check +``` + +### Stage Acceptance + +Chunk-plan writes honor both controlled deployment policies, existing +directories are not converged, and all rooted atomicity and recovery coverage +passes. This stage is small enough for one implementation prompt. + +## Stage 3: Convert Checkpoints And Prove Assembled Cache Behavior + +### Goal + +Move checkpoints to the new shared writer, remove stale private-cache +expectations at the CLI boundary, and prove one assembled run applies the same +deployment policy to both cache families. + +### Required Changes + +1. Change the checkpoint recorder's JSON write boundary in + `internal/framework/checkpoint` to call + `fileio.WriteJSONWithProcessPermissions`. Do not thread modes through the + recorder or introduce a cache permission configuration value. +2. Replace the checkpoint round-trip test's forced `0700`/`0600` subsection + with focused POSIX permission coverage. Use a cache root that does not exist + before the recorder writes so root creation is covered. +3. Preserve checkpoint JSON representation, path layout, manifests, payloads, + identity, recording enablement, reuse eligibility, loader decisions, and + corrupt/incompatible-state behavior. +4. In `internal/cli/state_hardening_test.go`, stop applying + `assertRestrictedTree` to checkpoints. Retain that helper and assertion for + debug state, whose permission contract is unchanged. +5. Add one supported-Unix CLI integration test using the existing offline + state-test harness: + - configure both chunk-plan caching and checkpoint recording beneath + previously absent roots; + - set and reliably restore umask `0007` around the run; + - prove representative directories and final files in both cache families + are group-accessible (`0770`/`0660`); + - prove the run completes and the stored state remains reusable; and + - do not duplicate every package-level permission or recovery case. +6. Keep output and debug mode assertions unchanged and confirm that no + `cache.access` parsing or composition code exists. + +### Tests + +- Add checkpoint-level cases for umask `0077` and `0007`, covering + representative nested directories and both manifest and payload files. +- Prove a pre-existing checkpoint root is not chmod modified. +- Retain the existing all-stage round trip as the owner of serialization and + loader behavior rather than cloning it into each permission case. +- Ensure every umask-changing test is non-parallel and restores global state. + +### Validation + +Run: + +```sh +gofmt -w internal/framework/checkpoint/*.go internal/cli/*.go +go test ./internal/framework/checkpoint ./internal/cli +go test -race ./internal/framework/checkpoint ./internal/cli +go test ./internal/framework/chunkplan ./internal/core/fileio +git diff --check +``` + +### Stage Acceptance + +Both cache families follow process/filesystem policy through the real CLI +composition, checkpoint reuse remains intact, and output/debug permission +contracts have not changed. This stage is small enough for one implementation +prompt. + +## Stage 4: Update Canonical Documentation + +### Goal + +Make the implemented behavior discoverable without duplicating volatile +details across documentation owners. + +### Required Changes + +1. Update `docs/policy/architecture.md` with the normative boundary: + deployments own cache access policy; Notarius owns confined paths, atomic + publication, validation, and recovery. Do not generalize the rule to output + or debug state in this work. +2. Update `docs/operations.md`, the canonical operational owner, to: + - replace claims that cache entries are forcibly `0700`/`0600`; + - explain requested maximum modes `0777`/`0666`, umask filtering, default + ACLs, and setgid group inheritance without claiming literal mode-bit + inheritance; + - provide concise private (`0077`) and trusted-group (`0007` plus a + deployment-provisioned setgid root, or an equivalent default ACL) + examples; + - state that cache data is reconstructible but may contain source-derived + content and operators own access and retention; + - explain that pre-feature cache trees retain their existing inode modes and + must be deliberately recreated or administratively migrated; and + - accurately limit concurrent-use claims to the existing atomicity and + validation guarantees. +3. Update `docs/config.md` only to clarify that selecting chunk-plan or + checkpoint directories does not configure their filesystem access policy. + Do not document a nonexistent permission field. +4. Update `docs/internal/state.md` with the two implemented writer paths: + rooted chunk-plan atomic creation and checkpoint use of the + process-permission file writer. Link to Operations for operator-facing mode + and provisioning guidance instead of duplicating it. +5. Review `docs/internal/overview.md` and `docs/development.md`; update links or + routing only if the existing navigation no longer reaches the canonical + state and operations documents. +6. Do not edit a historical release note. The release procedure will summarize + this change when the next release candidate is prepared. + +### Validation + +- Verify every changed relative Markdown link and heading anchor. +- Search current-behavior documentation for stale assertions that cache modes + are always `0700`/`0600`; distinguish cache from still-valid debug claims. +- Confirm current-behavior documentation outside `docs/roadmap/` contains + neither `cache.access` nor instructions for Notarius-managed + chmod/chown/chgrp/ACL mutation. +- Run `git diff --check`. + +### Stage Acceptance + +Each policy and operational fact has one canonical owner, current docs describe +only implemented behavior, and the feature roadmap remains the owner of future +status until final review. This stage is small enough for one implementation +prompt. + +## Stage 5: Repository-Wide Verification And Contract Audit + +### Goal + +Prove the complete target state and catch accidental permission, safety, or +documentation drift before review. + +### Required Review + +1. Inspect production cache code for remaining `0700`, `0600`, or `Chmod` + operations. Every remaining occurrence must be a fixture, an unrelated + output/debug contract, or otherwise justified; neither cache writer may + enforce those values. +2. Search configuration, CLI, examples, and documentation to confirm no + `cache.access` surface or private/group application policy was introduced. +3. Confirm output and debug production paths and their contract tests are + unchanged except for necessary documentation disambiguation. +4. Confirm cache reads remain side-effect free where previously guaranteed and + every cache write remains confined and atomic. +5. Review all new tests against `docs/policy/testing.md`. Consolidate redundant + mode assertions while retaining one owner for each realistic regression: + kernel filtering, existing-root preservation, each cache writer, and + assembled CLI wiring. +6. Review the ADR and canonical docs for the agreed scope. Do not describe + Unix permissions as literal inheritance from a parent directory. + +### Validation + +Run the complete offline suite: + +```sh +gofmt -w internal/core/fileio/*.go internal/framework/chunkplan/*.go internal/framework/checkpoint/*.go internal/cli/*.go +go test ./... +go test -race ./internal/core/fileio ./internal/framework/chunkplan ./internal/framework/checkpoint ./internal/cli +go vet ./... +go build ./cmd/notarius +go run ./cmd/notarius config validate --config examples/dnd-minimal.config.yml --pipeline dnd-session +OPENROUTER_API_KEY=validation-placeholder go run ./cmd/notarius config validate --config examples/dnd-complete.config.yml --pipeline dnd-session +git diff --check +``` + +The config validations must perform no provider requests. Also inspect +`git status --short --untracked-files=all` for unintended files, generated +artifacts, or unrelated edits. + +### Final Acceptance + +Every acceptance criterion in the feature roadmap is supported by direct code, +test, or documentation evidence; all prescribed commands pass; the working +diff contains only this feature; and the two roadmap documents remain in place +for the post-implementation review. This stage is small enough for one final +verification prompt.