Plan deployment-managed cache permissions

docs/roadmap/deployment-managed-cache-permissions.md

@@ -0,0 +1,236 @@
+# Deployment-Managed Cache Permissions
+
+Status: proposed
+
+## Summary
+
+Stop treating cache access permissions as application policy. Notarius should
+create chunk-plan and checkpoint state through conventional Unix filesystem
+semantics and leave the effective owner, group, and access policy to the
+operator's provisioned directories, process umask, setgid inheritance, and
+default ACLs.
+
+Notarius continues to own safe paths, symlink rejection, atomic publication,
+cache validation, and recovery. It does not decide whether a configured cache
+is private to one account, shared by a trusted group, or readable under some
+other deployment policy.
+
+## Motivation And Current Problem
+
+The current cache implementations enforce a policy that was stronger than the
+application requires:
+
+- the chunk-plan store creates and repeatedly converges cache directories to
+  `0700` and atomic files to `0600`; and
+- the checkpoint recorder supplies `0700` directory and `0600` file modes to
+  the shared confined writer, which explicitly applies the file mode during
+  atomic publication.
+
+This prevents an administrator from using ordinary Unix provisioning to share
+one cache between a service account and trusted operator accounts. Adding a
+second Notarius-specific `private` or `group` configuration policy would solve
+that immediate symptom, but it would preserve the underlying ownership error:
+Notarius would still decide access policy instead of respecting the deployment.
+
+Routine Notarius operation does not require an application-enforced private
+cache. Cache records can contain source-derived data, so operators must be told
+what they are storing, but sensitivity does not imply that Notarius should
+override a deliberately configured filesystem policy. API credentials remain
+outside cache payloads and continue to follow their existing configuration and
+redaction boundaries.
+
+## Target Policy
+
+The target contract is:
+
+> Notarius creates cache entries with conventional maximum creation modes and
+> allows the operating system to determine effective access through the
+> process umask, inherited group ownership, and default ACLs.
+
+Concretely:
+
+- newly created cache directories request `0777`;
+- newly created cache files, including atomic temporary files, request `0666`;
+- the kernel applies the invoking process's umask and any applicable default
+  ACL;
+- setgid directories provisioned by the operator control inherited group
+  ownership and, on supported Unix systems, setgid propagation to descendants;
+- Notarius does not call `chmod`, `chown`, or `chgrp` to impose or repair a
+  cache access policy; and
+- Notarius does not reject a cache merely because its effective modes are more
+  or less permissive than a Notarius-selected default.
+
+This is policy inheritance, not literal copying of a parent directory's mode
+bits. Directories and non-executable data files appropriately have different
+maximum modes. A parent mode alone also does not determine child modes: an
+operator who needs a particular result must configure the service umask and,
+where appropriate, a setgid parent or default ACL.
+
+For example, a service started with umask `0077` ordinarily creates `0700`
+directories and `0600` files. A trusted-group deployment can provision a
+setgid cache root and use umask `0007`, or an equivalent default ACL, to obtain
+`0770` directories and `0660` files. These are deployment choices rather than
+configuration values interpreted by Notarius.
+
+## Public Contract
+
+No new Notarius configuration field, environment variable, or CLI flag is
+introduced. In particular, do not add the previously considered
+`cache.access: private|group` setting.
+
+Existing cache enablement, locations, chunk-plan modes, resume controls,
+identity, and compatibility rules remain unchanged. The behavioral change is
+that effective permissions on newly created cache entries are controlled by
+the deployment environment rather than forcibly converged by Notarius.
+
+The policy applies uniformly to both cache families:
+
+- canonical chunk plans; and
+- invocation checkpoints.
+
+Output and debug storage are outside this feature's scope. Their current
+contracts must not change incidentally through a shared helper refactor. A
+separate review may later consider whether the same deployment-owned policy is
+appropriate for those surfaces.
+
+## Filesystem And Atomic-Publication Requirements
+
+Permission deference must cover every cache write path, not only final files:
+
+- configured cache-family roots created by Notarius;
+- chunk-plan source-digest directories;
+- checkpoint identity and stage directories;
+- atomic temporary files; and
+- final files published by rename.
+
+Atomic publication must not bypass the deployment policy. Go's
+`os.CreateTemp` creates a file with `0600`, and applying `chmod(0666)` afterward
+would bypass the process umask. Cache writers therefore need an atomic
+temporary-file creation path that chooses a collision-resistant name and opens
+it with `O_CREATE|O_EXCL` and requested mode `0666`, allowing the kernel to
+apply umask and default ACL policy at creation time.
+
+Replacement writes publish a newly created temporary inode. Its effective
+permissions and ACL are therefore those selected for new entries by the
+current parent and process policy; Notarius does not copy mode bits or ACLs
+from the replaced file. Operators who require stable replacement permissions
+must express that policy through a stable service umask and directory default
+ACL.
+
+Opening an existing cache for writing must not alter its root or descendant
+modes. Read-only lookup of a missing cache must remain side-effect free.
+
+## Preserved Safety And Correctness Boundaries
+
+Deferring access policy does not weaken Notarius's filesystem correctness
+rules. The implementation must preserve:
+
+- confined resolution beneath the selected cache root;
+- rejection of symlinked or unexpected cache entries;
+- validation of source digests, checkpoint identities, schemas, and recorded
+  provenance;
+- atomic replacement, existing synchronization behavior, cleanup of abandoned
+  temporary files, and complete-record visibility;
+- existing missing, malformed, incompatible, and operational-error recovery
+  classifications; and
+- separation of output, cache, and debug roots.
+
+An administrator's decision to permit multiple accounts to reach one cache
+does not create a new Notarius guarantee for simultaneous writers. Preserve
+the existing atomicity and validation behavior, but do not add locking or claim
+transactional coordination across concurrent processes as part of this work.
+Concurrent writes to the same multi-file checkpoint identity remain subject to
+the currently documented cache behavior; the implementation documentation
+must state any existing limitation accurately.
+
+## Architectural Boundaries
+
+The target removes application-owned cache modes without creating a general
+permission-policy framework. Chunk-plan and checkpoint storage may use
+different filesystem APIs internally, but both must implement the same
+deployment-owned result. Pipeline modules and the runner remain unaware of
+physical roots and filesystem access policy.
+
+Any shared writer API must distinguish an application-enforced exact mode from
+an umask-filtered maximum creation mode. The cache change must not silently
+alter output or debug behavior merely because checkpoints currently share
+filesystem helpers with those surfaces.
+
+## Existing Cache Migration
+
+Stopping permission convergence does not change existing inode modes. A cache
+tree created by an older Notarius version may therefore remain `0700`/`0600`
+and inaccessible to another account after upgrade.
+
+Operators who want a different policy must either:
+
+- remove the reconstructible cache and allow it to be recreated beneath a
+  correctly provisioned parent and service environment; or
+- stop writers and deliberately update ownership, modes, and ACLs using normal
+  system-administration tools.
+
+Notarius must not perform a recursive startup migration. The same rule applies
+when an operator wants to make a previously shared cache private: change the
+deployment policy and deliberately migrate or recreate existing state when an
+immediate whole-tree change is required.
+
+## Documentation And Decision Records
+
+The final documentation set includes an accepted ADR titled along the lines of
+"Defer cache access policy to the deployment." The ADR records the durable
+ownership boundary so later maintenance does not reintroduce
+application-enforced modes solely because cache data can contain source-derived
+content.
+
+The new ADR need not supersede ADR-0005 or ADR-0007: those decisions govern
+canonical chunk-plan reuse and checkpoint recording/reuse, while leaving exact
+storage permissions to operational contracts. It should reference those ADRs
+and the output/cache/debug separation established by ADR-0006 and refined by
+ADR-0007.
+
+In the final current-behavior documentation, `docs/operations.md` owns cache
+provisioning, umask, setgid, default ACL, migration, sensitivity, and
+concurrent-use guidance. Architecture owns the deployment/application boundary;
+the configuration reference clarifies that selecting a cache path does not
+select its permissions; and the internal state guide explains the implemented
+write mechanics. The release that first contains the feature summarizes the
+operational change without becoming its canonical reference.
+
+Current-behavior documentation must not describe this target behavior before
+the implementation lands.
+
+## Verification Expectations
+
+Behavioral tests protect kernel-selected permissions for both cache families,
+non-mutation of existing roots, atomic replacement, and representative
+assembled CLI wiring. Existing confinement, symlink, validation, identity,
+reuse, and recovery coverage remains authoritative. Umask tests are POSIX-only,
+serialized, and restore process-global state. The default suite does not create
+privileged accounts or groups; setgid and default-ACL examples may be verified
+in a purpose-built Linux deployment environment.
+
+## Acceptance Criteria
+
+- No cache access-mode field or application-specific private/group policy is
+  added.
+- Both chunk-plan and checkpoint writers request `0777` for new directories
+  and `0666` for new files, including atomic temporary files.
+- Effective permissions reflect the process umask and applicable default ACLs
+  rather than a later Notarius chmod.
+- Existing cache roots and descendants are not chmod-converged during reads or
+  writes.
+- A private deployment can obtain private cache state through its filesystem
+  provisioning and umask without a Notarius-specific setting.
+- A trusted-group deployment can obtain shared cache state through setgid,
+  umask, and/or default ACL policy without a Notarius-specific setting.
+- Existing cache trees are not recursively migrated.
+- Chunk-plan and checkpoint enablement, lookup, recording, resume, identity,
+  validation, confinement, atomicity, and recovery semantics remain unchanged.
+- Output and debug permission behavior does not change as an incidental effect.
+- Notarius performs no user/group management, ownership mutation, ACL
+  management, privilege escalation, or cross-process locking.
+- The ADR and canonical operations, architecture, configuration, and internal
+  documentation accurately describe the implemented boundary; the next
+  release note summarizes the change when that release is prepared.
+- Focused tests and repository-wide `go test ./...`, `go vet ./...`, and
+  `go build ./cmd/notarius` checks pass.