Add ADR, roadmap, and staged implementation plan for filesystem surface refactoring

docs/adr/0006-separate-output-cache-and-debug-state.md

@@ -0,0 +1,114 @@
+# ADR-0006: Separate output, cache, and debug state
+
+**Status:** Proposed
+**Date:** 2026-07-17
+
+## Context
+
+Notarius currently exposes a workspace as a shared parent for checkpoints,
+debug artifacts, and preferred diagnostics settings. Diagnostics are a second
+inspection surface with their own enablement, directory, retention, and legacy
+configuration. Durable output uses a separate CLI-selected root, while the
+canonical chunk-plan cache introduced by ADR-0005 correctly uses an independent
+cache root.
+
+These concepts reflect implementation history more than operator intent. A user
+must understand differences among workspace state, diagnostics, debug artifacts,
+checkpoints, and chunk plans before deciding where Notarius may write. Some of
+those distinctions are important internally: a redacted run summary has a
+different sensitivity from a trace containing source material, prompts, and
+model responses. They do not require separate public filesystem categories.
+
+Notarius needs a smaller state model that communicates why data exists, how it
+may be treated, and whether it is reconstructible.
+
+## Decision
+
+Notarius exposes three filesystem surfaces: output, cache, and debug. The
+public workspace concept and diagnostics as a separate output surface are
+removed.
+
+### Output
+
+Output is the durable result of a run and the only surface intended for normal
+consumption. It contains the logical files produced by the output stage,
+including the maintained result, manifest, warning, and rejection contracts.
+Output is not cache or inspection state.
+
+### Cache
+
+Cache contains reconstructible state used to avoid repeated work or resume an
+interrupted workflow. Canonical chunk plans and invocation checkpoints are
+distinct cache families with independent identities, compatibility rules,
+enablement policies, locations, and cleanup lifecycles.
+
+ADR-0005 continues to govern canonical chunk-plan selection and reuse. Grouping
+chunk plans and checkpoints under the public cache category does not permit a
+checkpoint to compete with canonical plan reuse or couple their storage roots.
+
+Checkpointing is an invocation policy rather than a prerequisite hidden in
+persistent workspace configuration. An explicit resume invocation may read
+compatible checkpoints and record replacement checkpoint state for work it
+executes. Runs that do not request resume perform no checkpoint I/O.
+
+### Debug
+
+Debug is an explicitly requested per-run inspection bundle intended for
+developers and troubleshooting. It is off by default. When enabled, one bundle
+contains both redacted run summaries and detailed stage and LLM traces. The
+internal distinction between a safe summary and a sensitive trace remains, but
+there is one public enablement and location model.
+
+Debug data is never a cache input and has no automatic retention policy.
+Notarius does not create a debug directory unless debug is requested, and it
+does not automatically delete a requested bundle. Credentials remain redacted
+at every level, while the bundle as a whole is treated as potentially sensitive 
+because traces may contain source, reference, prompt, model-response, and 
+intermediate artifact content.
+
+Concise progress, warnings, and failures continue to use stdout or stderr. A
+run without debug may fail without producing a filesystem inspection record.
+
+Exact configuration fields, CLI flags, default paths, layouts, compatibility
+handling, and migration mechanics are configuration and operational contracts
+rather than part of this decision.
+
+## Alternatives considered
+
+- Keep workspace, diagnostics, checkpoints, debug, and chunk-plan cache as
+  separate public concepts. This preserves compatibility and the current safe
+  default-on failure records, but retains overlapping configuration and asks
+  operators to reason about implementation-specific categories.
+- Keep diagnostics as an always-available redacted operational surface and use
+  debug only for sensitive traces. This distinction is useful for a daemon or
+  managed service with an operational logging contract, but the current CLI can
+  report concise failures on stderr and provide inspection data when explicitly
+  requested.
+- Put all non-output state beneath one physical root. This minimizes path
+  configuration, but couples reconstructible caches to per-run inspection data
+  and couples cache families whose identity, sensitivity, and cleanup policies
+  differ.
+- Treat checkpoints as durable run state rather than cache. This emphasizes
+  resumability, but checkpoints are derived, compatibility-checked data that may
+  be deleted and recomputed. Cache more accurately describes their lifecycle.
+
+## Consequences
+
+The operator model becomes smaller: normal runs produce output and may use
+cache; developers explicitly request debug. Public configuration no longer
+exposes a workspace or overlapping diagnostics and debug systems.
+
+The implementation retains separate collaborators and serializers where their
+security or lifecycle boundaries differ. Redacted summaries remain useful as
+the index to a debug bundle, and chunk plans and checkpoints retain separate
+stores even though both are cache.
+
+Existing configuration, environment variables, flags, examples, and
+documentation require a deliberate compatibility transition. Default-on
+diagnostic directories disappear, so failures without debug are inspectable
+only through stderr and any durable output completed before the failure.
+
+Debug becomes easier to request and substantially more complete, but enabling
+it creates sensitive files that the operator must protect and remove. Cache
+cleanup is recoverable but may repeat expensive work, while deleting output is
+data loss from the user's perspective.