Prepare roadmap for documentation policy update

docs/roadmap/documentation.md

@@ -0,0 +1,355 @@
+# Documentation Policy And Structure Alignment
+
+## Status
+
+Proposed.
+
+## Purpose
+
+Align Narratio's documentation policy, organization, and maintenance practices
+with the current Notarius approach while preserving Narratio-specific product,
+operational, and integration needs.
+
+The resulting documentation set should assign each authoritative topic to one
+canonical owner, distinguish current behavior from future work, make audience
+boundaries explicit, and minimize duplicated contracts that can drift as the
+application changes.
+
+This roadmap defines the scope and intended final state. Execution guidance is
+maintained separately in the [Implementation Plan](implementation.md).
+
+## Motivation
+
+Narratio already has substantial user, operator, integration, and internal
+documentation, but its documentation policy predates the current Notarius
+policy. The existing policy prescribes document profiles and detailed outlines
+without defining ownership boundaries precisely enough. Several policy files
+also refer to obsolete or nonexistent paths, and some concrete inventories and
+contract details are repeated across documents.
+
+The alignment should retain useful Narratio documentation while adopting the
+stronger Notarius principles:
+
+- one canonical owner for each authoritative fact;
+- current behavior outside the roadmap and future behavior inside it;
+- audience-appropriate detail;
+- maintained, valid, secret-free examples;
+- explicit boundaries among user, operator, integration, architecture, and
+  internal documentation;
+- a defined lifecycle for architectural decision records; and
+- verification of documentation against implemented behavior.
+
+## Target Documentation Model
+
+### Canonical Ownership
+
+The final documentation policy should assign the following responsibilities.
+
+| Topic | Canonical owner | Intended responsibility |
+| --- | --- | --- |
+| Product orientation and minimal end-to-end quickstart | `README.md` | Explain what Narratio is, why it is useful, show the shortest successful invocation, and route readers onward. |
+| Contributor entry point | `docs/development.md` | Provide a task-oriented reading guide, baseline validation commands, and links to canonical policies and contracts. |
+| Current application architecture | `docs/policy/architecture.md` | Define system shape, normative ownership, dependency direction, boundaries, invariants, safety properties, and non-goals. |
+| Documentation organization | `docs/policy/documentation.md` | Define canonical ownership, audience boundaries, maintenance rules, and the ADR and document lifecycle. |
+| Testing policy | `docs/policy/testing.md` | Define test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, and test lifecycle decisions. |
+| CLI contract | `docs/cli.md` | Define commands, arguments, flags, invocation semantics, output conventions, and exit behavior. |
+| Configuration contract | `docs/config.md` | Define discovery, precedence, schemas, fields, defaults, environment overrides, and validation rules. |
+| Operations | `docs/operations.md` | Define runtime workflows, physical state layout, resume, cleanup, permissions, recovery, and operational limits. |
+| Troubleshooting | `docs/troubleshooting.md` | Provide symptom-driven diagnosis and safe remedies, linking to the owning CLI, configuration, operations, or integration contract. |
+| External and durable integration contracts | `docs/integrations/` | Define the external formats, protocols, logical artifact shapes, and compatibility behavior Narratio relies on. |
+| Implemented component inventory | `docs/internal/overview.md` | Inventory current packages and components, summarize their implemented responsibilities, and route readers to focused internal documents. |
+| Internal component behavior | Other files under `docs/internal/` | Explain implementation flow, internal collaborators, state transitions, package-local guarantees and failures, and relevant tests. |
+| Architectural decision history | `docs/adr/` | Record significant decisions, alternatives, rationale, consequences, and supersession history when ADRs are warranted. |
+| Future work and implementation status | `docs/roadmap/` | Describe proposed, accepted, deferred, rejected, or completed work and its implementation status. |
+| Complete copyable artifacts | `examples/` | Hold maintained configuration, inputs, and other files intended to be copied or run. |
+
+Documents for interfaces that Narratio does not expose should not be created as
+placeholders. In particular, public API or consumer documentation should be
+added only if a corresponding public interface exists.
+
+### Structural Parity With Notarius
+
+Narratio should follow the current Notarius policy layout where the same
+responsibility exists. In the aligned structure:
+
+- `docs/internal/overview.md` is the internal component inventory;
+- `docs/development.md` is the first-read contributor landing page;
+- all navigation and relative links use those canonical paths;
+- `docs/adr/` and sequentially numbered ADR filenames are used when architectural
+  decisions need durable records; and
+- Narratio-specific canonical documents remain, including the troubleshooting
+  guide, stage documentation, and integration contracts.
+
+Structural parity does not require placeholder documents, removal of useful
+Narratio-specific material, or identical prose where the applications have
+different contracts.
+
+## Policy Alignment
+
+### Documentation Policy
+
+The documentation policy should follow the current Notarius policy, adapted to
+Narratio. It should:
+
+- define one canonical owner for every contract or authoritative fact;
+- identify volatile details that must not be maintained in multiple places;
+- permit non-owning documents to provide only short, stable summaries with
+  links;
+- describe implemented behavior only outside `docs/roadmap/`;
+- allow accepted ADRs to precede implementation without presenting the
+  decision as implemented behavior;
+- distinguish user, operator, contributor, integration, and internal detail;
+- require complete copyable files to live under `examples/`;
+- define documentation security and privacy requirements;
+- specify ownership boundaries for orientation, CLI, configuration,
+  operations, troubleshooting, integrations, architecture, and internals;
+- define the ADR format and supersession rules; and
+- require contract, example, link, and sensitive-data checks when behavior or
+  documentation changes.
+
+### Architecture Policy
+
+The architecture policy should remain the normative owner of Narratio's system
+boundaries and invariants rather than a concrete package inventory or secondary
+testing and documentation policy. It should:
+
+- preserve Narratio's explicit, stage-driven orchestration model;
+- preserve adapter, manifest, artifact, path-safety, publish-commit, security,
+  privacy, and determinism invariants;
+- link to the testing and documentation policies for their general rules;
+- link to internal documentation for implemented component mechanics;
+- link to external contracts rather than redefining them; and
+- use the canonical `docs/policy/` paths consistently.
+
+### Testing Policy
+
+The copied Notarius testing policy is Narratio's canonical testing policy. Its
+place in the documentation set requires:
+
+- linking it from architecture and contributor guidance;
+- removing or reducing duplicated general test philosophy elsewhere;
+- retaining subsystem-specific test guidance only where it helps maintain a
+  concrete contract; and
+- ensuring project commands and examples remain accurate for Narratio.
+
+The alignment should not require Narratio and Notarius to have identical test
+suites. The shared policy governs how Narratio evaluates test value,
+boundaries, doubles, regression protection, coverage, and sufficiency.
+
+### Contributor Policy
+
+The development document should be a concise contributor entry point. It should
+route maintainers and coding agents to canonical documentation based on the
+task at hand and provide the minimum repository orientation and validation
+commands needed to begin work.
+
+Detailed package inventories belong in `docs/internal/overview.md`;
+architecture rules belong in the architecture policy; application contracts
+belong in their user, operator, or integration documents; and general testing
+rules belong in the testing policy. Any retained change recipes should avoid
+redefining those owners.
+
+## Documentation Set Alignment
+
+### README And Navigation
+
+Keep the README short and outward-facing. It should own product orientation and
+one minimal successful workflow, then link to the CLI, configuration,
+operations, troubleshooting, contributor, architecture, testing, integration,
+and internal entry points as appropriate.
+
+Navigation should use the final canonical paths and should not rely on obsolete
+aliases or duplicate index files.
+
+### CLI, Configuration, And Operations
+
+Separate these contracts consistently:
+
+- CLI documentation answers how Narratio is invoked and what its command-line
+  interface means;
+- configuration documentation answers how configuration is discovered,
+  interpreted, defaulted, overridden, and validated; and
+- operations documentation answers what happens to runtime state and how an
+  operator runs, resumes, cleans, diagnoses, or recovers the application.
+
+Cross-cutting workflows should have one task-oriented owner and link to the
+other contracts rather than copying their flags, fields, defaults, or path
+definitions.
+
+### Troubleshooting
+
+Keep troubleshooting as a Narratio-specific canonical document. Each entry
+should begin from an observable symptom and provide a likely cause, a safe
+diagnostic step, a safe remedy, and links to the canonical contract or
+operational procedure.
+
+Troubleshooting should not become a second CLI, configuration, or operations
+reference.
+
+### Integrations
+
+Integration documents should own the externally observable contracts Narratio
+uses: subprocess behavior, file formats, protocols, logical artifact paths and
+schemas, compatibility expectations, and upstream or downstream
+responsibilities.
+
+They should describe only the portions of WhisperX, Seriatim, Audita,
+Scriptorium, object storage, or future integrations that Narratio actually
+depends on. Internal adapter mechanics belong under `docs/internal/`, while
+configuration defaults, CLI syntax, and physical runtime placement remain with
+their respective owners.
+
+### Internal Documentation
+
+`docs/internal/overview.md` is the canonical implemented component inventory. It
+should summarize current components and route readers to focused documents
+without restating normative architecture.
+
+Focused internal documents should describe implemented behavior at useful
+component boundaries, including collaborators, data and state transitions,
+failures, and tests worth consulting. They may identify an external field,
+file, or protocol when explaining a dependency, but should link to the
+canonical contract for its definition.
+
+The internal set should be reviewed for duplicate definitions of:
+
+- stage order and stage contracts;
+- manifest states and transitions;
+- artifact identities, paths, and schemas;
+- workspace and object-storage layout;
+- restore, resume, cleanup, and publish behavior;
+- adapter invocation and compatibility rules; and
+- configuration fields and defaults.
+
+### Examples
+
+Complete copyable configuration and input files should remain under
+`examples/`. Documentation may include small illustrative fragments, but it
+should link to maintained examples instead of embedding alternate complete
+files.
+
+Examples should remain secret-free, loadable, valid, and covered by automated
+tests where practical. Their commands, filenames, fields, defaults, and
+templates must agree with implemented Narratio behavior.
+
+### Roadmaps And ADRs
+
+Roadmaps should own future work, implementation status, and sequencing. They
+must not be treated as current behavior references. When roadmap work lands,
+the relevant current-behavior documents should be updated in the same change,
+and the roadmap status should accurately reflect completion.
+
+Use ADRs for significant architectural decisions whose context, alternatives,
+and consequences should remain durable. Use the lightweight Nygard structure:
+title, status, date, context, decision, alternatives considered, and
+consequences. Accepted decision content is immutable; a later change should
+supersede it with a new ADR. Rejected architectural alternatives belong in the
+ADR, while rejected product ideas remain roadmap material.
+
+No empty `docs/adr/` directory or retrospective ADR catalog is required merely
+to satisfy structural parity.
+
+## Canonical-Ownership Audit
+
+The existing documentation should be audited for repeated authoritative facts,
+with priority given to details most likely to drift:
+
+- commands, arguments, flags, output conventions, and exit behavior;
+- configuration discovery, fields, defaults, environment variables, and
+  validation rules;
+- stage order, prerequisites, invalidation, skip, force, resume, and failure
+  semantics;
+- manifest states, artifact identities, and publish commit behavior;
+- local workspace, cache, spool, log, report, and remote object paths;
+- integration arguments, formats, schemas, timeouts, and compatibility rules;
+  and
+- security, credential, permission, and sensitive-artifact handling.
+
+For each repeated fact, choose the canonical owner defined by policy. Remove
+the duplicate definition or reduce it to the smallest stable summary needed
+for orientation, with a link to the owner.
+
+This audit should preserve useful task-oriented guidance. Canonical ownership
+means eliminating parallel contract definitions, not forcing readers to
+assemble every workflow from isolated reference fragments.
+
+## Intended Final State
+
+When this roadmap is complete:
+
+- Narratio's documentation policy closely matches the current Notarius policy
+  in principles, organization, terminology, and lifecycle rules;
+- every authoritative documentation topic has one stated canonical owner;
+- `docs/internal/overview.md` is the internal component index and no references
+  to `docs/internal/README.md` remain;
+- all policy and navigation links use real canonical paths;
+- architecture, documentation, testing, and contributor policies have distinct
+  responsibilities and link to one another;
+- current-behavior documentation contains no unimplemented claims;
+- roadmap documents clearly own future behavior and implementation status;
+- user and operator documents avoid unnecessary implementation detail;
+- internal documents do not redefine external or user-facing contracts;
+- volatile commands, fields, defaults, schemas, paths, and guarantees are not
+  maintained authoritatively in multiple places;
+- complete examples live under `examples/` and remain valid and secret-free;
+- the documentation remains concise enough to navigate but complete enough for
+  users, operators, developers, integrators, and coding agents; and
+- future behavior changes can identify the documentation that must change by
+  consulting the ownership table.
+
+## Validation
+
+Completion should include:
+
+- a repository-wide review of Markdown links and canonical paths;
+- verification of documented commands and flags against the CLI
+  implementation;
+- verification of configuration fields, defaults, environment overrides, and
+  validation rules against the configuration implementation;
+- verification of stage, manifest, artifact, restore, resume, cleanup, and
+  publish claims against implemented behavior;
+- validation of maintained examples through existing runtime config paths and
+  tests;
+- `go test ./...`;
+- confirmation that non-owning documents summarize and link rather than
+  redefine volatile contracts;
+- confirmation that unimplemented behavior appears only under
+  `docs/roadmap/`, subject to the accepted-ADR exception; and
+- a review for credentials, private campaign content, sensitive environment
+  data, and private infrastructure details.
+
+If the repository has no automated Markdown link checker, validation should use
+a focused, reproducible scripted link check supplemented by manual review of
+directory links and anchors.
+
+## Acceptance Criteria
+
+- `docs/policy/documentation.md` expresses the Narratio-adapted canonical
+  ownership model and boundary rules from the current Notarius policy.
+- `docs/policy/architecture.md`, `docs/policy/testing.md`, and
+  `docs/development.md` have clear, non-overlapping responsibilities.
+- `docs/policy/development.md` has moved to `docs/development.md`, with all
+  inbound links updated.
+- `docs/internal/README.md` has been renamed to
+  `docs/internal/overview.md`, with all inbound links updated.
+- README, CLI, configuration, operations, troubleshooting, integration,
+  internal, roadmap, example, and ADR responsibilities match the ownership
+  model.
+- No known obsolete documentation paths remain.
+- High-volatility contracts have one authoritative definition.
+- Current and future behavior are clearly separated.
+- Maintained examples and documented commands agree with the implementation.
+- Documentation validation and the full Go test suite pass, or any unrelated
+  pre-existing failure is recorded precisely.
+
+## Non-Goals
+
+- Changing Narratio runtime behavior solely to make existing documentation
+  true.
+- Implementing features described by other roadmap documents.
+- Rewriting every document for stylistic uniformity when its ownership and
+  content are already correct.
+- Copying Notarius product, CLI, configuration, integration, or internal
+  contracts into Narratio.
+- Creating placeholder API, consumer, ADR, or integration documents for
+  interfaces that do not exist.