diff --git a/docs/roadmap/documentation.md b/docs/roadmap/documentation.md index a77b108..a8edfc7 100644 --- a/docs/roadmap/documentation.md +++ b/docs/roadmap/documentation.md @@ -1,199 +1,759 @@ # Documentation Roadmap -## Status +## Purpose -This document captures planned documentation decisions for Notarius. It records -policy choices while the application architecture is still being shaped. It does -not describe implemented behavior. +This roadmap defines the work required to bring Notarius documentation into +compliance with [`docs/policy/documentation.md`](../policy/documentation.md) and +the current implementation. It is a planning document only. Future implementation +agents should use it to rewrite documentation in stages without documenting +unimplemented behavior outside `docs/roadmap/`. -## Documentation Goals +## Repository Documentation Inventory -Notarius documentation should make three boundaries obvious: +- `README.md`: keep and rewrite. It still contains template text + (`go-application-template`) and does not describe the implemented Notarius CLI. +- `AGENTS.md`: keep and lightly update if needed. It correctly directs agents to + read `docs/policy`, and should stay short. +- `docs/policy/documentation.md`: keep and do not update. It is the controlling documentation policy. +- `docs/policy/architecture.md`: keep and do not update. It is the controlling architecture policy. +- `docs/integrations/seriatim.md`: keep and lightly update. It is close to the + current Seriatim adapter behavior and belongs in `docs/integrations/` because + Seriatim minimal transcript JSON is an external file-format contract. +- `docs/integrations/dnd-spells.md`: split. It mixes implemented extractor + internals, artifact payload details, and stale roadmap text claiming the CLI + `run` workflow does not exist. Move module-internal behavior to + `docs/internal/modules.md` or a module-specific internal doc; move durable + spell artifact output shape to an integration/artifact contract doc. +- `docs/roadmap/documentation.md`: keep during the documentation migration. After + the migration is complete, either delete it or reduce it to any remaining + documentation work only. +- `docs/roadmap/implementation.md`: delete after migration. It is a completed MVP + implementation plan and is no longer an active roadmap. +- `docs/roadmap/initial-architecture.md`: delete after migration. The implemented + architecture now belongs in policy and internal docs; historical planning + detail should not remain as active roadmap content. +- `docs/roadmap/mvp.md`: delete or rewrite after migration. It currently records + implemented MVP behavior and deferred documentation work; once current-behavior + docs exist, keep only genuine future work under `docs/roadmap/`. +- `examples/dnd-spells.config.yml`: keep and lightly update. It is a maintained + example covered by CLI tests, but should be checked for user-facing clarity and + secret-free LLM configuration. +- `examples/seriatim-minimal-transcript.json`: keep and lightly update. It is a + maintained Seriatim input fixture covered by CLI tests. -- source-format support belongs to input-stage modules; -- extraction-domain behavior belongs to extract-stage modules; -- core framework behavior is source-agnostic and domain-agnostic. +## Policy Compliance Assessment -Documentation should avoid making the MVP look more transcript-specific or -D&D-specific than the architecture intends. +Required documents missing under the documentation policy: -## Current Policy Decisions +- `docs/cli.md`: required because Notarius is a CLI application. +- `docs/config.md`: required because Notarius is config-driven. +- `docs/operations.md`: required because Notarius runs multi-step workflows, + writes durable artifacts, and maintains diagnostics directories. +- `docs/internal/`: required because Notarius is modular and orchestration-based. +- `docs/policy/development.md`: required because the project is maintained by + humans and LLM coding agents. -### Planned Work Stays In Roadmap Docs +Recommended documents that should be added: -Until code exists, planned behavior belongs under `docs/roadmap/`. +- `docs/troubleshooting.md`: recommended because implemented commands have common + failure modes: missing config, unknown pipeline, invalid lane selection, + invalid Seriatim input, LLM configuration failures, provider failures, output + write failures, and diagnostics retention behavior. +- Maintained examples under `examples/`: already present and should remain + linked from README, CLI, config, and integration docs. -Implemented behavior should later move into canonical docs. Roadmap files may -then link to those docs or be reduced to remaining future work. +Documents that exist but are stale or in the wrong canonical home: -### Core Docs Should Use Generic Terms +- `README.md` is stale template content. +- `docs/integrations/dnd-spells.md` includes stale limits and belongs partly in + internal/module docs and partly in artifact integration docs. +- Completed roadmap documents describe implemented behavior and should not be the + only source of current-behavior documentation after the migration. -Core architecture docs should prefer: +Content describing deprecated, historical, planned, or unimplemented behavior +outside `docs/roadmap/`: -- source document; -- source unit; -- source reference; -- input adapter; -- extractor; -- chunker; -- merger; -- normalizer; -- output encoder; -- artifact; -- validator; -- run manifest. +- `docs/integrations/dnd-spells.md` says the CLI `run` workflow does not exist; + that is now false. +- No other non-roadmap stale claims were found during this pass, but future + implementation agents should grep before rewriting. -Core docs should avoid transcript-specific terms such as segment, speaker, -timestamp, and transcript range unless discussing an input adapter or an example. +Examples: -Core docs should avoid D&D-specific terms such as spell, NPC, item, combat, and -encounter unless discussing extract modules, artifact docs, or examples. +- `examples/dnd-spells.config.yml` and + `examples/seriatim-minimal-transcript.json` are covered by + `internal/cli/run_test.go`. +- There is no maintained expected-output example under `examples/`. Do not add + one unless it can be validated or regenerated from implemented behavior. -### Input Module Docs Own Source Formats +Links likely needing verification: -Each implemented input-stage module should have a canonical integration -document. +- New README links to `docs/cli.md`, `docs/config.md`, + `docs/operations.md`, `docs/troubleshooting.md`, `docs/internal/`, and + `docs/integrations/` will need verification after those files are created. +- Existing roadmap links to old implementation documents should be removed or + deleted with the roadmap cleanup stage. -Likely future files: +Documents not applicable: + +- `docs/api.md` is not needed because Notarius does not implement a public HTTP + API. +- `docs/consumers/` is not needed for this pass because the repository exposes a + CLI, not a supported public package, SDK, plugin API, or HTTP consumer API. + +## Target Documentation Set + +Desired final tree after the migration: ```text -docs/integrations/seriatim-transcript.md -docs/integrations/markdown-source.md +AGENTS.md +README.md +docs/cli.md +docs/config.md +docs/operations.md +docs/troubleshooting.md +docs/policy/architecture.md +docs/policy/development.md +docs/policy/documentation.md +docs/internal/overview.md +docs/internal/pipeline.md +docs/internal/modules.md +docs/internal/llm.md +docs/internal/diagnostics.md +docs/integrations/seriatim.md +docs/integrations/openai-compatible.md +docs/integrations/json-output.md +docs/integrations/dnd-spell-artifacts.md +docs/roadmap/mvp.md +examples/dnd-spells.config.yml +examples/seriatim-minimal-transcript.json ``` -Input module docs should cover: +`docs/roadmap/mvp.md` should exist only if it contains genuine remaining future +work after the migration. `docs/roadmap/documentation.md` exists to drive this +migration and should be deleted or reduced when the migration is complete. -- accepted external schema or file shape; -- mapping into `SourceDocument` and `SourceUnit`; -- metadata preserved by the module; -- validation rules and failure behavior; -- examples. +### `AGENTS.md` -The Seriatim input module doc should reference the Seriatim schema it supports and -explain how transcript segment IDs become source-unit IDs. +- Audience: LLM coding agents. +- Purpose: shortest repository-specific agent instruction. +- Canonical scope: policy files that agents must read before changes. +- Recommended outline: one or two sentences linking to policy docs. +- Source-of-truth areas: `docs/policy/architecture.md`, + `docs/policy/documentation.md`. +- Acceptance criteria: stays short; no duplicated architecture or documentation + policy; links remain accurate. -### Stage Module Docs Own Business Logic +### `README.md` -Each implemented stage-module family should have canonical internal or -integration docs. +- Audience: users, administrators, operators. +- Purpose: concise outward-facing orientation. +- Canonical scope: project purpose, shortest useful command, and links to + targeted docs. +- Recommended outline: description; elevator pitch; shortest useful command + using maintained examples; where output appears; links. +- Source-of-truth areas: `cmd/notarius/main.go`, `internal/cli/run.go`, + `examples/dnd-spells.config.yml`, + `examples/seriatim-minimal-transcript.json`, + `internal/cli/run_test.go`. +- Acceptance criteria: no template text; no long manual content; commands match + real CLI syntax; links resolve; no unimplemented features. -Likely future files: +### `docs/policy/architecture.md` -```text -docs/internal/stage-modules.md -docs/integrations/artifacts-dnd.md -``` +- Audience: developers, LLM coding agents. +- Purpose: durable development architecture policy. +- Canonical scope: stable architecture invariants and boundaries. +- Recommended outline: keep the current concise policy shape. +- Source-of-truth areas: current architecture policy and actual package + boundaries. +- Acceptance criteria: does not grow into CLI/config/reference documentation; + no planned behavior outside roadmap language. -Stage module docs should cover: +### `docs/policy/documentation.md` -- module key; -- stage; -- artifact type; -- schema version; -- required source-reference behavior; -- validator chain; -- prompt and response-schema ownership; -- examples. +- Audience: developers, LLM coding agents. +- Purpose: controlling documentation policy. +- Canonical scope: documentation layout, audience, maintenance rules, and + canonical homes. +- Recommended outline: keep current policy unless project-wide documentation + rules change. +- Source-of-truth areas: current documentation policy and this migration. +- Acceptance criteria: remains policy-oriented; no project-specific + current-behavior reference material that belongs elsewhere. -D&D concepts should be documented in D&D extract-module or artifact docs, not in -generic runner or framework docs. +### `docs/cli.md` -### CLI Docs Should Reflect Extensibility +- Audience: users, administrators, operators. +- Purpose: canonical CLI reference. +- Canonical scope: implemented commands, flags, exit behavior, common workflows, + and command output. +- Recommended outline: shortest useful command; command overview; `help`; + `run`; `config validate`; `pipelines list`; exit codes; common workflows; + diagnostics/output pointers. +- Source-of-truth areas: `internal/cli/run.go`, `internal/cli/run_test.go`, + `internal/cli/catalog.go`. +- Acceptance criteria: documents only implemented commands and flags; explains + `--config`, `--input`, `--only`, `--output-dir`, `--diagnostics-dir`, + `--llm-profile`, `--pipeline`, and `--json`; does not duplicate full config + reference; examples are covered by tests or are direct variants of tested + commands. -The CLI reference should present named pipeline profiles as the primary -user-facing abstraction. Individual stage modules should be visible through -pipeline configuration and discovery commands, not through ad hoc structural -run flags. +### `docs/config.md` -Provisional command shape: +- Audience: administrators, operators, advanced users. +- Purpose: canonical configuration reference. +- Canonical scope: config discovery, YAML version 1 structure, defaults, + environment overrides, pipeline profiles, module bindings, LLM profiles, + diagnostics settings, validation behavior, and secrets handling. +- Recommended outline: config discovery and precedence; minimal working config; + production-oriented config; `version`; `llm_profiles`; `pipelines`; + module-binding shorthand and object form; `concurrency`; `diagnostics`; + environment overrides; validation and capability checks; examples. +- Source-of-truth areas: `internal/core/config/*.go`, + `internal/framework/pipeline/profile.go`, `internal/framework/pipeline/module.go`, + `internal/cli/run.go`, `examples/dnd-spells.config.yml`, config tests. +- Acceptance criteria: full config reference is here and not duplicated + elsewhere; lists implemented defaults and supported env vars; explains that + raw secrets should not be committed; examples load successfully. + +### `docs/operations.md` + +- Audience: administrators, operators. +- Purpose: canonical operational behavior and recovery reference. +- Canonical scope: run output directory layout, diagnostics directory layout, + retention behavior, manifests, warnings, failure inspection, safe cleanup, and + operational caveats. +- Recommended outline: normal run workflow; output files; diagnostics files; + retention modes; failure behavior; warnings; atomic writes and path safety; + safe cleanup; what is not resumable yet. +- Source-of-truth areas: `internal/cli/run.go`, + `internal/core/diagnostics/*.go`, `internal/modules/output/json/encoder.go`, + output and diagnostics tests. +- Acceptance criteria: accurately names implemented durable files; states that + resume/retry is not implemented as a command; explains when diagnostics are + retained; does not describe unimplemented remote storage or archives. + +### `docs/troubleshooting.md` + +- Audience: users, administrators, operators. +- Purpose: task-oriented fixes for recurring failure modes. +- Canonical scope: symptoms, likely causes, inspection steps, and safe fixes. +- Recommended outline: config not found; unsupported config version; unknown + pipeline; invalid `--only`; Seriatim validation failure; missing LLM base URL + or model; provider HTTP failures; output write failure; diagnostics retention + surprise. +- Source-of-truth areas: `internal/cli/run.go`, `internal/cli/run_test.go`, + `internal/core/config/*_test.go`, + `internal/modules/input/seriatim/*_test.go`, + `internal/framework/llm/*_test.go`. +- Acceptance criteria: every entry maps to implemented behavior; no speculative + troubleshooting for future modules; links to CLI/config/operations docs + instead of duplicating reference material. + +### `docs/policy/development.md` + +- Audience: developers, LLM coding agents. +- Purpose: contributor workflow and change discipline. +- Canonical scope: repository layout, build/test commands, coding conventions, + dependency policy, how to add config fields, CLI flags, modules, adapters, and + examples. +- Recommended outline: repository layout; required policy reads; build/test + commands; Go conventions; dependency policy; adding CLI flags; adding config; + adding stage modules; adding examples; docs update rules. +- Source-of-truth areas: `AGENTS.md`, `go.mod`, `internal/cli`, + `internal/core/config`, `internal/framework/pipeline`, `internal/modules`, + `docs/policy`. +- Acceptance criteria: procedural and concise; links to `architecture.md` for + invariants; does not restate full CLI/config reference. + +### `docs/internal/overview.md` + +- Audience: developers, LLM coding agents. +- Purpose: high-level implemented internals map. +- Canonical scope: package roles and import/boundary expectations. +- Recommended outline: executable and CLI; core packages; framework packages; + module packages; tests and fixtures; where not to put domain/source-specific + logic. +- Source-of-truth areas: repository tree, `docs/policy/architecture.md`, + `internal/cli/catalog.go`. +- Acceptance criteria: concise package map; reflects current code; does not + become a generated file listing every source file. + +### `docs/internal/pipeline.md` + +- Audience: developers, LLM coding agents. +- Purpose: implemented pipeline orchestration reference. +- Canonical scope: runner inputs/outputs, stage ordering, registries, resolved + profiles, artifact lanes, validation, warnings, and manifest population. +- Recommended outline: stage flow; profile resolution; registries and module + specs; runner execution; validator behavior; output encoder handoff; failure + behavior; tests. +- Source-of-truth areas: `internal/framework/contracts/contracts.go`, + `internal/framework/pipeline/*.go`, pipeline tests. +- Acceptance criteria: explains implemented serial execution; does not promise + parallel execution or resume; links to config docs for profile syntax. + +### `docs/internal/modules.md` + +- Audience: developers, LLM coding agents. +- Purpose: implemented stage modules and how to add new ones. +- Canonical scope: current production modules and their boundaries. +- Recommended outline: module contract pattern; `seriatim`; `generic`; + `dnd/spells`; `appendorder`; `noop`; `json`; module registration; tests to + add for new modules. +- Source-of-truth areas: `internal/modules/**`, `internal/cli/catalog.go`, + module tests. +- Acceptance criteria: documents only implemented modules; moves relevant + non-external content from `docs/integrations/dnd-spells.md`; keeps detailed + external file-format contracts in `docs/integrations/`. + +### `docs/internal/llm.md` + +- Audience: developers, LLM coding agents. +- Purpose: implemented LLM runtime internals. +- Canonical scope: structured LLM client contract, OpenAI-compatible adapter, + scheduler, schema registry, retry/redaction behavior, and prompt boundary. +- Recommended outline: contract; provider request shape at a high level; + scheduler; retries and timeouts; structured output decoding; secret redaction; + tests. +- Source-of-truth areas: `internal/framework/contracts/contracts.go`, + `internal/framework/llm/*.go`, `internal/framework/prompt/*.go`, + `internal/modules/extract/dnd/spells/assets`. +- Acceptance criteria: no provider marketing claims; no API keys; links to + OpenAI-compatible integration doc for the external wire contract. + +### `docs/internal/diagnostics.md` + +- Audience: developers, LLM coding agents. +- Purpose: implemented diagnostics internals. +- Canonical scope: run directory creation, artifact writers, retention decision + mechanics, and redacted effective config payloads. +- Recommended outline: purpose; inputs and outputs; files written; retention; + failure behavior; tests; invariants. +- Source-of-truth areas: `internal/core/diagnostics/*.go`, + `internal/core/config/redaction.go`, CLI diagnostics tests. +- Acceptance criteria: complements `docs/operations.md`; does not duplicate + operator procedures. + +### `docs/integrations/seriatim.md` + +- Audience: developers, LLM coding agents. +- Purpose: Seriatim minimal transcript input contract. +- Canonical scope: accepted JSON shape, validation, source mapping, metadata + keys, capabilities, and compatibility limits. +- Recommended outline: adapter; accepted shape; validation; source mapping; + metadata keys; capabilities; example; limits. +- Source-of-truth areas: `internal/modules/input/seriatim/*.go`, + `internal/modules/input/seriatim/*_test.go`, + `examples/seriatim-minimal-transcript.json`. +- Acceptance criteria: matches implemented validation exactly; clearly states + only the minimal Seriatim shape is supported. + +### `docs/integrations/openai-compatible.md` + +- Audience: administrators, developers, LLM coding agents. +- Purpose: external LLM provider wire contract currently used by Notarius. +- Canonical scope: OpenAI-compatible chat completions endpoint assumptions, + structured output response format, request/response fields used, retry + behavior, and secret handling. +- Recommended outline: provider type; base URL handling; endpoint path; + request shape summary; response assumptions; retryable statuses; auth header; + configuration links; limits. +- Source-of-truth areas: `internal/framework/llm/openai_compatible_client.go`, + `internal/framework/llm/openai_compatible_client_test.go`, + `internal/core/config/effective_config.go`. +- Acceptance criteria: documents only what the client implements; avoids + copying external API docs wholesale; links to config docs for settings. + +### `docs/integrations/json-output.md` + +- Audience: users, administrators, downstream developers, LLM coding agents. +- Purpose: durable JSON output file-format contract. +- Canonical scope: output directory contents and JSON file shapes produced by + the `json` output module. +- Recommended outline: output encoder; file list; `index.json`; + `manifest.json`; `artifacts/.json`; `rejected.json`; `warnings.json`; + artifact envelope; filename sanitization; limits. +- Source-of-truth areas: `internal/modules/output/json/encoder.go`, + `internal/core/artifacts/artifacts.go`, output tests, + `internal/cli/run_test.go`. +- Acceptance criteria: exactly matches implemented file names and shapes; does + not document future output formats. + +### `docs/integrations/dnd-spell-artifacts.md` + +- Audience: downstream developers, LLM coding agents. +- Purpose: D&D spell-cast artifact payload contract. +- Canonical scope: durable `dnd.spell_cast` payload fields, schema version, + source-reference expectations, prompt/schema provenance in manifests, and + validator rejection reasons. +- Recommended outline: artifact type; payload fields; source references; + validators; manifest metadata; example payload; limits. +- Source-of-truth areas: `internal/modules/extract/dnd/spells/*.go`, + `internal/modules/extract/dnd/spells/assets/schemas/dnd_spells.v1.json`, + spell extractor tests. +- Acceptance criteria: replaces the external-contract parts of + `docs/integrations/dnd-spells.md`; no stale statement that CLI run is missing; + no item/NPC/combat claims outside limitations. + +### `docs/roadmap/mvp.md` + +- Audience: maintainers, developers, LLM coding agents. +- Purpose: remaining roadmap work only, if any remains after the documentation + migration. +- Canonical scope: future, planned, deferred, or unimplemented work. +- Recommended outline: remaining future work; deferred ideas; release checklist, + if still useful. +- Source-of-truth areas: current code and this documentation roadmap. +- Acceptance criteria: contains no current-behavior reference material that + belongs in README, CLI, config, operations, internal, or integration docs. + +### `examples/dnd-spells.config.yml` + +- Audience: users, administrators, operators, LLM coding agents. +- Purpose: maintained minimal config example for the implemented spell + extraction workflow. +- Canonical scope: one valid, secret-free config example. +- Recommended outline: YAML fixture only, with comments only if they do not + interfere with tests. +- Source-of-truth areas: config parser/tests and CLI example-fixture tests. +- Acceptance criteria: remains loadable by tests; contains no secrets; is linked + from README, CLI, and config docs. + +### `examples/seriatim-minimal-transcript.json` + +- Audience: users, developers, LLM coding agents. +- Purpose: maintained minimal input example for the Seriatim adapter. +- Canonical scope: one valid minimal transcript JSON fixture. +- Recommended outline: JSON fixture only. +- Source-of-truth areas: Seriatim adapter/tests and CLI example-fixture tests. +- Acceptance criteria: remains valid; contains no private transcript data; is + linked from README, CLI, and Seriatim integration docs. + +## File-by-File Rewrite Guidance + +- `README.md`: cover what Notarius does, the shortest useful command, where + outputs go, and links. Avoid package diagrams, full config reference, and + roadmap content. Inspect `internal/cli/run.go`, `examples/`, and CLI tests. + Do not carry forward `go-application-template`. +- `docs/cli.md`: cover implemented commands and flags. Avoid full YAML field + reference and internal package details. Link to `docs/config.md` for config + fields and `docs/operations.md` for output/diagnostics. +- `docs/config.md`: cover config schema, defaults, discovery, env overrides, + validation, and secrets. Avoid command walkthroughs except minimal examples. + Link to `docs/cli.md` for command syntax and integration docs for module + contracts. +- `docs/operations.md`: cover output and diagnostics behavior, retention, and + failure inspection. Avoid developer implementation details that belong in + `docs/internal/diagnostics.md`. +- `docs/troubleshooting.md`: cover recurring implemented failures. Avoid generic + LLM advice not grounded in Notarius errors or tests. +- `docs/policy/development.md`: cover contributor workflow. Avoid copying + `docs/policy/architecture.md`; link to it for invariants. +- `docs/internal/overview.md`: cover package roles and boundaries. Avoid + user-facing command details. +- `docs/internal/pipeline.md`: cover runner and registry mechanics. Avoid config + syntax beyond linking to the canonical config doc. +- `docs/internal/modules.md`: cover implemented modules and how they fit stage + contracts. Avoid external Seriatim schema detail and durable JSON output + schemas beyond links to integration docs. +- `docs/internal/llm.md`: cover runtime internals. Avoid claiming support for + non-implemented providers. +- `docs/internal/diagnostics.md`: cover diagnostics implementation. Avoid + operator runbooks; link to operations. +- `docs/integrations/seriatim.md`: keep focused on Seriatim minimal transcript + JSON. Avoid D&D extractor behavior. +- `docs/integrations/openai-compatible.md`: document the implemented external + LLM contract. Avoid extensive copied provider docs. +- `docs/integrations/json-output.md`: document durable output files. Avoid + diagnostics run-directory details except a link to operations. +- `docs/integrations/dnd-spell-artifacts.md`: document the spell artifact + contract. Avoid extractor implementation details better covered internally. +- `docs/integrations/dnd-spells.md`: replace by split docs above, then delete or + leave a temporary redirect-style note only if that is useful during migration. +- `docs/roadmap/*.md`: remove completed implementation history or rewrite it as + future work. Do not leave implemented behavior solely in roadmap docs. + +## Examples Plan + +`examples/` exists and should remain because Notarius has non-trivial +configuration and workflows. + +- `examples/dnd-spells.config.yml` + - Purpose: maintained minimal D&D spell extraction pipeline config. + - Expected validity check: `go test ./internal/cli` currently validates this + fixture with `config validate`, `pipelines list`, and fake-LLM `run`. + - Documentation links: README, `docs/cli.md`, `docs/config.md`, + `docs/integrations/seriatim.md`, `docs/integrations/dnd-spell-artifacts.md`. + - Update guidance: keep secret-free; prefer environment-based API key + behavior; preserve test coverage. + +- `examples/seriatim-minimal-transcript.json` + - Purpose: maintained minimal Seriatim transcript input. + - Expected validity check: `go test ./internal/cli` uses it in the fake-LLM run + acceptance tests; Seriatim adapter tests cover compatible shapes separately. + - Documentation links: README, `docs/cli.md`, + `docs/integrations/seriatim.md`. + +Do not add an expected-output example unless it can be kept in sync by tests or +a small documented regeneration procedure. Do not invent examples for NPC, item, +combat, Markdown, or Obsidian extraction until those features exist. + +## Internal Documentation Plan + +### Pipeline Orchestration + +- Path: `docs/internal/pipeline.md` +- Purpose: explain implemented runner orchestration. +- Inputs and outputs: `pipeline.RunInput`, `pipeline.RunOutput`, + `contracts.*Request`, `contracts.*Result`, `artifacts.RunManifest`. +- Boundaries: orchestration and registries only; no source-format or D&D logic. +- Config fields used: resolved pipeline bindings and lane selection from + `internal/core/config`. +- Adapters used: input, chunk, extract, merge, normalize, validator, output + registries. +- Failure behavior: stage errors wrap module keys/lane/chunk context; partial + manifest may be returned on failure. +- Tests to inspect before changing: `internal/framework/pipeline/*_test.go`, + `internal/framework/pipeline/walking_skeleton_test.go`, + `internal/cli/run_test.go`. +- Architectural invariants: fixed six-stage flow; generic source references; + module selection via registries. + +### Stage Modules + +- Path: `docs/internal/modules.md` +- Purpose: summarize implemented production modules and extension pattern. +- Inputs and outputs: module-specific contract methods under + `internal/framework/contracts`. +- Boundaries: module logic under `internal/modules//...`; CLI only wires + production modules. +- Config fields used: module bindings, `llm_profile`, and `options`. +- Adapters used: Seriatim input, generic chunker, D&D spells extractor, + append-order merger, noop normalizer, JSON output encoder. +- Failure behavior: module constructors and stage methods return contextual + errors; validators reject candidates rather than panicking. +- Tests to inspect before changing: module package tests and + `internal/cli/catalog.go` coverage in `internal/cli/run_test.go`. +- Architectural invariants: source/domain concerns stay in modules; framework + remains source- and domain-agnostic. + +### LLM Runtime + +- Path: `docs/internal/llm.md` +- Purpose: explain structured LLM execution internals. +- Inputs and outputs: `StructuredCompletionRequest`, + `StructuredCompletionResponse`, schema registry entries, scheduler calls. +- Boundaries: provider wire details stay in `internal/framework/llm`; prompts + and schemas are owned by modules or generic prompt helpers. +- Config fields used: `llm_profiles`, timeout, retries, concurrency, API key + environment handling. +- Adapters used: OpenAI-compatible HTTP client and scheduler. +- Failure behavior: validation errors before calls; retryable provider errors; + redacted error messages. +- Tests to inspect before changing: `internal/framework/llm/*_test.go`, + `internal/framework/prompt/*_test.go`, spell prompt/schema tests. +- Architectural invariants: no secrets in errors/diagnostics; no domain prompt + logic in provider adapters. + +### Diagnostics + +- Path: `docs/internal/diagnostics.md` +- Purpose: explain diagnostics implementation. +- Inputs and outputs: invocation metadata, redacted effective config, resolved + pipeline, run manifest, warnings, run report, error log. +- Boundaries: diagnostics writes inspection artifacts; durable user output is + written by CLI/output modules. +- Config fields used: diagnostics work dir and retention mode. +- Adapters used: none external. +- Failure behavior: diagnostics write failures fail the CLI command with + context; retention removes or preserves the run directory based on result and + warnings. +- Tests to inspect before changing: `internal/core/diagnostics/*_test.go`, + CLI diagnostics tests. +- Architectural invariants: diagnostics must not expose secrets; cleanup must + use narrow paths. + +## Integration Documentation Plan + +### Seriatim Minimal Transcript JSON + +- Path: `docs/integrations/seriatim.md` +- External system or contract: Seriatim minimal transcript JSON. +- Current usage: parsed by the `seriatim` input adapter into a generic source + document. +- Version or compatibility notes: repository currently supports only the minimal + shape implemented by `internal/modules/input/seriatim`. +- What should be documented: accepted fields, validation, source mapping, + metadata keys, capabilities, example. +- What should not be documented: future Seriatim variants or D&D extraction + behavior. + +### OpenAI-Compatible Structured Output + +- Path: `docs/integrations/openai-compatible.md` +- External system or contract: OpenAI-compatible chat completions API with + `response_format.type=json_schema`. +- Current usage: production LLM client posts to `/chat/completions` + and decodes assistant JSON content. +- Version or compatibility notes: compatibility is defined by the implemented + request and response fields in `internal/framework/llm`. +- What should be documented: endpoint construction, auth header behavior, + request shape summary, response assumptions, retry behavior, config links. +- What should not be documented: comprehensive provider API reference or + unsupported providers. + +### Durable JSON Output + +- Path: `docs/integrations/json-output.md` +- External system or contract: file-format contract for Notarius JSON output. +- Current usage: produced by the `json` output encoder and written by the CLI. +- Version or compatibility notes: manifest includes schema version and resolved + pipeline provenance; exact envelope versioning should match implemented + fields. +- What should be documented: file names, JSON shapes, artifact grouping, source + references, rejected artifacts, warnings, filename sanitization. +- What should not be documented: future output formats or remote storage. + +### D&D Spell-Cast Artifacts + +- Path: `docs/integrations/dnd-spell-artifacts.md` +- External system or contract: durable `dnd.spell_cast` artifact payload and + validation contract. +- Current usage: produced by the `dnd/spells` extractor and serialized by JSON + output. +- Version or compatibility notes: artifact type `dnd.spell_cast`, schema version + `v1`; response schema asset `dnd_spells.v1.json` is internal to extraction. +- What should be documented: payload fields, source references, rejection reason + codes, prompt/schema manifest metadata, limits. +- What should not be documented: unimplemented D&D item, NPC, combat, or rules + extraction. + +## Recommended Implementation Sequence + +### Stage 1: Root Orientation And Core References + +- Goal: make the project immediately understandable and add required public + references. +- Files to create/update/delete/move: rewrite `README.md`; create + `docs/cli.md`; create `docs/config.md`; update examples only if needed for + clarity and tests. +- Repository areas to inspect: `internal/cli/run.go`, `internal/cli/run_test.go`, + `internal/core/config`, `examples/`. +- Acceptance criteria: README is concise; CLI and config docs are canonical and + describe only implemented behavior; maintained example commands match tests. +- Suggested validation commands: `go test ./internal/cli ./internal/core/config`; + `go test ./...` if examples changed. +- One prompt? Yes, if kept focused on README, CLI, and config only. + +### Stage 2: Operations And Troubleshooting + +- Goal: document durable output, diagnostics, retention, and common recovery + paths. +- Files to create/update/delete/move: create `docs/operations.md`; create + `docs/troubleshooting.md`. +- Repository areas to inspect: `internal/cli/run.go`, + `internal/core/diagnostics`, `internal/modules/output/json`, + `internal/framework/llm`, relevant tests. +- Acceptance criteria: operations doc names implemented output and diagnostics + files; troubleshooting entries map to implemented errors; no remote storage, + resume command, or archive behavior is documented. +- Suggested validation commands: `go test ./internal/cli ./internal/core/diagnostics ./internal/modules/output/json ./internal/framework/llm`. +- One prompt? Yes. + +### Stage 3: Developer Policy And Internal Docs + +- Goal: add required developer workflow and internal component documentation. +- Files to create/update/delete/move: create `docs/policy/development.md`; + create `docs/internal/overview.md`; create `docs/internal/pipeline.md`; + create `docs/internal/modules.md`; create `docs/internal/llm.md`; create + `docs/internal/diagnostics.md`. +- Repository areas to inspect: `AGENTS.md`, `go.mod`, `docs/policy`, + `internal/framework`, `internal/core`, `internal/modules`, + `internal/cli/catalog.go`. +- Acceptance criteria: docs preserve architecture-policy boundaries; internal + docs cover implemented components only; development doc gives actionable + commands and contribution patterns without duplicating reference docs. +- Suggested validation commands: `go test ./...`; `go vet ./...`; + `go build ./cmd/notarius`. +- One prompt? Possibly too large. If needed, split into Stage 3A + (`development.md` and `overview.md`) and Stage 3B (component docs). + +### Stage 4: Integration And Artifact Contracts + +- Goal: place external and durable file-format contracts in canonical + integration docs. +- Files to create/update/delete/move: update `docs/integrations/seriatim.md`; + create `docs/integrations/openai-compatible.md`; create + `docs/integrations/json-output.md`; create + `docs/integrations/dnd-spell-artifacts.md`; delete or replace + `docs/integrations/dnd-spells.md` after its content is split. +- Repository areas to inspect: `internal/modules/input/seriatim`, + `internal/framework/llm`, `internal/modules/output/json`, + `internal/core/artifacts`, `internal/modules/extract/dnd/spells`. +- Acceptance criteria: integration docs document actual external/file-format + contracts; no stale claim that CLI run is unimplemented; D&D details are kept + out of generic docs. +- Suggested validation commands: `go test ./internal/modules/input/seriatim ./internal/framework/llm ./internal/modules/output/json ./internal/modules/extract/dnd/spells`. +- One prompt? Yes. + +### Stage 5: Roadmap Cleanup And Link Verification + +- Goal: remove completed planning clutter and verify the final documentation + tree. +- Files to create/update/delete/move: delete completed roadmap files or reduce + them to genuine future work; update links across README and docs; keep + `docs/roadmap/documentation.md` only if remaining documentation work exists. +- Repository areas to inspect: all docs, examples, roadmap files. +- Acceptance criteria: future/planned behavior appears only under + `docs/roadmap/`; completed MVP implementation details are documented in + canonical current-behavior docs; links resolve; no stale template text remains. +- Suggested validation commands: `go test ./...`; `go vet ./...`; + `go build ./cmd/notarius`; grep checks from the validation plan. +- One prompt? Yes. + +## Validation Plan + +Run these checks during or after implementation: + +- `go test ./...` +- `go vet ./...` +- `go build ./cmd/notarius` +- `go test ./internal/cli` after CLI docs or examples change. +- `go test ./internal/core/config` after config docs or examples change. +- `go test ./internal/modules/input/seriatim` after Seriatim docs/examples + change. +- `go test ./internal/modules/extract/dnd/spells` after spell artifact docs + change. +- `go test ./internal/modules/output/json` after JSON output docs change. +- `go test ./internal/framework/llm` after OpenAI-compatible docs change. + +No automated documentation link checker currently exists in the repository. +Future implementation agents should manually verify Markdown links or add a +tool only if the project adopts one deliberately. + +Recommended grep checks: ```sh -notarius run dnd-session --input ./source.json -notarius run dnd-session --input ./source.json --only spells,npcs -notarius config validate -notarius pipelines list +rg -n 'go-application-template|does not yet|not yet implemented|provisional|Likely future|CLI `run` workflow' README.md docs examples --glob '!docs/policy/**' --glob '!docs/roadmap/**' +rg -n 'NPC|item|combat|Obsidian|Markdown' README.md docs examples --glob '!docs/roadmap/**' +rg -n 'TODO|FIXME' README.md docs examples ``` -Once implemented, `docs/cli.md` should document: +Manual review items: -- pipeline ID selection; -- required input path flags; -- `--only` artifact-lane selection; -- config path behavior; -- operational overrides such as output path, model, concurrency, and diagnostics - directory; -- output path behavior; -- diagnostics and report behavior; -- exit codes. +- Confirm README remains short and orientation-focused. +- Confirm `docs/cli.md` contains every implemented command and flag from + `internal/cli/run.go`. +- Confirm `docs/config.md` contains every implemented file config field and env + override from `internal/core/config`. +- Confirm examples are secret-free and covered by tests where practical. +- Confirm roadmap docs do not masquerade as current-behavior docs. +- Confirm no docs mention unimplemented NPC, item, combat, Markdown, Obsidian, + remote storage, resume, or DAG behavior except under roadmap limitations or + explicit out-of-scope notes. -### Config Docs Should Separate Framework And Plugin-Like Options +## Open Questions -`docs/config.md` should describe named pipeline profiles and the resolved -pipeline model. - -It should cover: - -- config file locations and precedence; -- `llm_profiles`; -- `pipelines..input`; -- `pipelines..chunk`; -- `pipelines..artifacts..extract`; -- lane `merge`, `normalize`, and validator settings; -- output module selection; -- string shorthand versus inline module-binding object form; -- defaults for omitted slots; -- capability validation; -- pipeline digest and manifest provenance. - -Module-specific config should stay inline with the pipeline slot that owns it. -Top-level named module instances should not be introduced until repeated inline -settings create real drift. `llm_profiles` are the cross-cutting exception. - -### Examples Should Stay Real - -Examples should be added only when the matching behavior exists and should be -covered by tests where practical. - -Likely future examples: - -```text -examples/seriatim-minimal-transcript.json -examples/minimal-config.yml -examples/dnd-spells.artifacts.json -examples/dnd-session.config.yml -``` - -Examples should be secret-free and should use the same command shapes documented -in `docs/cli.md`. - -## Canonical Documentation Targets - -When the first vertical slice is implemented, add or update: - -- `README.md`: concise purpose, shortest useful command, links. -- `docs/cli.md`: implemented command behavior. -- `docs/config.md`: implemented config behavior. -- `docs/operations.md`: diagnostics, retention, failure inspection. -- `docs/troubleshooting.md`: common failures. -- `docs/internal/overview.md`: implemented package map. -- `docs/internal/pipeline.md`: implemented extraction flow. -- `docs/internal/stage-modules.md`: stage contracts and implemented modules. -- `docs/internal/input-modules.md`: input adapter contract and implemented input modules. -- `docs/internal/validators.md`: validator contract and built-ins. -- `docs/integrations/seriatim-transcript.md`: Seriatim input contract. -- `docs/integrations/artifacts.md`: output artifact envelope. - -## Review Checklist For Future Documentation Changes - -Before merging docs, check: - -- Does the document describe implemented behavior outside `docs/roadmap/`? -- Are source-format details isolated to input module or integration docs? -- Are D&D details isolated to extract module or artifact docs? -- Is there one canonical home for the topic? -- Do command examples match implemented CLI syntax? -- Do config examples use named pipeline profiles rather than ad hoc module - flags? -- Are examples valid, maintained, and free of secrets? -- Did any architecture, config, CLI, stage module, validator, or artifact - contract change require a docs update? +None. The repository evidence is sufficient to plan the documentation migration.