From 6ceabf40bba6ede69bc7ed5496d6bb77db299a9c Mon Sep 17 00:00:00 2001
From: Eric Rakestraw <eric@rakestraw.us>
Date: Sat, 4 Jul 2026 03:07:56 +0000
Subject: [PATCH] Add developer and internal architecture documentation

---
 docs/internal/diagnostics.md |  88 +++++++++++++++++++
 docs/internal/llm.md         | 115 ++++++++++++++++++++++++
 docs/internal/modules.md     | 164 +++++++++++++++++++++++++++++++++++
 docs/internal/overview.md    |  86 ++++++++++++++++++
 docs/internal/pipeline.md    | 127 +++++++++++++++++++++++++++
 docs/policy/development.md   | 139 +++++++++++++++++++++++++++++
 6 files changed, 719 insertions(+)
 create mode 100644 docs/internal/diagnostics.md
 create mode 100644 docs/internal/llm.md
 create mode 100644 docs/internal/modules.md
 create mode 100644 docs/internal/overview.md
 create mode 100644 docs/internal/pipeline.md
 create mode 100644 docs/policy/development.md

diff --git a/docs/internal/diagnostics.md b/docs/internal/diagnostics.md
new file mode 100644
index 0000000..0771a05
--- /dev/null
+++ b/docs/internal/diagnostics.md
@@ -0,0 +1,88 @@
+# Diagnostics Internals
+
+Diagnostics internals live in `internal/core/diagnostics`. Operator-facing run
+behavior is documented in [Operations](../operations.md).
+
+## Purpose
+
+Diagnostics provide local inspection artifacts for a run without becoming the
+durable output contract. Durable user output is produced by output modules and
+written by the CLI.
+
+Diagnostics must not expose secrets.
+
+## Run Directory
+
+`NewRunDirectory(workDir, retention)` creates:
+
+```text
+<workDir>/run-<unix-nanoseconds>/
+```
+
+If `workDir` is empty, it defaults to `/tmp/notarius`. Empty retention defaults
+to `auto`.
+
+The writer makes the work directory if needed, then attempts to create a unique
+run directory. It retries run ID creation a bounded number of times if a
+collision occurs.
+
+## Artifact Writers
+
+Implemented artifact names:
+
+- `invocation.json`
+- `effective-config.json`
+- `resolved-pipeline.json`
+- `source-document.json`
+- `run-manifest.json`
+- `run-report.json`
+- `warnings.json`
+- `error.log`
+
+JSON artifacts are encoded with indentation and a trailing newline. Writes are
+atomic through a temporary file in the target directory followed by rename.
+
+Artifact names must be single relative file names. Absolute paths, path
+separators, and names resolving outside the run directory are rejected.
+
+## Redacted Effective Config
+
+Diagnostics writers accept payloads that implement
+`RedactedDiagnosticsPayload`. `internal/core/config` uses this to redact API
+keys in effective config diagnostics while preserving resolved pipeline context.
+
+The redaction path clones config data before replacing secret values.
+
+## Retention
+
+Retention is decided by `ShouldRetainRunDirectory`.
+
+- Failed runs are always retained.
+- `always` retains successful runs.
+- `never` removes successful runs.
+- `auto` retains successful runs only when warnings exist.
+- Unknown retention values are treated as retain by the retention decision, but
+  config validation rejects unsupported values before normal runs.
+
+`ApplyRetention` removes only the specific run directory.
+
+## CLI Failure Behavior
+
+The CLI creates the diagnostics run directory after config loading and before
+pipeline resolution. Failures before that point do not have diagnostics.
+
+After diagnostics creation, run failures call `WriteErrorLog` and apply
+retention with `RunSucceeded: false`, so the run directory remains available.
+
+When the pipeline returns a partial manifest on failure, the CLI writes that
+manifest before logging the failure.
+
+## Invariants
+
+- Diagnostics paths must be narrow and run-directory scoped.
+- Writes should be atomic where practical.
+- Secrets must be redacted.
+- Diagnostics write failures are command failures because they can hide the
+  information needed for recovery.
+- Durable output file contracts belong to output modules and integration docs,
+  not to diagnostics.
diff --git a/docs/internal/llm.md b/docs/internal/llm.md
new file mode 100644
index 0000000..90284a8
--- /dev/null
+++ b/docs/internal/llm.md
@@ -0,0 +1,115 @@
+# LLM Runtime
+
+The implemented LLM runtime lives in `internal/framework/llm`. It provides
+transport-neutral structured completion contracts, an OpenAI-compatible HTTP
+adapter, concurrency scheduling, schema registry helpers, retry behavior, and
+secret redaction.
+
+## Contract
+
+Modules depend on `contracts.StructuredLLMClient`:
+
+```go
+CompleteStructured(ctx, request, out) (response, error)
+```
+
+The request contains messages, optional model override, response schema name,
+and response schema JSON. The caller supplies a pointer target for decoded
+structured output.
+
+Extractors own prompts and schemas. Provider adapters should not contain
+domain-specific prompt logic.
+
+## Production Client Construction
+
+`internal/cli` builds the production LLM client from the effective config:
+
+1. find the effective LLM profile;
+2. build `OpenAICompatibleClientConfig`;
+3. create an OpenAI-compatible client;
+4. create a scheduler from profile or global concurrency;
+5. wrap the client with `NewScheduledClient`;
+6. return non-secret LLM profile manifest metadata.
+
+The current run command requires exactly one distinct effective LLM profile for
+the resolved pipeline.
+
+## OpenAI-Compatible Adapter
+
+`OpenAICompatibleClient` posts JSON to:
+
+```text
+<base_url>/chat/completions
+```
+
+It sends:
+
+- `model`
+- `messages`
+- `response_format.type = "json_schema"`
+- `response_format.json_schema.name`
+- `response_format.json_schema.strict = true`
+- `response_format.json_schema.schema`
+
+If an API key is configured, the adapter sends an `Authorization: Bearer ...`
+header.
+
+The adapter accepts assistant content either as a JSON string containing JSON or
+as raw JSON content. It then unmarshals that content into the caller-provided
+target.
+
+External wire-contract details belong in the OpenAI-compatible integration doc.
+
+## Retries And Timeouts
+
+The adapter retries:
+
+- provider request failures;
+- response read failures;
+- HTTP `429`;
+- HTTP `5xx`;
+- malformed provider envelopes;
+- malformed assistant JSON;
+- structured-output decode failures.
+
+Non-retryable `4xx` responses are returned without retry. Request timeout comes
+from the effective LLM profile. Context cancellation is respected.
+
+## Scheduler
+
+`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a
+FIFO queue of waiters. Cancellation removes queued waiters or releases granted
+permits.
+
+`NewScheduledClient` wraps any structured LLM client and runs each completion
+inside the scheduler.
+
+Effective concurrency is:
+
+1. `llm_profiles.<id>.max_concurrency`, when greater than zero;
+2. `concurrency.total_llm`, when greater than zero;
+3. `1`.
+
+## Schema Registry
+
+The framework schema registry embeds generic test schemas. It also exposes
+helpers for caller-owned schemas:
+
+- `LoadResponseSchema`
+- `LookupResponseSchema`
+- `MustLookupResponseSchema`
+- `ResponseSchema.DiagnosticsMap`
+
+`DiagnosticsMap` omits raw schema content and includes metadata such as key,
+ID, version, name, and SHA-256.
+
+The D&D spell extractor owns and loads its own embedded response schema.
+
+## Secret Redaction
+
+Provider errors are passed through `ErrorWithSecretsRedacted` with the API key
+and bearer-token value. Config diagnostics use redacted effective config
+payloads.
+
+Do not add raw provider request bodies, response bodies, API keys, or prompt
+payloads to diagnostics by default.
diff --git a/docs/internal/modules.md b/docs/internal/modules.md
new file mode 100644
index 0000000..102df36
--- /dev/null
+++ b/docs/internal/modules.md
@@ -0,0 +1,164 @@
+# Modules
+
+Production modules live under `internal/modules`. Each module implements one
+contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and
+registers itself with the matching pipeline registry.
+
+The CLI production catalog currently registers only the modules listed here.
+
+## Contract Pattern
+
+A production module package should provide:
+
+- a stable module key;
+- a constructor such as `New`;
+- the relevant contract implementation;
+- `ModuleSpec`;
+- `Register`;
+- focused tests for registration, options, contract behavior, and errors.
+
+Module specs should describe capabilities accurately. Resolution uses specs to
+reject incompatible pipelines before execution.
+
+## `seriatim` Input
+
+Package: `internal/modules/input/seriatim`
+
+The `seriatim` adapter parses Seriatim minimal transcript JSON into a generic
+source document. It owns transcript JSON details, source ID selection, source
+digest creation, transcript segment validation, and segment metadata mapping.
+
+Provides:
+
+- `source.transcript`
+- `transcript.speaker`
+- `transcript.timestamps`
+
+External JSON shape belongs in the Seriatim integration doc.
+
+## `generic` Chunker
+
+Package: `internal/modules/chunk/generic`
+
+The `generic` chunker splits source units into ordered chunks. It validates the
+source document, clones source units, assigns chunk IDs such as `chunk-000001`,
+and records chunk metadata for start unit, end unit, and unit count.
+
+Options:
+
+- `max_units`: positive integer, default `50`;
+- `overlap_units`: non-negative integer, default `0`, and less than
+  `max_units`.
+
+Provides:
+
+- `chunks`
+
+## `dnd/spells` Extractor
+
+Package: `internal/modules/extract/dnd/spells`
+
+The `dnd/spells` extractor owns D&D spell-cast artifact semantics. It renders
+embedded prompts, loads the embedded structured response schema, calls the
+structured LLM client, converts spell-cast responses into artifact candidates,
+and supplies deterministic validators.
+
+Requires:
+
+- `chunks`
+- `source.transcript`
+
+Provides:
+
+- `dnd.spell_casts`
+
+Artifact type and schema version:
+
+- artifact type: `dnd.spell_cast`
+- schema version: `v1`
+
+The extractor adds prompt and response-schema provenance to lane manifest
+metadata. Durable artifact payload details belong in the D&D spell artifact
+integration doc.
+
+## D&D Spell Validators
+
+The spell extractor returns two built-in validators:
+
+- `dnd/spells/shape`: rejects malformed payloads and missing required fields.
+- `dnd/spells/source_refs`: rejects candidates without valid source references.
+
+Reason codes include:
+
+- `invalid_payload`
+- `missing_required_field`
+- `missing_source_ref`
+- `invalid_source_ref`
+
+These validators are supplied by the extractor when no validators are configured
+for the lane.
+
+## `appendorder` Merger
+
+Package: `internal/modules/merge/appendorder`
+
+The `appendorder` merger clones and appends candidates in chunk order. It does
+not deduplicate or reconcile candidates.
+
+Provides:
+
+- `merged`
+
+## `noop` Normalizer
+
+Package: `internal/modules/normalize/noop`
+
+The `noop` normalizer clones merged candidates and returns them unchanged.
+
+Requires:
+
+- `merged`
+
+Provides:
+
+- `normalized`
+
+## `json` Output
+
+Package: `internal/modules/output/json`
+
+The `json` output encoder converts approved artifacts, rejected artifacts,
+warnings, and the run manifest into logical JSON output files. It groups
+approved artifacts by artifact type and sanitizes artifact-type file names.
+
+Requires:
+
+- `normalized`
+
+Provides:
+
+- `encoded`
+
+Durable output file shapes belong in the JSON output integration doc. Operator
+behavior belongs in [Operations](../operations.md).
+
+## Production Registration
+
+Production registration is centralized in `internal/cli/catalog.go`.
+
+Do not make framework code import production modules. The CLI wires production
+modules at the application boundary; tests may provide fake registries or fake
+catalogs directly.
+
+## Adding A Module
+
+When adding a module, keep source-format and extraction-domain boundaries clear:
+
+- input modules may know external source formats;
+- extract modules may know artifact semantics and prompt/schema assets;
+- merge and normalize modules own candidate combination and reconciliation;
+- output modules own serialization, not diagnostics or CLI reporting.
+
+Update [Development](../policy/development.md), [Configuration](../config.md),
+internal docs, integration docs, and examples when the new module becomes
+implemented production behavior.
diff --git a/docs/internal/overview.md b/docs/internal/overview.md
new file mode 100644
index 0000000..5cca600
--- /dev/null
+++ b/docs/internal/overview.md
@@ -0,0 +1,86 @@
+# Internal Overview
+
+This directory documents implemented Notarius internals for developers and LLM
+coding agents. It complements [Architecture](../policy/architecture.md), which
+is the durable policy for boundaries and invariants.
+
+## Executable And CLI
+
+`cmd/notarius` calls the CLI package. `internal/cli` owns:
+
+- command parsing and usage;
+- config discovery and loading;
+- production module catalog and registry wiring;
+- production LLM client construction;
+- run directory creation;
+- durable output writes;
+- user-facing stdout, stderr, and exit codes.
+
+The CLI should stay thin around framework contracts. Domain extraction behavior
+belongs in modules, not in command handlers.
+
+## Core Packages
+
+- `internal/core/artifacts`: artifact candidates, approved artifacts, rejected
+  artifacts, validation decisions, and run manifests.
+- `internal/core/config`: defaults, YAML config parsing, environment overrides,
+  validation, redaction, and resolved pipeline config.
+- `internal/core/diagnostics`: per-run diagnostics directory creation,
+  diagnostics artifact writers, atomic writes, and retention decisions.
+- `internal/core/source`: source documents, source units, source references, and
+  validation.
+
+Core packages should remain deterministic and concrete. They should not import
+production modules.
+
+## Framework Packages
+
+- `internal/framework/contracts`: interfaces and request/result structs for
+  input adapters, chunkers, extractors, mergers, normalizers, validators, output
+  encoders, and structured LLM clients.
+- `internal/framework/pipeline`: module registries, module specs, profile
+  resolution, capability checks, run orchestration, warnings, validation, and
+  manifest population.
+- `internal/framework/llm`: OpenAI-compatible structured-output client,
+  scheduler, schema registry, retries, and secret redaction.
+- `internal/framework/prompt`: embedded prompt registry and template rendering.
+- `internal/framework/validate`: validator decision helpers and cardinality
+  enforcement.
+
+Framework code should stay source-agnostic and domain-agnostic.
+
+## Module Packages
+
+Production module packages live under `internal/modules`:
+
+- `input/seriatim`
+- `chunk/generic`
+- `extract/dnd/spells`
+- `merge/appendorder`
+- `normalize/noop`
+- `output/json`
+
+Each module package owns its contract implementation, module spec,
+registration, options, focused tests, and module-specific errors.
+
+## Fixtures And Tests
+
+The repository uses focused package tests plus a fixture-driven CLI workflow.
+
+- CLI acceptance tests cover maintained examples under `examples/`.
+- Pipeline tests cover registry composition and end-to-end framework behavior
+  with fakes.
+- Module tests cover implemented module contracts without requiring real
+  provider calls.
+- LLM tests use local test servers and fakes.
+
+Do not use real external services in tests. Use fakes, fixtures, or local test
+servers.
+
+## Boundary Reminders
+
+- Source-format details stay in input modules and integration docs.
+- Extraction-domain details stay in extract modules and artifact docs.
+- Provider wire details stay in the LLM runtime and provider integration docs.
+- Durable output contracts belong in integration docs.
+- Operator procedures belong in `docs/operations.md`, not internal docs.
diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md
new file mode 100644
index 0000000..96b8267
--- /dev/null
+++ b/docs/internal/pipeline.md
@@ -0,0 +1,127 @@
+# Pipeline Internals
+
+The implemented pipeline runner lives in `internal/framework/pipeline`. It
+executes the fixed workflow defined by the architecture policy:
+
+```text
+input -> chunk -> extract -> merge -> normalize -> output
+```
+
+Pipeline execution is serial. The runner executes the resolved lanes one after
+another in the fixed workflow order.
+
+## Profile Resolution
+
+Config loading produces `pipeline.PipelineProfile` values. Resolution happens
+before execution:
+
+1. `internal/core/config.Config.Resolve` validates config and finds the named
+   pipeline.
+2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
+3. Module bindings are defaulted:
+   - chunk: `generic`
+   - merge: `appendorder`
+   - normalize: `noop`
+   - output: `json`
+   - LLM profile: `default`
+4. The module catalog is checked for each bound module key.
+5. Module capabilities are checked in workflow order.
+6. A digest is calculated from the resolved pipeline without the digest field.
+
+The CLI writes the resolved pipeline and digest to diagnostics.
+
+## Registries And Module Specs
+
+`pipeline.Registries` holds concrete constructors for execution. A
+`pipeline.ModuleCatalog` exposes module specs for config validation and
+resolution.
+
+Every production module registers a `ModuleSpec` with:
+
+- `Key`: module key used in config;
+- `Stage`: module kind such as input, chunk, extract, merge, normalize,
+  validate, or output;
+- `Provides`: capabilities added after that module runs;
+- `Requires`: capabilities that must already be available.
+
+Capability checks prevent incompatible pipeline composition before a run starts.
+
+## Runner Input And Output
+
+`pipeline.RunInput` carries:
+
+- a `ResolvedPipeline`;
+- optional source ID, input path, and raw input bytes;
+- a structured LLM client;
+- run ID, start time, LLM profile manifest metadata, and CLI metadata.
+
+`pipeline.RunOutput` carries:
+
+- run manifest;
+- approved artifacts;
+- rejected artifacts;
+- warnings;
+- logical output files returned by the output encoder.
+
+The CLI owns durable file writes and diagnostics writes after the runner returns.
+
+## Execution
+
+The runner:
+
+1. validates run input and registries;
+2. builds the input adapter and parses the raw input into a source document;
+3. validates the source document;
+4. builds the chunker and produces source chunks;
+5. runs each selected artifact lane in sorted resolved order;
+6. builds the output encoder and validates logical output file names.
+
+Within an artifact lane, the runner:
+
+1. builds the extractor, merger, and normalizer;
+2. records module manifest metadata when modules provide it;
+3. extracts candidates from each chunk;
+4. normalizes candidate envelope fields such as index, extractor key, artifact
+   type, and schema version;
+5. merges candidates;
+6. normalizes merged candidates;
+7. validates candidate envelope consistency;
+8. runs validators;
+9. converts approved candidates to artifacts.
+
+## Validators
+
+If a lane declares validators in config, the runner builds those validators from
+the validator registry. Otherwise it uses validators returned by the extractor.
+
+Each validator must return exactly one decision for each eligible candidate. The
+runner enforces decision cardinality with `internal/framework/validate`.
+Rejected candidates are removed before the next validator runs. Approved
+candidates continue through the chain.
+
+The production CLI currently registers no standalone validator modules. The
+current D&D spell extractor supplies deterministic shape and source-reference
+validators.
+
+## Warnings And Failures
+
+Warnings from chunking, extraction, merging, normalization, validation, and
+output encoding are accumulated in `RunOutput.Warnings`.
+
+Errors wrap the operation and module key or lane context. If execution fails
+after a manifest exists, the returned manifest is marked `failed` and receives a
+completion timestamp.
+
+On successful execution, the manifest validation status is:
+
+- `approved` when no candidates were rejected;
+- `rejected` when at least one candidate was rejected.
+
+## Manifest Population
+
+The manifest records run ID, pipeline ID, pipeline digest, module keys, artifact
+lanes, LLM profile metadata, source digest, validation status, and timing.
+
+Modules can add non-secret manifest metadata by implementing
+`contracts.ManifestMetadataProvider`. The D&D spell extractor uses this for
+prompt and response-schema provenance.
diff --git a/docs/policy/development.md b/docs/policy/development.md
new file mode 100644
index 0000000..25c9c96
--- /dev/null
+++ b/docs/policy/development.md
@@ -0,0 +1,139 @@
+# Development
+
+This document defines contributor workflow for Notarius. For architectural
+invariants and package boundaries, read [Architecture](architecture.md) first.
+
+## Required Reading
+
+Before changing the repository, review:
+
+- [Architecture](architecture.md)
+- [Documentation Policy](documentation.md)
+
+Keep current-behavior documentation limited to implemented behavior. Put planned
+or deferred behavior under `docs/roadmap/`.
+
+## Repository Layout
+
+- `cmd/notarius`: executable entry point.
+- `internal/cli`: CLI parsing, production catalog wiring, config loading, run
+  command orchestration, output writes, and user-facing errors.
+- `internal/core`: deterministic models and policy for artifacts, source
+  documents, config, and diagnostics.
+- `internal/framework`: reusable contracts, pipeline orchestration, prompt
+  helpers, validation helpers, and LLM runtime plumbing.
+- `internal/modules`: concrete input, chunk, extract, merge, normalize, and
+  output modules.
+- `docs`: policy, user/operator docs, internal docs, integration docs, and
+  roadmap files.
+- `examples`: maintained, secret-free examples covered by tests where practical.
+
+## Validation Commands
+
+Run focused tests for the area changed, then run the broader checks when the
+change affects shared contracts, CLI behavior, or documentation examples.
+
+```sh
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```
+
+Useful focused checks:
+
+```sh
+go test ./internal/cli
+go test ./internal/core/config
+go test ./internal/framework/pipeline
+go test ./internal/framework/llm
+go test ./internal/modules/input/seriatim
+go test ./internal/modules/extract/dnd/spells
+go test ./internal/modules/output/json
+```
+
+## Go Conventions
+
+- Prefer the standard library unless a dependency is justified by correctness,
+  security, interoperability, or substantial complexity reduction.
+- Keep package names short, lowercase, and idiomatic.
+- Preserve import direction: framework and core code must not depend on concrete
+  production modules.
+- Use `context.Context` for long-running operations and external calls.
+- Return contextual errors that name the operation and relevant module, path, or
+  resource.
+- Do not include secrets in errors, logs, diagnostics, manifests, or docs.
+
+## Adding Config Fields
+
+Config behavior is centralized under `internal/core/config`.
+
+When adding a file config field:
+
+1. Update file config structs and YAML parsing in `file_config.go`.
+2. Apply the field over defaults in config application code.
+3. Add validation in `validation.go` when the field has constraints.
+4. Add environment override support in `env.go` only for operational overrides.
+5. Update redaction if the field can contain secrets.
+6. Add focused config tests.
+7. Update [Configuration](../config.md) and maintained examples when behavior
+   changes.
+
+Pipeline composition should remain config-driven. Do not add command flags that
+silently replace structural pipeline definitions.
+
+## Adding CLI Flags Or Commands
+
+CLI behavior lives in `internal/cli`.
+
+When adding CLI surface:
+
+1. Keep syntax explicit and update usage text.
+2. Validate arguments before running expensive work.
+3. Convert internal errors into concise user-facing messages.
+4. Add CLI tests for success, syntax errors, and failure modes.
+5. Update [CLI Reference](../cli.md), and update
+   [Operations](../operations.md) or [Troubleshooting](../troubleshooting.md)
+   if run behavior changes.
+
+## Adding Modules Or Adapters
+
+Concrete modules live under `internal/modules/<kind>/...` and implement the
+interfaces in `internal/framework/contracts`.
+
+For a new production module:
+
+1. Implement the relevant contract.
+2. Expose a `ModuleSpec` with the correct module key, module kind, provided
+   capabilities, and required capabilities.
+3. Expose a `Register` function that registers the module with its registry.
+4. Add focused module tests for contract behavior, registration, options,
+   validation, and errors.
+5. Register the module in `internal/cli/catalog.go` only when it is production
+   ready.
+6. Update internal docs and user-facing docs only for implemented behavior.
+
+Source-format behavior belongs in input modules and integration docs.
+Extraction-domain behavior belongs in extract modules and artifact docs.
+
+## Updating Examples
+
+Examples must be valid, secret-free, and small.
+
+- Prefer environment-based secret configuration.
+- Keep `examples/dnd-spells.config.yml` loadable by CLI tests.
+- Keep `examples/seriatim-minimal-transcript.json` compatible with the Seriatim
+  adapter.
+- Do not add expected-output fixtures unless they are validated or have a clear
+  regeneration procedure.
+
+## Documentation Updates
+
+Update docs in the same change when behavior changes.
+
+- CLI syntax: `docs/cli.md`
+- Config fields and defaults: `docs/config.md`
+- Output, diagnostics, retention, or recovery: `docs/operations.md`
+- Common user-facing failures: `docs/troubleshooting.md`
+- Internal architecture and contracts: `docs/internal/`
+- External file formats and durable integration contracts: `docs/integrations/`
+- Future or planned work only: `docs/roadmap/`