notarius/docs/internal/overview.md

Internal Overview

This document inventories the implemented Notarius components. Normative boundaries and dependency direction belong in Architecture; external behavior belongs in the CLI, Configuration, Operations, and integration contracts.

Execution Path

cmd/notarius delegates to internal/cli, the production composition root. The CLI loads configuration, builds the production catalogs and runtime collaborators, invokes internal/framework/pipeline, and places the logical output files returned by the runner. Diagnostics, checkpoints, and debug recorders are optional side-channel collaborators supplied at this boundary.

Resolution produces a fixed ordered workflow and a sorted set of artifact lanes. Preparation constructs the complete module and validator set before the runner receives source bytes. Source parsing and chunking are serial; extraction uses a bounded run-wide worker pool, followed by serial per-lane merge and normalize continuations that may overlap across lanes.

Application Boundary

Package	Implemented responsibility
cmd/notarius	Executable entry point and process exit delegation.
internal/cli	Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results.

Core Packages

Package	Implemented responsibility
internal/core/artifacts	Run-manifest and provenance models.
internal/core/config	Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution.
internal/core/diagnostics	Scoped run directories, diagnostics writers, atomic writes, and retention decisions.
internal/core/source	Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests.
internal/core/workspace	Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models.

Framework Packages

Package	Implemented responsibility
internal/framework/contracts	Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types.
internal/framework/pipeline	Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population.
internal/framework/validate	Shared validator decision and cardinality helpers.
internal/framework/llm	Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction.
internal/framework/promptfs	Builds module prompt filesystems from module-owned and caller-provided shared prompt assets.
internal/framework/checkpoint	Workspace-backed checkpoint loading, recording, and payload serialization.
internal/framework/debug	Workspace-backed framework and LLM debug recording.

Framework contracts provide typed artifact, provenance-wrapper, chunk-validator, serialized-validator, and typed-validator interfaces. The runner owns handoff provenance, validation sequencing, rejection handling, checkpoint and debug boundaries, and final manifest assembly.

Artifact registries support heterogeneous typed extraction entries and kind-specific merger, normalizer, and validator variants. Resolution derives a lane's kind from its extractor, requires the matching codec, verifies exact Go type equality across the lane, and records schema identity in the resolved lane and pipeline digest. Registry entries carry separate option-validation and run-local construction closures. Preparation injects shared dependencies and constructs input, chunk, validators, ordered lanes, and output before source parsing. Production modules use strict construction-time option decoding, and LLM-backed modules retain the injected shared client. The D&D family registers the canonical dnd/spell-list codec, typed spell extractor and validators, and kind-specific generic merge and normalize strategies; generic JSON validators use the serialized-validation contract. The runner executes lanes through private exact-type-checked closures, coordinates extract results independently of completion timing, and serializes artifacts only through their codec at checkpoint, debug, and output boundaries.

Production Extensions

The canonical catalogs of user-selectable module and validator keys are in Configuration. The implemented module packages are:

Package	Implemented responsibility
internal/modules/seriatim/input/transcript	Parses the supported Seriatim transcript format into the generic source model.
internal/modules/generic/chunk/units	Splits ordered source units by unit count and overlap.
internal/modules/dnd/chunk/scenes	Produces contiguous D&D scene chunks from structured model output.
internal/modules/dnd	Owns the canonical D&D spell-list and spell-cast artifact types.
internal/modules/dnd/codec/spells	Strictly decodes and stably encodes the durable D&D spell-list representation.
internal/modules/dnd/extract/spells	Maps private structured model output to canonical source-grounded D&D spell lists.
internal/modules/generic/merge/appendorder	Combines accepted extraction results in chunk order.
internal/modules/generic/normalize/noop	Preserves accepted merged output.
internal/modules/generic/output/json	Encodes manifests, lane payloads, warnings, and rejections as logical JSON files.

internal/modules/dnd/shared owns reusable D&D prompt fragments, reference declarations, prompt input assembly, and source-unit reference helpers. Domain-neutral prompt filesystem composition lives in internal/framework/promptfs.

Generic validators under internal/modules/generic/validate provide unconditional test decisions, JSON syntax validation, and JSON Schema validation. D&D spell validators under internal/modules/dnd/validate/spells consume the canonical spell-list type directly to provide shape, source-reference, and source-relatedness decisions.

Production composition is grouped behind package-family registrars, and every implemented production extension uses its domain-first tree:

Package	Implemented responsibility
internal/modules/generic/register	Registers domain-neutral chunk, merge, normalize, output, and validator implementations.
internal/modules/seriatim/register	Registers the Seriatim input adapter.
internal/modules/dnd/register	Registers D&D modules, validators, default validator policy, and prompt/schema assets.

The CLI allocates the framework registries and asset registry, then invokes these registrars in generic, Seriatim, and D&D order.

Implementation details for all production extensions are in Module Internals.

Run-State Components

Surface	Implemented owners	Internal purpose
Durable output	Output module, pipeline runner, and CLI writer	Return logical consumer files and place them for a run.
Diagnostics	internal/core/diagnostics and internal/cli	Record redacted invocation, resolution, result, and failure inspection data.
Checkpoints	internal/framework/checkpoint and internal/core/workspace	Validate and serialize reusable stage outcomes.
Debug artifacts	internal/framework/debug and pipeline instrumentation	Capture sensitive framework-boundary and LLM-call material.

Physical layout, retention, recovery, and sensitive-data handling are defined in Operations. Concrete stage modules receive recorder interfaces and request data, not workspace paths.

Focused Documentation

• Pipeline Internals: resolution, execution, validation, retries, checkpoint/debug hooks, and result assembly.
• Module Internals: production modules, validators, assets, registration, and the contributor recipe for adding an extension.
• LLM Runtime: structured completion contracts, Scriptorium adapter, assets, scheduling, profile recording, and redaction.
• Diagnostics Internals: scoped writers, retention coordination, CLI failure flow, and path safety.