notarius/docs/internal/overview.md

Internal Overview

This document maps the implemented Notarius components and their ownership. It complements the durable invariants in Architecture and links to focused internal documentation for deeper behavior.

Execution Path

The executable delegates to the CLI, which resolves configuration and wires the production application around the framework runner:

cmd/notarius
    -> internal/cli
       -> config resolution + production registries + LLM client
       -> input -> chunk -> extract -> merge -> normalize -> output
       -> durable output writes

Pipeline side channels:
    diagnostics     checkpoints     debug artifacts

Pipeline execution is serial. Configuration selects modules for the fixed stage shape; registries construct them after profile, capability, validator, and reference resolution.

Application Boundary

cmd/notarius contains the executable entry point and delegates process exit behavior to internal/cli.

internal/cli owns command parsing, configuration discovery, production module and validator registration, prompt asset collection, production LLM client construction, reference preparation, workspace recorder setup, durable output writes, and user-facing stdout, stderr, and exit codes. It is the composition root for concrete production packages.

Core Packages

Package	Implemented responsibility
internal/core/artifacts	Run manifests and artifact serialization shapes.
internal/core/config	Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline configuration.
internal/core/diagnostics	Diagnostics run directories, artifact writers, atomic writes, and retention decisions.
internal/core/source	Generic source documents, units, references, and validation.
internal/core/workspace	Workspace settings, safe paths and writes, checkpoint identities, and checkpoint manifest types.

These packages provide concrete, deterministic models and policy. Production module registration occurs at the CLI boundary rather than in core packages.

Framework Packages

Package	Implemented responsibility
internal/framework/contracts	Stage, validator, reference, output, and structured LLM interfaces and request/result types.
internal/framework/pipeline	Module registries, profile resolution, capability checks, reference materialization, validation chains, retries, orchestration, warnings, and manifest population.
internal/framework/validate	Shared validator decision and cardinality helpers.
internal/framework/llm	Scriptorium-backed structured completions, prompt/schema asset registration, scheduling, profile recording, and secret redaction.
internal/framework/checkpoint	Workspace-backed checkpoint loading, recording, and payload envelopes.
internal/framework/debug	Workspace-backed framework and LLM debug artifacts.

Framework contracts carry raw stage outputs between modules. The runner owns provenance, validation sequencing, rejection handling, checkpoint boundaries, debug boundaries, and final manifest assembly.

Production Modules

Production implementations live under internal/modules and register through the CLI catalog.

Stage	Module key	Package	Role
Input	seriatim	internal/modules/input/seriatim	Converts Seriatim transcript JSON into the generic source model.
Chunk	generic	internal/modules/chunk/generic	Splits ordered source units by configured unit counts and overlap.
Chunk	dnd/scenes	internal/modules/chunk/dnd/scenes	Uses structured LLM output to create contiguous D&D scene chunks.
Extract	dnd/spells	internal/modules/extract/dnd/spells	Extracts source-grounded D&D spell-cast artifacts.
Merge	appendorder	internal/modules/merge/appendorder	Combines accepted extract outputs in chunk order.
Normalize	noop	internal/modules/normalize/noop	Preserves accepted merged output unchanged.
Output	json	internal/modules/output/json	Encodes manifests, indexes, warnings, rejections, and accepted lane payloads as logical JSON files.

internal/modules/sharedassets composes shared prompt filesystems. internal/modules/sharedassets/dnd owns shared D&D prompt fragments, reference slots, prompt input assembly, and source-unit reference helpers.

Validators

Concrete validators live under internal/validators and register separately from stage modules. Generic validators cover unconditional test decisions, JSON syntax, and JSON Schema. D&D spell validators cover artifact shape, source reference validity, and source relatedness.

The production default chain for dnd/spells extract output is registered centrally in internal/cli; module packages produce raw output but do not own the production approve/reject policy.

Files And Run State

Notarius keeps distinct output and inspection surfaces:

Surface	Owner	Purpose
Durable output	Output module and CLI writer	User-consumable run files.
Diagnostics	internal/core/diagnostics and CLI	Redacted run inspection, reports, warnings, and failures.
Checkpoints	internal/framework/checkpoint	Validated stage reuse for explicit resume.
Debug artifacts	internal/framework/debug and pipeline instrumentation	Sensitive framework-boundary and LLM call inspection.

Workspace settings determine whether and where diagnostics, checkpoints, and debug artifacts are written. Concrete stage modules do not receive workspace paths.

Focused Internal Documentation

• Pipeline Internals: resolution, execution, validators, references, retries, checkpoints, outputs, and manifests.
• Module Internals: production module contracts, capabilities, options, prompts, schemas, and registration.
• LLM Runtime: structured completion contracts, Scriptorium adapter, assets, scheduling, profile recording, and redaction.
• Diagnostics Internals: diagnostics files, retention, failure behavior, and path safety.

Test Surfaces

The repository uses focused package tests, registry and pipeline composition tests, a fake-backed walking skeleton, fixture-driven CLI coverage, and local test servers for LLM integration behavior. Tests do not require real provider calls.