From 6e6375521d80441a71c838f993e160ef9ad68ca5 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 17 Jul 2026 03:06:23 +0000 Subject: [PATCH] Refocus developer and internal documentation --- docs/development.md | 45 +--- docs/internal/diagnostics.md | 151 +++++++------ docs/internal/llm.md | 192 +++++++++-------- docs/internal/modules.md | 368 +++++++++++++------------------- docs/internal/overview.md | 160 ++++++-------- docs/internal/pipeline.md | 396 +++++++++++++---------------------- docs/policy/architecture.md | 22 +- 7 files changed, 549 insertions(+), 785 deletions(-) diff --git a/docs/development.md b/docs/development.md index 4f7b2d6..04b68b9 100644 --- a/docs/development.md +++ b/docs/development.md @@ -4,41 +4,16 @@ This is the first-read landing page for people and LLM coding agents working on Notarius. It provides a concise repository orientation and routes each kind of change to its canonical documentation. -## Orientation - -Notarius is a small Go application for extracting structured data from source -material. It is a general extraction platform with an initial Seriatim and D&D -implementation. Configured modules run through a fixed workflow: - -```text -input -> chunk -> extract -> merge -> normalize -> output -``` - -The CLI is the application boundary. Core packages own deterministic models and -policy, framework packages own reusable contracts and orchestration, and module -and validator packages own concrete behavior. - -## Repository Map - -- `cmd/notarius`: executable entry point. -- `internal/cli`: CLI behavior and production composition. -- `internal/core`: deterministic source, config, artifact, diagnostics, and - workspace packages. -- `internal/framework`: contracts, pipeline orchestration, validation helpers, - checkpoints, debug recording, and LLM runtime plumbing. -- `internal/modules`: concrete implementations of the six pipeline stages. -- `internal/validators`: concrete output validators. -- `docs`: canonical policy, reference, integration, internal, ADR, and roadmap - documentation. -- `examples`: maintained, secret-free example inputs and configuration. - -See [Internal Overview](internal/overview.md) for the implemented component -map and links to focused internal documentation. +Notarius is a Go CLI for configured structured extraction workflows. Start with +the [README](../README.md) for product context, [Architecture](policy/architecture.md) +for system boundaries, and [Internal Overview](internal/overview.md) for the +implemented component map. ## What To Read | When working on | Read | Why | | --- | --- | --- | +| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internals. | | Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. | | Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. | | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. | @@ -63,13 +38,3 @@ go test ./... go vet ./... go build ./cmd/notarius ``` - -## Universal Reminders - -- Keep non-roadmap documentation limited to implemented behavior. -- Update affected documentation and maintained examples in the same change as - behavior. -- Use fakes, fixtures, or local test servers instead of real external services - in tests. -- Do not expose secrets in code, errors, logs, diagnostics, manifests, - documentation, or examples. diff --git a/docs/internal/diagnostics.md b/docs/internal/diagnostics.md index 7878dad..b7ad032 100644 --- a/docs/internal/diagnostics.md +++ b/docs/internal/diagnostics.md @@ -1,102 +1,93 @@ # Diagnostics Internals -Diagnostics internals live in `internal/core/diagnostics`. Operator-facing run -behavior is documented in [Operations](../operations.md). - -## Purpose - -Diagnostics provide local inspection artifacts for a run without becoming the -durable output contract. Durable user output is produced by output modules and -written by the CLI. - -Diagnostics must not expose secrets. +`internal/core/diagnostics` provides the scoped writer and retention decision +used by `internal/cli`. The physical layout, artifact inventory, retention +semantics, failure inspection, and cleanup procedures are canonical in +[Operations](../operations.md#diagnostics-directory). Configuration fields and +defaults are canonical in [Configuration](../config.md#diagnostics). ## Run Directory -`NewRunDirectory(workDir, retention)` creates: +`NewRunDirectory` normalizes empty constructor inputs, creates the effective +diagnostics root when needed, and allocates a unique timestamp-based child +directory. It retries a bounded number of collisions before failing. The +resulting `RunDirectory` retains its creation time and retention mode for later +metadata and cleanup decisions. -```text -/run-/ -``` +The package does not resolve workspace configuration. `internal/cli` derives +effective workspace settings first and passes the diagnostics root into the +constructor. -If `workDir` is empty, it defaults to `/tmp/notarius`. Empty retention defaults -to `auto`. +## Scoped Writers -The CLI passes the effective diagnostics root from workspace configuration. -When `workspace.directory` is set and diagnostics are enabled, that root is -`/diagnostics`. The legacy diagnostics work directory and -`--diagnostics-dir` still pass a diagnostics-only root to this constructor. +Typed methods on `RunDirectory` write invocation metadata, redacted effective +configuration, resolved pipeline/reference data, checkpoint events, source data +when explicitly requested, manifests, reports, warnings, and error text. The +current filenames and their operator-facing contents are listed in +[Operations](../operations.md#diagnostics-directory). -The writer makes the work directory if needed, then attempts to create a unique -run directory. It retries run ID creation a bounded number of times if a -collision occurs. +JSON methods indent their payload and append a newline. All artifact writes use +a temporary file in the target directory, apply the requested permissions, and +rename it into place. Artifact resolution accepts only a single relative base +name; absolute paths, separators, and paths escaping the run directory fail +before writing. -## Artifact Writers +## Redacted Configuration -Implemented artifact names: +`WriteRedactedEffectiveConfig` accepts a `RedactedDiagnosticsPayload` provider +rather than a raw config value. `internal/core/config` implements that contract +by cloning effective config data and removing secret-shaped values before JSON +encoding. The diagnostics package therefore never needs configuration-specific +field knowledge. -- `invocation.json` -- `effective-config.json` -- `resolved-pipeline.json` -- `resolved-references.json` -- `checkpoint-events.json` -- `source-document.json` -- `run-manifest.json` -- `run-report.json` -- `warnings.json` -- `error.log` +## Retention Coordination -JSON artifacts are encoded with indentation and a trailing newline. Writes are -atomic through a temporary file in the target directory followed by rename. +`ShouldRetainRunDirectory` is a pure decision over the effective retention mode, +run success, and warning presence. `ApplyRetention` uses that result to remove +only its own run directory. Unsupported modes retain data as a fail-safe, though +normal CLI execution rejects them during config validation. -Artifact names must be single relative file names. Absolute paths, path -separators, and names resolving outside the run directory are rejected. +The meaning of each supported mode belongs in +[Operations](../operations.md#retention); this package implements that contract +without loading config or inspecting run artifacts. -## Redacted Effective Config +## CLI State Flow -Diagnostics writers accept payloads that implement -`RedactedDiagnosticsPayload`. `internal/core/config` uses this to redact API -keys in effective config diagnostics while preserving resolved pipeline context. +When diagnostics are enabled, `internal/cli` creates the run directory after +configuration loading and before pipeline resolution. It then writes artifacts +as state becomes available: invocation data, effective resolution data, +pipeline results, and the final report. This ordering permits later failures to +retain the context already established. -The redaction path clones config data before replacing secret values. +Failures before construction have no `RunDirectory`. Later failures write an +error log, preserve any available partial manifest, and apply a failed-run +retention decision. A diagnostics write failure is itself a command failure so +the CLI does not report success after losing requested inspection data. -## Retention +When diagnostics are disabled, the CLI carries a nil run directory and the +shared `writeDiagnostics` helper turns writes into no-ops. User-facing errors +still go to stderr; that invocation behavior is documented in +[Operations](../operations.md#failures). -Retention is decided by `ShouldRetainRunDirectory`. +## Package Guarantees -- Failed runs are always retained. -- `always` retains successful runs. -- `never` removes successful runs. -- `auto` retains successful runs only when warnings exist. -- Unknown retention values are treated as retain by the retention decision, but - config validation rejects unsupported values before normal runs. +- A `RunDirectory` writes and removes only within its allocated directory. +- JSON and error artifacts use atomic replacement. +- Nil receivers and invalid typed payloads return errors rather than panicking. +- Retention never removes a failed run and never targets the diagnostics root. +- Diagnostics models contain inspection metadata, not the durable output + contract. +- Checkpoint and debug serializers remain separate framework components. +- Secret-handling follows the invariant in + [Architecture](../policy/architecture.md#state-output-and-safety). -`ApplyRetention` removes only the specific run directory. +## Tests To Inspect -## CLI Failure Behavior - -When diagnostics are enabled, the CLI creates the diagnostics run directory -after config loading and before pipeline resolution. Failures before that point -do not have diagnostics. - -When workspace diagnostics are explicitly disabled, the CLI does not create a -diagnostics run directory and skips diagnostics artifact writes. Failures are -still printed to stderr. - -After diagnostics creation, run failures call `WriteErrorLog` and apply -retention with `RunSucceeded: false`, so the run directory remains available. - -When the pipeline returns a partial manifest on failure, the CLI writes that -manifest before logging the failure. - -## Invariants - -- Diagnostics paths must be narrow and run-directory scoped. -- Writes should be atomic where practical. -- Secrets must be redacted. -- Diagnostics write failures are command failures because they can hide the - information needed for recovery. -- Durable output file contracts belong to output modules and integration docs, - not to diagnostics. -- Checkpoint and debug workspace files are separate framework-owned artifacts, - not diagnostics artifacts. +- `internal/core/diagnostics/run_dir_test.go`: allocation, artifact confinement, + atomic writes, retention, and failure behavior. +- `internal/core/diagnostics/artifacts_test.go`: stable artifact identifiers. +- `internal/core/config/redaction_test.go`: clone-and-redact payload behavior. +- `internal/core/workspace/settings_test.go`: effective diagnostics-root and + enablement handoff. +- `internal/cli/run_test.go`: creation timing, artifact sequencing, disabled + diagnostics, overrides, failures, and retention integration. diff --git a/docs/internal/llm.md b/docs/internal/llm.md index e359bc5..c6cf7cf 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -1,114 +1,132 @@ -# LLM Runtime +# LLM Runtime Internals -The implemented LLM runtime lives in `internal/framework/llm`. It provides -transport-neutral structured completion contracts, a Scriptorium-backed -production client, concurrency scheduling, prompt/schema asset registration, -schema registry helpers, and secret redaction. +`internal/framework/llm` implements Notarius's transport boundary for structured +completion. It contains the Scriptorium adapter, concurrency scheduler, +prompt/schema registries, selected-profile recording, and provider-error +redaction. -## Contract +Provider-neutral ownership rules are defined in +[Architecture](../policy/architecture.md#llm-boundary). Profile sources, +credentials, and concurrency settings are defined in +[Configuration](../config.md). -Modules depend on `contracts.StructuredLLMClient`: +## Structured Contract -```go -CompleteStructured(ctx, request, out) (response, error) -``` +Modules and LLM-backed validators depend on +`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a +prompt and optional profile/session, supplies named input materials and +variables, and provides a caller-owned decoding target. A successful response +contains the validated raw structured bytes plus non-secret provider, model, +profile, and token metadata. -The request contains prompt ID/version, profile ID, session ID, prompt input -materials, and variables. The caller supplies a pointer target for decoded -structured output. The response also carries the raw structured output bytes -returned by the runtime so modules can preserve raw payloads in pipeline stage -outputs. +The caller owns prompt selection, response-schema selection, and interpretation +of the decoded result. `LLMInputMaterial` keeps source and reference bytes with +their origin metadata so the adapter can pass named artifacts to Scriptorium +without exposing Scriptorium types through stage contracts. -Modules that call the LLM own their prompts, schemas, prompt IDs, and -domain-specific interpretation. Validator packages own approve/reject policy, -and central catalog mappings decide which validators run by default. Provider -adapters should not contain domain-specific prompt logic. +## Production Construction -Prompt input materials carry source or reference bytes with optional origin -metadata. The Scriptorium-backed runtime receives them as named artifacts rather -than rendered prompt strings owned by Notarius modules. +`internal/cli` constructs the production runtime by: -## Production Client Construction +1. collecting embedded prompt and response-schema assets from production module + packages; +2. creating a `ScriptoriumClient` from the effective profile source; +3. attaching an `LLMProfileRecorder`; +4. creating a scheduler from the effective concurrency limit; +5. returning a `ScheduledClient` wrapper. -`internal/cli` builds the production LLM client from the effective config: - -1. collect production Scriptorium prompt and schema assets from module packages; -2. create a Scriptorium-backed structured client using effective Scriptorium - profile source settings from `scriptorium.profile_dir` or - `scriptorium.profile_file`; -3. create a scheduler from global LLM concurrency; -4. wrap the client with `NewScheduledClient`; -5. let the runtime report non-secret profile manifest metadata after calls. - -The runtime records the actual selected Scriptorium profile, provider, and model -used during execution. Manifest population does not rely on a precomputed -profile ID before pipeline execution. - -Explicit profile validation applies to LLM-capable pipeline stages: chunk, -extract, merge, normalize, and LLM-backed validators with explicit -`llm_profile` values. Input, output, and deterministic validators do not call -the LLM. The `--llm-profile` run flag overrides effective chunk, extract, merge, -and normalize bindings; it does not override validator-specific profiles. +The CLI separately gathers explicit profile IDs from resolved LLM-capable stage +and validator bindings. It prepares a small internal check prompt for each ID so +missing or invalid profiles fail before pipeline execution. The runtime profile +override syntax and scope are defined in the +[CLI reference](../cli.md#run); binding rules are defined in +[Configuration](../config.md#module-bindings). ## Scriptorium Adapter -`ScriptoriumClient` implements `contracts.StructuredLLMClient` by converting -Notarius prompt requests into Scriptorium `RunRequest` values. It: +`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`. +It validates the decoding target and prompt identity, maps named input materials +to inline artifacts, forwards explicit profile and session context, delegates +rendering/provider execution/structured validation, and unmarshals successful +JSON into the caller target. -- validates the caller output target and prompt ID; -- converts `LLMInputMaterial` values into inline Scriptorium artifacts, using a - single space for empty material so optional blank references remain explicit; -- passes `session_id` through Scriptorium variables and request metadata when - present; -- sends explicit profile IDs only when the request supplies one; -- lets Scriptorium render prompts, call the configured provider, and validate - structured output; -- unmarshals successful JSON into the caller-provided target; -- returns the validated raw structured output bytes to the caller; -- maps token usage and selected profile/model metadata into the Notarius - response and manifest profile recorder. +Empty optional input material is represented by a single space so Scriptorium +retains the named input. The client returns Scriptorium's validated structured +bytes rather than re-encoding the caller target, allowing modules to preserve +the runtime result exactly. -Generated-output validation failures are returned as Notarius errors. Provider -and runtime errors are wrapped with prompt context and bearer tokens are -redacted from error strings. Prompt text, raw source input, reference content, -schema JSON, API keys, and bearer tokens are not added to default diagnostics or -run manifests. +Selected profile, provider, model, and token metadata are mapped into the +Notarius response. The recorder deduplicates profiles by identity and supplies +manifest-safe profile summaries after actual calls; manifest population does +not guess the selected prompt default in advance. -## Scheduler +Generated-output validation failures and provider failures are wrapped with +prompt context. Error strings pass through bearer-token redaction before they +cross the runtime boundary. -`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a -FIFO queue of waiters. Cancellation removes queued waiters or releases granted -permits. +## Scheduling -`NewScheduledClient` wraps any structured LLM client and runs each completion -inside the scheduler. +`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate +acquisition increments the in-flight count; queued acquisition waits for a +permit or context cancellation. Cancellation removes a queued waiter, while a +cancelled waiter that has already received a permit releases it. -Effective concurrency is: +`ScheduledClient` acquires a permit around each structured completion and +defers release on every result path. The effective limit and default are +configuration facts in [Configuration](../config.md#defaults). -1. `concurrency.total_llm`, when greater than zero; -2. `1`. +## Prompt And Schema Assets -## Schema Registry +`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes +and rejects invalid or conflicting registrations. Production module packages +register their own prompt and schema assets; generic framework code contains no +D&D prompt content. -The framework schema registry embeds generic test schemas. It also exposes -helpers for caller-owned schemas: +Schema helpers load embedded JSON Schema with identity and digest metadata, +return defensive copies, and expose a diagnostics map that omits schema bytes. +The small framework registry contains only generic test schemas; production +schemas remain package-owned. -- `LoadResponseSchema` -- `LookupResponseSchema` -- `MustLookupResponseSchema` -- `ResponseSchema.DiagnosticsMap` +## Debug And Redaction Boundaries -`DiagnosticsMap` omits raw schema content and includes metadata such as key, -ID, version, name, and SHA-256. +The pipeline may wrap the client with a debug recorder that captures prepared +prompt/response material for an explicitly enabled debug run. Default +diagnostics and manifests receive identities, hashes, usage, and selected +profile summaries rather than prompt, source, reference, schema, or response +content. -Production modules own and register their Scriptorium prompt and schema assets. -Framework packages may collect those files but must not contain D&D-specific -prompt content. +The Scriptorium error wrapper removes bearer credential values from surfaced +provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known +secret values elsewhere in the runtime. Config diagnostics use a separate +clone-and-redact path in `internal/core/config`. These mechanisms implement the +security invariant in +[Architecture](../policy/architecture.md#state-output-and-safety); operator +handling of debug data is defined in [Operations](../operations.md#debug). -## Secret Redaction +## Failure Behavior -Provider errors are redacted before surfacing through the Scriptorium-backed -client. Config diagnostics use redacted effective config payloads. +- Invalid targets, missing prompt IDs, malformed structured output, and + Scriptorium failures return contextual errors to the calling module. +- Scheduler construction rejects non-positive limits; acquisition respects + context cancellation. +- Asset registration rejects invalid roots, missing content, and path conflicts. +- Schema loading distinguishes missing assets, invalid JSON, and invalid + metadata. +- Profile validation errors occur during CLI preparation when an explicit + selected ID cannot be prepared. -Do not add raw provider request bodies, response bodies, API keys, or prompt -payloads to diagnostics by default. +## Tests To Inspect + +- `internal/framework/llm/scriptorium_client_test.go` and + `scriptorium_api_test.go`: adapter mapping and local HTTP integration. +- `internal/framework/llm/scheduler_test.go` and + `scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper + release. +- `internal/framework/llm/asset_registry_test.go` and + `schema_registry_test.go`: asset composition, validation, and defensive + copies. +- `internal/framework/llm/secrets_test.go`: provider-error redaction. +- `internal/cli/run_test.go`: profile validation, production client wiring, + manifest recording, and debug integration. +- Module-local `scriptorium_assets_test.go` files: prompt inputs and package + asset registration. diff --git a/docs/internal/modules.md b/docs/internal/modules.md index 391493e..ceeea2c 100644 --- a/docs/internal/modules.md +++ b/docs/internal/modules.md @@ -1,268 +1,194 @@ -# Modules - -Production modules live under `internal/modules`. Each module implements one -contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and -registers itself with the matching pipeline registry. +# Module And Validator Internals -The CLI production catalog currently registers only the modules listed here. -Validator implementations live under `internal/validators` and are registered -separately from modules. Production default validator chains are central CLI -catalog policy; module packages do not own their default validation chains. +Production stage implementations live under `internal/modules`; production +validators live under `internal/validators`. The selectable keys, configuration +options, reference slots, and default validator chain are canonical in the +[module](../config.md#implemented-production-modules) and +[validator](../config.md#implemented-production-validators) catalogs in +Configuration. -## Contract Pattern +## Extension Pattern -A production module package should provide: +A stage module package provides a stable key, constructor, contract +implementation, `ModuleSpec`, `Register`, and focused behavior and registration +tests. A validator package follows the same pattern with `ValidatorSpec` and the +validator registry. -- a stable module key; -- a constructor such as `New`; -- the relevant contract implementation; -- `ModuleSpec`; -- `Register`; -- focused tests for registration, options, contract behavior, and errors. +Specs expose capability and execution metadata without constructing an +implementation. Chunk, extract, merge, and normalize modules that accept +auxiliary material declare identical reference slots from both +`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce +that agreement. Runtime delivery uses the corresponding stage request's +`References` field. -Module specs should describe capabilities accurately. Resolution uses specs to -reject incompatible pipelines before execution. +LLM-backed extensions own their prompt definitions and response schemas under +package-local embedded assets. Shared filesystem composition belongs in +`internal/modules/sharedassets`; reusable D&D prompt fragments, reference +declarations, prompt-input assembly, and source-unit helpers belong in +`internal/modules/sharedassets/dnd`. Stage contracts expose only Notarius +structured-completion types, not Scriptorium public types. -Chunk, extract, merge, and normalize modules that accept auxiliary reference material -must declare slots through both `ReferenceSlots()` and -`ModuleSpec().ReferenceSlots`. The runtime slot list and registry metadata -should match so config validation can inspect slots without constructing module -instances. A slot declaration names the slot, whether it is required, accepted -media types, whether multiple items are allowed, and any byte limit. Empty -`AcceptedMediaTypes` means any inferred media type is accepted, though the file -must still be UTF-8 text. When a slot declares accepted media types, Notarius -compares the canonical base media type inferred from the file extension, -case-insensitively and without parameters. +Reference material may inform a module or prompt but must not become source +evidence. The resolver and materializer behavior is described in +[Pipeline Internals](pipeline.md#reference-materialization). -The resolver materializes reference content for chunk, extractor, merger, and -normalizer targets. Runtime delivery uses `contracts.ChunkRequest.References`, -`contracts.ExtractionRequest.References`, `contracts.MergeRequest.References`, -and `contracts.NormalizeRequest.References`. Reference material is not source -evidence and must not be converted into `SourceRef` values. If a module prompt -uses references, pass them as prompt input materials through the structured LLM -request. Prompt metadata hashes remain based on prompt asset source, not -rendered reference bytes. +## Input Adapter -LLM-backed modules own Scriptorium prompt definitions and response schemas in -their embedded assets. Module-owned prompts live under each module's shallow -`assets/prompts` tree and schemas live under `assets/schemas`. Generic shared -prompt filesystem composition lives under `internal/modules/sharedassets`. -Common D&D prompt fragments, reference slot helpers, prompt input assembly, and -reference rendering live under `internal/modules/sharedassets/dnd`. Module -contracts should expose prompt IDs, versions, input material names, and -non-secret prompt/schema hashes through manifest metadata; they should not -expose Scriptorium public types through chunk, extract, merge, or normalize contracts. +### `internal/modules/input/seriatim` -Chunk modules receive the structured LLM client, configured Scriptorium profile -ID, prompt session ID, and raw source input material through -`contracts.ChunkRequest` when they need model-backed chunking. The pipeline -runner validates generic chunk result invariants before extraction; module-owned -policies may be stricter but must stay within the module package. +The adapter decodes the supported transcript JSON, selects the source identity, +computes the raw-input digest, validates segments, and maps each segment into a +generic source unit with speaker and timestamp metadata. Its spec advertises the +transcript capabilities consumed by D&D modules. -Normalize modules receive the structured LLM client, configured Scriptorium -profile ID, prompt session ID, and reference material through -`contracts.NormalizeRequest` when they need model-backed reconciliation. +Parsing is strict about required values and duplicate unit IDs but deliberately +ignores unrelated Seriatim fields. The external format and derived-identity +rules are defined in the +[Seriatim contract](../integrations/seriatim.md). -Merge modules receive the structured LLM client, configured Scriptorium profile -ID, prompt session ID, raw source input material, and reference material through -`contracts.MergeRequest` when they need model-backed merge behavior. +## Chunkers -## `seriatim` Input +### `internal/modules/chunk/generic` -Package: `internal/modules/input/seriatim` +The generic chunker validates the source document, walks units in configured +windows, clones each selected unit, and emits deterministic ordered chunk IDs. +Overlap changes the next window start but never reorders units. It records the +first and last unit and unit count in chunk metadata. -The `seriatim` adapter parses Seriatim transcript JSON into a generic source -document. It owns transcript JSON details, source ID selection, source digest -creation, transcript segment validation, and segment metadata mapping. +The accepted options and defaults are defined in +[Configuration](../config.md#implemented-production-modules). Generic +framework validation canonicalizes the returned unit slices before extraction. -Provides: +### `internal/modules/chunk/dnd/scenes` -- `source.transcript` -- `transcript.speaker` -- `transcript.timestamps` +The scene chunker prepares a structured Scriptorium request from the full +transcript, session, and optional D&D reference inputs. It validates the model's +scene boundaries against source-unit IDs and converts them into deterministic +chunks. -External JSON shape belongs in the Seriatim integration doc. +Scene validation requires sequential, contiguous, non-overlapping coverage from +the first source unit through the last. Each chunk contains JSON scene content +and module-owned metadata for the scene description, boundaries, confidence, +participants, and unit count. Boundary caveats become warnings. Malformed +structured output is returned as an error; there is no fallback chunker. -## `generic` Chunker +The package embeds its prompt and response schema and reports their non-secret +identity and hashes through singleton module metadata. Shared D&D assets supply +reference declarations and prompt inputs; their user-facing keys and accepted +file types remain canonical in [Configuration](../config.md). -Package: `internal/modules/chunk/generic` +## Extractor -The `generic` chunker splits source units into ordered chunks. It validates the -source document, clones source units, assigns chunk IDs such as `chunk-000001`, -and records chunk metadata for start unit, end unit, and unit count. +### `internal/modules/extract/dnd/spells` -The pipeline runner canonicalizes chunk units from the source document by -integer ID before extractors and mergers run. Chunkers also populate chunk -start and end unit IDs, content bytes, and media type. Chunker-owned context -should stay in `SourceChunk.Metadata`. +The spell extractor prepares a structured request from one chunk, the +chunk-scoped source input, the session, and optional D&D reference inputs. It +decodes the model response, assigns the generic source identity to every source +reference, canonicalizes duplicate references, orders spell casts by their +earliest cited unit, and returns raw JSON plus response-schema provenance. -Options: +The package owns its embedded prompt, response schemas, and prompt/schema +manifest metadata. Shared D&D helpers keep prompt input names and source-unit +reference conversion consistent with the scene chunker. The extractor produces +raw output; production validators own approval policy. -- `max_units`: positive integer, default `50`; -- `overlap_units`: non-negative integer, default `0`, and less than - `max_units`. +The durable payload and manifest metadata shapes are defined in the +[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md). -Provides: +## Merger And Normalizer -- `chunks` +### `internal/modules/merge/appendorder` -## `dnd/scenes` Chunker +The merger preserves extract-result order. It passes through one JSON result, +concatenates a common top-level array field across multiple JSON objects, and +otherwise emits an array of the decoded values. It rejects invalid JSON and +non-JSON media types, and it preserves compatible schema provenance. -Package: `internal/modules/chunk/dnd/scenes` +### `internal/modules/normalize/noop` -The `dnd/scenes` chunker uses the structured LLM client to divide transcript -source units into coherent D&D scenes. It supplies the embedded Scriptorium -prompt ID, prompt version, transcript input material, response schema, and -session ID to the runtime; validates model-authored source-unit boundaries; and -converts each scene into a deterministic source chunk. +The normalizer defensively clones the accepted merge result, including payload +bytes, metadata, warnings, and schema provenance, without changing its logical +content. -Its prompt definition lives under `assets/prompts` and its schema under -`assets/schemas`. Shared reusable D&D prompt fragments are provided by -`internal/modules/sharedassets/dnd` and referenced from prompt definitions under -`./sharedassets/`. +## Output Encoder -Requires: +### `internal/modules/output/json` -- `source.transcript` +The JSON encoder sorts normalized results by lane, derives collision-checked +safe logical names, pretty-prints JSON payloads, and assembles the logical index, +manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported +media types, unsafe names, and sanitized-name collisions are errors. -Provides: +The encoder returns logical files only. The CLI places them on disk, and the +[JSON output contract](../integrations/json-output.md) defines their external +paths and schemas. -- `chunks` -- `chunks.scenes` +## Generic Validators -Options: none. Non-empty options are rejected. +The unconditional accept and reject validators provide deterministic production +registrations used primarily for controlled composition and tests. -The chunker enforces full source-unit coverage from the first source unit to the -last, sequential contiguous scenes, and no overlap. Its LLM-facing schema uses -integer `start_unit_id` and `end_unit_id` values matching source-unit IDs. It -assigns chunk IDs such as `scene-000001`, emits JSON chunk content, and stores -scene metadata including title, primary mode, participants, summary, boundary -note, confidence, boundary unit IDs, and unit count. Boundary caveats become -warnings with reason code -`scene_boundary_caveat`. Whitespace-only caveats are treated as malformed -structured output rather than silently dropped. +The JSON syntax validator uses `encoding/json` to reject malformed payloads. The +JSON Schema validator requires schema bytes on the validation request, parses +the instance and schema with `jsonschema`, and distinguishes payload rejection +from schema loading or compilation errors. Neither validator calls the LLM. -Malformed model output fails explicitly rather than falling back to another -chunker. The chunker exposes prompt and response-schema provenance through -top-level `module_metadata.chunker` without raw prompts, raw schemas, source -text, or secrets. +## D&D Spell Validators -## `dnd/spells` Extractor +`internal/validators/extract/dnd/spells/spellpayload` provides strict decoding, +shape checks, source-reference candidates, and cited-text lookup shared by the +three validators. -Package: `internal/modules/extract/dnd/spells` +The shape validator rejects malformed JSON, unknown fields, missing or empty +spell fields, and empty reference lists. The source-reference validator applies +generic source-reference validation to every cited range. The relatedness +validator approves structurally valid payloads but warns when a case-insensitive +spell name is absent from all cited source text. It leaves malformed payloads to +the earlier validators in the configured chain. -The `dnd/spells` extractor owns D&D spell-cast extraction semantics. It -supplies the embedded Scriptorium prompt ID, prompt version, chunk-scoped -transcript input material, reference input materials, response schema, and -session ID to the runtime; then returns the structured LLM `spell_casts` -response as raw JSON. -Its LLM-facing source-reference schema uses integer `start_unit_id` and -`end_unit_id` values matching source-unit IDs. - -Its prompt definition lives under `assets/prompts` and its schema under -`assets/schemas`. Shared reusable D&D prompt fragments are provided by -`internal/modules/sharedassets/dnd` and referenced from prompt definitions under -`./sharedassets/`. - -Requires: - -- `chunks` -- `source.transcript` - -Provides: - -- `dnd.spell_casts` - -Response schema identity: - -- schema ID: `notarius.dnd.spells` -- schema name: `notarius_dnd_spells_v1` -- schema version: `v1` - -The extractor adds prompt and response-schema provenance to lane manifest -metadata under `artifact_lanes[].metadata.extractor`. Durable raw output -details belong in the -[D&D spell raw output contract](../integrations/dnd-spell-artifacts.md). - -The production catalog validates `dnd/spells` raw extract output with generic -JSON validators followed by D&D spell validators under -`internal/validators/extract/dnd/spells`. The extractor itself remains -responsible for prompt, schema, and raw output production rather than -approve/reject policy. - -The `dnd/scenes` chunker and `dnd/spells` extractor declare optional `players`, -`party`, and `glossary` reference slots accepting UTF-8 plain text, Markdown, -YAML, or JSON. They also accept `roster` as a deprecated compatibility alias for -`party`. Their prompts frame references as supporting disambiguation material -only; spell-cast artifacts must still be grounded in the source transcript. - -## `appendorder` Merger - -Package: `internal/modules/merge/appendorder` - -The `appendorder` merger preserves chunk order for raw extract outputs. A -single JSON extract output is passed through as the merge output. Multiple JSON -object outputs with one common top-level array field are merged by concatenating -that array field in chunk order. Other valid JSON shapes are merged as a JSON -array of decoded values in chunk order. Non-JSON media types and invalid JSON -are rejected. - -Provides: - -- `merged` - -## `noop` Normalizer - -Package: `internal/modules/normalize/noop` - -The `noop` normalizer clones the raw merge output and returns it unchanged. - -Requires: - -- `merged` - -Provides: - -- `normalized` - -## `json` Output - -Package: `internal/modules/output/json` - -The `json` output encoder converts normalized raw outputs, rejected raw outputs, -warnings, and the run manifest into logical JSON output files. It writes one -payload file per lane under `lanes/` and sanitizes lane IDs for file names. -Normalized output payloads must be valid `application/json`. - -Requires: - -- `normalized` - -Provides: - -- `encoded` - -Durable output file shapes belong in the -[JSON output contract](../integrations/json-output.md). Operator behavior -belongs in [Operations](../operations.md). +These validators are deterministic. Their selectable keys and production order +are defined in +[Configuration](../config.md#implemented-production-validators); their durable +payload rules are defined in the +[artifact contract](../integrations/dnd-spell-artifacts.md). ## Production Registration -Production registration is centralized in `internal/cli/catalog.go`. +`internal/cli/catalog.go` builds the production registries, registers module and +validator constructors, installs default validator-chain mappings, and exposes +the matching catalog for resolution. It also collects prompt assets from +LLM-backed packages before constructing the production client. -Do not make framework code import production modules. The CLI wires production -modules at the application boundary; tests may provide fake registries or fake -catalogs directly. +Framework packages must not import production extensions. Tests may compose +registries and catalogs directly with fakes. -## Adding A Module +## Adding An Extension -When adding a module, keep source-format and extraction-domain boundaries clear: +When adding a production module or validator: -- input modules may know external source formats; -- extract modules may know artifact semantics and prompt/schema assets; -- merge and normalize modules own raw output combination and reconciliation; -- output modules own serialization, not diagnostics or CLI reporting. +1. implement the stage or validator contract and package-local key; +2. expose and test its spec, constructor, and registration function; +3. keep format or domain parsing inside the concrete package; +4. add package-owned prompt/schema assets when the extension is LLM-backed; +5. register it in `internal/cli/catalog.go` and add a default chain only when + production policy requires one; +6. add resolution and composition coverage for capabilities, options, + references, and validation behavior; +7. update the selectable-key catalog in [Configuration](../config.md), the + relevant external contract, this inventory, and maintained examples when + user-visible behavior changes. -Update [Development](../development.md), [Configuration](../config.md), -internal docs, integration docs, and examples when the new module becomes -implemented production behavior. +Do not add the extension to `docs/development.md`; that file routes by task and +does not inventory implementations. + +## Tests To Inspect + +- Package-local `*_test.go` files under the module or validator being changed. +- `internal/framework/pipeline/registry_integration_test.go`: registry and spec + composition. +- `internal/framework/pipeline/default_modules_test.go`: framework binding + defaults. +- `internal/cli/run_test.go`: production catalog, config resolution, and + end-to-end CLI composition. +- `internal/modules/sharedassets/**/*_test.go`: shared prompt and reference + assembly. diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 1be9452..50d8e03 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -1,127 +1,105 @@ # Internal Overview -This document maps the implemented Notarius components and their ownership. It -complements the durable invariants in [Architecture](../policy/architecture.md) -and links to focused internal documentation for deeper behavior. +This document inventories the implemented Notarius components. Normative +boundaries and dependency direction belong in +[Architecture](../policy/architecture.md); external behavior belongs in the +[CLI](../cli.md), [Configuration](../config.md), +[Operations](../operations.md), and [integration contracts](../integrations/). ## Execution Path -The executable delegates to the CLI, which resolves configuration and wires the -production application around the framework runner: +`cmd/notarius` delegates to `internal/cli`, the production composition root. +The CLI loads configuration, builds the production catalogs and runtime +collaborators, invokes `internal/framework/pipeline`, and places the logical +output files returned by the runner. Diagnostics, checkpoints, and debug +recorders are optional side-channel collaborators supplied at this boundary. -```text -cmd/notarius - -> internal/cli - -> config resolution + production registries + LLM client - -> input -> chunk -> extract -> merge -> normalize -> output - -> durable output writes - -Pipeline side channels: - diagnostics checkpoints debug artifacts -``` - -Pipeline execution is serial. Configuration selects modules for the fixed stage -shape; registries construct them after profile, capability, validator, and -reference resolution. +Pipeline execution is serial. Resolution produces a fixed ordered workflow and +a sorted set of artifact lanes before the runner constructs any stage module. ## Application Boundary -`cmd/notarius` contains the executable entry point and delegates process exit -behavior to `internal/cli`. - -`internal/cli` owns command parsing, configuration discovery, production module -and validator registration, prompt asset collection, production LLM client -construction, reference preparation, workspace recorder setup, durable output -writes, and user-facing stdout, stderr, and exit codes. It is the composition -root for concrete production packages. +| Package | Implemented responsibility | +| --- | --- | +| `cmd/notarius` | Executable entry point and process exit delegation. | +| `internal/cli` | Command parsing, config discovery, production registration, prompt asset collection, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. | ## Core Packages | Package | Implemented responsibility | | --- | --- | -| `internal/core/artifacts` | Run manifests and artifact serialization shapes. | -| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline configuration. | -| `internal/core/diagnostics` | Diagnostics run directories, artifact writers, atomic writes, and retention decisions. | -| `internal/core/source` | Generic source documents, units, references, and validation. | -| `internal/core/workspace` | Workspace settings, safe paths and writes, checkpoint identities, and checkpoint manifest types. | - -These packages provide concrete, deterministic models and policy. Production -module registration occurs at the CLI boundary rather than in core packages. +| `internal/core/artifacts` | Run-manifest and provenance models. | +| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. | +| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. | +| `internal/core/source` | Generic source documents, units, references, lookup, and validation. | +| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. | ## Framework Packages | Package | Implemented responsibility | | --- | --- | -| `internal/framework/contracts` | Stage, validator, reference, output, and structured LLM interfaces and request/result types. | -| `internal/framework/pipeline` | Module registries, profile resolution, capability checks, reference materialization, validation chains, retries, orchestration, warnings, and manifest population. | +| `internal/framework/contracts` | Stage, validator, reference, output, and structured-completion interfaces and data types. | +| `internal/framework/pipeline` | Registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. | | `internal/framework/validate` | Shared validator decision and cardinality helpers. | -| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema asset registration, scheduling, profile recording, and secret redaction. | -| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload envelopes. | -| `internal/framework/debug` | Workspace-backed framework and LLM debug artifacts. | +| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. | +| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. | +| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. | -Framework contracts carry raw stage outputs between modules. The runner owns -provenance, validation sequencing, rejection handling, checkpoint boundaries, -debug boundaries, and final manifest assembly. +Framework contracts carry raw stage results between implementations. The +runner owns handoff provenance, validation sequencing, rejection handling, +checkpoint and debug boundaries, and final manifest assembly. -## Production Modules +## Production Extensions -Production implementations live under `internal/modules` and register through -the CLI catalog. +The canonical catalogs of user-selectable +[module](../config.md#implemented-production-modules) and +[validator](../config.md#implemented-production-validators) keys are in +Configuration. The implemented module packages are: -| Stage | Module key | Package | Role | -| --- | --- | --- | --- | -| Input | `seriatim` | `internal/modules/input/seriatim` | Converts Seriatim transcript JSON into the generic source model. | -| Chunk | `generic` | `internal/modules/chunk/generic` | Splits ordered source units by configured unit counts and overlap. | -| Chunk | `dnd/scenes` | `internal/modules/chunk/dnd/scenes` | Uses structured LLM output to create contiguous D&D scene chunks. | -| Extract | `dnd/spells` | `internal/modules/extract/dnd/spells` | Extracts source-grounded D&D spell-cast artifacts. | -| Merge | `appendorder` | `internal/modules/merge/appendorder` | Combines accepted extract outputs in chunk order. | -| Normalize | `noop` | `internal/modules/normalize/noop` | Preserves accepted merged output unchanged. | -| Output | `json` | `internal/modules/output/json` | Encodes manifests, indexes, warnings, rejections, and accepted lane payloads as logical JSON files. | +| Package | Implemented responsibility | +| --- | --- | +| `internal/modules/input/seriatim` | Parses the supported Seriatim transcript format into the generic source model. | +| `internal/modules/chunk/generic` | Splits ordered source units by unit count and overlap. | +| `internal/modules/chunk/dnd/scenes` | Produces contiguous D&D scene chunks from structured model output. | +| `internal/modules/extract/dnd/spells` | Produces source-grounded D&D spell-cast raw output. | +| `internal/modules/merge/appendorder` | Combines accepted extraction results in chunk order. | +| `internal/modules/normalize/noop` | Preserves accepted merged output. | +| `internal/modules/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. | `internal/modules/sharedassets` composes shared prompt filesystems. -`internal/modules/sharedassets/dnd` owns shared D&D prompt fragments, reference -slots, prompt input assembly, and source-unit reference helpers. +`internal/modules/sharedassets/dnd` owns reusable D&D prompt fragments, +reference declarations, prompt input assembly, and source-unit reference +helpers. -## Validators +Concrete validators live under `internal/validators`. Generic packages provide +unconditional test decisions, JSON syntax validation, and JSON Schema +validation. D&D spell packages provide shape, source-reference, and +source-relatedness decisions, with `spellpayload` holding their shared parser +and lookup helpers. Production chain composition is owned by `internal/cli`. -Concrete validators live under `internal/validators` and register separately -from stage modules. Generic validators cover unconditional test decisions, JSON -syntax, and JSON Schema. D&D spell validators cover artifact shape, source -reference validity, and source relatedness. +Implementation details for all production extensions are in +[Module Internals](modules.md). -The production default chain for `dnd/spells` extract output is registered -centrally in `internal/cli`; module packages produce raw output but do not own -the production approve/reject policy. +## Run-State Components -## Files And Run State - -Notarius keeps distinct output and inspection surfaces: - -| Surface | Owner | Purpose | +| Surface | Implemented owners | Internal purpose | | --- | --- | --- | -| Durable output | Output module and CLI writer | User-consumable run files. | -| Diagnostics | `internal/core/diagnostics` and CLI | Redacted run inspection, reports, warnings, and failures. | -| Checkpoints | `internal/framework/checkpoint` | Validated stage reuse for explicit resume. | -| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Sensitive framework-boundary and LLM call inspection. | +| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. | +| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. | +| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. | +| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. | -Workspace settings determine whether and where diagnostics, checkpoints, and -debug artifacts are written. Concrete stage modules do not receive workspace -paths. +Physical layout, retention, recovery, and sensitive-data handling are defined +in [Operations](../operations.md). Concrete stage modules receive recorder +interfaces and request data, not workspace paths. -## Focused Internal Documentation +## Focused Documentation -- [Pipeline Internals](pipeline.md): resolution, execution, validators, - references, retries, checkpoints, outputs, and manifests. -- [Module Internals](modules.md): production module contracts, capabilities, - options, prompts, schemas, and registration. +- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries, + checkpoint/debug hooks, and result assembly. +- [Module Internals](modules.md): production modules, validators, assets, + registration, and the contributor recipe for adding an extension. - [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter, assets, scheduling, profile recording, and redaction. -- [Diagnostics Internals](diagnostics.md): diagnostics files, retention, - failure behavior, and path safety. - -## Test Surfaces - -The repository uses focused package tests, registry and pipeline composition -tests, a fake-backed walking skeleton, fixture-driven CLI coverage, and local -test servers for LLM integration behavior. Tests do not require real provider -calls. +- [Diagnostics Internals](diagnostics.md): scoped writers, retention + coordination, CLI failure flow, and path safety. diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 2537706..3486e00 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -1,295 +1,185 @@ # Pipeline Internals -The implemented pipeline runner lives in `internal/framework/pipeline`. It -executes the fixed workflow defined by the architecture policy: +The implemented resolver and runner live in `internal/framework/pipeline`. +Their fixed workflow and ownership boundaries are defined by +[Architecture](../policy/architecture.md#system-shape). Configuration fields, +defaults, and selectable keys are defined in +[Configuration](../config.md#pipelines). -```text -input -> chunk -> extract -> merge -> normalize -> output -``` +Pipeline execution is serial. Resolution fixes the selected lanes and all +stage bindings before the runner constructs stage implementations. -Pipeline execution is serial. The runner executes the resolved lanes one after -another in the fixed workflow order. +## Resolution -## Profile Resolution +`internal/core/config.Config.Resolve` validates the loaded configuration, +selects the named profile, applies the runtime inputs supplied by the CLI, and +calls `pipeline.ResolvePipeline`. -Config loading produces `pipeline.PipelineProfile` values. Resolution happens -before execution: +`ResolvePipeline`: -1. `internal/core/config.Config.Resolve` validates config and finds the named - pipeline. -2. The optional lane selection is passed to `pipeline.ResolvePipeline`. -3. Module bindings are defaulted: - - chunk: `generic` - - merge: `appendorder` - - normalize: `noop` - - output: `json` - - LLM profile: empty, which lets Scriptorium prompt defaults choose a - profile. -4. The module catalog is checked for each bound module key. -5. Module capabilities are checked in workflow order. -6. A digest is calculated from the resolved pipeline without the digest field. +1. selects and sorts artifact lanes; +2. completes omitted bindings using the documented configuration defaults; +3. looks up each module and validator spec without constructing it; +4. checks required and provided capabilities in workflow order; +5. resolves target-aware reference bindings and validator chains; +6. calculates a digest over the resolved structure. -The CLI writes the resolved pipeline and digest to diagnostics. +Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete +bindings, validator chains, reference targets, and the digest. It does not read +reference bytes or construct runtime modules. CLI lane and reference selector +syntax is defined in the [CLI reference](../cli.md#run). -Pipeline profiles and artifact lanes may include reference binding maps keyed by -reference slot name. During resolution, pipeline-level bindings act as defaults -for selected chunk, extractor, merger, and normalizer targets that declare the -slot; target-local bindings override or add bindings for that target. Runtime -`--reference` requests override target config bindings, and runtime unbinds -remove optional target bindings. Flat runtime slot names are resolved only when -exactly one selected target declares the slot; otherwise the CLI requires a more -specific selector such as `chunk.slot`, `lane.extract.slot`, -`lane.merge.slot`, or `lane.normalize.slot`. Resolution validates bindings -against the declaring target specs and stores the bindings in target-aware -resolved reference holders. It does not read reference files or include -reference bytes in source digests. +## Reference Materialization -During run preparation, resolved file references for chunk, extractor, merger, -and normalizer targets are materialized before any LLM-backed pipeline work. Config -bindings resolve relative to the config file, and CLI bindings resolve relative -to the current working directory. Materialization accepts UTF-8 text files, -computes `sha256:` content digests, records file origins, infers canonical base -media types from file extensions, enforces declared byte limits, and warns for -empty bound files. Media-type acceptance is checked only when a slot declares -`AcceptedMediaTypes`; unknown extensions are recorded as -`application/octet-stream`. Reference content is omitted from diagnostics and -manifests. The CLI writes provenance-only resolved reference diagnostics, and -the run manifest records target-stage reference provenance separately from -source digests. Runtime reference content is passed to the matching chunker, -extractor, merger, or normalizer request. LLM-backed modules pass that material -onward as named Scriptorium prompt inputs. +The CLI calls `MaterializeReferences` after resolution and before constructing +the LLM client or running the pipeline. The materializer checks each binding +against its resolved target declaration, reads and validates the file, and +builds both a `contracts.ReferenceSet` and provenance-only metadata on the +corresponding `ResolvedReferenceTarget`. -The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse -those bytes into the source document. Chunk, merge, and normalize requests -receive the original source material as `SourceInput`; extraction requests -receive chunk-scoped source material built from the current `SourceChunk` -content, media type, and origin metadata. The raw input payload is not written -to manifests or default diagnostics. +The runner clones the resulting set into the chunk, extract, merge, or normalize +request that owns the target. LLM-backed extensions may convert those items into +named prompt inputs. Reference content remains separate from source evidence and +source digests. -The CLI also carries an optional run `session_id`. The runner makes it available -to chunk, extract, merge, and normalize requests; LLM-backed modules forward it -through their structured completion requests so Scriptorium can include it in -prompt execution metadata. +Binding precedence, path resolution, accepted content, and media-type behavior +are configuration contracts; see [Configuration](../config.md#pipelines). +Durable provenance is defined in the +[JSON output contract](../integrations/json-output.md#manifestjson), while +runtime sensitive-data handling belongs in [Operations](../operations.md). -When workspace resume checkpointing is enabled, the CLI constructs a checkpoint -recorder after pipeline resolution and reference materialization and passes it -through `pipeline.RunInput`. The runner records source, chunk, extract, merge, -and normalize outcomes through that interface. Concrete modules do not receive -workspace paths and do not write checkpoint files directly. +## Registries And Specs -For `run --resume`, the CLI also passes a checkpoint loader. The runner consults -the loader in workflow order and reuses only checkpoints whose manifest schema, -status, identity digest, dependency fingerprints, payload files, and payload -digests validate for the current invocation. The identity includes the resolved -pipeline, selected lanes, source/input digest, runtime overrides that affect -execution, and materialized reference digests. Missing or invalid checkpoints -fall back to normal execution and are refreshed by the recorder. +`pipeline.Registries` holds constructors used during execution. +`pipeline.ModuleCatalog` exposes their specs during configuration validation and +resolution. Separate registries exist for every stage and for validators; +`ValidatorChainRegistry` stores production default-chain mappings. -When workspace debug output is enabled, the CLI passes a debug recorder for the -current run ID. The runner writes framework-boundary inputs, outputs, -structured LLM calls, validator calls, timing, and retry attempt metadata -through that interface. Each retry or validator attempt records any LLM calls -made within that attempt in an `llm_calls` array and writes paired -`prompt-000N.json` and `response-000N.json` metadata files under the attempt -directory. LLM response bodies are written as sibling `response-content-000N.*` -files, using pretty-printed JSON when the content is valid JSON and raw text -otherwise. Debug output is not used for resume and can contain sensitive source, -reference, prompt, and model-output material. Concrete modules still do not -receive workspace paths. +A `ModuleSpec` declares its stage plus required and provided capabilities. +Chunk, extract, merge, and normalize specs may also declare reference slots. +Registry implementations defensively copy spec metadata, reject duplicate keys, +and verify that a constructed implementation reports the registered key. -## Registries And Module Specs +A `ValidatorSpec` declares a validator key and execution class. Resolution uses +the execution class to reject incompatible profile bindings before execution. +The current production catalog and default chain are listed only in +[Configuration](../config.md#implemented-production-validators). -`pipeline.Registries` holds concrete constructors for execution. A -`pipeline.ModuleCatalog` exposes module specs for config validation and -resolution. The catalog also exposes validator specs and central default -validator-chain mappings without constructing modules or validators. +## Runner Boundary -Every production module registers a `ModuleSpec` with: +`pipeline.RunInput` carries the resolved pipeline, raw source input, structured +LLM client, run identity and timing, optional session and profile metadata, and +checkpoint/debug collaborators. The runner parses source bytes through the +selected input adapter. Later stage requests receive the generic source model; +extract requests receive chunk-scoped input material, while chunk, merge, and +normalize requests retain access to the original source material. -- `Key`: module key used in config; -- `Stage`: module kind such as input, chunk, extract, merge, normalize, - validate, or output; -- `Provides`: capabilities added after that module runs; -- `Requires`: capabilities that must already be available. +`pipeline.RunOutput` carries the run manifest, accepted normalized results, +rejected results, warnings, checkpoint events, and logical files returned by the +output encoder. The CLI owns diagnostics and durable filesystem writes after the +runner returns. -Chunk, extract, merge, and normalize specs may also declare reference slots. Slot -declarations are available from registry metadata without constructing module -instances. Input, validate, and output specs must not declare reference slots. - -Capability checks prevent incompatible pipeline composition before a run starts. - -Every production validator registers a `ValidatorSpec` with: - -- `Key`: validator key used in config and manifests; -- `ExecutionClass`: `deterministic` or `llm_backed`. - -Default validator chains are keyed by workflow stage and module key. Production -currently registers a default chain for `extract` module `dnd/spells` only. - -## Runner Input And Output - -`pipeline.RunInput` carries: - -- a `ResolvedPipeline`; -- optional source ID, input path, and raw input bytes; -- a structured LLM client; -- run ID, start time, LLM profile manifest metadata, and CLI metadata. - -`pipeline.RunOutput` carries: - -- run manifest; -- normalized raw outputs; -- rejected raw outputs; -- warnings; -- logical output files returned by the output encoder. - -The CLI owns durable file writes and diagnostics writes after the runner returns. - -## Execution +## Execution Flow The runner: -1. validates run input and registries; -2. builds the input adapter and parses the raw input into a source document; -3. validates the source document; -4. builds the chunker and produces source chunks, retrying when configured; -5. validates source chunks against framework invariants and the resolved chunk - validator chain; -6. runs each selected artifact lane in sorted resolved order; -7. builds the output encoder and validates logical output file names. -8. passes accepted normalized raw outputs, rejected output records, warnings, - and the manifest to the output encoder. +1. validates its input and registries; +2. builds the input adapter, parses the raw input, and validates the generic + source document; +3. obtains or executes the chunk result; +4. validates and canonicalizes chunks; +5. executes each resolved artifact lane in order; +6. builds the output encoder and validates its logical file results; +7. returns the assembled manifest, outcomes, warnings, and files. -## Chunk Results +Within each artifact lane, it builds the extractor, merger, and normalizer, +then performs these transitions: -Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest` -with the validated source document, reference set, structured LLM client, the -configured LLM profile, module options, and run metadata. Deterministic and -LLM-backed chunkers use the same contract; provider construction stays outside -chunk modules. +1. extract once per accepted chunk and add runner-owned lane, source, and chunk + provenance; +2. validate each raw extract result and omit rejected results from merge input; +3. skip the rest of the lane when no extract result is accepted; +4. merge accepted extract results in their existing order; +5. validate the merge result and skip normalization on rejection; +6. normalize the accepted merge result; +7. validate and append the accepted normalized result. -When chunking succeeds, the runner validates generic chunk invariants before -running extractors: +Module-provided warnings and payload warnings are promoted only from attempts +whose results are accepted and used. -- chunk IDs must be non-empty and unique in the chunk result; -- each chunk `SourceID` must match the source document ID; -- each chunk `Index` must match its zero-based returned order; -- each chunk start and end unit ID must exist in the source document, with the - start unit at or before the end unit; -- each chunk must include non-empty extraction content and media type; -- each chunk must contain at least one source unit; -- a chunk must not repeat a source unit; -- every chunk source unit must exist in the source document; -- source units inside each chunk must appear in source-document order. +## Chunk Canonicalization -After validation, the runner rebuilds each chunk from source-document units by -integer ID, preserving chunk boundaries, content bytes, media type, and cloned -chunk metadata. Extractors and downstream stages therefore see canonical source -units, while `SourceChunk.Metadata` remains the supported place for -chunker-owned context. +Before lane execution, generic validation requires unique chunk IDs, matching +source identity, indexes matching returned order, valid ordered boundaries, +non-empty content and media type, and at least one valid source unit per chunk. +Units may not repeat inside a chunk and must preserve source-document order. -If chunk validation rejects the chunk result after configured retries, the runner -records a rejected raw output and skips downstream lane execution. Framework-level -chunking or validation errors that remain after configured retries fail the run. +The runner then rebuilds each chunk's unit slice from the source document by +unit ID. It preserves the module-owned boundaries, content, media type, and +cloned metadata. The framework permits gaps and overlap between separate +chunks; stricter coverage policy belongs to the chunk implementation. -The framework does not require complete source-unit coverage and does not reject -overlap between different chunks. Stricter policies, such as full coverage or -non-overlap, belong to individual chunk modules when they are part of that -module's contract. +## Validation And Retries -Within an artifact lane, the runner: +Chunk, extract, merge, and normalize results pass through the resolved validator +chain for their stage and module. Each validator receives the raw payload plus +the relevant source, chunk, prior-stage, schema, reference, session, LLM, option, +and run context. Validators execute in resolved order and stop at the first +error or rejection. An empty chain approves the result. -1. builds the extractor, merger, and normalizer; -2. records module manifest metadata when modules provide it; -3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when - configured; -4. fills runner-owned provenance on each extract output, including lane ID, - extractor key, source ID, chunk ID, and chunk index; -5. validates raw extract outputs and omits rejected outputs from merge input; -6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying - when configured; -7. validates raw merge output and skips normalization for rejected merge output; -8. normalizes the accepted merge output into one raw `NormalizeOutput`, - retrying when configured; -9. validates raw normalize output and appends accepted normalized raw output to - `RunOutput.NormalizeOutputs`. +`runWithRetry` performs the initial module call plus the configured additional +attempts. Each attempt includes module execution and its complete validation +chain. A module or validator error retries and becomes a framework error after +the final attempt. A rejection retries and becomes a recorded `RejectedOutput` +after the final attempt. Cancellation stops retry processing immediately. -## Validators +Rejected output is a non-fatal pipeline outcome and does not advance. Warnings +from discarded attempts are not promoted. Configuration owns retry counts and +validator overrides; see [Module Bindings](../config.md#module-bindings). -The runner handoff is raw-output based. Chunkers, extractors, mergers, and -normalizers do not advertise validator chains through their module interfaces. -Resolved validation chains receive the raw module output plus stage, lane, -module, source, chunk, schema, session, reference, LLM client/profile, binding -option, and run metadata context. Chunk validators receive the chunk result -collection, merge validators receive the ordered extract outputs used by the -merge, and normalize validators receive the accepted merge output. Empty chains -approve output by default. Response-schema provenance may include in-memory JSON -schema bytes for validators. Those bytes are omitted from manifests, -diagnostics, and encoded output files. +## Checkpoint And Debug Hooks -Resolved validator chains come from central default mappings unless a -stage-local config override is set on `chunk`, lane `extract`, lane `merge`, or -lane `normalize`. Explicit empty overrides are valid and are recorded as empty -chains in manifests. Explicit non-empty overrides replace the default chain and -preserve configured order. +The runner depends on recorder and loader interfaces, using no-op +implementations when collaborators are absent. Each checkpointed workflow +boundary records a running, succeeded, or failed transition. Reuse decisions +are consulted in workflow order and accepted payloads are cloned before +entering the normal handoff path. Dependency fingerprints connect later +checkpoints to the exact accepted results on which they depend. -The production default chain for `extract` module `dnd/spells` is: +Debug instrumentation wraps run, stage, attempt, validator, and structured LLM +boundaries. Context scopes associate nested LLM calls with the module or +validator attempt that made them. Debug-write failures are framework errors; +debug data is never used as a checkpoint source. -1. `generic/valid_json` -2. `generic/valid_json_schema` -3. `extract/dnd/spells/shape` -4. `extract/dnd/spells/source_refs` -5. `extract/dnd/spells/source_relatedness` +Checkpoint identity, physical layout, reuse behavior, and debug artifact +handling are operator contracts in [Operations](../operations.md). Serialization +and recorder implementation are inventoried in +[Internal Overview](overview.md#run-state-components). -No other production module currently has a default validator chain. +## Results And Failures -Validator rejection is a non-fatal run outcome: the rejected output is recorded -in `RunOutput.Rejected` and does not pass to the next stage. Validator execution -errors are framework-level errors and retry according to the relevant binding. -Warning-only validators return approved results with warnings; those warnings -are promoted only from successful attempts whose outputs are used. +The runner owns manifest assembly and handoff summaries but not the durable JSON +schema. It records resolved module and lane provenance, validator chains, +source/reference identities, selected LLM profiles, normalized and rejected +summaries, status, and timing. Raw payload bytes remain outside the manifest. +Module metadata providers may add non-secret singleton or lane-scoped metadata. -## Warnings And Failures +Execution errors include stage, module, lane, or validator context. Once a +manifest exists, a failing run returns it with failed status and completion +time. Successful status reflects whether any raw result was rejected. The +durable manifest and logical file schemas are defined in the +[JSON output contract](../integrations/json-output.md). -Warnings from the successful chunking, extraction, merging, and normalization -attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along -with output encoder warnings. Warnings from discarded retry attempts are not -promoted to final warnings. +## Tests To Inspect -Errors wrap the operation and module key or lane context. If execution fails -after a manifest exists, the returned manifest is marked `failed` and receives a -completion timestamp. - -On successful execution, the manifest validation status is: - -- `approved` when no raw outputs were rejected; -- `rejected` when at least one raw output was rejected. - -## Manifest Population - -The manifest records run ID, pipeline ID, pipeline digest, module keys, top-level -module metadata, artifact lanes, LLM profile metadata, source digest, -reference provenance, normalized raw output summaries, rejected output -summaries, validation status, and timing. Raw output summaries include lane ID, -normalizer module key, media type, source ID, and response-schema provenance -when present. Rejected output summaries include stage, lane, module, chunk, -validator or reason, message, attempt count, and optional diagnostic artifact -path. The manifest does not include raw output payload bytes. - -Singleton pipeline modules may add non-secret metadata by implementing -`contracts.ManifestMetadataProvider`. The runner records that metadata under -`module_metadata` with stable keys for `input`, `chunker`, and `output`. - -Lane-owned modules may add non-secret metadata through -`artifact_lanes[].metadata`. The runner records extractor, merger, and -normalizer metadata there. The D&D spell extractor uses lane metadata for -prompt and response-schema provenance. - -## JSON Output - -The production JSON output encoder writes `manifest.json`, `index.json`, -`warnings.json`, `rejected.json`, and one pretty-printed JSON file per accepted -normalized lane output under `lanes/`. It accepts only normalized outputs with -valid `application/json` payloads. Unsupported media types, invalid JSON, unsafe -logical paths, and duplicate sanitized lane file names fail the run before -durable output files are written. +- `internal/core/config/effective_config_test.go`: config-to-resolution boundary. +- `internal/framework/pipeline/profile_test.go`: selection, defaults, + capabilities, validator chains, and digest behavior. +- `internal/framework/pipeline/references_test.go`: target resolution and + materialization. +- `internal/framework/pipeline/runner_test.go`: stage transitions, retries, + rejections, warnings, checkpoints, debug hooks, and manifests. +- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete + workflow composition. +- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse + collaborators. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index d55a09a..9d1b878 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -32,21 +32,17 @@ Notarius is contract-first without being abstraction-heavy. Interfaces and extension points should protect demonstrated boundaries. New abstraction is not itself an architectural goal. -## Package Layout And Dependency Direction +## Layers And Dependency Direction -| Area | Ownership | -| --- | --- | -| `cmd/notarius` | Executable entry point; delegates to the CLI. | -| `internal/cli` | Application boundary, production composition, runtime setup, durable writes, and user-facing results. | -| `internal/core` | Generic deterministic models and policy for source material, configuration, manifests, diagnostics, and workspace identity. | -| `internal/framework` | Reusable contracts, registries, pipeline orchestration, validation mechanics, checkpoints, debug boundaries, and LLM runtime plumbing. | -| `internal/modules` | Concrete pipeline stage behavior. | -| `internal/validators` | Concrete approve/reject policies. | +The application boundary is the composition root and may depend on concrete +implementations. Domain-neutral model and framework layers provide reusable +policy, contracts, and orchestration. Concrete input, pipeline, output, and +validation extensions depend inward on those generic layers. -The CLI is the composition root and may import concrete implementations. Core -and framework packages cooperate as generic application layers; neither may -depend on production modules or validators. Concrete implementations may depend -on core models and framework contracts. +Generic layers must not depend on production extensions. Concrete extensions +must not compose the application or take ownership of process behavior. The +current packages implementing these layers are inventoried in +[Internal Overview](../internal/overview.md). The following dependency boundaries are mandatory: