eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refocus developer and internal documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-17 03:06:23 +00:00
parent b1fe9dc5a7
commit 6e6375521d
7 changed files with 549 additions and 785 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
docs/development.md
View File

@@ -4,41 +4,16 @@ This is the first-read landing page for people and LLM coding agents working on
Notarius. It provides a concise repository orientation and routes each kind of Notarius. It provides a concise repository orientation and routes each kind of
change to its canonical documentation. change to its canonical documentation.
## Orientation Notarius is a Go CLI for configured structured extraction workflows. Start with
the [README](../README.md) for product context, [Architecture](policy/architecture.md)
Notarius is a small Go application for extracting structured data from source for system boundaries, and [Internal Overview](internal/overview.md) for the
material. It is a general extraction platform with an initial Seriatim and D&D implemented component map.
implementation. Configured modules run through a fixed workflow:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
The CLI is the application boundary. Core packages own deterministic models and
policy, framework packages own reusable contracts and orchestration, and module
and validator packages own concrete behavior.
## Repository Map
- `cmd/notarius`: executable entry point.
- `internal/cli`: CLI behavior and production composition.
- `internal/core`: deterministic source, config, artifact, diagnostics, and
workspace packages.
- `internal/framework`: contracts, pipeline orchestration, validation helpers,
checkpoints, debug recording, and LLM runtime plumbing.
- `internal/modules`: concrete implementations of the six pipeline stages.
- `internal/validators`: concrete output validators.
- `docs`: canonical policy, reference, integration, internal, ADR, and roadmap
documentation.
- `examples`: maintained, secret-free example inputs and configuration.
See [Internal Overview](internal/overview.md) for the implemented component
map and links to focused internal documentation.
## What To Read ## What To Read
| When working on | Read | Why | | When working on | Read | Why |
| --- | --- | --- | | --- | --- | --- |
| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internals. |
| Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. | | Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. | | Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. | | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
@@ -63,13 +38,3 @@ go test ./...
go vet ./... go vet ./...
go build ./cmd/notarius go build ./cmd/notarius
``` ```
## Universal Reminders
- Keep non-roadmap documentation limited to implemented behavior.
- Update affected documentation and maintained examples in the same change as
behavior.
- Use fakes, fixtures, or local test servers instead of real external services
in tests.
- Do not expose secrets in code, errors, logs, diagnostics, manifests,
documentation, or examples.

151
docs/internal/diagnostics.md
View File

@@ -1,102 +1,93 @@
# Diagnostics Internals # Diagnostics Internals
Diagnostics internals live in `internal/core/diagnostics`. Operator-facing run `internal/core/diagnostics` provides the scoped writer and retention decision
behavior is documented in [Operations](../operations.md). used by `internal/cli`. The physical layout, artifact inventory, retention
semantics, failure inspection, and cleanup procedures are canonical in
## Purpose [Operations](../operations.md#diagnostics-directory). Configuration fields and
defaults are canonical in [Configuration](../config.md#diagnostics).
Diagnostics provide local inspection artifacts for a run without becoming the
durable output contract. Durable user output is produced by output modules and
written by the CLI.
Diagnostics must not expose secrets.
## Run Directory ## Run Directory
`NewRunDirectory(workDir, retention)` creates: `NewRunDirectory` normalizes empty constructor inputs, creates the effective
diagnostics root when needed, and allocates a unique timestamp-based child
directory. It retries a bounded number of collisions before failing. The
resulting `RunDirectory` retains its creation time and retention mode for later
metadata and cleanup decisions.
```text The package does not resolve workspace configuration. `internal/cli` derives
<workDir>/run-<unix-nanoseconds>/ effective workspace settings first and passes the diagnostics root into the
``` constructor.
If `workDir` is empty, it defaults to `/tmp/notarius`. Empty retention defaults ## Scoped Writers
to `auto`.
The CLI passes the effective diagnostics root from workspace configuration. Typed methods on `RunDirectory` write invocation metadata, redacted effective
When `workspace.directory` is set and diagnostics are enabled, that root is configuration, resolved pipeline/reference data, checkpoint events, source data
`<workspace.directory>/diagnostics`. The legacy diagnostics work directory and when explicitly requested, manifests, reports, warnings, and error text. The
`--diagnostics-dir` still pass a diagnostics-only root to this constructor. current filenames and their operator-facing contents are listed in
[Operations](../operations.md#diagnostics-directory).
The writer makes the work directory if needed, then attempts to create a unique JSON methods indent their payload and append a newline. All artifact writes use
run directory. It retries run ID creation a bounded number of times if a a temporary file in the target directory, apply the requested permissions, and
collision occurs. rename it into place. Artifact resolution accepts only a single relative base
name; absolute paths, separators, and paths escaping the run directory fail
before writing.
## Artifact Writers ## Redacted Configuration
Implemented artifact names: `WriteRedactedEffectiveConfig` accepts a `RedactedDiagnosticsPayload` provider
rather than a raw config value. `internal/core/config` implements that contract
by cloning effective config data and removing secret-shaped values before JSON
encoding. The diagnostics package therefore never needs configuration-specific
field knowledge.
- `invocation.json` ## Retention Coordination
- `effective-config.json`
- `resolved-pipeline.json`
- `resolved-references.json`
- `checkpoint-events.json`
- `source-document.json`
- `run-manifest.json`
- `run-report.json`
- `warnings.json`
- `error.log`
JSON artifacts are encoded with indentation and a trailing newline. Writes are `ShouldRetainRunDirectory` is a pure decision over the effective retention mode,
atomic through a temporary file in the target directory followed by rename. run success, and warning presence. `ApplyRetention` uses that result to remove
only its own run directory. Unsupported modes retain data as a fail-safe, though
normal CLI execution rejects them during config validation.
Artifact names must be single relative file names. Absolute paths, path The meaning of each supported mode belongs in
separators, and names resolving outside the run directory are rejected. [Operations](../operations.md#retention); this package implements that contract
without loading config or inspecting run artifacts.
## Redacted Effective Config ## CLI State Flow
Diagnostics writers accept payloads that implement When diagnostics are enabled, `internal/cli` creates the run directory after
`RedactedDiagnosticsPayload`. `internal/core/config` uses this to redact API configuration loading and before pipeline resolution. It then writes artifacts
keys in effective config diagnostics while preserving resolved pipeline context. as state becomes available: invocation data, effective resolution data,
pipeline results, and the final report. This ordering permits later failures to
retain the context already established.
The redaction path clones config data before replacing secret values. Failures before construction have no `RunDirectory`. Later failures write an
error log, preserve any available partial manifest, and apply a failed-run
retention decision. A diagnostics write failure is itself a command failure so
the CLI does not report success after losing requested inspection data.
## Retention When diagnostics are disabled, the CLI carries a nil run directory and the
shared `writeDiagnostics` helper turns writes into no-ops. User-facing errors
still go to stderr; that invocation behavior is documented in
[Operations](../operations.md#failures).
Retention is decided by `ShouldRetainRunDirectory`. ## Package Guarantees
- Failed runs are always retained. - A `RunDirectory` writes and removes only within its allocated directory.
- `always` retains successful runs. - JSON and error artifacts use atomic replacement.
- `never` removes successful runs. - Nil receivers and invalid typed payloads return errors rather than panicking.
- `auto` retains successful runs only when warnings exist. - Retention never removes a failed run and never targets the diagnostics root.
- Unknown retention values are treated as retain by the retention decision, but - Diagnostics models contain inspection metadata, not the durable output
config validation rejects unsupported values before normal runs. contract.
- Checkpoint and debug serializers remain separate framework components.
- Secret-handling follows the invariant in
[Architecture](../policy/architecture.md#state-output-and-safety).
`ApplyRetention` removes only the specific run directory. ## Tests To Inspect
## CLI Failure Behavior - `internal/core/diagnostics/run_dir_test.go`: allocation, artifact confinement,
atomic writes, retention, and failure behavior.
When diagnostics are enabled, the CLI creates the diagnostics run directory - `internal/core/diagnostics/artifacts_test.go`: stable artifact identifiers.
after config loading and before pipeline resolution. Failures before that point - `internal/core/config/redaction_test.go`: clone-and-redact payload behavior.
do not have diagnostics. - `internal/core/workspace/settings_test.go`: effective diagnostics-root and
enablement handoff.
When workspace diagnostics are explicitly disabled, the CLI does not create a - `internal/cli/run_test.go`: creation timing, artifact sequencing, disabled
diagnostics run directory and skips diagnostics artifact writes. Failures are diagnostics, overrides, failures, and retention integration.
still printed to stderr.
After diagnostics creation, run failures call `WriteErrorLog` and apply
retention with `RunSucceeded: false`, so the run directory remains available.
When the pipeline returns a partial manifest on failure, the CLI writes that
manifest before logging the failure.
## Invariants
- Diagnostics paths must be narrow and run-directory scoped.
- Writes should be atomic where practical.
- Secrets must be redacted.
- Diagnostics write failures are command failures because they can hide the
information needed for recovery.
- Durable output file contracts belong to output modules and integration docs,
not to diagnostics.
- Checkpoint and debug workspace files are separate framework-owned artifacts,
not diagnostics artifacts.

192
docs/internal/llm.md
View File

@@ -1,114 +1,132 @@
# LLM Runtime # LLM Runtime Internals
The implemented LLM runtime lives in `internal/framework/llm`. It provides `internal/framework/llm` implements Notarius's transport boundary for structured
transport-neutral structured completion contracts, a Scriptorium-backed completion. It contains the Scriptorium adapter, concurrency scheduler,
production client, concurrency scheduling, prompt/schema asset registration, prompt/schema registries, selected-profile recording, and provider-error
schema registry helpers, and secret redaction. redaction.
## Contract Provider-neutral ownership rules are defined in
[Architecture](../policy/architecture.md#llm-boundary). Profile sources,
credentials, and concurrency settings are defined in
[Configuration](../config.md).
Modules depend on `contracts.StructuredLLMClient`: ## Structured Contract
```go Modules and LLM-backed validators depend on
CompleteStructured(ctx, request, out) (response, error) `contracts.StructuredLLMClient.CompleteStructured`. A request identifies a
``` prompt and optional profile/session, supplies named input materials and
variables, and provides a caller-owned decoding target. A successful response
contains the validated raw structured bytes plus non-secret provider, model,
profile, and token metadata.
The request contains prompt ID/version, profile ID, session ID, prompt input The caller owns prompt selection, response-schema selection, and interpretation
materials, and variables. The caller supplies a pointer target for decoded of the decoded result. `LLMInputMaterial` keeps source and reference bytes with
structured output. The response also carries the raw structured output bytes their origin metadata so the adapter can pass named artifacts to Scriptorium
returned by the runtime so modules can preserve raw payloads in pipeline stage without exposing Scriptorium types through stage contracts.
outputs.
Modules that call the LLM own their prompts, schemas, prompt IDs, and ## Production Construction
domain-specific interpretation. Validator packages own approve/reject policy,
and central catalog mappings decide which validators run by default. Provider
adapters should not contain domain-specific prompt logic.
Prompt input materials carry source or reference bytes with optional origin `internal/cli` constructs the production runtime by:
metadata. The Scriptorium-backed runtime receives them as named artifacts rather
than rendered prompt strings owned by Notarius modules.
## Production Client Construction 1. collecting embedded prompt and response-schema assets from production module
packages;
2. creating a `ScriptoriumClient` from the effective profile source;
3. attaching an `LLMProfileRecorder`;
4. creating a scheduler from the effective concurrency limit;
5. returning a `ScheduledClient` wrapper.
`internal/cli` builds the production LLM client from the effective config: The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
and validator bindings. It prepares a small internal check prompt for each ID so
1. collect production Scriptorium prompt and schema assets from module packages; missing or invalid profiles fail before pipeline execution. The runtime profile
2. create a Scriptorium-backed structured client using effective Scriptorium override syntax and scope are defined in the
profile source settings from `scriptorium.profile_dir` or [CLI reference](../cli.md#run); binding rules are defined in
`scriptorium.profile_file`; [Configuration](../config.md#module-bindings).
3. create a scheduler from global LLM concurrency;
4. wrap the client with `NewScheduledClient`;
5. let the runtime report non-secret profile manifest metadata after calls.
The runtime records the actual selected Scriptorium profile, provider, and model
used during execution. Manifest population does not rely on a precomputed
profile ID before pipeline execution.
Explicit profile validation applies to LLM-capable pipeline stages: chunk,
extract, merge, normalize, and LLM-backed validators with explicit
`llm_profile` values. Input, output, and deterministic validators do not call
the LLM. The `--llm-profile` run flag overrides effective chunk, extract, merge,
and normalize bindings; it does not override validator-specific profiles.
## Scriptorium Adapter ## Scriptorium Adapter
`ScriptoriumClient` implements `contracts.StructuredLLMClient` by converting `ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`.
Notarius prompt requests into Scriptorium `RunRequest` values. It: It validates the decoding target and prompt identity, maps named input materials
to inline artifacts, forwards explicit profile and session context, delegates
rendering/provider execution/structured validation, and unmarshals successful
JSON into the caller target.
- validates the caller output target and prompt ID; Empty optional input material is represented by a single space so Scriptorium
- converts `LLMInputMaterial` values into inline Scriptorium artifacts, using a retains the named input. The client returns Scriptorium's validated structured
single space for empty material so optional blank references remain explicit; bytes rather than re-encoding the caller target, allowing modules to preserve
- passes `session_id` through Scriptorium variables and request metadata when the runtime result exactly.
present;
- sends explicit profile IDs only when the request supplies one;
- lets Scriptorium render prompts, call the configured provider, and validate
structured output;
- unmarshals successful JSON into the caller-provided target;
- returns the validated raw structured output bytes to the caller;
- maps token usage and selected profile/model metadata into the Notarius
response and manifest profile recorder.
Generated-output validation failures are returned as Notarius errors. Provider Selected profile, provider, model, and token metadata are mapped into the
and runtime errors are wrapped with prompt context and bearer tokens are Notarius response. The recorder deduplicates profiles by identity and supplies
redacted from error strings. Prompt text, raw source input, reference content, manifest-safe profile summaries after actual calls; manifest population does
schema JSON, API keys, and bearer tokens are not added to default diagnostics or not guess the selected prompt default in advance.
run manifests.
## Scheduler Generated-output validation failures and provider failures are wrapped with
prompt context. Error strings pass through bearer-token redaction before they
cross the runtime boundary.
`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a ## Scheduling
FIFO queue of waiters. Cancellation removes queued waiters or releases granted
permits.
`NewScheduledClient` wraps any structured LLM client and runs each completion `Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate
inside the scheduler. acquisition increments the in-flight count; queued acquisition waits for a
permit or context cancellation. Cancellation removes a queued waiter, while a
cancelled waiter that has already received a permit releases it.
Effective concurrency is: `ScheduledClient` acquires a permit around each structured completion and
defers release on every result path. The effective limit and default are
configuration facts in [Configuration](../config.md#defaults).
1. `concurrency.total_llm`, when greater than zero; ## Prompt And Schema Assets
2. `1`.
## Schema Registry `AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
and rejects invalid or conflicting registrations. Production module packages
register their own prompt and schema assets; generic framework code contains no
D&D prompt content.
The framework schema registry embeds generic test schemas. It also exposes Schema helpers load embedded JSON Schema with identity and digest metadata,
helpers for caller-owned schemas: return defensive copies, and expose a diagnostics map that omits schema bytes.
The small framework registry contains only generic test schemas; production
schemas remain package-owned.
- `LoadResponseSchema` ## Debug And Redaction Boundaries
- `LookupResponseSchema`
- `MustLookupResponseSchema`
- `ResponseSchema.DiagnosticsMap`
`DiagnosticsMap` omits raw schema content and includes metadata such as key, The pipeline may wrap the client with a debug recorder that captures prepared
ID, version, name, and SHA-256. prompt/response material for an explicitly enabled debug run. Default
diagnostics and manifests receive identities, hashes, usage, and selected
profile summaries rather than prompt, source, reference, schema, or response
content.
Production modules own and register their Scriptorium prompt and schema assets. The Scriptorium error wrapper removes bearer credential values from surfaced
Framework packages may collect those files but must not contain D&D-specific provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known
prompt content. secret values elsewhere in the runtime. Config diagnostics use a separate
clone-and-redact path in `internal/core/config`. These mechanisms implement the
security invariant in
[Architecture](../policy/architecture.md#state-output-and-safety); operator
handling of debug data is defined in [Operations](../operations.md#debug).
## Secret Redaction ## Failure Behavior
Provider errors are redacted before surfacing through the Scriptorium-backed - Invalid targets, missing prompt IDs, malformed structured output, and
client. Config diagnostics use redacted effective config payloads. Scriptorium failures return contextual errors to the calling module.
- Scheduler construction rejects non-positive limits; acquisition respects
context cancellation.
- Asset registration rejects invalid roots, missing content, and path conflicts.
- Schema loading distinguishes missing assets, invalid JSON, and invalid
metadata.
- Profile validation errors occur during CLI preparation when an explicit
selected ID cannot be prepared.
Do not add raw provider request bodies, response bodies, API keys, or prompt ## Tests To Inspect
payloads to diagnostics by default.
- `internal/framework/llm/scriptorium_client_test.go` and
`scriptorium_api_test.go`: adapter mapping and local HTTP integration.
- `internal/framework/llm/scheduler_test.go` and
`scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper
release.
- `internal/framework/llm/asset_registry_test.go` and
`schema_registry_test.go`: asset composition, validation, and defensive
copies.
- `internal/framework/llm/secrets_test.go`: provider-error redaction.
- `internal/cli/run_test.go`: profile validation, production client wiring,
manifest recording, and debug integration.
- Module-local `scriptorium_assets_test.go` files: prompt inputs and package
asset registration.

368
docs/internal/modules.md
View File

@@ -1,268 +1,194 @@
# Modules # Module And Validator Internals
Production modules live under `internal/modules`. Each module implements one
contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and
registers itself with the matching pipeline registry.
The CLI production catalog currently registers only the modules listed here. Production stage implementations live under `internal/modules`; production
Validator implementations live under `internal/validators` and are registered validators live under `internal/validators`. The selectable keys, configuration
separately from modules. Production default validator chains are central CLI options, reference slots, and default validator chain are canonical in the
catalog policy; module packages do not own their default validation chains. [module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) catalogs in
Configuration.
## Contract Pattern ## Extension Pattern
A production module package should provide: A stage module package provides a stable key, constructor, contract
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
tests. A validator package follows the same pattern with `ValidatorSpec` and the
validator registry.
- a stable module key; Specs expose capability and execution metadata without constructing an
- a constructor such as `New`; implementation. Chunk, extract, merge, and normalize modules that accept
- the relevant contract implementation; auxiliary material declare identical reference slots from both
- `ModuleSpec`; `ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
- `Register`; that agreement. Runtime delivery uses the corresponding stage request's
- focused tests for registration, options, contract behavior, and errors. `References` field.
Module specs should describe capabilities accurately. Resolution uses specs to LLM-backed extensions own their prompt definitions and response schemas under
reject incompatible pipelines before execution. package-local embedded assets. Shared filesystem composition belongs in
`internal/modules/sharedassets`; reusable D&D prompt fragments, reference
declarations, prompt-input assembly, and source-unit helpers belong in
`internal/modules/sharedassets/dnd`. Stage contracts expose only Notarius
structured-completion types, not Scriptorium public types.
Chunk, extract, merge, and normalize modules that accept auxiliary reference material Reference material may inform a module or prompt but must not become source
must declare slots through both `ReferenceSlots()` and evidence. The resolver and materializer behavior is described in
`ModuleSpec().ReferenceSlots`. The runtime slot list and registry metadata [Pipeline Internals](pipeline.md#reference-materialization).
should match so config validation can inspect slots without constructing module
instances. A slot declaration names the slot, whether it is required, accepted
media types, whether multiple items are allowed, and any byte limit. Empty
`AcceptedMediaTypes` means any inferred media type is accepted, though the file
must still be UTF-8 text. When a slot declares accepted media types, Notarius
compares the canonical base media type inferred from the file extension,
case-insensitively and without parameters.
The resolver materializes reference content for chunk, extractor, merger, and ## Input Adapter
normalizer targets. Runtime delivery uses `contracts.ChunkRequest.References`,
`contracts.ExtractionRequest.References`, `contracts.MergeRequest.References`,
and `contracts.NormalizeRequest.References`. Reference material is not source
evidence and must not be converted into `SourceRef` values. If a module prompt
uses references, pass them as prompt input materials through the structured LLM
request. Prompt metadata hashes remain based on prompt asset source, not
rendered reference bytes.
LLM-backed modules own Scriptorium prompt definitions and response schemas in ### `internal/modules/input/seriatim`
their embedded assets. Module-owned prompts live under each module's shallow
`assets/prompts` tree and schemas live under `assets/schemas`. Generic shared
prompt filesystem composition lives under `internal/modules/sharedassets`.
Common D&D prompt fragments, reference slot helpers, prompt input assembly, and
reference rendering live under `internal/modules/sharedassets/dnd`. Module
contracts should expose prompt IDs, versions, input material names, and
non-secret prompt/schema hashes through manifest metadata; they should not
expose Scriptorium public types through chunk, extract, merge, or normalize contracts.
Chunk modules receive the structured LLM client, configured Scriptorium profile The adapter decodes the supported transcript JSON, selects the source identity,
ID, prompt session ID, and raw source input material through computes the raw-input digest, validates segments, and maps each segment into a
`contracts.ChunkRequest` when they need model-backed chunking. The pipeline generic source unit with speaker and timestamp metadata. Its spec advertises the
runner validates generic chunk result invariants before extraction; module-owned transcript capabilities consumed by D&D modules.
policies may be stricter but must stay within the module package.
Normalize modules receive the structured LLM client, configured Scriptorium Parsing is strict about required values and duplicate unit IDs but deliberately
profile ID, prompt session ID, and reference material through ignores unrelated Seriatim fields. The external format and derived-identity
`contracts.NormalizeRequest` when they need model-backed reconciliation. rules are defined in the
[Seriatim contract](../integrations/seriatim.md).
Merge modules receive the structured LLM client, configured Scriptorium profile ## Chunkers
ID, prompt session ID, raw source input material, and reference material through
`contracts.MergeRequest` when they need model-backed merge behavior.
## `seriatim` Input ### `internal/modules/chunk/generic`
Package: `internal/modules/input/seriatim` The generic chunker validates the source document, walks units in configured
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
Overlap changes the next window start but never reorders units. It records the
first and last unit and unit count in chunk metadata.
The `seriatim` adapter parses Seriatim transcript JSON into a generic source The accepted options and defaults are defined in
document. It owns transcript JSON details, source ID selection, source digest [Configuration](../config.md#implemented-production-modules). Generic
creation, transcript segment validation, and segment metadata mapping. framework validation canonicalizes the returned unit slices before extraction.
Provides: ### `internal/modules/chunk/dnd/scenes`
- `source.transcript` The scene chunker prepares a structured Scriptorium request from the full
- `transcript.speaker` transcript, session, and optional D&D reference inputs. It validates the model's
- `transcript.timestamps` scene boundaries against source-unit IDs and converts them into deterministic
chunks.
External JSON shape belongs in the Seriatim integration doc. Scene validation requires sequential, contiguous, non-overlapping coverage from
the first source unit through the last. Each chunk contains JSON scene content
and module-owned metadata for the scene description, boundaries, confidence,
participants, and unit count. Boundary caveats become warnings. Malformed
structured output is returned as an error; there is no fallback chunker.
## `generic` Chunker The package embeds its prompt and response schema and reports their non-secret
identity and hashes through singleton module metadata. Shared D&D assets supply
reference declarations and prompt inputs; their user-facing keys and accepted
file types remain canonical in [Configuration](../config.md).
Package: `internal/modules/chunk/generic` ## Extractor
The `generic` chunker splits source units into ordered chunks. It validates the ### `internal/modules/extract/dnd/spells`
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
and records chunk metadata for start unit, end unit, and unit count.
The pipeline runner canonicalizes chunk units from the source document by The spell extractor prepares a structured request from one chunk, the
integer ID before extractors and mergers run. Chunkers also populate chunk chunk-scoped source input, the session, and optional D&D reference inputs. It
start and end unit IDs, content bytes, and media type. Chunker-owned context decodes the model response, assigns the generic source identity to every source
should stay in `SourceChunk.Metadata`. reference, canonicalizes duplicate references, orders spell casts by their
earliest cited unit, and returns raw JSON plus response-schema provenance.
Options: The package owns its embedded prompt, response schemas, and prompt/schema
manifest metadata. Shared D&D helpers keep prompt input names and source-unit
reference conversion consistent with the scene chunker. The extractor produces
raw output; production validators own approval policy.
- `max_units`: positive integer, default `50`; The durable payload and manifest metadata shapes are defined in the
- `overlap_units`: non-negative integer, default `0`, and less than [D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
`max_units`.
Provides: ## Merger And Normalizer
- `chunks` ### `internal/modules/merge/appendorder`
## `dnd/scenes` Chunker The merger preserves extract-result order. It passes through one JSON result,
concatenates a common top-level array field across multiple JSON objects, and
otherwise emits an array of the decoded values. It rejects invalid JSON and
non-JSON media types, and it preserves compatible schema provenance.
Package: `internal/modules/chunk/dnd/scenes` ### `internal/modules/normalize/noop`
The `dnd/scenes` chunker uses the structured LLM client to divide transcript The normalizer defensively clones the accepted merge result, including payload
source units into coherent D&D scenes. It supplies the embedded Scriptorium bytes, metadata, warnings, and schema provenance, without changing its logical
prompt ID, prompt version, transcript input material, response schema, and content.
session ID to the runtime; validates model-authored source-unit boundaries; and
converts each scene into a deterministic source chunk.
Its prompt definition lives under `assets/prompts` and its schema under ## Output Encoder
`assets/schemas`. Shared reusable D&D prompt fragments are provided by
`internal/modules/sharedassets/dnd` and referenced from prompt definitions under
`./sharedassets/`.
Requires: ### `internal/modules/output/json`
- `source.transcript` The JSON encoder sorts normalized results by lane, derives collision-checked
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
media types, unsafe names, and sanitized-name collisions are errors.
Provides: The encoder returns logical files only. The CLI places them on disk, and the
[JSON output contract](../integrations/json-output.md) defines their external
paths and schemas.
- `chunks` ## Generic Validators
- `chunks.scenes`
Options: none. Non-empty options are rejected. The unconditional accept and reject validators provide deterministic production
registrations used primarily for controlled composition and tests.
The chunker enforces full source-unit coverage from the first source unit to the The JSON syntax validator uses `encoding/json` to reject malformed payloads. The
last, sequential contiguous scenes, and no overlap. Its LLM-facing schema uses JSON Schema validator requires schema bytes on the validation request, parses
integer `start_unit_id` and `end_unit_id` values matching source-unit IDs. It the instance and schema with `jsonschema`, and distinguishes payload rejection
assigns chunk IDs such as `scene-000001`, emits JSON chunk content, and stores from schema loading or compilation errors. Neither validator calls the LLM.
scene metadata including title, primary mode, participants, summary, boundary
note, confidence, boundary unit IDs, and unit count. Boundary caveats become
warnings with reason code
`scene_boundary_caveat`. Whitespace-only caveats are treated as malformed
structured output rather than silently dropped.
Malformed model output fails explicitly rather than falling back to another ## D&D Spell Validators
chunker. The chunker exposes prompt and response-schema provenance through
top-level `module_metadata.chunker` without raw prompts, raw schemas, source
text, or secrets.
## `dnd/spells` Extractor `internal/validators/extract/dnd/spells/spellpayload` provides strict decoding,
shape checks, source-reference candidates, and cited-text lookup shared by the
three validators.
Package: `internal/modules/extract/dnd/spells` The shape validator rejects malformed JSON, unknown fields, missing or empty
spell fields, and empty reference lists. The source-reference validator applies
generic source-reference validation to every cited range. The relatedness
validator approves structurally valid payloads but warns when a case-insensitive
spell name is absent from all cited source text. It leaves malformed payloads to
the earlier validators in the configured chain.
The `dnd/spells` extractor owns D&D spell-cast extraction semantics. It These validators are deterministic. Their selectable keys and production order
supplies the embedded Scriptorium prompt ID, prompt version, chunk-scoped are defined in
transcript input material, reference input materials, response schema, and [Configuration](../config.md#implemented-production-validators); their durable
session ID to the runtime; then returns the structured LLM `spell_casts` payload rules are defined in the
response as raw JSON. [artifact contract](../integrations/dnd-spell-artifacts.md).
Its LLM-facing source-reference schema uses integer `start_unit_id` and
`end_unit_id` values matching source-unit IDs.
Its prompt definition lives under `assets/prompts` and its schema under
`assets/schemas`. Shared reusable D&D prompt fragments are provided by
`internal/modules/sharedassets/dnd` and referenced from prompt definitions under
`./sharedassets/`.
Requires:
- `chunks`
- `source.transcript`
Provides:
- `dnd.spell_casts`
Response schema identity:
- schema ID: `notarius.dnd.spells`
- schema name: `notarius_dnd_spells_v1`
- schema version: `v1`
The extractor adds prompt and response-schema provenance to lane manifest
metadata under `artifact_lanes[].metadata.extractor`. Durable raw output
details belong in the
[D&D spell raw output contract](../integrations/dnd-spell-artifacts.md).
The production catalog validates `dnd/spells` raw extract output with generic
JSON validators followed by D&D spell validators under
`internal/validators/extract/dnd/spells`. The extractor itself remains
responsible for prompt, schema, and raw output production rather than
approve/reject policy.
The `dnd/scenes` chunker and `dnd/spells` extractor declare optional `players`,
`party`, and `glossary` reference slots accepting UTF-8 plain text, Markdown,
YAML, or JSON. They also accept `roster` as a deprecated compatibility alias for
`party`. Their prompts frame references as supporting disambiguation material
only; spell-cast artifacts must still be grounded in the source transcript.
## `appendorder` Merger
Package: `internal/modules/merge/appendorder`
The `appendorder` merger preserves chunk order for raw extract outputs. A
single JSON extract output is passed through as the merge output. Multiple JSON
object outputs with one common top-level array field are merged by concatenating
that array field in chunk order. Other valid JSON shapes are merged as a JSON
array of decoded values in chunk order. Non-JSON media types and invalid JSON
are rejected.
Provides:
- `merged`
## `noop` Normalizer
Package: `internal/modules/normalize/noop`
The `noop` normalizer clones the raw merge output and returns it unchanged.
Requires:
- `merged`
Provides:
- `normalized`
## `json` Output
Package: `internal/modules/output/json`
The `json` output encoder converts normalized raw outputs, rejected raw outputs,
warnings, and the run manifest into logical JSON output files. It writes one
payload file per lane under `lanes/` and sanitizes lane IDs for file names.
Normalized output payloads must be valid `application/json`.
Requires:
- `normalized`
Provides:
- `encoded`
Durable output file shapes belong in the
[JSON output contract](../integrations/json-output.md). Operator behavior
belongs in [Operations](../operations.md).
## Production Registration ## Production Registration
Production registration is centralized in `internal/cli/catalog.go`. `internal/cli/catalog.go` builds the production registries, registers module and
validator constructors, installs default validator-chain mappings, and exposes
the matching catalog for resolution. It also collects prompt assets from
LLM-backed packages before constructing the production client.
Do not make framework code import production modules. The CLI wires production Framework packages must not import production extensions. Tests may compose
modules at the application boundary; tests may provide fake registries or fake registries and catalogs directly with fakes.
catalogs directly.
## Adding A Module ## Adding An Extension
When adding a module, keep source-format and extraction-domain boundaries clear: When adding a production module or validator:
- input modules may know external source formats; 1. implement the stage or validator contract and package-local key;
- extract modules may know artifact semantics and prompt/schema assets; 2. expose and test its spec, constructor, and registration function;
- merge and normalize modules own raw output combination and reconciliation; 3. keep format or domain parsing inside the concrete package;
- output modules own serialization, not diagnostics or CLI reporting. 4. add package-owned prompt/schema assets when the extension is LLM-backed;
5. register it in `internal/cli/catalog.go` and add a default chain only when
production policy requires one;
6. add resolution and composition coverage for capabilities, options,
references, and validation behavior;
7. update the selectable-key catalog in [Configuration](../config.md), the
relevant external contract, this inventory, and maintained examples when
user-visible behavior changes.
Update [Development](../development.md), [Configuration](../config.md), Do not add the extension to `docs/development.md`; that file routes by task and
internal docs, integration docs, and examples when the new module becomes does not inventory implementations.
implemented production behavior.
## Tests To Inspect
- Package-local `*_test.go` files under the module or validator being changed.
- `internal/framework/pipeline/registry_integration_test.go`: registry and spec
composition.
- `internal/framework/pipeline/default_modules_test.go`: framework binding
defaults.
- `internal/cli/run_test.go`: production catalog, config resolution, and
end-to-end CLI composition.
- `internal/modules/sharedassets/**/*_test.go`: shared prompt and reference
assembly.

160
docs/internal/overview.md
View File

@@ -1,127 +1,105 @@
# Internal Overview # Internal Overview
This document maps the implemented Notarius components and their ownership. It This document inventories the implemented Notarius components. Normative
complements the durable invariants in [Architecture](../policy/architecture.md) boundaries and dependency direction belong in
and links to focused internal documentation for deeper behavior. [Architecture](../policy/architecture.md); external behavior belongs in the
[CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [integration contracts](../integrations/).
## Execution Path ## Execution Path
The executable delegates to the CLI, which resolves configuration and wires the `cmd/notarius` delegates to `internal/cli`, the production composition root.
production application around the framework runner: The CLI loads configuration, builds the production catalogs and runtime
collaborators, invokes `internal/framework/pipeline`, and places the logical
output files returned by the runner. Diagnostics, checkpoints, and debug
recorders are optional side-channel collaborators supplied at this boundary.
```text Pipeline execution is serial. Resolution produces a fixed ordered workflow and
cmd/notarius a sorted set of artifact lanes before the runner constructs any stage module.
-> internal/cli
-> config resolution + production registries + LLM client
-> input -> chunk -> extract -> merge -> normalize -> output
-> durable output writes
Pipeline side channels:
diagnostics checkpoints debug artifacts
```
Pipeline execution is serial. Configuration selects modules for the fixed stage
shape; registries construct them after profile, capability, validator, and
reference resolution.
## Application Boundary ## Application Boundary
`cmd/notarius` contains the executable entry point and delegates process exit | Package | Implemented responsibility |
behavior to `internal/cli`. | --- | --- |
| `cmd/notarius` | Executable entry point and process exit delegation. |
`internal/cli` owns command parsing, configuration discovery, production module | `internal/cli` | Command parsing, config discovery, production registration, prompt asset collection, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
and validator registration, prompt asset collection, production LLM client
construction, reference preparation, workspace recorder setup, durable output
writes, and user-facing stdout, stderr, and exit codes. It is the composition
root for concrete production packages.
## Core Packages ## Core Packages
| Package | Implemented responsibility | | Package | Implemented responsibility |
| --- | --- | | --- | --- |
| `internal/core/artifacts` | Run manifests and artifact serialization shapes. | | `internal/core/artifacts` | Run-manifest and provenance models. |
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline configuration. | | `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
| `internal/core/diagnostics` | Diagnostics run directories, artifact writers, atomic writes, and retention decisions. | | `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
| `internal/core/source` | Generic source documents, units, references, and validation. | | `internal/core/source` | Generic source documents, units, references, lookup, and validation. |
| `internal/core/workspace` | Workspace settings, safe paths and writes, checkpoint identities, and checkpoint manifest types. | | `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
These packages provide concrete, deterministic models and policy. Production
module registration occurs at the CLI boundary rather than in core packages.
## Framework Packages ## Framework Packages
| Package | Implemented responsibility | | Package | Implemented responsibility |
| --- | --- | | --- | --- |
| `internal/framework/contracts` | Stage, validator, reference, output, and structured LLM interfaces and request/result types. | | `internal/framework/contracts` | Stage, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Module registries, profile resolution, capability checks, reference materialization, validation chains, retries, orchestration, warnings, and manifest population. | | `internal/framework/pipeline` | Registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. | | `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema asset registration, scheduling, profile recording, and secret redaction. | | `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload envelopes. | | `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
| `internal/framework/debug` | Workspace-backed framework and LLM debug artifacts. | | `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
Framework contracts carry raw stage outputs between modules. The runner owns Framework contracts carry raw stage results between implementations. The
provenance, validation sequencing, rejection handling, checkpoint boundaries, runner owns handoff provenance, validation sequencing, rejection handling,
debug boundaries, and final manifest assembly. checkpoint and debug boundaries, and final manifest assembly.
## Production Modules ## Production Extensions
Production implementations live under `internal/modules` and register through The canonical catalogs of user-selectable
the CLI catalog. [module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) keys are in
Configuration. The implemented module packages are:
| Stage | Module key | Package | Role | | Package | Implemented responsibility |
| --- | --- | --- | --- | | --- | --- |
| Input | `seriatim` | `internal/modules/input/seriatim` | Converts Seriatim transcript JSON into the generic source model. | | `internal/modules/input/seriatim` | Parses the supported Seriatim transcript format into the generic source model. |
| Chunk | `generic` | `internal/modules/chunk/generic` | Splits ordered source units by configured unit counts and overlap. | | `internal/modules/chunk/generic` | Splits ordered source units by unit count and overlap. |
| Chunk | `dnd/scenes` | `internal/modules/chunk/dnd/scenes` | Uses structured LLM output to create contiguous D&D scene chunks. | | `internal/modules/chunk/dnd/scenes` | Produces contiguous D&D scene chunks from structured model output. |
| Extract | `dnd/spells` | `internal/modules/extract/dnd/spells` | Extracts source-grounded D&D spell-cast artifacts. | | `internal/modules/extract/dnd/spells` | Produces source-grounded D&D spell-cast raw output. |
| Merge | `appendorder` | `internal/modules/merge/appendorder` | Combines accepted extract outputs in chunk order. | | `internal/modules/merge/appendorder` | Combines accepted extraction results in chunk order. |
| Normalize | `noop` | `internal/modules/normalize/noop` | Preserves accepted merged output unchanged. | | `internal/modules/normalize/noop` | Preserves accepted merged output. |
| Output | `json` | `internal/modules/output/json` | Encodes manifests, indexes, warnings, rejections, and accepted lane payloads as logical JSON files. | | `internal/modules/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
`internal/modules/sharedassets` composes shared prompt filesystems. `internal/modules/sharedassets` composes shared prompt filesystems.
`internal/modules/sharedassets/dnd` owns shared D&D prompt fragments, reference `internal/modules/sharedassets/dnd` owns reusable D&D prompt fragments,
slots, prompt input assembly, and source-unit reference helpers. reference declarations, prompt input assembly, and source-unit reference
helpers.
## Validators Concrete validators live under `internal/validators`. Generic packages provide
unconditional test decisions, JSON syntax validation, and JSON Schema
validation. D&D spell packages provide shape, source-reference, and
source-relatedness decisions, with `spellpayload` holding their shared parser
and lookup helpers. Production chain composition is owned by `internal/cli`.
Concrete validators live under `internal/validators` and register separately Implementation details for all production extensions are in
from stage modules. Generic validators cover unconditional test decisions, JSON [Module Internals](modules.md).
syntax, and JSON Schema. D&D spell validators cover artifact shape, source
reference validity, and source relatedness.
The production default chain for `dnd/spells` extract output is registered ## Run-State Components
centrally in `internal/cli`; module packages produce raw output but do not own
the production approve/reject policy.
## Files And Run State | Surface | Implemented owners | Internal purpose |
Notarius keeps distinct output and inspection surfaces:
| Surface | Owner | Purpose |
| --- | --- | --- | | --- | --- | --- |
| Durable output | Output module and CLI writer | User-consumable run files. | | Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
| Diagnostics | `internal/core/diagnostics` and CLI | Redacted run inspection, reports, warnings, and failures. | | Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
| Checkpoints | `internal/framework/checkpoint` | Validated stage reuse for explicit resume. | | Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. |
| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Sensitive framework-boundary and LLM call inspection. | | Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
Workspace settings determine whether and where diagnostics, checkpoints, and Physical layout, retention, recovery, and sensitive-data handling are defined
debug artifacts are written. Concrete stage modules do not receive workspace in [Operations](../operations.md). Concrete stage modules receive recorder
paths. interfaces and request data, not workspace paths.
## Focused Internal Documentation ## Focused Documentation
- [Pipeline Internals](pipeline.md): resolution, execution, validators, - [Pipeline Internals](pipeline.md): resolution, execution, validation, retries,
references, retries, checkpoints, outputs, and manifests. checkpoint/debug hooks, and result assembly.
- [Module Internals](modules.md): production module contracts, capabilities, - [Module Internals](modules.md): production modules, validators, assets,
options, prompts, schemas, and registration. registration, and the contributor recipe for adding an extension.
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter, - [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
assets, scheduling, profile recording, and redaction. assets, scheduling, profile recording, and redaction.
- [Diagnostics Internals](diagnostics.md): diagnostics files, retention, - [Diagnostics Internals](diagnostics.md): scoped writers, retention
failure behavior, and path safety. coordination, CLI failure flow, and path safety.
## Test Surfaces
The repository uses focused package tests, registry and pipeline composition
tests, a fake-backed walking skeleton, fixture-driven CLI coverage, and local
test servers for LLM integration behavior. Tests do not require real provider
calls.

396
docs/internal/pipeline.md
View File

@@ -1,295 +1,185 @@
# Pipeline Internals # Pipeline Internals
The implemented pipeline runner lives in `internal/framework/pipeline`. It The implemented resolver and runner live in `internal/framework/pipeline`.
executes the fixed workflow defined by the architecture policy: Their fixed workflow and ownership boundaries are defined by
[Architecture](../policy/architecture.md#system-shape). Configuration fields,
defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines).
```text Pipeline execution is serial. Resolution fixes the selected lanes and all
input -> chunk -> extract -> merge -> normalize -> output stage bindings before the runner constructs stage implementations.
```
Pipeline execution is serial. The runner executes the resolved lanes one after ## Resolution
another in the fixed workflow order.
## Profile Resolution `internal/core/config.Config.Resolve` validates the loaded configuration,
selects the named profile, applies the runtime inputs supplied by the CLI, and
calls `pipeline.ResolvePipeline`.
Config loading produces `pipeline.PipelineProfile` values. Resolution happens `ResolvePipeline`:
before execution:
1. `internal/core/config.Config.Resolve` validates config and finds the named 1. selects and sorts artifact lanes;
pipeline. 2. completes omitted bindings using the documented configuration defaults;
2. The optional lane selection is passed to `pipeline.ResolvePipeline`. 3. looks up each module and validator spec without constructing it;
3. Module bindings are defaulted: 4. checks required and provided capabilities in workflow order;
- chunk: `generic` 5. resolves target-aware reference bindings and validator chains;
- merge: `appendorder` 6. calculates a digest over the resolved structure.
- normalize: `noop`
- output: `json`
- LLM profile: empty, which lets Scriptorium prompt defaults choose a
profile.
4. The module catalog is checked for each bound module key.
5. Module capabilities are checked in workflow order.
6. A digest is calculated from the resolved pipeline without the digest field.
The CLI writes the resolved pipeline and digest to diagnostics. Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
bindings, validator chains, reference targets, and the digest. It does not read
reference bytes or construct runtime modules. CLI lane and reference selector
syntax is defined in the [CLI reference](../cli.md#run).
Pipeline profiles and artifact lanes may include reference binding maps keyed by ## Reference Materialization
reference slot name. During resolution, pipeline-level bindings act as defaults
for selected chunk, extractor, merger, and normalizer targets that declare the
slot; target-local bindings override or add bindings for that target. Runtime
`--reference` requests override target config bindings, and runtime unbinds
remove optional target bindings. Flat runtime slot names are resolved only when
exactly one selected target declares the slot; otherwise the CLI requires a more
specific selector such as `chunk.slot`, `lane.extract.slot`,
`lane.merge.slot`, or `lane.normalize.slot`. Resolution validates bindings
against the declaring target specs and stores the bindings in target-aware
resolved reference holders. It does not read reference files or include
reference bytes in source digests.
During run preparation, resolved file references for chunk, extractor, merger, The CLI calls `MaterializeReferences` after resolution and before constructing
and normalizer targets are materialized before any LLM-backed pipeline work. Config the LLM client or running the pipeline. The materializer checks each binding
bindings resolve relative to the config file, and CLI bindings resolve relative against its resolved target declaration, reads and validates the file, and
to the current working directory. Materialization accepts UTF-8 text files, builds both a `contracts.ReferenceSet` and provenance-only metadata on the
computes `sha256:` content digests, records file origins, infers canonical base corresponding `ResolvedReferenceTarget`.
media types from file extensions, enforces declared byte limits, and warns for
empty bound files. Media-type acceptance is checked only when a slot declares
`AcceptedMediaTypes`; unknown extensions are recorded as
`application/octet-stream`. Reference content is omitted from diagnostics and
manifests. The CLI writes provenance-only resolved reference diagnostics, and
the run manifest records target-stage reference provenance separately from
source digests. Runtime reference content is passed to the matching chunker,
extractor, merger, or normalizer request. LLM-backed modules pass that material
onward as named Scriptorium prompt inputs.
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse The runner clones the resulting set into the chunk, extract, merge, or normalize
those bytes into the source document. Chunk, merge, and normalize requests request that owns the target. LLM-backed extensions may convert those items into
receive the original source material as `SourceInput`; extraction requests named prompt inputs. Reference content remains separate from source evidence and
receive chunk-scoped source material built from the current `SourceChunk` source digests.
content, media type, and origin metadata. The raw input payload is not written
to manifests or default diagnostics.
The CLI also carries an optional run `session_id`. The runner makes it available Binding precedence, path resolution, accepted content, and media-type behavior
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it are configuration contracts; see [Configuration](../config.md#pipelines).
through their structured completion requests so Scriptorium can include it in Durable provenance is defined in the
prompt execution metadata. [JSON output contract](../integrations/json-output.md#manifestjson), while
runtime sensitive-data handling belongs in [Operations](../operations.md).
When workspace resume checkpointing is enabled, the CLI constructs a checkpoint ## Registries And Specs
recorder after pipeline resolution and reference materialization and passes it
through `pipeline.RunInput`. The runner records source, chunk, extract, merge,
and normalize outcomes through that interface. Concrete modules do not receive
workspace paths and do not write checkpoint files directly.
For `run --resume`, the CLI also passes a checkpoint loader. The runner consults `pipeline.Registries` holds constructors used during execution.
the loader in workflow order and reuses only checkpoints whose manifest schema, `pipeline.ModuleCatalog` exposes their specs during configuration validation and
status, identity digest, dependency fingerprints, payload files, and payload resolution. Separate registries exist for every stage and for validators;
digests validate for the current invocation. The identity includes the resolved `ValidatorChainRegistry` stores production default-chain mappings.
pipeline, selected lanes, source/input digest, runtime overrides that affect
execution, and materialized reference digests. Missing or invalid checkpoints
fall back to normal execution and are refreshed by the recorder.
When workspace debug output is enabled, the CLI passes a debug recorder for the A `ModuleSpec` declares its stage plus required and provided capabilities.
current run ID. The runner writes framework-boundary inputs, outputs, Chunk, extract, merge, and normalize specs may also declare reference slots.
structured LLM calls, validator calls, timing, and retry attempt metadata Registry implementations defensively copy spec metadata, reject duplicate keys,
through that interface. Each retry or validator attempt records any LLM calls and verify that a constructed implementation reports the registered key.
made within that attempt in an `llm_calls` array and writes paired
`prompt-000N.json` and `response-000N.json` metadata files under the attempt
directory. LLM response bodies are written as sibling `response-content-000N.*`
files, using pretty-printed JSON when the content is valid JSON and raw text
otherwise. Debug output is not used for resume and can contain sensitive source,
reference, prompt, and model-output material. Concrete modules still do not
receive workspace paths.
## Registries And Module Specs A `ValidatorSpec` declares a validator key and execution class. Resolution uses
the execution class to reject incompatible profile bindings before execution.
The current production catalog and default chain are listed only in
[Configuration](../config.md#implemented-production-validators).
`pipeline.Registries` holds concrete constructors for execution. A ## Runner Boundary
`pipeline.ModuleCatalog` exposes module specs for config validation and
resolution. The catalog also exposes validator specs and central default
validator-chain mappings without constructing modules or validators.
Every production module registers a `ModuleSpec` with: `pipeline.RunInput` carries the resolved pipeline, raw source input, structured
LLM client, run identity and timing, optional session and profile metadata, and
checkpoint/debug collaborators. The runner parses source bytes through the
selected input adapter. Later stage requests receive the generic source model;
extract requests receive chunk-scoped input material, while chunk, merge, and
normalize requests retain access to the original source material.
- `Key`: module key used in config; `pipeline.RunOutput` carries the run manifest, accepted normalized results,
- `Stage`: module kind such as input, chunk, extract, merge, normalize, rejected results, warnings, checkpoint events, and logical files returned by the
validate, or output; output encoder. The CLI owns diagnostics and durable filesystem writes after the
- `Provides`: capabilities added after that module runs; runner returns.
- `Requires`: capabilities that must already be available.
Chunk, extract, merge, and normalize specs may also declare reference slots. Slot ## Execution Flow
declarations are available from registry metadata without constructing module
instances. Input, validate, and output specs must not declare reference slots.
Capability checks prevent incompatible pipeline composition before a run starts.
Every production validator registers a `ValidatorSpec` with:
- `Key`: validator key used in config and manifests;
- `ExecutionClass`: `deterministic` or `llm_backed`.
Default validator chains are keyed by workflow stage and module key. Production
currently registers a default chain for `extract` module `dnd/spells` only.
## Runner Input And Output
`pipeline.RunInput` carries:
- a `ResolvedPipeline`;
- optional source ID, input path, and raw input bytes;
- a structured LLM client;
- run ID, start time, LLM profile manifest metadata, and CLI metadata.
`pipeline.RunOutput` carries:
- run manifest;
- normalized raw outputs;
- rejected raw outputs;
- warnings;
- logical output files returned by the output encoder.
The CLI owns durable file writes and diagnostics writes after the runner returns.
## Execution
The runner: The runner:
1. validates run input and registries; 1. validates its input and registries;
2. builds the input adapter and parses the raw input into a source document; 2. builds the input adapter, parses the raw input, and validates the generic
3. validates the source document; source document;
4. builds the chunker and produces source chunks, retrying when configured; 3. obtains or executes the chunk result;
5. validates source chunks against framework invariants and the resolved chunk 4. validates and canonicalizes chunks;
validator chain; 5. executes each resolved artifact lane in order;
6. runs each selected artifact lane in sorted resolved order; 6. builds the output encoder and validates its logical file results;
7. builds the output encoder and validates logical output file names. 7. returns the assembled manifest, outcomes, warnings, and files.
8. passes accepted normalized raw outputs, rejected output records, warnings,
and the manifest to the output encoder.
## Chunk Results Within each artifact lane, it builds the extractor, merger, and normalizer,
then performs these transitions:
Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest` 1. extract once per accepted chunk and add runner-owned lane, source, and chunk
with the validated source document, reference set, structured LLM client, the provenance;
configured LLM profile, module options, and run metadata. Deterministic and 2. validate each raw extract result and omit rejected results from merge input;
LLM-backed chunkers use the same contract; provider construction stays outside 3. skip the rest of the lane when no extract result is accepted;
chunk modules. 4. merge accepted extract results in their existing order;
5. validate the merge result and skip normalization on rejection;
6. normalize the accepted merge result;
7. validate and append the accepted normalized result.
When chunking succeeds, the runner validates generic chunk invariants before Module-provided warnings and payload warnings are promoted only from attempts
running extractors: whose results are accepted and used.
- chunk IDs must be non-empty and unique in the chunk result; ## Chunk Canonicalization
- each chunk `SourceID` must match the source document ID;
- each chunk `Index` must match its zero-based returned order;
- each chunk start and end unit ID must exist in the source document, with the
start unit at or before the end unit;
- each chunk must include non-empty extraction content and media type;
- each chunk must contain at least one source unit;
- a chunk must not repeat a source unit;
- every chunk source unit must exist in the source document;
- source units inside each chunk must appear in source-document order.
After validation, the runner rebuilds each chunk from source-document units by Before lane execution, generic validation requires unique chunk IDs, matching
integer ID, preserving chunk boundaries, content bytes, media type, and cloned source identity, indexes matching returned order, valid ordered boundaries,
chunk metadata. Extractors and downstream stages therefore see canonical source non-empty content and media type, and at least one valid source unit per chunk.
units, while `SourceChunk.Metadata` remains the supported place for Units may not repeat inside a chunk and must preserve source-document order.
chunker-owned context.
If chunk validation rejects the chunk result after configured retries, the runner The runner then rebuilds each chunk's unit slice from the source document by
records a rejected raw output and skips downstream lane execution. Framework-level unit ID. It preserves the module-owned boundaries, content, media type, and
chunking or validation errors that remain after configured retries fail the run. cloned metadata. The framework permits gaps and overlap between separate
chunks; stricter coverage policy belongs to the chunk implementation.
The framework does not require complete source-unit coverage and does not reject ## Validation And Retries
overlap between different chunks. Stricter policies, such as full coverage or
non-overlap, belong to individual chunk modules when they are part of that
module's contract.
Within an artifact lane, the runner: Chunk, extract, merge, and normalize results pass through the resolved validator
chain for their stage and module. Each validator receives the raw payload plus
the relevant source, chunk, prior-stage, schema, reference, session, LLM, option,
and run context. Validators execute in resolved order and stop at the first
error or rejection. An empty chain approves the result.
1. builds the extractor, merger, and normalizer; `runWithRetry` performs the initial module call plus the configured additional
2. records module manifest metadata when modules provide it; attempts. Each attempt includes module execution and its complete validation
3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when chain. A module or validator error retries and becomes a framework error after
configured; the final attempt. A rejection retries and becomes a recorded `RejectedOutput`
4. fills runner-owned provenance on each extract output, including lane ID, after the final attempt. Cancellation stops retry processing immediately.
extractor key, source ID, chunk ID, and chunk index;
5. validates raw extract outputs and omits rejected outputs from merge input;
6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying
when configured;
7. validates raw merge output and skips normalization for rejected merge output;
8. normalizes the accepted merge output into one raw `NormalizeOutput`,
retrying when configured;
9. validates raw normalize output and appends accepted normalized raw output to
`RunOutput.NormalizeOutputs`.
## Validators Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
from discarded attempts are not promoted. Configuration owns retry counts and
validator overrides; see [Module Bindings](../config.md#module-bindings).
The runner handoff is raw-output based. Chunkers, extractors, mergers, and ## Checkpoint And Debug Hooks
normalizers do not advertise validator chains through their module interfaces.
Resolved validation chains receive the raw module output plus stage, lane,
module, source, chunk, schema, session, reference, LLM client/profile, binding
option, and run metadata context. Chunk validators receive the chunk result
collection, merge validators receive the ordered extract outputs used by the
merge, and normalize validators receive the accepted merge output. Empty chains
approve output by default. Response-schema provenance may include in-memory JSON
schema bytes for validators. Those bytes are omitted from manifests,
diagnostics, and encoded output files.
Resolved validator chains come from central default mappings unless a The runner depends on recorder and loader interfaces, using no-op
stage-local config override is set on `chunk`, lane `extract`, lane `merge`, or implementations when collaborators are absent. Each checkpointed workflow
lane `normalize`. Explicit empty overrides are valid and are recorded as empty boundary records a running, succeeded, or failed transition. Reuse decisions
chains in manifests. Explicit non-empty overrides replace the default chain and are consulted in workflow order and accepted payloads are cloned before
preserve configured order. entering the normal handoff path. Dependency fingerprints connect later
checkpoints to the exact accepted results on which they depend.
The production default chain for `extract` module `dnd/spells` is: Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
boundaries. Context scopes associate nested LLM calls with the module or
validator attempt that made them. Debug-write failures are framework errors;
debug data is never used as a checkpoint source.
1. `generic/valid_json` Checkpoint identity, physical layout, reuse behavior, and debug artifact
2. `generic/valid_json_schema` handling are operator contracts in [Operations](../operations.md). Serialization
3. `extract/dnd/spells/shape` and recorder implementation are inventoried in
4. `extract/dnd/spells/source_refs` [Internal Overview](overview.md#run-state-components).
5. `extract/dnd/spells/source_relatedness`
No other production module currently has a default validator chain. ## Results And Failures
Validator rejection is a non-fatal run outcome: the rejected output is recorded The runner owns manifest assembly and handoff summaries but not the durable JSON
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution schema. It records resolved module and lane provenance, validator chains,
errors are framework-level errors and retry according to the relevant binding. source/reference identities, selected LLM profiles, normalized and rejected
Warning-only validators return approved results with warnings; those warnings summaries, status, and timing. Raw payload bytes remain outside the manifest.
are promoted only from successful attempts whose outputs are used. Module metadata providers may add non-secret singleton or lane-scoped metadata.
## Warnings And Failures Execution errors include stage, module, lane, or validator context. Once a
manifest exists, a failing run returns it with failed status and completion
time. Successful status reflects whether any raw result was rejected. The
durable manifest and logical file schemas are defined in the
[JSON output contract](../integrations/json-output.md).
Warnings from the successful chunking, extraction, merging, and normalization ## Tests To Inspect
attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along
with output encoder warnings. Warnings from discarded retry attempts are not
promoted to final warnings.
Errors wrap the operation and module key or lane context. If execution fails - `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
after a manifest exists, the returned manifest is marked `failed` and receives a - `internal/framework/pipeline/profile_test.go`: selection, defaults,
completion timestamp. capabilities, validator chains, and digest behavior.
- `internal/framework/pipeline/references_test.go`: target resolution and
On successful execution, the manifest validation status is: materialization.
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
- `approved` when no raw outputs were rejected; rejections, warnings, checkpoints, debug hooks, and manifests.
- `rejected` when at least one raw output was rejected. - `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
workflow composition.
## Manifest Population - `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
collaborators.
The manifest records run ID, pipeline ID, pipeline digest, module keys, top-level
module metadata, artifact lanes, LLM profile metadata, source digest,
reference provenance, normalized raw output summaries, rejected output
summaries, validation status, and timing. Raw output summaries include lane ID,
normalizer module key, media type, source ID, and response-schema provenance
when present. Rejected output summaries include stage, lane, module, chunk,
validator or reason, message, attempt count, and optional diagnostic artifact
path. The manifest does not include raw output payload bytes.
Singleton pipeline modules may add non-secret metadata by implementing
`contracts.ManifestMetadataProvider`. The runner records that metadata under
`module_metadata` with stable keys for `input`, `chunker`, and `output`.
Lane-owned modules may add non-secret metadata through
`artifact_lanes[].metadata`. The runner records extractor, merger, and
normalizer metadata there. The D&D spell extractor uses lane metadata for
prompt and response-schema provenance.
## JSON Output
The production JSON output encoder writes `manifest.json`, `index.json`,
`warnings.json`, `rejected.json`, and one pretty-printed JSON file per accepted
normalized lane output under `lanes/`. It accepts only normalized outputs with
valid `application/json` payloads. Unsupported media types, invalid JSON, unsafe
logical paths, and duplicate sanitized lane file names fail the run before
durable output files are written.

22
docs/policy/architecture.md
View File

@@ -32,21 +32,17 @@ Notarius is contract-first without being abstraction-heavy. Interfaces and
extension points should protect demonstrated boundaries. New abstraction is not extension points should protect demonstrated boundaries. New abstraction is not
itself an architectural goal. itself an architectural goal.
## Package Layout And Dependency Direction ## Layers And Dependency Direction
| Area | Ownership | The application boundary is the composition root and may depend on concrete
| --- | --- | implementations. Domain-neutral model and framework layers provide reusable
| `cmd/notarius` | Executable entry point; delegates to the CLI. | policy, contracts, and orchestration. Concrete input, pipeline, output, and
| `internal/cli` | Application boundary, production composition, runtime setup, durable writes, and user-facing results. | validation extensions depend inward on those generic layers.
| `internal/core` | Generic deterministic models and policy for source material, configuration, manifests, diagnostics, and workspace identity. |
| `internal/framework` | Reusable contracts, registries, pipeline orchestration, validation mechanics, checkpoints, debug boundaries, and LLM runtime plumbing. |
| `internal/modules` | Concrete pipeline stage behavior. |
| `internal/validators` | Concrete approve/reject policies. |
The CLI is the composition root and may import concrete implementations. Core Generic layers must not depend on production extensions. Concrete extensions
and framework packages cooperate as generic application layers; neither may must not compose the application or take ownership of process behavior. The
depend on production modules or validators. Concrete implementations may depend current packages implementing these layers are inventoried in
on core models and framework contracts. [Internal Overview](../internal/overview.md).
The following dependency boundaries are mandatory: The following dependency boundaries are mandatory: