eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: eric:v0.2.0
Branches Tags
eric:main
eric:v0.3.0 eric:v0.2.0 eric:v0.1.0
..
compare: eric:6f7d5258052c6bacd763f06db05dcc5c10f6f643
Branches Tags
eric:main
eric:v0.3.0 eric:v0.2.0 eric:v0.1.0

1 Commits
v0.2.0 ... 6f7d525805

Author SHA1 Message Date
Eric Rakestraw
6f7d525805 Complete checkpoint review 2026-07-03 06:11:44 +00:00
536 changed files with 2903 additions and 84528 deletions
Expand all files Collapse all files

10
.gitignore vendored
View File

@@ -1,9 +1,3 @@
# build and testing artifacts
notarius
notarius-output
workspace/
.codebase-memory/
# ---> Go
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
@@ -53,8 +47,7 @@ go.work.sum
.LSOverride
# Icon must end with two \r
Icon
Icon
# Thumbnails
@@ -74,3 +67,4 @@ Icon
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk

5
AGENTS.md
View File

@@ -1,2 +1,3 @@
Please review `docs/development.md` for initial orientation in this repository
and follow its task-specific reading guide.
Please carefully review the documents in `docs/policy` before making any changes to this repository.
- `architecture.md` provides the canonical high-level architecture policy for this repository.
- `documentation.md` provides the canonical documentation policy for this repository.

45
README.md
View File

@@ -1,45 +1,2 @@
# Notarius
# go-application-template
Notarius is a Go CLI for turning source material into structured artifacts with
configured extraction pipelines. The implemented D&D workflow reads Seriatim
transcript JSON and can produce scene descriptions, item and currency events,
NPC identities, combat turns, NPC interactions, and spell casts.
## Quickstart
Provide an OpenRouter API key through the environment, then run the maintained
minimal example:
~~~
OPENROUTER_API_KEY=your-api-key \
go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json
~~~
The command publishes a JSON output bundle. Its command syntax and exit
behavior are documented in the [CLI reference](docs/cli.md); configuration,
credentials, and module selection are owned by the
[configuration reference](docs/config.md).
For the complete ordered D&D workflow, use
[the complete configuration](examples/dnd-complete.config.yml) with
[its synthetic transcript](examples/dnd-complete-transcript.json). It
demonstrates all implemented D&D lanes and the supporting campaign references.
## Documentation
- [CLI reference](docs/cli.md) — commands, flags, output streams, and exits.
- [Configuration reference](docs/config.md) — configuration files, profiles,
validation, and module selection.
- [Operations](docs/operations.md) — output, state, recovery, and debug
handling.
- [Integration contracts](docs/integrations/) — Seriatim input and published
artifact formats.
- [Subprocess consumer guide](docs/consumers/subprocess.md) — invoke Notarius
from an orchestrator and consume a published result.
- [Internal overview](docs/internal/overview.md) — implemented component map
for maintainers.
- [Developer guide](docs/development.md) — contributor orientation and
validation guidance.
- [Future work](docs/roadmap/future.md) — unimplemented ideas and priorities.

23
docs/adr/0001-record-architecture-decisions.md
View File

@@ -1,23 +0,0 @@
# ADR-0001: Record architecture decisions as ADRs
**Status:** Accepted
**Date:** 2026-07-13
## Context
Architectural reasoning made during design (pattern choices, rejected
alternatives, trigger conditions for revisiting) is lost if only the final
state is documented.
## Decision
We keep a living overview in docs/policy/architecture.md describing current
intended state, and immutable, numbered ADRs (Nygard format) in docs/adr/
recording each significant decision, its alternatives, and its consequences.
Changed decisions get a new ADR that marks the old one Superseded.
## Alternatives considered
- Overview doc only: loses the "why" and the rejected options.
- arc42 / RFC-style design docs: heavier than warranted for a solo repo.
## Consequences
Small ongoing writing cost; durable reasoning trail; cheap onboarding for
future contributors (including future-us).

51
docs/adr/0002-linear-pipes-and-filters-pipeline.md
View File

@@ -1,51 +0,0 @@
# ADR-0002: Linear pipes-and-filters pipeline, not a general DAG
**Status:** Accepted
**Date:** 2026-07-13
## Context
Notarius processes source material through one known workflow:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
Input and chunking apply to the source as a whole. Each selected artifact lane
then performs extract, merge, and normalize, after which output aggregates the
lane outcomes. Chunk extraction has a natural scatter-gather shape, but no
current use case requires arbitrary branches, joins, or user-defined stage
topology.
## Decision
Notarius implements a fixed six-stage pipes-and-filters pipeline. Configuration
selects implementations for these stages but cannot add stages, reorder them,
or define an arbitrary graph.
The framework owns stage sequencing and the scatter-gather boundary between
chunk, extract, and merge. Extract results are handed to merge in deterministic
source-chunk order regardless of execution strategy. Each artifact lane remains
logically linear. Output runs after every selected lane has either produced an
accepted normalized artifact or reached a recorded rejection. A framework
execution failure aborts the pipeline.
The runner's concrete internal representation and stage-specific scheduling
policies are implementation details. Concurrency must preserve the pipeline's
deterministic handoffs, validation behavior, and provenance, and all execution
strategies must continue to honor context cancellation.
## Alternatives considered
- Build a general DAG engine now. This would support hypothetical branching
topologies, but would add scheduling, topology validation, configuration, and
state-management complexity without a current consumer. Revisit this choice
only when a concrete workflow requires a topology the fixed pipeline cannot
express.
## Consequences
The runner, configuration model, and operator mental model remain small. Stage
ownership stays visible, and general chunking, merging, or normalization cannot
be hidden inside extractors. A future DAG requirement will require an explicit
architectural change rather than incremental exceptions to the fixed pipeline.

119
docs/adr/0003-typed-interfaces-with-two-zone-data-model.md
View File

@@ -1,119 +0,0 @@
# ADR-0003: Strongly typed stage interfaces with a two-zone data model
**Status:** Accepted
**Date:** 2026-07-13
## Context
Pipeline stages must exchange source data and extracted artifacts. Universal
source data has one engine-wide meaning, while extracted artifacts have
domain-specific shapes. Passing opaque bytes or `any` between all stages would
make invalid wiring and merge behavior runtime concerns. Requiring JSON at
every handoff would preserve interoperability but discard useful Go type safety
while all modules are in-process.
The framework must also support multiple configured artifact domains, durable
checkpoints, diagnostics, and output encoders without making those consumers
depend on every domain's Go types.
## Decision
Notarius uses two typed data zones followed by one serialized boundary.
### Source zone
Input and chunk stages use conservative, engine-owned document, segment, chunk,
and source-reference types. Their exact Go names are implementation details.
Every segment carries engine-owned source provenance identifying the source
location from which it was produced. Chunks preserve the ordered provenance of
their segments.
Source-format-specific fields remain in input modules or explicitly namespaced
metadata; they do not become framework contracts.
### Domain artifact zone
Each artifact lane has one domain-owned Go artifact type `T`. Its extract,
merge, normalize, and domain-aware validation implementations use generic,
strongly typed contracts over the same `T`. Raw JSON, opaque bytes, and `any`
are not stage-handoff contracts within a lane.
Each registered domain artifact type supplies a codec for `T`. The codec owns:
- stable schema identity and an explicit schema version;
- JSON serialization and deserialization;
- the media type and schema metadata required at serialized boundaries; and
- rejection of data that cannot be represented by the declared artifact
schema.
An artifact type's JSON representation is a maintained domain contract.
Changing it incompatibly requires a new schema version.
Extract, merge, and normalize may change the contents of `T`, but they do not
change the lane's canonical Go artifact type or artifact schema identity. An
extractor maps any provider- or prompt-specific response type into `T` before
returning. A future lane that requires different artifact types at different
stages requires a new architectural decision.
### Serialized boundary
After normalization, each typed artifact is converted into an engine-owned
serialized artifact containing bytes, media type, and schema metadata. Output
aggregation and output encoders consume this type-erased form. Intermediate
checkpoint and debug encodings do not become stage-handoff contracts.
LLM transport, checkpoints, and opt-in debug recording are also explicit
serialization boundaries. They may encode or decode a typed artifact through
its domain codec, but they do not change the in-memory type used between
extract, merge, normalize, and typed validators. Checkpoint reuse requires a
compatible schema identity and version.
An LLM structured-response schema is a module transport contract and may differ
from the domain artifact schema. The calling module owns the response type and
maps it into the canonical `T`; the artifact codec remains authoritative for
artifact checkpoints and output serialization.
The framework may use private type-erased adapters to store heterogeneous lane
registrations and execute configured domains. Such an adapter must assemble a
type-consistent lane before execution and must not expose `any` or raw payloads
as module-facing handoffs inside the domain artifact zone.
### Construction and dependencies
Every module operation accepts `context.Context`. Modules receive stable runtime
collaborators through an injected dependency set at construction time. In
particular, LLM-using modules receive the application-provided structured LLM
client and do not construct provider clients or bypass shared scheduling.
The application boundary enforces one configurable global upper bound on
in-flight LLM calls across all stages, lanes, retries, and validators.
Configuration options are parsed and validated while a module is constructed,
before that module executes. Per-run data such as source material, references,
session identity, and lane identity remains operation input rather than a
construction dependency.
## Alternatives considered
- Pass raw bytes between stages. This maximizes decoupling but moves wiring,
parsing, and merge errors to runtime and prevents domain types from being the
canonical in-process contract.
- Require JSON plus schemas at every stage boundary. This is appropriate for an
out-of-process boundary, but adds serialization and parsing inside the current
in-process pipeline. The stable codec contract preserves this upgrade path if
remote plugins are introduced.
- Use a uniform `Process(any) (any, error)` contract. This simplifies a fully
dynamic engine but turns incompatible module composition into type assertions
and runtime failures. The fixed topology does not require that tradeoff.
## Consequences
Domain pipelines gain compile-time handoff safety and explicit merge semantics.
Serialization, schema compatibility, checkpoint decoding, and output erasure
have named owners. Dynamic registration requires a small erased adapter around
each typed lane, and generic stage implementations must be instantiated for a
specific artifact type or behavior rather than manipulating arbitrary JSON.
The engine-owned source model becomes a long-lived contract and must evolve
conservatively. Domain authors must maintain a codec and versioned schema in
addition to their Go artifact type.

81
docs/adr/0004-package-modules-by-domain.md
View File

@@ -1,81 +0,0 @@
# ADR-0004: Package modules by domain, not by stage
**Status:** Accepted
**Date:** 2026-07-13
## Context
Module packages can be grouped first by pipeline stage, such as
`modules/chunk/dnd/scenes`, or first by domain, such as
`modules/dnd/chunk/scenes`. A domain's extract, merge, normalize, validation,
schema, prompt, and artifact-codec implementations collaborate around the same
artifact types and are likely to evolve together.
Go package dependencies also constrain registration. If shared types live in a
domain root package, that package cannot import child implementation packages
to register them because the children already import the root types.
## Decision
Production extensions are grouped by domain under:
```text
internal/modules/<domain>/<stage>/<name>
```
Shared artifact types live at the domain root, for example
`internal/modules/dnd/types.go`. Domain-specific validators, prompt fragments,
schemas, reference helpers, and codecs also live within that domain tree.
Each domain exposes one production registration entry point from a sibling
registrar package, for example `internal/modules/dnd/register`. The registrar
may import the domain root and its child implementations; the domain root does
not import its registrar or child packages. This keeps shared types available
as `dnd.SpellList` without creating a Go import cycle.
The `generic` tree is a peer extension family for reusable implementations that
contain no concrete source-format or artifact-domain knowledge. Source-format
and output-format families, such as Seriatim and JSON output, follow the same
domain-first organization even when they do not define a type in the
[domain artifact zone](0003-typed-interfaces-with-two-zone-data-model.md#domain-artifact-zone).
Concrete domain implementation packages do not import another concrete domain.
Generic extension packages never import concrete domains. A domain registrar
may import domain-neutral generic extension packages to instantiate a reusable
strategy for that domain's artifact type; the generic implementation remains
unaware of the concrete type's domain semantics. Reuse needed directly by a
domain implementation lives in a domain-neutral framework or helper package,
not in a peer extension package.
The application composition root may import multiple registrar packages, and
black-box integration tests may compose multiple domains. Other cross-domain
reuse occurs through engine contracts and composition-time registration rather
than concrete peer-domain imports.
A domain registrar owns registration of that domain's modules, validators,
default validator chains, artifact codecs, schemas, and prompt assets. It does
not take ownership of application execution or process behavior.
## Alternatives considered
- Group modules by stage. This keeps interchangeable strategies side by side,
but scatters a domain's shared artifact model and collaborating extensions
across the repository. It is preferable when generic strategy libraries
dominate or when the project is primarily a stage-extension framework rather
than an application composed from domain suites.
- Put both shared types and `Register` in the domain root. This gives the
shortest import path but creates an import cycle once child implementations
import the root artifact types.
## Consequences
The repository layout makes supported domains immediately visible, and adding
or extracting a domain affects one cohesive subtree. The CLI composition root
depends on a small set of domain registrars instead of every leaf package.
Package moves must preserve user-visible module and validator keys unless a
separate compatibility decision changes them. Shared behavior that cannot be
expressed through framework contracts may need to move into a domain-neutral
framework package rather than creating a concrete peer-domain import. Registrar
packages become explicit composition points for instantiating generic typed
strategies, in addition to registering domain-owned implementations.

141
docs/adr/0005-cache-canonical-chunk-plans-by-source.md
View File

@@ -1,141 +0,0 @@
# ADR-0005: Cache one canonical chunk plan per source
**Status:** Accepted
**Date:** 2026-07-17
## Context
Notarius may run several extraction passes over the same source. A D&D
transcript, for example, may first produce NPC artifacts and later produce
spell or combat artifacts, with output from an earlier pass supplied as a
reference to a later pass.
An LLM-backed chunker may process an entire, potentially large source in one
expensive request. Recomputing boundaries for every pipeline or pass repeats
that cost and can make otherwise comparable extraction runs use different
source partitions. Stable chunk material also gives later extraction requests
a better opportunity to benefit from provider-side prompt caching.
Chunk boundaries can affect extraction quality. Evidence may span a boundary,
overlap may produce duplicates, and different partitions may change the context
available to a model. Merge and normalization should remove structural signs
of chunking from durable output, but they cannot guarantee recovery of evidence
that an extractor did not receive.
Notarius therefore needs an explicit policy for choosing between automatically
applying the latest chunking configuration and preserving one stable partition
for repeated work on the same source.
## Decision
Notarius assigns one active canonical chunk plan to a source and reuses that
plan by default across pipelines and invocations.
The canonical source identity is derived from the validated generic source
document and covers the source-unit identity, order, and content needed to
interpret plan boundaries. Input-adapter and chunk-producer identities are
recorded as provenance, but the active-plan lookup does not vary with:
- pipeline identity or selected artifact lanes;
- the configured chunk module or its options;
- references;
- LLM provider, model, profile, prompt, or response schema; or
- configuration for later pipeline stages.
When an active plan exists, Notarius uses it even if the current pipeline
configures a different chunk module or different chunk-module settings. The
configured chunk module generates a plan only when none exists or when the
operator explicitly requests recomputation.
The framework-owned minimum plan contract is an ordered, non-empty set of
source-unit ranges. Each range identifies the inclusive start and end unit for
one chunk. A chunk module may also provide namespaced, domain-specific
annotations at plan or range scope. Those annotations are stored with the plan
and passed through the pipeline when present, but they remain optional.
Downstream stages must not assume that annotations associated with the
currently configured chunk module are present on a reused plan produced by a
different module.
The cache stores the plan rather than fully materialized chunks. The framework
validates a reused plan against the current source and deterministically
materializes its ranges into chunks. The same source and plan must produce
byte-stable chunk input for later stages.
Canonical plan storage is a distinct cache surface with an independently
configurable location. It is not coupled to the roots or lifecycles of
invocation checkpoints, diagnostics, debug artifacts, or durable output. This
allows per-user and system-service deployments to apply cache-specific
ownership, permissions, placement, and cleanup policy without relocating other
Notarius state.
One mutable active plan is stored under the canonical source identity and
retains provenance for the module and relevant runtime inputs that produced it.
Refreshing the active plan atomically replaces that one mutable record; readers
must observe either the previous complete plan or the replacement complete
plan, never a partial update.
The effective plan producer is reported separately from the chunk module
requested by the current pipeline; reuse must not attribute cached boundaries
or annotations to a module that did not produce them.
Reuse is enabled by default. Operators can explicitly:
- bypass cached plans for an invocation without changing the active plan; or
- recompute a plan with the configured chunk module and make it active for
later work.
Exact storage layout, configuration fields, CLI syntax, publication mechanics,
recovery behavior, and diagnostics are implementation and operational
contracts rather than part of this decision.
## Alternatives considered
- Recompute chunks on every invocation. This always applies the current
chunking configuration, but repeats the most expensive stage and weakens
provider-side caching and cross-pass comparability.
- Cache every distinct chunking request by including module options,
references, prompts, profiles, and other runtime inputs in its identity. This
closely associates a cached result with its producing request, but reduces
reuse and permits boundary drift across operationally different passes.
- Key plans by source plus chunk module and options. This shares plans across
pipelines using the same strategy, but changing the configured strategy
silently selects a different partition rather than preserving one canonical
partition for the source.
- Require operators to name or supply a plan for every run. Explicit selection
is reproducible and may be useful as an advanced operation, but adds friction
to the default workflow and does not provide automatic reuse.
- Store fully materialized chunks. This simplifies loading, but duplicates
source content and couples durable state to the current chunk representation
rather than the stable boundary decision.
- Store canonical plans beneath the general workspace root. This would reuse an
existing location setting, but it couples a reusable application cache to
checkpoint, diagnostic, and debug state that have different ownership,
sensitivity, retention, and deployment requirements.
## Consequences
Independent pipelines and passes over the same source use stable boundaries by
default. This reduces repeated LLM work, improves cross-pass comparability, and
increases the opportunity for cached provider reads.
The configured chunk module may not execute, and its settings may have no
effect, when an active plan already exists. Domain-specific annotations reflect
the plan's original producer and may be absent or differ from those the current
module would produce. User-visible provenance must make the effective plan
clear.
A poor or outdated partition remains active until an operator replaces it.
This can preserve suboptimal context boundaries and affect extraction recall or
duplication even when merge and normalization hide the partition structure in
durable output. Stable reuse is an intentional priority over automatically
incorporating later chunk-strategy changes.
The framework gains a durable minimal chunk-plan contract and deterministic
materialization responsibility. Chunk modules must separate required boundary
output from optional annotations, and downstream modules may rely only on the
minimal boundary contract unless a future decision introduces an explicit plan
compatibility mechanism.
Operators must configure and secure canonical plan storage independently from
other workspace state when the per-user default is not appropriate. Removing
that cache remains recoverable because Notarius can regenerate it from the
source, but doing so may repeat an expensive LLM operation.

114
docs/adr/0006-separate-output-cache-and-debug-state.md
View File

@@ -1,114 +0,0 @@
# ADR-0006: Separate output, cache, and debug state
**Status:** Superseded by [ADR-0007](0007-separate-checkpoint-recording-from-reuse.md)
**Date:** 2026-07-17
## Context
Notarius currently exposes a workspace as a shared parent for checkpoints,
debug artifacts, and preferred diagnostics settings. Diagnostics are a second
inspection surface with their own enablement, directory, retention, and legacy
configuration. Durable output uses a separate CLI-selected root, while the
canonical chunk-plan cache introduced by ADR-0005 correctly uses an independent
cache root.
These concepts reflect implementation history more than operator intent. A user
must understand differences among workspace state, diagnostics, debug artifacts,
checkpoints, and chunk plans before deciding where Notarius may write. Some of
those distinctions are important internally: a redacted run summary has a
different sensitivity from a trace containing source material, prompts, and
model responses. They do not require separate public filesystem categories.
Notarius needs a smaller state model that communicates why data exists, how it
may be treated, and whether it is reconstructible.
## Decision
Notarius exposes three filesystem surfaces: output, cache, and debug. The
public workspace concept and diagnostics as a separate output surface are
removed.
### Output
Output is the durable result of a run and the only surface intended for normal
consumption. It contains the logical files produced by the output stage,
including the maintained result, manifest, warning, and rejection contracts.
Output is not cache or inspection state.
### Cache
Cache contains reconstructible state used to avoid repeated work or resume an
interrupted workflow. Canonical chunk plans and invocation checkpoints are
distinct cache families with independent identities, compatibility rules,
enablement policies, locations, and cleanup lifecycles.
ADR-0005 continues to govern canonical chunk-plan selection and reuse. Grouping
chunk plans and checkpoints under the public cache category does not permit a
checkpoint to compete with canonical plan reuse or couple their storage roots.
Checkpointing is an invocation policy rather than a prerequisite hidden in
persistent workspace configuration. An explicit resume invocation may read
compatible checkpoints and record replacement checkpoint state for work it
executes. Runs that do not request resume perform no checkpoint I/O.
### Debug
Debug is an explicitly requested per-run inspection bundle intended for
developers and troubleshooting. It is off by default. When enabled, one bundle
contains both redacted run summaries and detailed stage and LLM traces. The
internal distinction between a safe summary and a sensitive trace remains, but
there is one public enablement and location model.
Debug data is never a cache input and has no automatic retention policy.
Notarius does not create a debug directory unless debug is requested, and it
does not automatically delete a requested bundle. Credentials remain redacted
at every level, while the bundle as a whole is treated as potentially sensitive
because traces may contain source, reference, prompt, model-response, and
intermediate artifact content.
Concise progress, warnings, and failures continue to use stdout or stderr. A
run without debug may fail without producing a filesystem inspection record.
Exact configuration fields, CLI flags, default paths, layouts, compatibility
handling, and migration mechanics are configuration and operational contracts
rather than part of this decision.
## Alternatives considered
- Keep workspace, diagnostics, checkpoints, debug, and chunk-plan cache as
separate public concepts. This preserves compatibility and the current safe
default-on failure records, but retains overlapping configuration and asks
operators to reason about implementation-specific categories.
- Keep diagnostics as an always-available redacted operational surface and use
debug only for sensitive traces. This distinction is useful for a daemon or
managed service with an operational logging contract, but the current CLI can
report concise failures on stderr and provide inspection data when explicitly
requested.
- Put all non-output state beneath one physical root. This minimizes path
configuration, but couples reconstructible caches to per-run inspection data
and couples cache families whose identity, sensitivity, and cleanup policies
differ.
- Treat checkpoints as durable run state rather than cache. This emphasizes
resumability, but checkpoints are derived, compatibility-checked data that may
be deleted and recomputed. Cache more accurately describes their lifecycle.
## Consequences
The operator model becomes smaller: normal runs produce output and may use
cache; developers explicitly request debug. Public configuration no longer
exposes a workspace or overlapping diagnostics and debug systems.
The implementation retains separate collaborators and serializers where their
security or lifecycle boundaries differ. Redacted summaries remain useful as
the index to a debug bundle, and chunk plans and checkpoints retain separate
stores even though both are cache.
Existing configuration, environment variables, flags, examples, and
documentation require a deliberate compatibility transition. Default-on
diagnostic directories disappear, so failures without debug are inspectable
only through stderr and any durable output completed before the failure.
Debug becomes easier to request and substantially more complete, but enabling
it creates sensitive files that the operator must protect and remove. Cache
cleanup is recoverable but may repeat expensive work, while deleting output is
data loss from the user's perspective.

50
docs/adr/0007-separate-checkpoint-recording-from-reuse.md
View File

@@ -1,50 +0,0 @@
# ADR-0007: Separate checkpoint recording from reuse
**Status:** Accepted
**Date:** 2026-07-19
## Context
ADR-0006 made checkpoint I/O conditional on an explicit `--resume` invocation.
That policy requires an operator to anticipate the need for recovery before a
run begins. A failed ordinary run cannot reuse completed work because it did not
record checkpoints.
Recording reconstructible state and authorizing reuse are separate operational
decisions. Recording consumes storage and retains sensitive derived application
data, while reuse may change which module operations execute during a run.
## Decision
ADR-0006's separation of output, cache, and debug surfaces remains in effect;
this decision supersedes only its checkpoint invocation policy.
Checkpoint recording is controlled by an explicit persistent Boolean
configuration setting and remains disabled by default. When recording is
enabled, every run records checkpoint transitions and reusable approved stage
results.
Checkpoint loading remains an invocation policy. Only a run with `--resume`
loads and reuses compatible completed work. A recording-enabled run without
`--resume` executes every stage normally and never loads checkpoints. A resume
request while recording is disabled is rejected.
The existing checkpoint identities, compatibility rules, payload format,
filesystem root behavior, and pipeline collaborator contracts remain unchanged.
## Alternatives considered
- Continue coupling reads and writes to `--resume`. This is safe by default but
prevents recovery unless resume was anticipated on the earlier run.
- Always record checkpoints. This maximizes recovery but creates potentially
sensitive state without explicit operator consent.
- Add a multi-value recording policy. This preserves the old behavior as an
option but adds configuration complexity without a current need.
## Consequences
Operators can opt into recovery-ready runs while keeping checkpoint reuse
explicit. Enabled successful, rejected, and failed runs may all leave sensitive
checkpoint state, so operators remain responsible for access and retention.
Disabled configurations perform no checkpoint I/O, and `--resume` requires the
operator to enable recording first.

50
docs/adr/0008-ordered-pipeline-steps.md
View File

@@ -1,50 +0,0 @@
# ADR-0008: Bounded ordered pipeline steps and explicit artifact references
**Status:** Accepted
**Date:** 2026-07-21
## Context
Notarius currently models one pipeline-wide input, chunking plan, artifact
lanes, and output boundary. Some workflows need a deterministic handoff from
one set of normalized artifacts to a later set of artifacts, such as using
extracted NPC records while grounding later combat events. The workflow needs
an explicit topology without turning the pipeline into a general-purpose
workflow engine.
## Decision
Add an ordered collection of pipeline steps. Each step owns one or more
artifact lanes, and lanes within a step retain the existing independent
execution model. The pipeline continues to have one input, chunk plan, output,
and failure boundary. Steps are barriers: a later step may consume only
normalized artifacts from an earlier step.
Generated references use an explicit step-and-lane selector. Reference slots
declare the generated artifact kinds and media types they accept. The resolver
validates the topology, ordering, lane identity, artifact kind, schema, and
codec compatibility before execution. External references remain supported as
path sources, and the legacy top-level artifact map is interpreted as an
implicit `default` step.
Pipeline-level references may not select generated artifacts. General DAGs,
branches, loops, conditional execution, joins, and inferred dependencies are
not part of this model.
## Alternatives considered
- A general DAG would provide more flexibility but would also require a new
scheduler, lifecycle model, failure semantics, and provenance model.
- Separate pipeline runs connected through filesystem paths would lose the
static topology and typed compatibility checks.
- Inferring dependencies from module or lane names would make ordering and
configuration errors difficult to detect reliably.
## Consequences
The resolved pipeline has a deterministic, inspectable topology and can
include it in its identity digest. Configuration validation can reject invalid
generated bindings before any work begins. Existing single-step profiles keep
their behavior through the implicit `default` step. Execution handoff and
multi-step scheduling require follow-up work in the runner and checkpoint
layers.

98
docs/adr/0009-minimal-evidence-grounded-extraction-artifacts.md
View File

@@ -1,98 +0,0 @@
# ADR-0009: Prefer minimal evidence-grounded extraction artifacts
**Status:** Accepted
**Date:** 2026-07-22
## Context
Notarius is intended to extract structured facts from source material. Several
early D&D artifacts grew to include descriptive prose, inferred relationships,
immediate outcomes, summaries, and other enrichment alongside the facts that
identify an event or entity. Those fields make one model call responsible for
both extraction and synthesis.
In practice, the richer contracts have produced overlapping or weakly grounded
fields and have made structurally valid, semantically coherent output harder for
cost-effective smaller models. They also increase prompt size, validation and
normalization policy, durable schema surface, downstream coupling, and the
number of claims whose provenance must be evaluated.
The application needs a consistent rule for deciding what belongs in an
extractor before redesigning the current D&D spell, NPC, and combat-turn
contracts or adding new artifact families.
## Decision
An extraction module answers one narrowly stated question and returns the
smallest durable structured artifact that usefully answers it.
Every model-produced field in an extraction artifact must:
- be necessary to answer the extractor's stated question or serve a known
downstream consumer;
- represent a fact or bounded classification that can be supported directly by
cited source ranges;
- remain independently meaningful without model-generated explanatory prose;
and
- justify the additional prompt, schema, validation, normalization, and
compatibility surface it creates.
Source references are required provenance for extracted records. Auxiliary
references may disambiguate identities or canonical names, but they do not
establish source facts and are not copied into evidence.
Extraction artifacts do not include narrative summaries, general analysis,
speculative enrichment, inferred biography or relationships, or redundant
free-text descriptions by default. When such output has a demonstrated use, it
belongs in an explicitly named extraction, classification, enrichment, or
analysis module with its own contract and evidence policy.
Occurrence-level facts are not forced into entity-level attributes. A fact
that can change between encounters, such as an NPC's role in a scene, belongs
on an occurrence artifact rather than as one scalar property of a normalized
NPC registry entry.
Deterministic mapping and normalization may assign application-owned
identifiers, canonicalize known catalog values, order and deduplicate evidence,
and collapse records under an explicit identity rule. They must not manufacture
removed descriptive fields or synthesize missing claims to satisfy an older
contract.
This is a default design rule, not a prohibition on rich artifacts. A richer
field is appropriate when its consumer, evidence semantics, and ownership are
explicit.
## Alternatives considered
- Keep rich schemas and improve prompts or use larger models. This retains
potentially convenient prose but does not resolve overlapping field
responsibilities, weak provenance, higher cost, or unnecessary downstream
coupling.
- Make enrichment fields optional. This reduces rejection pressure but leaves
ambiguous artifact semantics and inconsistent records, and many strict
structured-output providers still require nullable placeholders.
- Keep minimal private LLM schemas while preserving rich durable artifacts.
Deterministic code would have to invent, default, or separately derive the
missing fields, hiding synthesis behind the extraction boundary.
- Use one broad session-analysis module. This reduces the number of lanes but
couples unrelated facts, schemas, retries, evaluation, and downstream
consumers into one model call.
## Consequences
Extraction prompts and response schemas become smaller, more focused, and more
suitable for lower-cost models. Artifacts carry fewer unsupported claims, and
their evidence and validation policies become easier to explain and evaluate.
Independent extractors can evolve, retry, and be consumed without requiring
unrelated enrichment.
Some descriptive convenience fields will disappear from primary artifacts.
Consumers that genuinely need them may require a separate module and explicit
pipeline step. Entity registries may no longer resolve aliases or relationships
unless a dedicated, evidence-grounded capability supplies them.
Removing durable fields is a schema compatibility change. Each affected
artifact requires an explicit version and reference policy; private prompt
changes alone are insufficient. Current-behavior integration and internal
documentation must change with implementation, while the roadmap owns the
proposed contract until then.

49
docs/adr/0010-workload-oriented-llm-profile-defaults.md
View File

@@ -1,49 +0,0 @@
# ADR-0010: Use workload-oriented LLM profile defaults
**Status:** Accepted
**Date:** 2026-08-03
## Context
LLM-backed D&D operations share an execution-policy choice, but repeating a
provider or model-named profile on every module binding ties pipeline structure
to a deployment decision. Different environments may require different model,
backend, timeout, or reasoning settings while retaining the same workload.
Notarius also needs a usable default for maintained D&D prompts without making
an operator profile mandatory. That default must remain owned by the D&D
family, while generic LLM infrastructure stays unaware of domain-specific
policy.
## Decision
Pipelines may name one workload-oriented default profile, inherited only by
selected LLM-backed bindings and validators. Binding-level profile IDs remain
intentional exceptions, and the run-wide CLI profile override has highest
precedence.
The D&D family owns an embedded fallback profile named `dnd-extraction`.
Operators may provide a complete profile with the same ID through a PromptKit
filesystem source. PromptKit selects the higher-precedence matching definition;
Notarius does not merge profile documents. Production, development, and local
deployments can therefore use different execution policy behind one unchanged
pipeline ID.
## Alternatives considered
- Repeat a model-named profile on every binding. This makes routine deployment
policy changes noisy and obscures the shared workload intent.
- Require every deployment to install a profile file. This adds configuration
friction and leaves maintained D&D prompts without an application-owned
fallback.
- Put D&D profile policy in generic LLM infrastructure. This breaks domain
ownership and makes generic code depend on one workload.
## Consequences
Pipeline configuration expresses workload intent rather than a specific
provider or model. Operators can replace the complete execution policy without
editing bindings, while binding-level and run-wide exceptions remain available.
Profile changes affect resolved pipeline and checkpoint identity, so they may
intentionally cause work to be recomputed. The D&D fallback becomes a
maintained application execution-policy asset.

165
docs/cli.md
View File

@@ -1,165 +0,0 @@
# CLI Reference
This is the canonical reference for the implemented Notarius command-line
interface. For the shortest successful run, see the [README](../README.md).
Configuration fields, discovery rules, and selectable module keys are defined
in [Configuration](config.md); runtime state and recovery procedures are
defined in [Operations](operations.md).
## Command Summary
~~~
notarius help
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list [--config path/to/config.yml] [--json]
~~~
Running Notarius without arguments, or with **help**, **--help**, or **-h**,
writes the command summary to standard output and exits with status 0.
## run
~~~
notarius run <pipeline-id> --input path/to/source.json [--json] [flags]
~~~
The **run** command executes the named pipeline for one input file. The
pipeline ID and **--input** are required.
| Flag | Meaning |
| --- | --- |
| **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). |
| **--input path** | Source input file to process. Required. |
| **--output-dir path** | Override the configured output root for this run. |
| **--json** | Write the successful run-result receipt as JSON to standard output. |
| **--chunk_cache auto\|bypass\|refresh** | Override chunk-plan cache handling for this run. |
| **--resume** | Reuse compatible recorded checkpoints when checkpoint recording is enabled. |
| **--recompute-step step-id** | With **--resume**, recompute the selected ordered step and its dependent lanes. It cannot be combined with **--only**. |
| **--debug** | Retain a debug bundle for this run. |
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
| **--llm-profile id** | Highest-precedence configured profile for selected LLM-backed bindings and validators; it replaces binding and [pipeline](config.md#pipelines) defaults. |
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
| **--reasoning-effort value** | Replace the selected PromptKit profile's reasoning effort for every LLM-backed call in this run. The value must be non-empty and the flag may be specified only once. |
| **--clear-reasoning-effort** | Clear reasoning effort inherited from the selected PromptKit profile for every LLM-backed call in this run. |
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
**--debug-dir**, **--output-dir**, **--session-id**, and
**--reasoning-effort**, and **--recompute-step** reject explicit empty values.
**--reasoning-effort** and **--clear-reasoning-effort** are mutually exclusive.
When neither is present, reasoning effort comes from the selected PromptKit
profile. These controls apply to the shared run client, including retries and
LLM-backed validators, and do not modify configuration or profile files.
Persistent reasoning settings remain a PromptKit profile concern.
**--recompute-step** requires **--resume**; checkpoint requirements and reuse
behavior are documented in [Operations](operations.md).
### Reference selectors
Use **--reference** only for a reference slot declared by the selected
configured target. The accepted selector forms are:
| Form | Target |
| --- | --- |
| slot=path | The unique selected target that declares slot. |
| chunk.slot=path | The chunker. |
| merge.slot=path | The unique selected merger that declares slot. |
| lane.slot=path | The unique extractor, merger, or normalizer in lane that declares slot. |
| lane.extract.slot=path | The extractor in lane. |
| lane.merge.slot=path | The merger in lane. |
| lane.normalize.slot=path | The normalizer in lane. |
**--without-reference** uses the same selector forms without =path. Slot
names, requiredness, and configured bindings are part of the
[configuration contract](config.md).
### Run output
Without **--json**, standard output contains the completed pipeline ID, counts
of normalized and rejected outputs, and the output directory. A debug-enabled
run also prints its debug-bundle path to standard output. A successful run with
warnings reports the warning count to standard error. The published JSON bundle
is defined by the [JSON output contract](integrations/json-output.md).
With **--json**, successful standard output is exactly one
`notarius.run-result.v1` JSON document followed by a newline, with no
human-oriented status or debug-path line. Its fields and compatibility policy
are defined by the [run-result contract](integrations/run-result.md). A caller
must check for exit status 0 before decoding this output; a failed write can
leave incomplete standard-output bytes that are not a result document.
Example:
~~~
OPENROUTER_API_KEY=your-api-key \
go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json
~~~
## config validate
~~~
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
~~~
This command loads and validates a configuration. With **--pipeline**, it also
resolves that pipeline against the production module catalog. **--only** selects
lanes during that resolution and requires **--pipeline**.
Success is written to standard output as either config "<path>" is valid or
config "<path>" is valid for pipeline "<pipeline-id>".
Examples:
~~~
go run ./cmd/notarius config validate \
--config examples/dnd-minimal.config.yml \
--pipeline dnd-session
OPENROUTER_API_KEY=validation-placeholder \
go run ./cmd/notarius config validate \
--config examples/dnd-complete.config.yml \
--pipeline dnd-session
~~~
The placeholder in the second command is sufficient only for offline
validation; it cannot run a provider-backed pipeline.
## pipelines list
~~~
notarius pipelines list [--config path/to/config.yml] [--json]
~~~
This command lists configured pipeline IDs in sorted order. By default, it
writes one ID per line to standard output. **--json** writes an object shaped as
{"pipelines":[...]} instead.
~~~
go run ./cmd/notarius pipelines list \
--config examples/dnd-minimal.config.yml
~~~
## Output Streams And Exit Statuses
Successful commands write their primary result to standard output. Warnings and
errors are written to standard error.
For **run --json**, warnings remain on standard error and standard output is a
machine-readable success result only. Syntax and runtime diagnostics remain on
standard error. Parse the result only after the process exits with status 0.
| Status | Meaning |
| --- | --- |
| 0 | The command completed successfully, including root help. |
| 1 | Command syntax was valid but configuration loading or validation, pipeline resolution or execution, provider use, output, or requested debug handling failed. |
| 2 | The command or flag syntax was invalid, including unknown commands, missing required arguments, invalid flag values, or invalid flag combinations. |
The root help spellings are the supported help path. Invoking **--help** on
**run**, **config validate**, or **pipelines list** is handled by the flag
parser as a usage error: it writes an error to standard error and exits with
status 2.

446
docs/config.md
View File

@@ -1,446 +0,0 @@
# Configuration
This is the canonical reference for Notarius configuration. Configuration files
are YAML and must declare version 4. They select pipelines and their modules;
the [CLI reference](cli.md) owns invocation syntax, and
[Operations](operations.md) owns run-state procedures.
## Configuration Discovery And Precedence
Commands that load configuration choose a file in this order:
1. a non-empty **--config** CLI value;
2. a non-empty **NOTARIUS_CONFIG** environment value;
3. the installed default file at **/usr/local/etc/notarius/config.yml**, when
it exists.
The command fails if none of these paths provides a configuration file.
For configuration values, precedence is:
1. built-in defaults;
2. the selected YAML file;
3. supported operational environment variables; and
4. the CLI run overrides that apply to a command.
Environment variables do not provide a second configuration schema. They only
override the fields listed below.
## Maintained Examples
- [Minimal D&D configuration](../examples/dnd-minimal.config.yml) is a
single-lane Seriatim-to-spell pipeline.
- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses
ordered steps, all implemented D&D lanes, generated references, state
settings, bounded LLM concurrency, and the maintained
[operator profile](../examples/profiles/dnd-extraction.yml).
Use these complete files as starting points rather than combining the
illustrative fragments in this reference.
## File Shape And Defaults
Unknown fields, duplicate mapping keys, empty identifiers, and identifiers that
become duplicates after trimming whitespace are rejected. Every top-level field
other than **version** is optional.
| Field | Type | Default | Rules |
| --- | --- | --- | --- |
| **version** | integer | none | Required; must be 4. |
| **promptkit** | object | none | Profile source and optional local-backend configuration. |
| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. |
| **concurrency** | object | see below | Global LLM and extraction limits. |
| **output** | object | see below | Published output settings. |
| **cache** | object | see below | Chunk-plan and checkpoint settings. |
| **debug** | object | see below | Debug-bundle root only; it does not enable capture. |
Built-in defaults are:
| Field | Default |
| --- | --- |
| **concurrency.total_llm** | 1 |
| **concurrency.stage_workers.extract** | Effective **total_llm** |
| **output.directory** | **./notarius-output** |
| **cache.chunk_plans.mode** | **auto** |
| **cache.chunk_plans.directory** | Empty, selecting the per-user chunk-plan root |
| **cache.checkpoints.enabled** | false |
| **cache.checkpoints.directory** | Empty, selecting the per-user checkpoint root |
| **debug.directory** | **./notarius-debug** |
An empty cache directory in YAML deliberately selects the corresponding
per-user root. An explicit empty output or debug directory is invalid.
## PromptKit Profiles
The optional **promptkit** object selects one source of profile definitions and
may register one conventional local OpenAI-compatible backend:
~~~yaml
version: 4
promptkit:
profile_dir: ./profiles
# profile_file: ./profiles.yml
local_backend:
endpoint: http://localhost:8000/v1
concurrency_limit: 2
~~~
| Field | Type | Rules |
| --- | --- | --- |
| **profile_dir** | string | Non-empty directory containing profile files. |
| **profile_file** | string | Non-empty profile file. |
| **local_backend** | object | Optional registration for the conventional PromptKit backend ID **local**. |
| **local_backend.endpoint** | string | Required when **local_backend** is present; absolute HTTP or HTTPS URL with a host. |
| **local_backend.concurrency_limit** | integer | Optional non-negative limit; defaults to 0. |
Set at most one of **profile_dir** and **profile_file**. Relative values use
the process working directory, not the configuration file's directory. The
complete example's `./examples/profiles/dnd-extraction.yml` value is therefore
valid when Notarius is launched from the repository root; use an absolute path
for services and containers.
An operator source is optional. For a requested ID, PromptKit checks the
configured operator source first, then Notarius's embedded fallback profiles,
then its own built-in catalog. A matching profile is complete: it replaces a
lower-precedence definition rather than merging with it. The maintained
[`dnd-extraction` operator profile](../examples/profiles/dnd-extraction.yml)
is a secret-free deployment artifact; production, development, and local
deployments can each provide a complete definition with that same workload ID.
Use workload-oriented IDs for new profiles instead of model names.
[Operations](operations.md#promptkit-profile-deployment) owns the deployment
workflow and credential-handling guidance.
When **local_backend** is present, its endpoint is trimmed and must use HTTP or
HTTPS case-insensitively, be absolute, and have a non-empty host. URL paths are
allowed. User information, queries, and fragments are rejected. A zero
**concurrency_limit** leaves the local backend unrestricted inside PromptKit;
a positive value limits simultaneous local generations. The application-wide
**concurrency.total_llm** limit still applies in both cases. Neither local
backend field has an environment override. Omitting **local_backend** registers
nothing and preserves existing built-in and endpoint-only profile behavior.
A file-backed PromptKit profile selects the registration by its case-sensitive
backend ID:
~~~yaml
id: local-summary
backend: local
model: example-model
~~~
Keep credentials out of the local-backend object. A PromptKit profile may name
its credential environment variable through `api_key_env`; set that variable
only in the run environment. PromptKit owns the
[pinned profile-file format](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md).
The [PromptKit upstream boundary](integrations/pkg-promptkit.md) identifies the
supported package API, and [Operations](operations.md#operational-limits)
describes the effective concurrency layers.
`notarius config validate --pipeline <id>` resolves the selected pipeline and
inspects every explicit effective profile without contacting a provider or
requiring credential values. It rejects absent, malformed, or incompatible
profiles before a run prepares modules. Credential availability is checked only
when a generation is prepared.
## Migrating Version 3 Configuration
Version 3 files are not decoded or rewritten. Change **version: 3** to
**version: 4** and rename the top-level **scriptorium:** section to
**promptkit:**. Version 4 decoding is strict, so a remaining **scriptorium**
field is rejected as unknown.
## Operational Environment Variables
These variables are applied after YAML values:
| Variable | Overrides | Rules |
| --- | --- | --- |
| **NOTARIUS_TOTAL_LLM_CONCURRENCY** | **concurrency.total_llm** | Integer. |
| **NOTARIUS_STAGE_WORKERS_EXTRACT** | **concurrency.stage_workers.extract** | Integer. |
| **NOTARIUS_OUTPUT_DIR** | **output.directory** | Non-empty path. |
| **NOTARIUS_CACHE_CHUNK_PLANS_MODE** | **cache.chunk_plans.mode** | **auto**, **bypass**, or **refresh**. |
| **NOTARIUS_CACHE_CHUNK_PLANS_DIR** | **cache.chunk_plans.directory** | Non-empty path. |
| **NOTARIUS_CACHE_CHECKPOINTS_DIR** | **cache.checkpoints.directory** | Non-empty path. |
| **NOTARIUS_DEBUG_DIR** | **debug.directory** | Non-empty path. |
Integer values are trimmed then parsed as base-10 integers. Directory and
output values reject NUL characters. **NOTARIUS_CONFIG** participates only in
configuration discovery.
## Concurrency, Output, Cache, And Debug
~~~yaml
concurrency:
total_llm: 2
stage_workers:
extract: 2
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: auto
directory: ./notarius-cache/chunk-plans
checkpoints:
enabled: true
directory: ./notarius-cache/checkpoints
debug:
directory: ./notarius-debug
~~~
**concurrency.total_llm** must be greater than zero. The only supported
**concurrency.stage_workers** key is **extract**; its value must be from 1
through **total_llm**. When omitted, it is recalculated from the effective
**total_llm** after YAML and environment precedence.
**cache.chunk_plans.mode** accepts **auto**, **bypass**, or **refresh**.
**cache.checkpoints.enabled** is a boolean. The CLI can override the output
directory and chunk-plan mode for one run; see [CLI reference](cli.md#run).
## Pipelines
Each **pipelines** entry has a unique, non-empty ID and the following shape:
~~~yaml
pipelines:
dnd-session:
llm_profile: dnd-extraction
input: seriatim
chunk: generic
output: json
artifacts:
spells:
extract: dnd/spells
merge: appendorder
normalize: dnd/spells
~~~
| Field | Type | Default | Rules |
| --- | --- | --- | --- |
| **llm_profile** | string | none | Optional non-empty default PromptKit profile ID for selected LLM-backed bindings and validators. An explicitly present blank value is invalid. |
| **input** | module binding | none | Required. |
| **chunk** | module binding | **generic** | Optional. |
| **output** | module binding | **json** | Optional. |
| **artifacts** | map | none | Compact single-step lane map. |
| **steps** | list | none | Ordered lane definitions. Mutually exclusive with **artifacts**. |
| **references** | map | none | External reference defaults for eligible targets. |
Use either **artifacts** or **steps**. The compact **artifacts** form is an
implicit single step. An explicit **steps** list must be non-empty; every step
needs a unique non-empty **id**, an **artifacts** map, and may have
**references**. A lane ID must not appear more than once in a pipeline,
including across explicit steps.
For each selected LLM-backed binding or validator, profile selection occurs
after module, validator, and `--only` lane selection. It uses the
run-level **--llm-profile** value first, then the binding's **llm_profile**,
then the pipeline's **llm_profile**, and finally the PromptKit default.
Deterministic bindings do not receive these defaults or run overrides.
A lane has these fields:
| Field | Type | Default | Rules |
| --- | --- | --- | --- |
| **extract** | module binding | none | Required. |
| **merge** | module binding | **appendorder** | Optional. |
| **normalize** | module binding | **noop** | Optional. |
| **references** | map | none | Supported compatibility alias for **extract.references**. |
| **validators** | list | none | Non-empty lane-level lists are rejected. Set validator overrides on a binding instead. |
The lane-level **references** alias remains accepted. When the alias and
**extract.references** bind the same slot, **extract.references** wins. Use
the binding-local form in new configurations.
## Module Bindings And Validators
Use a module key directly when no other binding fields are needed:
~~~yaml
input: seriatim
~~~
Use an object for fields:
~~~yaml
extract:
module: dnd/spells
llm_profile: dnd-extraction
retries: 2
references:
spell_catalog: ./dnd-spell-catalog.json
~~~
| Binding field | Type | Default | Rules |
| --- | --- | --- | --- |
| **module** | string | none | Required for an object binding. Must be a registered compatible key. |
| **llm_profile** | string | none | Optional non-empty PromptKit profile ID for an LLM-backed binding. It overrides the pipeline default unless the run supplies **--llm-profile**. |
| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. |
| **options** | object | none | Must satisfy the selected module. |
| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. |
| **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. |
Omitting **validators** uses the registered chain. **validators: []** selects
an empty chain; a non-empty list replaces the chain in the listed order.
Validator bindings accept only **module**, **llm_profile**, and **options**.
They reject **references**, **retries**, and nested **validators**. Deterministic
validators reject an explicit **llm_profile**. Deterministic module bindings
also reject an explicit **llm_profile**.
The **json** output module accepts optional **include_chunk_map** and
**evidence_context** settings:
~~~yaml
output:
module: json
options:
include_chunk_map: true
evidence_context:
enabled: true
window_units: 3
lanes:
- npcs
- spells
~~~
**include_chunk_map** is a boolean and defaults to false. It adds the accepted
chunk map when one exists; its wire format is defined in the
[chunk-map contract](integrations/chunk-map.md).
Omitting **evidence_context** disables evidence publication. When present, it
is an object with these strict fields:
| Field | Type | Rules |
| --- | --- | --- |
| **enabled** | boolean | Required. `false` permits no other evidence fields. |
| **lanes** | array of strings | Required and non-empty when enabled. Each value is trimmed and must be unique; every value must name a configured pipeline lane. |
| **window_units** | non-negative integer | Optional when enabled; defaults to 3. Zero retains only directly cited units. |
Unknown outer or nested option fields are rejected, as are incompatible YAML
types. The allowlist remains valid when a run uses lane filtering: a configured
lane that is not active for that invocation simply contributes no evidence.
Evidence publication is opt-in because it can persist source text and metadata.
Its payload contract is [Published Evidence Context](integrations/evidence-context.md).
## References And Ordered Handoffs
Reference maps bind named slots that the selected target declares. A scalar is
an external path. Pipeline-level maps accept only external paths; step-local
and binding-local maps may also select a normalized artifact from an earlier
step:
~~~yaml
steps:
- id: describe-session
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: extract-events
references:
npcs:
artifact:
step: describe-session
lane: npcs
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
~~~
An artifact selector contains only **step** and **lane**. The producer must be
an earlier step and the selected artifact must be compatible with the consumer
slot. A generated binding supplies one accepted normalized artifact; it does
not name a file. A configured generated dependency remains required even when
that consumer slot is otherwise optional.
Pipeline references are defaults. A matching step-local or binding-local
external path overrides a pipeline default. Required slots must be bound after
these configuration values and any CLI reference overrides are applied.
Reference paths in YAML are resolved relative to the configuration file.
### D&D Reference Slots
The following slot names are accepted by the implemented D&D modules when the
selected target declares them:
| Slot | Source and use |
| --- | --- |
| **party** | Optional text campaign context. This is the canonical party-roster spelling. |
| **roster** | Accepted compatibility alias for **party**. |
| **players** | Optional text player context. |
| **glossary** | Optional text campaign glossary. |
| **spell_catalog** | Optional JSON spell-catalog overlay for spell extraction and normalization. See [spell-catalog overlays](integrations/dnd-spell-catalog-overlays.md). |
| **npcs** | Normalized NPC registry. Optional for spells and combat turns; required for NPC interactions. |
| **scene_descriptions** | Required normalized scene-description artifact for combat-turn extraction. |
Scene descriptions accept **party**, **players**, and **glossary**, but not
**roster**. NPC interactions require **npcs** for both extraction and
normalization. Combat turns require **scene_descriptions** for extraction; the
normalized combat-turn module may use optional **npcs**. The complete example
shows generated **npcs** and **scene_descriptions** bindings.
## Production Module Keys
| Kind | Keys |
| --- | --- |
| Input | **seriatim** |
| Chunk | **generic**, **dnd/scenes** |
| Extract | **dnd/spells**, **dnd/npcs**, **dnd/combat-turns**, **dnd/item-events**, **dnd/npc-interactions**, **dnd/scene-descriptions** |
| Merge | **appendorder** |
| Normalize | **noop**, **dnd/spells**, **dnd/npcs**, **dnd/combat-turns**, **dnd/item-events**, **dnd/npc-interactions**, **dnd/scene-descriptions** |
| Output | **json** |
The D&D artifact contracts define each emitted schema:
[spells](integrations/dnd-spell-artifacts.md),
[NPCs](integrations/dnd-npc-artifacts.md),
[NPC interactions](integrations/dnd-npc-interaction-artifacts.md),
[combat turns](integrations/dnd-combat-turn-artifacts.md),
[item events](integrations/dnd-item-event-artifacts.md), and
[scene descriptions](integrations/dnd-scene-description-artifacts.md).
## Production Validator Keys And Default Chains
Available validator keys are:
| Family | Keys |
| --- | --- |
| Generic | **generic/always_accept**, **generic/always_reject**, **generic/valid_json**, **generic/valid_json_schema** |
| Spells | **extract/dnd/spells/shape**, **extract/dnd/spells/catalog**, **extract/dnd/spells/source_refs**, **extract/dnd/spells/source_relatedness** |
| NPCs | **extract/dnd/npcs/shape**, **extract/dnd/npcs/source_refs**, **extract/dnd/npcs/source_relatedness**, **normalize/dnd/npcs/identity** |
| Combat turns | **extract/dnd/combat-turns/shape**, **extract/dnd/combat-turns/source_refs**, **extract/dnd/combat-turns/source_relatedness**, **normalize/dnd/combat-turns/invariants** |
| Item events | **extract/dnd/item-events/shape**, **extract/dnd/item-events/source_refs**, **extract/dnd/item-events/source_relatedness**, **normalize/dnd/item-events/invariants** |
| NPC interactions | **extract/dnd/npc-interactions/shape**, **extract/dnd/npc-interactions/registry**, **extract/dnd/npc-interactions/source_refs**, **extract/dnd/npc-interactions/source_relatedness**, **normalize/dnd/npc-interactions/invariants** |
| Scene descriptions | **extract/dnd/scene-descriptions/shape**, **extract/dnd/scene-descriptions/source_refs**, **extract/dnd/scene-descriptions/source_relatedness**, **normalize/dnd/scene-descriptions/invariants** |
When no override is configured, production D&D bindings use the following
ordered chains. Each row lists extract then normalize; spell chains are the
same at both stages.
| Lane | Extract | Normalize |
| --- | --- | --- |
| Spells | generic/valid_json, extract/dnd/spells/shape, extract/dnd/spells/catalog, extract/dnd/spells/source_refs, generic/valid_json_schema, extract/dnd/spells/source_relatedness | Same as extract |
| NPCs | generic/valid_json, extract/dnd/npcs/shape, extract/dnd/npcs/source_refs, generic/valid_json_schema, extract/dnd/npcs/source_relatedness | generic/valid_json, extract/dnd/npcs/shape, normalize/dnd/npcs/identity, extract/dnd/npcs/source_refs, generic/valid_json_schema, extract/dnd/npcs/source_relatedness |
| Combat turns | generic/valid_json, extract/dnd/combat-turns/shape, extract/dnd/combat-turns/source_refs, generic/valid_json_schema, extract/dnd/combat-turns/source_relatedness | generic/valid_json, extract/dnd/combat-turns/shape, normalize/dnd/combat-turns/invariants, extract/dnd/combat-turns/source_refs, generic/valid_json_schema, extract/dnd/combat-turns/source_relatedness |
| Item events | generic/valid_json, extract/dnd/item-events/shape, extract/dnd/item-events/source_refs, generic/valid_json_schema, extract/dnd/item-events/source_relatedness | generic/valid_json, extract/dnd/item-events/shape, normalize/dnd/item-events/invariants, extract/dnd/item-events/source_refs, generic/valid_json_schema, extract/dnd/item-events/source_relatedness |
| NPC interactions | generic/valid_json, extract/dnd/npc-interactions/shape, extract/dnd/npc-interactions/registry, extract/dnd/npc-interactions/source_refs, generic/valid_json_schema, extract/dnd/npc-interactions/source_relatedness | generic/valid_json, extract/dnd/npc-interactions/shape, extract/dnd/npc-interactions/registry, normalize/dnd/npc-interactions/invariants, extract/dnd/npc-interactions/source_refs, generic/valid_json_schema, extract/dnd/npc-interactions/source_relatedness |
| Scene descriptions | generic/valid_json, extract/dnd/scene-descriptions/shape, extract/dnd/scene-descriptions/source_refs, generic/valid_json_schema, extract/dnd/scene-descriptions/source_relatedness | generic/valid_json, extract/dnd/scene-descriptions/shape, normalize/dnd/scene-descriptions/invariants, extract/dnd/scene-descriptions/source_refs, generic/valid_json_schema, extract/dnd/scene-descriptions/source_relatedness |
Chains are only registered for the D&D extract and normalize modules shown
above; select an explicit override when a different compatible chain is
required.
## Validation
Validate a file and one pipeline before running it:
~~~sh
go run ./cmd/notarius config validate \
--config examples/dnd-minimal.config.yml \
--pipeline dnd-session
~~~
Configuration validation rejects invalid YAML, unsupported fields, invalid
defaults or environment overrides, incompatible module keys, unknown options,
invalid reference bindings, missing required reference slots, invalid validator
overrides, and incompatible generated artifact handoffs. Use
[pipelines list](cli.md#pipelines-list) to inspect configured IDs.

74
docs/consumers/subprocess.md
View File

@@ -1,74 +0,0 @@
# Using Notarius As A Subprocess
Use this workflow when an orchestrator runs Notarius and consumes its published
artifacts. The [CLI reference](../cli.md) owns invocation syntax and exit
statuses, while the [run-result receipt](../integrations/run-result.md) and
[Published JSON Output contract](../integrations/json-output.md) own the
durable result formats.
## Run And Check The Process
Optionally preflight a selected configuration and pipeline before work starts:
```sh
notarius config validate --config /path/to/notarius.yml --pipeline pipeline-id
```
Invoke the run with explicit paths and machine-readable output. Capture
standard output and standard error separately; do not combine them before
processing the result.
```sh
notarius run pipeline-id \
--config /path/to/notarius.yml \
--input /path/to/source.json \
--output-dir /path/to/output-root \
--json
```
Use absolute paths for supplied input, configuration, output-root, and
reference files. When a stable prompt session identifier or references are
needed, pass the supported CLI flags. Supply credentials through Notarius's
documented configuration and environment mechanisms, never as command-line
arguments or generated secret-bearing configuration.
Wait for the process before interpreting standard output. Only an exit status
of 0 permits decoding the receipt. On a nonzero exit, retain standard error for
diagnosis and ignore all standard-output bytes: a failed receipt write may have
left a partial document.
## Discover Required Artifacts
Decode the successful receipt and accept the schema versions supported by the
caller. Use its `output_directory` as the bundle root. For the production JSON
output, resolve `index_file` under that root with a confinement check and reject
an absolute path or a result that escapes the root.
Read the resulting `index.json` and locate each artifact by `lane_id`, not by a
guessed filename. Before decoding a selected payload, verify its descriptor's
media type and schema identity against the relevant published artifact
contract. The JSON bundle contract links to the available lane contracts.
If `index.json` has an `evidence_context` descriptor, treat it as a
pipeline-wide artifact rather than a lane entry. Verify its six descriptor
fields before decoding the linked file according to the [Published Evidence
Context contract](../integrations/evidence-context.md). Use each
`evidence_refs` entry as the citation to source material. Its surrounding
context range and included units explain the citation, but do not widen or
replace the cited source reference.
A zero exit status may still report rejected outputs, warnings, or absent
lanes. The caller decides which lane IDs are required for its own work and
which are optional; it should make that decision explicitly rather than infer
failure from the receipt counts alone.
## Preserve Provenance And Handle Data Carefully
Keep the receipt with the published `manifest.json`, and retain
`rejected.json` and `warnings.json` when review or later provenance requires
them. Treat the input, output bundle, cache, debug bundle, and captured process
logs as potentially sensitive data. Apply the caller's access controls and
retention policy, and avoid copying secrets into arguments, logs, or
provenance records. An evidence-context artifact contains source-unit text and
metadata, and selected lanes can cover most of an input; preserve and share it
only when that source content is authorized for the recipient.

43
docs/development.md
View File

@@ -1,43 +0,0 @@
# Development
This is the first-read landing page for people and LLM coding agents working on
Notarius. It provides a concise repository orientation and routes each kind of
change to its canonical documentation.
Notarius is a Go CLI for configured structured extraction workflows. Start with
the [README](../README.md) for product context, [Architecture](policy/architecture.md)
for system boundaries, and [Internal Overview](internal/overview.md) for the
implemented component map.
## What To Read
| When working on | Read | Why |
| --- | --- | --- |
| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internals. |
| Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
| Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. |
| CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. |
| Building a subprocess caller or changing its result protocol | [Subprocess Consumer Guide](consumers/subprocess.md), [Run Result Receipt](integrations/run-result.md), and [CLI Internals](internal/cli.md) | These separate caller workflow, durable receipt contract, and CLI implementation behavior. |
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and PromptKit integration. |
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
For an existing subsystem, also inspect its focused tests and the package-local
types and contracts before changing behavior.
## Validation
Use focused package tests while iterating. Run the repository-wide checks when
a change affects shared contracts, application behavior, or maintained
documentation examples:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
```

81
docs/integrations/chunk-map.md
View File

@@ -1,81 +0,0 @@
# Accepted Chunk Map
This document defines the optional durable `chunk-map.json` artifact in a
[published JSON bundle](json-output.md). It describes the accepted,
materialized chunk plan used by one run. It is not a lane payload and is never
an input to a later pipeline step.
## Contract Identity
| Property | Value |
| --- | --- |
| Artifact kind | `source/chunk-map` |
| Logical file | `chunk-map.json` |
| Media type | `application/json` |
| Schema ID | `notarius.source.chunk_map` |
| Schema name | `notarius_source_chunk_map_v1` |
| Schema version | `v1` |
The optional `chunk_map` descriptor in `index.json` identifies this artifact.
Export is controlled by the JSON output binding described in
[Configuration](../config.md#module-bindings-and-validators).
## Wire Shape
Every payload has these required fields:
| Field | Meaning |
| --- | --- |
| `source_id` | Accepted source-document identity. |
| `source_digest` | Lower-case `sha256:` digest of that source document. |
| `plan_digest` | Lower-case `sha256:` digest of the logical chunk plan. |
| `requested_chunker` | Chunk module selected by the resolved pipeline. |
| `producer` | Original accepted-plan producer. `input_module` and `chunk_module` are required; `llm_profile` is optional. |
| `plan_annotations` | Plan-level annotation namespace map; `{}` when none are present. |
| `chunks` | Non-empty execution-order chunk collection. |
Each `chunks` entry contains non-empty `id`, zero-based `index`, `source_ref`,
positive `unit_count`, and an explicit `annotations` map. `source_ref` contains
the same `source_id` as the top-level value plus positive inclusive
`start_unit_id` and `end_unit_id` values. Endpoints identify source units; their
numeric values do not by themselves establish source-document order.
Annotation namespaces are non-empty trimmed strings. Their values are arbitrary
valid JSON and are retained without interpreting a module-specific namespace.
## Ordering And Validation
`chunks` are in execution order. Their indexes are contiguous, start at zero,
and equal their array positions; chunk IDs are unique. The emitted map is built
only after the selected plan has been accepted and materialized against the
source document, so its ranges, unit counts, annotations, and digests describe
that exact plan.
The codec rejects malformed JSON, trailing content, unknown fixed-object
fields, invalid identities or digests, invalid annotations, duplicate chunk
IDs, non-contiguous indexes, and a `plan_digest` that does not match the
reconstructed logical plan. The checked-in
[schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
defines the strict JSON shape.
## Valid Example
The compact
[source chunk-map fixture](../../internal/framework/chunkmap/testdata/source_chunk_map.v1.json)
is decoded by the production codec and demonstrates an accepted map with
annotations, producer identity, and ordered chunks.
## Publication And Compatibility
The map is present only when a chunk plan was accepted and its export is
enabled. It remains publishable if a later lane is rejected, but is absent when
chunk-plan validation rejects the plan. `requested_chunker` identifies the
current pipeline selection, while `producer` identifies the component that
originally produced the accepted plan; they may differ when an accepted plan is
reused.
The map contains structure rather than source content: it excludes transcript
bytes, source-unit metadata, chunk text, private model output, reference
content, debug data, and filesystem paths. Treat the exported map with the
same care as other published output. Publication location and retention are
defined in [Operations](../operations.md#output-bundles).

69
docs/integrations/dnd-combat-turn-artifacts.md
View File

@@ -1,69 +0,0 @@
# D&D Combat-Turn Artifact
This contract defines the durable combat-action occurrence list produced by
`dnd/combat-turns`. It records source-grounded turns and actions; it is not a
complete initiative tracker, combat summary, or state model.
## Identity and compatibility
| Property | Value |
| --- | --- |
| Artifact kind | `dnd/combat-turn-list` |
| Schema ID | `notarius.dnd.combat_turns` |
| Schema name | `notarius_dnd_combat_turns_v1` |
| Schema version | `v1` |
| Media type | `application/json` |
`v1` is a strict JSON object with required `combat_turns`; the array may be
empty. Turn and source-reference objects reject unknown fields. An incompatible
shape change requires a new schema version.
## Wire shape
Each combat turn has these required fields:
| Field | Contract |
| --- | --- |
| `actor` | Non-empty acting character or creature name. |
| `turn_kind` | `turn`, `reaction`, `legendary_action`, `lair_action`, or `other`. |
| `source_refs` | One or more transcript evidence ranges. |
Each source reference has exactly `source_id`, `start_unit_id`, and
`end_unit_id`. It identifies an inclusive current-transcript range; unit IDs
are positive and the start may not follow the end.
```json
{
"combat_turns": [
{
"actor": "Mira Thorn",
"turn_kind": "turn",
"source_refs": [
{"source_id": "session-7", "start_unit_id": 31, "end_unit_id": 32}
]
}
]
}
```
## Eligibility, evidence, and normalized form
The extractor requires an approved [scene-description artifact](dnd-scene-description-artifacts.md).
It emits combat turns only for a chunk with an exact matching scene classified
`combat`; an exact non-combat scene produces an accepted empty list. The scene
record controls eligibility only: its title, summary, and reference do not
become turn evidence. No exact matching scene also produces an empty list and
the `scene_classification_unavailable` warning.
An optional normalized [NPC artifact](dnd-npc-artifacts.md) can ground an
actor name. Its registry references are provenance, never combat evidence.
Normalization trims and, where possible, canonicalizes actor names; orders and
deduplicates exact source references; orders valid-evidence turns by source
chronology; and collapses only duplicates with the same actor identity, turn
kind, and complete valid evidence. It does not infer turns, initiative, or
actions from registry or scene data.
The [NPC-interaction artifact](dnd-npc-interaction-artifacts.md) records
broader NPC occurrences. The [JSON output contract](json-output.md) defines
publication, and [D&D module internals](../internal/dnd.md) describes routing
and validation mechanics.

78
docs/integrations/dnd-item-event-artifacts.md
View File

@@ -1,78 +0,0 @@
# D&D Item-Event Artifact
This contract defines the durable item and currency occurrence list produced by
`dnd/item-events`. It records source-grounded discoveries and possession
changes; it does not maintain an inventory, balance, or ledger.
## Identity and compatibility
| Property | Value |
| --- | --- |
| Artifact kind | `dnd/item-event-list` |
| Schema ID | `notarius.dnd.item_events` |
| Schema name | `notarius_dnd_item_events_v1` |
| Schema version | `v1` |
| Media type | `application/json` |
`v1` is a strict JSON object with required `events`; the array may be empty.
Event and source-reference objects reject unknown fields. An incompatible
shape change requires a new schema version.
## Wire shape
Every event has required `name`, `kind`, and `source_refs`. `quantity`, `from`,
and `to` are optional where the event kind permits them.
| Field | Contract |
| --- | --- |
| `name` | Non-empty item or currency display name. |
| `kind` | `discovered`, `acquired`, `lost`, `consumed`, or `transferred`. |
| `quantity` | Optional positive integer; omit it when no count is established. |
| `from` | Optional non-empty losing holder, when allowed by `kind`. |
| `to` | Optional non-empty gaining holder, when allowed by `kind`. |
| `source_refs` | One or more transcript evidence ranges. |
Each source reference has exactly `source_id`, `start_unit_id`, and
`end_unit_id`. It identifies an inclusive current-transcript range; unit IDs
are positive and the start may not follow the end.
```json
{
"events": [
{
"name": "Silver Pieces",
"kind": "acquired",
"quantity": 20,
"to": "party",
"source_refs": [
{"source_id": "session-7", "start_unit_id": 2, "end_unit_id": 2}
]
}
]
}
```
## Holder rules and minimal extraction
`discovered` has neither holder; `acquired` requires `to` and forbids `from`;
`lost` and `consumed` require `from` and forbid `to`; `transferred` requires
both holders. `party` denotes collective possession. A transfer cannot use
`party` for either holder and its two normalized holders must differ.
Only an evidenced discovery or possession change belongs in this artifact.
It does not infer quantities or holders, convert currency denominations,
calculate balances, or merge nearby events. Campaign references may
disambiguate names but are never event evidence. Currency uses the ordinary
`name` field and an explicit `quantity` only when the transcript establishes
one; each denomination remains a separate event.
Normalization trims display whitespace, orders and removes exact duplicate
source references, then orders events by valid source chronology, name identity
and display value, kind, holders, quantity, and reference sequence. It
collapses only entries with the same normalized durable fields and complete
valid evidence.
The [JSON output contract](json-output.md) defines publication. See
[D&D module internals](../internal/dnd.md) for implementation details and the
[NPC-interaction artifact](dnd-npc-interaction-artifacts.md) for a distinct
kind of occurrence.

69
docs/integrations/dnd-npc-artifacts.md
View File

@@ -1,69 +0,0 @@
# D&D NPC Artifact
This contract defines the durable NPC registry produced by `dnd/npcs`. It is a
minimal, source-grounded identity registry for other D&D artifacts, not a
character sheet or a relationship summary.
## Identity and compatibility
| Property | Value |
| --- | --- |
| Artifact kind | `dnd/npc-list` |
| Schema ID | `notarius.dnd.npcs` |
| Schema name | `notarius_dnd_npcs_v1` |
| Schema version | `v1` |
| Media type | `application/json` |
| Identity policy | `dnd.npcs.identity.v1` |
`v1` accepts one strict JSON object with required `npcs`; the array may be
empty. NPC and source-reference objects reject unknown fields. An incompatible
artifact shape or identity-policy change uses a new version or policy.
## Wire shape and identity
Each NPC has these required fields:
| Field | Contract |
| --- | --- |
| `id` | `npc:sha256:` followed by 64 lowercase hexadecimal characters. |
| `name` | Non-empty canonical display name. |
| `source_refs` | One or more transcript evidence ranges for the identity. |
A source reference has exactly `source_id`, `start_unit_id`, and `end_unit_id`.
The source ID identifies the transcript, unit IDs are positive inclusive unit
identifiers, and the start may not follow the end.
```json
{
"npcs": [
{
"id": "npc:sha256:99a16589618a04f535a7d21fdcc71a0b1c05d22f752cd492065b1086d97bc3d7",
"name": "Mira Thorn",
"source_refs": [
{"source_id": "session-7", "start_unit_id": 4, "end_unit_id": 5}
]
}
]
}
```
The ID is deterministic: normalize the name to Unicode NFKC, normalize the
supported apostrophe forms, collapse whitespace, case-fold it, SHA-256 the
result, then prefix the lowercase hexadecimal digest with `npc:sha256:`. Each
canonical identity and ID appears at most once. Normalization collapses records
with the same canonical identity, retains their earliest position, and merges
their canonicalized evidence; it does not add aliases, roles, descriptions, or
relationship fields.
## Scope and consumers
Only individually identifiable NPC names with transcript evidence belong in
this artifact. Groups, generic roles, invented labels, and descriptive
enrichment are excluded. Its source references prove registry provenance; they
do not become evidence for a spell, interaction, or combat occurrence.
This registry can ground actor or caster names in the [spell](dnd-spell-artifacts.md)
and [combat-turn](dnd-combat-turn-artifacts.md) artifacts. It is required to
resolve the canonical `name` in an [NPC interaction](dnd-npc-interaction-artifacts.md).
The [JSON output contract](json-output.md) defines publication, and
[D&D module internals](../internal/dnd.md) owns pipeline mechanics.

78
docs/integrations/dnd-npc-interaction-artifacts.md
View File

@@ -1,78 +0,0 @@
# D&D NPC Interaction Artifact
This contract defines the durable occurrence list produced by
`dnd/npc-interactions`. It records discrete, source-grounded interactions with
NPCs already present in a normalized registry; it does not extend that registry
or summarize the session.
## Identity and compatibility
| Property | Value |
| --- | --- |
| Artifact kind | `dnd/npc-interaction-list` |
| Schema ID | `notarius.dnd.npc_interactions` |
| Schema name | `notarius_dnd_npc_interactions_v1` |
| Schema version | `v1` |
| Media type | `application/json` |
`v1` is a strict JSON object with required `interactions`; the array may be
empty. Interaction and source-reference objects reject unknown fields. An
incompatible shape change requires a new schema version.
## Wire shape
Each interaction has these required fields:
| Field | Contract |
| --- | --- |
| `name` | Non-empty canonical display name from the required NPC registry. |
| `kind` | One of the interaction categories below. |
| `source_refs` | One or more transcript evidence ranges. |
Each source reference has exactly `source_id`, `start_unit_id`, and
`end_unit_id`. It identifies an inclusive range in the current transcript;
unit IDs are positive and the start may not follow the end. Extraction evidence
for an interaction is confined to its accepted chunk.
```json
{
"interactions": [
{
"name": "Mira Thorn",
"kind": "dialogue",
"source_refs": [
{"source_id": "session-7", "start_unit_id": 12, "end_unit_id": 13}
]
}
]
}
```
## Interaction categories
| Kind | Meaning |
| --- | --- |
| `mentioned` | The NPC is referred to but is not established as present or communicating. |
| `noncombat_presence` | The NPC is present and relevant without meaningful dialogue or combat participation. |
| `dialogue` | The NPC speaks, responds, or meaningfully participates in a non-combat exchange. |
| `combat_ally` | The NPC actively participates in combat on the party's side. |
| `combat_opponent` | The NPC actively participates in combat against the party. |
| `other` | A clearly evidenced direct occurrence not covered by another category. |
The categories do not represent motives, relationships, state, or events that
the cited transcript does not establish. An `other` entry is not a substitute
for uncertain classification.
## Identity, evidence, and order
The required normalized [NPC artifact](dnd-npc-artifacts.md) resolves `name`.
Registry references are provenance only and never replace an interaction's own
evidence. Normalization canonicalizes recognized registry names, orders and
deduplicates exact source references, then orders interactions by valid source
chronology, NPC comparison identity, display name, kind, and reference sequence.
Only entries with the same canonical name, kind, and complete valid evidence
sequence are collapsed; distinct categories or evidence remain separate.
See the [combat-turn artifact](dnd-combat-turn-artifacts.md) for combat-action
occurrences and the [JSON output contract](json-output.md) for publication.
Pipeline mechanics are described in [D&D module internals](../internal/dnd.md).

69
docs/integrations/dnd-scene-description-artifacts.md
View File

@@ -1,69 +0,0 @@
# D&D Scene-Description Artifact
This contract defines the durable output of `dnd/scene-descriptions`. Each
record classifies one accepted transcript chunk and gives it a minimal
source-grounded title and summary.
## Identity and compatibility
| Property | Value |
| --- | --- |
| Artifact kind | `dnd/scene-description-list` |
| Schema ID | `notarius.dnd.scene_descriptions` |
| Schema name | `notarius_dnd_scene_descriptions_v1` |
| Schema version | `v1` |
| Media type | `application/json` |
`v1` is a strict JSON object with required non-empty `scenes`. Scene and
source-reference objects reject unknown fields. An incompatible shape change
requires a new schema version.
## Wire shape
Each scene has exactly these required fields:
| Field | Contract |
| --- | --- |
| `id` | Non-empty accepted chunk ID, assigned by Notarius. |
| `source_ref` | The assigned inclusive source range for that chunk. |
| `kind` | `combat`, `narrative`, `recap`, or `meta`. |
| `title` | Non-empty, trimmed, source-grounded title. |
| `summary` | Non-empty, trimmed, source-grounded summary. |
`source_ref` has exactly `source_id`, `start_unit_id`, and `end_unit_id`.
Its source ID identifies the input transcript; its positive unit IDs identify
the chunk's inclusive range, with the start no later than the end.
```json
{
"scenes": [
{
"id": "chunk-000001",
"source_ref": {"source_id": "session-7", "start_unit_id": 1, "end_unit_id": 3},
"kind": "narrative",
"title": "Arrival at the watchtower",
"summary": "The party reaches the ruined watchtower and begins to investigate it."
}
]
}
```
## Meaning and normalized form
`combat` identifies a chunk where active combat is the central activity.
`narrative` is current in-world play that is not principally combat, recap, or
meta discussion. `recap` is primarily a recounting of an earlier session, and
`meta` is primarily out-of-character discussion. The artifact does not add
participants, confidence, events, or information absent from the chunk.
Normalization trims title and summary, orders scenes by source position and
then ID, and removes exact duplicate records. A reused ID with different
durable fields, or the same source range with different kind, title, or
summary, is invalid. It does not merge adjacent ranges, alter prose, or infer
missing scenes.
The [combat-turn artifact](dnd-combat-turn-artifacts.md) uses an exact matching
`combat` scene only as eligibility control; scene title, summary, and source
reference never become combat evidence. Publication is defined by the
[JSON output contract](json-output.md); implementation details live in
[D&D module internals](../internal/dnd.md).

73
docs/integrations/dnd-spell-artifacts.md
View File

@@ -1,73 +0,0 @@
# D&D Spell Artifact
This contract defines the durable output of the `dnd/spells` extractor and
normalizer. It records source-grounded spell-casting occurrences; it is not a
spellbook, a rules lookup result, or a record of hypothetical casts.
## Identity and compatibility
| Property | Value |
| --- | --- |
| Artifact kind | `dnd/spell-list` |
| Schema ID | `notarius.dnd.spells` |
| Schema name | `notarius_dnd_spells_v1` |
| Schema version | `v1` |
| Media type | `application/json` |
`v1` is a single strict JSON object. It requires `spell_casts`; the array may
be empty. Each spell-cast object and source-reference object rejects unknown
fields. An incompatible shape change requires a new schema version.
## Wire shape
Each `spell_casts` entry has these required fields:
| Field | Contract |
| --- | --- |
| `caster` | Non-empty in-world character or creature name. |
| `spell` | Non-empty spell name. |
| `source_refs` | One or more transcript evidence ranges. |
Every source reference has exactly `source_id`, `start_unit_id`, and
`end_unit_id`. The source ID identifies the input transcript; the unit IDs are
positive inclusive unit identifiers, and the start may not follow the end in
that source. References are evidence for the cast, not campaign-reference or
NPC-registry provenance.
```json
{
"spell_casts": [
{
"caster": "Mira Thorn",
"spell": "Fireball",
"source_refs": [
{"source_id": "session-7", "start_unit_id": 12, "end_unit_id": 13}
]
}
]
}
```
## Evidence and normalized form
An entry represents an actual cast or an unambiguous declared attempt. A spell
mention, rules discussion, plan, or catalog match alone is not an occurrence.
The configured catalog checks the name; it does not establish evidence.
When normalization is selected, recognized spell names use the effective
catalog's canonical display name. Source references are put in canonical source
order and exact duplicate references are removed. A later entry is collapsed
only when it has the same canonical spell, the same case- and
whitespace-insensitive caster identity, and the same complete valid reference
sequence. Remaining entries retain their merged order.
The optional normalized [NPC artifact](dnd-npc-artifacts.md) can ground a
caster name. Its own references remain registry provenance and are never copied
into `source_refs`.
## Related contracts
The [spell-catalog overlay contract](dnd-spell-catalog-overlays.md) defines
the configured catalog additions. The [JSON output contract](json-output.md)
defines where this logical artifact is published; [D&D module internals](../internal/dnd.md)
describes extraction and validation mechanics.

72
docs/integrations/dnd-spell-catalog-overlays.md
View File

@@ -1,72 +0,0 @@
# D&D Spell-Catalog Overlays
This document defines the optional JSON overlay consumed by the D&D spell
extractor. An overlay contributes campaign spell names and aliases for
recognition. It does not define spell rules, effects, levels, classes, or
transcript evidence. Bind the optional `spell_catalog` reference as described
in [Configuration](../config.md#references-and-ordered-handoffs).
## Contract Identity
| Property | Value |
| --- | --- |
| Consumer | D&D spell extraction and normalization |
| Reference slot | `spell_catalog` |
| Media type | `application/json` |
| Required schema version | `notarius.dnd.spell-catalog-overlay.v1` |
| Base catalog | Embedded D&D 5e 2014 SRD catalog |
At most one overlay document may be bound. The maintained example is
[dnd-spell-catalog.json](../../examples/dnd-spell-catalog.json).
## Wire Shape
This is a minimal valid overlay:
```json
{
"schema_version": "notarius.dnd.spell-catalog-overlay.v1",
"catalogs": [
{
"id": "campaign.example",
"ruleset": "dnd-5e-2014",
"source": {"title": "Example campaign spells"},
"spells": [{"name": "Aegis of Emberfall"}]
}
]
}
```
| Field | Required | Meaning and constraints |
| --- | --- | --- |
| `schema_version` | Yes | Exactly `notarius.dnd.spell-catalog-overlay.v1`. |
| `catalogs` | Yes | Non-empty array of catalog objects with unique IDs. |
| `catalogs[].id` | Yes | Non-empty trimmed string. |
| `catalogs[].ruleset` | Yes | Exactly `dnd-5e-2014`. |
| `catalogs[].source.title` | Yes | Non-empty trimmed string. |
| `catalogs[].source.version` | No | String when present. |
| `catalogs[].source.url` | No | String when present. |
| `catalogs[].source.license` | No | String when present. |
| `catalogs[].spells` | Yes | Non-empty array of spell objects. |
| `catalogs[].spells[].name` | Yes | Non-empty trimmed string. |
| `catalogs[].spells[].aliases` | No | Array of non-empty trimmed strings when present. |
Unknown fields are rejected at every object level. The document must contain
one JSON value; `null` is not accepted for optional strings or aliases.
## Composition And Compatibility
Notarius starts with the embedded base catalog, then applies overlay catalogs
in ascending catalog-ID order. A new canonical spell name adds a recognition
entry. If an overlay names an existing canonical spell, it augments that spell
with aliases while retaining the established display spelling.
Repeated aliases for the same spell are accepted. A canonical-name, canonical-
to-alias, or alias-to-alias collision between different spells is rejected,
including a collision with the embedded catalog. Matching uses the catalog’s
case, whitespace, and apostrophe normalization, so authors should avoid names
or aliases that normalize to another spell.
The overlay is a recognition aid only. The durable spell-artifact schema and
source-evidence rules are defined by the
[D&D spell artifact contract](dnd-spell-artifacts.md).

116
docs/integrations/evidence-context.md
View File

@@ -1,116 +0,0 @@
# Published Evidence Context
This contract defines the optional `source/evidence-context` artifact emitted
by the production JSON output. Its configuration is owned by
[Configuration](../config.md#module-bindings-and-validators); its logical-file
discovery is owned by [Published JSON Output](json-output.md).
## Identity And Discovery
When enabled, the JSON bundle contains `evidence-context.json` and an
`index.json` `evidence_context` descriptor with the same six fields as other
pipeline-wide artifact descriptors.
| Property | Value |
| --- | --- |
| Artifact kind | `source/evidence-context` |
| Media type | `application/json` |
| Schema ID | `notarius.source.evidence_context` |
| Schema name | `notarius_source_evidence_context_v1` |
| Schema version | `v1` |
| Logical file | `evidence-context.json` |
Consumers must discover the file from the descriptor, verify all six descriptor
fields, and decode only a supported schema version. The descriptor is optional:
its absence means evidence publication was not enabled for that bundle.
## Payload
The v1 payload is a JSON object with required `source_id`, `source_digest`,
`window_units`, `selected_lanes`, and `contexts` fields. `selected_lanes` and
`contexts` are always arrays; an enabled configuration with no accepted direct
evidence publishes `contexts: []`.
```json
{
"source_id": "session-alpha",
"source_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"window_units": 1,
"selected_lanes": ["npcs", "spells"],
"contexts": [
{
"context_ref": {
"source_id": "session-alpha",
"start_unit_id": 10,
"end_unit_id": 20
},
"evidence_refs": [
{
"lane_id": "spells",
"source_ref": {
"source_id": "session-alpha",
"start_unit_id": 10,
"end_unit_id": 10
}
}
],
"units": [
{
"id": 10,
"kind": "transcript_segment",
"text": "Aria casts Cure Wounds.",
"ref": {
"source_id": "session-alpha",
"start_unit_id": 10,
"end_unit_id": 10
}
},
{
"id": 20,
"kind": "transcript_segment",
"text": "The party regroups.",
"ref": {
"source_id": "session-alpha",
"start_unit_id": 20,
"end_unit_id": 20
}
}
]
}
]
}
```
Each context requires a `context_ref` object and `evidence_refs` and `units`
arrays. `context_ref` identifies the first and last included unit. Each
evidence entry contains a selected `lane_id` and an original `source_ref`. A
unit uses the existing source-unit shape: required `id`, `kind`, `text`, and
self `ref`, plus optional JSON-object `metadata`. Fixed payload objects reject
unknown fields; unit metadata may contain application-defined JSON values.
## Citations And Context
`evidence_refs` are the authoritative citations. They identify the direct
references emitted by accepted normalized artifacts. `context_ref` and the
units collection include those cited units plus nearby source units selected by
the configured window. They are explanatory context, not widened citations.
Only accepted outputs from the configured lane allowlist contribute. Rejected,
failed, absent, and lane-filtered outputs do not contribute. The artifact never
contains raw input bytes, prompts, model responses, auxiliary reference
content, credentials, or filesystem paths.
## Ordering And Compatibility
The selected lane allowlist is lexical. Contexts and units are in source
document position order, not numeric unit-ID order. Direct evidence entries
are deterministically ordered by lane and source reference. Overlapping or
contiguous windows merge, and each source unit appears at most once in the
resulting contexts.
The artifact is additive to the JSON bundle and is not a lane payload,
normalized-output count, checkpoint, or generated reference. Consumers that
do not need it must tolerate the absent optional descriptor. Consumers that do
use it should preserve the artifact and its schema identity with the run
provenance, and should treat its source text and metadata as sensitive durable
content.

133
docs/integrations/json-output.md
View File

@@ -1,133 +0,0 @@
# Published JSON Output
This document defines the logical JSON bundle emitted by the production JSON
output encoder. The bundle’s physical destination, atomic publication, and
retention are operational concerns; see [Operations](../operations.md#output-bundles).
Output configuration, including chunk-map and evidence-context publication, belongs in
[Configuration](../config.md#module-bindings-and-validators).
## Bundle Layout
All paths below are logical, relative, slash-separated bundle paths. The
encoder always emits the first four JSON files below and adds lane or
pipeline-wide artifact files when their corresponding artifacts are available:
A subprocess caller first obtains the physical bundle root from the
[run-result receipt](run-result.md), then resolves `index.json` beneath that
root for the logical discovery described here.
| Path | Purpose |
| --- | --- |
| `index.json` | Entry point that names the other published files and lane payloads. |
| `manifest.json` | Run provenance and result summaries. |
| `rejected.json` | Rejected pipeline outputs. |
| `warnings.json` | Accepted-output and run warnings. |
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
| `evidence-context.json` | Optional source-context artifact, when evidence publication is enabled. |
JSON files are pretty-printed with a trailing newline. Lane payloads are
accepted only when their media type is `application/json`.
## `index.json`
`index.json` is the bundle’s discovery document. An approved run with no
normalized lanes has this valid minimal index:
```json
{
"manifest_file": "manifest.json",
"output_files": [],
"rejected_file": "rejected.json",
"warnings_file": "warnings.json"
}
```
| Field | Required | Meaning |
| --- | --- | --- |
| `manifest_file` | Yes | Always `manifest.json`. |
| `output_files` | Yes | Lane descriptors sorted by `lane_id`. |
| `rejected_file` | Yes | Always `rejected.json`. |
| `warnings_file` | Yes | Always `warnings.json`. |
| `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. |
| `evidence_context` | No | Descriptor for the pipeline-wide `evidence-context.json`; never a lane descriptor. |
Each lane descriptor has required `lane_id` and `file`. It may also include
`media_type`, `module_key`, `schema_id`, `schema_name`, and `schema_version`
when supplied by the normalized artifact. Each pipeline-wide artifact
descriptor (`chunk_map` or `evidence_context`) contains `artifact_kind`,
`file`, `media_type`, `schema_id`, `schema_name`, and `schema_version`. Their
payloads are defined by the [Accepted Chunk Map contract](chunk-map.md) and
[Published Evidence Context](evidence-context.md), respectively.
The lane path is derived from its lane ID. Characters outside letters, digits,
periods, underscores, and hyphens become underscores; `..` sequences are
neutralized; leading and trailing periods and underscores are removed. A lane
that produces an empty name, or two lanes that produce the same path, makes
output encoding fail.
## Lane Payloads
Each `lanes/<safe-lane-id>.json` file is the codec-owned normalized JSON for
that lane. Consumers should use the index descriptor’s schema identity rather
than infer a lane schema from its name. The current D&D payload contracts are
[spells](dnd-spell-artifacts.md), [NPCs](dnd-npc-artifacts.md),
[NPC interactions](dnd-npc-interaction-artifacts.md),
[combat turns](dnd-combat-turn-artifacts.md),
[item events](dnd-item-event-artifacts.md), and
[scene descriptions](dnd-scene-description-artifacts.md).
## `manifest.json`
`manifest.json` is published provenance, not a copy of lane payloads or a
checkpoint store. Fields without a value may be omitted. Its top-level fields
group into the following externally observable summaries:
| Group | Fields |
| --- | --- |
| Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
| Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
| Source and references | `source_digests`, `references` |
| Published result summaries | `normalized_outputs`, `rejected_outputs` |
| Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
`references` records provenance such as the target, slot, origin, digest,
media type, size, and generated-artifact identity. It does not contain
reference content. `normalized_outputs` and `rejected_outputs` likewise
summarize results without embedding lane payload bytes. A chunk-plan summary is
provenance for the plan used by this run; cache records, debug artifacts, and
other operational state are not published as bundle files.
Each `llm_profiles` entry identifies effective, non-secret LLM execution
provenance:
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | Yes | Selected PromptKit profile identifier. |
| `provider` | No | Notarius adapter provider identifier. |
| `model` | No | Effective provider model identifier. |
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
These values describe observed execution; they are not a backend-registration
interface. Entries that differ by backend or effective reasoning remain
distinct even when their profile, provider, and model are otherwise equal.
## Rejections And Warnings
`rejected.json` is always an object with a `rejected` array. Each entry has
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
`chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
`diagnostic_artifact_path` are present only when applicable.
`warnings.json` is always an object with a `warnings` array. Each warning has
`reason_code` and `message`; `scope` is optional. Both arrays are empty when
there is nothing to report.
## Compatibility
The index is the authoritative map from a logical lane to its published
payload. Consumers must tolerate omitted optional manifest and descriptor
fields, and should rely on the linked artifact contract for each lane’s JSON
shape. This contract describes the published logical bundle only; it does not
promise a filesystem layout or expose internal state formats.

106
docs/integrations/pkg-promptkit.md
View File

@@ -1,106 +0,0 @@
# PromptKit Integration
Notarius pins
[`gitea.maximumdirect.net/eric/promptkit` v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0)
as its in-process prompt engine. The upstream
[Go package consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/consumers/pkg-promptkit.md)
owns the public engine API, and the upstream
[format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
owns prompt, profile, and schema file contracts.
## Supported Boundary
Notarius relies on the root `promptkit` package to:
- construct an `Engine` with filesystem-backed prompt, schema, and optional
operator and application-fallback profile sources;
- prepare one frozen execution from a `RunRequest` with named inline artifacts,
variables, a direct session ID, prompt identity, and profile selection, then
record credential-redacted details and run that exact execution;
- return rendered debug material, validated structured output, selected
profile, backend, effective model metadata, and token usage;
- register the optional conventional `local` backend through `BackendLocal`,
`LocalBackend`, and `WithBackend`;
- distinguish structured-output validation failure from execution failure; and
- identify a missing explicit profile through `ErrProfileNotFound` and backend
admission exhaustion through `ErrCapacityExceeded`.
The pinned
[`BackendLocal`, `LocalBackend`, and `WithBackend` API](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/backends.go)
owns the registration and backend-capacity contract.
For one completion, the adapter calls `PrepareExecution`, takes a
caller-owned `Details` snapshot, and calls `RunPrepared` for that same opaque
prepared execution. It defers `Discard` for every unexecuted handle. Explicit
profile preflight uses `Engine.InspectProfile`; it does not prepare a synthetic
prompt. PromptKit's prepared handle, inspection result, and capacity-error
types stay inside the Notarius LLM adapter.
When a PromptKit profile and runtime override leave `temperature`, `max_tokens`,
or `top_p` unset, Notarius leaves that control unset as well. Compatible
providers therefore apply their own defaults; an operator that requires a
specific sampling value must select it explicitly in the profile or runtime
override.
Notarius does not use PromptKit's optional `ArtifactReader`. It materializes
source and reference content itself and supplies owned inline artifacts at the
adapter boundary. It also retains responsibility for pipeline retries,
scheduling, debug persistence, redaction, profile provenance, and conversion
from private model responses into durable domain artifacts.
Notarius sends its trimmed run session through PromptKit's direct session
field, which is authoritative for provider session behavior. It also retains
the same value as the `session_id` prompt variable for maintained prompt
compatibility. Session IDs are stable, non-secret correlation identifiers and
may be exposed to providers and provider observability.
Notarius records PromptKit's selected backend ID and effective reasoning
setting as optional run-manifest provenance. Endpoint-only profiles have no
backend ID. Debug prompt material also retains the selected backend ID and
PromptKit's stable lower-case `effective_model_params` JSON, which may include
`backend_id`. Notarius production configuration exposes one optional
conventional `local` registration. It does not expose a general user-defined
PromptKit backend registry. Endpoint-only profiles remain supported unchanged.
Notarius retains its application-wide scheduled client around the PromptKit
adapter. PromptKit may apply a narrower limit for the selected backend;
endpoint-only profiles have no such backend limit. The adapter translates
PromptKit capacity rejection into the provider-neutral Notarius
`ErrLLMCapacityExceeded` contract. It may include the normalized selected
backend ID in safe diagnostic context, without exposing PromptKit's capacity
error type, and leaves retries to the calling pipeline stage.
## Profile Sources And Compatibility
Notarius gives PromptKit the configured operator profile source, registered
application fallback profile assets, and optional backend registration through
the same construction path for inspection and execution. PromptKit owns the
resulting source precedence and strict profile parsing: a matching operator
profile is a complete replacement for a fallback or built-in profile, while an
invalid matching document fails instead of falling through. The operator
configuration and deployment workflow are defined in
[Configuration](../config.md#promptkit-profiles) and
[Operations](../operations.md#promptkit-profile-deployment).
Notarius supports this boundary against PromptKit v0.5.0. Its fallback source,
prepared-execution, inspection, and typed capacity APIs are used as public
upstream contracts; other PromptKit APIs or file-format behavior are not
implicitly supported. A dependency upgrade requires reviewing the adapter,
profile-source construction, and this compatibility statement against the
pinned upstream documentation.
## Notarius Ownership
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
module assets, maps its transport-neutral completion contract, prepares and
executes requests, validates output, records provenance, captures debug
material, redacts errors, and preserves timeout ownership.
[D&D Module Internals](../internal/dnd.md) owns the embedded
`dnd-extraction` fallback profile and the maintained D&D prompt defaults.
[Configuration](../config.md#promptkit-profiles) defines how a Notarius
configuration selects one PromptKit profile source and optionally registers
the conventional local backend.
PromptKit API or format changes outside this boundary are not implicitly
supported. Updating the pinned version requires reviewing the adapter and
profile/configuration contracts against the upstream documentation.

68
docs/integrations/run-result.md
View File

@@ -1,68 +0,0 @@
# Run Result Receipt
`notarius run --json` writes this receipt to standard output when a run
completes successfully. It lets a subprocess caller discover the physical root
of the published output bundle without parsing interactive command output.
Command syntax, streams, and exit statuses are defined in the
[CLI reference](../cli.md); logical files within the bundle are defined in the
[Published JSON Output contract](json-output.md).
## Schema
The current schema version is `notarius.run-result.v1`.
| Field | Required | Meaning |
| --- | --- | --- |
| `schema_version` | Yes | Exactly `notarius.run-result.v1`. |
| `run_id` | Yes | The finalized Notarius run identifier. |
| `pipeline_id` | Yes | The effective pipeline identifier. |
| `output_directory` | Yes | Absolute path to the published, run-specific output bundle. |
| `index_file` | For the production JSON output | Logical path `index.json`; omitted for other output modules. |
| `normalized_output_count` | Yes | Number of final normalized outputs. |
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
| `warning_count` | Yes | Number of final run warnings. |
| `validation_status` | Yes | The final run manifest validation status. |
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
For the production `json` output module, `index_file` is present only when the
completed run returned exactly one logical output file named `index.json`.
For another output module, its absence does not indicate a failed run.
```json
{
"schema_version": "notarius.run-result.v1",
"run_id": "run-1770000000000000000-0123456789abcdef0123456789abcdef",
"pipeline_id": "dnd-session",
"output_directory": "/work/results/run-1770000000000000000-0123456789abcdef0123456789abcdef",
"index_file": "index.json",
"normalized_output_count": 6,
"rejected_output_count": 2,
"warning_count": 1,
"validation_status": "rejected"
}
```
## Paths And Bundle Discovery
`output_directory` and `debug_directory`, when present, are lexical absolute
paths. They identify the paths used by Notarius and do not resolve symlinks.
`output_directory` is the run-specific bundle, not the configured output root.
The receipt is a summary and discovery document. It does not contain lane
descriptors, payloads, manifest data, rejections, warnings, or file contents.
For the production JSON output, resolve `index_file` beneath
`output_directory`, reject path escapes, and use the
[Published JSON Output contract](json-output.md) to discover logical files and
lane payloads.
## Delivery And Compatibility
Notarius writes the receipt only after the output bundle has been published and
any requested debug terminal reporting has completed. Standard output is not
transactional: a result-write failure returns a nonzero status and can leave
partial bytes. Consumers must ignore standard output unless the process exits
with status 0.
Future versions may add optional fields to this schema. Consumers must tolerate
unknown fields. An incompatible field or semantic change requires a new
`schema_version` value.

73
docs/integrations/seriatim.md
View File

@@ -1,73 +0,0 @@
# Seriatim Transcript Input
This document defines the JSON transcript accepted by the production Seriatim
input adapter. It is a source input, not a durable lane artifact. Configure the
input adapter through [Configuration](../config.md#production-module-keys).
## Contract Identity
| Property | Value |
| --- | --- |
| Consumer | Seriatim input adapter |
| Media type | `application/vnd.seriatim+json` |
| Source document kind | `transcript` |
| Source-unit kind | `transcript_segment` |
## Accepted Shape
The input is one JSON object containing `metadata` and a non-empty `segments`
array. This minimal document is valid:
```json
{
"metadata": {"id": "session-alpha"},
"segments": [
{
"id": 1,
"start": 0,
"end": 4,
"speaker": "Aria",
"text": "Aria casts Cure Wounds."
}
]
}
```
The maintained two-segment input is
[seriatim-minimal-transcript.json](../../examples/seriatim-minimal-transcript.json).
| Field | Required | Meaning and constraints |
| --- | --- | --- |
| `metadata` | Yes | JSON object. Its entries become source metadata; no particular metadata key is otherwise required. |
| `segments` | Yes | Non-empty array of segment objects, kept in input order. |
| `segments[].id` | Yes | Positive canonical decimal integer, supplied as a JSON number or string. IDs must be unique. |
| `segments[].start` | Yes | Finite, non-negative numeric value, supplied as a JSON number or string. |
| `segments[].end` | Yes | Finite, non-negative numeric value that is not earlier than `start`. |
| `segments[].speaker` | Yes | String that is non-empty after trimming. |
| `segments[].text` | Yes | String that is non-empty after trimming. Its original text is retained. |
Additional top-level and segment fields are ignored. A missing required field,
`null` in place of an object or array, malformed JSON, or more than one
top-level JSON value is rejected.
## Source Identity And References
The adapter chooses the source ID in this order:
1. a non-empty source ID supplied by the calling request;
2. non-empty string `metadata.id`;
3. non-empty string `metadata.source_id`;
4. `seriatim:` followed by the first 16 hexadecimal characters of the raw
input’s SHA-256 digest.
Each accepted segment becomes one source unit whose unit ID is `segments[].id`.
Its self-reference uses the derived source ID and the same segment ID for both
range endpoints. Artifact contracts use those segment IDs when they cite
transcript evidence.
## Compatibility
This adapter accepts only the shape described here. A broader Seriatim export
is usable only when it supplies this object, metadata, and segment shape with
the stated types and constraints. Unknown additional fields do not add
Notarius behavior.

163
docs/internal/cli.md
View File

@@ -1,163 +0,0 @@
# CLI Internals
This document describes **internal/cli**, Notarius's production composition
root. The [CLI reference](../cli.md) owns command syntax and exit statuses;
[Configuration](../config.md) owns configuration values; and
[Operations](../operations.md) owns filesystem layout, recovery, and operator
procedures.
## Inputs, Outputs, And Boundaries
The CLI accepts process arguments, standard streams, and injectable options
used by tests and embedding code. It writes command results to the supplied
streams and returns a process exit status. For a run, it also creates the
production catalog and runtime collaborators, hands a prepared pipeline and
source bytes to the framework, and places the logical files returned by the
runner.
It is the only boundary allowed to compose concrete registries, LLM clients,
cache/checkpoint collaborators, debug recorders, and physical output paths.
Pipeline modules receive interfaces and request data rather than CLI streams or
filesystem roots. The [Architecture](../policy/architecture.md) defines this
composition-root boundary; [Pipeline Internals](pipeline.md) owns resolution,
preparation, and runner mechanics after their inputs are supplied.
## Dispatch And Configuration Handoff
The root dispatcher handles help, configuration validation, pipeline listing,
and a pipeline run. It normalizes injectable options before dispatch so that a
missing production dependency fails as a command error rather than reaching
execution.
Commands that need configuration use one shared loader. The CLI discovers the
file, parses it through **internal/core/config**, starts from defaults, applies
the file and supported environment overrides, and then validates it for the
command. The configured discovery and precedence contract is in
[Configuration](../config.md), while the loading and resolution mechanics are
in [Configuration Internals](configuration.md).
Configuration validation without a selected pipeline checks structural
configuration only. Validation with a selected pipeline also builds the
effective catalog, resolves the pipeline, and verifies every explicit effective
PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
profiles are inspected
against the configured PromptKit source and backend registrations without
loading a prompt or performing generation, so an unknown or invalid profile
fails before pipeline preparation. Credential availability remains an
execution-time concern. Pipeline listing validates configuration before
returning normalized, sorted identifiers.
## Production Composition
The production composition helper allocates every framework registry and the
prompt-asset registry, then registers the generic, Seriatim, and D&D module
families in that order. The resulting registries provide both the module
catalog used for resolution and the concrete constructors used for preparation.
Tests may provide a catalog or registries instead; production code must not
silently merge an injected partial catalog with production registrations.
The production LLM factory builds one PromptKit-backed client from the resolved
**promptkit.profile_dir** or **promptkit.profile_file** source, attaches the
profile-provenance recorder, creates one scheduler from the effective global
LLM limit, and wraps the client before it reaches modules. Registration and LLM
construction errors are returned before a pipeline is prepared. Configuration
field definitions remain in [Configuration](../config.md#promptkit-profiles);
the D&D registrar's fallback profile assets and the adapter mechanics remain in
[LLM Runtime](llm.md).
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
preserves inherit, replace, and clear states across the composition boundary.
Run orchestration constructs this value from the mutually exclusive
`--reasoning-effort` and `--clear-reasoning-effort` controls. Absence preserves
a nil pointer, replacement is trimmed, and clear uses a non-nil empty string.
The same override reaches the one shared production client, checkpoint
identity, and debug invocation metadata. Persistent reasoning configuration
remains owned by PromptKit profiles; Notarius configuration has no reasoning
field.
## Run Orchestration
After parsing and validating a run invocation, the CLI performs this ordered
handoff:
1. load and validate configuration, then apply command-level operational
overrides;
2. create and validate a safe run identity, then allocate a debug bundle only
when requested;
3. build the effective catalog, resolve requested reference changes, resolve
the effective pipeline, and inspect its explicit effective PromptKit
profiles;
4. materialize external or generated references and record redacted invocation
and resolution provenance when debug capture is enabled;
5. construct registries, the scheduled LLM client, prepared modules, and the
requested cache/checkpoint collaborators;
6. read the source input and invoke the framework runner; and
7. write the runner's logical output files only after a successful run, then
complete the command report and user-facing result.
Preparation happens before source parsing, so module construction and
dependency failures cannot begin stage execution. The CLI also preserves the
framework's result and warning information when it writes summaries and the
final command result. Detailed state lifecycle, resume handling, and physical
path confinement are maintained in [Run State Internals](state.md) and
[Operations](../operations.md).
For `run --json`, the CLI constructs and encodes its private run-result receipt
after a successful runner result is available, before it publishes logical
output files. It writes the prepared receipt to standard output only after
output publication and requested debug terminalization succeed. A receipt-write
failure exits with runtime status 1 and may leave partial standard-output bytes,
but the already-published output bundle remains complete and requested debug
reporting remains successfully terminalized. The CLI reports a bounded
command-owned error and does not repeat terminal reporting. The receipt remains
a CLI reporting concern rather than a framework or output-module responsibility;
its public contract is the
[run-result receipt](../integrations/run-result.md).
## Failure Mapping And Terminal Reporting
Argument, flag, and invocation-combination failures are reported to standard
error before runtime composition and use the syntax error class. Once an
invocation is syntactically valid, configuration loading and validation,
resolution, registration, profile checks, reference materialization, module
construction, input reads, runner failures, output publication, and requested
debug handling use the runtime failure class. The public status numbers and
stream contract are defined in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
When debug capture has been allocated, one command-state value records the
known run result. Guarded terminalization writes a success report once, or
attempts a failure report and error record once. A persistence failure is
reported in addition to the original failure and never replaces it. If a debug
path exists, failure output includes that path so the retained diagnostic data
is discoverable.
## Invariants To Preserve
- Only the CLI composes production implementations and physical runtime roots.
- Configuration and resolved composition failures occur before module
preparation or source parsing.
- A runner's logical files are published only after a successful run.
- Production registries and a caller-supplied catalog or registries are
alternative composition sources, not an implicit mixture.
- A requested debug bundle has one terminal report attempt; its persistence
errors supplement rather than obscure the primary command error.
- User-facing flags, paths, exit codes, and configuration fields are defined
by their public documentation, not duplicated here.
## Focused Tests
- **internal/cli/command_contract_test.go** covers dispatch, help, syntax and
runtime error classes, discovery, validation, and listing.
- **internal/cli/run_contract_test.go** covers the run handoff, publication,
debug reporting, and command-owned state collaborators.
- **internal/cli/production_contract_test.go** covers registrar composition,
production catalog contents, assets, and representative configuration
validation.
- **internal/cli/reference_contract_test.go** covers CLI reference overrides,
origin separation, and materialization boundaries.
- **internal/cli/state_hardening_test.go** covers safe run identity, state
roots, and failure ordering.
Run **go test ./internal/cli** after changing command composition or command
behavior. Pair it with **go test ./internal/core/config** when the configuration
handoff changes.

144
docs/internal/configuration.md
View File

@@ -1,144 +0,0 @@
# Configuration Internals
This document describes the maintainer-facing configuration boundary in
**internal/core/config**. The [Configuration](../config.md) reference owns the
file format, fields, defaults, precedence contract, and selectable keys. The
[CLI reference](../cli.md) owns command syntax; this document does not redefine
either interface.
## Boundary
The configuration package turns a selected YAML file and supported environment
values into a validated, independently owned configuration. It then resolves a
requested pipeline against a module catalog before the framework prepares or
runs anything.
| Boundary | Inputs | Outputs | Does not own |
| --- | --- | --- | --- |
| Loading | Selected file path and environment lookup | Parsed file model and a populated **Config** | Choosing the file path or reporting a command result. |
| Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
| Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. |
| Summary | **Config** or **EffectiveConfig** | Detached redacted payload suitable for debug summaries | Redacting arbitrary process state or provider traffic. |
The CLI discovers a configuration file, invokes this package, and supplies the
result to the framework. Configuration never reads an input file, constructs a
module, or creates output, cache, or debug paths. Those responsibilities remain
at their respective [CLI](cli.md), [pipeline](pipeline.md), and
[run-state](state.md) boundaries.
## Loading And Validation
The CLI loads configuration in this order:
1. parse the selected YAML file strictly into the file model;
2. start from **Default**;
3. apply the file model; and
4. apply the supported environment overrides.
This establishes the public precedence order without giving environment input a
second file schema. Loading and application reject malformed YAML, unsupported
file versions, unknown fields, invalid values, and identifiers that are empty
or collide after whitespace normalization. The file application also makes the
effective extraction-worker default follow the effective LLM limit. A present
PromptKit local-backend object requires and trims its endpoint, defaults its
omitted concurrency limit to zero, and is copied so the parsed file model
cannot alias the populated **Config**. A pipeline `llm_profile` is
presence-aware: omission remains empty, while a present blank value is
rejected and a non-empty file value is trimmed before it reaches **Config**.
**Config.Validate** checks configuration-only invariants before resolution. It
rejects incompatible profile sources, invalid state-surface values, unsupported
concurrency settings, malformed bindings and references, invalid retries, and
invalid pipeline, step, or lane structure. PromptKit local-backend validation
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
information, query, or fragment, and rejects a negative local concurrency
limit. Its errors retain the closest known pipeline, lane, and binding context.
It deliberately does not require modules to be registered: that requires a
catalog and belongs to resolution.
The exact user-selectable values and validation rules are defined in
[Configuration](../config.md). Keep additions to the file model, an
environment override, its validation, and that reference in the same change.
## Effective Resolution
**Config.Resolve** first recomputes derived concurrency defaults and validates
the configuration. It normalizes the requested pipeline ID, copies the selected
profile, and passes the non-empty command-level LLM profile override, requested
lane selection, and reference changes to the framework resolver.
After module and validator selection, the resolver applies the effective
profile policy to LLM-backed bindings only: command override, binding profile,
pipeline profile, then the prompt default. Deterministic bindings remain
profile-free, and no second inheritance decision occurs during execution. The
public field definitions and precedence are owned by
[Configuration](../config.md#pipelines).
The framework resolver supplies defaults, selects lanes, resolves validator
chains, checks registered module and artifact compatibility, validates module
options, and returns the fixed ordered pipeline shape. The resulting
**EffectiveConfig** retains the selected ID, requested selection and reference
changes, a clone of the input configuration, and the resolved pipeline.
Callers may therefore retain or modify their input slices and maps without
changing the resolved result, and later consumers cannot mutate the original
configuration through the effective value. This ownership includes the nested
PromptKit local-backend value.
Resolution failures stop before module construction and source parsing. They
include an error path for an unconfigured pipeline, missing module, missing
capability, incompatible artifact variant, invalid option, invalid reference,
or invalid lane selection. CLI code maps these valid-invocation failures to the
runtime error class described in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
## Resolved Identity And Redaction
The framework assigns the resolved pipeline a deterministic SHA-256 digest
after defaults, lane selection, module bindings, reference bindings, validator
chains, effective LLM profiles, and artifact schema identity have been
resolved. The digest excludes
its own stored value. It identifies resolved composition rather than raw YAML
bytes, a debug payload, or all runtime state. The CLI records it as invocation
provenance before execution; cache and checkpoint identity have additional
owners in [Run State Internals](state.md).
Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
**RedactedResolvedPipelinePayload**, never a direct configuration marshal.
Those methods copy every binding and nested option container, replace values
whose key is credential-shaped with **[REDACTED]**, and omit materialized
reference content while retaining safe binding and reference provenance. The
payload must not alias the source configuration or resolved pipeline.
PromptKit's local endpoint and concurrency limit are preserved as non-secret
configuration metadata in the independently owned summary; the object contains
no credential value. This redaction is deliberately narrow: it protects
configuration summaries and does not authorize recording arbitrary environment
values or provider requests.
## Invariants To Preserve
- Defaults, YAML values, and environment values are applied in one direction;
later sources may override only their supported operational settings.
- A configuration is structurally valid before it is resolved, and a resolved
pipeline is compatible with the supplied catalog before preparation begins.
- Whitespace-normalized identifiers are unique wherever they identify a
pipeline, step, lane, worker, or reference slot.
- Resolution and summary generation return detached data. Redaction must cover
every configured and resolved binding, including nested validator bindings.
- The resolved digest changes when resolved composition changes and never
includes itself.
## Focused Tests
- **internal/core/config/file_config_contract_test.go** covers strict file
parsing, normalization, file application, and structural rejection.
- **internal/core/config/env_contract_test.go** covers supported operational
overrides and their precedence.
- **internal/core/config/validation_contract_test.go** covers configuration
invariants and contextual failures.
- **internal/core/config/effective_config_contract_test.go** covers defaults,
selections, overrides, resolution context, digest changes, and ownership.
- **internal/core/config/redaction_test.go** covers recursive credential
redaction, reference-content exclusion, and non-aliasing payloads.
Run **go test ./internal/core/config** after changing this boundary. Changes to
the handoff or resolved-composition semantics also need the focused framework
pipeline tests.

155
docs/internal/dnd.md
View File

@@ -1,155 +0,0 @@
# D&D Module Internals
This guide records the conventions shared by the production D&D module family.
It complements [Module Internals](modules.md), which owns generic registration
and extension mechanics, and [Configuration](../config.md), which owns the
selectable keys, bindings, reference syntax, and default validator chains.
## Durable Artifact Contracts
The six lanes have separate durable wire contracts. This guide deliberately
does not repeat their JSON shapes or schemas.
| Lane | Durable contract |
| --- | --- |
| Spells | [spell artifacts](../integrations/dnd-spell-artifacts.md) |
| NPCs | [NPC artifacts](../integrations/dnd-npc-artifacts.md) |
| Combat turns | [combat-turn artifacts](../integrations/dnd-combat-turn-artifacts.md) |
| Item events | [item-event artifacts](../integrations/dnd-item-event-artifacts.md) |
| NPC interactions | [NPC-interaction artifacts](../integrations/dnd-npc-interaction-artifacts.md) |
| Scene descriptions | [scene-description artifacts](../integrations/dnd-scene-description-artifacts.md) |
## Family Composition
The D&D registrar registers the family’s artifact codecs, extractors, typed
append-order mergers, normalizers, validators, prompt assets, fallback LLM
profile asset, and default validator chains. Each extractor and normalizer has
a stable module spec, explicit execution class, strict option decoding, and a
typed builder. Scene chunking, every extractor, and NPC normalization are
registered as `llm_backed`; the remaining current D&D mergers and normalizers
are `deterministic`. The metadata is available to catalog inspection and
resolved-pipeline debug data and determines which selected bindings inherit the
pipeline profile. Configuration remains the canonical owner of the exact keys,
profile precedence, and validator order.
Private structured-LLM response schemas are deliberately minimal. They reject
invalid JSON structure, missing required fields, incompatible types, and
unknown fields, while preserving semantic candidates for deterministic
validation. Do not promote a private response envelope into a durable schema;
the contracts above define durable data.
## Prompt Construction
D&D extractors assemble prompts from an ordered manifest of shared and
module-owned assets. Reuse the shared D&D system, evidence, identity,
reference, and transcript assets instead of copying their text into individual
modules. A manifest’s declared sequence, including cache-control placement, is
part of the prompt behavior.
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
profile. The D&D registrar embeds that fallback profile with the maintained
OpenRouter model, timeout, and service-tier policy. An operator may provide a
complete profile with the same ID through the configured PromptKit source; that
definition replaces the fallback rather than merging with it. The fallback
leaves reasoning and optional sampling controls unspecified. Deployment profile
selection and the maintained operator example are documented in
[Configuration](../config.md#promptkit-profiles).
All extraction prompts share this four-message rendered prefix: the system
message without cache control, the identity message without cache control, the
campaign-reference message with ephemeral cache control, and the chunk
transcript message with ephemeral cache control. This gives equivalent
extraction requests the same reusable prefix through their source material.
Extraction-evidence policy, generated NPC registries, spell catalogs, module
tasks, and instructions follow the transcript because they are not universal
across all extraction lanes. The final instructions message carries ephemeral
cache control; evidence, registry, catalog, and task messages do not. Preserve
this division when changing an extractor or its assets so prompt-cache behavior
remains stable.
The other D&D LLM prompts intentionally follow different patterns. Scene
chunking has no sibling extraction lane with which to share its full transcript,
so it renders campaign references before its task and instructions, then places
the cacheable full transcript last. NPC normalization keeps its task and
cacheable instructions before the candidate collection, followed by the
cacheable transcript windows: candidates must be available before their
supporting evidence is evaluated, and those windows are not a cross-lane
prefix. Mounted assets and their declared message order determine the prompt
fingerprint, so intentional prompt edits continue to invalidate stale
checkpoints.
All extractors use the shared prompt-input preparation rules. The current chunk
is copied into transcript material; player, party, glossary, and compatible
campaign references are context for disambiguation, not source evidence.
Reference prompt material is canonically ordered before it is rendered, which
keeps equivalent inputs stable across runs.
## Evidence, Candidates, And Normalization
The current transcript is the only durable evidence source. Extractors assign
the current source identity, preserve candidate evidence ranges for validators,
and canonically order or remove exact duplicate ranges without asking the
model to repair semantic errors. Campaign context and generated artifacts may
ground names or control routing, but they never establish evidence for a D&D
result.
Default chains keep responsibilities separate: structural validators assess the
candidate, source-reference validators resolve cited ranges against the current
source, durable-schema validation checks an approved representation, and
relatedness validators report advisory evidence concerns. The configured order
is documented in
[Configuration](../config.md#production-validator-keys-and-default-chains).
Normalizers are deterministic for spells, combat turns, item events, NPC
interactions, and scene descriptions. They canonicalize display values and
evidence, use source-document order for stable output, and issue bounded
warnings for changes or collapsed duplicates. The NPC normalizer is the
intentional exception: it first produces a deterministic candidate set, then
uses a bounded structured-LLM proposal to reconcile identity groups. Invalid
or unusable proposals retain the deterministic result and surface retry or
fallback diagnostics; the model does not directly replace durable records.
## Generated References And Grounding
Normalized D&D artifacts can be handed to a later step through a generated
reference binding. The framework verifies artifact compatibility and retains
producer provenance; consumers resolve the handed-off artifact into an
immutable, validated projection for each operation. External files are checked
during preparation, while generated artifacts are resolved at the handoff.
NPC registries are names-only grounding projections: they may canonicalize
actors for spells and combat turns and are required for NPC interactions, but
they do not supply evidence. Scene-description registries are eligibility-only
projections: they retain the current chunk’s classification data, not scene
prose or evidence, and exist to route combat extraction.
## Lane-Specific Rules
The following differences are intentional and should remain explicit when a
shared helper changes.
| Lane | Intentional behavior |
| --- | --- |
| Spells | May use a spell-catalog overlay and optional NPC grounding; the catalog validator supplies domain-specific semantic checks. |
| NPCs | Does not consume an NPC registry. Its normalizer is the LLM-assisted reconciliation exception described above. |
| Combat turns | Requires a scene-description artifact. It calls the LLM only for an exact `combat` classification; exact non-combat classifications return an accepted empty result, while missing or mismatched classifications return an empty result with a bounded warning. Optional NPC grounding never becomes evidence. |
| Item events | Uses campaign context for disambiguation but has no NPC-registry or scene-description dependency. |
| NPC interactions | Requires the normalized NPC registry at extraction and normalization, using it for canonical actor grounding only. |
| Scene descriptions | Produces the classifications consumed by combat routing; it does not consume an NPC registry or provide evidence for combat artifacts. |
The combat and scene-description contracts describe their exact handoff and
empty-result behavior in more detail:
[combat turns](../integrations/dnd-combat-turn-artifacts.md) and
[scene descriptions](../integrations/dnd-scene-description-artifacts.md).
## Focused Verification
When changing D&D behavior, test the affected codec, extractor, normalizer,
validator, prompt-asset manifest, and registry projection. Also test generated
handoffs at the integration boundary and run the full D&D module suite:
~~~sh
go test ./internal/modules/dnd/...
go test ./internal/modules/integration/...
~~~

251
docs/internal/llm.md
View File

@@ -1,251 +0,0 @@
# LLM Runtime Internals
`internal/framework/llm` is Notarius’s provider-independent structured
completion boundary. It adapts framework requests to PromptKit, bounds
provider calls, assembles registered prompt and schema assets, records selected
profiles, and redacts provider errors. The architectural boundary is defined in
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
credentials, and concurrency settings belong in
[Configuration](../config.md#promptkit-profiles) and
[Configuration](../config.md#concurrency-output-cache-and-debug).
## Structured Completion Boundary
Modules and LLM-backed validators depend only on
`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and
version, optional profile and session IDs, named input material, variables, and
a caller-owned decode target. The successful response returns the validated raw
structured bytes together with non-secret provider, model, profile, and token
metadata.
The caller owns the domain behavior: it chooses the prompt, prepares inputs,
selects the private response schema, and interprets the decoded result. The
adapter does not own source evidence, artifact conversion, normalization, or
durable schemas. Those responsibilities remain with the module and its
[integration contract](../integrations/).
`PromptKitClient` validates the request target and prompt identity, maps each
named material to a PromptKit inline artifact while preserving its origin URI,
maps the trimmed request session to PromptKit's direct per-run session field,
retains the same value as the `session_id` prompt variable for maintained
prompt compatibility, and forwards profile selection. It then creates one
frozen prepared execution, captures its caller-owned credential-redacted
details for debug material, and executes that exact snapshot through
PromptKit's prepared-execution boundary. The direct field
is authoritative for provider session behavior. A session ID is a stable,
non-secret correlation identifier and may be exposed to providers and provider
observability. The adapter returns PromptKit’s validated raw bytes rather than
re-encoding the decoded target. An empty optional material is represented as
one space so its named input is retained by PromptKit.
Client construction may also receive a run-wide reasoning-effort override from
the CLI factory boundary. The adapter copies the caller-owned pointer and
creates a fresh PromptKit execution override for each request: a nil pointer
inherits the selected profile, a non-empty value replaces it, and an empty
value clears inherited reasoning. The CLI's mutually exclusive
`--reasoning-effort` and `--clear-reasoning-effort` controls select those
states. With neither flag, profile behavior remains unchanged. Because
production constructs one shared client, the selected state applies uniformly
to module calls, retries, and LLM-backed validators for the whole run.
An empty request profile lets the prompt select its configured default. Before a
run begins, the CLI asks the adapter to inspect every explicit profile on the
resolved selected LLM-backed bindings and validators, including inherited
pipeline profiles. Inspection resolves the profile and its selected backend and
target without loading a prompt, reading credentials, admitting capacity, or
contacting a provider, so a missing or invalid explicit profile fails before
stage execution while a valid `api_key_env` may remain unset. Calls record the
profile actually selected by PromptKit. The recorder trims and deduplicates
non-secret profile identity, provider, model, selected backend ID, and
effective reasoning values for manifest use. Entries that differ in backend or
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
retain an empty backend ID, which the published JSON omits. Successful
completion responses and recorded profile manifests identify the adapter
provider as `promptkit`.
The CLI's profile-inspection engine and the production adapter use the same
profile-source construction to apply the configured profile directory or file,
the optional registered fallback profile assets, and the optional conventional
`local` backend. Preflight therefore resolves the same profile sources and
backend membership as runtime without performing generation. Fallback assets
are mounted only when at least one source is registered. The production D&D
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
prompts select that logical ID by default. PromptKit owns source precedence and
profile parsing: an operator-provided matching profile takes precedence over a
fallback profile without Notarius merging either document.
When the registration is absent, a profile selecting `backend: local` fails
inspection instead of falling back to a built-in or endpoint-only target.
Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the
identity of PromptKit's compiled-in profile catalog with a deterministic digest
of every YAML profile in the configured profile directory, or of the configured
profile file, and a deterministic digest of the flattened fallback profile
assets. The fingerprint contains neither profile content nor source paths. It
covers inherited pipeline profiles, explicit binding profiles, and
prompt-selected defaults, so changing a model or other profile setting cannot
reuse checkpoints created under the
prior profile source. This cache identity is independent of durable
profile provenance: run manifests continue to list only profiles actually
observed during LLM calls. When the local backend is registered, a second
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
stored in checkpoint identity, and the local concurrency limit is excluded
because it changes scheduling rather than execution semantics.
## Shared Provider-Call Limit
Production construction creates one PromptKit client and wraps it in one
scheduled client. The scheduler has a fixed, positive permit limit, serves
queued calls in FIFO order, and removes a queued call when its context is
cancelled. A granted permit is released exactly once on every completion path.
The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent
lanes, pipeline retries, and LLM-backed validators share the same provider-call
ceiling. This ceiling is independent of pipeline worker concurrency; changing
worker counts cannot exceed the configured LLM limit. The configuration field
and its effective default are owned by
[Configuration](../config.md#concurrency-output-cache-and-debug).
PromptKit applies a second, independent admission limit when the selected
profile names a limited backend. It sits beneath the Notarius scheduled client,
so it may narrow but cannot expand the application-wide limit. Built-in
OpenRouter profiles select PromptKit's reserved backend and its upstream
capacity policy. A positive configured local-backend limit bounds active local
generations inside PromptKit; zero leaves that backend unlimited there.
Endpoint-only profiles do not select a PromptKit backend and remain limited
only by the Notarius scheduler.
## Prompt And Schema Assets
An `AssetRegistry` collects prompt, schema, and optional fallback-profile
filesystems from production module families. It flattens registered roots into
the corresponding PromptKit filesystems and rejects invalid roots, unreadable
assets, duplicate paths, and missing prompt or schema files during preparation.
Fallback assets receive a safe content digest for checkpoint identity; raw
paths and bytes are never included. The framework’s `promptfs` helper combines
module-owned prompt files with reusable domain fragments without making the
framework depend on D&D content.
Each LLM-backed module owns its prompt declaration, package-specific assets,
and private response schema. Shared D&D wording is owned by the D&D shared
asset package; the detailed D&D conventions are in
[D&D Module Internals](dnd.md). The mounted prompt assets used by a module also
determine its prompt fingerprint. Schema loaders validate JSON, attach identity
and digest metadata, make defensive copies, and expose diagnostics without raw
schema bytes.
Private response schemas validate a model transport envelope. They are not the
durable artifact schema and should not be documented as an external wire
contract. Durable formats and compatibility rules remain in the
[integration contracts](../integrations/).
## Prompt Maintenance And Backend Caching
Prompt message order and shared asset bytes are runtime behavior. Backend cache
reuse depends on identical preceding roles, rendered bytes, and cache-control
metadata—not merely equivalent meaning. Keep reusable shared assets
byte-identical and preserve each prompt’s declared ordering and cache controls
when editing it.
For sibling prompts that can reuse the same source material, order universal
shared context first, request source material next, and module-specific
suffixes last. Put a cache boundary at a reusable prefix that is useful to the
backend. Redundant intermediate cache boundaries do not extend that reusable
prefix and add no value.
Prompt-family owners may choose a different sequence when their inputs and
reuse pattern differ. The D&D family’s extraction, scene-chunking, and NPC
normalization policies are maintained in [D&D Module Internals](dnd.md#prompt-construction).
Do not add tests that enforce prompt prose; prompt tests should verify the
meaningful input placement and cache controls of the prompt being changed.
## Validation, Repair, And Retries
PromptKit performs prompt rendering, provider execution, and the prompt’s
structured-output validation. The adapter reports an empty result, validation
failure, empty structured body, or decode failure as
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
material when they exist. Provider failures remain operational errors rather
than output-validation failures.
When PromptKit rejects backend admission before generation, the adapter maps
`promptkit.ErrCapacityExceeded` to
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
type as a framework contract. When supplied, the normalized selected backend
ID appears only in that safe application-owned diagnostic context. A canceled
caller context takes precedence. The adapter does not retry capacity failures;
the pipeline's existing binding attempt policy sees the operational error and
decides whether to rerun the complete operation.
Prompt-declared repair is executed within PromptKit’s structured-output flow.
The current production D&D prompt manifests set repair attempts to zero. That
setting does not replace pipeline retry behavior: a binding’s configured retry
count reruns its stage attempt after an error or rejection, and an exhausted
rejection is a recorded output rather than a provider error. The pipeline owns
attempt lifecycle, validation chains, and retry diagnostics; see
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
[binding reference](../config.md#module-bindings-and-validators).
## Timeout Ownership
The caller context remains the outer cancellation authority. PromptKit applies
a positive effective generation timeout as an inner request deadline; an
explicit zero disables only that generation deadline. The HTTP client timeout
is a separate transport-wide cap. Notarius forwards the caller context and
does not install another timeout wrapper around PromptKit.
The selected PromptKit profile owns generation settings. Notarius binding
retries remain outside the adapter and repeat the complete module operation
and validation chain. PromptKit does not add a provider retry loop.
Operator-facing behavior is summarized in
[Operations](../operations.md#operational-limits), and the pinned upstream
contract is identified in
[PromptKit Integration](../integrations/pkg-promptkit.md).
## Observability And Redaction
When debug recording is enabled, the pipeline decorates the shared client. The
wrapper records prepared prompt and response material, timing, selected profile
and backend, effective model parameters, and call identifiers in the run’s
debug bundle, including material available from a failed structured completion.
Effective parameters use PromptKit's stable lower-case JSON field names and may
include `backend_id`. For a successful completion, a debug-write failure is
surfaced; when the completion already failed, its call error remains the
result. Debug-bundle location, retention, and handling are operational concerns
documented in [Operations](../operations.md#debug-bundles).
Run manifests receive selected profile summaries, including optional effective
backend and reasoning provenance, and component identities—not prompt, schema,
source, reference, or response content. The published field semantics belong
to the [JSON output contract](../integrations/json-output.md#manifestjson).
Provider error text is wrapped with prompt context and bearer credentials are
redacted before it crosses the runtime boundary. Known-secret redaction is
available to other runtime collaborators; it does not make prompt or response
contents safe for general logging.
## Failure Boundaries
- Construction fails for missing asset registries, mutually exclusive profile
sources, invalid asset registration, or a non-positive scheduler limit.
- Preparation failures, unavailable explicit profiles, provider failures, and
context cancellation propagate to the calling stage with context.
- Backend admission exhaustion is a provider-neutral operational error and is
not classified as invalid structured output or validator rejection.
- Malformed or schema-invalid provider output is classified separately as
invalid structured output so the module or pipeline can apply its own retry
and rejection policy.
- Domain semantic checks, evidence decisions, and deterministic normalization
run outside the provider adapter.
## Focused Verification
Read the LLM adapter, scheduler, asset registry, schema loader, and redaction
tests when changing this boundary. Prompt changes also require the owning
module’s asset tests, and retry or debug changes require focused pipeline or
CLI coverage. The focused runtime and D&D checks are:
~~~sh
go test ./internal/framework/llm/... ./internal/modules/dnd/...
~~~

113
docs/internal/modules.md
View File

@@ -1,113 +0,0 @@
# Module Internals
This guide owns the mechanics for implementing and registering production
modules. [Configuration](../config.md) owns selectable keys, binding syntax,
reference configuration, and default validator chains. Durable input and output
shapes belong in [integration contracts](../integrations/).
The D&D family has additional shared conventions and domain-specific
exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
## Module Boundary
A module is a typed implementation registered for one pipeline stage. Its
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
stage, execution class, required and provided capabilities, artifact kind, and
accepted reference slots. The execution class states whether a module is
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
resolved-pipeline debug data without constructing the module. The framework
uses the declaration to resolve a configured binding before it builds the
implementation. After selection, the resolver applies profile inheritance only
to bindings whose declared execution class is `llm_backed` and rejects a
binding-specific profile on a deterministic module. The user-facing precedence
contract belongs in [Configuration](../config.md#pipelines).
Implementations that accept options must provide both an option validator and
a builder. The validator is used while resolving configuration; the builder
decodes the same options and constructs the implementation from the prepared
`BuildRequest`. Reject unknown options in both paths. A builder receives only
the dependencies and materialized references that the framework prepared for
that operation, so it must not re-read configuration or files.
Registry helpers register the typed builder for a stage-specific registry.
They are preferable to hand-written untyped registration because they retain
the artifact type at the framework boundary. Registrars validate the registries
they need, register each leaf implementation, and add any family-owned assets
or default validator chains. They return contextual errors so production
composition fails at startup rather than at the first run.
An artifact family can register an optional typed evidence projector alongside
its codec. The projector returns defensive copies of the artifact's direct
generic source references and must use the codec's exact Go type. It does not
interpret surrounding context or publish files; the pipeline validates the
capability during preparation and the output boundary owns publication. See
the [Published Evidence Context contract](../integrations/evidence-context.md)
for the durable result.
## Production Composition
Production composition is intentionally split by family:
- The generic registrar provides the unit chunker, generic JSON validators,
and JSON output encoder.
- The Seriatim registrar provides the transcript input adapter. Its external
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
are documented in [D&D Module Internals](dnd.md).
The CLI owns the composition that invokes these registrars. A module package
may register its own family but must not assemble the CLI or make framework
packages depend on production extensions.
## Adding Or Changing A Module
1. Choose the pipeline stage and the typed artifact boundary. Put external
input or durable artifact formats in the relevant integration contract,
not in this guide or in a private LLM response type.
2. Define a stable `ModuleSpec` with an explicit execution class, the exact
capabilities, and reference slots needed for the operation. Model a
producer/consumer handoff as an artifact-compatible slot; configuration
then chooses an external file or a generated binding.
3. Implement strict option decoding, construction, and the typed stage
interface. Preserve caller ownership: do not retain mutable request data
and return defensive copies where an implementation exposes stored data.
4. Register the module through its typed registry helper and add it to the
owning family registrar. Add a default validator chain only when that
family owns the behavior; otherwise require an explicit compatible chain.
5. Update the selectable-key and chain reference in
[Configuration](../config.md#production-module-keys), the applicable
integration contract, and focused tests. Keep the configuration document
as the sole list of production keys and validator order.
## Validation And References
Validators operate on the value produced at their configured stage. A default
chain is ordered behavior, not a set: JSON parsing, structural checks,
domain-specific checks, durable-schema checks, and advisory checks may have
different responsibilities and failure handling. The active default chains and
override rules are maintained in
[Configuration](../config.md#production-validator-keys-and-default-chains).
Reference slots are part of the module specification. They describe the
accepted artifact kind, media type, size, and whether a binding is required;
the framework validates those constraints before construction. An external
reference is materialized during preparation. A generated reference is a
compatible normalized artifact handed from an earlier pipeline step at
operation time. The configuration reference rules, including precedence and
ordered-handoff requirements, are maintained in
[Configuration](../config.md#references-and-ordered-handoffs).
## Focused Verification
Exercise the leaf implementation and its registration path when changing a
module. Registry and registrar tests cover duplicate keys, required registries,
and typed construction; pipeline resolution tests cover capabilities, options,
and reference compatibility. Domain packages should additionally test their
codecs, validators, normalizers, and any integration handoffs they own.
Run the affected package tests while iterating. The complete module suite is:
~~~sh
go test ./internal/modules/...
~~~

58
docs/internal/overview.md
View File

@@ -1,58 +0,0 @@
# Internal Overview
This document is the implemented component map for Notarius. Normative
boundaries and dependency direction belong in
[Architecture](../policy/architecture.md). User and operator contracts belong
in the [CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [integration contracts](../integrations/).
## Execution Path
~~~
cmd/notarius -> internal/cli -> configuration and production composition
-> internal/framework/pipeline -> logical output files
-> internal/cli -> durable output and optional state/debug data
~~~
The CLI is the application boundary: it discovers configuration, composes
production registries and runtime collaborators, invokes the framework, and
places returned files. The framework resolves and prepares a fixed extraction
pipeline, then returns logical results without owning process behavior or
physical state roots.
## Components
| Area | Implemented owners | Responsibility |
| --- | --- | --- |
| Executable and command boundary | **cmd/notarius**, **internal/cli** | Process entry, command dispatch, configuration discovery, production composition, runtime collaborator setup, durable file placement, and user-facing reporting. |
| Configuration | **internal/core/config** | Defaults, strict YAML parsing, environment overrides, structural validation, effective resolution, redaction, and resolved-composition summaries. |
| Generic models | **internal/core/source**, **internal/core/artifacts**, **internal/framework/contracts** | Source documents and chunks, manifests and provenance, plus typed artifact, reference, validation, output, and structured-completion contracts. |
| Pipeline framework | **internal/framework/pipeline** | Registries, profile and reference resolution, typed preparation, validation, retry coordination, ordered execution, handoff, and result assembly. |
| LLM and prompt runtime | **internal/framework/llm**, **internal/framework/promptfs** | Provider-neutral structured completions, scheduling, profile recording, prompt assets, schema registration, and credential-shaped-value redaction. |
| Runtime state | **internal/core/fileio**, **internal/core/debugbundle**, **internal/framework/checkpoint**, **internal/framework/chunkplan**, **internal/framework/chunkmap**, **internal/framework/debug** | Confined atomic files, debug bundles, checkpoint and chunk-plan state, accepted chunk maps, and pipeline-facing debug recording. |
| Production extensions | **internal/modules/generic**, **internal/modules/seriatim**, **internal/modules/dnd** | Domain-neutral extensions, Seriatim input support, and D&D extraction families registered into the production catalog. |
Generic core and framework packages do not depend on production extensions.
Concrete extensions depend inward on their contracts and are registered only at
the CLI composition boundary.
## Focused Documentation
- [Configuration Internals](configuration.md): loading, validation, effective
resolution, redaction, and resolved-composition identity.
- [CLI Internals](cli.md): command dispatch, production composition, run
orchestration, and terminal reporting.
- [Pipeline Internals](pipeline.md): resolution, preparation, execution,
validation, typed handoff, and framework state hooks.
- [Run State Internals](state.md): output, cache, debug collaborator
composition, and path safety.
- [LLM Runtime](llm.md): structured completion, scheduling, prompt assets,
profiles, and secret handling.
- [Module Internals](modules.md): generic extension registration, module
construction, validation, and reference mechanics.
- [D&D Module Internals](dnd.md): shared D&D extractor conventions, generated
reference projections, and lane-specific exceptions. Durable D&D and
Seriatim data shapes remain in the [integration contracts](../integrations/).
Use this map to find an owner, then read the focused document and its tests
before changing behavior.

185
docs/internal/pipeline.md
View File

@@ -1,185 +0,0 @@
# Pipeline Internals
This document describes the framework-owned pipeline mechanics in
**internal/framework/pipeline**. [Configuration](../config.md) owns selectable
profiles, bindings, and retry settings; [Operations](../operations.md) owns
state lifecycle and recovery; and the [integration contracts](../integrations/)
own durable output shapes. Concrete production extensions are covered by
[Module Internals](modules.md).
## Boundary
The pipeline framework accepts a resolved composition, registries, shared
dependencies, input bytes, and state/debug collaborators. It returns logical
output files, normalized artifacts, recorded rejections and warnings, manifest
provenance, and checkpoint decisions. The CLI owns process arguments,
configuration discovery, physical roots, and placement of returned output
files.
The framework has one fixed shape:
~~~
input -> chunk -> extract -> merge -> normalize -> output
~~~
Input and chunking are pipeline-wide. A selected artifact lane owns extract,
merge, and normalize; output aggregates the terminal lane outcomes. A pipeline
is an ordered list of steps, not an arbitrary workflow graph.
## Resolve, Materialize, Prepare
Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
It normalizes the pipeline and lane identities, applies stage defaults, selects
requested lanes where that is supported, resolves validator chains, checks
module capabilities and typed artifact compatibility, validates options, and
assigns a deterministic resolved-composition digest. The resolved pipeline
contains bindings and declared reference targets, not external reference bytes.
After selection, the resolver applies command, binding, and pipeline profile
precedence to LLM-backed bindings and validators only; prompt defaults remain
an empty resolved binding profile. Deterministic bindings remain profile-free.
These effective values are part of the digest, so execution and checkpoint
consumers do not repeat profile inheritance.
Configuration resolution supplies the selected profile and catalog; see
[Configuration Internals](configuration.md).
External reference materialization happens before preparation. The materializer
checks that each slot is declared by the selected module, resolves a file path
relative to the correct configuration or working-directory origin, reads
UTF-8 text, verifies media type and size limits, and retains bounded
provenance. A generated-artifact selector remains declared but has no bytes
until its producing step completes.
Preparation is the construction boundary. It validates the resolved shape and
registry set, clones the resolved data, then constructs the input adapter,
chunker, stage-local validators, every typed lane, and output encoder with
cloned options, references, and shared dependencies. It also collects stable
checkpoint fingerprints. Missing registrations, incompatible typed entries,
nil implementations, and constructor failures are reported before source
parsing or any stage operation begins.
An output encoder can opt into source-evidence publication through its output
policy. Preparation keeps the configured lane allowlist and active lanes
separate, then verifies an exact typed evidence projector and registered codec
for each active lane. The resulting private plan is immutable; lanes excluded
by invocation filtering remain configured but do not acquire a projector for
that run.
## Typed Lanes And References
Each resolved lane has one artifact kind, codec, and exact Go type. The
framework uses private type erasure only around those typed operations; every
handoff checks exact type and codec identity and reports incompatibility as an
error rather than panicking. Encoding through the registered codec is the
boundary for output, checkpoints, debug records, and generated references.
Reference targets are stage- and lane-specific. External reference bytes are
cloned into the operation request. Generated references are built at the next
step boundary from exactly one accepted normalized producer output. The
framework decodes and re-encodes that output with the registered producer
codec, checks its complete schema and media identity, and records a content
digest plus bounded producer provenance. A missing, ambiguous, invalid, or
incompatible producer prevents the consumer step from starting.
## Execution And Ordering
The runner validates its input, installs no-op state collaborators when none
were supplied, and serially performs source parsing and chunk-plan selection.
An accepted plan is materialized into source-addressed chunks and passes the
configured chunk validators before any lane runs. A chunk rejection is a
recorded pipeline outcome: lanes do not start, but the output stage can encode
the terminal result.
For each ordered step, the runner first builds generated reference sets from
the accepted normalized outputs of earlier steps. It then executes the step's
lanes. Later steps do not begin until the current step is terminal and its
generated handoffs have succeeded.
Within a step, the lane engine dispatches extraction jobs in deterministic
chunk-first, lane-second order to a bounded worker group. When all extraction
jobs for one lane are terminal, a bounded continuation group can run that
lane's merge and normalize work while extraction for other lanes continues.
The framework does not create an unbounded goroutine per chunk or lane.
Completion timing does not determine public results. The coordinator restores
lane and chunk order before merging results, and selects a framework error by
stable stage, lane, and chunk position. A validator rejection records a lane
outcome without cancelling unrelated work. A framework error or parent
cancellation cancels derived work, prevents queued work from starting, waits
for started workers, and prevents output encoding.
## Validation, Retries, And Output
Every chunk, extract, merge, and normalize candidate passes its resolved
validator chain. Validators receive immutable canonical input appropriate to
their target: chunks, typed values, or serialized codec bytes. They may
approve, approve with warnings, reject, or fail. A rejection is an ordinary
pipeline result; a validator error is a framework error.
The runner applies the binding's retry policy around a stage operation and its
complete validation chain. It preserves warnings only from the final accepted
or rejected attempt. Cancellation stops retries. Normalizer-specific retry
directives consume this same budget and validate any final safe fallback through
the normalizer chain.
After terminal lane work, the runner assembles manifest provenance, normalized
artifacts, rejections, warnings, and an optional accepted chunk map. When an
output policy selected evidence lanes, it decodes accepted serialized normalize
outputs through their registered codecs and invokes the prepared typed
projectors. Rejected or absent lanes contribute nothing. This reconstruction is
also used after normalized-checkpoint reuse, so no second typed output channel
is retained. The runner passes the resulting owned artifact to the output
encoder, which returns logical files and does not choose a physical directory.
The CLI publishes those files only after the runner returns without a framework
error. Logical file names and schemas are defined by the [output integration
contracts](../integrations/).
## Checkpoint And Debug Hooks
The runner receives checkpoint and debug interfaces rather than roots. It
records workflow transitions and reuse decisions through the supplied
collaborators, and clones reusable artifacts before they re-enter normal typed
handoff. Generated-reference dependencies participate in checkpoint decisions.
Selective recomputation can require a canonical accepted normalized predecessor
before a dependent lane starts.
Debug recording is attempt-scoped and application-owned. A failure to persist
required debug data is a framework error. State roots, persistence, reason-code
meanings, resume, and cleanup are intentionally owned by
[Run State Internals](state.md) and [Operations](../operations.md).
## Invariants To Preserve
- The six fixed stages remain explicit; a pipeline is not a general DAG.
- Resolution and preparation reject statically discoverable incompatibility
before parsing or execution.
- Every typed lane uses one compatible artifact kind, codec, and exact Go type.
- Generated references come only from one earlier accepted normalized producer
and carry canonical identity rather than an unverified value.
- Rejections are recorded outcomes; framework errors cancel derived work and
prevent output encoding.
- Public ordering and selected errors are independent of goroutine completion
order.
- Pipeline modules receive collaborators and data, never CLI streams or
physical output, cache, or debug roots.
## Focused Tests
- **internal/framework/pipeline/profile_test.go** and
**typed_resolution_test.go** cover resolution, defaults, ordered steps,
compatibility, validators, references, and resolved identity.
- **internal/framework/pipeline/preparation_test.go** covers complete
construction before execution and contextual construction failures.
- **internal/framework/pipeline/references_test.go** and **handoff_test.go**
cover external materialization, generated references, provenance, and typed
producer checks.
- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded
execution, ordered steps, stable error selection, rejections, and
cancellation.
- **internal/framework/pipeline/runner_chunk_plan_test.go**,
**runner_typed_checkpoint_test.go**, and
**runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior.
- **internal/framework/pipeline/runner_attempt_debug_test.go** and
**runner_terminal_debug_test.go** cover attempt and terminal debug behavior.
Run **go test ./internal/framework/pipeline ./internal/cli** after changing a
pipeline boundary. Use the more focused tests above while iterating.

147
docs/internal/state.md
View File

@@ -1,147 +0,0 @@
# Run State Internals
This document describes the implementation collaborators behind output, cache,
and debug state. User-visible fields belong in [Configuration](../config.md),
and physical layout, retention, recovery, reason codes, and cleanup belong in
[Operations](../operations.md).
## Composition
`internal/cli` is the only physical-path composition root. It resolves the
effective configuration, selects exact roots, allocates requested debug bundles,
constructs cache collaborators, writes logical output files, and reports paths.
Pipeline modules receive interfaces and request data, never output, cache, or
debug roots.
The CLI creates no chunk-plan store in bypass mode. It creates a checkpoint
recorder only when recording is enabled and a checkpoint loader only for a
resume invocation. It allocates debug state only after a safe run identity has
been generated and only when debug capture was requested. These choices keep
the three state families independently composable.
## Output And Cache
The pipeline runner returns logical output files. After validating every
logical name, the CLI exclusively creates the run directory beneath the
selected output root and performs confined, atomic file writes within it.
The runner supplies an accepted chunk map as an optional, defensively owned
output-request artifact. The JSON encoder alone decides whether its explicit
option writes the map and optional index descriptor; neither the map payload
nor its annotations are copied into the run manifest. The durable fields are
owned by the [Accepted Chunk Map contract](../integrations/chunk-map.md).
`internal/framework/chunkplan` owns source-addressed plan storage, validation,
and atomic publication. Its store is constructed only when the selected mode is
not `bypass`.
`internal/framework/checkpoint` owns checkpoint identity, manifests, payload
codecs, loader, and recorder. The CLI constructs a recorder whenever checkpoint
recording is enabled and constructs a loader only for a `--resume` invocation.
Identity incorporates explicit stable semantic fingerprints collected from
prepared modules and validators in addition to configuration, input,
references, runtime overrides, observed LLM profiles, and the LLM runtime's
non-secret effective profile-source identity. A profile source change therefore
causes a cold miss even when the configured profile ID remains unchanged.
The serialized
`workspace_schema_version` identifiers are frozen wire-compatibility fields;
they do not describe a current public state surface.
Ordered-step lane checkpoints include the step identity in their storage scope.
When a later lane consumes a generated artifact, its dependency fingerprints
include the producer's artifact kind, complete schema identity, media type,
canonical content digest, and size. Ordinary resume compares those fingerprints
when progressively loading consumer stage checkpoints, so changed producer
content produces `dependency_invalidated` rather than stale downstream reuse.
Selective recomputation instead requires each unselected producer's accepted
normalized artifact; invalid accepted state records its specific bounded reason
and stops before the dependent. The selected step and its transitive dependents
record `forced_recompute`.
Ordinary resume loads extract, merge, and normalize checkpoints progressively
and may execute later lane stages after an earlier cache miss. Selective
recomputation instead asks the loader for the required producer's accepted
normalize artifact. That lookup reuses the existing normalize files, requires
workspace schema v3 plus an exact non-empty invocation identity, and deliberately
does not require extract or merge checkpoint files or dependency fingerprints.
The runner performs canonical codec and producer-provenance validation before
cloning the artifact into normal step output. Success restores only stored
normalize warnings and emits one normalize decision; failure retains the files,
records the decision, and stops without executing the producer or consumer.
The loader assigns a typed category and reason code at each validation site;
diagnostic prose is not classified after the fact. The runner then applies
forced-execution policy, validates reusable artifact bytes through the prepared
codec once, returns the canonical hydrated value to the stage, and records the
final decision before enforcing a required-predecessor failure. That failure
names only the step, lane, and stable reason code. Decision detail is selected
from code-owned descriptions by reason code and then UTF-8 normalized and
bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and
codes remain intact through pipeline events and become strings only in manifest
and debug-summary JSON.
[Operations](../operations.md#checkpoint-recording-resume-and-recompute) is the
canonical operator-facing reason-code reference.
`internal/core/fileio` provides confined atomic file writes used by state
collaborators. The chunk-plan store retains its stronger entry validation.
The CLI constructs selective-recomputation policy from resolved generated
artifact dependencies. It forces the selected step and transitive consumers,
while marking unforced producers as required reusable inputs. The runner owns
the actual hydration and rejection decisions; the [Operations guide](../operations.md#checkpoint-recording-resume-and-recompute)
owns the operator workflow and stable reason-code meanings.
## Debug Bundles
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command,
resolution, run, warning, and failure artifacts. `internal/framework/debug`
implements the pipeline-facing trace recorder under the trace root.
The CLI allocates a bundle before pipeline resolution and treats requested
summary or trace persistence failures as command failures. The pipeline's debug
boundaries redact sensitive metadata and credential-shaped bytes while allowing
application-owned trace material. Debug data is never a checkpoint source or
cache input.
Generated reference bytes exist only in cloned operation requests and are not
written as paths into checkpoints, manifests, or debug summaries. Those state
surfaces retain canonical identities and bounded producer provenance so that a
resume decision can be explained without copying generated campaign content.
After allocation, one CLI-owned state value accumulates the known report paths,
pipeline outcome counts, and validation status. A single guarded terminalization
operation writes the success report, or makes one attempt each to write the
failure report and error log. Terminal persistence failures are reported
separately and never replace the command's primary error.
## Invariants To Preserve
- Modules receive state collaborators and request data, never physical roots.
- Output logical paths are validated before a run directory is allocated, and
files are atomically written within that directory.
- Chunk-plan publication occurs only for accepted plans; bypass does not
construct or touch a plan store.
- Checkpoint recording and checkpoint loading remain separate collaborators.
- Debug state is opt-in, is not cache input, and terminal reporting does not
obscure the command's primary failure.
## Tests To Inspect
- `internal/cli/run_contract_test.go`: command-owned state allocation,
terminalization, and output/report boundaries.
- `internal/cli/cache_contract_test.go`: cache-mode precedence, root selection,
and resume collaborator construction.
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
permissions, cleanup, and redaction.
- `internal/cli/recompute_policy_test.go`: forced dependents and required
reusable predecessors for selective recomputation.
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
filesystem recovery, deterministic decisions, and failed predecessor state.
- `internal/cli/production_contract_test.go`: production composition and
configuration validation at the CLI boundary.
- `internal/cli/example_contract_test.go`: maintained example ownership.
- `internal/core/debugbundle/*_test.go`: bundle allocation and summary writes.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and
reuse.
- `internal/framework/chunkplan/store_test.go`: plan envelope, confinement,
publication, and permissions.

291
docs/operations.md
View File

@@ -1,291 +0,0 @@
# Operations
This is the canonical guide for operating Notarius runtime state. The
[CLI reference](cli.md) owns command syntax and exit statuses, while
[Configuration](config.md) owns fields, defaults, and precedence. Maintainers
who need implementation mechanics should read [Run State Internals](internal/state.md).
## State Surfaces
Each run can use independent roots with different retention and access-control
needs.
| Surface | Purpose | Created when | Retention |
| --- | --- | --- | --- |
| Output | Durable user-facing result bundle | A pipeline completes and returns logical output files | Keep until consumers no longer need it. |
| Chunk-plan cache | Reconstructible source-addressed plan | The configured cache mode permits cache I/O | Keep while reuse is useful. |
| Checkpoint cache | Reconstructible execution and recovery state | Checkpoint recording is enabled | Keep only while recovery or reuse is useful. |
| Debug bundle | Explicit diagnostic record | A run requests debug collection | Keep only under an intentional sensitive-data retention policy. |
Output, cache, and debug roots are never merged or cleaned automatically. Use
separate locations and permissions for operators or services that must not
share application data.
## Roots And Permissions
The configured output and debug directories are exact roots. An empty cache
directory selects a per-user root:
~~~
<os.UserCacheDir>/notarius/chunk-plans
<os.UserCacheDir>/notarius/checkpoints
~~~
The field definitions and configuration examples are in [Configuration](config.md).
On supported Unix systems, output directories and files are created with
requested modes **0755** and **0644**. Chunk-plan, checkpoint, and debug
directories and files use **0700** and **0600**. The operating system's umask
may impose stricter output modes. Cache and debug roots may contain sensitive
source-derived data, so provision them for one trusted account or service. An
output bundle can also contain source content when its JSON output enables
evidence publication. Apply an appropriate umask and output-root access policy
before enabling that option; the requested output modes alone may not be
suitable for transcript-bearing bundles.
## PromptKit Profile Deployment
Profile deployment has four distinct layers:
| Layer | Owner | Operational role |
| --- | --- | --- |
| Prompts and schemas | Notarius module families | Embedded request and structured-output definitions. They are not deployment profile files. |
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
| Operator profiles | Deployment filesystem | Complete environment-specific definitions selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
embedded fallback makes that ID usable without an operator file. Production,
development, and local deployments can each install a different complete
definition for the same ID, retaining the pipeline while choosing their own
model, backend, timeout, or reasoning policy. An operator definition wins over
the fallback; it is not merged with it. The configuration field and full
precedence rules are owned by [Configuration](config.md#promptkit-profiles).
Use a profile source owned by the service account, keep it readable only by
the intended operator, and supply provider credentials through the service
environment—not in the Notarius configuration or profile YAML. The maintained
[operator profile](../examples/profiles/dnd-extraction.yml) is secret-free and
can be copied as a format starting point. Validate a deployment without a
provider call or credentials:
~~~sh
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
~~~
Profile paths are currently resolved from the process working directory, not
from the configuration file. The complete example's
`./examples/profiles/dnd-extraction.yml` path is valid for a repository-root
invocation only. Use absolute paths such as
`/etc/notarius/profiles/dnd-extraction.yml` for services and containers.
## Run Lifecycle
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
and resolves configuration before module preparation and source parsing. It
then performs any permitted cache lookup, executes the pipeline, and publishes
logical output files only after a successful runner result.
On success, the command reports the output bundle path. A warning-bearing run
still succeeds and reports its warning count on standard error. Errors and
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
## Output Bundles
Each successful run receives a generated safe run identifier and writes beneath:
~~~
<output-root>/<run-id>/
~~~
The [JSON output contract](integrations/json-output.md) owns the logical files
and their schemas. Before creating the run directory, Notarius validates every
logical output path. It refuses an existing run directory without changing it.
Files are written atomically; if a later write fails, the newly created partial
run directory remains for inspection and is never removed automatically.
Treat an output bundle as durable user data. Do not use cache-cleanup policy to
remove it. An optional accepted chunk map is also durable output and can carry
source- or model-derived annotations; its content and compatibility contract
are defined in [Accepted Chunk Map](integrations/chunk-map.md). An optional
[evidence context](integrations/evidence-context.md) contains source-unit text
and metadata. It is not a cache or debug artifact: retain it with the output
bundle only for as long as consumers need it, and apply source-content access
controls to the entire bundle. Selected lanes may collectively cite most of a
transcript, so a broad allowlist can make the evidence artifact nearly as
sensitive and large as the source itself.
## Chunk-Plan Cache
Chunk plans live beneath the selected chunk-plan root:
~~~
<chunk-plan-root>/<source-sha256-hex>/plan.json
~~~
One validated canonical plan is active for each source digest. The plan stores
boundaries and provenance, not a second copy of the entire source. This
source-addressed policy is recorded in [ADR-0005](adr/0005-cache-canonical-chunk-plans-by-source.md).
The configured cache mode controls one invocation:
- **auto** looks for a valid active plan. Missing or invalid state causes a new
plan to be generated; an accepted new plan is atomically published.
- **refresh** skips lookup, generates a plan with the configured chunker, and
atomically replaces the active plan after it is accepted.
- **bypass** performs no chunk-plan cache I/O. It does not resolve or create a
chunk-plan root.
A reused plan is still materialized and validated against the current source.
If a prior plan no longer gives acceptable results, use a refresh run rather
than editing cache files. Deleting a plan is recoverable but can repeat costly
chunking work.
## Checkpoint Recording, Resume, And Recompute
Checkpoint recording is an explicit configuration choice and is disabled by
default. When enabled, each run records stage transitions and the state needed
for compatible recovery. A run records checkpoints even when it does not ask
to reuse them. Checkpoint payloads can contain source-derived and intermediate
application data, so treat the entire root as sensitive.
Checkpoint loading is separate: [**--resume**](cli.md#run) asks a run to reuse
compatible recorded work. A resume request fails when checkpoint recording is
disabled. Without **--resume**, a recording-enabled run executes normally and
does not load checkpoint state. Compatibility includes the resolved pipeline,
input, selected lanes, runtime overrides, reference provenance, LLM-profile
provenance, the effective PromptKit profile-source fingerprint, and
prepared-component fingerprints. When a local PromptKit backend is configured,
compatibility also includes a non-secret fingerprint of its endpoint. Changing
profile content or the local endpoint causes a cold miss; changing only the
local concurrency limit does not. A changed identity produces a cold miss;
Notarius does not migrate, rewrite, or delete older checkpoint directories.
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
runtime identities, so checkpoints created under one state are not reused by
either of the others.
Checkpoint state is confined below an identity-specific path:
~~~
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest-prefix>/<pipeline-digest-prefix>/<identity-digest-prefix>/
~~~
### Selective Recompute
[**--recompute-step**](cli.md#run) requires both **--resume** and enabled
checkpoint recording. It forces the selected ordered step and every lane that
depends on it through generated artifact references. Unrelated lanes remain
eligible for reuse.
For an earlier producer required by a forced consumer, Notarius requires a
compatible accepted normalized artifact. It validates that artifact before
hydrating it and does not silently rerun the producer. If that state is
missing, rejected, corrupt, non-canonical, or incompatible, the run stops
before its dependent starts. Rerun the required producer deliberately instead
of copying or editing checkpoint files.
## Checkpoint Decisions And Recovery
Checkpoint events classify work as **executed**, **reused**,
**forced_recompute**, or **dependency_invalidated**. Their stable reason codes
are written to run diagnostics and provenance. Use the code, not a copied
error message, to decide what to repair.
| Reason code | Recovery meaning |
| --- | --- |
| **loading_disabled** | This invocation did not permit checkpoint loading. |
| **checkpoint_missing**, **checkpoint_path_invalid**, **checkpoint_read_failed**, **checkpoint_decode_failed** | The stored checkpoint could not be located or read safely; normal resume work can execute again. |
| **workspace_schema_incompatible**, **identity_mismatch**, **stage_mismatch**, **step_mismatch**, **lane_mismatch**, **module_mismatch** | Stored state belongs to a different compatible scope or identity; allow a fresh run to create new state. |
| **status_not_reusable** | The recorded operation did not end in reusable state. |
| **dependency_mismatch** | A dependency changed; dependent work is invalidated rather than reused. |
| **artifact_payload_invalid**, **artifact_digest_mismatch**, **artifact_codec_incompatible**, **artifact_not_canonical** | A stored artifact cannot safely be hydrated; rerun the producer instead of modifying the cache. |
| **checkpoint_reused** | A normal checkpoint passed compatibility checks. |
| **accepted_artifact_reused** | A required predecessor's accepted normalized artifact was safely hydrated. |
| **recompute_step** | Selective recomputation deliberately forced this work. |
Reason detail is bounded code-owned text. It is diagnostic information, not a
path-discovery or data-recovery mechanism, and does not contain checkpoint,
source, reference, credential, or environment content.
## Debug Bundles
Only a [debug-enabled run](cli.md#run) creates a bundle:
~~~
<debug-root>/<run-id>/
summary/
trace/
~~~
The summary contains redacted invocation and resolution information plus run,
warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
contains allowlisted application diagnostic records and can include source or
derived application data. Neither surface is a cache input. Do not treat a
debug bundle as safe to share merely because its configuration summary is
redacted. Invocation metadata omits reasoning effort when it is inherited,
records the replacement value when one is supplied, and records an empty value
when inherited reasoning was explicitly cleared.
Notarius never creates debug state without an explicit request and never
automatically deletes a requested bundle. If allocation succeeds, the command
reports its path on both success and later failure. A summary, trace, or
terminal-report persistence failure fails the command while preserving any
already-written diagnostic data for inspection.
## Cleanup
Cleanup is manual and destructive. First inspect the exact leaf directory,
then remove only that leaf; do not use a glob or a parent root as the target.
~~~
rm -rf -- /srv/notarius/output/run-1721300000000000000-0123456789abcdef0123456789abcdef
rm -rf -- /srv/notarius/chunk-plans/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
rm -rf -- /srv/notarius/checkpoints/example/seriatim-0123456789abcdef/0123456789abcdef/0123456789abcdef
rm -rf -- /srv/notarius/debug/run-1721300000000000000-0123456789abcdef0123456789abcdef
~~~
Deleting output permanently removes user data. Deleting chunk plans or
checkpoints is recoverable but may repeat expensive provider or pipeline work.
Deleting a debug bundle removes troubleshooting evidence and a retained copy of
application data. Notarius has no cache garbage collector, rollback operation,
or automatic cleanup command.
## Operational Limits
Provider execution settings and the generation timeout come from the selected
PromptKit profile. The invocation-only **--reasoning-effort** and
**--clear-reasoning-effort** controls may replace or clear that profile setting
for all LLM-backed calls in one run without changing the profile. PromptKit
v0.5.0 does not add a provider retry loop. Notarius binding retries rerun the
complete module operation and validation chain as defined by
[module bindings](config.md#module-bindings-and-validators).
Timeouts are layered. Caller cancellation is the outer authority. A positive
effective generation timeout adds an inner request deadline, while zero
disables only that generation deadline. The HTTP client timeout remains a
transport-wide cap. Notarius does not add another timeout around PromptKit.
The pinned upstream boundary and profile-format links are in
[PromptKit Integration](integrations/pkg-promptkit.md).
Concurrency has two independent layers. Notarius **total_llm** is the
application-wide provider-call limit shared by all backends, modules, retries,
and validators. PromptKit may impose a narrower admission limit for the
selected backend. The effective active-generation bound is the intersection of
both limits and can therefore be lower than **total_llm**. Built-in OpenRouter
profiles use PromptKit's upstream backend limit; endpoint-only profiles have no
PromptKit backend limit and remain bounded by Notarius. For the configured
local backend, a zero **concurrency_limit** leaves only the Notarius scheduler
as a call limit. A positive value makes the effective active local-generation
bound the smaller of **total_llm** and that local limit.
For a positive local limit, PromptKit owns its default waiting capacity and
admission behavior. When a PromptKit backend has admitted all active and queued
work, a new call fails as capacity exhaustion before generation. The adapter
maps that failure to Notarius's existing provider-neutral capacity error and
does not retry it. The calling stage's configured retry policy applies
normally, and the run fails if those attempts are exhausted. Caller
cancellation remains authoritative. Configuration contracts are documented
under [PromptKit profiles](config.md#promptkit-profiles) and
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
limits and actual provider-call limits are independent. Notarius writes local
filesystem state only; remote storage, archival, and retention automation are
outside the implemented CLI.

382
docs/policy/architecture.md
View File

@@ -1,237 +1,253 @@
# Architecture
This document defines the intended high-level architecture of Notarius and the
invariants that changes must preserve. Implemented component details belong in
[Internal Overview](../internal/overview.md) and its linked documents. The
reasoning behind significant architectural choices belongs in
[ADRs](../adr/).
This document defines the development principles for Notarius. It is
inward-facing: developers and LLM coding agents should use it to preserve the
project's shape, boundaries, and invariants as the code evolves.
## System Shape
## Project Shape
Notarius is a small, dependency-light Go application for extracting structured
artifacts from source material. It is a general extraction platform whose
source formats, extraction domains, validation policies, LLM providers, and
output formats are isolated behind explicit boundaries.
Notarius is a small, explicit, dependency-light Go application for extracting
structured artifacts from source material using modular extractors.
The application has one fixed pipeline shape:
The application should be contract-first but not abstraction-heavy. Add
interfaces and extension points when they protect a real boundary:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
- external source formats;
- extractor modules;
- validators;
- LLM providers and runtime plumbing;
- output schemas and embedded assets.
Pipelines are configured compositions of this shape. They are not arbitrary
DAGs or a general workflow language. Every stage remains explicit; general
chunking, merging, or normalization behavior must not be hidden inside an
extractor.
Avoid abstractions that only anticipate hypothetical complexity. Prefer narrow
contracts that can be exercised by tests and real modules.
Input and chunking are pipeline-wide. Each selected artifact lane owns its
extract, merge, and normalize stages, and the output stage aggregates the run's
lane outcomes.
## Core Invariants
Notarius is contract-first without being abstraction-heavy. Interfaces and
extension points should protect demonstrated boundaries. New abstraction is not
itself an architectural goal.
The core framework must remain source-agnostic and domain-agnostic.
## Layers And Dependency Direction
Source-format details belong in input adapters. Transcript-specific concepts
such as segments, speakers, timestamps, and transcript schemas must not spread
into runner, extractor, or validator framework code.
The application boundary is the composition root and may depend on concrete
implementations. Domain-neutral model and framework layers provide reusable
policy, contracts, and orchestration. Concrete input, pipeline, output, and
validation extensions depend inward on those generic layers.
Extraction-domain details belong in extractor packages. D&D-specific concepts
such as spells, NPCs, items, combat turns, and encounters must not spread into
core source, runner, or LLM framework packages.
Generic layers must not depend on production extensions. Concrete extensions
must not compose the application or take ownership of process behavior. The
current packages implementing these layers are inventoried in
[Internal Overview](../internal/overview.md).
Extracted facts should be grounded with source references. Source references
should point to generic source units, not to transcript-only structures.
The following dependency boundaries are mandatory:
## Dependency Policy
- extractors and validators do not depend on concrete input adapters;
- provider-specific types do not cross the LLM runtime boundary;
- external dependency types do not leak across internal package boundaries
unless that dependency is the package's explicit contract.
Prefer the Go standard library where practical.
Shared helpers may support demonstrated common needs, but must not move
source-format or extraction-domain knowledge into generic framework packages.
External dependencies require a clear correctness, security, interoperability,
or complexity benefit.
Use external dependencies only when justified by correctness, security,
interoperability, or substantial complexity reduction. Good reasons include
widely used file formats, complex validation behavior, or secure transport
handling.
## Source And Domain Boundaries
Avoid dependencies for small conveniences. Do not let external dependency types
leak across internal package boundaries unless the dependency is itself the
explicit public contract of that package.
Input modules translate external source formats into the generic source model.
Format-specific schemas, fields, and validation remain with the input module
and its integration contract.
## Package Layout
Framework stages operate on source documents, source units, and source
references rather than format-specific structures. A source reference identifies
an ordered range of generic source units. Framework code preserves those ranges
and does not merge or rewrite them unless a stage module explicitly owns that
behavior. Every source unit carries a validated self-reference to its containing
document and its own unit ID.
Use this layout unless a change documents a better project-specific reason.
Extract modules own artifact semantics, prompt use, response schemas, and
domain interpretation. Domain-specific concepts remain in the relevant module,
validator, shared domain helper, and artifact contract.
CLI and executable entrypoint:
Typed artifact registrations declare one stable artifact kind and exact Go
type from extraction through merge, normalization, and semantic validation.
Pipeline resolution requires a compatible codec and matching kind-specific
variants before a typed lane can be accepted. Framework-owned erasure remains
private and must report type incompatibility as an error rather than a panic.
- `cmd/notarius`: executable entrypoint.
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
An artifact kind may additionally provide a typed evidence projection that
copies its direct generic source references. Preparation proves that projection
matches the artifact codec's exact Go type before retaining it for an output
policy. The runner reconstructs evidence only from accepted serialized
normalized artifacts, and the output boundary owns any resulting publication.
Generic framework code never infers evidence by inspecting domain JSON or
depends on domain artifact types.
Core deterministic model and policy:
Auxiliary references provide context or disambiguation. They are not source
evidence and must not be converted into source references.
- `internal/core/config`: configuration structs, defaults, loading, precedence, and validation.
- `internal/core/source`: source document, source unit, and source reference types.
- `internal/core/sourcechunking`: deterministic chunking of ordered source units.
- `internal/core/artifacts`: artifact envelope, artifact candidates, rejected artifacts, and manifests.
- `internal/core/diagnostics`: run directories and diagnostics artifact paths.
- `internal/core/reporting`: process reports and report serialization.
- `internal/core/inputcatalog`: known input adapter keys and metadata.
- `internal/core/extractorcatalog`: known extractor keys and metadata.
## Pipeline Composition And Ownership
External source and provider adapters:
Module selection is configuration- and registry-driven. The framework resolves
named pipeline definitions, applies explicit defaults and runtime overrides,
and verifies module availability and capabilities before execution. Structural
pipeline choices must not be scattered through conditionals or hidden behind
ad hoc command flags.
- `internal/adapters/input/<name>`: source-format adapters that parse external input into core source documents.
- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
Resolution validates every selected module and validator option set. A separate
preparation boundary then constructs the complete input, chunk, lane,
validation, and output implementation set in pipeline order. The runner accepts
only that prepared set, so construction and dependency failures occur before
source parsing or any other module operation.
Reusable framework plumbing:
Stage ownership is explicit:
- `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts.
- `internal/framework/runner`: orchestration across adapters, extractors, validators, and artifact output.
- `internal/framework/extraction`: shared extraction helper code.
- `internal/framework/validators`: shared validator runtime behavior and decision checks.
- `internal/framework/llm`: LLM runtime, scheduling, and provider adapters.
- `internal/framework/responseschema`: embedded structured-output schema registry.
- `internal/framework/structuredoutput`: structured-output parsing and malformed-response handling.
- `internal/framework/promptcontext`: source-document prompt rendering helpers.
- `internal/framework/warnings`: shared warning records.
- input modules convert external material into the generic source model;
- chunk modules partition source material for extraction;
- extract modules produce domain artifacts from chunks;
- merge modules combine accepted extraction outputs;
- normalize modules reconcile merged output;
- output modules encode accepted results and run outcomes into logical files.
Domain implementations:
Chunk modules produce source-addressed chunk plans rather than materialized
chunks. The framework validates and materializes those plans into the generic
chunk representation before chunk validation and lane execution. Plan reuse is
therefore independent of the configured pipeline, module options, references,
lanes, validators, and LLM profile: the canonical source digest selects the
plan, while the current run still applies its configured chunk validators to
the materialized chunks.
- `internal/extractors/<domain>/<extractor>`: domain-specific extractor packages.
- `internal/validators/<validator>`: built-in validator implementations.
- `internal/prompts`: embedded prompt assets and prompt metadata registry.
The framework owns orchestration and handoff provenance. Modules return logical
results and warnings; they do not own CLI reporting, physical output, cache, or
debug roots, durable file placement, or checkpoint and debug lifecycle.
Package-private implementation constants may live near the package that owns
them, preferably in `constants.go` when useful.
After pipeline-wide chunking, extraction uses bounded framework concurrency.
One run-wide worker pool receives chunk-scoped lane jobs in deterministic
chunk-first, lane-second order. A lane may begin its merge and normalize
continuation only after all of its extract jobs are terminal; that continuation
remains serial within the lane, while bounded continuations for different lanes
may overlap. The framework must not create unbounded goroutines per lane or
chunk.
## Input Adapters
Completion timing does not choose public ordering or errors. The coordinator
orders accepted artifacts, warnings, rejections, checkpoint events, and
framework errors by stable pipeline scope. Rejections do not cancel unrelated
work. A framework error cancels derived work, prevents undispatched work from
starting, waits for started work, and prevents output encoding.
Use a hexagonal architecture style for source input.
## Validation
Input adapters translate external source formats into the core source model.
Adapters may know about external schema details, source-specific metadata, and
format-specific validation rules. They should not own extraction-domain
decisions.
Validation is a framework-managed boundary around outputs from chunk, extract,
merge, and normalize stages. Validators receive immutable stage output
and make an explicit whole-output decision: approve, approve with warnings, or
reject.
Other packages should interact with source input through adapter contracts and
core source types. Adapter implementation details and external dependency types
must not leak into framework or extractor packages.
Typed artifact validators receive the domain value directly. Chunk validators
receive source-zone chunks, while serialized validators receive immutable
representation bytes and declared schema metadata. A validator registered for
one target or artifact kind cannot satisfy an incompatible selection.
Adapter metadata may preserve source-specific facts such as transcript speaker,
timestamps, Markdown heading path, page number, or block ID. Framework code may
carry metadata through, but should not require a specific adapter's metadata
shape.
Rejection is a recorded pipeline outcome, not a framework execution error.
Validator execution failures are framework errors. Rejected output does not
advance to the next stage.
## Extractors
Default validator chains are production composition policy and are registered
centrally by stage and module. Configuration may replace a stage-local default,
including with an explicitly empty chain. Configured validator order is
authoritative; the framework must not silently reorder it.
Extractors are independent modules that produce one kind of structured artifact.
Each extractor package owns:
## LLM Boundary
- its artifact semantics;
- its prompt usage;
- its structured response schema selection;
- its validator chain;
- any domain-specific mapping or interpretation.
Modules and validators use transport-neutral structured completion contracts.
Provider request and response types, authentication, transport behavior, and
provider error adaptation remain inside the LLM runtime.
Extractors should depend on framework contracts and core source/artifact types.
They should not depend on concrete input adapter packages.
The caller of the LLM owns prompt selection, prompt inputs, response schema,
and interpretation of structured output. Provider adapters do not own source-
or domain-specific prompt logic.
The runner should be able to compose, skip, resume, or run individual extractors
when their prerequisites are satisfied. Ordering should be explicit through
configuration, a default sequence, or documented orchestration rules.
LLM calls and other external operations accept cancellation and respect
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
individual modules.
Extractor selection must go through a registry or equivalent mechanism rather
than scattered conditionals.
The application-wide LLM scheduler bounds actual provider calls independently
of framework worker limits. Every LLM-backed module, retry, and validator uses
the single injected scheduled client, including work performed by overlapping
lanes. Provider runtime adapters may enforce a narrower backend-specific limit
beneath this mandatory application-wide scheduler.
## Validators
## Configuration And Provenance
Validators should be independently testable and composable.
Configuration loading, precedence, defaults, environment overrides, redaction,
and validation are centralized. Named pipeline definitions make structural
composition explicit and discoverable. Operational overrides are permitted
when they do not obscure the configured pipeline structure.
Deterministic validators should run before LLM-backed validators when both are
present. Validator decision semantics should be explicit: each candidate
artifact should receive exactly one decision from each validator that evaluates
it.
Run preparation fails before stage execution when statically discoverable
modules, capabilities, reference bindings, or explicitly selected profiles are
invalid or incompatible.
Shared validator runtime mechanics belong under `internal/framework/validators`.
Concrete validator behavior belongs under `internal/validators/<validator>`.
Run manifests record enough resolved pipeline, module, source, reference, and
LLM provenance to make a run auditable after configuration changes. Manifests
record identities and summaries rather than secret or large payload content.
## LLM Runtime
## State, Output, And Safety
LLM provider details belong behind transport-neutral framework contracts.
Notarius exposes three filesystem surfaces with independent roots and
lifecycle:
Provider-specific HTTP request and response types should stay inside the LLM
runtime package. Prompt construction should stay in extractors, validators, or
shared prompt-context helpers; provider adapters should not own domain prompt
logic.
- output is durable user data; output modules define logical files and the CLI
owns their placement;
- cache is reconstructible state, with separate chunk-plan and checkpoint
families; and
- debug is explicitly requested inspection data, combining a redacted summary
with a detailed trace.
Errors, diagnostics, reports, and redacted config must not expose secrets.
Chunk plans are keyed only by canonical source digest. Configured checkpoint
recording is independent of checkpoint reuse; checkpoints are loaded only for
an invocation that explicitly requests resume. Debug is never a cache input and
is never created without an explicit request. Pipeline modules receive
collaborator interfaces and never physical roots.
## Configuration
Writes are atomic where practical. Paths for writes, moves, overwrites, and
deletion must be narrow and explicit. Notarius never automatically deletes
output or requested debug bundles; cache cleanup is explicit and recoverable.
Centralize configuration loading, processing, precedence, defaults, and
validation in `internal/core/config`.
Secrets must not appear in errors, logs, output, cache, debug summaries,
traces, manifests, documentation, examples, or redacted configuration. Debug
collection is allowlisted to application-owned payloads and must not capture
unrelated process environment values or filesystem content. Trace data may
contain application data and therefore inherits its sensitivity; operators own
access controls and retention. Physical layout and operation are defined in
[Operations](../operations.md).
The goal is to make configuration discoverable and avoid implicit or hidden
operational values. User-visible defaults and cross-package operational defaults
should be defined in config code.
## Architectural Non-Goals
Unless documented otherwise, precedence is:
Notarius does not aim to provide:
1. CLI flags
2. environment variables
3. configuration file
4. built-in defaults
- an arbitrary workflow graph or general workflow language;
- source-format or extraction-domain behavior in generic framework packages;
- provider-specific contracts exposed to modules;
- structural pipeline composition through ad hoc CLI flags;
- implicit cross-stage behavior that bypasses the fixed pipeline;
- abstractions introduced solely for hypothetical future complexity.
Prefer YAML configuration unless the project has a strong reason to use another
format. Config files should be discoverable at
`/usr/local/etc/notarius/config.yml`, with a CLI override via `--config`.
Configuration files should not contain raw secrets unless the application is
explicitly designed for that. Prefer environment variables or secret files for
secrets.
Adapter-specific and extractor-specific configuration should remain grouped by
the adapter or extractor that owns it.
## Embedded Assets
Store embedded JSON schemas, Markdown prompts, templates, and similar assets as
separate files, not inline string literals, unless there is a strong reason
otherwise.
Embedded prompts and response schemas should have stable IDs, versions, source
metadata, and hashes suitable for diagnostics and run manifests.
## Errors and Logging
Errors should be actionable and preserve context. Wrap errors with operation and
path/resource context. CLI code should convert internal errors into concise
user-facing messages.
Errors and logs must not expose secrets.
Use structured logging where practical. Logs should describe operations, paths,
external calls, retries, and failure causes, but should not include large source
or artifact payloads by default.
## Context, Timeouts, and Cancellation
Long-running operations should accept `context.Context`. External calls,
subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
workflows should respect cancellation and timeouts.
## State, Files, and Safety
If the application writes durable state, writes should be atomic where
practical. Multi-step workflows should preserve enough state to support
inspection, retry, or resume after failure.
Code that deletes, moves, or overwrites files must use narrow, explicit paths.
Avoid broad parent-directory operations. Cleanup that can cause data loss must
be opt-in.
## Testing
Core logic should be testable without real external services. Use fakes,
fixtures, or local test doubles for adapters, extractors, validators, and LLM
clients where practical.
Contract-first work should include fake implementations that prove interfaces
compose before real adapters or extractors depend on them.
Config examples should be load-tested once config files exist. Important CLI
workflows should have parser or command tests. Adapter, extractor, and validator
contracts should have focused tests that do not require running the full
application unless end-to-end coverage is intentional.
## Documentation
Documentation should follow the project documentation policy. Keep user docs
focused on implemented behavior. Put future, planned, or aspirational work only
under `docs/roadmap/`.
Core documentation should use generic terms such as source document, source
unit, source reference, input adapter, extractor, artifact, validator, and run
manifest.
Source-format details belong in adapter or integration docs. Domain-specific
extraction details belong in extractor or artifact docs.
When changing architecture, config, CLI behavior, adapters, extractor contracts,
validator contracts, LLM runtime behavior, or artifact schemas, update the
relevant docs and examples in the same change.

512
docs/policy/documentation.md
View File

@@ -1,144 +1,444 @@
# Documentation Policy
# Go Project Documentation Policy
## Purpose
This policy assigns each documentation topic to one canonical owner. Its goal is
to keep Notarius documentation accurate, concise, discoverable, and resistant
to drift for users, operators, developers, integrators, and LLM coding agents.
Project documentation must help five audiences:
1. users who need to run the application;
2. administrators/operators who need to configure and operate it;
3. developers who need to understand and change it safely;
4. LLM coding agents that need clear scope, boundaries, and invariants;
5. developers and LLM coding agents integrating this project from another codebase.
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
## Core Rules
### One Canonical Owner
### 1. Keep docs concise
Each authoritative fact belongs in one document. A non-owning document may give
a short, stable summary for orientation, but it must link to the canonical owner
instead of repeating volatile details.
Each document should cover a defined scope and only the essentials for that scope.
Volatile details include commands, flags, configuration fields and defaults,
module keys, schemas, file names, paths, status codes, retry behavior, and
runtime guarantees. If readers could reasonably treat a statement as a
contract, maintain it only in the owning document.
Avoid:
- long background explanations;
- repeated reference material;
- implementation detail in user-facing docs;
- aspirational language outside roadmap docs;
- verbose examples where one minimal example is clearer.
### Current And Future Behavior
### 2. Document only implemented behavior outside roadmap files
Outside `docs/roadmap/`, documentation describes implemented behavior only.
Partial features may be described only to their implemented boundary.
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
ADRs are the narrow exception: an ADR may record an accepted architectural
decision before implementation, but acceptance must not be presented as proof
that the behavior exists. The roadmap owns implementation status and sequencing
until the decision is implemented. Current architecture, user, operator,
integration, and internal documentation are updated when the behavior lands.
- `docs/roadmap/`
### Audience And Detail
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
Write for the document's stated audience and include only the detail needed for
its owned topic. User and operator docs should not expose implementation detail.
Developer docs should link to user-facing and external contracts rather than
restate them.
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
### Examples
### 3. Use canonical homes
Complete copyable files belong in `examples/`. Documentation may use the
smallest illustrative snippet needed to explain its owned topic, but should link
to maintained examples instead of embedding a second complete copy.
Each type of information should have one canonical location.
Examples must be valid, secret-free, and tested where practical. Commands and
configuration used in documentation should match the application.
Canonical homes:
### Security And Privacy
- project purpose and quickstart: `README.md`
- development principles: `docs/policy/architecture.md`
- public HTTP API reference: `docs/api.md`
- configuration reference: `docs/config.md`
- CLI reference: `docs/cli.md`
- operations and recovery: `docs/operations.md`
- troubleshooting: `docs/troubleshooting.md`
- public API/package consumer guidance: `docs/consumers/`
- implemented internals: `docs/internal/`
- external protocol, service, and file-format contracts: `docs/integrations/`
- future work: `docs/roadmap/`
- contributor workflow: `docs/policy/development.md`
- copyable examples: `examples/`
Documentation and examples must not contain real credentials, private keys,
private environment dumps, sensitive source material, or private infrastructure
details unless intentionally public. Document secret-handling mechanisms, not
secret values.
Other files should summarize briefly and link to the canonical source.
## Canonical Ownership
### 4. Keep examples real
| Topic | Canonical owner | Owned content | Content owned elsewhere |
| --- | --- | --- | --- |
| Product orientation and minimal end-to-end quickstart | `README.md` | What Notarius is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, detailed change recipes. |
| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, and exit codes. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, module implementation details. |
| Configuration contract | `docs/config.md` | Discovery and precedence, file schema, fields, defaults, environment overrides, validation rules, and user-selectable module or validator keys. | Complete example files, CLI syntax, runtime state lifecycle, module implementation details. |
| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and state layout, output, cache, and debug handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical output schemas, implementation mechanics. |
| Public HTTP contract, if introduced | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
| Consumer guidance, if a public package or API is introduced | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. |
| External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical output bundle paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
| Implemented component inventory | `docs/internal/overview.md` | Current packages and components, their implemented responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
| Internal component behavior | Other files under `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |
| Architectural decision history | `docs/adr/` | Significant decisions, context, alternatives, rationale, consequences, and supersession history. | Current behavior reference, implementation status, task sequencing. |
| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, deferred, or rejected work; implementation status; sequencing; and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
| Complete copyable artifacts | `examples/` | Maintained configuration, inputs, and other files intended to be copied or run. | Field-by-field reference, command reference, prose explanation. |
Examples should be valid, maintained, and free of secrets.
Documents that do not exist are required only when the corresponding interface
or responsibility exists. Do not create placeholder API, consumer, integration,
or operations documents for behavior the application does not have.
Where practical:
- example configs should load successfully;
- example commands should match real CLI syntax;
- important examples should be covered by tests.
## Boundary Rules
## Documentation Profiles
### Orientation
All projects require:
The README owns product orientation. The developer guide routes contributors.
Architecture owns normative structure. Internal overview owns the current
concrete component map. These documents may link to one another but should not
maintain parallel package or behavior descriptions.
- `README.md`
- `docs/policy/architecture.md`
### Commands, Configuration, And Operations
Additional docs depend on the project.
CLI documentation answers how to invoke the application. Configuration
documentation answers what settings mean. Operations answers what happens to
runtime state and how to operate or recover the application. When a workflow
crosses these topics, choose the document that owns the task and link to the
other contracts.
### Small library
### Contracts And Implementation
Recommended:
- `docs/policy/development.md`, if contributor conventions are non-obvious
Integration and API documents define externally observable shapes and
semantics. Internal documents explain how Notarius implements or consumes those
contracts. Internal docs may name a field, file, or protocol to identify a
dependency, but must link to its canonical contract for the definition.
### Simple CLI
### Security Topics
Required:
- `docs/cli.md`
This policy owns what documentation and examples may contain. Architecture owns
application security invariants. Configuration owns credential-supply
mechanisms. Operations owns permissions and handling of sensitive runtime
artifacts. Internal docs own implementation mechanisms only.
Recommended:
- `docs/policy/development.md`
## Architecture Decision Records
### Config-driven CLI
Use sequentially numbered ADR filenames such as
`0001-record-architecture-decisions.md`. Follow the lightweight Nygard format:
Required:
- `docs/cli.md`
- `docs/config.md`
1. title;
2. status;
3. date;
4. context;
5. decision;
6. alternatives considered;
7. consequences.
Recommended:
- `examples/`
- `docs/policy/development.md`
Treat the decision content of an accepted ADR as immutable. When a decision
changes, create a new ADR and update the earlier ADR's status to superseded.
Rejected architectural alternatives belong in the ADR; rejected product ideas
belong in the roadmap.
### Stateful or operator-facing application
## Maintenance
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
When behavior changes, update its canonical owner in the same change. If
ownership moves, remove the old definition and replace it with a link where
navigation remains useful.
Recommended:
- `docs/troubleshooting.md`
- `examples/`
- `docs/policy/development.md`
Before completing documentation work:
### Modular, service-oriented, or orchestration application
- verify affected behavior and examples;
- check commands, flags, fields, defaults, schemas, and paths against their
implementation;
- keep unimplemented behavior in the roadmap, subject to the ADR exception;
- remove stale references and validate links;
- confirm that non-owning documents summarize and link rather than redefine;
- confirm that no secrets or sensitive private data were added.
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
- `docs/internal/`
- `docs/policy/development.md`
Recommended:
- `docs/troubleshooting.md`
- validated examples under `examples/`
### Public HTTP API service
Required:
- `docs/api.md`
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
- `docs/internal/`
- `docs/policy/development.md`
Recommended:
- `docs/troubleshooting.md`
- `docs/consumers/`, for task-oriented client integration guides
- `docs/integrations/`, for upstream/downstream service contracts
- validated examples under `examples/`
### Project with public packages or consumer APIs
Required:
- `docs/consumers/api.md`
- one `docs/consumers/pkg-<name>.md` file per public package, if public packages exist
Recommended:
- copyable consumer examples under `examples/`, if practical
## Required Documents
### README.md
**Audience:** users, administrators, operators
The README is the outward-facing project orientation page.
It should include, in order:
1. concise description;
2. elevator pitch;
3. shortest useful command or usage example;
4. links to targeted docs.
The README should be short. It is not a manual.
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
### docs/policy/architecture.md
**Audience:** developers, LLM coding agents
`docs/policy/architecture.md` is required for every project.
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
It should include:
- project shape;
- core design principles;
- package and boundary philosophy;
- state/persistence philosophy, if applicable;
- external integration philosophy, if applicable;
- error-handling and logging principles;
- testing expectations;
- documentation expectations;
- architectural invariants;
- explicit non-goals, if useful.
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
### docs/api.md
**Audience:** external HTTP API consumers, developers, LLM coding agents integrating by HTTP
Required for projects whose primary public interface is HTTP.
`docs/api.md` is the canonical public HTTP API contract. It should be normative for external consumers and should not be duplicated by README, operations docs, consumer guides, or integration docs.
It should include:
1. base URL conventions;
2. authentication and authorization behavior, if implemented;
3. response envelope;
4. supported media types and content negotiation behavior;
5. shared query parameters;
6. endpoint reference grouped by route family;
7. request parameters and validation rules;
8. response fields, units, nullability, and optionality;
9. error response shape and status codes;
10. pagination, caching, rate-limit, idempotency, and retry behavior, if implemented;
11. compact request and response examples.
It must document only implemented endpoints and behavior. Planned endpoints, proposed fields, future filters, and experimental response shapes belong only under `docs/roadmap/`.
For HTTP API projects, `docs/consumers/` may provide task-oriented client integration guides, but those guides should link to `docs/api.md` for the authoritative endpoint contract.
### docs/policy/development.md
**Audience:** developers, LLM coding agents
Required for projects maintained by humans and LLM coding agents.
It should include:
- repository layout;
- build/test commands;
- coding conventions;
- dependency policy;
- how to add config fields;
- how to add CLI flags;
- how to add modules or adapters, if applicable;
- how to update examples;
- documentation update expectations.
### docs/config.md
**Audience:** administrators, operators, advanced users
Required for applications with configuration files.
It should include, in order:
1. config file locations and discovery precedence;
2. minimal working config;
3. production-oriented config;
4. full configuration reference;
5. secrets handling, if applicable;
6. links to maintained examples.
The full configuration reference should be canonical.
### docs/cli.md
**Audience:** users, administrators, operators
Required for CLI applications.
It should include, in order:
1. shortest useful command;
2. command overview;
3. complete flag reference;
4. common workflows;
5. diagnostic or recovery commands, if applicable.
Explain when commands are useful, not just their syntax.
### docs/operations.md
**Audience:** administrators, operators
Required for applications that maintain state, support resume behavior, run multi-step workflows, write durable artifacts, use remote storage, or require recovery procedures.
It should cover:
- normal workflow;
- filesystem layout;
- remote storage layout, if applicable;
- logs and manifests;
- resume/retry behavior;
- cleanup behavior;
- archive/backup behavior;
- safe recovery procedures;
- operational caveats.
### docs/troubleshooting.md
**Audience:** administrators, operators
Recommended once recurring failure modes exist.
Each entry should include:
- symptom;
- likely cause;
- diagnostic command or inspection step;
- safe fix;
- relevant links.
### docs/consumers/
**Audience:** developers and LLM coding agents integrating this project from another codebase
Required for projects with public packages, SDKs, client APIs, plugin APIs, or other application-facing integration surfaces.
This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
For projects whose public API is HTTP, `docs/consumers/` is not required, and it should not duplicate the endpoint reference in `docs/api.md`. If present, it may provide practical integration workflows, client-specific examples, or migration notes that link back to `docs/api.md`.
`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
1. intended consumer audience and use cases;
2. required inputs supplied by operators or deployment configuration;
3. recommended public package or API workflow;
4. minimal copyable example;
5. consumer responsibilities and boundaries;
6. retry, idempotency, or status behavior, if applicable;
7. links to package-specific docs and canonical integration contracts.
Package-specific docs should be named `pkg-<name>.md` and should include:
1. import path;
2. intended use cases;
3. primary types and functions needed by consumers;
4. minimal examples;
5. validation, error, retry, and boundary behavior;
6. links to canonical file-format or wire-protocol contracts.
### docs/internal/
**Audience:** developers, LLM coding agents
Required for modular, service-oriented, or orchestration projects.
This directory describes implemented internal components. It is not the roadmap.
Use one file per major component where useful.
Each component doc should include:
1. purpose;
2. inputs and outputs;
3. boundaries;
4. config fields used;
5. external adapters used;
6. state or manifest behavior, if applicable;
7. skip/resume behavior, if applicable;
8. failure behavior;
9. tests to inspect before changing;
10. architectural invariants.
### docs/roadmap/
**Audience:** maintainers, developers, LLM coding agents
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
Roadmap docs should clearly distinguish:
- proposed work;
- accepted plans;
- deferred ideas;
- rejected ideas;
- implementation prompts or task breakdowns, if useful.
Roadmap docs should not be confused with current behavior.
### docs/integrations/
**Audience:** developers, LLM coding agents
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
For public HTTP API services, `docs/integrations/` should document upstream, downstream, storage, protocol, or runtime contracts that the service depends on or bridges. It should not become a second copy of the public HTTP endpoint reference; that belongs in `docs/api.md`.
Use one file per integration where useful.
## Examples Directory
Projects with non-trivial configuration or workflows should include `examples/`.
Useful examples include:
- minimal working config;
- production-oriented config;
- full annotated config;
- local development config;
- remote/object-storage config;
- minimal session/input file.
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
## Security and Privacy
Docs and examples must not include:
- real API keys;
- tokens;
- passwords;
- private keys;
- private environment dumps;
- sensitive user data;
- raw private transcripts;
- private infrastructure details unless intentionally public.
Document secret-handling mechanisms, not actual secret values.
## Maintenance Rules
When docs change, verify the affected behavior.
Where practical:
- load example config files in tests;
- test CLI examples or command parser behavior;
- validate documented flags against real flags;
- remove stale references;
- update links after renames;
- keep roadmap content out of non-roadmap docs.
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
Documentation is complete only when it matches the current code.
## Documentation Change Checklist
Before merging documentation changes, verify:
- README is concise and orientation-focused.
- `docs/policy/architecture.md` describes development principles.
- `docs/api.md` is the canonical HTTP contract for HTTP API services.
- Future work appears only under `docs/roadmap/`.
- User-facing docs avoid unnecessary internals.
- Consumer-facing docs explain public APIs without duplicating HTTP endpoint or integration contracts.
- Developer-facing docs preserve boundaries and invariants.
- Config examples match the schema.
- CLI examples match real commands and flags.
- Defaults appear in the canonical config reference.
- No secrets or private data are included.
- Links are accurate.

296
docs/policy/testing.md
View File

@@ -1,296 +0,0 @@
# Testing Policy
## Purpose
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
We do not optimize for test count, line coverage, exhaustive isolation, or the fewest possible tests. We optimize for sufficient confidence in important behavior while imposing as little unnecessary friction as possible on future development.
## Every test has a cost
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
A test must be:
- written and reviewed;
- understood by future maintainers and coding agents;
- executed in local and CI workflows;
- diagnosed when it fails;
- updated when legitimate behavior changes;
- maintained as fixtures, APIs, and dependencies evolve; and
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
A test is warranted only when the confidence it provides justifies these costs.
Apply this cost-benefit analysis at two levels:
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
## Default testing style
Use a **classical/Detroit-style** approach:
- Test observable behavior, resulting state, contracts, and invariants.
- Use real internal collaborators when they are fast and deterministic.
- Use fakes, stubs, or mocks primarily at expensive, nondeterministic, destructive, or external boundaries.
- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.
Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.
## Test execution requirements
Tests in the default suite must be deterministic, offline, and independent of real credentials. They must not invoke paid APIs or depend on mutable external services. Tests that require live infrastructure must be explicitly opt-in and clearly separated from the default suite.
Control clocks, randomness, environment variables, and other process-global or machine-specific state when they affect behavior. Tests should be safe to run repeatedly and alongside other tests without depending on execution order or state left by an earlier test.
## What deserves tests
Prioritize tests for:
1. Public and package-level contracts.
2. Domain rules and important invariants.
3. Boundary conditions and malformed input.
4. Failure handling, cancellation, retries, recovery, and partial success.
5. Serialization, schemas, compatibility, and round trips.
6. Previously observed or plausible regressions.
7. Representative integration and end-to-end workflows.
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
## Choose the right test boundary
Test through the narrowest stable boundary that expresses the behavior clearly.
This is often the package API, but it may instead be:
- a smaller pure function when dense domain logic is most clearly isolated there;
- a package-level operation when several internal collaborators jointly produce the behavior; or
- a larger integration boundary when correctness emerges from interaction with a real dependency.
Do not force all behavior through oversized end-to-end tests. Do not test every private helper merely because it exists. Choose the boundary that gives durable confidence with the least incidental coupling.
## Test behavior, not implementation
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
Before adding or retaining a test, ask:
> What realistic defect would this test catch?
A test is suspect when its main purpose is to detect that someone:
- changed an internal constant;
- renamed or split a private helper;
- reordered equivalent internal operations;
- changed incidental formatting;
- replaced one correct algorithm with another; or
- refactored internal object structure without changing behavior.
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
## Expected effects of different changes
Use the following expectations when evaluating test failures and test maintenance:
| Change | Expected effect on tests |
|---|---|
| Internal refactor that preserves behavior | Existing tests should normally remain unchanged and continue to pass. |
| Change to an internal default with no contractual significance | Behavioral tests should normally remain unchanged; tests should derive expectations from configuration or relationships rather than duplicate the old value. |
| Intentional change to public behavior, policy, schema, or compatibility guarantees | The relevant tests should be reviewed and changed deliberately. |
| Accidental violation of a contract or invariant | Tests should fail; fix the production code rather than rewriting the tests to accept the defect. |
A test failing is not the same as a test needing to be edited. Many tests may correctly fail because of one production defect. The maintenance smell is a correct internal change that requires unrelated expectation updates throughout the suite.
## Separate mechanism from policy
Configurable thresholds and defaults must not be duplicated throughout the test suite.
For example, do not encode an internal concurrency limit indirectly:
```go
// Production policy:
const maxConcurrency = 4
// Brittle test:
err := startProcesses(5)
require.Error(t, err)
```
Instead, test the mechanism relationally:
```go
const limit = 2
runner := NewRunner(limit)
require.NoError(t, runner.Start(limit))
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
```
The test should prove:
- the configured limit is accepted; and
- one beyond the configured limit is rejected.
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
For concurrency limits, test both kinds of behavior when relevant:
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
## Avoid semantic duplication across layers
Each behavior should have a clear test owner.
- Parser tests own parsing cases.
- Validator tests own validation rules.
- Domain tests own transformations and invariants.
- Adapter tests own external integration behavior.
- Orchestrator tests own coordination and failure propagation.
- CLI tests own argument and configuration mapping.
- End-to-end tests prove that representative assembled workflows work.
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
## Use test doubles deliberately
Choose the least elaborate test double that provides the required control or observation.
As a default:
1. Prefer real collaborators when they are fast and deterministic.
2. Use small in-memory fakes when realistic stateful behavior is helpful.
3. Use stubs when a dependency only needs to provide controlled responses.
4. Use mocks when the interaction itself is contractual.
Mocks are appropriate when the contract includes facts such as:
- a notification is sent exactly once;
- a transaction is committed only after successful writes;
- cancellation reaches a subprocess;
- an expensive API is called no more than once; or
- a security audit event is emitted.
Do not use mocks merely to isolate every object or reproduce the implementation's call graph.
## Go-specific guidance
Use:
- table-driven tests for meaningful behavioral categories and boundaries;
- `t.TempDir()` for real filesystem behavior;
- `httptest.Server` for realistic HTTP interactions;
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
- golden files only when the complete output is intentionally stable;
- integration tests where correctness depends on component interaction; and
- a small number of representative end-to-end tests.
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
At CLI boundaries, prefer exit classifications, structured output, and the smallest stable semantic fragment needed to identify the error. Do not snapshot complete diagnostic wording unless it is contractual.
Golden-file updates must require an explicit local flag. CI must not update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.
Keep tests readable and direct. Test helpers and fixture frameworks must earn their own maintenance cost; do not build elaborate test infrastructure for small or isolated needs.
## Coverage
Coverage is a diagnostic, not a target.
Use it to find untested critical branches and unexpectedly weak packages. Do not write low-value tests solely to increase a percentage, and do not infer test quality from coverage alone.
Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
## Regression tests
A bug fix should normally include a regression test that fails before the fix and passes afterward.
Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant; do not preserve accidental implementation details from the original bug.
Not every historical bug requires a permanent test. If the underlying design has made recurrence impossible, the test has become redundant, or a stronger invariant test now subsumes it, remove or consolidate it.
## Deleting or rewriting tests
Tests are maintained code, not permanent historical artifacts.
Delete or rewrite a test when its maintenance cost exceeds the confidence it provides.
Strong candidates include tests that:
- require updates after harmless internal changes;
- directly assert private constants without protecting a real contract;
- duplicate the same policy across several layers;
- verify mock choreography rather than outcomes;
- snapshot large amounts of incidental output;
- test trivial private helpers already exercised through stable package behavior;
- protect risks already covered more effectively elsewhere;
- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
- no longer correspond to a plausible failure mode.
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
## Reviewing a proposed test
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
1. What realistic defect would it catch?
2. How likely is that defect?
3. How consequential would it be?
4. Is the behavior already protected elsewhere?
5. At which layer should this behavior be owned?
6. Does the test assert a durable contract or an incidental implementation detail?
7. Could the implementation be refactored without changing the behavior and without editing this test?
8. What should cause this test to fail?
9. What legitimate changes should not cause this test to fail?
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
11. Is there a smaller or more direct test that protects the same risk?
Do not add the test when its expected lifetime cost exceeds its expected protective value.
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
## Definition of sufficient
A test suite is sufficient when:
- important contracts and invariants are protected;
- meaningful boundaries and failure modes are exercised;
- realistic and consequential regressions are credibly protected against silent recurrence;
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
- important external boundaries have realistic integration coverage;
- representative complete workflows are tested;
- failures provide useful signal rather than redundant noise;
- legitimate internal changes usually do not require test edits; and
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
The governing rule is:
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.

70
docs/roadmap/1-core-contracts-and-skeleton.md Normal file
View File

@@ -0,0 +1,70 @@
# Checkpoint 1: Core Contracts And Skeleton
## Status
This document describes planned work, not implemented behavior.
## Goal
Define the stable vocabulary and core interfaces that adapters, extractors,
validators, and runners will build against.
This checkpoint should produce a compileable Go repository with a minimal CLI
shell and contract-level tests. It does not need to process real input or
produce useful artifacts.
## Scope
In scope:
- Go module bootstrap;
- executable entrypoint;
- minimal CLI package;
- core source, artifact, manifest, and contract types;
- fake implementation tests proving the interfaces are usable.
Out of scope:
- real input adapters;
- real extractors;
- LLM provider calls;
- prompt or response-schema assets;
- diagnostics run directory;
- production config loading.
## Target End State
The repository should contain a compileable Go application shell and stable core
contract packages:
- `cmd/notarius` provides the executable entrypoint.
- `internal/cli` provides a minimal CLI shell.
- `internal/core/source` defines generic source documents, source units, source
references, and source validation helpers.
- `internal/core/artifacts` defines extractor-neutral artifact candidate,
approved artifact, rejected artifact, and run manifest types.
- `internal/framework/contracts` defines the adapter, extractor, validator, and
structured LLM interfaces used by later checkpoints.
The contracts should be proven with fake implementations in tests. Those tests
should demonstrate composition without real source adapters, real extractors,
LLM provider calls, prompt assets, or diagnostics infrastructure.
Implementation staging belongs in
[`implementation.md`](implementation.md).
## Done Criteria
- `go test ./...` passes.
- `go build ./cmd/notarius` passes.
- Core types and contracts exist in stable package locations.
- Tests prove fake implementations can compose at the type-contract level.
- No real Seriatim, D&D, LLM, or Audita-specific behavior has been added yet.
## Review Questions
- Are the interfaces small enough?
- Are source-format details absent from core packages?
- Are D&D concepts absent from framework and core packages?
- Is the shell compileable without placeholder behavior that will be hard to
unwind?

108
docs/roadmap/2-framework-composition.md Normal file
View File

@@ -0,0 +1,108 @@
# Checkpoint 2: Framework Composition
## Status
This document describes planned work, not implemented behavior.
## Goal
Prove the core contracts compose before adding real adapters, real extractors,
or portable Audita infrastructure.
This checkpoint should produce a minimal runner that can execute fake registered
components from source input to artifact output in tests.
## Scope
In scope:
- input adapter registry;
- extractor registry;
- validator decision model;
- decision-cardinality checks;
- minimal runner;
- fake-component runner tests.
Out of scope:
- real input parsing;
- real LLM calls;
- prompt assets;
- response schema assets;
- diagnostics run directory;
- real D&D artifact schemas.
## Proposed Stages
### Stage 1: Input Adapter Registry
Add a registry for `InputAdapter` constructors or instances.
The registry should:
- reject empty keys;
- reject duplicate registrations;
- return clear errors for unknown keys;
- avoid importing concrete adapter packages from core contracts.
### Stage 2: Extractor Registry
Add a registry for extractor constructors or instances.
The registry should:
- support stable extractor keys;
- support repeated extractor instances if needed later;
- return clear errors for unknown keys;
- avoid domain-specific logic.
### Stage 3: Validator Decisions
Add validator decision types and cardinality checks.
Each validator should return exactly one decision for each candidate artifact it
receives.
Decision fields should include:
- candidate index;
- approved flag;
- reason code;
- message;
- optional diagnostics path.
### Stage 4: Minimal Runner
Add a runner that can:
1. receive a `SourceDocument`;
2. execute configured extractors;
3. validate candidate artifacts;
4. return approved and rejected artifacts.
Keep source chunking optional or stubbed at this checkpoint. The runner may
operate on whole documents only until source-unit chunking is added later.
### Stage 5: Runner Tests With Fakes
Add tests with fake components that prove:
- registered fake extractors run in configured order;
- validators filter candidates deterministically;
- decision cardinality failures are surfaced;
- approved and rejected artifacts are returned in stable order.
## Done Criteria
- `go test ./...` passes.
- Fake adapter/extractor/validator registrations work in tests.
- The runner operates on `SourceDocument`, not transcript-specific structures.
- The runner does not import concrete D&D extractor packages.
## Review Questions
- Does the runner know only about sources, extractors, validators, and artifacts?
- Are registries simple enough to evolve?
- Are validation decisions expressive enough for deterministic and LLM-backed
validators?
- Is any domain-specific behavior creeping into framework packages?

122
docs/roadmap/3-portable-audita-infrastructure.md Normal file
View File

@@ -0,0 +1,122 @@
# Checkpoint 3: Portable Audita Infrastructure
## Status
This document describes planned work, not implemented behavior.
## Goal
Port or adapt reusable Audita infrastructure that directly supports Notarius
contracts while avoiding Audita's transcript-correction model.
This checkpoint should add reusable runtime plumbing, not real extraction
behavior.
## Scope
In scope:
- structured LLM client interface implementation;
- LLM scheduler;
- prompt registry pattern;
- response-schema registry pattern;
- diagnostics run directory pattern;
- minimal config structs and defaults for implemented runtime pieces.
Out of scope:
- correction proposals;
- replacement policies;
- transcript mutation;
- correction ledger terminology;
- Audita module or validator behavior;
- real D&D prompts or schemas unless needed as inert registry tests.
## Proposed Stages
### Stage 1: LLM Runtime
Port or adapt the OpenAI-compatible structured-output client and scheduler.
Keep the contract transport-neutral:
- framework code should depend on a `StructuredLLMClient` interface;
- provider-specific HTTP details should remain in the LLM runtime package;
- errors must redact configured secrets.
### Stage 2: Response Schema Registry
Port or adapt the embedded JSON response-schema registry pattern.
The registry should track:
- schema key;
- schema ID;
- schema version;
- schema name;
- JSON schema content;
- schema hash.
Use placeholder or test schemas if real extractor schemas are not ready.
### Stage 3: Prompt Registry
Port or adapt the embedded prompt registry pattern.
The registry should track:
- prompt ID;
- prompt version;
- prompt source;
- embedded path;
- prompt hash.
Do not add D&D prompt assets here unless the implementation naturally overlaps
with checkpoint 5. Test prompts are acceptable for registry tests.
### Stage 4: Diagnostics Run Directory
Port or adapt the diagnostics run directory pattern.
Initial diagnostics should cover:
- invocation metadata;
- redacted effective config;
- source document artifact;
- run report placeholder;
- error log on failure.
Avoid Audita-specific artifact names such as correction ledger.
### Stage 5: Minimal Runtime Config
Add config structs and defaults only for infrastructure that now exists.
Initial config areas:
- input adapter key;
- extractor keys;
- primary LLM settings;
- validation LLM settings if needed;
- concurrency;
- work directory;
- diagnostics retention.
Config loading can remain minimal unless the implementation needs full file/env
precedence at this checkpoint.
## Done Criteria
- `go test ./...` passes.
- Audita runtime pieces are adapted to Notarius package names and contracts.
- No correction proposal, replacement policy, transcript mutation, or correction
ledger code has been copied.
- Runtime tests cover secret redaction, schema registry lookup, prompt metadata,
and scheduler behavior where applicable.
## Review Questions
- Did we copy only reusable infrastructure?
- Do provider-specific types stay behind adapter/runtime boundaries?
- Are diagnostics names and report concepts extraction-oriented?
- Is config limited to implemented behavior?

116
docs/roadmap/4-seriatim-input-adapter.md Normal file
View File

@@ -0,0 +1,116 @@
# Checkpoint 4: Seriatim Input Adapter
## Status
This document describes planned work, not implemented behavior.
## Goal
Add the first real input source while keeping transcript-specific behavior
isolated inside an input adapter.
This checkpoint should allow Seriatim minimal transcript JSON to become a
generic `SourceDocument`.
## Scope
In scope:
- `internal/adapters/input/seriatim`;
- parser for Seriatim minimal output JSON;
- mapping into `SourceDocument` and `SourceUnit`;
- source-document validation;
- adapter registry wiring;
- fixtures and tests;
- CLI/config path to select the adapter if the CLI shell exists.
Out of scope:
- D&D extraction;
- LLM extraction calls;
- transcript-specific behavior in runner/core packages;
- support for every possible Seriatim schema variant.
## Proposed Stages
### Stage 1: Seriatim Source Model
Define adapter-local structs for the Seriatim minimal output schema.
Expected external shape:
- top-level `metadata`;
- top-level `segments`;
- segment `id`;
- segment `start`;
- segment `end`;
- segment `speaker`;
- segment `text`.
Keep these structs in the adapter package.
### Stage 2: Parse And Validate
Implement parser and validation behavior.
Validation should cover:
- valid JSON;
- required metadata fields;
- required segment fields;
- unique segment IDs;
- non-empty segment text;
- valid start/end values as appropriate.
Prefer clear adapter-specific errors.
### Stage 3: Map To SourceDocument
Map Seriatim data into the generic source model:
- segment `id` becomes `SourceUnit.ID`;
- segment `text` becomes `SourceUnit.Text`;
- unit kind should identify transcript-like units without requiring core
packages to know transcript semantics;
- `speaker`, `start`, and `end` become unit metadata;
- Seriatim metadata becomes document metadata.
The resulting `SourceDocument` should pass core source validation.
### Stage 4: Registry And CLI Wiring
Register the adapter under a stable key, likely `seriatim`.
If CLI support exists, add provisional selection:
```sh
notarius extract ./transcript.json --input seriatim
```
The command may still use fake extractors until checkpoint 5.
### Stage 5: Fixtures And Tests
Add fixtures and tests for:
- valid Seriatim minimal transcript;
- malformed JSON;
- missing metadata;
- missing or duplicate segment IDs;
- empty segment text;
- source-reference compatibility with generated unit IDs.
## Done Criteria
- `go test ./...` passes.
- Seriatim minimal transcript JSON maps into `SourceDocument`.
- Transcript fields do not appear in core runner contracts.
- The adapter is selectable through the registry.
- Tests prove transcript-specific assumptions are isolated to the adapter.
## Review Questions
- Are segment, speaker, and timestamp assumptions contained inside the adapter?
- Are unit IDs stable and suitable for source references?
- Does the adapter preserve enough metadata for transcript-oriented output later?
- Should the adapter accept only Seriatim minimal output for now?

131
docs/roadmap/5-dnd-spells-extractor.md Normal file
View File

@@ -0,0 +1,131 @@
# Checkpoint 5: D&D Spells Extractor
## Status
This document describes planned work, not implemented behavior.
## Goal
Implement the first useful extraction module: D&D spell casts from a Seriatim
transcript source document.
This checkpoint should produce the first meaningful vertical slice from real
source input to validated artifact output.
## Scope
In scope:
- D&D spell artifact schema and Go structs;
- structured response schema asset;
- prompt assets;
- `internal/extractors/dnd/spells`;
- source-reference and schema validators in the extractor chain;
- fake LLM tests;
- CLI-level integration test if the CLI path is ready.
Out of scope:
- D&D item extraction;
- NPC extraction;
- combat extraction;
- cross-slice deduplication beyond simple deterministic merging;
- broad D&D rules validation.
## Proposed Stages
### Stage 1: Spell Artifact Schema
Define the D&D spell artifact model.
Initial shape:
```go
type SpellCast struct {
Player string `json:"player"`
Spell string `json:"spell"`
Effect string `json:"effect"`
NarrativeDescription string `json:"narrative_description"`
SourceRefs []SourceRef `json:"source_refs"`
}
```
Keep this schema inside the D&D spells extractor or a D&D artifact package, not
inside core framework packages.
### Stage 2: Structured Response Schema
Add a structured response schema asset for spell extraction.
The schema should require:
- spell-cast array;
- non-empty player, spell, effect, and narrative description fields;
- at least one source reference per spell cast.
### Stage 3: Prompt Assets
Add embedded prompt assets for D&D spell extraction.
Prompts should:
- describe the generic source-unit input format;
- explain that source references must use source-unit IDs;
- avoid relying on transcript-specific fields except as optional metadata;
- request only spell-cast artifacts.
### Stage 4: Extractor Implementation
Implement `internal/extractors/dnd/spells`.
The extractor should:
- satisfy the framework `Extractor` contract;
- build LLM messages from a source document or source slice;
- call the structured LLM client;
- return artifact candidates with source references;
- attach its validator chain.
### Stage 5: Validators And Tests
Wire deterministic validators:
- schema/shape validation;
- source-reference validation;
- required-field validation if not covered by schema handling.
Add tests using a fake structured LLM client:
- successful spell extraction;
- empty result;
- invalid source reference rejection;
- malformed structured output handling;
- stable output ordering.
### Stage 6: CLI Integration
If the CLI path is ready, add an end-to-end test using:
```sh
notarius extract ./transcript.json --input seriatim --extractors dnd.spells --output ./artifacts.json
```
The test should use fake LLM wiring and fixture input.
## Done Criteria
- `go test ./...` passes.
- Seriatim input can flow through the runner into the D&D spells extractor.
- Spell artifacts include valid source references.
- D&D concepts are contained in extractor/artifact packages and docs.
- The first meaningful vertical slice is available through tests, and through
CLI if the CLI path is ready.
## Review Questions
- Is the spell extractor domain-specific without making the framework
D&D-specific?
- Are source references valid and useful for downstream validation?
- Is prompt/schema ownership clear?
- Does this vertical slice reveal contract changes needed before adding items,
NPCs, or combat?

174
docs/roadmap/documentation.md Normal file
View File

@@ -0,0 +1,174 @@
# Documentation Roadmap
## Status
This document captures planned documentation decisions for Notarius. It records
policy choices while the application architecture is still being shaped. It does
not describe implemented behavior.
## Documentation Goals
Notarius documentation should make three boundaries obvious:
- source-format support belongs to input adapters;
- extraction-domain behavior belongs to extractor packages;
- core framework behavior is source-agnostic and domain-agnostic.
Documentation should avoid making the MVP look more transcript-specific or
D&D-specific than the architecture intends.
## Current Policy Decisions
### Planned Work Stays In Roadmap Docs
Until code exists, planned behavior belongs under `docs/roadmap/`.
Implemented behavior should later move into canonical docs. Roadmap files may
then link to those docs or be reduced to remaining future work.
### Core Docs Should Use Generic Terms
Core architecture docs should prefer:
- source document;
- source unit;
- source reference;
- input adapter;
- extractor;
- artifact;
- validator;
- run manifest.
Core docs should avoid transcript-specific terms such as segment, speaker,
timestamp, and transcript range unless discussing an input adapter or an example.
Core docs should avoid D&D-specific terms such as spell, NPC, item, combat, and
encounter unless discussing extractor packages or examples.
### Adapter Docs Own Source Formats
Each implemented input adapter should have a canonical integration document.
Likely future files:
```text
docs/integrations/seriatim-transcript.md
docs/integrations/markdown-source.md
```
Adapter docs should cover:
- accepted external schema or file shape;
- mapping into `SourceDocument` and `SourceUnit`;
- metadata preserved by the adapter;
- validation rules and failure behavior;
- examples.
The Seriatim adapter doc should reference the Seriatim schema it supports and
explain how transcript segment IDs become source-unit IDs.
### Extractor Docs Own Domains
Each implemented extractor family should have canonical internal or integration
docs.
Likely future files:
```text
docs/internal/extractors.md
docs/integrations/artifacts-dnd.md
```
Extractor docs should cover:
- extractor key;
- artifact type;
- schema version;
- required source-reference behavior;
- validator chain;
- prompt and response-schema ownership;
- examples.
D&D concepts should be documented in D&D extractor docs, not in generic runner
or framework docs.
### CLI Docs Should Reflect Extensibility
The CLI reference should present input adapters and extractors as selectable
components.
Provisional command shape:
```sh
notarius extract ./source.json --input seriatim --extractors dnd.spells --output ./artifacts.json
```
Once implemented, `docs/cli.md` should document:
- positional source input path;
- input adapter selection;
- extractor selection;
- config path behavior;
- output path behavior;
- diagnostics and report behavior;
- exit codes.
### Config Docs Should Separate Framework And Plugin-Like Options
`docs/config.md` should group fields by responsibility:
- input adapter selection and adapter-specific options;
- extractor selection and extractor-specific options;
- LLM runtime;
- validation runtime;
- source chunking;
- output and diagnostics.
Adapter-specific and extractor-specific config should not leak into unrelated
core config sections.
### Examples Should Stay Real
Examples should be added only when the matching behavior exists and should be
covered by tests where practical.
Likely future examples:
```text
examples/seriatim-minimal-transcript.json
examples/minimal-config.yml
examples/dnd-spells.artifacts.json
```
Examples should be secret-free and should use the same command shapes documented
in `docs/cli.md`.
## Canonical Documentation Targets
When the first vertical slice is implemented, add or update:
- `README.md`: concise purpose, shortest useful command, links.
- `docs/cli.md`: implemented command behavior.
- `docs/config.md`: implemented config behavior.
- `docs/operations.md`: diagnostics, retention, failure inspection.
- `docs/troubleshooting.md`: common failures.
- `docs/internal/overview.md`: implemented package map.
- `docs/internal/pipeline.md`: implemented extraction flow.
- `docs/internal/adapters.md`: adapter contract and implemented adapters.
- `docs/internal/extractors.md`: extractor contract and built-ins.
- `docs/internal/validators.md`: validator contract and built-ins.
- `docs/integrations/seriatim-transcript.md`: Seriatim input contract.
- `docs/integrations/artifacts.md`: output artifact envelope.
## Review Checklist For Future Documentation Changes
Before merging docs, check:
- Does the document describe implemented behavior outside `docs/roadmap/`?
- Are source-format details isolated to adapter docs?
- Are D&D details isolated to extractor or artifact docs?
- Is there one canonical home for the topic?
- Do command examples match implemented CLI syntax?
- Are examples valid, maintained, and free of secrets?
- Did any architecture, config, CLI, adapter, extractor, validator, or artifact
contract change require a docs update?

223
docs/roadmap/future.md
View File

@@ -1,223 +0,0 @@
# Future Work
Current Notarius behavior is documented in the canonical README, CLI,
configuration, operations, internal, and integration docs. This roadmap records
future work only. Items are ordered roughly by current value and specificity,
not as committed release dates.
## Near-Term D&D Pipeline
### Combat Enemy Ledger
- Add a D&D artifact that identifies enemies faced during combat and supports
an end-of-session encounter ledger.
- Track each enemy's observed state using a small controlled vocabulary such as
`active`, `killed`, `fled`, `captured`, or `incapacitated`, while preserving
an explicit unresolved state when the transcript does not establish an
outcome.
- Preserve the evidence for enemy participation and state changes rather than
inferring a terminal outcome from combat ending or an enemy disappearing
from the conversation.
- Define how repeated mentions, groups of unnamed enemies, summoned or allied
creatures, and the same enemy appearing in multiple combats affect identity
and ledger entries.
- Evaluate whether the ledger should be extracted directly, derived from
combat-turn artifacts, or use a sequential pipeline that consumes combat
turns and the normalized NPC registry as grounding references.
### Location Extraction
- Add a D&D artifact for locations visited by the party or otherwise mentioned
in the transcript.
- Distinguish observed visits from references, plans, recalled places, and
uncertain or inferred locations so a mention alone is not reported as a
visit.
- Preserve transcript evidence for each visit or mention and reconcile aliases,
nested places, and repeated appearances without collapsing distinct
locations that share a generic name.
- Define how the location artifact should ground later narrative reports and
whether future event artifacts should retain canonical location identities.
### Evaluate Spell Extraction And Normalization
- Evaluate ordinary extraction retries and the completed normalization path
against a human-reviewed transcript set before adding repair-aware retries or
an LLM-backed semantic validator.
- Maintain a small set of human-reviewed transcripts and outputs for prompt,
validator, and normalizer development. Treat model-quality review as an
iterative human evaluation aid, not a deterministic correctness gate.
### Evaluate The Shared D&D Scene Plan
- Reassess whether one shared scene plan provides enough context for NPC,
spell, combat, interaction, and scene-description lanes after real-world use.
Add more complex chunking only in response to demonstrated failures.
## Cross-Cutting LLM Runtime
### Deterministic Prompt Session Identity
- Replace the source-document-ID default for prompt sessions with one
predictable, procedurally generated session ID for the complete
source-processing workload.
- Preserve an explicit non-empty `--session-id` as the highest-precedence
override. Otherwise, derive the default only from the effective input module
identity and the exact raw input bytes.
- Use a versioned, bounded representation such as
`notarius:v1:<sha256(input-module + NUL + raw-input)>`. The exact encoding
must fit PromptKit's session length contract and must not embed source
content.
- Keep the derived session stable across runs, pipelines, selected lanes,
ordered steps, retries, resume, recomputation, LLM profiles, reasoning
overrides, and output, debug, or cache settings.
- Do not include file-backed references, generated references, reference
contents, or the composition of a reference bundle in session derivation.
References may change between prompt calls within one pipeline without
changing routing affinity.
- Resolve the authoritative session before checkpoint construction and use the
same value for checkpoint runtime identity, every prompt-facing module,
PromptKit's direct session field, the compatibility `session_id` prompt
variable, run-manifest metadata, and debug metadata.
- Keep routing identity separate from cache and checkpoint content identity.
Exact prompt prefixes, reference contents, model settings, and other
generation-affecting inputs must continue to participate in their existing
hashes and checkpoint fingerprints even though they do not change the
session.
- Treat the generated value as a provider-visible, stable pseudonymous
correlation identifier. Do not introduce an installation-specific HMAC or
secret unless a concrete multi-tenant or privacy requirement justifies
sacrificing deterministic identity across installations.
### Raise The Default Application-Wide LLM Limit
- Raise the default `concurrency.total_llm` value from 1 to 16 so ordinary
single-backend runs can use PromptKit's expected OpenRouter capacity and
lower-capacity local backends without an unnecessarily narrower Notarius
limit.
- Keep the Notarius application-wide scheduler mandatory and require
`total_llm` to remain a positive integer. Do not make the default unlimited:
endpoint-only profiles, an unrestricted local backend, injected clients, and
aggregate work across several backends may have no narrower PromptKit limit.
- Continue defaulting `concurrency.stage_workers.extract` to the effective
`total_llm`, making its default 16 as part of the same change. Preserve an
explicit lower extract-worker setting when an operator wants less queued or
concurrent extraction work.
- Define effective provider concurrency as the intersection of the Notarius
application-wide limit, the selected PromptKit backend limit when present,
and the work made available by stage execution. A Notarius limit of 16 does
not narrow a backend already limited to 16, while a local backend limited to
4 remains bounded at 4.
- Treat the default as an application-wide safety ceiling across profiles,
backends, modules, retries, and validators. A run that intentionally needs
the combined capacity of several backends may configure a higher
`total_llm` and an appropriate extract-worker count explicitly.
- Retain the existing configuration and environment override surfaces. Update
canonical configuration, operations, and internal documentation together
when the default changes.
- Reconsider decoupling the extract-worker default from `total_llm` only after
mixed-backend workloads demonstrate a need for a high global emergency
ceiling with a lower default work-production rate.
## Shared Normalization And Quality Work
### Generic LLM-Assisted Deduplication
- Add a reusable normalizer that asks an LLM to identify duplicate sets in a
list and propose one replacement element for each set.
- Define the minimum domain-neutral input contract, initially an ordered list
whose elements have stable unique IDs. Artifact-kind registrations or
adapters may expose that structure without moving domain rules into the
generic package.
- Keep mutation deterministic: parse and validate the model's duplicate groups,
require every referenced ID to exist, reject overlapping or malformed groups,
prevent unrelated insertion or deletion, and apply only approved replacement
operations in code.
- Preserve provenance needed for audit and downstream validation, and emit
warnings describing every collapsed group.
- Evaluate batching and context-window limits before applying the normalizer to
large artifact collections.
The model may use its own domain knowledge to judge semantic duplication; the
generic implementation is responsible only for the common proposal contract,
safety checks, and deterministic application of accepted changes.
### Validation And Review
- Add domain validators and production default chains alongside each new D&D
artifact.
- Add production LLM-backed validators only when a concrete review policy
benefits from model judgment and deterministic checks are insufficient.
- Add validator diagnostics and timing summaries if operators need more detail
than the current [durable output bundle](../integrations/json-output.md)
provides.
- Add validator compatibility metadata if deployments need config-time proof
that a validator is suitable for a particular stage, module, or artifact
kind.
- Add media-type validators when non-JSON artifact representations are
introduced.
## Further Reference Evolution
- Make prior-run artifacts easier to bind as references without changing the
existing module-facing reference-item contract.
- Add structured or parsed references, such as typed NPC registries, rosters,
or spell catalogs, when opaque UTF-8 prompt material is no longer sufficient.
- Add per-slot or per-chunk inclusion policies so large references are not
repeated in every prompt unnecessarily.
- Add token budgeting and model context-window management for reference
content.
- Add reference caching, preprocessing, summarization, embedding, or retrieval
only when reference size and observed model behavior justify them.
- Extend generated references to prior-run artifacts or derived summaries only
after same-run ordered handoffs establish the required provenance and
lifecycle semantics.
## Design Considerations To Revisit
These concerns are relevant to ordered artifact dependencies but are not
committed near-term features.
### Cross-artifact identity links
Evaluate whether downstream D&D artifacts should retain canonical NPC IDs from
the generated NPC reference in addition to normalized display names. Any such
contract must define player-character, unknown-actor, missing-NPC, and
superseded-identity behavior before implementation. Deterministic validation
may confirm that a linked ID exists in the consumed NPC artifact, but the link
must never substitute for transcript evidence that the downstream event
occurred.
### Artifact contract evolution
Define compatibility and migration policy before generated-reference chains
must span multiple schema versions or long-lived historical artifacts. The
policy should address stable identifier semantics, which schema changes permit
checkpoint reuse, when an older artifact may be decoded or adapted, and when a
producer or all dependents must be recomputed. Do not add a general migration
framework until an actual contract change requires one.
## Blue-Sky Platform And Operations
These ideas are intentionally less specified. Promote one into an earlier
section only after a concrete workflow, contract, and priority emerge.
### Platform Extensions
- Additional input adapters, such as Markdown or note-export formats.
- Additional output encoders.
- Concurrent cross-lane entity normalization or broader workflow composition.
- Batching or specialized context-window controls for LLM-backed validators.
### Distribution And Operations
- Packaged release artifacts for alpha distribution.
- A documented versioning and release process.
- Optional generated example-output fixtures with a regeneration procedure.
- Additional diagnostics or reporting views.
### Workspace And Storage
- Default-idempotent run behavior with an explicit force override.
- Remote workspace storage.
- Workspace garbage collection and archival policies.
- Cross-machine checkpoint reuse.

1032
docs/roadmap/implementation.md
View File

File diff suppressed because it is too large Load Diff

407
docs/roadmap/initial-architecture.md Normal file
View File

@@ -0,0 +1,407 @@
# Initial Architecture Roadmap
## Status
This document captures proposed architecture and implementation sequencing for
Notarius. It describes planned work, not implemented behavior.
## Goal
Notarius should extract structured JSON artifacts from primary source inputs
using modular, LLM-backed extractors.
The first MVP should target audio transcripts generated by Seriatim. That
choice should be implemented as an input adapter, not as a transcript-specific
assumption in the application core. Later input sources, such as unstructured
Markdown notes or Obsidian documents, should be addable through new adapters and
extractors without reshaping the framework.
The first extraction domain should be D&D session analysis, starting with spell
casts. That domain should live in extractor packages and related schemas, not in
core framework packages.
The application should follow the same broad architecture as Audita:
- deterministic core packages for config, source documents, artifacts, diagnostics, and reporting;
- input adapters that translate external source formats into a small internal source model;
- reusable framework packages for contracts, orchestration, LLM runtime, structured output, and validation;
- independent extractor packages that own domain-specific behavior;
- independent validator packages;
- embedded prompt and JSON schema assets;
- CLI orchestration that wires the pieces together without owning domain logic.
The main domain difference from Audita is that Notarius emits extracted
artifacts rather than proposing and applying transcript corrections.
## Architectural Principles
- Keep the core input model generic: ordered text units plus metadata.
- Keep source-format details in hexagonal input adapters.
- Keep extraction-domain details in extractor packages.
- Treat evidence as source references, not transcript references.
- Prefer narrow, useful abstractions over a universal document model.
- Preserve enough provenance for validation, replay, and downstream inspection.
## Proposed Package Shape
```text
cmd/notarius
internal/cli
internal/core/config
internal/core/source
internal/core/sourcechunking
internal/core/artifacts
internal/core/diagnostics
internal/core/reporting
internal/core/extractorcatalog
internal/core/inputcatalog
internal/adapters/input/seriatim
internal/adapters/input/markdown
internal/framework/contracts
internal/framework/extraction
internal/framework/runner
internal/framework/validators
internal/framework/llm
internal/framework/responseschema
internal/framework/structuredoutput
internal/framework/promptcontext
internal/framework/warnings
internal/extractors/dnd/spells
internal/extractors/dnd/items
internal/extractors/dnd/npcs
internal/extractors/dnd/combat
internal/validators/source_refs
internal/validators/schema_validity
internal/validators/domain_consistency
internal/validators/llm_review
internal/prompts
examples
docs/internal
```
The `markdown` adapter is listed as a likely future package. The MVP should only
implement the Seriatim adapter unless a second adapter is needed to test the
boundary.
## Core Concepts
### SourceDocument
Canonical internal representation of source material. This should be the object
extractors receive, regardless of whether the original input was a transcript,
Markdown file, note export, or another source type.
```go
type SourceDocument struct {
ID string `json:"id"`
Kind string `json:"kind"`
Format string `json:"format"`
Digest string `json:"digest"`
Units []SourceUnit `json:"units"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type SourceUnit struct {
ID string `json:"id"`
Kind string `json:"kind"`
Text string `json:"text"`
Metadata map[string]any `json:"metadata,omitempty"`
}
```
Initial source-unit assumptions:
- units are ordered;
- unit IDs are stable within a source document;
- each unit has extractable text;
- adapter-specific metadata may carry speaker, timestamps, heading paths, page
numbers, or other source details.
### Input Adapter
Hexagonal boundary for external source formats.
```go
type InputAdapter interface {
Key() string
Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
}
```
The MVP adapter should target Seriatim minimal transcript JSON. Seriatim segment
fields should map as follows:
- `id` becomes `SourceUnit.ID`;
- `text` becomes `SourceUnit.Text`;
- `speaker`, `start`, and `end` become unit metadata;
- Seriatim output metadata becomes document metadata.
The core runner should not know that these units came from transcript segments.
### SourceRef
Grounding reference from an extracted fact back to source units.
```go
type SourceRef struct {
SourceID string `json:"source_id"`
StartUnitID string `json:"start_unit_id"`
EndUnitID string `json:"end_unit_id"`
}
```
Initial source-reference validation should require:
- source ID exists for the current run;
- start and end unit IDs exist;
- start is less than or equal to end in document order;
- the referenced range is contiguous within the source document;
- every extracted fact has at least one source reference unless its schema
explicitly allows ungrounded metadata.
Transcript-oriented output can still present these as transcript segment ranges
when the adapter metadata makes that interpretation available.
### Extractor
Reusable module contract for producing one artifact type.
```go
type Extractor interface {
Key() string
ArtifactType() string
SchemaVersion() string
Validators() []Validator
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
}
```
An extractor should receive either a whole source document or a source slice,
depending on runner configuration. It should return typed artifact candidates
plus warnings. It should not mutate the source document.
Extractor packages own domain concepts. For example, D&D spell extraction should
live under `internal/extractors/dnd/spells`; a future to-do extractor for notes
should live under a different domain path and use the same framework contract.
### Validator
Reusable validation contract for artifact candidates.
Validators should cover:
- JSON/schema validity;
- source-reference validity;
- required-field and shape checks;
- domain consistency;
- optional LLM review for high-risk or ambiguous artifacts.
Validator output should follow Audita's decision-cardinality model: each
candidate artifact receives exactly one decision per validator.
### Artifact
Final approved JSON output from one or more extractors.
Artifacts should preserve enough metadata to support downstream validation,
debugging, and replay. The exact top-level envelope is still open, but should
include artifact type, schema version, extracted records, source references, and
run manifest data.
### RunManifest
Per-run provenance record.
```go
type RunManifest struct {
InputAdapter string `json:"input_adapter"`
SourceDigests []string `json:"source_digests"`
Extractors []string `json:"extractors"`
SchemaVersion string `json:"schema_version"`
ValidationStatus string `json:"validation_status"`
}
```
The manifest should eventually include model names, prompt IDs, prompt hashes,
response schema versions, config source, started/completed timestamps, and
diagnostics paths.
## Initial Extractor Targets
### D&D Spells
Recommended first vertical slice because it is narrow but representative.
```go
type SpellCast struct {
Player string `json:"player"`
Spell string `json:"spell"`
Effect string `json:"effect"`
NarrativeDescription string `json:"narrative_description"`
SourceRefs []SourceRef `json:"source_refs"`
}
```
The spell extractor should be D&D-specific. The framework should not know what a
spell is.
### D&D Items
Tracks items gained, lost, transferred, consumed, or transformed.
Open questions:
- Should currency be represented as items or as its own artifact type?
- Should item ownership be a required field?
- How should ambiguous ownership changes be represented?
### D&D NPCs
Tracks NPCs interacted with, newly introduced, renamed, described, or otherwise
made relevant to campaign state.
Open questions:
- Should NPC identity resolution happen inside this extractor or in a later
deduplication stage?
- Should location/faction/relationship facts be separate artifact types?
### D&D Combat
Likely warrants a dedicated schema rather than a generic event list.
Proposed first shape:
```go
type CombatTurn struct {
Actor string `json:"actor"`
Action string `json:"action"`
Outcome string `json:"outcome"`
NarrativeDescription string `json:"narrative_description"`
SourceRefs []SourceRef `json:"source_refs"`
}
```
Open questions:
- Should combat be extracted as turns, rounds, encounters, or all three?
- Should mechanical fields such as damage, conditions, saves, attacks, and spell
slots be normalized immediately or added later?
- How should uncertain initiative order be represented?
### Future Non-D&D Extractors
The architecture should support extractors outside the D&D domain. Examples:
- to-do items from Markdown or Obsidian notes;
- decisions and action items from meeting transcripts;
- named people, places, and dates from research notes.
These should be addable as extractor packages without changing runner,
validator, source-reference, or LLM framework contracts.
## Proposed Runner Flow
1. Load effective config.
2. Create diagnostics run directory.
3. Resolve the configured input adapter.
4. Read source input.
5. Parse source input into a `SourceDocument`.
6. Validate source-document invariants.
7. Chunk source units into deterministic source slices.
8. Resolve configured extractor instances through a registry.
9. Execute extractor instances in configured order.
10. Run deterministic validators before LLM-backed validators.
11. Retain approved artifacts and rejected-artifact diagnostics.
12. Merge approved slice artifacts deterministically.
13. Serialize final output JSON.
14. Write run manifest, diagnostics, and optional report JSON.
The runner should operate on source documents and source slices only. Any
transcript-specific behavior should happen before the runner, inside the input
adapter, or after the runner, inside output rendering that understands source
metadata.
## Audita Patterns To Reuse
Reuse these architectural patterns:
- deterministic parsing and schema validation style;
- deterministic chunking of ordered source units;
- explicit extractor registry;
- `contracts` package for transport-neutral interfaces;
- OpenAI-compatible structured LLM client;
- scheduler for bounded LLM concurrency;
- embedded prompt registry with prompt metadata and hashes;
- embedded response-schema registry with schema metadata and hashes;
- diagnostics run directory with redacted effective config;
- validator decision cardinality and deterministic validator ordering;
- CLI tests and fixture-driven integration tests.
Avoid copying these Audita concepts directly:
- transcript-specific core types;
- correction proposals;
- replacement policies;
- deterministic transcript mutation;
- correction ledger terminology.
Those concepts are specific to Audita's transcript-editing role and should be
replaced with source-document, artifact-candidate, artifact-validation, and
extraction-report concepts.
## Checkpoint Roadmap
The initial implementation should proceed through five coherent checkpoints.
Each checkpoint should leave the repository in a reviewable state, with the code
compiling and targeted tests covering the newly introduced contracts or behavior.
1. [Core Contracts And Skeleton](1-core-contracts-and-skeleton.md)
2. [Framework Composition](2-framework-composition.md)
3. [Portable Audita Infrastructure](3-portable-audita-infrastructure.md)
4. [Seriatim Input Adapter](4-seriatim-input-adapter.md)
5. [D&D Spells Extractor](5-dnd-spells-extractor.md)
The first useful vertical slice should arrive at checkpoint 5: Seriatim
transcript input to validated D&D spell artifact output. Earlier checkpoints are
intentionally contract-first and may not produce useful user output yet.
## Open Design Questions
- Should final output be one combined artifact envelope or one file per
extractor?
- Should extractor output use typed Go structs per artifact or a generic
artifact record with `json.RawMessage` payloads?
- Should schemas be versioned per extractor, globally, or both?
- Should every record require source references, or should some top-level
artifact metadata be allowed without source references?
- Should overlapping source-reference ranges be merged, preserved exactly, or
both?
- Should extraction run independently per source slice only, or should some
extractors receive whole-document context?
- Should a later reconciliation stage deduplicate entities and events across
source slices?
- Should LLM review be part of each extractor's validator chain or a separate
review phase?
- Should the Seriatim adapter accept only its minimal schema initially or also
support richer transcript schemas?
- Should source-unit metadata be untyped `map[string]any`, typed extension
structs, or both?
## Near-Term Documentation Tasks
Once behavior is implemented, move implemented contracts out of roadmap docs and
into canonical docs:
- `README.md` for purpose and shortest useful command;
- `docs/cli.md` for CLI behavior;
- `docs/config.md` for config fields and precedence;
- `docs/internal/` for implemented architecture and package boundaries;
- `docs/integrations/` for source input and artifact file formats;
- `examples/` for maintained source, config, and artifact examples.

318
docs/roadmap/promptkit.md
View File

@@ -1,318 +0,0 @@
# PromptKit v0.5 Integration And LLM Profile Policy
## Purpose
This roadmap defines the target state for upgrading Notarius from PromptKit
v0.3.0 to v0.5.0 and adopting the upstream runtime and profile facilities that
directly improve Notarius. It also defines the application policy for stable,
domain-oriented LLM profile names, operator overrides, pipeline inheritance,
profile validation, provider defaults, checkpoint identity, and documentation.
The ordered work needed to reach this state belongs in
[the implementation plan](implementation.md). Current behavior remains defined
by the canonical documentation outside `docs/roadmap/` until the corresponding
work is implemented.
## Background
Notarius currently pins PromptKit v0.3.0. Its adapter prepares a request once
for debug material and then independently runs the original request, causing
PromptKit to prepare the same logical call a second time. The CLI validates an
explicit profile by preparing a synthetic prompt. PromptKit profile selection
can be repeated on individual module bindings or replaced for one invocation
with `--llm-profile`, but a configured pipeline cannot yet declare one inherited
profile policy.
PromptKit v0.4.0 and v0.5.0 add the upstream boundaries needed to improve these
areas:
- [v0.4.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.4.0.md)
adds opaque prepared executions, exact profile and prompt inspection, and a
typed backend-capacity error;
- [v0.5.0](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/releases/v0.5.0.md)
adds application fallback profile filesystems and stops sending unset
optional sampling controls as framework-selected provider values; and
- the [v0.5.0 format contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.5.0/docs/formats.md)
defines the resulting profile-source and execution-setting precedence.
A source-compatibility test of the current Notarius repository against
PromptKit v0.5.0 completed successfully. The work is therefore primarily an
intentional runtime and configuration migration rather than a repair for a
breaking Go API change.
## Goals
- Pin and document PromptKit v0.5.0 as Notarius's supported upstream contract.
- Execute the exact prepared request snapshot whose safe details are recorded
in Notarius debug material.
- Validate configured PromptKit profiles through the upstream inspection API
without synthetic prompts, provider calls, or credential-value access.
- Give Notarius an application-owned, operator-overridable
`dnd-extraction` profile fallback.
- Let a pipeline choose one default LLM profile without repeating that ID on
every LLM-backed binding.
- Apply profile inheritance and run-wide overrides only where the resolved
module or validator can use an LLM.
- Preserve accurate checkpoint invalidation, effective profile provenance,
redaction, cancellation, concurrency, and provider-neutral module contracts.
- Provide operators with one clear deployment pattern for production,
development, and local profile definitions.
## Target End State
### PromptKit Runtime Boundary
Notarius depends on PromptKit v0.5.0 and uses its public APIs rather than
reimplementing source or execution resolution.
For each structured completion, the adapter:
1. builds one PromptKit run request from the provider-neutral Notarius request;
2. calls `PrepareExecution` once;
3. immediately arranges an idempotent `Discard` for every unexecuted handle;
4. obtains credential-redacted `Details` for debug and response metadata; and
5. calls `RunPrepared` so generation uses that exact frozen snapshot.
The debug prompt and successful result therefore describe the same selected
profile, rendered messages, input bytes, session, output contract, and effective
settings even when a filesystem-backed source changes concurrently. PromptKit
handle types remain private to `internal/framework/llm`.
PromptKit admission failures continue to match Notarius's provider-neutral
`ErrLLMCapacityExceeded` contract. When PromptKit supplies a `CapacityError`,
the adapter obtains the normalized backend ID through `errors.As` and may add it
to safe application-owned diagnostics without parsing upstream error wording.
The backend ID does not become a provider-specific module contract.
### Optional Provider Controls
Notarius accepts PromptKit v0.5.0's new behavior for `temperature`,
`max_tokens`, and `top_p`: an unset setting is omitted from compatible provider
requests and the provider chooses its own default. Notarius does not restore
PromptKit's former implicit `top_p: 1` value globally.
An operator who requires a particular value specifies it in the selected
PromptKit profile. The application fallback described below intentionally
leaves these controls unset. A human-reviewed D&D extraction comparison should
be performed after the upgrade, but paid or nondeterministic model output is
not part of the default automated test suite.
### Profile Inspection
Pipeline-aware configuration validation uses `Engine.InspectProfile` for every
effective explicit profile ID. It verifies that the profile exists, parses and
validates, resolves its backend and target, and is compatible with the engine's
registered backends. It does not create a synthetic prompt, load prompt inputs,
contact a provider, or require credential values to exist in the validation
process environment.
Credential availability is execution-time state. PromptKit preparation still
enforces the selected profile's credential contract before generation. This
keeps `notarius config validate` useful in build and deployment validation
environments where secrets are deliberately absent.
PromptKit construction for inspection and execution uses one shared internal
profile-source and backend-option path. The CLI does not expose PromptKit public
types across the Notarius LLM boundary merely to perform inspection.
`InspectPrompt` is not adopted merely because it exists. It remains available
for a later, separately defined module-to-prompt interface preflight if a
concrete validation requirement justifies that additional contract.
### Application And Operator Profile Sources
Notarius embeds one ordinary PromptKit YAML profile with the stable ID
`dnd-extraction`. It is an application fallback registered through
`WithFallbackProfileFS`, is owned by the D&D module family, and initially
preserves the current effective D&D baseline:
- backend: PromptKit's built-in `openrouter` backend;
- model: `openai/gpt-5.6-luna`;
- reasoning effort: unset, allowing OpenAI's backend to apply its default of
`medium`;
- generation timeout: 240 seconds;
- service tier: `flex`; and
- no application-selected `temperature`, `max_tokens`, or `top_p`.
All maintained D&D LLM prompt definitions use `dnd-extraction` as their
`default_profile`. The ID communicates workload intent rather than a provider,
model, or environment. Changing the embedded fallback is an intentional
Notarius execution-policy change and participates in checkpoint identity.
Effective profile definitions resolve in PromptKit's order:
1. programmatic in-memory profiles used by tests or explicit consumers;
2. the operator source configured by `promptkit.profile_file` or
`promptkit.profile_dir`;
3. the Notarius application fallback source; and
4. PromptKit's embedded built-in catalog.
Only an absent ID falls through to the next source. A matching profile is a
complete definition: fields are not merged with a lower-precedence definition,
and a malformed matching operator profile fails rather than silently selecting
the application fallback.
Production, development, and local deployments should normally provide
different complete definitions for the same `dnd-extraction` ID. An operator
source is optional because the application fallback keeps the maintained D&D
workflow usable, but a deployment that needs an intentional model or backend
policy should configure its own definition.
### Domain Ownership And Asset Assembly
The D&D fallback profile remains under `internal/modules/dnd` and is registered
by the D&D registrar, consistent with ADR-0004. Generic LLM plumbing knows how
to collect and flatten application fallback profile filesystems but contains no
D&D model or policy knowledge.
The shared asset registry detects invalid roots, unreadable sources, and
duplicate flattened paths. PromptKit remains responsible for strict profile
YAML parsing, duplicate profile-ID detection, source precedence, and effective
target resolution. The same assembled fallback source is supplied to runtime
execution and CLI profile inspection.
### Explicit Module Execution Metadata
Every registered input, chunk, extract, merge, normalize, and output module
declares one required execution class: `deterministic` or `llm_backed`.
Validator registrations continue to declare the same distinction through their
validator specifications.
The registered specification is authoritative for configuration resolution.
Current production classifications are:
- the D&D scene chunker, all D&D extractors, and the D&D NPC normalizer are
LLM-backed;
- the Seriatim input adapter, generic chunker, all current mergers, all other
current normalizers, and the JSON output encoder are deterministic; and
- current validators retain their declared classifications.
Missing or unsupported execution metadata is a registration error. Explicitly
assigning `llm_profile` to a deterministic module or validator is a pipeline
resolution error. The framework does not infer execution class by inspecting
domain package names or concrete implementation types at runtime.
The module specification replaces the chunk runner's special runtime
execution-class probe. Effective resolved bindings already express the result:
only LLM-backed bindings may retain a non-empty profile.
### Pipeline-Level Profile Default
Configuration version 4 gains one optional non-empty pipeline field:
```yaml
pipelines:
dnd-session:
llm_profile: dnd-extraction
```
No configuration-version increment is required because the field is additive
and existing files remain valid. An explicitly present blank value is invalid.
For every selected LLM-backed module and validator, the effective profile uses
this precedence:
1. non-empty run-wide `--llm-profile` override;
2. binding-specific `llm_profile`;
3. pipeline-level `llm_profile`; and
4. the prompt definition's `default_profile`, represented by an empty effective
Notarius binding profile.
The run-wide override and inherited pipeline default never attach to a
deterministic binding. Binding-specific exceptions remain available when one
operation needs a different cost, latency, quality, backend, or reasoning
policy.
Inheritance is resolved after module and validator selection, including
`--only` lane filtering, but before effective-pipeline validation, digest
construction, explicit-profile inspection, checkpoint construction,
preparation, execution, or provenance capture. Only profiles used by selected
LLM-backed bindings are inspected. An unused pipeline default in a pipeline
with no selected LLM-backed work does not require an otherwise unused profile
to exist.
The resolved pipeline contains effective binding profiles rather than a second
runtime inheritance mechanism. Two pipelines that differ only by spelling the
same effective policy once as a pipeline default and once on every LLM-backed
binding have the same semantic resolved digest. Changing an effective profile
changes the digest and applicable checkpoint identity.
### Provenance And Checkpoints
The PromptKit profile-source checkpoint fingerprint covers:
- the PromptKit v0.5.0 built-in profile catalog identity;
- exact application fallback profile asset content; and
- exact configured operator profile YAML content, when present.
The existing local-backend target fingerprint remains separate and continues
to exclude scheduling-only concurrency limits. Fingerprints contain hashes and
stable markers, not profile contents, filesystem paths, endpoints, credentials,
or other secrets.
Changing the PromptKit version, application fallback, operator profile, or
effective pipeline profile makes incompatible LLM checkpoints ineligible for
reuse. The dependency upgrade is expected to invalidate checkpoints produced
under v0.3.0.
Successful run manifests continue to record only profiles actually selected by
PromptKit, including their effective model, backend, and reasoning metadata.
Debug output reports the same effective execution snapshot used for generation.
### Operator Documentation And Examples
Canonical documentation clearly distinguishes:
- Notarius prompt and schema assets embedded in the application;
- Notarius application fallback profiles embedded in the application;
- PromptKit's own embedded built-in profiles; and
- operator profile files on the deployment filesystem.
The configuration reference owns the pipeline field, profile-source fields,
validation rules, and precedence. Operations owns deployment layout, working
directory behavior, credentials, and environment-specific profile management.
The PromptKit integration document owns the pinned upstream contract and
source-precedence boundary. Internal documents describe asset registration,
resolution, inspection, prepared execution, fingerprinting, and tests without
duplicating user-facing field definitions.
The maintained examples continue to include only the minimal and complete D&D
configurations. They use the stable `dnd-extraction` policy, and one maintained
PromptKit profile file under `examples/` demonstrates an operator override.
Examples remain secret-free and are validated without live provider calls.
## Out Of Scope
- Implementing the separate deterministic prompt-session identity roadmap
item.
- Changing the default `concurrency.total_llm` value; PromptKit's retained
OpenRouter capacity of 16 remains relevant to that separate item.
- Adding model evaluation as a deterministic or CI correctness gate.
- Automatically selecting production, development, or local environments.
Deployment configuration chooses the operator profile source.
- Profile inheritance, partial profile merging, or cross-profile aliases.
- Exposing PromptKit types to modules, validators, durable output contracts, or
public configuration structures.
- Adopting `InspectPrompt` without a separately justified prompt-interface
validation contract.
## Acceptance Criteria
- Notarius builds and its offline test suite passes with PromptKit v0.5.0.
- Every structured completion executes the exact snapshot used for safe debug
prompt details.
- Profile preflight uses profile inspection and no synthetic prompt.
- The embedded `dnd-extraction` fallback resolves without an operator source,
and a matching valid operator profile replaces it completely.
- Every production module has explicit, correct execution metadata.
- Pipeline, binding, CLI, and prompt-default precedence behaves as defined for
modules and validators, while deterministic bindings remain profile-free.
- Effective profiles participate in pipeline digests, profile inspection,
checkpoint identity, debug records, and run provenance at the appropriate
boundaries.
- The dependency and application fallback changes invalidate incompatible old
checkpoints without exposing profile or credential content.
- Canonical documentation and maintained examples accurately describe and
exercise the implemented operator workflow.
- Default tests remain deterministic, offline, credential-free, and focused on
Notarius-owned behavior rather than duplicating PromptKit's upstream suite.

85
examples/dnd-complete-transcript.json
View File

@@ -1,85 +0,0 @@
{
"metadata": {
"id": "session-ravenfall",
"title": "The Ravenfall Watchtower"
},
"segments": [
{
"id": 1,
"start": 0,
"end": 14,
"speaker": "DM",
"text": "Recap: last session, the party learned that Elder Rowan vanished near the Ravenfall watchtower."
},
{
"id": 2,
"start": 14,
"end": 25,
"speaker": "Player",
"text": "Out of character, we agree to investigate the watchtower before the next game."
},
{
"id": 3,
"start": 25,
"end": 39,
"speaker": "DM",
"text": "Aria and Borin arrive at the ruined Ravenfall watchtower as dusk settles over the road."
},
{
"id": 4,
"start": 39,
"end": 55,
"speaker": "Mira Thorn",
"text": "Mira Thorn steps from the doorway and says, \"Elder Rowan warned me that Kesh would return for the relic.\""
},
{
"id": 5,
"start": 55,
"end": 70,
"speaker": "DM",
"text": "Mira leads the party to a hidden cache. The party discovers a moonblade and acquires 20 silver pieces."
},
{
"id": 6,
"start": 70,
"end": 83,
"speaker": "Aria",
"text": "Aria hands her healing potion to Borin so he can carry it into the tower."
},
{
"id": 7,
"start": 83,
"end": 96,
"speaker": "DM",
"text": "Kesh, the goblin captain, orders the raiders to attack. Roll initiative."
},
{
"id": 8,
"start": 96,
"end": 110,
"speaker": "DM",
"text": "On Kesh's turn, he strikes Borin with his scimitar. Borin drinks the healing potion on his turn."
},
{
"id": 9,
"start": 110,
"end": 124,
"speaker": "Aria",
"text": "Aria casts Cure Wounds on Borin, then invokes Aegis of Emberfall as Kesh closes in."
},
{
"id": 10,
"start": 124,
"end": 137,
"speaker": "DM",
"text": "Kesh casts Shield as a reaction against Borin's counterattack, but the party drives the raiders away."
},
{
"id": 11,
"start": 137,
"end": 150,
"speaker": "DM",
"text": "After the battle, Aria pays 5 silver pieces to repair the watchtower gate."
}
]
}

103
examples/dnd-complete.config.yml
View File

@@ -1,103 +0,0 @@
version: 4
promptkit:
profile_file: ./examples/profiles/dnd-extraction.yml
concurrency:
total_llm: 2
stage_workers:
extract: 2
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: auto
directory: ./notarius-cache/chunk-plans
checkpoints:
enabled: true
directory: ./notarius-cache/checkpoints
debug:
directory: ./notarius-debug
pipelines:
dnd-session:
llm_profile: dnd-extraction
input: seriatim
# Stable campaign context is shared by every module that accepts these slots.
references:
party: ./dnd-party.txt
glossary: ./dnd-glossary.txt
chunk:
module: dnd/scenes
retries: 2
output:
module: json
options:
include_chunk_map: true
evidence_context:
enabled: true
window_units: 3
lanes:
- item-events
- npcs
- spells
- combat-turns
- npc-interactions
steps:
# Establish session-wide reference artifacts alongside independent item events.
- id: describe-session
artifacts:
item-events:
extract:
module: dnd/item-events
retries: 2
merge: appendorder
normalize: dnd/item-events
npcs:
extract:
module: dnd/npcs
retries: 2
merge: appendorder
normalize:
module: dnd/npcs
retries: 2
scene-descriptions:
extract:
module: dnd/scene-descriptions
retries: 2
merge: appendorder
normalize: dnd/scene-descriptions
- id: extract-events
# Accepted NPC grounding and scene-description eligibility artifacts are
# supplied in memory to their compatible consumers in this step.
references:
npcs:
artifact:
step: describe-session
lane: npcs
scene_descriptions:
artifact:
step: describe-session
lane: scene-descriptions
artifacts:
spells:
extract:
module: dnd/spells
retries: 2
references:
spell_catalog: ./dnd-spell-catalog.json
merge: appendorder
# Stage-local file references are intentionally bound at each stage.
normalize:
module: dnd/spells
references:
spell_catalog: ./dnd-spell-catalog.json
combat-turns:
extract:
module: dnd/combat-turns
retries: 2
merge: appendorder
normalize: dnd/combat-turns
npc-interactions:
extract:
module: dnd/npc-interactions
retries: 2
merge: appendorder
normalize: dnd/npc-interactions

8
examples/dnd-glossary.txt
View File

@@ -1,8 +0,0 @@
Ravenfall watchtower: a ruined watchtower near the party's current route.
Mira Thorn: the watchtower's keeper.
Elder Rowan: a missing local scholar.
Kesh: a goblin captain leading raiders.
Moonblade: a blade found in the watchtower's hidden cache.
Cure Wounds: a healing spell.
Shield: a defensive reaction spell.
Aegis of Emberfall: a campaign spell recorded in the supplied catalog overlay.

9
examples/dnd-minimal.config.yml
View File

@@ -1,9 +0,0 @@
version: 4
pipelines:
dnd-session:
llm_profile: dnd-extraction
input: seriatim
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells

2
examples/dnd-party.txt
View File

@@ -1,2 +0,0 @@
Aria: party cleric and recurring healer.
Borin: fighter ally.

21
examples/dnd-spell-catalog.json
View File

@@ -1,21 +0,0 @@
{
"schema_version": "notarius.dnd.spell-catalog-overlay.v1",
"catalogs": [
{
"id": "notarius.example-campaign",
"ruleset": "dnd-5e-2014",
"source": {
"title": "Notarius example campaign spell names",
"version": "1",
"url": "",
"license": ""
},
"spells": [
{
"name": "Aegis of Emberfall",
"aliases": ["Emberfall Aegis"]
}
]
}
]
}

5
examples/profiles/dnd-extraction.yml
View File

@@ -1,5 +0,0 @@
id: dnd-extraction
backend: openrouter
model: openai/gpt-5.6-luna
timeout_seconds: 240
service_tier: flex

22
examples/seriatim-minimal-transcript.json
View File

@@ -1,22 +0,0 @@
{
"metadata": {
"id": "session-alpha",
"title": "Synthetic D&D spell session"
},
"segments": [
{
"id": 1,
"start": 0,
"end": 4,
"speaker": "Aria",
"text": "Aria raises her holy symbol and casts Cure Wounds."
},
{
"id": 2,
"start": 4,
"end": 8,
"speaker": "DM",
"text": "The bandit mage casts Shield as the blow lands."
}
]
}

10
go.mod
View File

@@ -1,11 +1,3 @@
module gitea.maximumdirect.net/eric/notarius
go 1.25.5
require (
gitea.maximumdirect.net/eric/promptkit v0.5.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
gopkg.in/yaml.v3 v3.0.1
)
require golang.org/x/text v0.40.0
go 1.24.0

12
go.sum
View File

@@ -1,12 +0,0 @@
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

315
internal/cli/assembled_spell_pipeline_contract_test.go
View File

@@ -1,315 +0,0 @@
package cli
import (
"context"
"encoding/json"
"fmt"
"reflect"
"sort"
"strings"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
spellcatalog "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/spells/catalog"
)
const assembledSpellExtractorKey = "test/dnd/spell-casts"
func TestAssembledSpellPipelineNormalizesMergedCasts(t *testing.T) {
registries, resolved, extractor := assembledSpellPipeline(t, assembledSpellPipelineOptions{})
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
output, err := pipeline.New().Run(context.Background(), pipeline.RunInput{
Prepared: prepared,
RawInput: readRepositoryFile(t, "examples", "seriatim-minimal-transcript.json"),
ChunkCacheMode: pipeline.ChunkCacheBypass,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
chunkIndexes := extractor.chunkIndexesSnapshot()
sort.Ints(chunkIndexes)
if !reflect.DeepEqual(chunkIndexes, []int{0, 1}) {
t.Fatalf("extractor chunk indexes = %#v, want two chunk-boundary calls", chunkIndexes)
}
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
t.Fatalf("run output = %#v, want approved normalized output without rejections", output)
}
if output.NormalizeOutputs[0].NormalizerKey != spellnormalize.Key {
t.Fatalf("normalized output module = %q, want %q", output.NormalizeOutputs[0].NormalizerKey, spellnormalize.Key)
}
var normalized dnd.SpellList
if err := json.Unmarshal(output.NormalizeOutputs[0].Artifact.Content, &normalized); err != nil {
t.Fatalf("decode normalized output: %v", err)
}
if len(normalized.SpellCasts) != 2 {
t.Fatalf("normalized casts = %#v, want collapsed duplicate plus distinct evidence", normalized.SpellCasts)
}
first, distinct := normalized.SpellCasts[0], normalized.SpellCasts[1]
if first.Spell != "Cure Wounds" || first.Caster != " Aria \t" {
t.Fatalf("retained cast = %#v, want canonical spell with first occurrence caster", first)
}
if !reflect.DeepEqual(first.SourceRefs, []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}, {SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}}) {
t.Fatalf("retained refs = %#v, want sorted complete evidence", first.SourceRefs)
}
if distinct.Spell != "Cure Wounds" || distinct.Caster != "aria" || !reflect.DeepEqual(distinct.SourceRefs, []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}}) {
t.Fatalf("distinct cast = %#v, want separate evidence event", distinct)
}
wantWarningReasons := []string{
spellnormalize.ReasonCodeSpellNameCanonicalized,
spellnormalize.ReasonCodeSourceReferencesNormalized,
spellnormalize.ReasonCodeDuplicateSpellCastCollapsed,
"spell_not_near_source",
}
gotWarningReasons := make([]string, len(output.Warnings))
for index, warning := range output.Warnings {
gotWarningReasons[index] = warning.ReasonCode
}
if !reflect.DeepEqual(gotWarningReasons, wantWarningReasons) {
t.Fatalf("warnings = %#v, want deterministic normalize and validation warnings", output.Warnings)
}
if output.Warnings[2].Scope != "spell_casts[0]" || !strings.Contains(output.Warnings[2].Message, "retained input index 0") || !strings.Contains(output.Warnings[2].Message, "removed input indices [1]") {
t.Fatalf("duplicate warning = %#v, want retained and removed merged indices", output.Warnings[2])
}
warningsFile := decodeAssembledOutput[struct {
Warnings []contracts.Warning `json:"warnings"`
}](t, output.OutputFiles, "warnings.json")
if !reflect.DeepEqual(warningsFile.Warnings, output.Warnings) {
t.Fatalf("warnings file = %#v, run warnings = %#v, want manifest output path to preserve warnings", warningsFile.Warnings, output.Warnings)
}
manifest := decodeAssembledOutput[artifacts.RunManifest](t, output.OutputFiles, "manifest.json")
if len(manifest.ArtifactLanes) != 1 || manifest.ArtifactLanes[0].Normalizer != spellnormalize.Key {
t.Fatalf("manifest lanes = %#v, want assembled spell normalizer", manifest.ArtifactLanes)
}
normalizerMetadata, ok := manifest.ArtifactLanes[0].Metadata["normalizer"].(map[string]any)
_, hasOverlayIDs := normalizerMetadata["catalog_overlay_ids"]
if !ok || normalizerMetadata["catalog_base_id"] != spellcatalog.SRD5E2014ID || !strings.HasPrefix(stringValue(normalizerMetadata["catalog_digest"]), "sha256:") || !hasOverlayIDs {
t.Fatalf("normalizer manifest metadata = %#v, want base ID, digest, and overlay IDs", manifest.ArtifactLanes[0].Metadata)
}
}
func TestAssembledSpellPipelineHonorsNormalizeValidatorOverride(t *testing.T) {
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{normalizeValidatorOverride: true})
var normalizeChain *pipeline.ResolvedValidatorChain
for index := range resolved.ValidatorChains {
chain := &resolved.ValidatorChains[index]
if chain.Stage == pipeline.StageNormalize && chain.ModuleKey == spellnormalize.Key && chain.LaneID == "spells" {
normalizeChain = chain
break
}
}
if normalizeChain == nil || len(normalizeChain.Validators) != 1 || normalizeChain.Validators[0].Binding.Module != "generic/always_accept" {
t.Fatalf("normalize validator chain = %#v, want explicit always-accept override", normalizeChain)
}
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
output, err := pipeline.New().Run(context.Background(), pipeline.RunInput{
Prepared: prepared,
RawInput: readRepositoryFile(t, "examples", "seriatim-minimal-transcript.json"),
ChunkCacheMode: pipeline.ChunkCacheBypass,
})
if err != nil || output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
t.Fatalf("Run() error = %v output = %#v, want approved override run", err, output)
}
for _, warning := range output.Warnings {
if warning.ReasonCode == "spell_not_near_source" {
t.Fatalf("warnings = %#v, want explicit validator override to replace default relatedness chain", output.Warnings)
}
}
}
func TestAssembledSpellPipelineRejectsUnknownSpellWithoutPromotingAttemptWarning(t *testing.T) {
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{unknownSpell: true})
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
output, err := pipeline.New().Run(context.Background(), pipeline.RunInput{
Prepared: prepared,
RawInput: readRepositoryFile(t, "examples", "seriatim-minimal-transcript.json"),
ChunkCacheMode: pipeline.ChunkCacheBypass,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if output.Manifest.ValidationStatus != "rejected" || len(output.NormalizeOutputs) != 0 || len(output.Rejected) != 1 {
t.Fatalf("run output = %#v, want one rejected normalize candidate and no normalized output", output)
}
rejection := output.Rejected[0]
if rejection.Stage != string(pipeline.StageNormalize) || rejection.LaneID != "spells" || rejection.ModuleKey != spellnormalize.Key || rejection.ValidatorName != "extract/dnd/spells/catalog" || rejection.ReasonCode != "unknown_spell" {
t.Fatalf("rejection = %#v, want durable normalize catalog rejection", rejection)
}
rejectedFile := decodeAssembledOutput[struct {
Rejected []contracts.RejectedOutput `json:"rejected"`
}](t, output.OutputFiles, "rejected.json")
if !reflect.DeepEqual(rejectedFile.Rejected, output.Rejected) {
t.Fatalf("rejected file = %#v, run rejections = %#v, want durable rejection diagnostic", rejectedFile.Rejected, output.Rejected)
}
for _, warning := range output.Warnings {
if warning.ReasonCode == spellnormalize.ReasonCodeSpellNameUnresolved {
t.Fatalf("warnings = %#v, want rejected-attempt warning to remain non-durable", output.Warnings)
}
}
}
func TestAssembledSpellPipelinePromotesUnknownSpellWarningWhenOverrideAccepts(t *testing.T) {
registries, resolved, _ := assembledSpellPipeline(t, assembledSpellPipelineOptions{normalizeValidatorOverride: true, unknownSpell: true})
prepared, err := pipeline.Prepare(resolved, registries, pipeline.ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
output, err := pipeline.New().Run(context.Background(), pipeline.RunInput{
Prepared: prepared,
RawInput: readRepositoryFile(t, "examples", "seriatim-minimal-transcript.json"),
ChunkCacheMode: pipeline.ChunkCacheBypass,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
t.Fatalf("run output = %#v, want accepted unknown spell with explicit validator override", output)
}
var normalized dnd.SpellList
if err := json.Unmarshal(output.NormalizeOutputs[0].Artifact.Content, &normalized); err != nil {
t.Fatalf("decode normalized output: %v", err)
}
if len(normalized.SpellCasts) != 1 || normalized.SpellCasts[0].Spell != "Mysterious Burst" {
t.Fatalf("normalized casts = %#v, want unresolved name preserved", normalized.SpellCasts)
}
if len(output.Warnings) != 1 || output.Warnings[0].ReasonCode != spellnormalize.ReasonCodeSpellNameUnresolved || output.Warnings[0].Scope != "spell_casts[0]" {
t.Fatalf("warnings = %#v, want promoted scoped unresolved-name warning", output.Warnings)
}
warningsFile := decodeAssembledOutput[struct {
Warnings []contracts.Warning `json:"warnings"`
}](t, output.OutputFiles, "warnings.json")
if !reflect.DeepEqual(warningsFile.Warnings, output.Warnings) {
t.Fatalf("warnings file = %#v, run warnings = %#v, want durable unresolved-name warning", warningsFile.Warnings, output.Warnings)
}
}
type assembledSpellPipelineOptions struct {
normalizeValidatorOverride bool
unknownSpell bool
}
func assembledSpellPipeline(t *testing.T, options assembledSpellPipelineOptions) (pipeline.Registries, pipeline.ResolvedPipeline, *assembledSpellExtractor) {
t.Helper()
components := productionTestComponents(t)
extractor := &assembledSpellExtractor{unknownSpell: options.unknownSpell}
if err := pipeline.RegisterExtractor[dnd.SpellList](components.registries.Extractors, pipeline.ModuleSpec{
Key: assembledSpellExtractorKey,
Stage: pipeline.StageExtract,
ExecutionClass: contracts.ExecutionClassDeterministic,
Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.spell_casts"},
ArtifactKind: dnd.SpellListKind,
}, func() (contracts.Extractor[dnd.SpellList], error) {
return extractor, nil
}); err != nil {
t.Fatalf("register assembled extractor: %v", err)
}
normalize := pipeline.Binding(spellnormalize.Key)
if options.normalizeValidatorOverride {
normalize.Validators = pipeline.ValidatorOverride{
Set: true,
Validators: []pipeline.ModuleBinding{pipeline.Binding("generic/always_accept")},
}
}
resolved, err := pipeline.ResolvePipeline(pipeline.PipelineProfile{
ID: "assembled-dnd-spells",
Input: pipeline.Binding("seriatim"),
Chunk: pipeline.ModuleBinding{Module: "generic", Options: map[string]any{"max_units": 1}},
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"spells": {Extract: pipeline.Binding(assembledSpellExtractorKey), Normalize: normalize},
},
Output: pipeline.Binding("json"),
}, pipeline.ResolveOptions{}, catalogFromRegistries(components.registries))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
return components.registries, resolved, extractor
}
type assembledSpellExtractor struct {
mu sync.Mutex
chunkIndexes []int
unknownSpell bool
}
func (e *assembledSpellExtractor) Key() string { return assembledSpellExtractorKey }
func (*assembledSpellExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (e *assembledSpellExtractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.SpellList], error) {
if err := ctx.Err(); err != nil {
return contracts.TypedExtractionResult[dnd.SpellList]{}, err
}
if req.Chunk == nil || req.Source == nil {
return contracts.TypedExtractionResult[dnd.SpellList]{}, fmt.Errorf("assembled extractor requires source and chunk")
}
e.mu.Lock()
e.chunkIndexes = append(e.chunkIndexes, req.Chunk.Index)
e.mu.Unlock()
refOne := source.SourceRef{SourceID: req.Source.ID, StartUnitID: 1, EndUnitID: 1}
refTwo := source.SourceRef{SourceID: req.Source.ID, StartUnitID: 2, EndUnitID: 2}
if e.unknownSpell {
if req.Chunk.Index == 0 {
return contracts.TypedExtractionResult[dnd.SpellList]{Value: dnd.SpellList{SpellCasts: []dnd.SpellCast{{
Caster: "Aria", Spell: "Mysterious Burst", SourceRefs: []source.SourceRef{refOne},
}}}}, nil
}
return contracts.TypedExtractionResult[dnd.SpellList]{Value: dnd.SpellList{SpellCasts: []dnd.SpellCast{}}}, nil
}
switch req.Chunk.Index {
case 0:
return contracts.TypedExtractionResult[dnd.SpellList]{Value: dnd.SpellList{SpellCasts: []dnd.SpellCast{{
Caster: " Aria \t", Spell: " cure wounds ", SourceRefs: []source.SourceRef{refTwo, refOne},
}}}}, nil
case 1:
return contracts.TypedExtractionResult[dnd.SpellList]{Value: dnd.SpellList{SpellCasts: []dnd.SpellCast{
{Caster: "aria", Spell: "Cure Wounds", SourceRefs: []source.SourceRef{refOne, refTwo}},
{Caster: "aria", Spell: "Cure Wounds", SourceRefs: []source.SourceRef{refTwo}},
}}}, nil
default:
return contracts.TypedExtractionResult[dnd.SpellList]{}, fmt.Errorf("unexpected assembled chunk index %d", req.Chunk.Index)
}
}
func (e *assembledSpellExtractor) chunkIndexesSnapshot() []int {
e.mu.Lock()
defer e.mu.Unlock()
return append([]int(nil), e.chunkIndexes...)
}
func decodeAssembledOutput[T any](t *testing.T, files []contracts.OutputFile, name string) T {
t.Helper()
for _, file := range files {
if file.Name != name {
continue
}
var value T
if err := json.Unmarshal(file.Bytes, &value); err != nil {
t.Fatalf("decode %s: %v", name, err)
}
return value
}
t.Fatalf("output files = %#v, want %q", files, name)
return *new(T)
}

371
internal/cli/cache_contract_test.go
View File

@@ -1,371 +0,0 @@
package cli
import (
"bytes"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestRunChunkPlanModePrecedenceAndValidation(t *testing.T) {
tests := []struct {
name string
fileMode string
envMode string
cliMode string
wantStores int
}{
{name: "default", wantStores: 1},
{name: "file", fileMode: "bypass"},
{name: "environment", envMode: "bypass"},
{name: "cli", envMode: "refresh", cliMode: "bypass"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
if tt.name == "default" {
removeStateTestConfigLine(t, roots.config, " mode: auto\n")
} else if tt.fileMode != "" {
replaceStateTestConfigLine(t, roots.config, " mode: auto\n", " mode: "+tt.fileMode+"\n")
}
var stores []string
opts := newStateTestHarness().options()
opts.LookupEnv = func(name string) (string, bool) {
if name == "NOTARIUS_CACHE_CHUNK_PLANS_MODE" && tt.envMode != "" {
return tt.envMode, true
}
return "", false
}
opts.ChunkPlanStoreFactory = func(root string) (pipeline.ChunkPlanStore, error) {
stores = append(stores, root)
return chunkplan.NewFilesystemStore(root)
}
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input}
if tt.cliMode != "" {
args = append(args, "--chunk_cache", tt.cliMode)
}
var stdout, stderr bytes.Buffer
if code := RunWithOptions(args, &stdout, &stderr, opts); code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertStateTestOutput(t, roots.output)
if len(stores) != tt.wantStores {
t.Fatalf("chunk plan store roots = %v, want %d stores", stores, tt.wantStores)
}
if tt.wantStores == 1 && stores[0] != roots.plans {
t.Fatalf("chunk plan store root = %q, want %q", stores[0], roots.plans)
}
if tt.wantStores == 0 {
assertAbsent(t, roots.plans)
}
})
}
t.Run("invalid cli syntax is a usage error", func(t *testing.T) {
roots := newStateTestRoots(t)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "invalid"}, &stdout, &stderr, newStateTestHarness().options())
if code != 2 || stdout.Len() != 0 || stderr.Len() == 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertNoRunState(t, roots)
})
for _, tt := range []struct {
name string
fileConfig bool
}{
{name: "invalid environment mode"},
{name: "invalid file mode", fileConfig: true},
} {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
if tt.fileConfig {
replaceStateTestConfigLine(t, roots.config, " mode: auto\n", " mode: invalid\n")
} else {
opts.LookupEnv = func(name string) (string, bool) {
if name == "NOTARIUS_CACHE_CHUNK_PLANS_MODE" {
return "invalid", true
}
return "", false
}
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &stdout, &stderr, opts)
if code != 1 || stdout.Len() != 0 || stderr.Len() == 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertNoRunState(t, roots)
})
}
}
func TestRunChunkPlanRootSelectionAndFailures(t *testing.T) {
t.Run("empty configured root uses the per-user cache root", func(t *testing.T) {
roots := newStateTestRoots(t)
removeStateTestConfigLine(t, roots.config, fmt.Sprintf(" directory: %q\n", roots.plans))
userCache := filepath.Join(t.TempDir(), "user-cache")
var stores []string
opts := newStateTestHarness().options()
opts.UserCacheDir = func() (string, error) { return userCache, nil }
opts.ChunkPlanStoreFactory = func(root string) (pipeline.ChunkPlanStore, error) {
stores = append(stores, root)
return chunkplan.NewFilesystemStore(root)
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &stdout, &stderr, opts)
if code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
wantRoot := filepath.Join(userCache, "notarius", "chunk-plans")
if len(stores) != 1 || stores[0] != wantRoot {
t.Fatalf("chunk plan store roots = %v, want [%q]", stores, wantRoot)
}
assertFile(t, filepath.Join(wantRoot, strings.TrimPrefix(stateTestDigest, "sha256:"), "plan.json"))
assertAbsent(t, roots.plans)
})
t.Run("bypass avoids default cache dependencies", func(t *testing.T) {
roots := newStateTestRoots(t)
removeStateTestConfigLine(t, roots.config, fmt.Sprintf(" directory: %q\n", roots.plans))
userCacheCalls := 0
storeCalls := 0
opts := newStateTestHarness().options()
opts.UserCacheDir = func() (string, error) {
userCacheCalls++
return "", errors.New("user cache must not be resolved")
}
opts.ChunkPlanStoreFactory = func(string) (pipeline.ChunkPlanStore, error) {
storeCalls++
return nil, errors.New("chunk plan store must not be constructed")
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
if code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if userCacheCalls != 0 || storeCalls != 0 {
t.Fatalf("user cache calls=%d store calls=%d, want none", userCacheCalls, storeCalls)
}
assertStateTestOutput(t, roots.output)
assertAbsent(t, roots.plans)
})
t.Run("user cache resolution failure has context and no output", func(t *testing.T) {
roots := newStateTestRoots(t)
removeStateTestConfigLine(t, roots.config, fmt.Sprintf(" directory: %q\n", roots.plans))
opts := newStateTestHarness().options()
opts.UserCacheDir = func() (string, error) { return "", errors.New("cache home unavailable") }
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &stdout, &stderr, opts)
if code != 1 || !strings.Contains(stderr.String(), "resolve chunk plan root") || !strings.Contains(stderr.String(), "cache home unavailable") {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertNoRunState(t, roots)
})
t.Run("store construction failure has context and no output", func(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
opts.ChunkPlanStoreFactory = func(root string) (pipeline.ChunkPlanStore, error) {
return nil, fmt.Errorf("store unavailable")
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &stdout, &stderr, opts)
want := fmt.Sprintf("create chunk plan store at %q", roots.plans)
if code != 1 || !strings.Contains(stderr.String(), want) || !strings.Contains(stderr.String(), "store unavailable") {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertNoRunState(t, roots)
})
t.Run("checkpoint root resolution failure has context and no output", func(t *testing.T) {
roots := newStateTestRoots(t)
removeStateTestConfigLine(t, roots.config, fmt.Sprintf(" directory: %q\n", roots.checkpoints))
opts := newStateTestHarness().options()
opts.UserCacheDir = func() (string, error) { return "", errors.New("checkpoint cache unavailable") }
result := runStateTest(t, roots, opts, false, true, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "resolve checkpoint root") || !strings.Contains(result.stderr, "checkpoint cache unavailable") {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
assertNoRunState(t, roots)
})
}
func TestRunAutoReusesPlanWhenRunInputsChange(t *testing.T) {
roots := newStateTestRoots(t)
data, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
configText := replaceRequiredOnce(t, string(data), " chunk: test/chunk\n", ` chunk:
module: test/chunk
options:
strategy: first
`)
configText = replaceRequiredOnce(t, configText, " output: test/output\n", ` other:
extract: test/extract
merge: test/merge
normalize: test/normalize
output: test/output
`)
if err := os.WriteFile(roots.config, []byte(configText), 0o600); err != nil {
t.Fatal(err)
}
referencePath := filepath.Join(filepath.Dir(roots.input), "reference.txt")
if err := os.WriteFile(referencePath, []byte("reference content"), 0o600); err != nil {
t.Fatal(err)
}
harness := newStateTestHarness()
var firstStdout, firstStderr bytes.Buffer
first := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, &firstStdout, &firstStderr, harness.options())
if first != 0 {
t.Fatalf("first run code=%d stdout=%q stderr=%q", first, firstStdout.String(), firstStderr.String())
}
configData, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
configText = replaceRequiredOnce(t, string(configData), "strategy: first", "strategy: second")
if err := os.WriteFile(roots.config, []byte(configText), 0o600); err != nil {
t.Fatal(err)
}
var stdout, stderr bytes.Buffer
second := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--only", "items", "--reference", "chunk.cache-reference=" + referencePath,
}, &stdout, &stderr, harness.options())
if second != 0 {
t.Fatalf("second run code=%d stdout=%q stderr=%q", second, stdout.String(), stderr.String())
}
harness.mu.Lock()
chunkCalls := harness.chunkCalls
harness.mu.Unlock()
if chunkCalls != 1 {
t.Fatalf("chunk calls across changed run inputs = %d, want 1", chunkCalls)
}
assertFile(t, filepath.Join(roots.plans, strings.TrimPrefix(stateTestDigest, "sha256:"), "plan.json"))
assertAnyFile(t, roots.output)
}
func TestRunResumeSelectsConfiguredOrPerUserCheckpointRoot(t *testing.T) {
for _, configured := range []bool{true, false} {
name := "per-user root"
if configured {
name = "configured root"
}
t.Run(name, func(t *testing.T) {
roots := newStateTestRoots(t)
if !configured {
removeStateTestConfigLine(t, roots.config, fmt.Sprintf(" directory: %q\n", roots.checkpoints))
}
userCache := filepath.Join(t.TempDir(), "user-cache")
userCacheCalls := 0
opts := newStateTestHarness().options()
opts.UserCacheDir = func() (string, error) {
userCacheCalls++
return userCache, nil
}
result := runStateTest(t, roots, opts, false, true, "bypass")
if result.code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
wantRoot := roots.checkpoints
wantCalls := 0
if !configured {
wantRoot = filepath.Join(userCache, "notarius", "checkpoints")
wantCalls = 1
}
if userCacheCalls != wantCalls {
t.Fatalf("user cache calls = %d, want %d", userCacheCalls, wantCalls)
}
assertAnyFile(t, wantRoot)
if !configured {
assertAbsent(t, roots.checkpoints)
}
assertStateTestOutput(t, roots.output)
})
}
t.Run("disabled avoids checkpoint root resolution", func(t *testing.T) {
roots := newStateTestRoots(t)
replaceStateTestConfigLine(t, roots.config, " enabled: true\n", " enabled: false\n")
removeStateTestConfigLine(t, roots.config, fmt.Sprintf(" directory: %q\n", roots.checkpoints))
opts := newStateTestHarness().options()
opts.UserCacheDir = func() (string, error) { return "", errors.New("checkpoint cache must not be resolved") }
result := runStateTest(t, roots, opts, false, false, "bypass")
if result.code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
assertStateTestOutput(t, roots.output)
assertAbsent(t, roots.checkpoints)
})
t.Run("resume requires enabled checkpoint recording", func(t *testing.T) {
roots := newStateTestRoots(t)
replaceStateTestConfigLine(t, roots.config, " enabled: true\n", " enabled: false\n")
result := runStateTest(t, roots, newStateTestHarness().options(), true, true, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "--resume requires cache.checkpoints.enabled: true") {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
assertNoRunState(t, roots)
})
}
func TestConfigCommandsDoNotResolveRunState(t *testing.T) {
for _, args := range [][]string{
{"config", "validate", "--config"},
{"pipelines", "list", "--config"},
} {
name := strings.Join(args[:2], "-")
t.Run(name, func(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
opts.UserCacheDir = func() (string, error) { return "", errors.New("state root must not be resolved") }
opts.ChunkPlanStoreFactory = func(string) (pipeline.ChunkPlanStore, error) {
return nil, errors.New("chunk plan store must not be constructed")
}
command := append([]string(nil), args...)
command = append(command, roots.config)
var stdout, stderr bytes.Buffer
code := RunWithOptions(command, &stdout, &stderr, opts)
if code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertNoRunState(t, roots)
})
}
}
func replaceStateTestConfigLine(t *testing.T, path, old, new string) {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
text := replaceRequiredOnce(t, string(data), old, new)
if err := os.WriteFile(path, []byte(text), 0o600); err != nil {
t.Fatal(err)
}
}
func removeStateTestConfigLine(t *testing.T, path, line string) {
replaceStateTestConfigLine(t, path, line, "")
}
func assertNoRunState(t *testing.T, roots stateTestRoots) {
t.Helper()
assertAbsent(t, roots.output)
assertAbsent(t, roots.plans)
assertAbsent(t, roots.checkpoints)
assertAbsent(t, roots.debug)
}

176
internal/cli/catalog.go
View File

@@ -1,176 +0,0 @@
package cli
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
genericregister "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/register"
seriatimregister "gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/register"
)
type productionComponents struct {
registries pipeline.Registries
assets *llm.AssetRegistry
}
func newProductionComponents() (productionComponents, error) {
registries := pipeline.Registries{
Inputs: pipeline.NewInputAdapterRegistry(),
Chunkers: pipeline.NewChunkerRegistry(),
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
ArtifactEvidence: pipeline.NewArtifactEvidenceRegistry(),
Extractors: pipeline.NewExtractorRegistry(),
Mergers: pipeline.NewMergerRegistry(),
Normalizers: pipeline.NewNormalizerRegistry(),
Validators: pipeline.NewValidatorRegistry(),
ValidatorChains: pipeline.NewValidatorChainRegistry(),
Outputs: pipeline.NewOutputEncoderRegistry(),
}
assets := llm.NewAssetRegistry()
registrars := []struct {
name string
register func(pipeline.Registries, *llm.AssetRegistry) error
}{
{name: "generic", register: genericregister.Register},
{name: "seriatim", register: seriatimregister.Register},
{name: "dnd", register: dndregister.Register},
}
for _, registrar := range registrars {
if err := registrar.register(registries, assets); err != nil {
return productionComponents{}, fmt.Errorf("register %s module family: %w", registrar.name, err)
}
}
return productionComponents{registries: registries, assets: assets}, nil
}
func productionRegistries() (pipeline.Registries, error) {
components, err := newProductionComponents()
return components.registries, err
}
func productionCatalog() (pipeline.ModuleCatalog, error) {
registries, err := productionRegistries()
if err != nil {
return pipeline.ModuleCatalog{}, err
}
return catalogFromRegistries(registries), nil
}
func productionPromptAssets() (*llm.AssetRegistry, error) {
components, err := newProductionComponents()
return components.assets, err
}
func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
if !isEmptyCatalog(opts.Catalog) {
return opts.Catalog, nil
}
if !isEmptyRegistries(opts.Registries) {
return catalogFromRegistries(opts.Registries), nil
}
return productionCatalog()
}
func effectiveRegistries(opts Options) (pipeline.Registries, error) {
if !isEmptyRegistries(opts.Registries) {
return opts.Registries, nil
}
if !isEmptyCatalog(opts.Catalog) {
return registriesFromCatalog(opts.Catalog), nil
}
return productionRegistries()
}
func catalogFromRegistries(registries pipeline.Registries) pipeline.ModuleCatalog {
return pipeline.ModuleCatalog{
Inputs: registries.Inputs,
Chunkers: registries.Chunkers,
ArtifactCodecs: registries.ArtifactCodecs,
ArtifactEvidence: registries.ArtifactEvidence,
Extractors: registries.Extractors,
Mergers: registries.Mergers,
Normalizers: registries.Normalizers,
Validators: registries.Validators,
ValidatorChains: registries.ValidatorChains,
Outputs: registries.Outputs,
}
}
func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
return pipeline.Registries{
Inputs: catalog.Inputs,
Chunkers: catalog.Chunkers,
ArtifactCodecs: catalog.ArtifactCodecs,
ArtifactEvidence: catalog.ArtifactEvidence,
Extractors: catalog.Extractors,
Mergers: catalog.Mergers,
Normalizers: catalog.Normalizers,
Validators: catalog.Validators,
ValidatorChains: catalog.ValidatorChains,
Outputs: catalog.Outputs,
}
}
func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
return catalog.Inputs == nil &&
catalog.Chunkers == nil &&
catalog.ArtifactCodecs == nil &&
catalog.ArtifactEvidence == nil &&
catalog.Extractors == nil &&
catalog.Mergers == nil &&
catalog.Normalizers == nil &&
catalog.Validators == nil &&
catalog.ValidatorChains == nil &&
catalog.Outputs == nil
}
func isEmptyRegistries(registries pipeline.Registries) bool {
return registries.Inputs == nil &&
registries.Chunkers == nil &&
registries.ArtifactCodecs == nil &&
registries.ArtifactEvidence == nil &&
registries.Extractors == nil &&
registries.Mergers == nil &&
registries.Normalizers == nil &&
registries.Validators == nil &&
registries.ValidatorChains == nil &&
registries.Outputs == nil
}
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
return func(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return buildProductionLLMClient(ctx, cfg, profileID, overrides, assets)
}
}
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, overrides LLMRuntimeOverrides, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil {
return nil, nil, err
}
if assets == nil {
return nil, nil, fmt.Errorf("production asset registry must not be nil")
}
recorder := llm.NewLLMProfileRecorder()
client, err := llm.NewPromptKitClient(llm.PromptKitClientConfig{
ProfileDir: cfg.PromptKit.ProfileDir,
ProfileFile: cfg.PromptKit.ProfileFile,
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
Assets: assets,
Recorder: recorder,
ReasoningEffort: overrides.ReasoningEffort,
})
if err != nil {
return nil, nil, fmt.Errorf("create PromptKit-backed LLM client: %w", err)
}
scheduler, err := llm.NewScheduler(cfg.Concurrency.TotalLLM)
if err != nil {
return nil, nil, fmt.Errorf("create LLM scheduler: %w", err)
}
return llm.NewScheduledClient(client, scheduler), nil, nil
}

257
internal/cli/command_contract_test.go
View File

@@ -1,257 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
)
func TestCommandHelpSpellingsWriteUsageToStdout(t *testing.T) {
tests := [][]string{nil, {"help"}, {"--help"}, {"-h"}}
for _, args := range tests {
name := "no arguments"
if len(args) > 0 {
name = args[0]
}
t.Run(name, func(t *testing.T) {
var stdout, stderr bytes.Buffer
code := RunWithOptions(args, &stdout, &stderr, commandContractOptions(t))
if code != 0 || !strings.Contains(stdout.String(), "Usage:") || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
}
func TestCommandSyntaxErrorsUseStderrAndExitTwo(t *testing.T) {
tests := []struct {
name string
args []string
want string
}{
{name: "unknown command", args: []string{"unknown"}, want: "unknown command"},
{name: "missing config subcommand", args: []string{"config"}, want: "config requires a subcommand"},
{name: "unknown pipelines subcommand", args: []string{"pipelines", "unknown"}, want: "unknown pipelines subcommand"},
{name: "malformed run flag", args: []string{"run", "demo", "--chunk_cache", "invalid"}, want: "not supported"},
{name: "unknown flag", args: []string{"config", "validate", "--unknown"}, want: "flag provided but not defined"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var stdout, stderr bytes.Buffer
code := RunWithOptions(tt.args, &stdout, &stderr, commandContractOptions(t))
if code != 2 || !strings.Contains(stderr.String(), tt.want) || stdout.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
}
func TestConfigDiscoveryPrefersExplicitPathThenEnvironment(t *testing.T) {
explicit := writeCommandConfig(t, "explicit", "alpha")
environment := writeCommandConfig(t, "environment", "beta")
lookup := func(name string) (string, bool) {
if name == "NOTARIUS_CONFIG" {
return environment, true
}
return "", false
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"pipelines", "list", "--config", explicit}, &stdout, &stderr, commandContractOptionsWithLookup(t, lookup))
if code != 0 || stdout.String() != "alpha\nexplicit\n" || stderr.Len() != 0 {
t.Fatalf("explicit config: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
stdout.Reset()
stderr.Reset()
code = RunWithOptions([]string{"pipelines", "list"}, &stdout, &stderr, commandContractOptionsWithLookup(t, lookup))
if code != 0 || stdout.String() != "beta\nenvironment\n" || stderr.Len() != 0 {
t.Fatalf("environment config: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
}
func TestConfigDiscoveryUsesCompiledDefaultOnlyWhenAvailable(t *testing.T) {
info, statErr := os.Stat(defaultConfigPath)
if statErr != nil && !os.IsNotExist(statErr) {
t.Fatalf("stat compiled default config: %v", statErr)
}
if statErr == nil && !info.Mode().IsRegular() {
t.Skipf("compiled default config has unexpected host state: %s", info.Mode())
}
path, err := discoverConfigPath("", commandContractOptions(t))
if statErr == nil {
if err != nil || path != defaultConfigPath {
t.Fatalf("discoverConfigPath() = %q, %v; want compiled default", path, err)
}
return
}
if err == nil || !strings.Contains(err.Error(), "config file not found") {
t.Fatalf("discoverConfigPath() error = %v, want documented not-found context", err)
}
}
func TestConfigLoadingFailuresReturnOneWithPathContext(t *testing.T) {
missing := filepath.Join(t.TempDir(), "missing.yml")
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"config", "validate", "--config", missing}, &stdout, &stderr, commandContractOptions(t))
if code != 1 || !strings.Contains(stderr.String(), missing) || stdout.Len() != 0 {
t.Fatalf("missing config: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
malformed := filepath.Join(t.TempDir(), "malformed.yml")
if err := os.WriteFile(malformed, []byte("version: [\n"), 0o600); err != nil {
t.Fatal(err)
}
stdout.Reset()
stderr.Reset()
code = RunWithOptions([]string{"config", "validate", "--config", malformed}, &stdout, &stderr, commandContractOptions(t))
if code != 1 || !strings.Contains(stderr.String(), malformed) || !strings.Contains(stderr.String(), "parse config file") || stdout.Len() != 0 {
t.Fatalf("malformed config: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
}
func TestConfigValidateResolvesPipelineAndChecksSelection(t *testing.T) {
configPath := writeResolvableCommandConfig(t)
options := commandContractOptions(t)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo", "--only", "spells"}, &stdout, &stderr, options)
if code != 0 || !strings.Contains(stdout.String(), "valid for pipeline \"demo\"") || stderr.Len() != 0 {
t.Fatalf("valid resolution: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
stdout.Reset()
stderr.Reset()
code = RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "missing"}, &stdout, &stderr, options)
if code != 1 || !strings.Contains(stderr.String(), "pipeline \"missing\"") {
t.Fatalf("unknown pipeline: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
stdout.Reset()
stderr.Reset()
code = RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo", "--only", "missing"}, &stdout, &stderr, options)
if code != 1 || !strings.Contains(stderr.String(), "lane \"missing\"") {
t.Fatalf("unknown lane: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
stdout.Reset()
stderr.Reset()
code = RunWithOptions([]string{"config", "validate", "--config", configPath, "--only", "spells"}, &stdout, &stderr, options)
if code != 2 || !strings.Contains(stderr.String(), "--only requires --pipeline") {
t.Fatalf("missing pipeline for only: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
stdout.Reset()
stderr.Reset()
code = RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo", "--only", "spells,,other"}, &stdout, &stderr, options)
if code != 2 || !strings.Contains(stderr.String(), "--only must contain") {
t.Fatalf("malformed only: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
}
func TestConfigValidatePipelineDefaultProfileIsOffline(t *testing.T) {
configPath := writeCommandConfigContent(t, `version: 4
pipelines:
demo:
llm_profile: dnd-extraction
input: seriatim
artifacts:
spells:
extract: dnd/spells
`)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "demo"}, &stdout, &stderr, Options{})
if code != 0 || !strings.Contains(stdout.String(), "valid for pipeline \"demo\"") || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
}
func TestPipelinesListSortsNormalizedIDsInTextAndJSON(t *testing.T) {
configPath := writeCommandConfig(t, " zeta ", "alpha")
options := commandContractOptions(t)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"pipelines", "list", "--config", configPath}, &stdout, &stderr, options)
if code != 0 || stdout.String() != "alpha\nzeta\n" || stderr.Len() != 0 {
t.Fatalf("text list: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
stdout.Reset()
stderr.Reset()
code = RunWithOptions([]string{"pipelines", "list", "--config", configPath, "--json"}, &stdout, &stderr, options)
var payload struct {
Pipelines []string `json:"pipelines"`
}
if err := json.Unmarshal(stdout.Bytes(), &payload); err != nil {
t.Fatalf("JSON list = %q: %v", stdout.String(), err)
}
if code != 0 || len(payload.Pipelines) != 2 || payload.Pipelines[0] != "alpha" || payload.Pipelines[1] != "zeta" || stderr.Len() != 0 {
t.Fatalf("JSON list: code=%d payload=%#v stderr=%q", code, payload, stderr.String())
}
}
func TestRemovedStructuralFlagsAndRuntimeFailuresKeepExitClasses(t *testing.T) {
configPath := writeResolvableCommandConfig(t)
options := commandContractOptions(t)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "demo", "--input", "missing-input", "--config", configPath, "--diagnostics-dir", t.TempDir()}, &stdout, &stderr, options)
if code != 2 || !strings.Contains(stderr.String(), "flag provided but not defined") {
t.Fatalf("removed flag: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
stdout.Reset()
stderr.Reset()
code = RunWithOptions([]string{"run", "missing", "--input", "missing-input", "--config", configPath, "--chunk_cache", "bypass"}, &stdout, &stderr, options)
if code != 1 || !strings.Contains(stderr.String(), "pipeline \"missing\"") || stdout.Len() != 0 {
t.Fatalf("valid-runtime failure: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
}
func commandContractOptions(t *testing.T) Options {
return commandContractOptionsWithLookup(t, emptyLookup)
}
func commandContractOptionsWithLookup(t *testing.T, lookup func(string) (string, bool)) Options {
t.Helper()
components, err := newProductionComponents()
if err != nil {
t.Fatal(err)
}
return Options{
Catalog: catalogFromRegistries(components.registries),
Registries: components.registries,
LookupEnv: lookup,
}
}
func writeCommandConfig(t *testing.T, firstID, secondID string) string {
t.Helper()
content := fmt.Sprintf("version: 4\npipelines:\n %q:\n input: seriatim\n %q:\n input: seriatim\n", firstID, secondID)
return writeCommandConfigContent(t, content)
}
func writeResolvableCommandConfig(t *testing.T) string {
t.Helper()
return writeCommandConfigContent(t, `version: 4
pipelines:
demo:
input: seriatim
artifacts:
spells:
extract: dnd/spells
`)
}
func writeCommandConfigContent(t *testing.T, content string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
return path
}

16
internal/cli/contract_test_helpers_test.go
View File

@@ -1,16 +0,0 @@
package cli
import (
"strings"
"testing"
)
func replaceRequiredOnce(t *testing.T, input, old, replacement string) string {
t.Helper()
if count := strings.Count(input, old); count != 1 {
t.Fatalf("replacement marker %q occurs %d times, want exactly once", old, count)
}
return strings.Replace(input, old, replacement, 1)
}
func emptyLookup(string) (string, bool) { return "", false }

208
internal/cli/dnd_combat_contract_test.go
View File

@@ -1,208 +0,0 @@
package cli
import (
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
combatextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/combatturns"
combatnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/combatturns"
)
func TestProductionCombatConfigurationResolvesTypedLane(t *testing.T) {
components := productionTestComponents(t)
cfg := productionCombatContractConfig()
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-combat", Catalog: catalogFromRegistries(components.registries)})
if err != nil {
t.Fatalf("Resolve() error = %v, want nil", err)
}
if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule {
t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule)
}
if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("artifact lanes = %#v, want one combat lane", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
}
lane := effective.ResolvedPipeline.Steps[0].ArtifactLanes[0]
if lane.ID != "combat" || lane.ArtifactKind != dnd.CombatTurnListKind || lane.Extract.Module != combatextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != combatnormalize.Key {
t.Fatalf("resolved combat lane = %#v, want typed production composition", lane)
}
catalog := catalogFromRegistries(components.registries)
extractSpec, ok := catalog.Extractors.Spec(combatextract.Key)
if !ok || !reflect.DeepEqual(extractSpec.Requires, []string{"chunks", "source.transcript"}) || !reflect.DeepEqual(extractSpec.Provides, []string{"dnd.combat_turns"}) {
t.Fatalf("combat extractor spec = %#v, want source and artifact capabilities", extractSpec)
}
normalizeSpec, ok := catalog.Normalizers.SpecForArtifact(combatnormalize.Key, dnd.CombatTurnListKind)
if !ok || !reflect.DeepEqual(normalizeSpec.Requires, []string{"merged"}) || !reflect.DeepEqual(normalizeSpec.Provides, []string{"normalized"}) {
t.Fatalf("combat normalizer spec = %#v, want merged/normalized capabilities", normalizeSpec)
}
mergeSpec, ok := catalog.Mergers.SpecForArtifact(pipeline.DefaultMergeModule, dnd.CombatTurnListKind)
if !ok || !reflect.DeepEqual(mergeSpec.Provides, []string{"merged"}) {
t.Fatalf("combat merger spec = %#v, want merged capability", mergeSpec)
}
codecSpec, ok := catalog.ArtifactCodecs.Spec(dnd.CombatTurnListKind)
if !ok || codecSpec.Schema.ID != "notarius.dnd.combat_turns" || codecSpec.Schema.Version != "v1" {
t.Fatalf("combat codec spec = %#v, want compatible durable schema", codecSpec)
}
if !hasReferenceSlot(extractSpec.ReferenceSlots, "npcs") || !hasReferenceSlot(extractSpec.ReferenceSlots, "scene_descriptions") || !hasReferenceSlot(normalizeSpec.ReferenceSlots, "npcs") {
t.Fatalf("combat reference slots = %#v / %#v, want extraction scene and NPC slots plus normalization NPC slot", extractSpec.ReferenceSlots, normalizeSpec.ReferenceSlots)
}
sceneSlot := referenceSlot(extractSpec.ReferenceSlots, "scene_descriptions")
if !sceneSlot.Required || !reflect.DeepEqual(sceneSlot.AcceptedMediaTypes, []string{"application/json"}) || !reflect.DeepEqual(sceneSlot.AcceptedArtifactKinds, []contracts.ArtifactKind{dnd.SceneDescriptionListKind}) || sceneSlot.MaxBytes != 1048576 {
t.Fatalf("scene description slot = %#v, want required approved scene artifact", sceneSlot)
}
wantExtractChain := []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("extract/dnd/combat-turns/shape"),
pipeline.Binding("extract/dnd/combat-turns/source_refs"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/combat-turns/source_relatedness"),
}
wantNormalizeChain := []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("extract/dnd/combat-turns/shape"),
pipeline.Binding("normalize/dnd/combat-turns/invariants"),
pipeline.Binding("extract/dnd/combat-turns/source_refs"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/combat-turns/source_relatedness"),
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageExtract, combatextract.Key); !reflect.DeepEqual(got, wantExtractChain) {
t.Fatalf("combat extract chain = %#v, want %#v", got, wantExtractChain)
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageNormalize, combatnormalize.Key); !reflect.DeepEqual(got, wantNormalizeChain) {
t.Fatalf("combat normalize chain = %#v, want %#v", got, wantNormalizeChain)
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageMerge, pipeline.DefaultMergeModule); len(got) != 0 {
t.Fatalf("combat merge chain = %#v, want empty", got)
}
bound, err := cfg.Resolve(config.ResolveInput{
PipelineID: "dnd-combat",
Catalog: catalog,
ReferenceOverrides: []pipeline.ReferenceBinding{
{Stage: pipeline.StageExtract, LaneID: "combat", SlotName: "npcs", Source: "npc-run/lanes/npcs.json", BindingSource: contracts.ReferenceBindingSourceCLI},
{Stage: pipeline.StageNormalize, LaneID: "combat", SlotName: "npcs", Source: "npc-run/lanes/npcs.json", BindingSource: contracts.ReferenceBindingSourceCLI},
},
})
if err != nil {
t.Fatalf("Resolve(bound references) error = %v, want nil", err)
}
boundLane := bound.ResolvedPipeline.Steps[0].ArtifactLanes[0]
if len(boundLane.ExtractReferences.Bindings) != 2 || len(boundLane.NormalizeReferences.Bindings) != 1 || !hasReferenceBinding(boundLane.ExtractReferences.Bindings, "npcs") || !hasReferenceBinding(boundLane.ExtractReferences.Bindings, "scene_descriptions") || !hasReferenceBinding(boundLane.NormalizeReferences.Bindings, "npcs") {
t.Fatalf("bound combat references = %#v / %#v, want extraction scene and NPC bindings plus normalization NPC binding", boundLane.ExtractReferences, boundLane.NormalizeReferences)
}
}
func TestProductionCombatConfigurationRequiresSceneDescriptions(t *testing.T) {
components := productionTestComponents(t)
cfg := productionCombatContractConfig()
profile := cfg.Pipelines["dnd-combat"]
profile.References = nil
cfg.Pipelines["dnd-combat"] = profile
if _, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-combat", Catalog: catalogFromRegistries(components.registries)}); err == nil || !strings.Contains(err.Error(), "scene_descriptions") || !strings.Contains(err.Error(), "required") {
t.Fatalf("Resolve() error = %v, want required scene reference failure", err)
}
}
func TestProductionCombatConfigurationRejectsLooseOptionsAndLaneValidators(t *testing.T) {
components := productionTestComponents(t)
resolve := func(mutate func(*pipeline.PipelineProfile)) error {
cfg := productionCombatContractConfig()
profile := cfg.Pipelines["dnd-combat"]
mutate(&profile)
cfg.Pipelines["dnd-combat"] = profile
_, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-combat", Catalog: catalogFromRegistries(components.registries)})
return err
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["combat"]
lane.Extract.Options = map[string]any{"unexpected": true}
profile.Artifacts["combat"] = lane
}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("unknown extractor option error = %v, want strict option rejection", err)
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["combat"]
lane.Normalize.Options = map[string]any{"unexpected": true}
profile.Artifacts["combat"] = lane
}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("unknown normalizer option error = %v, want strict option rejection", err)
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["combat"]
lane.Validators = []pipeline.ModuleBinding{pipeline.Binding("generic/always_accept")}
profile.Artifacts["combat"] = lane
}); err == nil || !strings.Contains(err.Error(), "artifact lane level") {
t.Fatalf("lane-level validator error = %v, want invalid placement rejection", err)
}
}
func TestProductionCombatConfigurationResolvesTypedUnconditionalValidators(t *testing.T) {
components := productionTestComponents(t)
cfg := productionCombatContractConfig()
profile := cfg.Pipelines["dnd-combat"]
lane := profile.Artifacts["combat"]
lane.Extract.Validators = pipeline.ValidatorOverride{Set: true, Validators: []pipeline.ModuleBinding{pipeline.Binding("generic/always_accept")}}
lane.Normalize.Validators = pipeline.ValidatorOverride{Set: true, Validators: []pipeline.ModuleBinding{pipeline.Binding("generic/always_reject")}}
profile.Artifacts["combat"] = lane
cfg.Pipelines["dnd-combat"] = profile
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-combat", Catalog: catalogFromRegistries(components.registries)})
if err != nil {
t.Fatalf("Resolve() error = %v, want typed unconditional validators to resolve", err)
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageExtract, combatextract.Key); !reflect.DeepEqual(got, []pipeline.ModuleBinding{pipeline.Binding("generic/always_accept")}) {
t.Fatalf("extract override chain = %#v, want typed always-accept", got)
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageNormalize, combatnormalize.Key); !reflect.DeepEqual(got, []pipeline.ModuleBinding{pipeline.Binding("generic/always_reject")}) {
t.Fatalf("normalize override chain = %#v, want typed always-reject", got)
}
}
func productionCombatContractConfig() config.Config {
cfg := config.Default()
cfg.Pipelines["dnd-combat"] = pipeline.PipelineProfile{
ID: "dnd-combat",
Input: pipeline.Binding("seriatim"),
Chunk: pipeline.Binding(pipeline.DefaultChunkModule),
References: map[string]pipeline.ReferenceSource{"scene_descriptions": pipeline.ExternalReference("scenes.json")},
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"combat": {
Extract: pipeline.ModuleBinding{Module: combatextract.Key, Retries: 2},
Normalize: pipeline.Binding(combatnormalize.Key),
},
},
}
return cfg
}
func hasReferenceSlot(slots []contracts.ReferenceSlot, name string) bool {
for _, slot := range slots {
if slot.Name == name {
return true
}
}
return false
}
func hasReferenceBinding(bindings []pipeline.ReferenceBinding, name string) bool {
for _, binding := range bindings {
if binding.SlotName == name {
return true
}
}
return false
}
func referenceSlot(slots []contracts.ReferenceSlot, name string) contracts.ReferenceSlot {
for _, slot := range slots {
if slot.Name == name {
return slot
}
}
return contracts.ReferenceSlot{}
}

135
internal/cli/dnd_interactions_contract_test.go
View File

@@ -1,135 +0,0 @@
package cli
import (
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
interactioncodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcinteractions"
interactionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcinteractions"
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
interactionnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcinteractions"
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
)
func TestProductionNPCInteractionPipelineResolvesAndPrepares(t *testing.T) {
components := productionTestComponents(t)
resolved, err := pipeline.ResolvePipeline(npcInteractionProfile(pipeline.GeneratedReference("npcs", "npcs")), pipeline.ResolveOptions{}, catalogFromRegistries(components.registries))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v", err)
}
if len(resolved.Steps) != 2 || len(resolved.Steps[1].ArtifactLanes) != 1 {
t.Fatalf("resolved pipeline = %#v", resolved)
}
lane := resolved.Steps[1].ArtifactLanes[0]
if lane.ArtifactKind != dnd.NPCInteractionListKind || lane.Extract.Module != interactionextract.Key || lane.Normalize.Module != interactionnormalize.Key {
t.Fatalf("interaction lane = %#v", lane)
}
for _, bindings := range [][]pipeline.ReferenceBinding{lane.ExtractReferences.Bindings, lane.NormalizeReferences.Bindings} {
if len(bindings) != 1 || bindings[0].SlotName != "npcs" || bindings[0].Artifact == nil || bindings[0].Artifact.Step != "npcs" || bindings[0].Artifact.Lane != "npcs" {
t.Fatalf("generated bindings = %#v", bindings)
}
}
if _, err := pipeline.Prepare(resolved, components.registries, pipeline.ModuleDependencies{LLM: &productionFakeLLMClient{}}); err != nil {
t.Fatalf("Prepare() error = %v", err)
}
catalog := catalogFromRegistries(components.registries)
codecSpec, ok := catalog.ArtifactCodecs.Spec(dnd.NPCInteractionListKind)
if !ok || codecSpec.Schema.ID != interactioncodec.SchemaID || codecSpec.Schema.Version != interactioncodec.SchemaVersion {
t.Fatalf("NPC interaction codec spec = %#v", codecSpec)
}
}
func TestProductionNPCInteractionReferencesRequireEarlierCompatibleProducer(t *testing.T) {
components := productionTestComponents(t)
catalog := catalogFromRegistries(components.registries)
laterProfile := npcInteractionProfile(pipeline.GeneratedReference("npcs", "npcs"))
laterProfile.Steps[0].ID = "seed"
laterProfile.Steps[0].Artifacts["seed"] = laterProfile.Steps[0].Artifacts["npcs"]
delete(laterProfile.Steps[0].Artifacts, "npcs")
laterProfile.Steps = append(laterProfile.Steps, pipeline.PipelineStepProfile{ID: "future", Artifacts: map[string]pipeline.ArtifactLaneProfile{
"npcs": {Extract: pipeline.Binding(npcextract.Key), Normalize: pipeline.Binding(npcnormalize.Key)},
}})
laterProfile.Steps[1].References["npcs"] = pipeline.GeneratedReference("future", "npcs")
tests := []struct {
name string
profile pipeline.PipelineProfile
want string
}{
{name: "missing", profile: npcInteractionProfile(pipeline.ReferenceSource{}), want: "source must not be empty"},
{name: "same step", profile: npcInteractionProfile(pipeline.GeneratedReference("interactions", "interactions")), want: "earlier step"},
{name: "later step", profile: laterProfile, want: "earlier step"},
{name: "wrong artifact kind", profile: npcInteractionProfile(pipeline.GeneratedReference("npcs", "npcs")), want: "does not accept artifact kind"},
}
tests[3].profile.Steps[0].Artifacts["npcs"] = pipeline.ArtifactLaneProfile{Extract: pipeline.Binding("dnd/spells")}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
_, err := pipeline.ResolvePipeline(test.profile, pipeline.ResolveOptions{}, catalog)
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("ResolvePipeline() error = %v, want %q", err, test.want)
}
})
}
}
func TestProductionNPCInteractionReferencesRejectIncompatibleExternalRegistries(t *testing.T) {
components := productionTestComponents(t)
catalog := catalogFromRegistries(components.registries)
root := t.TempDir()
for _, test := range []struct {
name string
file string
content string
prepare bool
want string
}{
{name: "media type", file: "registry.txt", content: "not JSON", want: "media type"},
{name: "artifact schema", file: "registry.json", content: `{"npcs":[{"name":"missing required fields"}]}`, prepare: true, want: "NPC registry"},
} {
t.Run(test.name, func(t *testing.T) {
path := filepath.Join(root, test.file)
if err := os.WriteFile(path, []byte(test.content), 0o600); err != nil {
t.Fatal(err)
}
resolved, err := pipeline.ResolvePipeline(npcInteractionProfile(pipeline.ExternalReference(path)), pipeline.ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v", err)
}
materialized, _, err := pipeline.MaterializeReferences(resolved, catalog, pipeline.ReferenceMaterializationOptions{})
if !test.prepare {
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("MaterializeReferences() error = %v, want %q", err, test.want)
}
return
}
if err != nil {
t.Fatalf("MaterializeReferences() error = %v", err)
}
if _, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: &productionFakeLLMClient{}}); err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Prepare() error = %v, want %q", err, test.want)
}
})
}
}
func npcInteractionProfile(reference pipeline.ReferenceSource) pipeline.PipelineProfile {
profile := pipeline.PipelineProfile{
ID: "dnd-npc-interactions",
Input: pipeline.Binding("seriatim"),
Chunk: pipeline.ModuleBinding{Module: "generic", Options: map[string]any{"max_units": 1}},
Output: pipeline.Binding("json"),
Steps: []pipeline.PipelineStepProfile{
{ID: "npcs", Artifacts: map[string]pipeline.ArtifactLaneProfile{
"npcs": {Extract: pipeline.Binding(npcextract.Key), Normalize: pipeline.Binding(npcnormalize.Key)},
}},
{ID: "interactions", References: map[string]pipeline.ReferenceSource{"npcs": reference}, Artifacts: map[string]pipeline.ArtifactLaneProfile{
"interactions": {Extract: pipeline.Binding(interactionextract.Key), Normalize: pipeline.Binding(interactionnormalize.Key)},
}},
},
}
return profile
}

151
internal/cli/dnd_npc_contract_test.go
View File

@@ -1,151 +0,0 @@
package cli
import (
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
)
func TestProductionNPCConfigurationResolvesTypedLane(t *testing.T) {
components := productionTestComponents(t)
catalog := catalogFromRegistries(components.registries)
cfg := productionNPCContractConfig()
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalog})
if err != nil {
t.Fatalf("Resolve() error = %v, want nil", err)
}
if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule {
t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule)
}
if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 {
t.Fatalf("artifact lanes = %#v, want one NPC lane", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
}
lane := effective.ResolvedPipeline.Steps[0].ArtifactLanes[0]
if lane.ID != "npcs" || lane.ArtifactKind != dnd.NPCListKind || lane.Extract.Module != npcextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != npcnormalize.Key {
t.Fatalf("resolved NPC lane = %#v, want typed production composition", lane)
}
if len(lane.ExtractReferences.Bindings) != 0 || len(lane.NormalizeReferences.Bindings) != 0 {
t.Fatalf("unbound NPC references = %#v / %#v, want none", lane.ExtractReferences, lane.NormalizeReferences)
}
extractSpec, ok := catalog.Extractors.Spec(npcextract.Key)
if !ok || !reflect.DeepEqual(extractSpec.Requires, []string{"chunks", "source.transcript"}) || !reflect.DeepEqual(extractSpec.Provides, []string{"dnd.npcs"}) {
t.Fatalf("NPC extractor spec = %#v, want source and artifact capabilities", extractSpec)
}
mergeSpec, ok := catalog.Mergers.SpecForArtifact(pipeline.DefaultMergeModule, dnd.NPCListKind)
if !ok || !reflect.DeepEqual(mergeSpec.Provides, []string{"merged"}) {
t.Fatalf("NPC merger spec = %#v, want merged capability", mergeSpec)
}
normalizeSpec, ok := catalog.Normalizers.SpecForArtifact(npcnormalize.Key, dnd.NPCListKind)
if !ok || !reflect.DeepEqual(normalizeSpec.Requires, []string{"merged"}) || !reflect.DeepEqual(normalizeSpec.Provides, []string{"normalized"}) {
t.Fatalf("NPC normalizer spec = %#v, want merged/normalized capabilities", normalizeSpec)
}
codecSpec, ok := catalog.ArtifactCodecs.Spec(dnd.NPCListKind)
if !ok || codecSpec.Kind != dnd.NPCListKind || codecSpec.Schema.ID != npccodec.SchemaID || codecSpec.Schema.Version != npccodec.SchemaVersion {
t.Fatalf("NPC codec spec = %#v, want typed v1 durable schema", codecSpec)
}
wantExtractChain := []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("extract/dnd/npcs/shape"),
pipeline.Binding("extract/dnd/npcs/source_refs"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
}
wantNormalizeChain := []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("extract/dnd/npcs/shape"),
pipeline.Binding("normalize/dnd/npcs/identity"),
pipeline.Binding("extract/dnd/npcs/source_refs"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageExtract, npcextract.Key); !reflect.DeepEqual(got, wantExtractChain) {
t.Fatalf("NPC extract chain = %#v, want %#v", got, wantExtractChain)
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageNormalize, npcnormalize.Key); !reflect.DeepEqual(got, wantNormalizeChain) {
t.Fatalf("NPC normalize chain = %#v, want %#v", got, wantNormalizeChain)
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageMerge, pipeline.DefaultMergeModule); len(got) != 0 {
t.Fatalf("NPC merge chain = %#v, want empty", got)
}
}
func TestProductionNPCConfigurationValidatesOptionsReferencesAndPlacement(t *testing.T) {
components := productionTestComponents(t)
resolve := func(mutate func(*pipeline.PipelineProfile)) error {
cfg := productionNPCContractConfig()
profile := cfg.Pipelines["dnd-session"]
mutate(&profile)
cfg.Pipelines["dnd-session"] = profile
_, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(components.registries)})
return err
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["npcs"]
lane.Extract.Options = map[string]any{"unexpected": true}
profile.Artifacts["npcs"] = lane
}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("unknown extractor option error = %v, want strict option rejection", err)
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["npcs"]
lane.Normalize.Options = map[string]any{"unexpected": true}
profile.Artifacts["npcs"] = lane
}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("unknown normalizer option error = %v, want strict option rejection", err)
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
profile.References = pipeline.ExternalReferenceMap(map[string]string{
"players": "players.txt",
"party": "party.txt",
"glossary": "glossary.txt",
})
}); err != nil {
t.Fatalf("optional NPC references error = %v, want resolution success", err)
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["npcs"]
lane.Validators = []pipeline.ModuleBinding{pipeline.Binding("normalize/dnd/npcs/identity")}
profile.Artifacts["npcs"] = lane
}); err == nil || !strings.Contains(err.Error(), "artifact lane level") {
t.Fatalf("lane-level validator error = %v, want invalid placement rejection", err)
}
}
func productionNPCContractConfig() config.Config {
cfg := config.Default()
cfg.Pipelines["dnd-session"] = pipeline.PipelineProfile{
ID: "dnd-session",
Input: pipeline.Binding("seriatim"),
Chunk: pipeline.Binding(pipeline.DefaultChunkModule),
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"npcs": {
Extract: pipeline.ModuleBinding{Module: npcextract.Key, Retries: 2},
Normalize: pipeline.Binding(npcnormalize.Key),
},
},
}
return cfg
}
func validatorChain(resolved pipeline.ResolvedPipeline, stage pipeline.ModuleStage, module string) []pipeline.ModuleBinding {
for _, chain := range resolved.ValidatorChains {
if chain.Stage == stage && chain.ModuleKey == module {
bindings := make([]pipeline.ModuleBinding, len(chain.Validators))
for index, validator := range chain.Validators {
bindings[index] = validator.Binding
}
return bindings
}
}
return nil
}

147
internal/cli/dnd_scene_descriptions_contract_test.go
View File

@@ -1,147 +0,0 @@
package cli
import (
"context"
"encoding/json"
"fmt"
"reflect"
"strings"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
scenecodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/scenedescriptions"
sceneextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
scenenormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/scenedescriptions"
)
func TestProductionSceneDescriptionWorkflow(t *testing.T) {
components := productionTestComponents(t)
cfg := config.Default()
cfg.Pipelines["scene-descriptions"] = pipeline.PipelineProfile{
ID: "scene-descriptions",
Input: pipeline.Binding("seriatim"),
Chunk: pipeline.ModuleBinding{Module: "generic", Options: map[string]any{"max_units": 1}},
Output: pipeline.Binding("json"),
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"scene-descriptions": {
Extract: pipeline.ModuleBinding{Module: sceneextract.Key, LLMProfile: "scene-description-profile"},
Normalize: pipeline.Binding(scenenormalize.Key),
},
},
}
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "scene-descriptions", Catalog: catalogFromRegistries(components.registries)})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
lane := effective.ResolvedPipeline.Steps[0].ArtifactLanes[0]
if lane.ArtifactKind != dnd.SceneDescriptionListKind || lane.Extract.Module != sceneextract.Key || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != scenenormalize.Key {
t.Fatalf("resolved lane = %#v, want production scene-description composition", lane)
}
if len(lane.ExtractReferences.Bindings) != 0 || len(lane.NormalizeReferences.Bindings) != 0 {
t.Fatalf("resolved references = %#v / %#v, want no generated or required references", lane.ExtractReferences, lane.NormalizeReferences)
}
llmClient := &sceneDescriptionLLM{}
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, components.registries, pipeline.ModuleDependencies{LLM: llmClient})
if err != nil {
t.Fatalf("Prepare() error = %v", err)
}
output, err := pipeline.New().Run(context.Background(), pipeline.RunInput{
Prepared: prepared,
RawInput: readRepositoryFile(t, "examples", "seriatim-minimal-transcript.json"),
ChunkCacheMode: pipeline.ChunkCacheBypass,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if output.Manifest.ValidationStatus != "approved" || len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
t.Fatalf("run output = %#v, want one approved normalized artifact", output)
}
wantProfiles := []artifacts.LLMProfileManifest{{
ID: "scene-description-profile",
Provider: "promptkit",
Model: "deterministic",
}}
if !reflect.DeepEqual(output.Manifest.LLMProfiles, wantProfiles) {
t.Fatalf("manifest LLM profiles = %#v, want %#v", output.Manifest.LLMProfiles, wantProfiles)
}
normalizedOutput := output.NormalizeOutputs[0]
if normalizedOutput.NormalizerKey != scenenormalize.Key || normalizedOutput.Artifact.Kind != dnd.SceneDescriptionListKind || normalizedOutput.Artifact.Schema.ID != scenecodec.SchemaID || normalizedOutput.Artifact.Schema.Name != scenecodec.SchemaName || normalizedOutput.Artifact.Schema.Version != scenecodec.SchemaVersion {
t.Fatalf("normalized output = %#v, want registered durable scene-description schema", normalizedOutput)
}
var value dnd.SceneDescriptionList
if err := json.Unmarshal(normalizedOutput.Artifact.Content, &value); err != nil {
t.Fatalf("decode normalized artifact: %v", err)
}
want := dnd.SceneDescriptionList{Scenes: []dnd.SceneDescription{
{ID: "chunk-000001", SourceRef: source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}, Kind: dnd.SceneKindNarrative, Title: "Aria casts Cure Wounds", Summary: "Aria casts Cure Wounds."},
{ID: "chunk-000002", SourceRef: source.SourceRef{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}, Kind: dnd.SceneKindCombat, Title: "Bandit mage casts Shield", Summary: "The bandit mage casts Shield."},
}}
if !reflect.DeepEqual(value, want) {
t.Fatalf("normalized scene descriptions = %#v, want %#v", value, want)
}
durable := decodeAssembledOutput[dnd.SceneDescriptionList](t, output.OutputFiles, "lanes/scene-descriptions.json")
if !reflect.DeepEqual(durable, want) {
t.Fatalf("durable output payload = %#v, want %#v", durable, want)
}
if len(output.Warnings) != 0 {
t.Fatalf("warnings = %#v, want grounded descriptions without warnings", output.Warnings)
}
}
type sceneDescriptionLLM struct {
mu sync.Mutex
profile *artifacts.LLMProfileManifest
}
func (client *sceneDescriptionLLM) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
if err := ctx.Err(); err != nil {
return contracts.StructuredCompletionResponse{}, err
}
if req.PromptID != sceneextract.PromptID {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected prompt %q", req.PromptID)
}
transcript := string(req.Inputs["transcript"].Content)
var content string
switch {
case strings.Contains(transcript, "Cure Wounds"):
content = `{"kind":"narrative","title":" Aria casts Cure Wounds ","summary":" Aria casts Cure Wounds. "}`
case strings.Contains(transcript, "Shield"):
content = `{"kind":"combat","title":"Bandit mage casts Shield","summary":"The bandit mage casts Shield."}`
default:
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected transcript material %q", transcript)
}
if err := json.Unmarshal([]byte(content), out); err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate structured response: %w", err)
}
profile := artifacts.LLMProfileManifest{
ID: req.ProfileID,
Provider: "promptkit",
Model: "deterministic",
}
client.mu.Lock()
client.profile = &profile
client.mu.Unlock()
return contracts.StructuredCompletionResponse{
Content: []byte(content),
Provider: profile.Provider,
Model: profile.Model,
ProfileID: profile.ID,
}, nil
}
func (client *sceneDescriptionLLM) LLMProfileManifests() []artifacts.LLMProfileManifest {
client.mu.Lock()
defer client.mu.Unlock()
if client.profile == nil {
return nil
}
return []artifacts.LLMProfileManifest{*client.profile}
}

266
internal/cli/example_contract_test.go
View File

@@ -1,266 +0,0 @@
package cli
import (
"context"
"encoding/json"
"os"
"path/filepath"
"sort"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
)
func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
components := productionTestComponents(t)
for _, example := range maintainedExampleFiles(t) {
t.Run(example.name, func(t *testing.T) {
cfg := loadMaintainedExample(t, example.path)
raw, err := os.ReadFile(example.transcriptPath)
if err != nil {
t.Fatalf("read maintained transcript %q: %v", example.transcriptPath, err)
}
document, err := transcript.New().Parse(context.Background(), contracts.ParseRequest{
Path: example.transcriptPath,
Raw: raw,
})
if err != nil {
t.Fatalf("parse maintained transcript %q: %v", example.transcriptPath, err)
}
if len(document.Units) == 0 {
t.Fatalf("maintained transcript %q has no parsed units", example.transcriptPath)
}
for _, pipelineID := range example.pipelineIDs {
effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, pipelineID))
if err != nil {
t.Fatalf("resolve maintained example %q: %v", pipelineID, err)
}
materialized, _, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{
ConfigPath: example.path,
WorkingDir: filepath.Dir(example.path),
})
if err != nil {
t.Fatalf("materialize maintained example references for %q: %v", pipelineID, err)
}
if example.name == "complete" {
if got := exampleStepLaneIDs(materialized); strings.Join(got, "|") != "describe-session:item-events,npcs,scene-descriptions|extract-events:combat-turns,npc-interactions,spells" {
t.Fatalf("complete example steps and lanes = %v, want every D&D extractor in the documented two-step composition", got)
}
spellLane := referenceContractLane(t, materialized, "spells")
if len(spellLane.ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 ||
len(spellLane.NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 {
t.Fatalf("complete example spell catalog reference was not materialized: %#v", spellLane)
}
itemEventLane := referenceContractLane(t, materialized, "item-events")
for _, references := range []pipeline.ResolvedReferenceTarget{itemEventLane.ExtractReferences, itemEventLane.NormalizeReferences} {
if _, found := references.ReferenceSet.Slots["npcs"]; found {
t.Fatalf("item event lane unexpectedly depends on generated NPCs: %#v", itemEventLane)
}
if _, found := references.ReferenceSet.Slots["scene_descriptions"]; found {
t.Fatalf("item event lane unexpectedly depends on generated scene descriptions: %#v", itemEventLane)
}
}
}
}
var stdout, stderr strings.Builder
code := RunWithOptions([]string{"pipelines", "list", "--config", example.path}, &stdout, &stderr, productionOptionsFromComponents(components))
if code != 0 || stdout.String() != strings.Join(example.pipelineIDs, "\n")+"\n" || stderr.Len() != 0 {
t.Fatalf("pipelines list: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
}
func TestMaintainedConfigurationExampleSet(t *testing.T) {
entries, err := os.ReadDir(repositoryPath("examples"))
if err != nil {
t.Fatal(err)
}
var names []string
for _, entry := range entries {
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".config.yml") {
names = append(names, entry.Name())
}
}
sort.Strings(names)
if got := strings.Join(names, ","); got != "dnd-complete.config.yml,dnd-minimal.config.yml" {
t.Fatalf("maintained configuration examples = %q, want only the minimal and complete D&D examples", got)
}
profileEntries, err := os.ReadDir(repositoryPath("examples", "profiles"))
if err != nil {
t.Fatal(err)
}
names = names[:0]
for _, entry := range profileEntries {
if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".yml") {
names = append(names, entry.Name())
}
}
sort.Strings(names)
if got := strings.Join(names, ","); got != "dnd-extraction.yml" {
t.Fatalf("maintained operator profiles = %q, want dnd-extraction.yml", got)
}
}
func TestMaintainedExamplesValidateEffectiveProfilesOffline(t *testing.T) {
t.Chdir(repositoryPath())
t.Setenv("OPENROUTER_API_KEY", "")
for _, example := range maintainedExampleFiles(t) {
t.Run(example.name, func(t *testing.T) {
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"config", "validate", "--config", example.path, "--pipeline", "dnd-session",
}, &stdout, &stderr, Options{})
if code != 0 || stderr.Len() != 0 || !strings.Contains(stdout.String(), `valid for pipeline "dnd-session"`) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
}
func exampleStepLaneIDs(resolved pipeline.ResolvedPipeline) []string {
result := make([]string, 0, len(resolved.Steps))
for _, step := range resolved.Steps {
laneIDs := make([]string, 0, len(step.ArtifactLanes))
for _, lane := range step.ArtifactLanes {
laneIDs = append(laneIDs, lane.ID)
}
sort.Strings(laneIDs)
result = append(result, step.ID+":"+strings.Join(laneIDs, ","))
}
return result
}
func TestMaintainedMinimalInvocationProducesJSONBundle(t *testing.T) {
outputRoot := filepath.Join(t.TempDir(), "output")
fake := &productionFakeLLMClient{}
options := productionRunOptions(t, fake)
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"run", "dnd-session",
"--config", repositoryPath("examples", "dnd-minimal.config.yml"),
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
"--only", "spells", "--chunk_cache", "bypass", "--output-dir", outputRoot,
}, &stdout, &stderr, options)
if code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if !strings.Contains(stdout.String(), `pipeline "dnd-session"`) || !strings.Contains(stdout.String(), "outputs=1 rejected=0") {
t.Fatalf("stdout=%q, want completed pipeline and counts", stdout.String())
}
runRoot := filepath.Join(outputRoot, productionRunID)
index := readProductionJSON[exampleOutputIndex](t, filepath.Join(runRoot, "index.json"))
if index.ManifestFile != "manifest.json" || index.RejectedFile != "rejected.json" || index.WarningsFile != "warnings.json" || len(index.OutputFiles) != 1 {
t.Fatalf("index = %#v, want one spells output and fixed companion files", index)
}
entry := index.OutputFiles[0]
if entry.LaneID != "spells" || entry.File != "lanes/spells.json" || entry.MediaType != "application/json" || entry.SchemaID != "notarius.dnd.spells" || entry.SchemaVersion != "v1" {
t.Fatalf("index output entry = %#v, want spells JSON contract", entry)
}
manifest := readProductionJSON[artifacts.RunManifest](t, filepath.Join(runRoot, "manifest.json"))
if manifest.PipelineID != "dnd-session" || manifest.InputModule != "seriatim" || manifest.Chunker != "generic" || manifest.OutputEncoder != "json" || manifest.ValidationStatus != "approved" || manifest.ChunkPlan == nil || manifest.ChunkPlan.Action != "bypassed" {
t.Fatalf("manifest = %#v, want approved minimal run", manifest)
}
if len(manifest.ArtifactLanes) != 1 {
t.Fatalf("manifest lanes = %#v, want exactly spells", manifest.ArtifactLanes)
}
lane := manifest.ArtifactLanes[0]
if lane.ID != "spells" || lane.Extractor != "dnd/spells" || lane.Merger != "appendorder" || lane.Normalizer != spellnormalize.Key {
t.Fatalf("manifest lane = %#v, want production spells composition", lane)
}
if len(manifest.References) != 0 {
t.Fatalf("base-only manifest references = %#v, want no overlay provenance", manifest.References)
}
extractorMetadata, ok := lane.Metadata["extractor"].(map[string]any)
if !ok || len(stringValues(extractorMetadata["catalog_overlay_ids"])) != 0 {
t.Fatalf("base-only extractor metadata = %#v, want no overlay IDs", lane.Metadata)
}
artifact := readProductionJSON[dnd.SpellList](t, filepath.Join(runRoot, entry.File))
if len(artifact.SpellCasts) != 1 || artifact.SpellCasts[0].Spell != "Cure Wounds" || artifact.SpellCasts[0].SourceRefs[0].SourceID != "session-alpha" {
t.Fatalf("artifact = %#v, want one source-linked Cure Wounds cast", artifact)
}
rejected := readProductionJSON[struct {
Rejected []json.RawMessage `json:"rejected"`
}](t, filepath.Join(runRoot, "rejected.json"))
if len(rejected.Rejected) != 0 {
t.Fatalf("rejected = %#v, want empty rejection list", rejected.Rejected)
}
warnings := readProductionJSON[struct {
Warnings []json.RawMessage `json:"warnings"`
}](t, filepath.Join(runRoot, "warnings.json"))
if len(warnings.Warnings) != 0 {
t.Fatalf("warnings = %#v, want empty warning list", warnings.Warnings)
}
}
func TestMaintainedMalformedInputOnlyRecordsDebugFailureWhenRequested(t *testing.T) {
malformed := filepath.Join(t.TempDir(), "malformed.json")
if err := os.WriteFile(malformed, []byte("{not valid json"), 0o600); err != nil {
t.Fatal(err)
}
for _, debug := range []bool{false, true} {
name := "without debug"
if debug {
name = "with debug"
}
t.Run(name, func(t *testing.T) {
outputRoot := filepath.Join(t.TempDir(), "output")
debugRoot := filepath.Join(t.TempDir(), "debug")
options := productionRunOptions(t, &productionFakeLLMClient{})
args := []string{
"run", "dnd-session",
"--config", repositoryPath("examples", "dnd-minimal.config.yml"),
"--input", malformed, "--chunk_cache", "bypass", "--output-dir", outputRoot,
}
if debug {
args = append(args, "--debug", "--debug-dir", debugRoot)
}
var stdout, stderr strings.Builder
code := RunWithOptions(args, &stdout, &stderr, options)
if code != 1 || stdout.Len() != 0 || !strings.Contains(stderr.String(), "parse input") {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertAbsent(t, outputRoot)
if !debug {
assertAbsent(t, debugRoot)
return
}
bundle := onlyChildDir(t, debugRoot)
report := readProductionJSON[debugbundle.RunReport](t, filepath.Join(bundle, "summary", "run-report.json"))
if report.Succeeded || report.PipelineID != "dnd-session" {
t.Fatalf("failure report = %#v, want failed dnd-session report", report)
}
})
}
}
type exampleOutputIndex struct {
ManifestFile string `json:"manifest_file"`
OutputFiles []exampleOutputIndexEntry `json:"output_files"`
RejectedFile string `json:"rejected_file"`
WarningsFile string `json:"warnings_file"`
}
type exampleOutputIndexEntry struct {
LaneID string `json:"lane_id"`
MediaType string `json:"media_type"`
File string `json:"file"`
SchemaID string `json:"schema_id"`
SchemaVersion string `json:"schema_version"`
}
func resolveInputForMaintainedExample(components productionComponents, pipelineID string) config.ResolveInput {
return config.ResolveInput{PipelineID: pipelineID, Catalog: catalogFromRegistries(components.registries)}
}

80
internal/cli/npc_registry_contract_test.go
View File

@@ -1,80 +0,0 @@
package cli
import (
"context"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *testing.T) {
components := productionTestComponents(t)
npcPath := filepath.Join(t.TempDir(), "npcs.json")
if err := os.WriteFile(npcPath, []byte(strings.Repeat("x", 1048577)), 0o600); err != nil {
t.Fatal(err)
}
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
content := fmt.Sprintf(`version: 4
cache:
chunk_plans:
mode: bypass
checkpoints:
enabled: true
directory: %q
pipelines:
dnd-session:
input: seriatim
artifacts:
spells:
extract:
module: dnd/spells
references:
npcs: %q
normalize: dnd/spells
`, checkpointRoot, npcPath)
configPath := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(configPath, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
llmConstructed := false
chunkStoreConstructed := false
options := Options{
Catalog: catalogFromRegistries(components.registries),
Registries: components.registries,
LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
llmConstructed = true
return nil, nil, errors.New("LLM client must not be constructed")
},
ChunkPlanStoreFactory: func(string) (pipeline.ChunkPlanStore, error) {
chunkStoreConstructed = true
return nil, errors.New("chunk-plan store must not be constructed")
},
}
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"run", "dnd-session", "--config", configPath,
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
"--chunk_cache", "bypass", "--output-dir", t.TempDir(),
}, &stdout, &stderr, options)
for _, fragment := range []string{`pipeline "dnd-session"`, `reference slot "npcs"`, "1048577 bytes", "limit 1048576"} {
if code == 0 || !strings.Contains(stderr.String(), fragment) {
t.Fatalf("RunWithOptions() code = %d stderr = %q, want context fragment %q", code, stderr.String(), fragment)
}
}
if llmConstructed || chunkStoreConstructed {
t.Fatalf("runtime construction = LLM %t, chunk store %t; want materialization failure first", llmConstructed, chunkStoreConstructed)
}
if _, err := os.Stat(checkpointRoot); !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("checkpoint root stat error = %v, want no checkpoint allocation", err)
}
}

1077
internal/cli/production_contract_test.go
View File

File diff suppressed because it is too large Load Diff

46
internal/cli/promptkit_profiles.go
View File

@@ -1,46 +0,0 @@
package cli
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
func validateExplicitPromptKitProfiles(ctx context.Context, cfg config.Config, profileIDs []string, assets *llm.AssetRegistry) error {
if len(profileIDs) == 0 {
return nil
}
inspector, err := llm.NewPromptKitProfileInspector(llm.PromptKitProfileInspectorConfig{
Source: promptKitProfileSourceConfig(cfg),
Assets: assets,
})
if err != nil {
return fmt.Errorf("load PromptKit profiles: %w", err)
}
for _, profileID := range profileIDs {
if _, err := inspector.InspectProfile(ctx, profileID); err != nil {
return err
}
}
return nil
}
func promptKitProfileSourceConfig(cfg config.Config) llm.PromptKitProfileSourceConfig {
return llm.PromptKitProfileSourceConfig{
ProfileDir: cfg.PromptKit.ProfileDir,
ProfileFile: cfg.PromptKit.ProfileFile,
LocalBackend: mapPromptKitLocalBackend(cfg.PromptKit.LocalBackend),
}
}
func mapPromptKitLocalBackend(cfg *config.PromptKitLocalBackendConfig) *llm.PromptKitLocalBackendConfig {
if cfg == nil {
return nil
}
return &llm.PromptKitLocalBackendConfig{
Endpoint: cfg.Endpoint,
ConcurrencyLimit: cfg.ConcurrencyLimit,
}
}

159
internal/cli/promptkit_profiles_test.go
View File

@@ -1,159 +0,0 @@
package cli
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
func TestExplicitPromptKitProfileValidationInspectsProfilesWithoutGeneration(t *testing.T) {
var providerCalls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
providerCalls.Add(1)
}))
defer server.Close()
writeProfile := func(t *testing.T, name, content string) string {
t.Helper()
profilePath := filepath.Join(t.TempDir(), name+".yaml")
if err := os.WriteFile(profilePath, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
return profilePath
}
localProfile := "id: local-profile\nbackend: local\nmodel: local-model\n"
credentialProfile := `id: credential-profile
endpoint: ` + server.URL + `/v1
model: credential-model
api_key_env: NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY
`
t.Setenv("NOTARIUS_PROMPTKIT_PROFILE_INSPECTION_TEST_KEY", "")
tests := []struct {
name string
profilePath string
profileID string
profileDir bool
localBackend bool
canceled bool
wantErr []string
rejectErr []string
}{
{
name: "configured local backend",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
profileDir: true,
localBackend: true,
},
{
name: "missing local backend registration",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
wantErr: []string{`PromptKit profile "local-profile" is invalid or unreadable`},
},
{
name: "absent profile",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "absent-profile",
localBackend: true,
wantErr: []string{`PromptKit profile "absent-profile" is not configured`},
},
{
name: "malformed profile",
profilePath: writeProfile(t, "malformed-profile", "id: malformed-profile\nbackend: [\n"),
profileID: "malformed-profile",
wantErr: []string{`PromptKit profile "malformed-profile" is invalid or unreadable`},
rejectErr: []string{"malformed-profile.yaml", "backend: ["},
},
{
name: "invalid profile source",
profilePath: filepath.Join(t.TempDir(), "missing-profile.yaml"),
profileID: "missing-profile",
wantErr: []string{"load PromptKit profiles", "profile configuration is invalid or unreadable"},
},
{
name: "credential environment intentionally unset",
profilePath: writeProfile(t, "credential-profile", credentialProfile),
profileID: "credential-profile",
},
{
name: "canceled inspection",
profilePath: writeProfile(t, "local-profile", localProfile),
profileID: "local-profile",
localBackend: true,
canceled: true,
wantErr: []string{"context canceled"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := config.Default()
if tt.profileDir {
cfg.PromptKit.ProfileDir = filepath.Dir(tt.profilePath)
} else {
cfg.PromptKit.ProfileFile = tt.profilePath
}
if tt.localBackend {
cfg.PromptKit.LocalBackend = &config.PromptKitLocalBackendConfig{
Endpoint: server.URL + "/v1",
ConcurrencyLimit: 2,
}
}
ctx := context.Background()
if tt.canceled {
var cancel context.CancelFunc
ctx, cancel = context.WithCancel(ctx)
cancel()
}
err := validateExplicitPromptKitProfiles(ctx, cfg, []string{tt.profileID}, nil)
if len(tt.wantErr) == 0 {
if err != nil {
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
}
return
}
if err == nil {
t.Fatal("validateExplicitPromptKitProfiles() error = nil, want failure")
}
if tt.canceled && !errors.Is(err, context.Canceled) {
t.Fatalf("canceled inspection error = %v, want context canceled", err)
}
for _, want := range tt.wantErr {
if !strings.Contains(err.Error(), want) {
t.Fatalf("validation error = %q, want %q", err, want)
}
}
for _, rejected := range append(tt.rejectErr, tt.profilePath) {
if rejected != "" && strings.Contains(err.Error(), rejected) {
t.Fatalf("validation error = %q, must not expose %q", err, rejected)
}
}
})
}
if providerCalls.Load() != 0 {
t.Fatalf("provider calls during profile inspection = %d, want 0", providerCalls.Load())
}
}
func TestExplicitPromptKitProfileValidationUsesFallbackAssets(t *testing.T) {
assets := llm.NewAssetRegistry()
if err := assets.RegisterFallbackProfileFS(fstest.MapFS{
"profiles/fallback.yaml": {Data: []byte("id: fallback-profile\nendpoint: http://promptkit.test/v1\nmodel: fallback-model\n")},
}, "profiles"); err != nil {
t.Fatalf("RegisterFallbackProfileFS() error = %v, want nil", err)
}
if err := validateExplicitPromptKitProfiles(context.Background(), config.Default(), []string{"fallback-profile"}, assets); err != nil {
t.Fatalf("validateExplicitPromptKitProfiles() error = %v, want nil", err)
}
}

323
internal/cli/recompute_execution_contract_test.go
View File

@@ -1,323 +0,0 @@
package cli
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestRecomputeStepRecoversThroughFilesystemCheckpoints(t *testing.T) {
tests := []struct {
name string
invalidateOutput bool
wantCode int
}{
{name: "accepted producer is hydrated", wantCode: 0},
{name: "invalid producer stops dependents", invalidateOutput: true, wantCode: 1},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newRecomputeTestRoots(t)
harness := newRecomputeTestHarness()
fresh := runRecomputeCommand(roots, harness.options(), false)
if fresh.code != 0 {
t.Fatalf("fresh run code=%d stderr=%q", fresh.code, fresh.stderr)
}
removeCheckpointLaneStage(t, roots.checkpoints, "extract", "first", "producer")
removeCheckpointLaneStage(t, roots.checkpoints, "merge", "first", "producer")
if tt.invalidateOutput {
path := findCheckpointFile(t, roots.checkpoints, "normalize", "first", "producer", "output.json")
if err := os.WriteFile(path, []byte("{"), 0o600); err != nil {
t.Fatal(err)
}
}
harness.resetCalls()
resumed := runRecomputeCommand(roots, harness.options(), true)
if resumed.code != tt.wantCode {
t.Fatalf("resumed code=%d stdout=%q stderr=%q", resumed.code, resumed.stdout, resumed.stderr)
}
events := readLatestCheckpointEvents(t, roots.debug)
if tt.invalidateOutput {
if harness.callsFor("test/extract/middle") != 0 || harness.callsFor("test/extract/dependent") != 0 {
t.Fatalf("dependent calls after invalid producer = %#v", harness.callsSnapshot())
}
if !strings.Contains(resumed.stderr, string(pipeline.CheckpointReasonDecodeFailed)) {
t.Fatalf("stderr=%q, want stable checkpoint reason", resumed.stderr)
}
assertNormalizeDecisionSequence(t, events, []checkpointDecisionExpectation{{"first", "producer", pipeline.CheckpointDecisionExecuted, pipeline.CheckpointReasonDecodeFailed}})
return
}
if got := harness.callsSnapshot(); !reflect.DeepEqual(got, map[string]int{"test/extract/dependent": 1, "test/extract/middle": 1}) {
t.Fatalf("resumed extractor calls = %#v", got)
}
outputPath := filepath.Join(latestChildDir(t, roots.output), "result.json")
data, err := os.ReadFile(outputPath)
if err != nil {
t.Fatal(err)
}
if string(data) != "[\"producer\",\"unrelated\",\"middle\",\"dependent\"]\n" {
t.Fatalf("ordered output = %q", data)
}
assertNormalizeDecisionSequence(t, events, []checkpointDecisionExpectation{
{"first", "producer", pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonAcceptedArtifactReused},
{"first", "unrelated", pipeline.CheckpointDecisionReused, pipeline.CheckpointReasonReused},
{"second", "middle", pipeline.CheckpointDecisionForcedRecompute, pipeline.CheckpointReasonRecomputeStep},
{"third", "dependent", pipeline.CheckpointDecisionForcedRecompute, pipeline.CheckpointReasonRecomputeStep},
})
})
}
}
type checkpointDecisionExpectation struct {
step, lane string
action pipeline.CheckpointDecisionCategory
reason pipeline.CheckpointReasonCode
}
func assertNormalizeDecisionSequence(t *testing.T, events []pipeline.CheckpointEvent, want []checkpointDecisionExpectation) {
t.Helper()
var got []checkpointDecisionExpectation
for _, event := range events {
if event.Stage == string(pipeline.StageNormalize) {
got = append(got, checkpointDecisionExpectation{event.StepID, event.LaneID, event.Action, event.ReasonCode})
}
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("normalize decisions = %#v, want %#v", got, want)
}
}
type recomputeTestHarness struct {
base *stateTestHarness
mu sync.Mutex
calls map[string]int
}
func newRecomputeTestHarness() *recomputeTestHarness {
return &recomputeTestHarness{base: newStateTestHarness(), calls: make(map[string]int)}
}
func (h *recomputeTestHarness) options() Options {
opts := h.base.options()
for _, key := range []string{"test/extract/producer", "test/extract/unrelated", "test/extract/middle", "test/extract/dependent"} {
moduleKey := key
spec := pipeline.ModuleSpec{
Key: moduleKey, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind,
ReferenceSlots: []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}},
}
if err := pipeline.RegisterExtractor(opts.Registries.Extractors, spec, func() (contracts.Extractor[stateTestArtifact], error) {
return recomputeTestExtractor{key: moduleKey, harness: h}, nil
}); err != nil {
panic(err)
}
}
if err := opts.Registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/recompute-output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
return recomputeTestOutput{}, nil
}); err != nil {
panic(err)
}
opts.Catalog = catalogFromRegistries(opts.Registries)
return opts
}
func (h *recomputeTestHarness) record(key string) {
h.mu.Lock()
defer h.mu.Unlock()
h.calls[key]++
}
func (h *recomputeTestHarness) resetCalls() {
h.mu.Lock()
defer h.mu.Unlock()
h.calls = make(map[string]int)
}
func (h *recomputeTestHarness) callsFor(key string) int {
h.mu.Lock()
defer h.mu.Unlock()
return h.calls[key]
}
func (h *recomputeTestHarness) callsSnapshot() map[string]int {
h.mu.Lock()
defer h.mu.Unlock()
result := make(map[string]int, len(h.calls))
for key, value := range h.calls {
result[key] = value
}
return result
}
type recomputeTestExtractor struct {
key string
harness *recomputeTestHarness
}
func (e recomputeTestExtractor) Key() string { return e.key }
func (e recomputeTestExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return []contracts.ReferenceSlot{{Name: "upstream", AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{stateTestArtifactKind}}}
}
func (e recomputeTestExtractor) Extract(context.Context, contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[stateTestArtifact], error) {
e.harness.record(e.key)
return contracts.TypedExtractionResult[stateTestArtifact]{Value: stateTestArtifact{Value: e.key}}, nil
}
type recomputeTestOutput struct{}
func (recomputeTestOutput) Key() string { return "test/recompute-output" }
func (recomputeTestOutput) Encode(_ context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
lanes := make([]string, 0, len(req.NormalizeOutputs))
for _, output := range req.NormalizeOutputs {
lanes = append(lanes, output.LaneID)
}
data, err := json.Marshal(lanes)
if err != nil {
return contracts.OutputResult{}, err
}
return contracts.OutputResult{Files: []contracts.OutputFile{{Name: "result.json", Bytes: append(data, '\n')}}}, nil
}
func newRecomputeTestRoots(t *testing.T) stateTestRoots {
t.Helper()
roots := newStateTestRoots(t)
config := fmt.Sprintf(`version: 4
output:
directory: %q
cache:
chunk_plans:
directory: %q
mode: bypass
checkpoints:
enabled: true
directory: %q
debug:
directory: %q
pipelines:
sample:
input: test/input
chunk: test/chunk
steps:
- id: first
artifacts:
producer:
extract: test/extract/producer
merge: test/merge
normalize: test/normalize
unrelated:
extract: test/extract/unrelated
merge: test/merge
normalize: test/normalize
- id: second
references:
upstream:
artifact:
step: first
lane: producer
artifacts:
middle:
extract: test/extract/middle
merge: test/merge
normalize: test/normalize
- id: third
references:
upstream:
artifact:
step: second
lane: middle
artifacts:
dependent:
extract: test/extract/dependent
merge: test/merge
normalize: test/normalize
output: test/recompute-output
`, roots.output, roots.plans, roots.checkpoints, roots.debug)
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
t.Fatal(err)
}
return roots
}
func runRecomputeCommand(roots stateTestRoots, opts Options, recompute bool) stateTestResult {
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}
if recompute {
args = append(args, "--resume", "--recompute-step", "second", "--debug")
}
var stdout, stderr bytes.Buffer
return stateTestResult{code: RunWithOptions(args, &stdout, &stderr, opts), stdout: stdout.String(), stderr: stderr.String()}
}
func removeCheckpointLaneStage(t *testing.T, root, stage, step, lane string) {
t.Helper()
dir := filepath.Dir(findCheckpointFile(t, root, stage, step, lane, "manifest.json"))
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
func findCheckpointFile(t *testing.T, root, stage, step, lane, name string) string {
t.Helper()
want := filepath.Join(stage, step, lane, name)
var matches []string
err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error {
if err != nil {
return err
}
if !entry.IsDir() && strings.HasSuffix(path, want) {
matches = append(matches, path)
}
return nil
})
if err != nil {
t.Fatal(err)
}
if len(matches) != 1 {
t.Fatalf("checkpoint files ending in %q = %v", want, matches)
}
return matches[0]
}
func latestChildDir(t *testing.T, root string) string {
t.Helper()
entries, err := os.ReadDir(root)
if err != nil {
t.Fatal(err)
}
var dirs []string
for _, entry := range entries {
if entry.IsDir() {
dirs = append(dirs, filepath.Join(root, entry.Name()))
}
}
if len(dirs) == 0 {
t.Fatal("no child directory")
}
sort.Strings(dirs)
return dirs[len(dirs)-1]
}
func readLatestCheckpointEvents(t *testing.T, root string) []pipeline.CheckpointEvent {
t.Helper()
var events []pipeline.CheckpointEvent
data, err := os.ReadFile(filepath.Join(latestChildDir(t, root), "summary", "checkpoint-events.json"))
if err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(data, &events); err != nil {
t.Fatal(err)
}
return events
}

56
internal/cli/recompute_policy_test.go
View File

@@ -1,56 +0,0 @@
package cli
import (
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestRecomputePolicyIncludesDependentLanesAndReusablePredecessors(t *testing.T) {
producer := pipeline.ResolvedArtifactLane{StepID: "first", ID: "producer"}
unrelated := pipeline.ResolvedArtifactLane{StepID: "first", ID: "unrelated"}
consumer := pipeline.ResolvedArtifactLane{
StepID: "second", ID: "consumer",
ExtractReferences: pipeline.ResolvedReferenceTarget{Bindings: []pipeline.ReferenceBinding{{Artifact: &pipeline.ArtifactReference{Step: "first", Lane: "producer"}}}},
}
downstream := pipeline.ResolvedArtifactLane{
StepID: "third", ID: "downstream",
ExtractReferences: pipeline.ResolvedReferenceTarget{Bindings: []pipeline.ReferenceBinding{{Artifact: &pipeline.ArtifactReference{Step: "second", Lane: "consumer"}}}},
}
independent := pipeline.ResolvedArtifactLane{StepID: "third", ID: "independent"}
resolved := pipeline.ResolvedPipeline{Steps: []pipeline.ResolvedPipelineStep{
{ID: "first", ArtifactLanes: []pipeline.ResolvedArtifactLane{producer, unrelated}},
{ID: "second", ArtifactLanes: []pipeline.ResolvedArtifactLane{consumer}},
{ID: "third", ArtifactLanes: []pipeline.ResolvedArtifactLane{downstream, independent}},
}}
policy, err := recomputePolicy(resolved, "second")
if err != nil {
t.Fatal(err)
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("second", "consumer")]; !ok {
t.Fatal("selected lane was not forced")
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("third", "downstream")]; !ok {
t.Fatal("transitive dependent lane was not forced")
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("first", "producer")]; ok {
t.Fatal("predecessor was implicitly forced")
}
if _, ok := policy.RequireReusableLanes[pipeline.CheckpointLaneKey("first", "producer")]; !ok {
t.Fatal("required predecessor was not marked reusable")
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("first", "unrelated")]; ok {
t.Fatal("unrelated lane was forced")
}
if _, ok := policy.ForcedLanes[pipeline.CheckpointLaneKey("third", "independent")]; ok {
t.Fatal("unrelated later lane was forced")
}
}
func TestRecomputePolicyRejectsUnknownStep(t *testing.T) {
_, err := recomputePolicy(pipeline.ResolvedPipeline{Steps: []pipeline.ResolvedPipelineStep{{ID: "known"}}}, "missing")
if err == nil {
t.Fatal("unknown step was accepted")
}
}

456
internal/cli/reference_contract_test.go
View File

@@ -1,456 +0,0 @@
package cli
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestReferenceSelectorsParseAndApplyAllDocumentedForms(t *testing.T) {
tests := []struct {
name string
selector string
only []string
wantStage pipeline.ModuleStage
wantLane string
wantSlot string
}{
{name: "flat", selector: "alpha-slot", wantStage: pipeline.StageExtract, wantLane: "alpha", wantSlot: "alpha-slot"},
{name: "chunk", selector: "chunk.chunk-slot", wantStage: pipeline.StageChunk, wantSlot: "chunk-slot"},
{name: "merge", selector: "merge.alpha-merge", only: []string{"alpha"}, wantStage: pipeline.StageMerge, wantLane: "alpha", wantSlot: "alpha-merge"},
{name: "lane", selector: "alpha.alpha-slot", wantStage: pipeline.StageExtract, wantLane: "alpha", wantSlot: "alpha-slot"},
{name: "lane extract", selector: "alpha.extract.alpha-slot", wantStage: pipeline.StageExtract, wantLane: "alpha", wantSlot: "alpha-slot"},
{name: "lane merge", selector: "alpha.merge.alpha-merge", wantStage: pipeline.StageMerge, wantLane: "alpha", wantSlot: "alpha-merge"},
{name: "lane normalize", selector: "alpha.normalize.alpha-normalize", wantStage: pipeline.StageNormalize, wantLane: "alpha", wantSlot: "alpha-normalize"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := referenceContractConfig()
catalog := referenceContractCatalog(t, true, true)
selector, err := parseReferenceSelector(tt.selector, "--reference")
if err != nil {
t.Fatal(err)
}
overrides, _, err := resolveCLIReferenceRequests(cfg, "demo", tt.only, catalog, []cliReferenceRequest{{Selector: selector, Source: "reference.txt"}}, nil)
if err != nil {
t.Fatalf("resolve selector: %v", err)
}
if len(overrides) != 1 {
t.Fatalf("overrides = %#v, want one binding", overrides)
}
got := overrides[0]
if got.Stage != tt.wantStage || got.LaneID != tt.wantLane || got.SlotName != tt.wantSlot || got.BindingSource != contracts.ReferenceBindingSourceCLI {
t.Fatalf("binding = %#v, want %s/%s/%s from CLI", got, tt.wantStage, tt.wantLane, tt.wantSlot)
}
})
}
}
func TestReferenceSelectorsRejectAmbiguityWithSpecificSuggestions(t *testing.T) {
cfg := referenceContractConfig()
catalog := referenceContractCatalog(t, true, true)
for _, tt := range []struct {
name string
selector string
want []string
}{
{name: "flat shared slot", selector: "shared", want: []string{"alpha.extract.shared", "beta.extract.shared"}},
{name: "lane shared slot", selector: "alpha.shared", want: []string{"alpha.extract.shared", "alpha.merge.shared", "alpha.normalize.shared"}},
{name: "all mergers", selector: "merge.shared", want: []string{"alpha.merge.shared", "beta.merge.shared"}},
} {
t.Run(tt.name, func(t *testing.T) {
selector, err := parseReferenceSelector(tt.selector, "--reference")
if err != nil {
t.Fatal(err)
}
_, _, err = resolveCLIReferenceRequests(cfg, "demo", nil, catalog, []cliReferenceRequest{{Selector: selector, Source: "reference.txt"}}, nil)
if err == nil {
t.Fatal("resolve selector succeeded, want ambiguity error")
}
for _, fragment := range tt.want {
if !strings.Contains(err.Error(), fragment) {
t.Fatalf("error = %q, want suggestion %q", err, fragment)
}
}
})
}
}
func TestReferenceSelectorsRespectSelectedLanesBeforeMaterialization(t *testing.T) {
cfg := referenceContractConfig()
catalog := referenceContractCatalog(t, true, true)
for _, tt := range []struct {
name string
selector string
want string
}{
{name: "unselected lane", selector: "beta.extract.beta-slot", want: `reference lane "beta" is not selected`},
{name: "unknown lane", selector: "missing.extract.beta-slot", want: `reference lane "missing" is not selected`},
} {
t.Run(tt.name, func(t *testing.T) {
selector, err := parseReferenceSelector(tt.selector, "--reference")
if err != nil {
t.Fatal(err)
}
_, _, err = resolveCLIReferenceRequests(cfg, "demo", []string{"alpha"}, catalog, []cliReferenceRequest{{Selector: selector, Source: filepath.Join(t.TempDir(), "missing.txt")}}, nil)
if err == nil || !strings.Contains(err.Error(), tt.want) || strings.Contains(err.Error(), "missing.txt") {
t.Fatalf("error = %v, want selection failure before file access", err)
}
})
}
}
func TestReferenceSyntaxErrorsReturnTwo(t *testing.T) {
tests := []struct {
name string
args []string
}{
{name: "reference missing value", args: []string{"run", "demo", "--config", "missing.yml", "--input", "input.txt", "--reference"}},
{name: "reference missing selector", args: []string{"run", "demo", "--config", "missing.yml", "--input", "input.txt", "--reference", "=path.txt"}},
{name: "reference missing separator", args: []string{"run", "demo", "--config", "missing.yml", "--input", "input.txt", "--reference", "slot"}},
{name: "reference missing path", args: []string{"run", "demo", "--config", "missing.yml", "--input", "input.txt", "--reference", "slot="}},
{name: "reference excess segments", args: []string{"run", "demo", "--config", "missing.yml", "--input", "input.txt", "--reference", "a.b.c.d=path.txt"}},
{name: "unbind with path", args: []string{"run", "demo", "--config", "missing.yml", "--input", "input.txt", "--without-reference", "slot=path.txt"}},
{name: "unbind excess segments", args: []string{"run", "demo", "--config", "missing.yml", "--input", "input.txt", "--without-reference", "a.b.c.d"}},
{name: "unbind missing value", args: []string{"run", "demo", "--config", "missing.yml", "--input", "input.txt", "--without-reference"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var stdout, stderr bytes.Buffer
code := RunWithOptions(tt.args, &stdout, &stderr, Options{LookupEnv: emptyLookup})
if code != 2 || stdout.Len() != 0 || stderr.Len() == 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
}
func TestReferenceOverridesUseFinalExactTargetBinding(t *testing.T) {
cfg := referenceContractConfig()
catalog := referenceContractCatalog(t, true, true)
alphaShared, err := parseReferenceSelector("alpha.extract.shared", "--reference")
if err != nil {
t.Fatal(err)
}
betaShared, err := parseReferenceSelector("beta.extract.shared", "--reference")
if err != nil {
t.Fatal(err)
}
overrides, unbinds, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, []cliReferenceRequest{
{Selector: alphaShared, Source: "alpha-first.txt"},
{Selector: alphaShared, Source: "alpha-final.txt"},
{Selector: betaShared, Source: "beta-only.txt"},
}, nil)
if err != nil {
t.Fatal(err)
}
if len(unbinds) != 0 {
t.Fatalf("unbinds = %#v, want none", unbinds)
}
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "demo", Catalog: catalog, ReferenceOverrides: overrides})
if err != nil {
t.Fatalf("resolve pipeline: %v", err)
}
alpha := referenceContractLane(t, effective.ResolvedPipeline, "alpha")
beta := referenceContractLane(t, effective.ResolvedPipeline, "beta")
if source := referenceContractBindingSource(alpha.ExtractReferences.Bindings, "shared"); source != "alpha-final.txt" {
t.Fatalf("alpha shared source = %q, want final exact-target override", source)
}
if source := referenceContractBindingSource(beta.ExtractReferences.Bindings, "shared"); source != "beta-only.txt" {
t.Fatalf("beta shared source = %q, want target-specific override", source)
}
}
func TestReferenceUnbindsRemoveOptionalAndProtectRequiredSlots(t *testing.T) {
cfg := referenceContractConfig()
catalog := referenceContractCatalog(t, true, true)
optional, err := parseReferenceSelector("alpha.extract.alpha-slot", "--without-reference")
if err != nil {
t.Fatal(err)
}
_, without, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, nil, []cliReferenceUnbindRequest{{Selector: optional}})
if err != nil {
t.Fatal(err)
}
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "demo", Catalog: catalog, ReferenceUnbinds: without})
if err != nil {
t.Fatalf("optional unbind: %v", err)
}
if binding := referenceContractFindBinding(referenceContractLane(t, effective.ResolvedPipeline, "alpha").ExtractReferences.Bindings, "alpha-slot"); binding != nil {
t.Fatalf("optional binding after unbind = %#v, want absent", binding)
}
for _, tt := range []struct {
name string
selector string
}{
{name: "chunk", selector: "chunk.required-chunk"},
{name: "extract", selector: "alpha.extract.required-extract"},
{name: "merge", selector: "alpha.merge.required-merge"},
{name: "normalize", selector: "alpha.normalize.required-normalize"},
} {
t.Run(tt.name, func(t *testing.T) {
selector, err := parseReferenceSelector(tt.selector, "--without-reference")
if err != nil {
t.Fatal(err)
}
_, unbinds, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, nil, []cliReferenceUnbindRequest{{Selector: selector}})
if err != nil {
t.Fatal(err)
}
_, err = cfg.Resolve(config.ResolveInput{PipelineID: "demo", Catalog: catalog, ReferenceUnbinds: unbinds})
if err == nil || !strings.Contains(err.Error(), "required reference slot") {
t.Fatalf("resolve error = %v, want required-slot failure", err)
}
})
}
}
func TestReferenceMaterializationSeparatesCLIAndConfigPathOrigins(t *testing.T) {
configDir := t.TempDir()
workingDir := t.TempDir()
cfg := referenceContractConfig()
configPath := filepath.Join(configDir, "config.yml")
if err := os.WriteFile(configPath, []byte("version: 4\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(configDir, "required.txt"), []byte("config reference"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(configDir, "optional.txt"), []byte("optional reference"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(workingDir, "cli-reference.txt"), []byte("CLI reference"), 0o600); err != nil {
t.Fatal(err)
}
catalog := referenceContractCatalog(t, true, true)
selector, err := parseReferenceSelector("alpha.extract.alpha-slot", "--reference")
if err != nil {
t.Fatal(err)
}
overrides, unbinds, err := resolveCLIReferenceRequests(cfg, "demo", nil, catalog, []cliReferenceRequest{{Selector: selector, Source: "cli-reference.txt"}}, nil)
if err != nil {
t.Fatal(err)
}
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "demo", Catalog: catalog, ReferenceOverrides: overrides, ReferenceUnbinds: unbinds})
if err != nil {
t.Fatalf("resolve pipeline: %v", err)
}
materialized, _, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{ConfigPath: configPath, WorkingDir: workingDir})
if err != nil {
t.Fatalf("materialize references: %v", err)
}
alpha := referenceContractLane(t, materialized, "alpha")
cliItem := alpha.ExtractReferences.ReferenceSet.Slots["alpha-slot"].Items[0]
if string(cliItem.Content) != "CLI reference" || cliItem.BindingSource != contracts.ReferenceBindingSourceCLI || cliItem.Origin.URI != referenceContractFileURI(filepath.Join(workingDir, "cli-reference.txt")) {
t.Fatalf("CLI materialization = %#v, want working-directory provenance", cliItem)
}
configItem := alpha.ExtractReferences.ReferenceSet.Slots["required-extract"].Items[0]
if string(configItem.Content) != "config reference" || configItem.BindingSource != contracts.ReferenceBindingSourceConfig || configItem.Origin.URI != referenceContractFileURI(filepath.Join(configDir, "required.txt")) {
t.Fatalf("config materialization = %#v, want config-directory provenance", configItem)
}
}
func TestReferenceTargetLookupUsesArtifactVariantsAndReportsMissingContext(t *testing.T) {
cfg := referenceContractConfig()
full := referenceContractCatalog(t, true, true)
targets, err := selectedReferenceTargets(cfg, "demo", nil, full)
if err != nil {
t.Fatalf("select reference targets: %v", err)
}
var alphaMerge, betaMerge selectedReferenceTarget
for _, target := range targets {
if target.stage == pipeline.StageMerge && target.laneID == "alpha" {
alphaMerge = target
}
if target.stage == pipeline.StageMerge && target.laneID == "beta" {
betaMerge = target
}
}
if _, ok := alphaMerge.slots["alpha-merge"]; !ok {
t.Fatalf("alpha merger slots = %#v, want alpha artifact variant", alphaMerge.slots)
}
if _, ok := betaMerge.slots["beta-merge"]; !ok {
t.Fatalf("beta merger slots = %#v, want beta artifact variant", betaMerge.slots)
}
if _, ok := betaMerge.slots["alpha-merge"]; ok {
t.Fatalf("beta merger slots = %#v, must not use alpha variant", betaMerge.slots)
}
missingMerger := referenceContractCatalog(t, false, true)
_, err = selectedReferenceTargets(cfg, "demo", nil, missingMerger)
if err == nil || !strings.Contains(err.Error(), "merger") || !strings.Contains(err.Error(), string(referenceContractKindBeta)) {
t.Fatalf("missing merger error = %v, want artifact variant context", err)
}
missingNormalizer := referenceContractCatalog(t, true, false)
_, err = selectedReferenceTargets(cfg, "demo", nil, missingNormalizer)
if err == nil || !strings.Contains(err.Error(), "normalizer") || !strings.Contains(err.Error(), string(referenceContractKindBeta)) {
t.Fatalf("missing normalizer error = %v, want artifact variant context", err)
}
missingExtractor := referenceContractCatalog(t, true, true)
missingExtractor.Extractors = pipeline.NewExtractorRegistry()
_, err = selectedReferenceTargets(cfg, "demo", nil, missingExtractor)
if err == nil || !strings.Contains(err.Error(), `lane "alpha" extract module`) || !strings.Contains(err.Error(), "not registered") {
t.Fatalf("missing extractor error = %v, want lane/module context", err)
}
}
const (
referenceContractKindAlpha contracts.ArtifactKind = "reference/alpha"
referenceContractKindBeta contracts.ArtifactKind = "reference/beta"
)
func referenceContractConfig() config.Config {
cfg := config.Default()
cfg.Pipelines = map[string]pipeline.PipelineProfile{
"demo": {
ID: "demo",
Input: pipeline.Binding("reference/input"),
Chunk: pipeline.Binding("reference/chunk"),
Output: pipeline.Binding("reference/output"),
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"alpha": {
Extract: pipeline.Binding("reference/extract-alpha"),
Merge: pipeline.Binding("reference/shared-merge"),
Normalize: pipeline.Binding("reference/shared-normalize"),
References: pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"}),
},
"beta": {
Extract: pipeline.Binding("reference/extract-beta"),
Merge: pipeline.Binding("reference/shared-merge"),
Normalize: pipeline.Binding("reference/shared-normalize"),
References: pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"}),
},
},
},
}
profile := cfg.Pipelines["demo"]
profile.Chunk.References = pipeline.ExternalReferenceMap(map[string]string{"required-chunk": "required.txt"})
alpha := profile.Artifacts["alpha"]
alpha.Extract.References = pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt", "alpha-slot": "optional.txt"})
alpha.Merge.References = pipeline.ExternalReferenceMap(map[string]string{"required-merge": "required.txt"})
alpha.Normalize.References = pipeline.ExternalReferenceMap(map[string]string{"required-normalize": "required.txt"})
profile.Artifacts["alpha"] = alpha
beta := profile.Artifacts["beta"]
beta.Extract.References = pipeline.ExternalReferenceMap(map[string]string{"required-extract": "required.txt"})
beta.Merge.References = pipeline.ExternalReferenceMap(map[string]string{"required-merge": "required.txt"})
beta.Normalize.References = pipeline.ExternalReferenceMap(map[string]string{"required-normalize": "required.txt"})
profile.Artifacts["beta"] = beta
cfg.Pipelines["demo"] = profile
return cfg
}
func referenceContractCatalog(t *testing.T, includeBetaMerger, includeBetaNormalizer bool) pipeline.ModuleCatalog {
t.Helper()
registries := pipeline.Registries{
Inputs: pipeline.NewInputAdapterRegistry(),
Chunkers: pipeline.NewChunkerRegistry(),
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
Extractors: pipeline.NewExtractorRegistry(),
Mergers: pipeline.NewMergerRegistry(),
Normalizers: pipeline.NewNormalizerRegistry(),
Validators: pipeline.NewValidatorRegistry(),
ValidatorChains: pipeline.NewValidatorChainRegistry(),
Outputs: pipeline.NewOutputEncoderRegistry(),
}
register := func(err error) {
if err != nil {
t.Fatal(err)
}
}
register(registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "reference/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }))
register(registries.Chunkers.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-slot"}, {Name: "required-chunk", Required: true}}}, func() (contracts.Chunker, error) { return stateTestChunker{}, nil }))
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecA{}))
register(pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, referenceContractCodecB{}))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-alpha", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "reference/extract-beta", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-slot"}, {Name: "required-extract", Required: true}}}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{}, nil }))
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
if includeBetaMerger {
register(pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "reference/shared-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-merge"}, {Name: "required-merge", Required: true}}}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{}, nil }))
}
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindAlpha, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "alpha-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
if includeBetaNormalizer {
register(pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "reference/shared-normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: referenceContractKindBeta, ReferenceSlots: []contracts.ReferenceSlot{{Name: "shared"}, {Name: "beta-normalize"}, {Name: "required-normalize", Required: true}}}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{}, nil }))
}
register(registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "reference/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) { return stateTestOutput{}, nil }))
return catalogFromRegistries(registries)
}
type referenceContractCodecB struct{}
type referenceContractCodecA struct{}
func (referenceContractCodecA) Kind() contracts.ArtifactKind { return referenceContractKindAlpha }
func (referenceContractCodecA) Schema() contracts.ArtifactSchema {
return contracts.ArtifactSchema{ID: "reference.alpha", Name: "reference_alpha", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
}
func (referenceContractCodecA) MediaType() string { return "application/json" }
func (referenceContractCodecA) EncodeCandidate(stateTestArtifact) ([]byte, error) {
return []byte(`{"value":"ok"}`), nil
}
func (referenceContractCodecA) Encode(stateTestArtifact) ([]byte, error) {
return []byte(`{"value":"ok"}`), nil
}
func (referenceContractCodecA) Decode([]byte) (stateTestArtifact, error) {
return stateTestArtifact{Value: "ok"}, nil
}
func (referenceContractCodecB) Kind() contracts.ArtifactKind { return referenceContractKindBeta }
func (referenceContractCodecB) Schema() contracts.ArtifactSchema {
return contracts.ArtifactSchema{ID: "reference.beta", Name: "reference_beta", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
}
func (referenceContractCodecB) MediaType() string { return "application/json" }
func (referenceContractCodecB) EncodeCandidate(stateTestArtifact) ([]byte, error) {
return []byte(`{"value":"ok"}`), nil
}
func (referenceContractCodecB) Encode(stateTestArtifact) ([]byte, error) {
return []byte(`{"value":"ok"}`), nil
}
func (referenceContractCodecB) Decode([]byte) (stateTestArtifact, error) {
return stateTestArtifact{Value: "ok"}, nil
}
func referenceContractLane(t *testing.T, resolved pipeline.ResolvedPipeline, id string) pipeline.ResolvedArtifactLane {
t.Helper()
for _, step := range resolved.Steps {
for _, lane := range step.ArtifactLanes {
if lane.ID == id {
return lane
}
}
}
t.Fatalf("lane %q not found", id)
return pipeline.ResolvedArtifactLane{}
}
func referenceContractBindingSource(bindings []pipeline.ReferenceBinding, slot string) string {
for _, binding := range bindings {
if binding.SlotName == slot {
return binding.Source
}
}
return ""
}
func referenceContractFindBinding(bindings []pipeline.ReferenceBinding, slot string) *pipeline.ReferenceBinding {
for i := range bindings {
if bindings[i].SlotName == slot {
return &bindings[i]
}
}
return nil
}
func referenceContractFileURI(path string) string {
absolute, err := filepath.Abs(path)
if err != nil {
absolute = path
}
return "file://" + filepath.ToSlash(absolute)
}

1738
internal/cli/run.go
View File

File diff suppressed because it is too large Load Diff

654
internal/cli/run_contract_test.go
View File

@@ -1,654 +0,0 @@
package cli
import (
"bytes"
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestRunControlsRejectSyntaxWithoutAllocatingState(t *testing.T) {
tests := []struct {
name string
args func(stateTestRoots) []string
}{
{name: "missing pipeline", args: func(roots stateTestRoots) []string {
return []string{"run", "--config", roots.config, "--input", roots.input}
}},
{name: "missing input", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config}
}},
{name: "unknown flag", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--unknown"}
}},
{name: "blank output directory", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--output-dir", ""}
}},
{name: "blank debug directory", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", ""}
}},
{name: "debug directory without debug", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--debug-dir", filepath.Join(filepath.Dir(roots.debug), "requested-debug")}
}},
{name: "blank session ID", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--session-id", ""}
}},
{name: "multiple pipeline IDs", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "extra", "--config", roots.config, "--input", roots.input}
}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
var stdout, stderr bytes.Buffer
code := RunWithOptions(tt.args(roots), &stdout, &stderr, newStateTestHarness().options())
if code != 2 || stdout.Len() != 0 || stderr.Len() == 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertAbsent(t, roots.output)
assertAbsent(t, roots.debug)
})
}
}
func TestRecomputeStepCLIContract(t *testing.T) {
tests := []struct {
name string
configure func(*testing.T, stateTestRoots)
flags []string
wantCode int
wantOutput string
wantError string
}{
{
name: "explicit step",
configure: func(t *testing.T, roots stateTestRoots) {
replaceStateTestConfigLine(t, roots.config, " artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n", " steps:\n - id: chosen\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n")
},
flags: []string{"--resume", "--recompute-step", "chosen"},
wantCode: 0,
wantOutput: "outputs=1",
},
{name: "implicit default step", flags: []string{"--resume", "--recompute-step", "default"}, wantCode: 0, wantOutput: "outputs=1"},
{name: "repeated flag", flags: []string{"--resume", "--recompute-step", "default", "--recompute-step", "default"}, wantCode: 2, wantError: "specified only once"},
{name: "empty step", flags: []string{"--resume", "--recompute-step", ""}, wantCode: 2, wantError: "must not be empty"},
{name: "unknown step", flags: []string{"--resume", "--recompute-step", "missing"}, wantCode: 1, wantError: "unknown pipeline step"},
{name: "without resume", flags: []string{"--recompute-step", "default"}, wantCode: 2, wantError: "requires --resume"},
{
name: "checkpoint recording disabled",
configure: func(t *testing.T, roots stateTestRoots) {
replaceStateTestConfigLine(t, roots.config, " enabled: true\n", " enabled: false\n")
},
flags: []string{"--resume", "--recompute-step", "default"}, wantCode: 1, wantError: "cache.checkpoints.enabled",
},
{name: "with only", flags: []string{"--resume", "--recompute-step", "default", "--only", "items"}, wantCode: 2, wantError: "cannot be combined with --only"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
if tt.configure != nil {
tt.configure(t, roots)
}
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}
args = append(args, tt.flags...)
var stdout, stderr bytes.Buffer
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
if code != tt.wantCode || (tt.wantOutput != "" && !strings.Contains(stdout.String(), tt.wantOutput)) || (tt.wantError != "" && !strings.Contains(stderr.String(), tt.wantError)) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
}
func TestRunValidFailuresClassifyAndReportDebug(t *testing.T) {
tests := []struct {
name string
args func(stateTestRoots) []string
wantError string
wantDebug bool
}{
{name: "unknown pipeline", args: func(roots stateTestRoots) []string {
return []string{"run", "missing", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}
}, wantError: `pipeline "missing"`},
{name: "unknown lane", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--only", "missing", "--chunk_cache", "bypass", "--debug"}
}, wantError: `lane "missing"`, wantDebug: true},
{name: "unreadable input", args: func(roots stateTestRoots) []string {
return []string{"run", "sample", "--config", roots.config, "--input", filepath.Join(filepath.Dir(roots.input), "unreadable.txt"), "--chunk_cache", "bypass", "--debug"}
}, wantError: "read input", wantDebug: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
var stdout, stderr bytes.Buffer
code := RunWithOptions(tt.args(roots), &stdout, &stderr, newStateTestHarness().options())
if code != 1 || stdout.Len() != 0 || !strings.Contains(stderr.String(), tt.wantError) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if tt.wantDebug {
if !strings.Contains(stderr.String(), "debug=") {
t.Fatalf("stderr=%q, want debug path", stderr.String())
}
onlyChildDir(t, roots.debug)
} else {
assertAbsent(t, roots.debug)
}
assertAbsent(t, roots.output)
})
}
}
func TestRunOnlyExecutesSelectedLanes(t *testing.T) {
roots := newStateTestRoots(t)
data, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
data = []byte(replaceRequiredOnce(t, string(data), " output: test/output\n", " other:\n extract: test/extract\n output: test/output\n"))
if err := os.WriteFile(roots.config, data, 0o600); err != nil {
t.Fatal(err)
}
harness := newStateTestHarness()
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--only", "items", "--chunk_cache", "bypass"}, &stdout, &stderr, harness.options())
if code != 0 || !strings.Contains(stdout.String(), "outputs=1") || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
harness.mu.Lock()
extractCalls := harness.extractCalls
harness.mu.Unlock()
if extractCalls != 1 {
t.Fatalf("extract calls = %d, want only the selected lane", extractCalls)
}
}
func TestRunStateRootsHonorEnvironmentFlagsAndDefaults(t *testing.T) {
t.Run("environment roots", func(t *testing.T) {
roots := newStateTestRoots(t)
environmentOutput := filepath.Join(t.TempDir(), "environment-output")
environmentDebug := filepath.Join(t.TempDir(), "environment-debug")
opts := newStateTestHarness().options()
opts.LookupEnv = lookupRunContractEnv(map[string]string{
"NOTARIUS_OUTPUT_DIR": environmentOutput,
"NOTARIUS_DEBUG_DIR": environmentDebug,
})
result := runWithStateRoots(t, roots, opts, nil)
if result.code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
assertFile(t, filepath.Join(environmentOutput, filepath.Base(onlyChildDir(t, environmentOutput)), "result.json"))
onlyChildDir(t, environmentDebug)
assertAbsent(t, roots.output)
assertAbsent(t, roots.debug)
})
t.Run("command flags override environment", func(t *testing.T) {
roots := newStateTestRoots(t)
environmentOutput := filepath.Join(t.TempDir(), "environment-output")
environmentDebug := filepath.Join(t.TempDir(), "environment-debug")
flagOutput := filepath.Join(t.TempDir(), "flag-output")
flagDebug := filepath.Join(t.TempDir(), "flag-debug")
opts := newStateTestHarness().options()
opts.LookupEnv = lookupRunContractEnv(map[string]string{
"NOTARIUS_OUTPUT_DIR": environmentOutput,
"NOTARIUS_DEBUG_DIR": environmentDebug,
})
result := runWithStateRoots(t, roots, opts, []string{"--output-dir", flagOutput, "--debug-dir", flagDebug})
if result.code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
assertFile(t, filepath.Join(flagOutput, filepath.Base(onlyChildDir(t, flagOutput)), "result.json"))
onlyChildDir(t, flagDebug)
assertAbsent(t, environmentOutput)
assertAbsent(t, environmentDebug)
})
t.Run("built-in roots", func(t *testing.T) {
roots := newStateTestRoots(t)
data, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
text := string(data)
text = replaceRequiredOnce(t, text, fmt.Sprintf(" directory: %q\n", roots.output), "")
text = replaceRequiredOnce(t, text, fmt.Sprintf(" directory: %q\n", roots.debug), "")
if err := os.WriteFile(roots.config, []byte(text), 0o600); err != nil {
t.Fatal(err)
}
workDir := t.TempDir()
t.Chdir(workDir)
opts := newStateTestHarness().options()
result := runWithStateRoots(t, roots, opts, nil)
if result.code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
assertFile(t, filepath.Join(workDir, "notarius-output", filepath.Base(onlyChildDir(t, filepath.Join(workDir, "notarius-output"))), "result.json"))
onlyChildDir(t, filepath.Join(workDir, "notarius-debug"))
})
}
func TestRunLLMProfileOverrideAndValidationUseInjectedBoundaries(t *testing.T) {
t.Run("one effective profile reaches the factory and modules", func(t *testing.T) {
roots := newStateTestRoots(t)
profileDir := writeRunContractProfiles(t, "override-profile")
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
harness := newStateTestHarness()
var factoryProfiles []string
opts := harness.options()
var factoryOverrides []LLMRuntimeOverrides
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryProfiles = append(factoryProfiles, profileID)
factoryOverrides = append(factoryOverrides, overrides)
return nil, nil, nil
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--llm-profile", "override-profile"}, &stdout, &stderr, opts)
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
}
if len(factoryOverrides) != 1 || factoryOverrides[0].ReasoningEffort != nil {
t.Fatalf("factory overrides = %#v, want inherited reasoning", factoryOverrides)
}
harness.mu.Lock()
profiles := append([]string(nil), harness.moduleProfiles...)
harness.mu.Unlock()
if len(profiles) < 4 {
t.Fatalf("module profiles = %#v, want chunk and lane stage requests", profiles)
}
for _, profile := range profiles {
if profile != "override-profile" {
t.Fatalf("module profiles = %#v, want override on every request", profiles)
}
}
})
t.Run("runtime override applies to validators", func(t *testing.T) {
roots := newStateTestRoots(t)
profileDir := writeRunContractProfiles(t, "override-profile", "validator-profile")
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
harness := newStateTestHarness()
var validatorProfiles []string
opts := harness.options()
registerRunContractValidator(t, &opts, &validatorProfiles)
factoryProfiles := []string{}
opts.LLMClientFactory = func(_ context.Context, _ config.Config, profileID string, _ LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryProfiles = append(factoryProfiles, profileID)
return nil, nil, nil
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--llm-profile", "override-profile"}, &stdout, &stderr, opts)
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if len(factoryProfiles) != 1 || factoryProfiles[0] != "override-profile" {
t.Fatalf("factory profiles = %#v, want one override profile", factoryProfiles)
}
if len(validatorProfiles) != 1 || validatorProfiles[0] != "override-profile" {
t.Fatalf("validator profiles = %#v, want runtime override", validatorProfiles)
}
})
t.Run("unknown profile is rejected without factory access", func(t *testing.T) {
roots := newStateTestRoots(t)
profileDir := writeRunContractProfiles(t, "override-profile")
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
factoryCalls := 0
opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryCalls++
return nil, nil, nil
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--llm-profile", "missing-profile"}, &stdout, &stderr, opts)
if code != 1 || !strings.Contains(stderr.String(), "not configured") || factoryCalls != 0 || stdout.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
}
})
t.Run("pipeline default is rejected before factory access", func(t *testing.T) {
roots := newStateTestRoots(t)
profileDir := writeRunContractProfiles(t, "configured-profile")
prependRunContractConfig(t, roots, fmt.Sprintf("promptkit:\n profile_dir: %q\n", profileDir))
replaceStateTestConfigLine(t, roots.config, " sample:\n", " sample:\n llm_profile: missing-profile\n")
factoryCalls := 0
opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
factoryCalls++
return nil, nil, nil
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
if code != 1 || !strings.Contains(stderr.String(), "not configured") || factoryCalls != 0 || stdout.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q factoryCalls=%d", code, stdout.String(), stderr.String(), factoryCalls)
}
})
}
func TestRunReasoningEffortOverrideReachesFactory(t *testing.T) {
tests := []struct {
name string
flags []string
wantValue string
wantSet bool
}{
{name: "inherit"},
{name: "replace", flags: []string{"--reasoning-effort", " focused "}, wantValue: "focused", wantSet: true},
{name: "clear", flags: []string{"--clear-reasoning-effort"}, wantSet: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
var got []LLMRuntimeOverrides
opts.LLMClientFactory = func(_ context.Context, _ config.Config, _ string, overrides LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
got = append(got, overrides)
return nil, nil, nil
}
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, tt.flags...)
var stdout, stderr bytes.Buffer
if code := RunWithOptions(args, &stdout, &stderr, opts); code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
if len(got) != 1 {
t.Fatalf("factory overrides = %#v, want one call", got)
}
if !tt.wantSet {
if got[0].ReasoningEffort != nil {
t.Fatalf("reasoning effort = %q, want inherit", *got[0].ReasoningEffort)
}
return
}
if got[0].ReasoningEffort == nil || *got[0].ReasoningEffort != tt.wantValue {
t.Fatalf("reasoning effort = %#v, want %q", got[0].ReasoningEffort, tt.wantValue)
}
})
}
}
func TestRunReasoningEffortOverrideRejectsInvalidSyntax(t *testing.T) {
tests := []struct {
name string
flags []string
wantError string
}{
{
name: "mutually exclusive controls",
flags: []string{"--reasoning-effort", "focused", "--clear-reasoning-effort"},
wantError: "cannot be combined",
},
{
name: "empty replacement",
flags: []string{"--reasoning-effort", " "},
wantError: "must not be empty",
},
{
name: "duplicate replacement",
flags: []string{"--reasoning-effort", "low", "--reasoning-effort", "high"},
wantError: "may be specified only once",
},
{
name: "missing replacement",
flags: []string{"--reasoning-effort"},
wantError: "flag needs an argument",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input}, tt.flags...)
var stdout, stderr bytes.Buffer
code := RunWithOptions(args, &stdout, &stderr, newStateTestHarness().options())
if code != 2 || stdout.Len() != 0 || !strings.Contains(stderr.String(), tt.wantError) {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
assertNoRunState(t, roots)
})
}
}
func TestReasoningEffortOverrideSeparatesCheckpointIdentities(t *testing.T) {
replacement := " focused "
cleared := ""
states := []struct {
name string
overrides LLMRuntimeOverrides
wantValue string
wantSet bool
}{
{name: "inherit"},
{name: "replace", overrides: LLMRuntimeOverrides{ReasoningEffort: &replacement}, wantValue: "focused", wantSet: true},
{name: "clear", overrides: LLMRuntimeOverrides{ReasoningEffort: &cleared}, wantValue: "<cleared>", wantSet: true},
}
digests := make(map[string]string, len(states))
for _, state := range states {
fingerprints := runtimeOverrideFingerprints("", "", state.overrides)
var value string
var found bool
for _, fingerprint := range fingerprints {
if fingerprint.Name == "reasoning_effort_override" {
value, found = fingerprint.Value, true
}
}
if found != state.wantSet || (found && value != state.wantValue) {
t.Fatalf("%s fingerprint found=%t value=%q, want found=%t value=%q", state.name, found, value, state.wantSet, state.wantValue)
}
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
Pipeline: pipeline.ResolvedPipeline{ID: "sample", Digest: "sha256:pipeline", Input: pipeline.Binding("test/input")},
RawInputDigest: "sha256:input",
RuntimeOverrides: fingerprints,
})
if err != nil {
t.Fatal(err)
}
digests[state.name] = identity.Digest
}
if digests["inherit"] == digests["replace"] || digests["inherit"] == digests["clear"] || digests["replace"] == digests["clear"] {
t.Fatalf("checkpoint identity digests are not distinct: %#v", digests)
}
}
func TestEffectiveLLMProfileIDsAreSortedDeduplicatedAndLLMOnly(t *testing.T) {
resolved := pipeline.ResolvedPipeline{
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
InputExecutionClass: contracts.ExecutionClassLLMBacked,
Chunk: pipeline.ModuleBinding{LLMProfile: " zeta "},
ChunkExecutionClass: contracts.ExecutionClassLLMBacked,
Steps: []pipeline.ResolvedPipelineStep{{
ID: "default",
ArtifactLanes: []pipeline.ResolvedArtifactLane{{
Extract: pipeline.ModuleBinding{LLMProfile: "alpha"},
ExtractExecutionClass: contracts.ExecutionClassLLMBacked,
Merge: pipeline.ModuleBinding{LLMProfile: "deterministic-merge"},
MergeExecutionClass: contracts.ExecutionClassDeterministic,
Normalize: pipeline.ModuleBinding{LLMProfile: " gamma "},
NormalizeExecutionClass: contracts.ExecutionClassLLMBacked,
}},
}},
ValidatorChains: []pipeline.ResolvedValidatorChain{{Validators: []pipeline.ResolvedValidator{
{Binding: pipeline.ModuleBinding{LLMProfile: "deterministic-profile"}, ExecutionClass: contracts.ExecutionClassDeterministic},
{Binding: pipeline.ModuleBinding{LLMProfile: "beta"}, ExecutionClass: contracts.ExecutionClassLLMBacked},
}}},
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
OutputExecutionClass: contracts.ExecutionClassLLMBacked,
}
got := effectiveLLMProfileIDs(resolved)
want := []string{"alpha", "beta", "gamma", "input-profile", "output-profile", "zeta"}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("effective profiles = %#v, want %#v", got, want)
}
}
func TestRunSessionIDUsesExplicitValueOrSourceDocumentID(t *testing.T) {
for _, tt := range []struct {
name string
args []string
want string
}{
{name: "source document", want: "source"},
{name: "explicit trimmed value", args: []string{"--session-id", " explicit-session "}, want: "explicit-session"},
} {
t.Run(tt.name, func(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
args := append([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, tt.args...)
var stdout, stderr bytes.Buffer
code := RunWithOptions(args, &stdout, &stderr, harness.options())
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
harness.mu.Lock()
sessions := append([]string(nil), harness.sessionIDs...)
harness.mu.Unlock()
if len(sessions) < 4 {
t.Fatalf("session IDs = %#v, want all prompt-facing module requests", sessions)
}
for _, session := range sessions {
if session != tt.want {
t.Fatalf("session IDs = %#v, want %q", sessions, tt.want)
}
}
})
}
}
func TestRunFactoryAndPreparationFailuresAreProcessFailures(t *testing.T) {
t.Run("LLM factory", func(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
opts.LLMClientFactory = func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return nil, nil, errors.New("injected LLM factory failure")
}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
if code != 1 || !strings.Contains(stderr.String(), "injected LLM factory failure") || stdout.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
t.Run("pipeline preparation", func(t *testing.T) {
roots := newStateTestRoots(t)
data, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
data = []byte(replaceRequiredOnce(t, string(data), "extract: test/extract", "extract: test/failing-extract"))
if err := os.WriteFile(roots.config, data, 0o600); err != nil {
t.Fatal(err)
}
opts := newStateTestHarness().options()
if err := pipeline.RegisterExtractorBuilder(opts.Registries.Extractors, pipeline.ModuleSpec{Key: "test/failing-extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Extractor[stateTestArtifact], error) {
return nil, errors.New("injected extractor construction failure")
}); err != nil {
t.Fatal(err)
}
opts.Catalog = catalogFromRegistries(opts.Registries)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass"}, &stdout, &stderr, opts)
if code != 1 || !strings.Contains(stderr.String(), "injected extractor construction failure") || stdout.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
func TestRunWarningsRemainSuccessfulAndReachDurableSurfaces(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
harness.includeWarnings = true
harness.chunkWarnings = []contracts.Warning{{Scope: "chunk", ReasonCode: "contract-warning", Message: "warning retained"}}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--debug"}, &stdout, &stderr, harness.options())
if code != 0 || !strings.Contains(stdout.String(), "outputs=1") || !strings.Contains(stderr.String(), "1 warning(s)") {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
outputPath := filepath.Join(onlyChildDir(t, roots.output), "result.json")
output, err := os.ReadFile(outputPath)
if err != nil || !strings.Contains(string(output), "contract-warning") {
t.Fatalf("durable output = %q, %v", output, err)
}
bundle := onlyChildDir(t, roots.debug)
var warnings []contracts.Warning
readStateTestSummaryJSON(t, bundle, "warnings.json", &warnings)
if len(warnings) != 1 || warnings[0].ReasonCode != "contract-warning" {
t.Fatalf("debug warnings = %#v", warnings)
}
}
func runWithStateRoots(t *testing.T, roots stateTestRoots, opts Options, extra []string) stateTestResult {
t.Helper()
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--debug"}
args = append(args, extra...)
var stdout, stderr bytes.Buffer
return stateTestResult{code: RunWithOptions(args, &stdout, &stderr, opts), stdout: stdout.String(), stderr: stderr.String()}
}
func lookupRunContractEnv(values map[string]string) func(string) (string, bool) {
return func(name string) (string, bool) {
value, ok := values[name]
return value, ok
}
}
func prependRunContractConfig(t *testing.T, roots stateTestRoots, prefix string) {
t.Helper()
data, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(roots.config, append([]byte(prefix), data...), 0o600); err != nil {
t.Fatal(err)
}
}
func writeRunContractProfiles(t *testing.T, ids ...string) string {
t.Helper()
dir := t.TempDir()
for _, id := range ids {
profile := fmt.Sprintf("id: %s\nendpoint: http://127.0.0.1:1/v1\nmodel: %s-model\n", id, id)
if err := os.WriteFile(filepath.Join(dir, id+".yaml"), []byte(profile), 0o600); err != nil {
t.Fatal(err)
}
}
return dir
}
func registerRunContractValidator(t *testing.T, opts *Options, profiles *[]string) {
t.Helper()
if err := pipeline.RegisterTypedValidatorBuilder(opts.Registries.Validators, stateTestArtifactKind, pipeline.ValidatorSpec{Key: "run-contract-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.TypedValidator[stateTestArtifact], error) {
return runContractValidator{profiles: profiles}, nil
}); err != nil {
t.Fatal(err)
}
if err := opts.Registries.ValidatorChains.Register(pipeline.ValidatorChainMapping{Stage: pipeline.StageExtract, Module: "test/extract", Validators: []pipeline.ModuleBinding{{Module: "run-contract-validator", LLMProfile: "validator-profile"}}}); err != nil {
t.Fatal(err)
}
opts.Catalog = catalogFromRegistries(opts.Registries)
}
type runContractValidator struct {
profiles *[]string
}
func (v runContractValidator) Name() string { return "run-contract-validator" }
func (v runContractValidator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassLLMBacked
}
func (v runContractValidator) Validate(_ context.Context, req contracts.TypedValidationRequest[stateTestArtifact]) (contracts.ValidationResult, error) {
*v.profiles = append(*v.profiles, req.LLMProfile)
return contracts.ValidationResult{Approved: true}, nil
}

34
internal/cli/run_id.go
View File

@@ -1,34 +0,0 @@
package cli
import (
"crypto/rand"
"encoding/hex"
"fmt"
"io"
"path/filepath"
"strings"
"time"
)
type RunIDGenerator func(time.Time) (string, error)
func defaultRunIDGenerator(startedAt time.Time) (string, error) {
var suffix [16]byte
if _, err := io.ReadFull(rand.Reader, suffix[:]); err != nil {
return "", fmt.Errorf("read random run ID suffix: %w", err)
}
return fmt.Sprintf("run-%d-%s", startedAt.UnixNano(), hex.EncodeToString(suffix[:])), nil
}
func validateRunID(runID string) error {
if runID == "" {
return fmt.Errorf("run ID must not be empty")
}
if runID != strings.TrimSpace(runID) {
return fmt.Errorf("run ID %q must not have surrounding whitespace", runID)
}
if strings.ContainsAny(runID, `/\\`) || filepath.IsAbs(runID) || filepath.Clean(runID) != runID || runID == "." || runID == ".." {
return fmt.Errorf("run ID %q must be one safe path component", runID)
}
return nil
}

87
internal/cli/run_id_test.go
View File

@@ -1,87 +0,0 @@
package cli
import (
"os"
"path/filepath"
"regexp"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestDefaultRunIDGeneratorProducesUniqueSafeIDs(t *testing.T) {
startedAt := time.Unix(0, 123456789).UTC()
pattern := regexp.MustCompile(`^run-123456789-[0-9a-f]{32}$`)
seen := make(map[string]struct{}, 256)
for i := 0; i < 256; i++ {
runID, err := defaultRunIDGenerator(startedAt)
if err != nil {
t.Fatal(err)
}
if !pattern.MatchString(runID) {
t.Fatalf("run ID %q does not match production format", runID)
}
if err := validateRunID(runID); err != nil {
t.Fatalf("run ID %q is not path-safe: %v", runID, err)
}
if _, exists := seen[runID]; exists {
t.Fatalf("duplicate run ID %q", runID)
}
seen[runID] = struct{}{}
}
}
func TestWriteOutputFilesSupportsNestedLogicalPaths(t *testing.T) {
runPath := filepath.Join(t.TempDir(), "output", "run-safe")
if err := writeOutputFiles(runPath, []contracts.OutputFile{{Name: "nested/result.json", Bytes: []byte("result")}}); err != nil {
t.Fatal(err)
}
data, err := os.ReadFile(filepath.Join(runPath, "nested", "result.json"))
if err != nil || string(data) != "result" {
t.Fatalf("nested output = %q, %v", data, err)
}
}
func TestWriteOutputFilesRejectsUnsafeNamesBeforeAllocatingRunDirectory(t *testing.T) {
outputRoot := filepath.Join(t.TempDir(), "output")
runPath := filepath.Join(outputRoot, "run-safe")
for _, name := range []string{"", "../outside", "/absolute", `nested\\outside`, "nested/../outside"} {
t.Run(name, func(t *testing.T) {
if err := writeOutputFiles(runPath, []contracts.OutputFile{{Name: "safe.json"}, {Name: name}}); err == nil {
t.Fatalf("writeOutputFiles accepted %q", name)
}
if _, err := os.Stat(outputRoot); !os.IsNotExist(err) {
t.Fatalf("output root exists or stat failed after %q: %v", name, err)
}
})
}
}
func TestWriteOutputFilesRetainsNewPartialDirectoryAndPreservesSibling(t *testing.T) {
outputRoot := filepath.Join(t.TempDir(), "output")
siblingPath := filepath.Join(outputRoot, "sibling")
if err := os.MkdirAll(siblingPath, 0o755); err != nil {
t.Fatal(err)
}
sentinelPath := filepath.Join(siblingPath, "sentinel")
if err := os.WriteFile(sentinelPath, []byte("preserve sibling"), 0o644); err != nil {
t.Fatal(err)
}
runPath := filepath.Join(outputRoot, "run-safe")
err := writeOutputFiles(runPath, []contracts.OutputFile{
{Name: "blocked", Bytes: []byte("partial output")},
{Name: "blocked/nested.json", Bytes: []byte("unreachable")},
})
if err == nil || !strings.Contains(err.Error(), "create output directory") {
t.Fatalf("writeOutputFiles() error = %v, want later directory failure", err)
}
if got, err := os.ReadFile(filepath.Join(runPath, "blocked")); err != nil || string(got) != "partial output" {
t.Fatalf("partial output = %q, %v", got, err)
}
if got, err := os.ReadFile(sentinelPath); err != nil || string(got) != "preserve sibling" {
t.Fatalf("sibling sentinel = %q, %v", got, err)
}
}

111
internal/cli/run_result.go
View File

@@ -1,111 +0,0 @@
package cli
import (
"encoding/json"
"fmt"
"io"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const runResultSchemaVersion = "notarius.run-result.v1"
type runResult struct {
SchemaVersion string `json:"schema_version"`
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
OutputDirectory string `json:"output_directory"`
IndexFile string `json:"index_file,omitempty"`
NormalizedOutputCount int `json:"normalized_output_count"`
RejectedOutputCount int `json:"rejected_output_count"`
WarningCount int `json:"warning_count"`
ValidationStatus string `json:"validation_status"`
DebugDirectory string `json:"debug_directory,omitempty"`
}
func newRunResult(resolved pipeline.ResolvedPipeline, output pipeline.RunOutput, outputDirectory, debugDirectory string) (runResult, error) {
if strings.TrimSpace(output.Manifest.RunID) == "" {
return runResult{}, fmt.Errorf("run result requires a run ID")
}
if strings.TrimSpace(resolved.ID) == "" {
return runResult{}, fmt.Errorf("run result requires a resolved pipeline ID")
}
if strings.TrimSpace(output.Manifest.PipelineID) == "" {
return runResult{}, fmt.Errorf("run result requires a manifest pipeline ID")
}
if output.Manifest.PipelineID != resolved.ID {
return runResult{}, fmt.Errorf("run result pipeline ID does not match resolved pipeline")
}
if strings.TrimSpace(output.Manifest.ValidationStatus) == "" {
return runResult{}, fmt.Errorf("run result requires a validation status")
}
if strings.TrimSpace(outputDirectory) == "" {
return runResult{}, fmt.Errorf("run result requires an output directory")
}
absOutputDirectory, err := filepath.Abs(outputDirectory)
if err != nil {
return runResult{}, fmt.Errorf("make output directory absolute: %w", err)
}
result := runResult{
SchemaVersion: runResultSchemaVersion,
RunID: output.Manifest.RunID,
PipelineID: output.Manifest.PipelineID,
OutputDirectory: absOutputDirectory,
NormalizedOutputCount: len(output.NormalizeOutputs),
RejectedOutputCount: len(output.Rejected),
WarningCount: len(output.Warnings),
ValidationStatus: output.Manifest.ValidationStatus,
}
if strings.TrimSpace(debugDirectory) != "" {
absDebugDirectory, err := filepath.Abs(debugDirectory)
if err != nil {
return runResult{}, fmt.Errorf("make debug directory absolute: %w", err)
}
result.DebugDirectory = absDebugDirectory
}
if resolved.Output.Module == pipeline.DefaultOutputModule {
indexCount := 0
for _, file := range output.OutputFiles {
if file.Name == "index.json" {
indexCount++
}
}
if indexCount != 1 {
return runResult{}, fmt.Errorf("production JSON output must contain exactly one index.json file")
}
result.IndexFile = "index.json"
}
return result, nil
}
func encodeRunResult(result runResult) ([]byte, error) {
encoded, err := json.Marshal(result)
if err != nil {
return nil, fmt.Errorf("encode run result: %w", err)
}
return append(encoded, '\n'), nil
}
func writeRunResult(writer io.Writer, content []byte) error {
for len(content) > 0 {
written, err := writer.Write(content)
if written < 0 || written > len(content) {
return io.ErrShortWrite
}
content = content[written:]
if err != nil {
return err
}
if written == 0 {
return io.ErrShortWrite
}
}
return nil
}

199
internal/cli/run_result_command_test.go
View File

@@ -1,199 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/always_reject"
)
func TestMaintainedMinimalInvocationEmitsRunResult(t *testing.T) {
outputRoot := filepath.Join(t.TempDir(), "output")
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"run", "dnd-session",
"--config", repositoryPath("examples", "dnd-minimal.config.yml"),
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
"--only", "spells", "--chunk_cache", "bypass", "--output-dir", outputRoot, "--json",
}, &stdout, &stderr, productionRunOptions(t, &productionFakeLLMClient{}))
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
receipt := decodeRunResultDocument(t, stdout.String())
if got := receipt["schema_version"]; got != "notarius.run-result.v1" {
t.Fatalf("schema_version = %q", got)
}
if got := receipt["run_id"]; got != productionRunID {
t.Fatalf("run_id = %q", got)
}
if got := receipt["pipeline_id"]; got != "dnd-session" {
t.Fatalf("pipeline_id = %q", got)
}
if got := receipt["index_file"]; got != "index.json" {
t.Fatalf("index_file = %q", got)
}
if got := receipt["normalized_output_count"]; got != float64(1) {
t.Fatalf("normalized_output_count = %v", got)
}
if got := receipt["rejected_output_count"]; got != float64(0) {
t.Fatalf("rejected_output_count = %v", got)
}
if got := receipt["warning_count"]; got != float64(0) {
t.Fatalf("warning_count = %v", got)
}
if got := receipt["validation_status"]; got != "approved" {
t.Fatalf("validation_status = %q", got)
}
outputDirectory, ok := receipt["output_directory"].(string)
if !ok || !filepath.IsAbs(outputDirectory) || outputDirectory != filepath.Join(outputRoot, productionRunID) {
t.Fatalf("output_directory = %q", receipt["output_directory"])
}
indexFile := receipt["index_file"].(string)
assertFile(t, filepath.Join(outputDirectory, indexFile))
}
func TestRunResultReportsWarningsAndDebugBundle(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
harness.chunkWarnings = []contracts.Warning{{Scope: "chunk", ReasonCode: "contract-warning", Message: "warning retained"}}
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--chunk_cache", "bypass", "--debug", "--json",
}, &stdout, &stderr, harness.options())
if code != 0 || !strings.Contains(stderr.String(), "1 warning(s)") {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
receipt := decodeRunResultDocument(t, stdout.String())
if got := receipt["warning_count"]; got != float64(1) {
t.Fatalf("warning_count = %v", got)
}
debugDirectory, ok := receipt["debug_directory"].(string)
if !ok || !filepath.IsAbs(debugDirectory) || debugDirectory != onlyChildDir(t, roots.debug) {
t.Fatalf("debug_directory = %q", receipt["debug_directory"])
}
if strings.Contains(stdout.String(), "complete:") || strings.Contains(stdout.String(), "debug=") {
t.Fatalf("machine stdout contains human reporting: %q", stdout.String())
}
}
func TestRunResultReportsSuccessfulRejection(t *testing.T) {
roots := newStateTestRoots(t)
configBytes, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
configBytes = []byte(replaceRequiredOnce(t, string(configBytes), " normalize: test/normalize\n", " normalize:\n module: test/normalize\n validators:\n - generic/always_reject\n"))
if err := os.WriteFile(roots.config, configBytes, 0o600); err != nil {
t.Fatal(err)
}
harness := newStateTestHarness()
opts := harness.options()
if err := alwaysreject.RegisterTyped[stateTestArtifact](opts.Registries.Validators, stateTestArtifactKind); err != nil {
t.Fatal(err)
}
opts.Catalog = catalogFromRegistries(opts.Registries)
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--chunk_cache", "bypass", "--json",
}, &stdout, &stderr, opts)
if code != 0 || stderr.Len() != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
receipt := decodeRunResultDocument(t, stdout.String())
if got := receipt["normalized_output_count"]; got != float64(0) {
t.Fatalf("normalized_output_count = %v", got)
}
if got := receipt["rejected_output_count"]; got != float64(1) {
t.Fatalf("rejected_output_count = %v", got)
}
if got := receipt["validation_status"]; got != "rejected" {
t.Fatalf("validation_status = %q", got)
}
}
func TestRunResultIsAbsentForSyntaxAndRuntimeFailures(t *testing.T) {
t.Run("syntax", func(t *testing.T) {
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--json"}, &stdout, &stderr, newStateTestHarness().options())
if code != 2 || stdout.Len() != 0 || stderr.Len() == 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
t.Run("runtime", func(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
harness.extractErr = errors.New("injected extraction failure")
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--chunk_cache", "bypass", "--json",
}, &stdout, &stderr, harness.options())
if code != 1 || stdout.Len() != 0 || stderr.Len() == 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})
}
func TestRunResultDeliveryFailureRetainsPublishedBundles(t *testing.T) {
roots := newStateTestRoots(t)
writerErr := errors.New("result writer sentinel")
stdout := &resultDeliveryWriter{err: writerErr}
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "sample", "--config", roots.config, "--input", roots.input,
"--chunk_cache", "bypass", "--debug", "--json",
}, stdout, &stderr, newStateTestHarness().options())
if code != 1 || !strings.Contains(stderr.String(), "write run result") || strings.Contains(stderr.String(), writerErr.Error()) {
t.Fatalf("code=%d stderr=%q", code, stderr.String())
}
if stdout.accepted.Len() != 0 {
t.Fatalf("accepted stdout = %q", stdout.accepted.String())
}
assertStateTestOutput(t, roots.output)
debugBundle := onlyChildDir(t, roots.debug)
report := readStateTestRunReport(t, debugBundle)
if !report.Succeeded {
t.Fatalf("debug report = %#v, want successful persisted run", report)
}
if strings.Contains(readAllFiles(t, debugBundle), writerErr.Error()) {
t.Fatalf("debug bundle contains result writer error")
}
}
func decodeRunResultDocument(t *testing.T, stdout string) map[string]any {
t.Helper()
if strings.Count(stdout, "\n") != 1 {
t.Fatalf("stdout = %q, want one JSON document", stdout)
}
var receipt map[string]any
if err := json.Unmarshal([]byte(stdout), &receipt); err != nil {
t.Fatalf("decode run result: %v; stdout=%q", err, stdout)
}
return receipt
}
type resultDeliveryWriter struct {
err error
accepted bytes.Buffer
}
func (w *resultDeliveryWriter) Write(content []byte) (int, error) {
if w.err != nil {
return 0, w.err
}
return w.accepted.Write(content)
}

200
internal/cli/run_result_test.go
View File

@@ -1,200 +0,0 @@
package cli
import (
"bytes"
"encoding/json"
"errors"
"io"
"path/filepath"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestRunResultEncodesRequiredFieldsAndCounts(t *testing.T) {
result, err := newRunResult(testResolvedPipeline(pipeline.DefaultOutputModule), testRunOutput(), "relative-output", "relative-debug")
if err != nil {
t.Fatal(err)
}
encoded, err := encodeRunResult(result)
if err != nil {
t.Fatal(err)
}
if encoded[len(encoded)-1] != '\n' || bytes.Count(encoded, []byte{'\n'}) != 1 {
t.Fatalf("encoded result is not one newline-terminated object: %q", encoded)
}
var decoded map[string]any
if err := json.Unmarshal(encoded, &decoded); err != nil {
t.Fatal(err)
}
if got := decoded["schema_version"]; got != runResultSchemaVersion {
t.Fatalf("schema_version = %q", got)
}
if got := decoded["run_id"]; got != "run-123" {
t.Fatalf("run_id = %q", got)
}
if got := decoded["pipeline_id"]; got != "sample" {
t.Fatalf("pipeline_id = %q", got)
}
if got := decoded["validation_status"]; got != "rejected" {
t.Fatalf("validation_status = %q", got)
}
if got := decoded["index_file"]; got != "index.json" {
t.Fatalf("index_file = %q", got)
}
if got := decoded["normalized_output_count"]; got != float64(2) {
t.Fatalf("normalized_output_count = %v", got)
}
if got := decoded["rejected_output_count"]; got != float64(1) {
t.Fatalf("rejected_output_count = %v", got)
}
if got := decoded["warning_count"]; got != float64(1) {
t.Fatalf("warning_count = %v", got)
}
if got := decoded["output_directory"]; got != filepath.Join(mustWorkingDirectory(t), "relative-output") {
t.Fatalf("output_directory = %q", got)
}
if got := decoded["debug_directory"]; got != filepath.Join(mustWorkingDirectory(t), "relative-debug") {
t.Fatalf("debug_directory = %q", got)
}
}
func TestRunResultRejectsInvalidRequiredValues(t *testing.T) {
tests := []struct {
name string
resolved pipeline.ResolvedPipeline
output pipeline.RunOutput
directory string
}{
{name: "blank run ID", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutputWithout(func(output *pipeline.RunOutput) { output.Manifest.RunID = " " }), directory: "output"},
{name: "blank resolved pipeline ID", resolved: pipeline.ResolvedPipeline{Output: pipeline.ModuleBinding{Module: pipeline.DefaultOutputModule}}, output: testRunOutput(), directory: "output"},
{name: "blank manifest pipeline ID", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutputWithout(func(output *pipeline.RunOutput) { output.Manifest.PipelineID = "" }), directory: "output"},
{name: "mismatched pipeline IDs", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutputWithout(func(output *pipeline.RunOutput) { output.Manifest.PipelineID = "other" }), directory: "output"},
{name: "blank validation status", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutputWithout(func(output *pipeline.RunOutput) { output.Manifest.ValidationStatus = " " }), directory: "output"},
{name: "blank output directory", resolved: testResolvedPipeline(pipeline.DefaultOutputModule), output: testRunOutput(), directory: " "},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if _, err := newRunResult(tt.resolved, tt.output, tt.directory, ""); err == nil {
t.Fatal("newRunResult() succeeded")
}
})
}
}
func TestRunResultOmitsIndexFileForOtherOutputModules(t *testing.T) {
result, err := newRunResult(testResolvedPipeline("test/output"), testRunOutputWithout(func(output *pipeline.RunOutput) {
output.OutputFiles = nil
}), "output", "")
if err != nil {
t.Fatal(err)
}
if result.IndexFile != "" {
t.Fatalf("index_file = %q", result.IndexFile)
}
encoded, err := encodeRunResult(result)
if err != nil {
t.Fatal(err)
}
var decoded map[string]any
if err := json.Unmarshal(encoded, &decoded); err != nil {
t.Fatal(err)
}
if _, ok := decoded["index_file"]; ok {
t.Fatalf("encoded non-JSON result contains index_file: %s", encoded)
}
if _, ok := decoded["debug_directory"]; ok {
t.Fatalf("encoded result without debug capture contains debug_directory: %s", encoded)
}
}
func TestRunResultRequiresOneProductionIndexFile(t *testing.T) {
tests := []struct {
name string
files []contracts.OutputFile
}{
{name: "missing", files: nil},
{name: "duplicate", files: []contracts.OutputFile{{Name: "index.json"}, {Name: "index.json"}}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
output := testRunOutput()
output.OutputFiles = tt.files
if _, err := newRunResult(testResolvedPipeline(pipeline.DefaultOutputModule), output, "output", ""); err == nil {
t.Fatal("newRunResult() succeeded")
}
})
}
}
func TestWriteRunResultCompletesAndReportsWriterFailure(t *testing.T) {
content := []byte("result\n")
var target bytes.Buffer
if err := writeRunResult(partialResultWriter{writer: &target, limit: 2}, content); err != nil {
t.Fatal(err)
}
if got := target.String(); got != string(content) {
t.Fatalf("written result = %q", got)
}
writerErr := errors.New("result writer failed")
if err := writeRunResult(failingResultWriter{err: writerErr}, content); !errors.Is(err, writerErr) {
t.Fatalf("writeRunResult() error = %v", err)
}
if err := writeRunResult(zeroResultWriter{}, content); !errors.Is(err, io.ErrShortWrite) {
t.Fatalf("zero-progress error = %v", err)
}
}
func testResolvedPipeline(outputModule string) pipeline.ResolvedPipeline {
return pipeline.ResolvedPipeline{ID: "sample", Output: pipeline.ModuleBinding{Module: outputModule}}
}
func testRunOutput() pipeline.RunOutput {
return pipeline.RunOutput{
Manifest: artifacts.RunManifest{RunID: "run-123", PipelineID: "sample", ValidationStatus: "rejected"},
NormalizeOutputs: []contracts.SerializedOutput{{}, {}},
Rejected: []contracts.RejectedOutput{{}},
Warnings: []contracts.Warning{{}},
OutputFiles: []contracts.OutputFile{{Name: "index.json"}},
}
}
func testRunOutputWithout(change func(*pipeline.RunOutput)) pipeline.RunOutput {
output := testRunOutput()
change(&output)
return output
}
func mustWorkingDirectory(t *testing.T) string {
t.Helper()
workingDirectory, err := filepath.Abs(".")
if err != nil {
t.Fatal(err)
}
return workingDirectory
}
type partialResultWriter struct {
writer io.Writer
limit int
}
func (w partialResultWriter) Write(content []byte) (int, error) {
if len(content) > w.limit {
content = content[:w.limit]
}
return w.writer.Write(content)
}
type failingResultWriter struct{ err error }
func (w failingResultWriter) Write([]byte) (int, error) { return 0, w.err }
type zeroResultWriter struct{}
func (zeroResultWriter) Write([]byte) (int, error) { return 0, nil }

90
internal/cli/run_terminal.go
View File

@@ -1,90 +0,0 @@
package cli
import (
"errors"
"fmt"
"io"
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
type DebugTerminalWriter interface {
WriteRunReport(debugbundle.RunReport) error
WriteError(string) error
}
type pipelineCommandState struct {
report debugbundle.RunReport
terminalized bool
}
func newPipelineCommandState(runID, pipelineID, outputPath string) *pipelineCommandState {
return &pipelineCommandState{report: debugbundle.RunReport{
RunID: runID,
PipelineID: pipelineID,
OutputPath: outputPath,
}}
}
func (s *pipelineCommandState) setDebugPath(debugPath string) {
if s != nil {
s.report.DebugPath = debugPath
}
}
func (s *pipelineCommandState) observeOutput(output pipeline.RunOutput) {
if s == nil {
return
}
s.report.OutputCount = len(output.NormalizeOutputs)
s.report.RejectedCount = len(output.Rejected)
s.report.WarningCount = len(output.Warnings)
s.report.ValidationStatus = output.Manifest.ValidationStatus
}
func (s *pipelineCommandState) terminalize(writer DebugTerminalWriter, primaryErr error) (error, error) {
if s == nil || s.terminalized {
return primaryErr, nil
}
s.terminalized = true
if writer == nil {
return primaryErr, nil
}
report := s.report
report.Succeeded = primaryErr == nil
reportErr := writer.WriteRunReport(report)
if reportErr != nil {
reportErr = fmt.Errorf("write debug run report: %w", reportErr)
if primaryErr == nil {
primaryErr = reportErr
reportErr = nil
}
}
var errorLogErr error
if primaryErr != nil {
if err := writer.WriteError(primaryErr.Error()); err != nil {
errorLogErr = fmt.Errorf("write debug error log: %w", err)
}
}
return primaryErr, errors.Join(reportErr, errorLogErr)
}
func failPipelineCommand(stderr io.Writer, state *pipelineCommandState, writer DebugTerminalWriter, primaryErr error, persistenceErrs ...error) int {
primaryErr, terminalErr := state.terminalize(writer, primaryErr)
persistenceErrs = append(persistenceErrs, terminalErr)
return writePipelineCommandFailure(stderr, state, primaryErr, errors.Join(persistenceErrs...))
}
func writePipelineCommandFailure(stderr io.Writer, state *pipelineCommandState, primaryErr, persistenceErr error) int {
fmt.Fprintf(stderr, "notarius: %v\n", primaryErr)
if persistenceErr != nil {
fmt.Fprintf(stderr, "notarius: %v\n", persistenceErr)
}
if state != nil && state.report.DebugPath != "" {
fmt.Fprintf(stderr, "notarius: debug=%s\n", state.report.DebugPath)
}
return 1
}

75
internal/cli/run_test.go Normal file
View File

@@ -0,0 +1,75 @@
package cli
import (
"bytes"
"strings"
"testing"
)
func TestRunNoArgsWritesUsageToStdout(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Run(nil, &stdout, &stderr)
if code != 0 {
t.Fatalf("Run() code = %d, want 0", code)
}
if stdout.String() != usage {
t.Fatalf("stdout = %q, want %q", stdout.String(), usage)
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
}
func TestRunHelpArgsWriteUsageToStdout(t *testing.T) {
tests := []struct {
name string
args []string
}{
{name: "help", args: []string{"help"}},
{name: "long help flag", args: []string{"--help"}},
{name: "short help flag", args: []string{"-h"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Run(tt.args, &stdout, &stderr)
if code != 0 {
t.Fatalf("Run() code = %d, want 0", code)
}
if stdout.String() != usage {
t.Fatalf("stdout = %q, want %q", stdout.String(), usage)
}
if stderr.Len() != 0 {
t.Fatalf("stderr = %q, want empty", stderr.String())
}
})
}
}
func TestRunUnknownCommandWritesErrorAndUsageToStderr(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Run([]string{"extract"}, &stdout, &stderr)
if code != 2 {
t.Fatalf("Run() code = %d, want 2", code)
}
if stdout.Len() != 0 {
t.Fatalf("stdout = %q, want empty", stdout.String())
}
gotStderr := stderr.String()
if !strings.Contains(gotStderr, "notarius: unknown command \"extract\"") {
t.Fatalf("stderr = %q, want unknown command error", gotStderr)
}
if !strings.Contains(gotStderr, usage) {
t.Fatalf("stderr = %q, want usage", gotStderr)
}
}

503
internal/cli/spell_catalog_identity_contract_test.go
View File

@@ -1,503 +0,0 @@
package cli
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/checkpoint"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
spellcatalog "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/spells/catalog"
)
func TestSpellCatalogBytesAffectCheckpointIdentityButNotSemanticDigest(t *testing.T) {
components := productionTestComponents(t)
configPath := writeProductionSpellCatalogContractConfig(t)
effective, err := loadMaintainedExample(t, configPath).Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
if err != nil {
t.Fatalf("resolve production configuration: %v", err)
}
overlayPath := filepath.Join(t.TempDir(), "catalog.json")
resolved := effective.ResolvedPipeline
bindings := resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings
catalogBindingIndex := -1
for index, binding := range bindings {
if binding.SlotName == spellcatalog.SpellCatalogReferenceSlot {
catalogBindingIndex = index
break
}
}
if catalogBindingIndex < 0 {
t.Fatalf("spell catalog bindings = %#v, want catalog binding", bindings)
}
resolved.Steps[0].ArtifactLanes[0].ExtractReferences.Bindings[catalogBindingIndex].Source = overlayPath
normalizeBindings := resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings
normalizeCatalogBindingIndex := -1
for index, binding := range normalizeBindings {
if binding.SlotName == spellcatalog.SpellCatalogReferenceSlot {
normalizeCatalogBindingIndex = index
break
}
}
if normalizeCatalogBindingIndex < 0 {
t.Fatalf("normalize spell catalog bindings = %#v, want catalog binding", normalizeBindings)
}
resolved.Steps[0].ArtifactLanes[0].NormalizeReferences.Bindings[normalizeCatalogBindingIndex].Source = overlayPath
if err := os.WriteFile(overlayPath, []byte(reorderedOverlayA), 0o600); err != nil {
t.Fatal(err)
}
materializedA, _, err := pipeline.MaterializeReferences(resolved, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{ConfigPath: configPath, WorkingDir: filepath.Dir(configPath)})
if err != nil {
t.Fatalf("materialize first catalog: %v", err)
}
identityA := catalogCheckpointIdentity(t, materializedA)
metadataA := catalogExtractorMetadata(t, materializedA)
normalizerMetadataA := catalogNormalizerMetadata(t, materializedA)
referenceA := catalogReference(t, materializedA)
if err := os.WriteFile(overlayPath, []byte(reorderedOverlayB), 0o600); err != nil {
t.Fatal(err)
}
materializedB, _, err := pipeline.MaterializeReferences(resolved, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{ConfigPath: configPath, WorkingDir: filepath.Dir(configPath)})
if err != nil {
t.Fatalf("materialize reordered catalog: %v", err)
}
identityB := catalogCheckpointIdentity(t, materializedB)
metadataB := catalogExtractorMetadata(t, materializedB)
normalizerMetadataB := catalogNormalizerMetadata(t, materializedB)
referenceB := catalogReference(t, materializedB)
if identityA.Digest == identityB.Digest {
t.Fatalf("checkpoint identity digest = %q for both raw catalog files, want invalidation", identityA.Digest)
}
if referenceA.Digest == referenceB.Digest || referenceA.OriginURI != referenceB.OriginURI {
t.Fatalf("catalog reference provenance changed from %#v to %#v, want same origin and different raw digest", referenceA, referenceB)
}
digestA, ok := metadataA["catalog_digest"].(string)
if !ok {
t.Fatalf("first extractor catalog metadata = %#v, want digest", metadataA)
}
digestB, ok := metadataB["catalog_digest"].(string)
if !ok || digestA != digestB {
t.Fatalf("extractor catalog digests = %q and %q, want same semantic digest", digestA, digestB)
}
if got, want := metadataA["catalog_overlay_ids"], []string{"campaign.a", "campaign.b"}; !reflect.DeepEqual(got, want) || !reflect.DeepEqual(metadataB["catalog_overlay_ids"], want) {
t.Fatalf("extractor overlay IDs = %#v and %#v, want %#v", got, metadataB["catalog_overlay_ids"], want)
}
normalizerDigestA, ok := normalizerMetadataA["catalog_digest"].(string)
normalizerDigestB, okB := normalizerMetadataB["catalog_digest"].(string)
if !ok || !okB || normalizerDigestA != digestA || normalizerDigestB != digestB {
t.Fatalf("normalizer catalog digests = %#v and %#v, want extractor semantic digests %q and %q", normalizerMetadataA["catalog_digest"], normalizerMetadataB["catalog_digest"], digestA, digestB)
}
if got, want := normalizerMetadataA["catalog_overlay_ids"], []string{"campaign.a", "campaign.b"}; !reflect.DeepEqual(got, want) || !reflect.DeepEqual(normalizerMetadataB["catalog_overlay_ids"], want) {
t.Fatalf("normalizer overlay IDs = %#v and %#v, want %#v", got, normalizerMetadataB["catalog_overlay_ids"], want)
}
}
func TestConfiguredSpellCatalogBindingChangesResolvedPipelineIdentity(t *testing.T) {
base := productionSpellCatalogContractConfig(t)
changed := strings.Replace(base, repositoryPath("examples", "dnd-spell-catalog.json"), filepath.Join(t.TempDir(), "alternate-spell-catalog.json"), 1)
if changed == base {
t.Fatal("production configuration did not contain the maintained catalog binding")
}
root := t.TempDir()
firstPath := filepath.Join(root, "first.yml")
secondPath := filepath.Join(root, "second.yml")
if err := os.WriteFile(firstPath, []byte(base), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(secondPath, []byte(changed), 0o600); err != nil {
t.Fatal(err)
}
components := productionTestComponents(t)
first, err := loadMaintainedExample(t, firstPath).Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
if err != nil {
t.Fatalf("resolve first configuration: %v", err)
}
second, err := loadMaintainedExample(t, secondPath).Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
if err != nil {
t.Fatalf("resolve changed configuration: %v", err)
}
if first.ResolvedPipeline.Digest == second.ResolvedPipeline.Digest {
t.Fatalf("resolved pipeline digest = %q for different catalog bindings, want change", first.ResolvedPipeline.Digest)
}
}
func TestSemanticSpellCatalogFingerprintChangesCheckpointIdentityWithoutReferenceChange(t *testing.T) {
components := productionTestComponents(t)
configPath := writeProductionSpellCatalogContractConfig(t)
effective, err := loadMaintainedExample(t, configPath).Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
if err != nil {
t.Fatal(err)
}
materialized, _, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{ConfigPath: configPath, WorkingDir: filepath.Dir(configPath)})
if err != nil {
t.Fatal(err)
}
prepared, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: &productionFakeLLMClient{}})
if err != nil {
t.Fatal(err)
}
fingerprints := prepared.CheckpointFingerprints()
wantNames := map[string]struct{}{
"extract:spells:" + spells.Key + ":effective_catalog": {},
"extract:spells:" + spells.Key + ":validator:2:extract/dnd/spells/catalog:effective_catalog": {},
"normalize:spells:" + spellnormalize.Key + ":effective_catalog": {},
"normalize:spells:" + spellnormalize.Key + ":validator:2:extract/dnd/spells/catalog:effective_catalog": {},
}
seen := make(map[string]string, len(fingerprints))
for _, fingerprint := range fingerprints {
if _, ok := wantNames[fingerprint.Name]; ok {
seen[fingerprint.Name] = fingerprint.Value
}
}
if len(seen) != len(wantNames) {
t.Fatalf("prepared fingerprints = %#v, want scoped extractor and normalize catalog identities", fingerprints)
}
var catalogDigest string
for name, value := range seen {
if catalogDigest == "" {
catalogDigest = value
} else if value != catalogDigest {
t.Fatalf("prepared fingerprint %q = %q, want shared semantic catalog digest %q", name, value, catalogDigest)
}
}
identityFor := func(values []pipeline.CheckpointFingerprint) checkpoint.Identity {
identity, identityErr := checkpoint.NewIdentity(checkpoint.IdentityInput{
Pipeline: materialized,
InputKey: materialized.Input.Module,
RawInputDigest: "sha256:unchanged-input",
References: pipeline.ReferenceProvenance(materialized),
ProvenanceFingerprints: checkpointIdentityFingerprints(values),
})
if identityErr != nil {
t.Fatal(identityErr)
}
return identity
}
first := identityFor(fingerprints)
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
second := identityFor(changed)
if first.Digest == second.Digest || reflect.DeepEqual(first.ReferenceDigests, nil) || !reflect.DeepEqual(first.ReferenceDigests, second.ReferenceDigests) {
t.Fatalf("identities = %#v / %#v, want semantic invalidation with unchanged reference provenance", first, second)
}
}
func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t *testing.T) {
components := productionTestComponents(t)
configPath := writeProductionSpellCatalogContractConfig(t)
effective, err := loadMaintainedExample(t, configPath).Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
if err != nil {
t.Fatal(err)
}
materialized, _, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{ConfigPath: configPath, WorkingDir: filepath.Dir(configPath)})
if err != nil {
t.Fatal(err)
}
prepared, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: &productionFakeLLMClient{}})
if err != nil {
t.Fatal(err)
}
fingerprints := prepared.CheckpointFingerprints()
llmFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-one"}}
settings := config.CheckpointCacheConfig{Enabled: true, Directory: t.TempDir()}
recorder, _, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, false)
if err != nil {
t.Fatal(err)
}
doc := source.SourceDocument{ID: "source", Kind: "transcript", Format: "application/json"}
doc.Units = []source.SourceUnit{{ID: 1, Kind: "turn", Text: "Aria casts Cure Wounds.", Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 1}}}
doc.Digest, err = source.DigestDocument(&doc)
if err != nil {
t.Fatal(err)
}
if err := recorder.SourceSucceeded(materialized.Input.Module, &doc); err != nil {
t.Fatal(err)
}
normalizeDependencies := []pipeline.CheckpointFingerprint{{Name: "artifact[0]", Value: "sha256:merged-artifact"}}
normalizeSchema := contracts.ArtifactSchema{ID: "notarius.dnd.spells", Name: "notarius_dnd_spells", Version: "v1"}
normalizeArtifact := pipeline.CheckpointArtifact{
LaneID: "spells", ModuleKey: spellnormalize.Key, SourceID: doc.ID,
SchemaDigest: contracts.DigestArtifactSchema(normalizeSchema),
Artifact: contracts.SerializedArtifact{
Kind: dnd.SpellListKind, Schema: normalizeSchema, MediaType: "application/json", Content: []byte(`{"spell_casts":[]}`),
},
}
if err := recorder.NormalizeSucceeded("spells", spellnormalize.Key, normalizeDependencies, normalizeArtifact, nil); err != nil {
t.Fatal(err)
}
_, sameLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}
if _, decision := sameLoader.Source(materialized.Input.Module); !decision.Reused {
t.Fatalf("same fingerprint decision = %#v, want reuse", decision)
}
if restored, decision := sameLoader.Normalize("spells", spellnormalize.Key, normalizeDependencies); !decision.Reused || string(restored.Output.Artifact.Content) != `{"spell_casts":[]}` {
t.Fatalf("same normalize checkpoint = %#v, decision=%#v, want reuse", restored, decision)
}
changed := replaceCheckpointFingerprintValue(t, fingerprints, normalizeSpellCatalogFingerprintName(), "sha256:changed-effective-catalog")
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changed, normalizeSpellCatalogFingerprintName())
_, changedLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changed, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}
if _, decision := changedLoader.Source(materialized.Input.Module); decision.Reused {
t.Fatalf("changed fingerprint decision = %#v, want cold miss", decision)
}
if _, decision := changedLoader.Normalize("spells", spellnormalize.Key, normalizeDependencies); decision.Reused {
t.Fatalf("changed normalize fingerprint decision = %#v, want normalize checkpoint cold miss", decision)
}
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}
if _, decision := mappingLoader.Source(materialized.Input.Module); decision.Reused {
t.Fatalf("changed mapping policy decision = %#v, want cold miss", decision)
}
changedLLMFingerprints := []checkpoint.Fingerprint{{Name: "promptkit_profile_source", Value: "sha256:profile-source-two"}}
_, profileLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, fingerprints, changedLLMFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
if err != nil {
t.Fatal(err)
}
if _, decision := profileLoader.Source(materialized.Input.Module); decision.Reused {
t.Fatalf("changed PromptKit profile source decision = %#v, want cold miss", decision)
}
if _, decision := profileLoader.Normalize("spells", spellnormalize.Key, normalizeDependencies); decision.Reused {
t.Fatalf("changed PromptKit profile normalize decision = %#v, want cold miss", decision)
}
}
func normalizeSpellCatalogFingerprintName() string {
return "normalize:spells:" + spellnormalize.Key + ":effective_catalog"
}
func extractSpellMappingFingerprintName() string {
return "extract:spells:" + spells.Key + ":mapping_policy"
}
func replaceCheckpointFingerprintValue(t *testing.T, fingerprints []pipeline.CheckpointFingerprint, name, value string) []pipeline.CheckpointFingerprint {
t.Helper()
changed := append([]pipeline.CheckpointFingerprint(nil), fingerprints...)
matches := 0
for index := range changed {
if changed[index].Name == name {
changed[index].Value = value
matches++
}
}
if matches != 1 {
t.Fatalf("checkpoint fingerprints = %#v, want exactly one fingerprint named %q", fingerprints, name)
}
return changed
}
func assertOnlyCheckpointFingerprintChanged(t *testing.T, before, after []pipeline.CheckpointFingerprint, changedName string) {
t.Helper()
if len(before) != len(after) {
t.Fatalf("fingerprint lengths = %d and %d, want equal", len(before), len(after))
}
changes := 0
for index := range before {
if before[index].Name != after[index].Name {
t.Fatalf("fingerprint[%d] name changed from %q to %q", index, before[index].Name, after[index].Name)
}
if before[index].Value == after[index].Value {
continue
}
changes++
if before[index].Name != changedName {
t.Fatalf("fingerprint %q changed unexpectedly", before[index].Name)
}
}
if changes != 1 {
t.Fatalf("fingerprints changed %d values, want exactly %q", changes, changedName)
}
}
func TestMaintainedProductionOverlayRunAlignsGroundingValidationAndProvenance(t *testing.T) {
outputRoot := filepath.Join(t.TempDir(), "output")
fake := &productionFakeLLMClient{spellResponse: productionSpellResponse("Aegis of Emberfall")}
options := productionRunOptions(t, fake)
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"run", "dnd-session",
"--config", writeProductionSpellCatalogContractConfig(t),
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
"--only", "spells", "--chunk_cache", "bypass", "--output-dir", outputRoot,
}, &stdout, &stderr, options)
if code != 0 {
t.Fatalf("code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
runRoot := filepath.Join(outputRoot, productionRunID)
manifest := readProductionJSON[artifacts.RunManifest](t, filepath.Join(runRoot, "manifest.json"))
if manifest.ValidationStatus != "approved" || len(manifest.References) == 0 || len(manifest.ArtifactLanes) != 1 {
t.Fatalf("manifest = %#v, want approved overlay run with one lane and references", manifest)
}
lane := manifest.ArtifactLanes[0]
extractorMetadata, ok := lane.Metadata["extractor"].(map[string]any)
if !ok {
t.Fatalf("lane metadata = %#v, want extractor metadata", lane.Metadata)
}
if extractorMetadata["catalog_base_id"] != spellcatalog.SRD5E2014ID || !strings.HasPrefix(stringValue(extractorMetadata["catalog_digest"]), "sha256:") {
t.Fatalf("extractor catalog metadata = %#v, want base ID and semantic digest", extractorMetadata)
}
if got := stringValues(extractorMetadata["catalog_overlay_ids"]); !reflect.DeepEqual(got, []string{"notarius.example-campaign"}) {
t.Fatalf("catalog overlay IDs = %#v, want maintained overlay", got)
}
normalizerMetadata, ok := lane.Metadata["normalizer"].(map[string]any)
if !ok {
t.Fatalf("lane metadata = %#v, want normalizer metadata", lane.Metadata)
}
if normalizerMetadata["catalog_base_id"] != spellcatalog.SRD5E2014ID || !strings.HasPrefix(stringValue(normalizerMetadata["catalog_digest"]), "sha256:") || !reflect.DeepEqual(stringValues(normalizerMetadata["catalog_overlay_ids"]), []string{"notarius.example-campaign"}) {
t.Fatalf("normalizer catalog metadata = %#v, want base ID, semantic digest, and overlay IDs", normalizerMetadata)
}
if normalizerMetadata["catalog_digest"] != extractorMetadata["catalog_digest"] || !reflect.DeepEqual(stringValues(normalizerMetadata["catalog_overlay_ids"]), stringValues(extractorMetadata["catalog_overlay_ids"])) {
t.Fatalf("extractor metadata = %#v, normalizer metadata = %#v, want shared catalog identity", extractorMetadata, normalizerMetadata)
}
var catalogProvenances []artifacts.ReferenceProvenance
for index := range manifest.References {
reference := &manifest.References[index]
if reference.SlotName == spellcatalog.SpellCatalogReferenceSlot {
catalogProvenances = append(catalogProvenances, *reference)
}
}
if len(catalogProvenances) != 2 {
t.Fatalf("manifest references = %#v, want independently materialized extract and normalize catalog provenance", manifest.References)
}
overlayBytes := readRepositoryFile(t, "examples", "dnd-spell-catalog.json")
for _, catalogProvenance := range catalogProvenances {
if catalogProvenance.Stage != "extract" && catalogProvenance.Stage != "normalize" {
t.Fatalf("catalog provenance = %#v, want extract or normalize scope", catalogProvenance)
}
if catalogProvenance.LaneID != "spells" || catalogProvenance.OriginType != "file" || catalogProvenance.MediaType != "application/json" || catalogProvenance.SizeBytes != int64(len(overlayBytes)) || catalogProvenance.Digest != digestBytes(overlayBytes) || !strings.Contains(catalogProvenance.OriginURI, "dnd-spell-catalog.json") {
t.Fatalf("catalog provenance = %#v, want raw overlay provenance in both scopes", catalogProvenance)
}
}
manifestBytes, err := json.Marshal(manifest)
if err != nil {
t.Fatal(err)
}
for _, leaked := range []string{"Aegis of Emberfall", "Emberfall Aegis", "Notarius example campaign spell names"} {
if strings.Contains(string(manifestBytes), leaked) {
t.Fatalf("manifest leaked overlay content %q", leaked)
}
}
requests := fake.requestsFor(spells.PromptID)
if len(requests) != 1 {
t.Fatalf("spell requests = %d, want one", len(requests))
}
catalogInput, ok := requests[0].Inputs[spellcatalog.SpellCatalogReferenceSlot]
if !ok || !strings.Contains(string(catalogInput.Content), "Aegis of Emberfall") || strings.Contains(string(catalogInput.Content), "Emberfall Aegis") {
t.Fatalf("spell catalog prompt input = %#v, want canonical overlay name without alias", catalogInput)
}
artifact := readProductionJSON[dnd.SpellList](t, filepath.Join(runRoot, "lanes", "spells.json"))
if len(artifact.SpellCasts) != 1 || artifact.SpellCasts[0].Spell != "Aegis of Emberfall" {
t.Fatalf("artifact = %#v, want accepted overlay-only canonical spell", artifact)
}
rejected := readProductionJSON[struct {
Rejected []json.RawMessage `json:"rejected"`
}](t, filepath.Join(runRoot, "rejected.json"))
if len(rejected.Rejected) != 0 {
t.Fatalf("rejected = %#v, want no rejected output", rejected.Rejected)
}
}
func catalogCheckpointIdentity(t *testing.T, resolved pipeline.ResolvedPipeline) checkpoint.Identity {
t.Helper()
identity, err := checkpoint.NewIdentity(checkpoint.IdentityInput{
Pipeline: resolved,
InputKey: resolved.Input.Module,
RawInputDigest: "sha256:catalog-test-input",
References: pipeline.ReferenceProvenance(resolved),
})
if err != nil {
t.Fatalf("create checkpoint identity: %v", err)
}
return identity
}
func catalogExtractorMetadata(t *testing.T, resolved pipeline.ResolvedPipeline) map[string]any {
t.Helper()
lane := resolved.Steps[0].ArtifactLanes[0]
extractor, err := spells.New(&productionFakeLLMClient{}, spells.Options{}, lane.ExtractReferences.ReferenceSet)
if err != nil {
t.Fatalf("construct extractor: %v", err)
}
return extractor.ManifestMetadata()
}
func catalogNormalizerMetadata(t *testing.T, resolved pipeline.ResolvedPipeline) map[string]any {
t.Helper()
lane := resolved.Steps[0].ArtifactLanes[0]
normalizer, err := spellnormalize.New(spellnormalize.Options{}, lane.NormalizeReferences.ReferenceSet)
if err != nil {
t.Fatalf("construct normalizer: %v", err)
}
return normalizer.ManifestMetadata()
}
func catalogReference(t *testing.T, resolved pipeline.ResolvedPipeline) artifacts.ReferenceProvenance {
t.Helper()
for _, reference := range pipeline.ReferenceProvenance(resolved) {
if reference.SlotName == spellcatalog.SpellCatalogReferenceSlot && reference.Stage == "extract" && reference.LaneID == "spells" {
return reference
}
}
t.Fatalf("resolved references = %#v, want spell catalog provenance", pipeline.ReferenceProvenance(resolved))
return artifacts.ReferenceProvenance{}
}
func stringValue(value any) string {
result, _ := value.(string)
return result
}
func stringValues(value any) []string {
raw, err := json.Marshal(value)
if err != nil {
return nil
}
var values []string
if err := json.Unmarshal(raw, &values); err != nil {
return nil
}
return values
}
func digestBytes(value []byte) string {
sum := sha256.Sum256(value)
return "sha256:" + hex.EncodeToString(sum[:])
}
const reorderedOverlayA = `{
"schema_version": "notarius.dnd.spell-catalog-overlay.v1",
"catalogs": [
{"id":"campaign.a","ruleset":"dnd-5e-2014","source":{"title":"Campaign A"},"spells":[{"name":"Aegis of Emberfall","aliases":["Emberfall Aegis"]}]},
{"id":"campaign.b","ruleset":"dnd-5e-2014","source":{"title":"Campaign B"},"spells":[{"name":"Cinder Veil","aliases":["Veil of Cinder","Cinder Shroud"]}]}
]
}`
const reorderedOverlayB = `{"catalogs":[{"spells":[{"aliases":["Cinder Shroud","Veil of Cinder"],"name":"Cinder Veil"}],"source":{"title":"Campaign B"},"ruleset":"dnd-5e-2014","id":"campaign.b"},{"spells":[{"aliases":["Emberfall Aegis"],"name":"Aegis of Emberfall"}],"source":{"title":"Campaign A"},"ruleset":"dnd-5e-2014","id":"campaign.a"}],"schema_version":"notarius.dnd.spell-catalog-overlay.v1"}`

160
internal/cli/spell_catalog_retry_contract_test.go
View File

@@ -1,160 +0,0 @@
package cli
import (
"context"
"encoding/json"
"fmt"
"path/filepath"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
)
func TestProductionSpellCatalogValidationRetries(t *testing.T) {
const retries = 2
tests := []struct {
name string
responses []string
wantCalls int
wantRejected bool
wantSpell string
wantWarningCode string
}{
{
name: "unknown spell remains rejected after exhaustion",
responses: []string{
productionSpellResponse("Unknown Spell"),
productionSpellResponse("Unknown Spell"),
productionSpellResponse("Unknown Spell"),
},
wantCalls: retries + 1,
wantRejected: true,
},
{
name: "overlay spell becomes valid on retry",
responses: []string{
productionSpellResponse("Unknown Spell"),
productionSpellResponse("Aegis of Emberfall"),
},
wantCalls: 2,
wantSpell: "Aegis of Emberfall",
wantWarningCode: "spell_not_near_source",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
components := productionTestComponents(t)
configPath := writeProductionSpellCatalogContractConfig(t)
cfg := loadMaintainedExample(t, configPath)
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(components.registries)})
if err != nil {
t.Fatalf("resolve production configuration: %v", err)
}
materialized, _, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{
ConfigPath: configPath,
WorkingDir: filepath.Dir(configPath),
})
if err != nil {
t.Fatalf("materialize production references: %v", err)
}
materialized.Steps[0].ArtifactLanes[0].Extract.Retries = retries
llmClient := &catalogRetryLLMClient{responses: tt.responses}
prepared, err := pipeline.Prepare(materialized, components.registries, pipeline.ModuleDependencies{LLM: llmClient})
if err != nil {
t.Fatalf("prepare production pipeline: %v", err)
}
output, err := pipeline.New().Run(context.Background(), pipeline.RunInput{
Prepared: prepared,
RawInput: readRepositoryFile(t, "examples", "seriatim-minimal-transcript.json"),
ChunkCacheMode: pipeline.ChunkCacheBypass,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if calls := llmClient.CallCount(); calls > retries+1 || calls != tt.wantCalls {
t.Fatalf("LLM calls = %d, want %d and no more than %d", calls, tt.wantCalls, retries+1)
}
if tt.wantRejected {
if len(output.Rejected) != 1 || len(output.NormalizeOutputs) != 0 {
t.Fatalf("rejected = %#v normalized = %#v, want one nonfatal rejection and no merge output", output.Rejected, output.NormalizeOutputs)
}
rejection := output.Rejected[0]
if rejection.ReasonCode != "unknown_spell" || rejection.AttemptCount != retries+1 {
t.Fatalf("rejection = %#v, want exhausted unknown-spell rejection", rejection)
}
if len(output.Warnings) != 0 {
t.Fatalf("warnings = %#v, want no warnings from rejected attempts", output.Warnings)
}
return
}
if len(output.Rejected) != 0 || len(output.NormalizeOutputs) != 1 {
t.Fatalf("rejected = %#v normalized = %#v, want only accepted output", output.Rejected, output.NormalizeOutputs)
}
var value dnd.SpellList
if err := json.Unmarshal(output.NormalizeOutputs[0].Artifact.Content, &value); err != nil {
t.Fatalf("decode normalized spell list: %v", err)
}
if len(value.SpellCasts) != 1 || value.SpellCasts[0].Spell != tt.wantSpell {
t.Fatalf("normalized spell list = %#v, want accepted overlay spell", value)
}
if len(output.Warnings) != 2 || output.Warnings[0].ReasonCode != tt.wantWarningCode || output.Warnings[1].ReasonCode != tt.wantWarningCode {
t.Fatalf("warnings = %#v, want accepted-attempt warnings from extract and normalize validation", output.Warnings)
}
})
}
}
type catalogRetryLLMClient struct {
mu sync.Mutex
responses []string
calls int
}
func (client *catalogRetryLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
if err := ctx.Err(); err != nil {
return contracts.StructuredCompletionResponse{}, err
}
if req.PromptID != spells.PromptID {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected prompt %q", req.PromptID)
}
client.mu.Lock()
index := client.calls
client.calls++
client.mu.Unlock()
if index >= len(client.responses) {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("missing fake response %d", index)
}
content := []byte(client.responses[index])
if err := json.Unmarshal(content, out); err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate fake structured target: %w", err)
}
return contracts.StructuredCompletionResponse{Content: content, Provider: "test", Model: "deterministic", ProfileID: req.ProfileID}, nil
}
func (client *catalogRetryLLMClient) CallCount() int {
client.mu.Lock()
defer client.mu.Unlock()
return client.calls
}
func productionSpellResponse(name string) string {
content, err := json.Marshal(dnd.SpellList{SpellCasts: []dnd.SpellCast{{
Caster: "Aria",
Spell: name,
SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}},
}}})
if err != nil {
panic(err)
}
return string(content)
}

988
internal/cli/state_hardening_test.go
View File

@@ -1,988 +0,0 @@
package cli
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"runtime"
"strings"
"sync"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/chunkplan"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
frameworkdebug "gitea.maximumdirect.net/eric/notarius/internal/framework/debug"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const stateTestDigest = "sha256:e511d8906649b78eb639b11215fa57a9652a1a64f4aefa3ed68320dbda46f439"
func TestRunStateSurfaceMatrix(t *testing.T) {
for _, debug := range []bool{false, true} {
for _, resume := range []bool{false, true} {
for _, mode := range []string{"auto", "bypass", "refresh"} {
name := fmt.Sprintf("debug=%t/resume=%t/cache=%s", debug, resume, mode)
t.Run(name, func(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
opts := harness.options()
var storeRoots []string
opts.ChunkPlanStoreFactory = func(root string) (pipeline.ChunkPlanStore, error) {
storeRoots = append(storeRoots, root)
return chunkplan.NewFilesystemStore(root)
}
result := runStateTest(t, roots, opts, debug, resume, mode)
if result.code != 0 {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
assertStateTestOutput(t, roots.output)
if mode == "bypass" {
assertAbsent(t, roots.plans)
if len(storeRoots) != 0 {
t.Fatalf("chunk plan store roots = %v, want none", storeRoots)
}
} else {
assertFile(t, filepath.Join(roots.plans, strings.TrimPrefix(stateTestDigest, "sha256:"), "plan.json"))
if len(storeRoots) != 1 || storeRoots[0] != roots.plans {
t.Fatalf("chunk plan store roots = %v, want [%q]", storeRoots, roots.plans)
}
}
assertAnyFile(t, roots.checkpoints)
assertRestrictedTree(t, roots.checkpoints)
if debug {
bundle := onlyChildDir(t, roots.debug)
assertFile(t, filepath.Join(bundle, "summary", "invocation.json"))
assertAnyFile(t, filepath.Join(bundle, "trace"))
assertRestrictedTree(t, roots.debug)
} else {
assertAbsent(t, roots.debug)
}
})
}
}
}
}
func TestRunKeepsStateRootsIndependentAndReusesSelectedCheckpointRoot(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
first := runStateTest(t, roots, harness.options(), true, false, "auto")
if first.code != 0 {
t.Fatalf("first run code=%d stderr=%q", first.code, first.stderr)
}
planPath := filepath.Join(roots.plans, strings.TrimPrefix(stateTestDigest, "sha256:"), "plan.json")
initialPlan, err := os.ReadFile(planPath)
if err != nil {
t.Fatal(err)
}
firstBundle := onlyChildDir(t, roots.debug)
second := runStateTest(t, roots, harness.options(), false, false, "auto")
if second.code != 0 {
t.Fatalf("second run code=%d stderr=%q", second.code, second.stderr)
}
if harness.chunkCalls != 1 {
t.Fatalf("chunk calls after debug toggle = %d, want 1", harness.chunkCalls)
}
if harness.extractCalls != 2 {
t.Fatalf("extract calls after two recording-only runs = %d, want 2", harness.extractCalls)
}
if got, err := os.ReadFile(planPath); err != nil || !bytes.Equal(got, initialPlan) {
t.Fatalf("chunk plan changed after debug toggle: %v", err)
}
if _, err := os.Stat(firstBundle); err != nil {
t.Fatalf("initial debug bundle was removed: %v", err)
}
checkpointRoot := roots.checkpoints
extractCallsBeforeResume := harness.extractCalls
seed := runStateTest(t, roots, harness.options(), false, true, "auto")
if seed.code != 0 {
t.Fatalf("checkpoint seed code=%d stderr=%q", seed.code, seed.stderr)
}
if harness.extractCalls != extractCallsBeforeResume {
t.Fatalf("extract calls after reusing recording-only checkpoint = %d, want %d", harness.extractCalls, extractCallsBeforeResume)
}
extractCalls := harness.extractCalls
checkpointFiles := readTree(t, checkpointRoot)
reused := runStateTest(t, roots, harness.options(), false, true, "auto")
if reused.code != 0 {
t.Fatalf("checkpoint reuse code=%d stderr=%q", reused.code, reused.stderr)
}
if harness.extractCalls != extractCalls {
t.Fatalf("extract calls after checkpoint reuse = %d, want %d", harness.extractCalls, extractCalls)
}
if got := readTree(t, checkpointRoot); !sameFiles(got, checkpointFiles) {
t.Fatal("reused checkpoint was rewritten")
}
}
func TestRunRecomputesOnlyAfterExplicitChunkPlanRemoval(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
first := runStateTest(t, roots, harness.options(), true, false, "auto")
if first.code != 0 {
t.Fatalf("first run code=%d stderr=%q", first.code, first.stderr)
}
firstOutput := onlyChildDir(t, roots.output)
firstBundle := onlyChildDir(t, roots.debug)
entry := filepath.Join(roots.plans, strings.TrimPrefix(stateTestDigest, "sha256:"))
if err := os.RemoveAll(entry); err != nil {
t.Fatal(err)
}
second := runStateTest(t, roots, harness.options(), false, false, "auto")
if second.code != 0 {
t.Fatalf("second run code=%d stderr=%q", second.code, second.stderr)
}
if harness.chunkCalls != 2 {
t.Fatalf("chunk calls = %d, want 2 after removing exact cache entry", harness.chunkCalls)
}
assertFile(t, filepath.Join(firstOutput, "result.json"))
assertFile(t, filepath.Join(firstBundle, "summary", "run-report.json"))
}
func TestRunRetainsDebugBundlesAcrossFailures(t *testing.T) {
t.Run("configuration failure precedes allocation", func(t *testing.T) {
root := filepath.Join(t.TempDir(), "debug")
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "sample", "--config", filepath.Join(t.TempDir(), "missing.yml"), "--input", "missing", "--debug", "--debug-dir", root}, &stdout, &stderr, newStateTestHarness().options())
if code != 1 || !strings.Contains(stderr.String(), "config file") {
t.Fatalf("code=%d stderr=%q", code, stderr.String())
}
assertAbsent(t, root)
})
for _, failure := range []struct {
name string
expected string
setup func(*testing.T, stateTestRoots, *stateTestHarness) Options
}{
{"resolution", "pipeline \"missing\"", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options { return h.options() }},
{"pipeline", "synthetic extraction failure", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options {
h.extractErr = errors.New("synthetic extraction failure")
return h.options()
}},
{"output", "create output parent", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options {
if err := os.WriteFile(roots.output, []byte("not a directory"), 0o600); err != nil {
t.Fatal(err)
}
return h.options()
}},
{"summary", "write debug invocation metadata", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options {
opts := h.options()
opts.DebugRecorderFactory = func(traceRoot string) (pipeline.DebugRecorder, error) {
if err := os.RemoveAll(filepath.Join(filepath.Dir(traceRoot), "summary")); err != nil {
return nil, err
}
if err := os.WriteFile(filepath.Join(filepath.Dir(traceRoot), "summary"), []byte("blocked"), 0o600); err != nil {
return nil, err
}
return frameworkdebug.NewFilesystemRecorder(traceRoot)
}
return opts
}},
{"trace", "trace unavailable", func(t *testing.T, roots stateTestRoots, h *stateTestHarness) Options {
opts := h.options()
opts.DebugRecorderFactory = func(string) (pipeline.DebugRecorder, error) { return failingDebugRecorder{}, nil }
return opts
}},
} {
t.Run(failure.name, func(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
opts := failure.setup(t, roots, harness)
failureStderr := ""
if failure.name == "resolution" {
var stdout, stderr bytes.Buffer
code := RunWithOptions([]string{"run", "missing", "--config", roots.config, "--input", roots.input, "--debug"}, &stdout, &stderr, opts)
if code != 1 {
t.Fatalf("code=%d stderr=%q", code, stderr.String())
}
failureStderr = stderr.String()
} else {
result := runStateTest(t, roots, opts, true, false, "bypass")
if result.code != 1 {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
failureStderr = result.stderr
}
if !strings.Contains(failureStderr, failure.expected) || !strings.Contains(failureStderr, "debug=") {
t.Fatalf("stderr=%q, want %q and debug path", failureStderr, failure.expected)
}
bundle := onlyChildDir(t, roots.debug)
if !strings.Contains(readAllFiles(t, bundle), "synthetic") && failure.name == "pipeline" {
t.Fatal("pipeline failure was not retained in debug bundle")
}
})
}
}
func TestRunDebugArtifactsRedactSecretsButRetainApplicationData(t *testing.T) {
roots := newStateTestRoots(t)
t.Setenv("STATE_TEST_UNRELATED_ENV", "HOST_ONLY_SENTINEL")
if err := os.WriteFile(filepath.Join(filepath.Dir(roots.input), "unrelated.txt"), []byte("HOST_ONLY_FILE_SENTINEL"), 0o600); err != nil {
t.Fatal(err)
}
harness := newStateTestHarness()
result := runStateTest(t, roots, harness.options(), true, false, "bypass")
if result.code != 0 {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
bundle := onlyChildDir(t, roots.debug)
summary := readAllFiles(t, filepath.Join(bundle, "summary"))
trace := readAllFiles(t, filepath.Join(bundle, "trace"))
for _, forbidden := range []string{"sk-secretvalue", "Bearer secretvalue", "HOST_ONLY_SENTINEL", "HOST_ONLY_FILE_SENTINEL"} {
if strings.Contains(summary, forbidden) || strings.Contains(trace, forbidden) {
t.Fatalf("debug bundle contains %q", forbidden)
}
}
if strings.Contains(summary, "application content") {
t.Fatal("summary contains raw application input")
}
if !strings.Contains(trace, "application content") {
t.Fatal("trace does not retain expected application input")
}
}
func TestRunRedactsSensitiveModuleOptionsFromConfigAndPipelineSummaries(t *testing.T) {
roots := newStateTestRoots(t)
data, err := os.ReadFile(roots.config)
if err != nil {
t.Fatal(err)
}
configText := replaceRequiredOnce(t, string(data), " input: test/input\n", ` input:
module: test/input
options:
api_key: CONFIG_SUMMARY_SECRET_SENTINEL
safe: SAFE_OPTION_SENTINEL
nested:
- - password: PIPELINE_SUMMARY_SECRET_SENTINEL
neighbor: SAFE_NESTED_OPTION_SENTINEL
`)
if err := os.WriteFile(roots.config, []byte(configText), 0o600); err != nil {
t.Fatal(err)
}
result := runStateTest(t, roots, newStateTestHarness().options(), true, false, "bypass")
if result.code != 0 {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
summaryRoot := filepath.Join(onlyChildDir(t, roots.debug), "summary")
for _, name := range []string{"effective-config.json", "resolved-pipeline.json"} {
contents, err := os.ReadFile(filepath.Join(summaryRoot, name))
if err != nil {
t.Fatal(err)
}
text := string(contents)
for _, secret := range []string{"CONFIG_SUMMARY_SECRET_SENTINEL", "PIPELINE_SUMMARY_SECRET_SENTINEL"} {
if strings.Contains(text, secret) {
t.Fatalf("%s contains %q: %s", name, secret, text)
}
}
for _, retained := range []string{"[REDACTED]", "SAFE_OPTION_SENTINEL", "SAFE_NESTED_OPTION_SENTINEL"} {
if !strings.Contains(text, retained) {
t.Fatalf("%s does not contain %q: %s", name, retained, text)
}
}
}
}
func TestRunUsesOneInjectedIdentityForDebugOutputAndManifest(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
opts := harness.options()
const runID = "run-1000000000-11111111111111111111111111111111"
opts.RunIDGenerator = func(time.Time) (string, error) { return runID, nil }
result := runStateTest(t, roots, opts, true, false, "bypass")
if result.code != 0 {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
outputPath := filepath.Join(roots.output, runID)
debugPath := filepath.Join(roots.debug, runID)
assertFile(t, filepath.Join(outputPath, "result.json"))
assertFile(t, filepath.Join(debugPath, "summary", "run-manifest.json"))
if !strings.Contains(result.stdout, "output="+outputPath) || !strings.Contains(result.stdout, "debug="+debugPath) {
t.Fatalf("stdout=%q, want shared run identity", result.stdout)
}
data, err := os.ReadFile(filepath.Join(debugPath, "summary", "run-manifest.json"))
if err != nil {
t.Fatal(err)
}
var manifest artifacts.RunManifest
if err := json.Unmarshal(data, &manifest); err != nil {
t.Fatal(err)
}
if manifest.RunID != runID {
t.Fatalf("manifest run ID = %q, want %q", manifest.RunID, runID)
}
wantStartedAt := time.Unix(1, 0).UTC()
if manifest.StartedAt == nil || !manifest.StartedAt.Equal(wantStartedAt) {
t.Fatalf("manifest started at = %v, want %v", manifest.StartedAt, wantStartedAt)
}
var invocation debugbundle.Invocation
readStateTestSummaryJSON(t, debugPath, "invocation.json", &invocation)
if invocation.RunID != runID || !invocation.StartedAt.Equal(wantStartedAt) {
t.Fatalf("debug invocation identity = %#v, want run %q at %v", invocation, runID, wantStartedAt)
}
report := readStateTestRunReport(t, debugPath)
if !report.Succeeded || report.RunID != runID || report.PipelineID != "sample" || report.OutputPath != outputPath || report.DebugPath != debugPath || report.OutputCount != 1 || report.RejectedCount != 0 || report.WarningCount != 0 || report.ValidationStatus != "approved" {
t.Fatalf("success report = %#v", report)
}
if !strings.Contains(result.stdout, "outputs=1 rejected=0") {
t.Fatalf("stdout=%q, want report counts", result.stdout)
}
}
func TestRunWritesTerminalArtifactsForResolutionPipelineAndOutputFailures(t *testing.T) {
for _, tc := range []struct {
name string
pipelineID string
wantError string
wantOutputs int
wantValidation string
configureFailure func(*testing.T, stateTestRoots, *stateTestHarness)
}{
{name: "resolution", pipelineID: "missing", wantError: `pipeline "missing"`},
{name: "pipeline", pipelineID: "sample", wantError: "synthetic extraction failure", wantValidation: "failed", configureFailure: func(_ *testing.T, _ stateTestRoots, h *stateTestHarness) {
h.extractErr = errors.New("synthetic extraction failure")
}},
{name: "output", pipelineID: "sample", wantError: "create output parent", wantOutputs: 1, wantValidation: "approved", configureFailure: func(t *testing.T, roots stateTestRoots, _ *stateTestHarness) {
if err := os.WriteFile(roots.output, []byte("not a directory"), 0o600); err != nil {
t.Fatal(err)
}
}},
} {
t.Run(tc.name, func(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
if tc.configureFailure != nil {
tc.configureFailure(t, roots, harness)
}
opts := harness.options()
var stdout, stderr bytes.Buffer
args := []string{"run", tc.pipelineID, "--config", roots.config, "--input", roots.input, "--chunk_cache", "bypass", "--debug"}
code := RunWithOptions(args, &stdout, &stderr, opts)
if code != 1 || !strings.Contains(stderr.String(), tc.wantError) {
t.Fatalf("code=%d stderr=%q", code, stderr.String())
}
bundlePath := onlyChildDir(t, roots.debug)
runID := filepath.Base(bundlePath)
report := readStateTestRunReport(t, bundlePath)
if report.Succeeded || report.RunID != runID || report.PipelineID != tc.pipelineID || report.OutputPath != filepath.Join(roots.output, runID) || report.DebugPath != bundlePath || report.OutputCount != tc.wantOutputs || report.RejectedCount != 0 || report.WarningCount != 0 || report.ValidationStatus != tc.wantValidation {
t.Fatalf("failure report = %#v", report)
}
errorLog, err := os.ReadFile(filepath.Join(bundlePath, "summary", "error.log"))
if err != nil || !strings.Contains(string(errorLog), tc.wantError) {
t.Fatalf("error log = %q, %v", errorLog, err)
}
})
}
}
func TestRunRetainsPartialPipelineOutcomeInFailureSummary(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
harness.chunkWarnings = []contracts.Warning{{Scope: "chunk", ReasonCode: "partial-warning", Message: "warning retained before failure"}}
harness.extractErr = errors.New("synthetic partial pipeline failure")
result := runStateTest(t, roots, harness.options(), true, true, "bypass")
if result.code != 1 {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
bundlePath := onlyChildDir(t, roots.debug)
report := readStateTestRunReport(t, bundlePath)
if report.Succeeded || report.OutputCount != 0 || report.RejectedCount != 0 || report.WarningCount != 1 || report.ValidationStatus != "failed" {
t.Fatalf("partial failure report = %#v", report)
}
var manifest artifacts.RunManifest
readStateTestSummaryJSON(t, bundlePath, "run-manifest.json", &manifest)
if manifest.RunID != report.RunID || manifest.PipelineID != "sample" || manifest.ValidationStatus != "failed" {
t.Fatalf("partial manifest = %#v", manifest)
}
var warnings []contracts.Warning
readStateTestSummaryJSON(t, bundlePath, "warnings.json", &warnings)
if len(warnings) != 1 || warnings[0].ReasonCode != "partial-warning" {
t.Fatalf("partial warnings = %#v", warnings)
}
var events []pipeline.CheckpointEvent
readStateTestSummaryJSON(t, bundlePath, "checkpoint-events.json", &events)
if len(events) == 0 || events[0].Stage != "source" {
t.Fatalf("partial checkpoint events = %#v, want retained source decision", events)
}
var chunkPlan artifacts.ChunkPlanSummary
readStateTestSummaryJSON(t, bundlePath, "chunk-plan.json", &chunkPlan)
if chunkPlan.Mode != "bypass" || chunkPlan.ValidationStatus == "not_run" {
t.Fatalf("partial chunk plan = %#v", chunkPlan)
}
}
func TestRunTerminalPersistenceFailuresDoNotRecurseOrHidePrimaryError(t *testing.T) {
for _, tc := range []struct {
name string
reportErr error
errorLogErr error
wantSecondary string
}{
{name: "run report", reportErr: errors.New("injected run report failure"), wantSecondary: "injected run report failure"},
{name: "error log", errorLogErr: errors.New("injected error log failure"), wantSecondary: "injected error log failure"},
} {
t.Run(tc.name, func(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
harness.extractErr = errors.New("primary pipeline failure")
opts := harness.options()
var terminal *recordingTerminalWriter
opts.DebugTerminalFactory = func(delegate *debugbundle.SummaryWriter) DebugTerminalWriter {
terminal = &recordingTerminalWriter{delegate: delegate, reportErr: tc.reportErr, errorLogErr: tc.errorLogErr}
return terminal
}
result := runStateTest(t, roots, opts, true, false, "bypass")
if result.code != 1 {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
if terminal == nil {
t.Fatal("terminal writer was not constructed")
}
if terminal.reportCalls != 1 || terminal.errorLogCalls != 1 {
t.Fatalf("terminal calls = report:%d error:%d", terminal.reportCalls, terminal.errorLogCalls)
}
primaryIndex := strings.Index(result.stderr, "primary pipeline failure")
secondaryIndex := strings.Index(result.stderr, tc.wantSecondary)
debugIndex := strings.Index(result.stderr, "debug=")
if primaryIndex < 0 || secondaryIndex <= primaryIndex || debugIndex <= secondaryIndex {
t.Fatalf("stderr order = %q", result.stderr)
}
})
}
}
func TestRunReportFailureOnSuccessIsTerminalizedWithoutRetry(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
var terminal *recordingTerminalWriter
opts.DebugTerminalFactory = func(delegate *debugbundle.SummaryWriter) DebugTerminalWriter {
terminal = &recordingTerminalWriter{delegate: delegate, reportErr: errors.New("injected success report failure")}
return terminal
}
result := runStateTest(t, roots, opts, true, false, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "write debug run report") || !strings.Contains(result.stderr, "injected success report failure") {
t.Fatalf("code=%d stdout=%q stderr=%q", result.code, result.stdout, result.stderr)
}
if terminal == nil {
t.Fatal("terminal writer was not constructed")
}
if terminal.reportCalls != 1 || terminal.errorLogCalls != 1 {
t.Fatalf("terminal calls = report:%d error:%d", terminal.reportCalls, terminal.errorLogCalls)
}
if result.stdout != "" {
t.Fatalf("stdout=%q, want no success message", result.stdout)
}
bundlePath := onlyChildDir(t, roots.debug)
errorLog, err := os.ReadFile(filepath.Join(bundlePath, "summary", "error.log"))
if err != nil || !strings.Contains(string(errorLog), "injected success report failure") {
t.Fatalf("error log = %q, %v", errorLog, err)
}
}
func TestRunWithoutDebugDoesNotUseTerminalSummaryWriter(t *testing.T) {
roots := newStateTestRoots(t)
harness := newStateTestHarness()
harness.extractErr = errors.New("non-debug pipeline failure")
opts := harness.options()
factoryCalls := 0
opts.DebugTerminalFactory = func(delegate *debugbundle.SummaryWriter) DebugTerminalWriter {
factoryCalls++
return delegate
}
result := runStateTest(t, roots, opts, false, false, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "non-debug pipeline failure") {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
if factoryCalls != 0 {
t.Fatalf("terminal summary factory calls = %d, want 0", factoryCalls)
}
assertAbsent(t, roots.debug)
}
func TestRunRefusesExistingOutputDirectoryWithoutChangingIt(t *testing.T) {
roots := newStateTestRoots(t)
const runID = "run-1000000000-22222222222222222222222222222222"
runPath := filepath.Join(roots.output, runID)
if err := os.MkdirAll(filepath.Join(runPath, "nested"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(runPath, "sentinel"), []byte("existing output"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(runPath, "nested", "data"), []byte("preserve me"), 0o644); err != nil {
t.Fatal(err)
}
before := readTree(t, runPath)
opts := newStateTestHarness().options()
opts.RunIDGenerator = func(time.Time) (string, error) { return runID, nil }
result := runStateTest(t, roots, opts, true, false, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "output run directory") || !strings.Contains(result.stderr, "already exists") {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
if after := readTree(t, runPath); !sameFiles(after, before) {
t.Fatalf("existing output changed: before=%v after=%v", before, after)
}
bundlePath := filepath.Join(roots.debug, runID)
report := readStateTestRunReport(t, bundlePath)
if report.Succeeded || report.RunID != runID || report.OutputPath != runPath || report.DebugPath != bundlePath || report.OutputCount != 1 || report.ValidationStatus != "approved" {
t.Fatalf("output collision report = %#v", report)
}
errorLog, err := os.ReadFile(filepath.Join(bundlePath, "summary", "error.log"))
if err != nil || !strings.Contains(string(errorLog), "already exists") {
t.Fatalf("output collision error log = %q, %v", errorLog, err)
}
}
func TestRepeatedRunIdentityCannotOverwriteFirstOutput(t *testing.T) {
roots := newStateTestRoots(t)
const runID = "run-1000000000-33333333333333333333333333333333"
harness := newStateTestHarness()
opts := harness.options()
opts.RunIDGenerator = func(time.Time) (string, error) { return runID, nil }
first := runStateTest(t, roots, opts, false, false, "bypass")
if first.code != 0 {
t.Fatalf("first code=%d stderr=%q", first.code, first.stderr)
}
runPath := filepath.Join(roots.output, runID)
before := readTree(t, runPath)
second := runStateTest(t, roots, opts, false, false, "bypass")
if second.code != 1 || !strings.Contains(second.stderr, "already exists") {
t.Fatalf("second code=%d stderr=%q", second.code, second.stderr)
}
if after := readTree(t, runPath); !sameFiles(after, before) {
t.Fatalf("first output changed: before=%v after=%v", before, after)
}
}
func TestRunRefusesExistingDebugBundleWithoutChangingIt(t *testing.T) {
roots := newStateTestRoots(t)
const runID = "run-1000000000-44444444444444444444444444444444"
bundlePath := filepath.Join(roots.debug, runID)
if err := os.MkdirAll(bundlePath, 0o700); err != nil {
t.Fatal(err)
}
sentinelPath := filepath.Join(bundlePath, "sentinel")
if err := os.WriteFile(sentinelPath, []byte("existing debug"), 0o600); err != nil {
t.Fatal(err)
}
opts := newStateTestHarness().options()
opts.RunIDGenerator = func(time.Time) (string, error) { return runID, nil }
result := runStateTest(t, roots, opts, true, false, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "debug bundle") || !strings.Contains(result.stderr, "already exists") {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
if got, err := os.ReadFile(sentinelPath); err != nil || string(got) != "existing debug" {
t.Fatalf("sentinel = %q, %v", got, err)
}
assertAbsent(t, roots.output)
}
func TestRunIDGenerationFailurePrecedesDebugAllocation(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
opts.RunIDGenerator = func(time.Time) (string, error) { return "", errors.New("random source unavailable") }
result := runStateTest(t, roots, opts, true, false, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "generate run ID: random source unavailable") {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
assertAbsent(t, roots.debug)
assertAbsent(t, roots.output)
}
func TestRunRejectsUnsafeGeneratedIdentityBeforePathUse(t *testing.T) {
roots := newStateTestRoots(t)
opts := newStateTestHarness().options()
opts.RunIDGenerator = func(time.Time) (string, error) { return "../outside", nil }
result := runStateTest(t, roots, opts, true, false, "bypass")
if result.code != 1 || !strings.Contains(result.stderr, "invalid generated run ID") || !strings.Contains(result.stderr, "one safe path component") {
t.Fatalf("code=%d stderr=%q", result.code, result.stderr)
}
assertAbsent(t, roots.debug)
assertAbsent(t, roots.output)
}
type stateTestRoots struct{ config, input, output, plans, checkpoints, debug string }
func newStateTestRoots(t *testing.T) stateTestRoots {
t.Helper()
base := t.TempDir()
roots := stateTestRoots{input: filepath.Join(base, "input.txt"), output: filepath.Join(base, "output"), plans: filepath.Join(base, "plans"), checkpoints: filepath.Join(base, "checkpoints"), debug: filepath.Join(base, "debug")}
if err := os.WriteFile(roots.input, []byte("application content Bearer secretvalue sk-secretvalue"), 0o600); err != nil {
t.Fatal(err)
}
roots.config = filepath.Join(base, "config.yml")
config := fmt.Sprintf("version: 4\noutput:\n directory: %q\ncache:\n chunk_plans:\n directory: %q\n mode: auto\n checkpoints:\n enabled: true\n directory: %q\ndebug:\n directory: %q\npipelines:\n sample:\n input: test/input\n chunk: test/chunk\n artifacts:\n items:\n extract: test/extract\n merge: test/merge\n normalize: test/normalize\n output: test/output\n", roots.output, roots.plans, roots.checkpoints, roots.debug)
if err := os.WriteFile(roots.config, []byte(config), 0o600); err != nil {
t.Fatal(err)
}
return roots
}
type stateTestResult struct {
code int
stdout, stderr string
}
func runStateTest(t *testing.T, roots stateTestRoots, opts Options, debug, resume bool, mode string) stateTestResult {
t.Helper()
args := []string{"run", "sample", "--config", roots.config, "--input", roots.input, "--chunk_cache", mode}
if debug {
args = append(args, "--debug")
}
if resume {
args = append(args, "--resume")
}
var stdout, stderr bytes.Buffer
return stateTestResult{RunWithOptions(args, &stdout, &stderr, opts), stdout.String(), stderr.String()}
}
func assertStateTestOutput(t *testing.T, root string) {
t.Helper()
output := onlyChildDir(t, root)
data, err := os.ReadFile(filepath.Join(output, "result.json"))
if err != nil || string(data) != "{\"ok\":true}\n" {
t.Fatalf("output = %q, %v", data, err)
}
}
func onlyChildDir(t *testing.T, root string) string {
t.Helper()
entries, err := os.ReadDir(root)
if err != nil {
t.Fatal(err)
}
var dirs []string
for _, entry := range entries {
if entry.IsDir() {
dirs = append(dirs, filepath.Join(root, entry.Name()))
}
}
if len(dirs) != 1 {
t.Fatalf("directories in %q = %v, want one", root, dirs)
}
return dirs[0]
}
func assertFile(t *testing.T, path string) {
t.Helper()
if info, err := os.Stat(path); err != nil || info.IsDir() {
t.Fatalf("file %q: %v", path, err)
}
}
func assertAbsent(t *testing.T, path string) {
t.Helper()
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Fatalf("%q exists or stat failed: %v", path, err)
}
}
func assertAnyFile(t *testing.T, root string) {
t.Helper()
if text := readAllFiles(t, root); text == "" {
t.Fatalf("no files under %q", root)
}
}
func readAllFiles(t *testing.T, root string) string {
t.Helper()
var content strings.Builder
if err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error {
if err != nil {
return err
}
if entry.IsDir() {
return nil
}
data, err := os.ReadFile(path)
if err != nil {
return err
}
content.Write(data)
return nil
}); err != nil {
t.Fatal(err)
}
return content.String()
}
func readStateTestRunReport(t *testing.T, bundlePath string) debugbundle.RunReport {
t.Helper()
var report debugbundle.RunReport
readStateTestSummaryJSON(t, bundlePath, "run-report.json", &report)
return report
}
func readStateTestSummaryJSON(t *testing.T, bundlePath, name string, target any) {
t.Helper()
data, err := os.ReadFile(filepath.Join(bundlePath, "summary", name))
if err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(data, target); err != nil {
t.Fatal(err)
}
}
func assertRestrictedTree(t *testing.T, root string) {
t.Helper()
if runtime.GOOS == "windows" {
return
}
if err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
want := os.FileMode(0o600)
if info.IsDir() {
want = 0o700
}
if info.Mode().Perm() != want {
return fmt.Errorf("%s has mode %o, want %o", path, info.Mode().Perm(), want)
}
return nil
}); err != nil {
t.Fatal(err)
}
}
func readTree(t *testing.T, root string) map[string][]byte {
t.Helper()
files := map[string][]byte{}
if err := filepath.WalkDir(root, func(path string, entry os.DirEntry, err error) error {
if err != nil {
return err
}
if entry.IsDir() {
return nil
}
data, err := os.ReadFile(path)
if err != nil {
return err
}
relative, err := filepath.Rel(root, path)
if err != nil {
return err
}
files[relative] = data
return nil
}); err != nil {
t.Fatal(err)
}
return files
}
func sameFiles(left, right map[string][]byte) bool {
if len(left) != len(right) {
return false
}
for path, data := range left {
if !bytes.Equal(data, right[path]) {
return false
}
}
return true
}
type stateTestHarness struct {
mu sync.Mutex
chunkCalls, extractCalls int
runIDCalls uint64
extractErr error
chunkWarnings []contracts.Warning
moduleProfiles []string
sessionIDs []string
outputWarnings []contracts.Warning
includeWarnings bool
}
func newStateTestHarness() *stateTestHarness { return &stateTestHarness{} }
func (h *stateTestHarness) options() Options {
registries := pipeline.Registries{Inputs: pipeline.NewInputAdapterRegistry(), Chunkers: pipeline.NewChunkerRegistry(), ArtifactCodecs: pipeline.NewArtifactCodecRegistry(), Extractors: pipeline.NewExtractorRegistry(), Mergers: pipeline.NewMergerRegistry(), Normalizers: pipeline.NewNormalizerRegistry(), Validators: pipeline.NewValidatorRegistry(), ValidatorChains: pipeline.NewValidatorChainRegistry(), Outputs: pipeline.NewOutputEncoderRegistry()}
if err := pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, stateTestCodec{}); err != nil {
panic(err)
}
if err := registries.Inputs.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.InputAdapter, error) { return stateTestInput{}, nil }); err != nil {
panic(err)
}
if err := registries.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "test/chunk", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"source"}, Provides: []string{"chunks"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "cache-reference"}}}, func(map[string]any) error { return nil }, func(pipeline.BuildRequest) (contracts.Chunker, error) { return stateTestChunker{h}, nil }); err != nil {
panic(err)
}
if err := pipeline.RegisterExtractor(registries.Extractors, pipeline.ModuleSpec{Key: "test/extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks"}, Provides: []string{"artifact"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Extractor[stateTestArtifact], error) { return stateTestExtractor{h}, nil }); err != nil {
panic(err)
}
if err := pipeline.RegisterMerger(registries.Mergers, pipeline.ModuleSpec{Key: "test/merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Merger[stateTestArtifact], error) { return stateTestMerger{harness: h}, nil }); err != nil {
panic(err)
}
if err := pipeline.RegisterNormalizer(registries.Normalizers, pipeline.ModuleSpec{Key: "test/normalize", Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: stateTestArtifactKind}, func() (contracts.Normalizer[stateTestArtifact], error) { return stateTestNormalizer{harness: h}, nil }); err != nil {
panic(err)
}
if err := registries.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "test/output", Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}, Provides: []string{"output"}}, func() (contracts.OutputEncoder, error) {
return stateTestOutput{harness: h, includeWarnings: h.includeWarnings}, nil
}); err != nil {
panic(err)
}
return Options{Catalog: catalogFromRegistries(registries), Registries: registries, LookupEnv: emptyLookup, Now: func() time.Time { return time.Unix(1, 0) }, RunIDGenerator: func(startedAt time.Time) (string, error) {
h.mu.Lock()
defer h.mu.Unlock()
h.runIDCalls++
return fmt.Sprintf("run-%d-%032x", startedAt.UnixNano(), h.runIDCalls), nil
}, UserCacheDir: func() (string, error) { return "", errors.New("unexpected user cache lookup") }, LLMClientFactory: func(context.Context, config.Config, string, LLMRuntimeOverrides) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return nil, nil, nil
}}
}
type stateTestInput struct{}
func (stateTestInput) Key() string { return "test/input" }
func (stateTestInput) Parse(_ context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
doc := &source.SourceDocument{ID: "source", Kind: "text", Format: "text/plain", Units: []source.SourceUnit{{ID: 1, Kind: "text", Text: string(req.Raw), Ref: source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}}}}
digest, err := source.DigestDocument(doc)
if err != nil {
return nil, err
}
doc.Digest = digest
return doc, nil
}
type stateTestChunker struct{ harness *stateTestHarness }
func (stateTestChunker) Key() string { return "test/chunk" }
func (stateTestChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (c stateTestChunker) Plan(_ context.Context, req contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
c.harness.mu.Lock()
c.harness.moduleProfiles = append(c.harness.moduleProfiles, req.LLMProfile)
c.harness.sessionIDs = append(c.harness.sessionIDs, req.SessionID)
c.harness.mu.Unlock()
c.harness.mu.Lock()
c.harness.chunkCalls++
c.harness.mu.Unlock()
return contracts.ChunkPlanResult{Plan: source.ChunkPlan{SourceDigest: req.Source.Digest, Ranges: []source.ChunkRange{{StartUnitID: 1, EndUnitID: 1}}}, Warnings: append([]contracts.Warning(nil), c.harness.chunkWarnings...)}, nil
}
const stateTestArtifactKind contracts.ArtifactKind = "test/artifact"
type stateTestArtifact struct {
Value string `json:"value"`
}
type stateTestCodec struct{}
func (stateTestCodec) Kind() contracts.ArtifactKind { return stateTestArtifactKind }
func (stateTestCodec) Schema() contracts.ArtifactSchema {
return contracts.ArtifactSchema{ID: "test.artifact", Name: "test_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
}
func (stateTestCodec) MediaType() string { return "application/json" }
func (stateTestCodec) EncodeCandidate(v stateTestArtifact) ([]byte, error) {
return []byte(`{"value":"ok"}`), nil
}
func (stateTestCodec) Encode(v stateTestArtifact) ([]byte, error) {
return []byte(`{"value":"ok"}`), nil
}
func (stateTestCodec) Decode([]byte) (stateTestArtifact, error) {
return stateTestArtifact{Value: "ok"}, nil
}
type stateTestExtractor struct{ harness *stateTestHarness }
func (stateTestExtractor) Key() string { return "test/extract" }
func (stateTestExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (e stateTestExtractor) Extract(_ context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[stateTestArtifact], error) {
e.harness.mu.Lock()
defer e.harness.mu.Unlock()
e.harness.extractCalls++
e.harness.moduleProfiles = append(e.harness.moduleProfiles, req.LLMProfile)
e.harness.sessionIDs = append(e.harness.sessionIDs, req.SessionID)
if e.harness.extractErr != nil {
return contracts.TypedExtractionResult[stateTestArtifact]{}, e.harness.extractErr
}
return contracts.TypedExtractionResult[stateTestArtifact]{Value: stateTestArtifact{Value: "ok"}}, nil
}
type stateTestMerger struct{ harness *stateTestHarness }
func (stateTestMerger) Key() string { return "test/merge" }
func (m stateTestMerger) Merge(_ context.Context, req contracts.TypedMergeRequest[stateTestArtifact]) (contracts.TypedMergeResult[stateTestArtifact], error) {
m.harness.mu.Lock()
m.harness.moduleProfiles = append(m.harness.moduleProfiles, req.LLMProfile)
m.harness.sessionIDs = append(m.harness.sessionIDs, req.SessionID)
m.harness.mu.Unlock()
return contracts.TypedMergeResult[stateTestArtifact]{Value: req.ExtractOutputs[0].Value}, nil
}
type stateTestNormalizer struct{ harness *stateTestHarness }
func (stateTestNormalizer) Key() string { return "test/normalize" }
func (stateTestNormalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (n stateTestNormalizer) Normalize(_ context.Context, req contracts.TypedNormalizeRequest[stateTestArtifact]) (contracts.TypedNormalizeResult[stateTestArtifact], error) {
n.harness.mu.Lock()
n.harness.moduleProfiles = append(n.harness.moduleProfiles, req.LLMProfile)
n.harness.sessionIDs = append(n.harness.sessionIDs, req.SessionID)
n.harness.mu.Unlock()
return contracts.TypedNormalizeResult[stateTestArtifact]{Value: req.MergeOutput.Value}, nil
}
type stateTestOutput struct {
harness *stateTestHarness
includeWarnings bool
}
func (o stateTestOutput) Key() string { return "test/output" }
func (o stateTestOutput) Encode(_ context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
o.harness.mu.Lock()
o.harness.outputWarnings = append([]contracts.Warning(nil), req.Warnings...)
o.harness.mu.Unlock()
data := []byte("{\"ok\":true}\n")
if o.includeWarnings && len(req.Warnings) > 0 {
data = []byte(fmt.Sprintf("{\"ok\":true,\"warnings\":%q}\n", req.Warnings[0].ReasonCode))
}
return contracts.OutputResult{Files: []contracts.OutputFile{{Name: "result.json", Bytes: data}}}, nil
}
type failingDebugRecorder struct{}
func (failingDebugRecorder) Enabled() bool { return true }
func (failingDebugRecorder) WriteJSON(string, any) error { return errors.New("trace unavailable") }
func (failingDebugRecorder) WriteBytes(string, []byte) error { return errors.New("trace unavailable") }
type recordingTerminalWriter struct {
delegate DebugTerminalWriter
reportErr, errorLogErr error
reportCalls, errorLogCalls int
}
func (w *recordingTerminalWriter) WriteRunReport(report debugbundle.RunReport) error {
w.reportCalls++
if w.reportErr != nil {
return w.reportErr
}
return w.delegate.WriteRunReport(report)
}
func (w *recordingTerminalWriter) WriteError(message string) error {
w.errorLogCalls++
if w.errorLogErr != nil {
return w.errorLogErr
}
return w.delegate.WriteError(message)
}

6
internal/cli/testdata/invalid-seriatim-empty-segments.json vendored
View File

@@ -1,6 +0,0 @@
{
"metadata": {
"id": "session-alpha"
},
"segments": []
}

213
internal/core/artifacts/artifacts.go
View File

@@ -1,173 +1,68 @@
package artifacts
import (
"strings"
"encoding/json"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
)
type ArtifactLaneManifest struct {
StepID string `json:"step_id,omitempty"`
ID string `json:"id"`
Extractor string `json:"extractor"`
Merger string `json:"merger"`
Normalizer string `json:"normalizer"`
Metadata map[string]any `json:"metadata,omitempty"`
type Candidate struct {
Index int `json:"index"`
ExtractorKey string `json:"extractor_key"`
ArtifactType string `json:"artifact_type"`
SchemaVersion string `json:"schema_version"`
Payload json.RawMessage `json:"payload"`
SourceRefs []source.SourceRef `json:"source_refs,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type ValidatorChainManifest struct {
Stage string `json:"stage"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key"`
Validators []ValidatorManifest `json:"validators"`
type Artifact struct {
ExtractorKey string `json:"extractor_key"`
ArtifactType string `json:"artifact_type"`
SchemaVersion string `json:"schema_version"`
Payload json.RawMessage `json:"payload"`
SourceRefs []source.SourceRef `json:"source_refs,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type ValidatorManifest struct {
Key string `json:"key"`
ExecutionClass string `json:"execution_class"`
}
type LLMProfileManifest struct {
ID string `json:"id"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
BackendID string `json:"backend_id,omitempty"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
}
// Normalized returns the canonical representation used for manifest identity
// and publication.
func (profile LLMProfileManifest) Normalized() LLMProfileManifest {
profile.ID = strings.TrimSpace(profile.ID)
profile.Provider = strings.TrimSpace(profile.Provider)
profile.Model = strings.TrimSpace(profile.Model)
profile.BackendID = strings.TrimSpace(profile.BackendID)
profile.ReasoningEffort = strings.TrimSpace(profile.ReasoningEffort)
return profile
}
// IdentityKey returns an opaque, deterministic key for the effective profile.
func (profile LLMProfileManifest) IdentityKey() string {
profile = profile.Normalized()
return profile.ID + "\x00" +
profile.Provider + "\x00" +
profile.Model + "\x00" +
profile.BackendID + "\x00" +
profile.ReasoningEffort
}
type ReferenceProvenance struct {
Stage string `json:"stage,omitempty"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
SlotName string `json:"slot_name"`
OriginType string `json:"origin_type"`
OriginURI string `json:"origin_uri,omitempty"`
Digest string `json:"digest,omitempty"`
MediaType string `json:"media_type,omitempty"`
SizeBytes int64 `json:"size_bytes,omitempty"`
BindingSource string `json:"binding_source,omitempty"`
ArtifactKind string `json:"artifact_kind,omitempty"`
SchemaID string `json:"schema_id,omitempty"`
SchemaName string `json:"schema_name,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"`
SchemaDigest string `json:"schema_digest,omitempty"`
ProducerPipeline string `json:"producer_pipeline_id,omitempty"`
ProducerStep string `json:"producer_step_id,omitempty"`
ProducerLane string `json:"producer_lane_id,omitempty"`
ProducerModule string `json:"producer_module_key,omitempty"`
}
type OutputSchemaProvenance struct {
ID string `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Version string `json:"version,omitempty"`
}
type NormalizedOutputManifest struct {
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id"`
ModuleKey string `json:"module_key,omitempty"`
SourceID string `json:"source_id,omitempty"`
MediaType string `json:"media_type,omitempty"`
Schema OutputSchemaProvenance `json:"schema,omitempty"`
}
type RejectedOutputManifest struct {
Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
ValidatorName string `json:"validator_name,omitempty"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message,omitempty"`
AttemptCount int `json:"attempt_count,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
}
type CheckpointDecisionManifest struct {
Stage string `json:"stage"`
StepID string `json:"step_id,omitempty"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
Category string `json:"category"`
ReasonCode string `json:"reason_code,omitempty"`
Detail string `json:"detail,omitempty"`
}
type ChunkPlanManifest struct {
Mode string `json:"mode"`
Action string `json:"action,omitempty"`
SourceDigest string `json:"source_digest,omitempty"`
PlanDigest string `json:"plan_digest,omitempty"`
PlanSchemaVersion string `json:"plan_schema_version,omitempty"`
RequestedModule string `json:"requested_module"`
ProducerInputModule string `json:"producer_input_module,omitempty"`
ProducerModule string `json:"producer_module,omitempty"`
ProducerLLMProfile string `json:"producer_llm_profile,omitempty"`
ProducerReferences []ReferenceProvenance `json:"producer_references,omitempty"`
ProducerMetadata map[string]any `json:"producer_metadata,omitempty"`
CreatedAt *time.Time `json:"created_at,omitempty"`
}
// ChunkPlanSummary is deliberately limited to cache and validation decisions.
// It must never contain plan units, source content, annotations, or model I/O.
type ChunkPlanSummary struct {
Mode string `json:"mode"`
SourceDigest string `json:"source_digest,omitempty"`
CandidateDigest string `json:"candidate_digest,omitempty"`
RequestedModule string `json:"requested_module"`
LookupStatus string `json:"lookup_status"`
LookupReason string `json:"lookup_reason,omitempty"`
Action string `json:"action,omitempty"`
ValidationStatus string `json:"validation_status"`
PublicationStatus string `json:"publication_status"`
type RejectedArtifact struct {
Candidate Candidate `json:"candidate"`
ValidatorName string `json:"validator_name"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
}
type RunManifest struct {
RunID string `json:"run_id,omitempty"`
PipelineID string `json:"pipeline_id,omitempty"`
PipelineDigest string `json:"pipeline_digest,omitempty"`
InputModule string `json:"input_module,omitempty"`
Chunker string `json:"chunker,omitempty"`
ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"`
SourceDigests []string `json:"source_digests,omitempty"`
Extractors []string `json:"extractors,omitempty"`
Merger string `json:"merger,omitempty"`
Normalizer string `json:"normalizer,omitempty"`
OutputEncoder string `json:"output_encoder,omitempty"`
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
ValidatorChains []ValidatorChainManifest `json:"validator_chains,omitempty"`
References []ReferenceProvenance `json:"references,omitempty"`
NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"`
RejectedOutputs []RejectedOutputManifest `json:"rejected_outputs,omitempty"`
CheckpointDecisions []CheckpointDecisionManifest `json:"checkpoint_decisions,omitempty"`
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"`
ValidationStatus string `json:"validation_status,omitempty"`
StartedAt *time.Time `json:"started_at,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
RunID string `json:"run_id,omitempty"`
InputAdapter string `json:"input_adapter,omitempty"`
SourceDigests []string `json:"source_digests,omitempty"`
Extractors []string `json:"extractors,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"`
ValidationStatus string `json:"validation_status,omitempty"`
StartedAt *time.Time `json:"started_at,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
}
func ArtifactFromCandidate(candidate Candidate) Artifact {
return Artifact{
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: copyMetadata(candidate.Metadata),
}
}
func copyMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
copied := make(map[string]any, len(metadata))
for key, value := range metadata {
copied[key] = value
}
return copied
}

307
internal/core/artifacts/artifacts_test.go
View File

@@ -2,10 +2,116 @@ package artifacts
import (
"encoding/json"
"strings"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
)
func TestArtifactFromCandidatePreservesCandidateFields(t *testing.T) {
candidate := Candidate{
Index: 7,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"example"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u2"},
},
Metadata: map[string]any{
"confidence": 0.75,
},
}
artifact := ArtifactFromCandidate(candidate)
if artifact.ExtractorKey != candidate.ExtractorKey {
t.Fatalf("ExtractorKey = %q, want %q", artifact.ExtractorKey, candidate.ExtractorKey)
}
if artifact.ArtifactType != candidate.ArtifactType {
t.Fatalf("ArtifactType = %q, want %q", artifact.ArtifactType, candidate.ArtifactType)
}
if artifact.SchemaVersion != candidate.SchemaVersion {
t.Fatalf("SchemaVersion = %q, want %q", artifact.SchemaVersion, candidate.SchemaVersion)
}
if string(artifact.Payload) != string(candidate.Payload) {
t.Fatalf("Payload = %s, want %s", artifact.Payload, candidate.Payload)
}
if !reflect.DeepEqual(artifact.SourceRefs, candidate.SourceRefs) {
t.Fatalf("SourceRefs = %#v, want %#v", artifact.SourceRefs, candidate.SourceRefs)
}
if !reflect.DeepEqual(artifact.Metadata, candidate.Metadata) {
t.Fatalf("Metadata = %#v, want %#v", artifact.Metadata, candidate.Metadata)
}
candidate.Payload[0] = '['
candidate.SourceRefs[0].StartUnitID = "changed"
candidate.Metadata["confidence"] = 0.5
if string(artifact.Payload) != `{"name":"example"}` {
t.Fatalf("Payload changed after candidate mutation: %s", artifact.Payload)
}
if artifact.SourceRefs[0].StartUnitID != "u1" {
t.Fatalf("SourceRefs changed after candidate mutation: %#v", artifact.SourceRefs)
}
if artifact.Metadata["confidence"] != 0.75 {
t.Fatalf("Metadata changed after candidate mutation: %#v", artifact.Metadata)
}
}
func TestJSONMarshalUsesExpectedFieldNames(t *testing.T) {
candidate := Candidate{
Index: 1,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"value":true}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"reviewed": true,
},
}
rejected := RejectedArtifact{
Candidate: candidate,
ValidatorName: "generic-validator",
ReasonCode: "invalid",
Message: "candidate was not accepted",
}
gotJSON, err := json.Marshal(rejected)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
var got map[string]any
if err := json.Unmarshal(gotJSON, &got); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
assertHasKeys(t, got, "candidate", "validator_name", "reason_code", "message")
gotCandidate, ok := got["candidate"].(map[string]any)
if !ok {
t.Fatalf("candidate = %#v, want object", got["candidate"])
}
assertHasKeys(t, gotCandidate, "index", "extractor_key", "artifact_type", "schema_version", "payload", "source_refs", "metadata")
gotRefs, ok := gotCandidate["source_refs"].([]any)
if !ok {
t.Fatalf("source_refs = %#v, want array", gotCandidate["source_refs"])
}
if len(gotRefs) != 1 {
t.Fatalf("len(source_refs) = %d, want 1", len(gotRefs))
}
gotRef, ok := gotRefs[0].(map[string]any)
if !ok {
t.Fatalf("source_refs[0] = %#v, want object", gotRefs[0])
}
assertHasKeys(t, gotRef, "source_id", "start_unit_id", "end_unit_id")
}
func TestRunManifestOmitsEmptyOptionalFields(t *testing.T) {
gotJSON, err := json.Marshal(RunManifest{})
if err != nil {
@@ -17,205 +123,6 @@ func TestRunManifestOmitsEmptyOptionalFields(t *testing.T) {
}
}
func TestRunManifestChunkPlanIsAdditiveAndOmitsPlanContent(t *testing.T) {
manifest := RunManifest{ChunkPlan: &ChunkPlanManifest{
Mode: "auto", Action: "reused", SourceDigest: "sha256:source", PlanDigest: "sha256:plan",
PlanSchemaVersion: "notarius.chunk-plan.v2", RequestedModule: "chunk/current",
ProducerInputModule: "input/original", ProducerModule: "chunk/original",
}}
encoded, err := json.Marshal(manifest)
if err != nil {
t.Fatal(err)
}
text := string(encoded)
for _, want := range []string{`"chunk_plan"`, `"action":"reused"`, `"requested_module":"chunk/current"`, `"producer_module":"chunk/original"`} {
if !strings.Contains(text, want) {
t.Fatalf("manifest JSON %s does not contain %s", text, want)
}
}
for _, forbidden := range []string{`"plan"`, `"units"`, `"annotations"`} {
if strings.Contains(text, forbidden) {
t.Fatalf("manifest JSON contains forbidden field %s: %s", forbidden, text)
}
}
var legacy RunManifest
if err := json.Unmarshal([]byte(`{"pipeline_id":"legacy"}`), &legacy); err != nil {
t.Fatal(err)
}
if legacy.PipelineID != "legacy" || legacy.ChunkPlan != nil {
t.Fatalf("legacy manifest = %#v", legacy)
}
}
func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
manifest := RunManifest{
PipelineID: "pipeline-1",
PipelineDigest: "sha256:abc123",
LLMProfiles: []LLMProfileManifest{
{
ID: "default",
Provider: "promptkit",
Model: "model-a",
BackendID: "openrouter",
ReasoningEffort: "high",
},
},
ArtifactLanes: []ArtifactLaneManifest{
{
ID: "events",
Extractor: "event-extractor",
Merger: "appendorder",
Normalizer: "noop",
Metadata: map[string]any{
"extractor": map[string]any{"prompt_id": "test.prompt"},
},
},
},
ValidatorChains: []ValidatorChainManifest{
{
Stage: "extract",
LaneID: "events",
ModuleKey: "event-extractor",
Validators: []ValidatorManifest{
{Key: "grounded", ExecutionClass: "deterministic"},
},
},
},
}
gotJSON, err := json.Marshal(manifest)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
var got map[string]any
if err := json.Unmarshal(gotJSON, &got); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
assertHasKeys(t, got, "pipeline_id", "pipeline_digest", "artifact_lanes", "validator_chains", "llm_profiles")
profiles, ok := got["llm_profiles"].([]any)
if !ok {
t.Fatalf("llm_profiles = %#v, want array", got["llm_profiles"])
}
if len(profiles) != 1 {
t.Fatalf("len(llm_profiles) = %d, want 1", len(profiles))
}
profile, ok := profiles[0].(map[string]any)
if !ok {
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
}
assertHasKeys(t, profile, "id", "provider", "model", "backend_id", "reasoning_effort")
if profile["provider"] != "promptkit" {
t.Fatalf("llm_profiles[0].provider = %#v, want promptkit", profile["provider"])
}
if profile["backend_id"] != "openrouter" || profile["reasoning_effort"] != "high" {
t.Fatalf("llm_profiles[0] = %#v, want backend and reasoning provenance", profile)
}
lanes, ok := got["artifact_lanes"].([]any)
if !ok {
t.Fatalf("artifact_lanes = %#v, want array", got["artifact_lanes"])
}
if len(lanes) != 1 {
t.Fatalf("len(artifact_lanes) = %d, want 1", len(lanes))
}
lane, ok := lanes[0].(map[string]any)
if !ok {
t.Fatalf("artifact_lanes[0] = %#v, want object", lanes[0])
}
assertHasKeys(t, lane, "id", "extractor", "merger", "normalizer", "metadata")
chains, ok := got["validator_chains"].([]any)
if !ok {
t.Fatalf("validator_chains = %#v, want array", got["validator_chains"])
}
if len(chains) != 1 {
t.Fatalf("len(validator_chains) = %d, want 1", len(chains))
}
chain, ok := chains[0].(map[string]any)
if !ok {
t.Fatalf("validator_chains[0] = %#v, want object", chains[0])
}
assertHasKeys(t, chain, "stage", "lane_id", "module_key", "validators")
}
func TestRunManifestIncludesReferenceProvenance(t *testing.T) {
manifest := RunManifest{
References: []ReferenceProvenance{
{
Stage: "extract",
LaneID: "events",
SlotName: "roster",
OriginType: "file",
OriginURI: "file:///tmp/roster.txt",
Digest: "sha256:reference",
MediaType: "text/plain; charset=utf-8",
SizeBytes: 12,
BindingSource: "config",
},
},
}
gotJSON, err := json.Marshal(manifest)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
var got RunManifest
if err := json.Unmarshal(gotJSON, &got); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
if len(got.References) != 1 {
t.Fatalf("len(References) = %d, want 1", len(got.References))
}
reference := got.References[0]
if reference.Stage != "extract" || reference.LaneID != "events" || reference.SlotName != "roster" || reference.OriginType != "file" || reference.OriginURI != "file:///tmp/roster.txt" {
t.Fatalf("reference provenance = %#v, want lane-scoped origin details", reference)
}
if reference.Digest != "sha256:reference" || reference.MediaType != "text/plain; charset=utf-8" || reference.SizeBytes != 12 || reference.BindingSource != "config" {
t.Fatalf("reference provenance = %#v, want digest/media/size/source details", reference)
}
}
func TestRunManifestIncludesTopLevelModuleMetadata(t *testing.T) {
manifest := RunManifest{
ModuleMetadata: map[string]map[string]any{
"chunker": {
"prompt_id": "dnd.scenes",
"prompt_version": "v1",
"prompt_sha256": "sha256:abc123",
"response_schema_key": "dnd_scenes",
"response_schema_name": "dnd_scenes",
},
},
}
gotJSON, err := json.Marshal(manifest)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
var got map[string]any
if err := json.Unmarshal(gotJSON, &got); err != nil {
t.Fatalf("json.Unmarshal() error = %v", err)
}
moduleMetadata, ok := got["module_metadata"].(map[string]any)
if !ok {
t.Fatalf("module_metadata = %#v, want object", got["module_metadata"])
}
assertHasKeys(t, moduleMetadata, "chunker")
chunkerMetadata, ok := moduleMetadata["chunker"].(map[string]any)
if !ok {
t.Fatalf("module_metadata.chunker = %#v, want object", moduleMetadata["chunker"])
}
assertHasKeys(t, chunkerMetadata, "prompt_id", "prompt_version", "prompt_sha256", "response_schema_key", "response_schema_name")
}
func assertHasKeys(t *testing.T, values map[string]any, keys ...string) {
t.Helper()

31
internal/core/config/cache.go
View File

@@ -1,31 +0,0 @@
package config
import (
"fmt"
"path/filepath"
"strings"
)
// DefaultChunkPlanRoot resolves the existing per-user chunk-plan cache root.
func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error) {
return defaultCacheFamilyRoot(userCacheDir, "chunk-plans")
}
func DefaultCheckpointRoot(userCacheDir func() (string, error)) (string, error) {
return defaultCacheFamilyRoot(userCacheDir, "checkpoints")
}
func defaultCacheFamilyRoot(userCacheDir func() (string, error), family string) (string, error) {
if userCacheDir == nil {
return "", fmt.Errorf("user cache directory resolver must not be nil")
}
root, err := userCacheDir()
if err != nil {
return "", fmt.Errorf("resolve user cache directory: %w", err)
}
root = strings.TrimSpace(root)
if root == "" {
return "", fmt.Errorf("user cache directory must not be empty")
}
return filepath.Join(filepath.Clean(root), "notarius", family), nil
}

240
internal/core/config/config.go
View File

@@ -1,240 +0,0 @@
package config
import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const SupportedFileConfigVersion = 4
type Config struct {
PromptKit PromptKitConfig `json:"promptkit,omitempty"`
Pipelines map[string]pipeline.PipelineProfile `json:"pipelines"`
Concurrency ConcurrencyConfig `json:"concurrency"`
Output OutputConfig `json:"output"`
Cache CacheConfig `json:"cache"`
Debug DebugConfig `json:"debug"`
}
type PromptKitConfig struct {
ProfileDir string `json:"profile_dir,omitempty"`
ProfileFile string `json:"profile_file,omitempty"`
LocalBackend *PromptKitLocalBackendConfig `json:"local_backend,omitempty"`
}
type PromptKitLocalBackendConfig struct {
Endpoint string `json:"endpoint"`
ConcurrencyLimit int `json:"concurrency_limit"`
}
type ConcurrencyConfig struct {
TotalLLM int `json:"total_llm"`
StageWorkers map[string]int `json:"stage_workers"`
extractWorkersConfigured bool
defaultedExtractWorkers int
}
type OutputConfig struct {
Directory string `json:"directory"`
}
type CacheConfig struct {
ChunkPlans ChunkPlanCacheConfig `json:"chunk_plans"`
Checkpoints CheckpointCacheConfig `json:"checkpoints"`
}
type ChunkPlanCacheConfig struct {
Directory string `json:"directory,omitempty"`
Mode pipeline.ChunkCacheMode `json:"mode"`
}
type CheckpointCacheConfig struct {
Enabled bool `json:"enabled"`
Directory string `json:"directory,omitempty"`
}
type DebugConfig struct {
Directory string `json:"directory"`
}
func Default() Config {
return Config{
Pipelines: map[string]pipeline.PipelineProfile{},
Concurrency: ConcurrencyConfig{
TotalLLM: 1,
StageWorkers: map[string]int{"extract": 1},
defaultedExtractWorkers: 1,
},
Output: OutputConfig{Directory: "./notarius-output"},
Cache: CacheConfig{ChunkPlans: ChunkPlanCacheConfig{Mode: pipeline.ChunkCacheAuto}},
Debug: DebugConfig{Directory: "./notarius-debug"},
}
}
func cloneConfig(in Config) Config {
out := in
if in.PromptKit.LocalBackend != nil {
localBackend := *in.PromptKit.LocalBackend
out.PromptKit.LocalBackend = &localBackend
}
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
for key, profile := range in.Pipelines {
out.Pipelines[key] = clonePipelineProfile(profile)
}
return out
}
func cloneIntMap(in map[string]int) map[string]int {
if len(in) == 0 {
return nil
}
out := make(map[string]int, len(in))
for key, value := range in {
out[key] = value
}
return out
}
func (c *ConcurrencyConfig) recomputeStageWorkerDefaults() {
if c == nil {
return
}
if c.StageWorkers == nil {
c.StageWorkers = make(map[string]int)
}
if !c.extractWorkersConfigured {
if value, ok := c.StageWorkers["extract"]; ok && (c.defaultedExtractWorkers == 0 || value != c.defaultedExtractWorkers) {
c.extractWorkersConfigured = true
return
}
c.StageWorkers["extract"] = c.TotalLLM
c.defaultedExtractWorkers = c.TotalLLM
}
}
func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile {
out := in
out.Input = cloneModuleBinding(in.Input)
out.Chunk = cloneModuleBinding(in.Chunk)
out.Output = cloneModuleBinding(in.Output)
out.References = cloneReferenceSourceMap(in.References)
if len(in.Artifacts) > 0 {
out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts))
for key, lane := range in.Artifacts {
out.Artifacts[key] = cloneArtifactLaneProfile(lane)
}
}
if in.Steps != nil {
out.Steps = make([]pipeline.PipelineStepProfile, len(in.Steps))
for i, step := range in.Steps {
out.Steps[i] = clonePipelineStepProfile(step)
}
}
return out
}
func clonePipelineStepProfile(in pipeline.PipelineStepProfile) pipeline.PipelineStepProfile {
out := in
out.ID = in.ID
out.References = cloneReferenceSourceMap(in.References)
if len(in.Artifacts) > 0 {
out.Artifacts = make(map[string]pipeline.ArtifactLaneProfile, len(in.Artifacts))
for key, lane := range in.Artifacts {
out.Artifacts[key] = cloneArtifactLaneProfile(lane)
}
}
return out
}
func cloneArtifactLaneProfile(in pipeline.ArtifactLaneProfile) pipeline.ArtifactLaneProfile {
out := in
out.Extract = cloneModuleBinding(in.Extract)
out.Merge = cloneModuleBinding(in.Merge)
out.Normalize = cloneModuleBinding(in.Normalize)
out.References = cloneReferenceSourceMap(in.References)
if len(in.Validators) > 0 {
out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
for i, binding := range in.Validators {
out.Validators[i] = cloneModuleBinding(binding)
}
}
return out
}
func cloneStringMap(in map[string]string) map[string]string {
if len(in) == 0 {
return nil
}
out := make(map[string]string, len(in))
for key, value := range in {
out[key] = value
}
return out
}
func cloneReferenceSourceMap(in map[string]pipeline.ReferenceSource) map[string]pipeline.ReferenceSource {
if len(in) == 0 {
return nil
}
out := make(map[string]pipeline.ReferenceSource, len(in))
for key, source := range in {
out[key] = cloneReferenceSource(source)
}
return out
}
func cloneReferenceSource(in pipeline.ReferenceSource) pipeline.ReferenceSource {
out := in
if in.Artifact != nil {
artifact := *in.Artifact
out.Artifact = &artifact
}
return out
}
func cloneModuleBinding(in pipeline.ModuleBinding) pipeline.ModuleBinding {
out := in
if len(in.Options) > 0 {
out.Options = cloneOptions(in.Options)
}
out.References = cloneReferenceSourceMap(in.References)
out.Validators = cloneValidatorOverride(in.Validators)
return out
}
func cloneValidatorOverride(in pipeline.ValidatorOverride) pipeline.ValidatorOverride {
out := pipeline.ValidatorOverride{Set: in.Set}
if len(in.Validators) > 0 {
out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))
for i, binding := range in.Validators {
out.Validators[i] = cloneModuleBinding(binding)
}
}
return out
}
func cloneOptions(in map[string]any) map[string]any {
if len(in) == 0 {
return nil
}
out := make(map[string]any, len(in))
for key, value := range in {
out[key] = cloneOptionValue(value)
}
return out
}
func cloneOptionValue(value any) any {
switch typed := value.(type) {
case map[string]any:
return cloneOptions(typed)
case []any:
out := make([]any, len(typed))
for i, item := range typed {
out[i] = cloneOptionValue(item)
}
return out
default:
return typed
}
}

74
internal/core/config/effective_config.go
View File

@@ -1,74 +0,0 @@
package config
import (
"fmt"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
type ResolveInput struct {
PipelineID string
Only []string
Catalog pipeline.ModuleCatalog
LLMProfileOverride string
ReferenceOverrides []pipeline.ReferenceBinding
ReferenceUnbinds []pipeline.ReferenceUnbind
}
type EffectiveConfig struct {
Config Config
PipelineID string
Only []string
ReferenceOverrides []pipeline.ReferenceBinding
ReferenceUnbinds []pipeline.ReferenceUnbind
ResolvedPipeline pipeline.ResolvedPipeline
}
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
c.Concurrency.recomputeStageWorkerDefaults()
if err := c.Validate(); err != nil {
return EffectiveConfig{}, err
}
pipelineID := strings.TrimSpace(input.PipelineID)
if pipelineID == "" {
return EffectiveConfig{}, fmt.Errorf("pipeline id must not be empty")
}
profile, ok := lookupPipelineProfile(c.Pipelines, pipelineID)
if !ok {
return EffectiveConfig{}, fmt.Errorf("pipeline %q is not configured", pipelineID)
}
profile = clonePipelineProfile(profile)
profile.ID = pipelineID
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{
Only: input.Only,
LLMProfileOverride: input.LLMProfileOverride,
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
}, input.Catalog)
if err != nil {
return EffectiveConfig{}, fmt.Errorf("resolve pipeline %q: %w", pipelineID, err)
}
return EffectiveConfig{
Config: cloneConfig(c),
PipelineID: pipelineID,
Only: append([]string(nil), input.Only...),
ReferenceOverrides: append([]pipeline.ReferenceBinding(nil), input.ReferenceOverrides...),
ReferenceUnbinds: append([]pipeline.ReferenceUnbind(nil), input.ReferenceUnbinds...),
ResolvedPipeline: resolved,
}, nil
}
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
pipelineID = strings.TrimSpace(pipelineID)
for rawID, profile := range profiles {
if strings.TrimSpace(rawID) == pipelineID {
return profile, true
}
}
return pipeline.PipelineProfile{}, false
}

584
internal/core/config/effective_config_contract_test.go
View File

@@ -1,584 +0,0 @@
package config
import (
"context"
"encoding/json"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestEffectiveConfigRejectsEmptyAndUnknownPipelineIDs(t *testing.T) {
cfg := configForEffectiveTests(t, effectiveProfile())
for _, pipelineID := range []string{"", "missing"} {
name := pipelineID
if name == "" {
name = "empty"
}
t.Run(name, func(t *testing.T) {
_, err := cfg.Resolve(ResolveInput{PipelineID: pipelineID, Catalog: effectiveCatalog(t)})
if err == nil || !strings.Contains(err.Error(), "pipeline") {
t.Fatalf("Resolve(%q) error = %v, want pipeline context", pipelineID, err)
}
})
}
}
func TestEffectiveConfigResolvesTrimmedPipelineMapKeys(t *testing.T) {
profile := effectiveProfile()
profile.ID = " main "
cfg := Default()
cfg.Pipelines = map[string]pipeline.PipelineProfile{" main ": profile}
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if effective.PipelineID != "main" || effective.ResolvedPipeline.ID != "main" {
t.Fatalf("resolved IDs = %q, %q", effective.PipelineID, effective.ResolvedPipeline.ID)
}
}
func TestEffectiveConfigOnlySelectsRequestedLanesWithoutMutatingSource(t *testing.T) {
profile := effectiveProfile()
profile.Artifacts["other"] = pipeline.ArtifactLaneProfile{Extract: pipeline.Binding("extract")}
cfg := configForEffectiveTests(t, profile)
effective, err := cfg.Resolve(ResolveInput{
PipelineID: "main",
Only: []string{"other"},
Catalog: effectiveCatalog(t),
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if len(effective.ResolvedPipeline.Steps[0].ArtifactLanes) != 1 || effective.ResolvedPipeline.Steps[0].ArtifactLanes[0].ID != "other" {
t.Fatalf("resolved lanes = %#v", effective.ResolvedPipeline.Steps[0].ArtifactLanes)
}
if len(cfg.Pipelines["main"].Artifacts) != 2 {
t.Fatalf("source lanes were mutated: %#v", cfg.Pipelines["main"].Artifacts)
}
_, err = cfg.Resolve(ResolveInput{
PipelineID: "main",
Only: []string{"missing"},
Catalog: effectiveCatalog(t),
})
if err == nil || !strings.Contains(err.Error(), "lane \"missing\"") {
t.Fatalf("unknown lane error = %v", err)
}
}
func TestEffectiveConfigMaterializesDefaultBindingsThroughCatalog(t *testing.T) {
effective, err := resolveEffectiveProfile(t, effectiveProfile(), ResolveInput{})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
resolved := effective.ResolvedPipeline
if resolved.Chunk.Module != pipeline.DefaultChunkModule || resolved.Output.Module != pipeline.DefaultOutputModule {
t.Fatalf("default pipeline bindings = %#v, %#v", resolved.Chunk, resolved.Output)
}
if len(resolved.Steps[0].ArtifactLanes) != 1 || resolved.Steps[0].ArtifactLanes[0].Merge.Module != pipeline.DefaultMergeModule || resolved.Steps[0].ArtifactLanes[0].Normalize.Module != pipeline.DefaultNormalizeModule {
t.Fatalf("default lane bindings = %#v", resolved.Steps[0].ArtifactLanes)
}
}
func TestEffectiveConfigPreservesPromptKitProfileSource(t *testing.T) {
tests := []struct {
name string
profileSource PromptKitConfig
}{
{name: "profile directory", profileSource: PromptKitConfig{ProfileDir: "./profiles"}},
{name: "profile file", profileSource: PromptKitConfig{ProfileFile: "./profiles.yml"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := configForEffectiveTests(t, effectiveProfile())
cfg.PromptKit = tt.profileSource
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if effective.Config.PromptKit != cfg.PromptKit {
t.Fatalf("effective PromptKit config = %#v, want %#v", effective.Config.PromptKit, cfg.PromptKit)
}
})
}
}
func TestEffectiveConfigOwnsPromptKitLocalBackend(t *testing.T) {
cfg := configForEffectiveTests(t, effectiveProfile())
cfg.PromptKit.LocalBackend = &PromptKitLocalBackendConfig{
Endpoint: "http://localhost:8000/v1",
ConcurrencyLimit: 2,
}
effective, err := cfg.Resolve(ResolveInput{PipelineID: "main", Catalog: effectiveCatalog(t)})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if effective.Config.PromptKit.LocalBackend == nil {
t.Fatal("effective local backend = nil")
}
if effective.Config.PromptKit.LocalBackend == cfg.PromptKit.LocalBackend {
t.Fatal("effective local backend aliases input config")
}
cfg.PromptKit.LocalBackend.Endpoint = "http://changed-input.example/v1"
if effective.Config.PromptKit.LocalBackend.Endpoint != "http://localhost:8000/v1" {
t.Fatalf("input mutation changed effective config: %#v", effective.Config.PromptKit.LocalBackend)
}
effective.Config.PromptKit.LocalBackend.ConcurrencyLimit = 9
if cfg.PromptKit.LocalBackend.ConcurrencyLimit != 2 {
t.Fatalf("effective mutation changed input config: %#v", cfg.PromptKit.LocalBackend)
}
}
func TestEffectiveConfigResolutionFailuresRetainContext(t *testing.T) {
tests := []struct {
name string
mutate func(*pipeline.PipelineProfile)
want []string
}{
{
name: "unknown module",
mutate: func(profile *pipeline.PipelineProfile) {
profile.Input.Module = "missing-input"
},
want: []string{"pipeline \"main\"", "input"},
},
{
name: "missing capability",
mutate: func(profile *pipeline.PipelineProfile) {
profile.Chunk.Module = "needs-capability"
},
want: []string{"pipeline \"main\"", "chunk"},
},
{
name: "missing artifact variant",
mutate: func(profile *pipeline.PipelineProfile) {
profile.Artifacts["lane"] = pipeline.ArtifactLaneProfile{
Extract: pipeline.Binding("extract"),
Merge: pipeline.Binding("other-merge"),
}
},
want: []string{"pipeline \"main\"", "lane \"lane\"", "merge"},
},
{
name: "invalid module options",
mutate: func(profile *pipeline.PipelineProfile) {
profile.Chunk = pipeline.ModuleBinding{Module: "generic", Options: map[string]any{"unknown": true}}
},
want: []string{"pipeline \"main\"", "chunk", "generic", "options"},
},
{
name: "invalid validator options",
mutate: func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["lane"]
lane.Extract.Validators = pipeline.ValidatorOverride{
Set: true,
Validators: []pipeline.ModuleBinding{{
Module: "option-validator",
Options: map[string]any{"invalid": true},
}},
}
profile.Artifacts["lane"] = lane
},
want: []string{"pipeline \"main\"", "lane \"lane\"", "extract", "option-validator", "options"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
profile := effectiveProfile()
tt.mutate(&profile)
_, err := resolveEffectiveProfile(t, profile, ResolveInput{})
if err == nil {
t.Fatal("Resolve() error = nil, want failure")
}
for _, fragment := range tt.want {
if !strings.Contains(err.Error(), fragment) {
t.Fatalf("Resolve() error = %v, want context %q", err, fragment)
}
}
})
}
}
func TestEffectiveConfigLLMProfileOverrideChangesDigestAndOverridesValidators(t *testing.T) {
profile := effectiveProfile()
profile.Chunk.LLMProfile = "chunk-profile"
lane := profile.Artifacts["lane"]
lane.Extract.LLMProfile = "extract-profile"
lane.Merge.LLMProfile = "merge-profile"
lane.Normalize.LLMProfile = "normalize-profile"
lane.Extract.Validators = pipeline.ValidatorOverride{
Set: true,
Validators: []pipeline.ModuleBinding{{
Module: "llm-validator",
LLMProfile: "validator-profile",
}},
}
profile.Artifacts["lane"] = lane
base, err := resolveEffectiveProfile(t, profile, ResolveInput{})
if err != nil {
t.Fatalf("base Resolve() error = %v", err)
}
overridden, err := resolveEffectiveProfile(t, profile, ResolveInput{LLMProfileOverride: "override-profile"})
if err != nil {
t.Fatalf("overridden Resolve() error = %v", err)
}
if base.ResolvedPipeline.Digest == overridden.ResolvedPipeline.Digest {
t.Fatal("LLM profile override did not change the pipeline digest")
}
resolved := overridden.ResolvedPipeline
if resolved.Chunk.LLMProfile != "override-profile" || resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile != "override-profile" ||
resolved.Steps[0].ArtifactLanes[0].Merge.LLMProfile != "override-profile" || resolved.Steps[0].ArtifactLanes[0].Normalize.LLMProfile != "override-profile" {
t.Fatalf("pipeline profile override was not applied: %#v", resolved)
}
validators := findEffectiveValidatorChain(resolved, pipeline.StageExtract, "lane")
if len(validators.Validators) != 1 || validators.Validators[0].Binding.LLMProfile != "override-profile" {
t.Fatalf("validator profile = %#v, want runtime override", validators)
}
}
func TestEffectiveConfigPipelineLLMProfileIsInheritedWithoutMutatingConfig(t *testing.T) {
profile := effectiveProfile()
profile.LLMProfile = " configured-profile "
effective, err := resolveEffectiveProfile(t, profile, ResolveInput{})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if got := effective.Config.Pipelines["main"].LLMProfile; got != " configured-profile " {
t.Fatalf("effective config pipeline llm profile = %q, want preserved programmatic value", got)
}
resolved := effective.ResolvedPipeline
if got := resolved.Chunk.LLMProfile; got != "configured-profile" {
t.Fatalf("resolved chunk profile = %q, want inherited profile", got)
}
if got := resolved.Steps[0].ArtifactLanes[0].Extract.LLMProfile; got != "configured-profile" {
t.Fatalf("resolved extract profile = %q, want inherited profile", got)
}
}
func TestEffectiveConfigValidatorOverridesRemainDistinctAndOrdered(t *testing.T) {
tests := []struct {
name string
value pipeline.ValidatorOverride
want []string
}{
{
name: "omitted uses default",
want: []string{"default-validator"},
},
{
name: "explicit empty",
value: pipeline.ValidatorOverride{Set: true},
want: nil,
},
{
name: "configured order",
value: pipeline.ValidatorOverride{
Set: true,
Validators: []pipeline.ModuleBinding{
pipeline.Binding("configured-a"),
pipeline.Binding("configured-b"),
},
},
want: []string{"configured-a", "configured-b"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
profile := effectiveProfile()
lane := profile.Artifacts["lane"]
lane.Extract.Validators = tt.value
profile.Artifacts["lane"] = lane
effective, err := resolveEffectiveProfile(t, profile, ResolveInput{})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
chain := findEffectiveValidatorChain(effective.ResolvedPipeline, pipeline.StageExtract, "lane")
got := make([]string, len(chain.Validators))
for i, validator := range chain.Validators {
got[i] = validator.Binding.Module
}
if len(got) != len(tt.want) {
t.Fatalf("validator chain = %#v, want %v", got, tt.want)
}
for i := range got {
if got[i] != tt.want[i] {
t.Fatalf("validator chain = %#v, want %v", got, tt.want)
}
}
})
}
}
func TestEffectiveConfigAndResolutionInputsDoNotAliasSource(t *testing.T) {
profile := effectiveProfile()
profile.Chunk.Options = map[string]any{"nested": map[string]any{"safe": "source"}}
profile.Chunk.References = pipeline.ExternalReferenceMap(map[string]string{"chunk-ref": "chunk.txt"})
lane := profile.Artifacts["lane"]
lane.Extract.Validators = pipeline.ValidatorOverride{
Set: true,
Validators: []pipeline.ModuleBinding{{
Module: "configured-a",
Options: map[string]any{"nested": map[string]any{"safe": "validator-source"}},
}},
}
profile.Artifacts["lane"] = lane
cfg := configForEffectiveTests(t, profile)
only := []string{"lane"}
overrides := []pipeline.ReferenceBinding{{Stage: pipeline.StageChunk, SlotName: "chunk-ref", Source: "source.txt"}}
effective, err := cfg.Resolve(ResolveInput{
PipelineID: "main",
Only: only,
ReferenceOverrides: overrides,
Catalog: effectiveCatalog(t),
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
effective.Config.Pipelines["main"].Chunk.Options["nested"].(map[string]any)["safe"] = "effective-config"
effective.ResolvedPipeline.Chunk.Options["nested"].(map[string]any)["safe"] = "resolved-pipeline"
effective.ResolvedPipeline.ChunkReferences.Bindings[0].Source = "resolved-reference"
effective.ResolvedPipeline.ValidatorChains[1].Validators[0].Binding.Options["nested"].(map[string]any)["safe"] = "resolved-validator"
effective.Only[0] = "mutated-only"
effective.ReferenceOverrides[0].Source = "mutated-override"
if got := cfg.Pipelines["main"].Chunk.Options["nested"].(map[string]any)["safe"]; got != "source" {
t.Fatalf("source config option was aliased: %v", got)
}
if got := cfg.Pipelines["main"].Chunk.References["chunk-ref"].Path; got != "chunk.txt" {
t.Fatalf("source config references were aliased: %v", got)
}
if only[0] != "lane" || overrides[0].Source != "source.txt" {
t.Fatal("resolution inputs were aliased")
}
}
func effectiveProfile() pipeline.PipelineProfile {
return pipeline.PipelineProfile{
ID: "main",
Input: pipeline.Binding("input"),
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"lane": {Extract: pipeline.Binding("extract")},
},
}
}
func configForEffectiveTests(t *testing.T, profile pipeline.PipelineProfile) Config {
t.Helper()
cfg := Default()
cfg.Pipelines = map[string]pipeline.PipelineProfile{"main": profile}
return cfg
}
func resolveEffectiveProfile(t *testing.T, profile pipeline.PipelineProfile, input ResolveInput) (EffectiveConfig, error) {
t.Helper()
cfg := configForEffectiveTests(t, profile)
if input.PipelineID == "" {
input.PipelineID = "main"
}
if input.Catalog.Inputs == nil {
input.Catalog = effectiveCatalog(t)
}
return cfg.Resolve(input)
}
func findEffectiveValidatorChain(resolved pipeline.ResolvedPipeline, stage pipeline.ModuleStage, laneID string) pipeline.ResolvedValidatorChain {
for _, chain := range resolved.ValidatorChains {
if chain.Stage == stage && chain.LaneID == laneID {
return chain
}
}
return pipeline.ResolvedValidatorChain{}
}
type effectiveArtifact struct {
Value string `json:"value"`
}
const effectiveArtifactKind contracts.ArtifactKind = "test/effective"
type effectiveCodec struct{}
func (effectiveCodec) Kind() contracts.ArtifactKind { return effectiveArtifactKind }
func (effectiveCodec) Schema() contracts.ArtifactSchema {
return contracts.ArtifactSchema{
ID: "effective-schema",
Name: "Effective artifact",
Version: "1",
JSONSchema: []byte(`{"type":"object"}`),
}
}
func (effectiveCodec) MediaType() string { return "application/json" }
func (effectiveCodec) EncodeCandidate(value effectiveArtifact) ([]byte, error) {
return json.Marshal(value)
}
func (effectiveCodec) Encode(value effectiveArtifact) ([]byte, error) {
return json.Marshal(value)
}
func (effectiveCodec) Decode(content []byte) (effectiveArtifact, error) {
var value effectiveArtifact
err := json.Unmarshal(content, &value)
return value, err
}
type effectiveInput struct{ key string }
func (m effectiveInput) Key() string { return m.key }
func (m effectiveInput) Parse(context.Context, contracts.ParseRequest) (*source.SourceDocument, error) {
return &source.SourceDocument{}, nil
}
type effectiveChunker struct{ key string }
func (m effectiveChunker) Key() string { return m.key }
func (m effectiveChunker) ReferenceSlots() []contracts.ReferenceSlot {
if m.key == pipeline.DefaultChunkModule {
return []contracts.ReferenceSlot{{Name: "chunk-ref"}}
}
return nil
}
func (m effectiveChunker) Plan(context.Context, contracts.ChunkRequest) (contracts.ChunkPlanResult, error) {
return contracts.ChunkPlanResult{}, nil
}
type effectiveExtractor struct{ key string }
func (m effectiveExtractor) Key() string { return m.key }
func (m effectiveExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (m effectiveExtractor) Extract(context.Context, contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[effectiveArtifact], error) {
return contracts.TypedExtractionResult[effectiveArtifact]{}, nil
}
type effectiveMerger struct{ key string }
func (m effectiveMerger) Key() string { return m.key }
func (m effectiveMerger) Merge(context.Context, contracts.TypedMergeRequest[effectiveArtifact]) (contracts.TypedMergeResult[effectiveArtifact], error) {
return contracts.TypedMergeResult[effectiveArtifact]{}, nil
}
type effectiveNormalizer struct{ key string }
func (m effectiveNormalizer) Key() string { return m.key }
func (m effectiveNormalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (m effectiveNormalizer) Normalize(context.Context, contracts.TypedNormalizeRequest[effectiveArtifact]) (contracts.TypedNormalizeResult[effectiveArtifact], error) {
return contracts.TypedNormalizeResult[effectiveArtifact]{}, nil
}
type effectiveOutput struct{ key string }
func (m effectiveOutput) Key() string { return m.key }
func (m effectiveOutput) Encode(context.Context, contracts.OutputRequest) (contracts.OutputResult, error) {
return contracts.OutputResult{}, nil
}
type effectiveValidator struct {
name string
class contracts.ExecutionClass
}
func (v effectiveValidator) Name() string { return v.name }
func (v effectiveValidator) ExecutionClass() contracts.ExecutionClass { return v.class }
func (v effectiveValidator) Validate(context.Context, contracts.TypedValidationRequest[effectiveArtifact]) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}
func effectiveCatalog(t *testing.T) pipeline.ModuleCatalog {
t.Helper()
catalog := pipeline.ModuleCatalog{
Inputs: pipeline.NewInputAdapterRegistry(),
Chunkers: pipeline.NewChunkerRegistry(),
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
Extractors: pipeline.NewExtractorRegistry(),
Mergers: pipeline.NewMergerRegistry(),
Normalizers: pipeline.NewNormalizerRegistry(),
Validators: pipeline.NewValidatorRegistry(),
ValidatorChains: pipeline.NewValidatorChainRegistry(),
Outputs: pipeline.NewOutputEncoderRegistry(),
}
if err := pipeline.RegisterArtifactCodec(catalog.ArtifactCodecs, effectiveCodec{}); err != nil {
t.Fatal(err)
}
if err := catalog.Inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "input", Stage: pipeline.StageInput, ExecutionClass: contracts.ExecutionClassDeterministic, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
return effectiveInput{key: "input"}, nil
}); err != nil {
t.Fatal(err)
}
chunkSpec := pipeline.ModuleSpec{
Key: pipeline.DefaultChunkModule,
Stage: pipeline.StageChunk,
ExecutionClass: contracts.ExecutionClassLLMBacked,
Requires: []string{"source"},
Provides: []string{"chunk"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "chunk-ref"}},
}
chunkOptions := func(options map[string]any) error { return pipeline.RejectUnknownOptions(options, "size", "nested") }
if err := catalog.Chunkers.RegisterBuilderWithSpec(chunkSpec, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
return effectiveChunker{key: pipeline.DefaultChunkModule}, nil
}); err != nil {
t.Fatal(err)
}
if err := catalog.Chunkers.RegisterBuilderWithSpec(pipeline.ModuleSpec{Key: "needs-capability", Stage: pipeline.StageChunk, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"missing"}}, chunkOptions, func(pipeline.BuildRequest) (contracts.Chunker, error) {
return effectiveChunker{key: "needs-capability"}, nil
}); err != nil {
t.Fatal(err)
}
if err := pipeline.RegisterExtractor(catalog.Extractors, pipeline.ModuleSpec{Key: "extract", Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"chunk"}, Provides: []string{"candidate"}}, func() (contracts.Extractor[effectiveArtifact], error) {
return effectiveExtractor{key: "extract"}, nil
}); err != nil {
t.Fatal(err)
}
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: pipeline.DefaultMergeModule, Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
return effectiveMerger{key: pipeline.DefaultMergeModule}, nil
}); err != nil {
t.Fatal(err)
}
if err := pipeline.RegisterMerger(catalog.Mergers, pipeline.ModuleSpec{Key: "other-merge", Stage: pipeline.StageMerge, ExecutionClass: contracts.ExecutionClassDeterministic, ArtifactKind: "other-kind", Requires: []string{"candidate"}, Provides: []string{"merged"}}, func() (contracts.Merger[effectiveArtifact], error) {
return effectiveMerger{key: "other-merge"}, nil
}); err != nil {
t.Fatal(err)
}
if err := pipeline.RegisterNormalizer(catalog.Normalizers, pipeline.ModuleSpec{Key: pipeline.DefaultNormalizeModule, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, ArtifactKind: effectiveArtifactKind, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer[effectiveArtifact], error) {
return effectiveNormalizer{key: pipeline.DefaultNormalizeModule}, nil
}); err != nil {
t.Fatal(err)
}
if err := catalog.Outputs.RegisterWithSpec(pipeline.ModuleSpec{Key: pipeline.DefaultOutputModule, Stage: pipeline.StageOutput, ExecutionClass: contracts.ExecutionClassDeterministic, Requires: []string{"normalized"}}, func() (contracts.OutputEncoder, error) {
return effectiveOutput{key: pipeline.DefaultOutputModule}, nil
}); err != nil {
t.Fatal(err)
}
for _, validator := range []struct {
key string
class contracts.ExecutionClass
}{
{key: "default-validator", class: contracts.ExecutionClassDeterministic},
{key: "configured-a", class: contracts.ExecutionClassDeterministic},
{key: "configured-b", class: contracts.ExecutionClassDeterministic},
{key: "llm-validator", class: contracts.ExecutionClassLLMBacked},
} {
if err := pipeline.RegisterTypedValidatorBuilder(catalog.Validators, effectiveArtifactKind, pipeline.ValidatorSpec{Key: validator.key, ExecutionClass: validator.class}, func(options map[string]any) error {
return pipeline.RejectUnknownOptions(options, "nested")
}, func(pipeline.BuildRequest) (contracts.TypedValidator[effectiveArtifact], error) {
return effectiveValidator{name: validator.key, class: validator.class}, nil
}); err != nil {
t.Fatal(err)
}
}
if err := pipeline.RegisterTypedValidatorBuilder(catalog.Validators, effectiveArtifactKind, pipeline.ValidatorSpec{Key: "option-validator", ExecutionClass: contracts.ExecutionClassDeterministic}, func(options map[string]any) error {
return pipeline.RejectUnknownOptions(options, "allowed")
}, func(pipeline.BuildRequest) (contracts.TypedValidator[effectiveArtifact], error) {
return effectiveValidator{name: "option-validator", class: contracts.ExecutionClassDeterministic}, nil
}); err != nil {
t.Fatal(err)
}
if err := catalog.ValidatorChains.Register(pipeline.ValidatorChainMapping{Stage: pipeline.StageExtract, Module: "extract", Validators: []pipeline.ModuleBinding{pipeline.Binding("default-validator")}}); err != nil {
t.Fatal(err)
}
return catalog
}

103
internal/core/config/env.go
View File

@@ -1,103 +0,0 @@
package config
import (
"fmt"
"os"
"strconv"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func LoadFromEnv() (Config, error) {
cfg := Default()
if err := cfg.applyEnvOverridesWithLookup(os.LookupEnv); err != nil {
return Config{}, err
}
return cfg, nil
}
func (c *Config) ApplyEnvOverrides() error {
return c.applyEnvOverridesWithLookup(os.LookupEnv)
}
func (c *Config) ApplyEnvOverridesWithLookup(lookup func(string) (string, bool)) error {
return c.applyEnvOverridesWithLookup(lookup)
}
func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool)) error {
if c == nil {
return fmt.Errorf("config must not be nil")
}
if raw, ok := lookup("NOTARIUS_TOTAL_LLM_CONCURRENCY"); ok {
value, err := parseIntEnv("NOTARIUS_TOTAL_LLM_CONCURRENCY", raw)
if err != nil {
return err
}
c.Concurrency.TotalLLM = value
}
if raw, ok := lookup("NOTARIUS_STAGE_WORKERS_EXTRACT"); ok {
value, err := parseIntEnv("NOTARIUS_STAGE_WORKERS_EXTRACT", raw)
if err != nil {
return err
}
if c.Concurrency.StageWorkers == nil {
c.Concurrency.StageWorkers = make(map[string]int)
}
c.Concurrency.StageWorkers["extract"] = value
c.Concurrency.extractWorkersConfigured = true
}
c.Concurrency.recomputeStageWorkerDefaults()
if raw, ok := lookup("NOTARIUS_OUTPUT_DIR"); ok {
c.Output.Directory = strings.TrimSpace(raw)
if c.Output.Directory == "" {
return fmt.Errorf("NOTARIUS_OUTPUT_DIR: must not be empty")
}
if strings.ContainsRune(c.Output.Directory, '\x00') {
return fmt.Errorf("NOTARIUS_OUTPUT_DIR: must not contain NUL")
}
}
if raw, ok := lookup("NOTARIUS_CACHE_CHUNK_PLANS_MODE"); ok {
mode, err := pipeline.ParseChunkCacheMode(raw)
if err != nil {
return fmt.Errorf("NOTARIUS_CACHE_CHUNK_PLANS_MODE: %w", err)
}
c.Cache.ChunkPlans.Mode = mode
}
if raw, ok := lookup("NOTARIUS_CACHE_CHUNK_PLANS_DIR"); ok {
c.Cache.ChunkPlans.Directory = cleanOptionalPath(raw)
if c.Cache.ChunkPlans.Directory == "" {
return fmt.Errorf("NOTARIUS_CACHE_CHUNK_PLANS_DIR: must not be empty")
}
if strings.ContainsRune(c.Cache.ChunkPlans.Directory, '\x00') {
return fmt.Errorf("NOTARIUS_CACHE_CHUNK_PLANS_DIR: must not contain NUL")
}
}
if raw, ok := lookup("NOTARIUS_CACHE_CHECKPOINTS_DIR"); ok {
c.Cache.Checkpoints.Directory = cleanOptionalPath(raw)
if c.Cache.Checkpoints.Directory == "" {
return fmt.Errorf("NOTARIUS_CACHE_CHECKPOINTS_DIR: must not be empty")
}
if strings.ContainsRune(c.Cache.Checkpoints.Directory, '\x00') {
return fmt.Errorf("NOTARIUS_CACHE_CHECKPOINTS_DIR: must not contain NUL")
}
}
if raw, ok := lookup("NOTARIUS_DEBUG_DIR"); ok {
c.Debug.Directory = strings.TrimSpace(raw)
if c.Debug.Directory == "" {
return fmt.Errorf("NOTARIUS_DEBUG_DIR: must not be empty")
}
if strings.ContainsRune(c.Debug.Directory, '\x00') {
return fmt.Errorf("NOTARIUS_DEBUG_DIR: must not contain NUL")
}
}
return nil
}
func parseIntEnv(name string, raw string) (int, error) {
value, err := strconv.Atoi(strings.TrimSpace(raw))
if err != nil {
return 0, fmt.Errorf("%s: must be an integer", name)
}
return value, nil
}

239
internal/core/config/env_contract_test.go
View File

@@ -1,239 +0,0 @@
package config
import (
"errors"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestPrecedenceFileValuesOverrideBuiltInDefaults(t *testing.T) {
cfg := applyFileConfig(t, `version: 4
concurrency:
total_llm: 4
stage_workers:
extract: 2
output:
directory: ./file-output
cache:
chunk_plans:
directory: ./file-plans
mode: refresh
checkpoints:
directory: ./file-checkpoints
debug:
directory: ./file-debug
`)
if cfg.Concurrency.TotalLLM != 4 || cfg.Concurrency.StageWorkers["extract"] != 2 ||
cfg.Output.Directory != "./file-output" || cfg.Cache.ChunkPlans.Directory != "file-plans" ||
cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheRefresh || cfg.Cache.Checkpoints.Directory != "file-checkpoints" ||
cfg.Debug.Directory != "./file-debug" {
t.Fatalf("file values did not override defaults: %#v", cfg)
}
}
func TestPrecedenceOperationalEnvironmentOverridesFileValues(t *testing.T) {
cfg := applyFileConfig(t, `version: 4
concurrency:
total_llm: 2
stage_workers:
extract: 1
output:
directory: ./file-output
cache:
chunk_plans:
directory: ./file-plans
mode: refresh
checkpoints:
directory: ./file-checkpoints
debug:
directory: ./file-debug
`)
env := map[string]string{
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "8",
"NOTARIUS_STAGE_WORKERS_EXTRACT": "6",
"NOTARIUS_OUTPUT_DIR": "/env/output",
"NOTARIUS_CACHE_CHUNK_PLANS_MODE": "bypass",
"NOTARIUS_CACHE_CHUNK_PLANS_DIR": "/env/plans",
"NOTARIUS_CACHE_CHECKPOINTS_DIR": "/env/checkpoints",
"NOTARIUS_DEBUG_DIR": "/env/debug",
}
if err := cfg.ApplyEnvOverridesWithLookup(lookupValues(env)); err != nil {
t.Fatal(err)
}
if cfg.Concurrency.TotalLLM != 8 || cfg.Concurrency.StageWorkers["extract"] != 6 ||
cfg.Output.Directory != "/env/output" || cfg.Cache.ChunkPlans.Directory != "/env/plans" ||
cfg.Cache.ChunkPlans.Mode != pipeline.ChunkCacheBypass || cfg.Cache.Checkpoints.Directory != "/env/checkpoints" ||
cfg.Debug.Directory != "/env/debug" {
t.Fatalf("environment values did not override file values: %#v", cfg)
}
}
func TestPrecedenceExtractWorkersFollowEffectiveConcurrencyUnlessExplicit(t *testing.T) {
tests := []struct {
name string
file string
env map[string]string
wantTotal int
wantWorker int
}{
{
name: "default follows environment total",
file: "version: 4\n",
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5"},
wantTotal: 5,
wantWorker: 5,
},
{
name: "file worker is retained",
file: "version: 4\nconcurrency:\n total_llm: 3\n stage_workers:\n extract: 2\n",
env: map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"},
wantTotal: 6,
wantWorker: 2,
},
{
name: "environment worker is retained",
file: "version: 4\nconcurrency:\n total_llm: 2\n",
env: map[string]string{
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6",
"NOTARIUS_STAGE_WORKERS_EXTRACT": "4",
},
wantTotal: 6,
wantWorker: 4,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := applyFileConfig(t, tt.file)
if err := cfg.ApplyEnvOverridesWithLookup(lookupValues(tt.env)); err != nil {
t.Fatal(err)
}
if cfg.Concurrency.TotalLLM != tt.wantTotal || cfg.Concurrency.StageWorkers["extract"] != tt.wantWorker {
t.Fatalf("concurrency = %#v, want total %d and extract %d", cfg.Concurrency, tt.wantTotal, tt.wantWorker)
}
})
}
}
func TestPrecedenceEmptyFileCacheDirectoriesDeferPerUserResolution(t *testing.T) {
cfg := applyFileConfig(t, `version: 4
cache:
chunk_plans:
directory: ""
checkpoints:
directory: ""
`)
if err := cfg.Validate(); err != nil {
t.Fatalf("empty file cache directories should be valid: %v", err)
}
if cfg.Cache.ChunkPlans.Directory != "" || cfg.Cache.Checkpoints.Directory != "" {
t.Fatalf("empty cache directories were not preserved for deferred resolution: %#v", cfg.Cache)
}
resolver := func() (string, error) { return "/user/cache", nil }
chunkPlans, err := DefaultChunkPlanRoot(resolver)
if err != nil {
t.Fatal(err)
}
checkpoints, err := DefaultCheckpointRoot(resolver)
if err != nil {
t.Fatal(err)
}
if chunkPlans != "/user/cache/notarius/chunk-plans" || checkpoints != "/user/cache/notarius/checkpoints" {
t.Fatalf("deferred cache roots = %q, %q", chunkPlans, checkpoints)
}
}
func TestDefaultCacheRootsRejectInvalidUserCacheResolvers(t *testing.T) {
tests := []struct {
name string
resolver func() (string, error)
want string
}{
{name: "nil resolver", want: "must not be nil"},
{
name: "resolver failure",
resolver: func() (string, error) {
return "", errors.New("cache home unavailable")
},
want: "resolve user cache directory",
},
{name: "empty directory", resolver: func() (string, error) { return " ", nil }, want: "must not be empty"},
}
families := []struct {
name string
root func(func() (string, error)) (string, error)
}{
{name: "chunk plans", root: DefaultChunkPlanRoot},
{name: "checkpoints", root: DefaultCheckpointRoot},
}
for _, family := range families {
for _, tt := range tests {
t.Run(family.name+"/"+tt.name, func(t *testing.T) {
_, err := family.root(tt.resolver)
if err == nil || !strings.Contains(err.Error(), tt.want) {
t.Fatalf("error = %v, want substring %q", err, tt.want)
}
})
}
}
}
func TestEnvEmptyDirectoryOverridesAreErrors(t *testing.T) {
tests := []string{
"NOTARIUS_OUTPUT_DIR",
"NOTARIUS_CACHE_CHUNK_PLANS_DIR",
"NOTARIUS_CACHE_CHECKPOINTS_DIR",
"NOTARIUS_DEBUG_DIR",
}
for _, name := range tests {
t.Run(name, func(t *testing.T) {
cfg := Default()
err := cfg.ApplyEnvOverridesWithLookup(lookupValues(map[string]string{name: " \t"}))
if err == nil || !strings.Contains(err.Error(), name) {
t.Fatalf("error = %v, want responsible environment variable", err)
}
})
}
}
func TestEnvInvalidIntegersAndChunkCacheModesReportTheirNames(t *testing.T) {
tests := map[string]string{
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "not-an-integer",
"NOTARIUS_STAGE_WORKERS_EXTRACT": "not-an-integer",
"NOTARIUS_CACHE_CHUNK_PLANS_MODE": "not-a-cache-mode",
}
for name, value := range tests {
t.Run(name, func(t *testing.T) {
cfg := Default()
err := cfg.ApplyEnvOverridesWithLookup(lookupValues(map[string]string{name: value}))
if err == nil || !strings.Contains(err.Error(), name) {
t.Fatalf("error = %v, want responsible environment variable", err)
}
})
}
}
func TestEnvRemovedProviderVariablesAreIgnored(t *testing.T) {
before := Default()
cfg := Default()
removed := map[string]string{
"NOTARIUS_LLM_DEFAULT_ENDPOINT": "ignored-provider-setting",
"NOTARIUS_LLM_DEFAULT_MODEL": "ignored-provider-setting",
}
if err := cfg.ApplyEnvOverridesWithLookup(lookupValues(removed)); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(cfg, before) {
t.Fatalf("removed provider variables changed configuration: %#v", cfg)
}
}
func lookupValues(values map[string]string) func(string) (string, bool) {
return func(name string) (string, bool) {
value, ok := values[name]
return value, ok
}
}

Some files were not shown because too many files have changed in this diff Show More