211 lines
8.4 KiB
Markdown
211 lines
8.4 KiB
Markdown
# Architecture
|
|
|
|
This document defines Notarius development policy. It is inward-facing:
|
|
developers and LLM coding agents should use it to preserve the project's shape,
|
|
boundaries, and invariants as the code evolves.
|
|
|
|
Keep this document concise. It should describe durable architectural rules, not
|
|
CLI syntax, configuration reference material, module catalogs, or roadmap items.
|
|
|
|
## Project Shape
|
|
|
|
Notarius is a small, explicit, dependency-light Go application for extracting
|
|
structured artifacts from source material using modular pipeline stages.
|
|
|
|
The application is contract-first but not abstraction-heavy. Add interfaces and
|
|
extension points when they protect a real boundary:
|
|
|
|
- external source formats;
|
|
- pipeline stage modules;
|
|
- validators;
|
|
- LLM providers and runtime plumbing;
|
|
- output schemas and embedded assets.
|
|
|
|
Avoid abstractions that only anticipate hypothetical complexity. Prefer narrow
|
|
contracts that can be exercised by tests and real modules.
|
|
|
|
## Core Invariants
|
|
|
|
The framework must remain source-agnostic and domain-agnostic.
|
|
|
|
Source-format details belong in input modules. Transcript-specific concepts such
|
|
as segments, speakers, timestamps, and transcript schemas must not spread into
|
|
runner, extractor, validator, or LLM framework code.
|
|
|
|
Extraction-domain details belong in domain modules. D&D-specific concepts such
|
|
as spells, NPCs, items, combat turns, and encounters must not spread into core
|
|
source, runner, or LLM framework packages.
|
|
|
|
Extracted facts should be grounded with source references. Source references
|
|
should point to generic source units, not transcript-only structures. Framework
|
|
code should preserve source-reference ranges exactly and should not merge or
|
|
rewrite overlapping ranges unless a module explicitly owns that behavior.
|
|
|
|
The application workflow is fixed:
|
|
|
|
```text
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
```
|
|
|
|
These stages should remain explicit in the architecture. Chunking, merging, and
|
|
normalization must not be hidden inside domain extractors when they represent
|
|
general pipeline behavior.
|
|
|
|
Pipelines are fixed-shape templates for this workflow, not arbitrary DAGs or a
|
|
general workflow language. Module selection should be configuration- and
|
|
registry-driven, not scattered through conditionals.
|
|
|
|
## Package Boundaries
|
|
|
|
Prefer fewer, larger framework packages until a boundary proves itself through
|
|
import direction, ownership, test seams, or substantial file size.
|
|
|
|
Core packages should contain deterministic models and policy. Framework
|
|
packages should contain reusable orchestration and provider plumbing. Concrete
|
|
business logic should live under stage-oriented module packages:
|
|
|
|
```text
|
|
internal/modules/input/...
|
|
internal/modules/chunk/...
|
|
internal/modules/extract/...
|
|
internal/modules/merge/...
|
|
internal/modules/normalize/...
|
|
internal/modules/output/...
|
|
```
|
|
|
|
Use short, lowercase, idiomatic Go package names. Avoid package names that repeat
|
|
parent-stage context.
|
|
|
|
Input modules translate external source formats into the core source model.
|
|
They may know about external schema details, source-specific metadata, and
|
|
format-specific validation rules. They should not own extraction-domain
|
|
decisions.
|
|
|
|
Extract modules own artifact semantics, prompt usage, structured response schema
|
|
selection, validator defaults, and domain-specific interpretation. They should
|
|
depend on framework contracts and core source/artifact types, not concrete input
|
|
module packages.
|
|
|
|
Merge modules combine extracted candidates. Normalize modules reconcile merged
|
|
candidates for semantic consistency. Generic behavior may exist for simple
|
|
artifact types, but domain-specific behavior belongs in modules for the relevant
|
|
stage.
|
|
|
|
Output modules serialize final artifacts and may report warnings out of band.
|
|
CLI, diagnostics, and reporting layers are responsible for surfacing those
|
|
warnings.
|
|
|
|
## Validation
|
|
|
|
Validators should be independently testable and composable.
|
|
|
|
Deterministic validators should run before LLM-backed validators when both are
|
|
present. Validator decision semantics should be explicit: each candidate
|
|
artifact evaluated by a validator should receive exactly one decision from that
|
|
validator.
|
|
|
|
LLM-backed review belongs in module-owned validator chains, not in an implicit
|
|
global review phase. Extract and normalize modules may both use deterministic
|
|
and LLM-backed validators.
|
|
|
|
Shared validator runtime mechanics belong in framework code. Concrete validator
|
|
behavior belongs in module or validator implementation packages.
|
|
|
|
## LLM Runtime
|
|
|
|
LLM provider details belong behind transport-neutral framework contracts.
|
|
|
|
Provider-specific HTTP request and response types should stay inside the LLM
|
|
runtime package. Prompt construction should stay in extractors, validators, or
|
|
shared prompt helpers; provider adapters should not own domain prompt logic.
|
|
|
|
Errors, diagnostics, reports, manifests, and redacted configuration must not
|
|
expose secrets.
|
|
|
|
## Configuration
|
|
|
|
Configuration should make pipeline composition explicit and discoverable.
|
|
|
|
Centralize configuration loading, precedence, defaults, and validation. Structural
|
|
pipeline choices should come from named pipeline definitions, not ad hoc command
|
|
flags. Operational overrides may be handled separately when they do not obscure
|
|
the configured pipeline structure.
|
|
|
|
Module registries should expose module metadata and capabilities without
|
|
requiring module construction. Configuration validation should fail fast when a
|
|
pipeline binds incompatible or unknown modules.
|
|
|
|
Run manifests should record enough resolved pipeline provenance to make a run
|
|
auditable after named configuration changes over time.
|
|
|
|
## Dependencies
|
|
|
|
Prefer the Go standard library where practical.
|
|
|
|
Use external dependencies only when justified by correctness, security,
|
|
interoperability, or substantial complexity reduction. Good reasons include
|
|
widely used file formats, complex validation behavior, or secure transport
|
|
handling.
|
|
|
|
Avoid dependencies for small conveniences. Do not let external dependency types
|
|
leak across internal package boundaries unless the dependency is itself the
|
|
explicit contract of that package.
|
|
|
|
## State, Files, and Safety
|
|
|
|
If the application writes durable state, writes should be atomic where
|
|
practical. Multi-step workflows should preserve enough diagnostics to support
|
|
inspection after failure.
|
|
|
|
Code that deletes, moves, or overwrites files must use narrow, explicit paths.
|
|
Avoid broad parent-directory operations. Cleanup that can cause data loss must
|
|
be opt-in.
|
|
|
|
## Errors and Logging
|
|
|
|
Errors should be actionable and preserve context. Wrap errors with operation and
|
|
path or resource context. CLI code should convert internal errors into concise
|
|
user-facing messages.
|
|
|
|
Errors and logs must not expose secrets. Logs should describe operations,
|
|
external calls, retries, and failure causes, but should not include large source
|
|
or artifact payloads by default.
|
|
|
|
Long-running operations should accept `context.Context`. External calls,
|
|
subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
|
|
workflows should respect cancellation and timeouts.
|
|
|
|
## Testing
|
|
|
|
Core logic should be testable without real external services. Use fakes,
|
|
fixtures, or local test doubles for input modules, extract modules, validators,
|
|
and LLM clients where practical.
|
|
|
|
Contract-first work should include fake implementations that prove interfaces
|
|
compose before real modules depend on them.
|
|
|
|
Maintain a fixture-driven walking skeleton that exercises the full pipeline with
|
|
fake modules and fake external clients. This protects stage composition as real
|
|
modules evolve.
|
|
|
|
Important CLI and configuration workflows should have tests. Adapter, extractor,
|
|
validator, and stage contracts should have focused tests that do not require
|
|
running the full application unless end-to-end coverage is intentional.
|
|
|
|
## Documentation
|
|
|
|
Documentation should follow the project documentation policy. Keep user docs
|
|
focused on implemented behavior. Put future, planned, or aspirational work only
|
|
under `docs/roadmap/`.
|
|
|
|
Core documentation should use generic terms such as source document, source
|
|
unit, source reference, input adapter, extractor, chunker, merger, normalizer,
|
|
artifact, validator, and run manifest.
|
|
|
|
Source-format details belong in input module or integration docs.
|
|
Domain-specific extraction details belong in extract module or artifact docs.
|
|
|
|
When changing architecture, config, CLI behavior, stage modules, extractor
|
|
contracts, validator contracts, LLM runtime behavior, or artifact schemas, update
|
|
the relevant docs and examples in the same change.
|