|
|
|
|
@@ -1,19 +1,22 @@
|
|
|
|
|
# Architecture
|
|
|
|
|
|
|
|
|
|
This document defines the development principles for Notarius. It is
|
|
|
|
|
inward-facing: developers and LLM coding agents should use it to preserve the
|
|
|
|
|
project's shape, boundaries, and invariants as the code evolves.
|
|
|
|
|
This document defines Notarius development policy. It is inward-facing:
|
|
|
|
|
developers and LLM coding agents should use it to preserve the project's shape,
|
|
|
|
|
boundaries, and invariants as the code evolves.
|
|
|
|
|
|
|
|
|
|
Keep this document concise. It should describe durable architectural rules, not
|
|
|
|
|
CLI syntax, configuration reference material, module catalogs, or roadmap items.
|
|
|
|
|
|
|
|
|
|
## Project Shape
|
|
|
|
|
|
|
|
|
|
Notarius is a small, explicit, dependency-light Go application for extracting
|
|
|
|
|
structured artifacts from source material using modular extractors.
|
|
|
|
|
structured artifacts from source material using modular pipeline stages.
|
|
|
|
|
|
|
|
|
|
The application should be contract-first but not abstraction-heavy. Add
|
|
|
|
|
interfaces and extension points when they protect a real boundary:
|
|
|
|
|
The application is contract-first but not abstraction-heavy. Add interfaces and
|
|
|
|
|
extension points when they protect a real boundary:
|
|
|
|
|
|
|
|
|
|
- external source formats;
|
|
|
|
|
- extractor modules;
|
|
|
|
|
- pipeline stage modules;
|
|
|
|
|
- validators;
|
|
|
|
|
- LLM providers and runtime plumbing;
|
|
|
|
|
- output schemas and embedded assets.
|
|
|
|
|
@@ -23,95 +26,43 @@ contracts that can be exercised by tests and real modules.
|
|
|
|
|
|
|
|
|
|
## Core Invariants
|
|
|
|
|
|
|
|
|
|
The core framework must remain source-agnostic and domain-agnostic.
|
|
|
|
|
The framework must remain source-agnostic and domain-agnostic.
|
|
|
|
|
|
|
|
|
|
Source-format details belong in input modules. Transcript-specific concepts
|
|
|
|
|
such as segments, speakers, timestamps, and transcript schemas must not spread
|
|
|
|
|
into runner, extractor, or validator framework code.
|
|
|
|
|
Source-format details belong in input modules. Transcript-specific concepts such
|
|
|
|
|
as segments, speakers, timestamps, and transcript schemas must not spread into
|
|
|
|
|
runner, extractor, validator, or LLM framework code.
|
|
|
|
|
|
|
|
|
|
Extraction-domain details belong in extract modules. D&D-specific concepts
|
|
|
|
|
such as spells, NPCs, items, combat turns, and encounters must not spread into
|
|
|
|
|
core source, runner, or LLM framework packages.
|
|
|
|
|
Extraction-domain details belong in domain modules. D&D-specific concepts such
|
|
|
|
|
as spells, NPCs, items, combat turns, and encounters must not spread into core
|
|
|
|
|
source, runner, or LLM framework packages.
|
|
|
|
|
|
|
|
|
|
Extracted facts should be grounded with source references. Source references
|
|
|
|
|
should point to generic source units, not to transcript-only structures.
|
|
|
|
|
Artifact records should require source references by default unless their schema
|
|
|
|
|
explicitly opts into ungrounded fields. Generic pipeline code should preserve
|
|
|
|
|
source-reference ranges exactly and should not merge or rewrite overlapping
|
|
|
|
|
ranges.
|
|
|
|
|
should point to generic source units, not transcript-only structures. Framework
|
|
|
|
|
code should preserve source-reference ranges exactly and should not merge or
|
|
|
|
|
rewrite overlapping ranges unless a module explicitly owns that behavior.
|
|
|
|
|
|
|
|
|
|
The application workflow is:
|
|
|
|
|
The application workflow is fixed:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
These stages should remain explicit in the architecture. Chunking, merging, and
|
|
|
|
|
normalization must not be hidden inside domain extract modules when they represent
|
|
|
|
|
normalization must not be hidden inside domain extractors when they represent
|
|
|
|
|
general pipeline behavior.
|
|
|
|
|
|
|
|
|
|
## Dependency Policy
|
|
|
|
|
Pipelines are fixed-shape templates for this workflow, not arbitrary DAGs or a
|
|
|
|
|
general workflow language. Module selection should be configuration- and
|
|
|
|
|
registry-driven, not scattered through conditionals.
|
|
|
|
|
|
|
|
|
|
Prefer the Go standard library where practical.
|
|
|
|
|
## Package Boundaries
|
|
|
|
|
|
|
|
|
|
Use external dependencies only when justified by correctness, security,
|
|
|
|
|
interoperability, or substantial complexity reduction. Good reasons include
|
|
|
|
|
widely used file formats, complex validation behavior, or secure transport
|
|
|
|
|
handling.
|
|
|
|
|
Prefer fewer, larger framework packages until a boundary proves itself through
|
|
|
|
|
import direction, ownership, test seams, or substantial file size.
|
|
|
|
|
|
|
|
|
|
Avoid dependencies for small conveniences. Do not let external dependency types
|
|
|
|
|
leak across internal package boundaries unless the dependency is itself the
|
|
|
|
|
explicit public contract of that package.
|
|
|
|
|
|
|
|
|
|
## Package Layout
|
|
|
|
|
|
|
|
|
|
Use this layout unless a change documents a better project-specific reason.
|
|
|
|
|
|
|
|
|
|
CLI and executable entrypoint:
|
|
|
|
|
|
|
|
|
|
- `cmd/notarius`: executable entrypoint.
|
|
|
|
|
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
|
|
|
|
|
|
|
|
|
|
Core deterministic model and policy:
|
|
|
|
|
|
|
|
|
|
- `internal/core/source`: source document, source unit, and source reference types.
|
|
|
|
|
- `internal/core/artifacts`: artifact envelope, artifact candidates, rejected artifacts, and manifests.
|
|
|
|
|
- `internal/core/config`: configuration structs, defaults, loading, precedence, and validation, once config exists.
|
|
|
|
|
|
|
|
|
|
Reusable framework plumbing:
|
|
|
|
|
|
|
|
|
|
- `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts.
|
|
|
|
|
- `internal/framework/pipeline`: runner, pipeline-stage orchestration, registries, and small shared stage helpers.
|
|
|
|
|
- `internal/framework/validate`: shared validator runtime behavior and decision checks.
|
|
|
|
|
- `internal/framework/llm`: LLM clients, scheduling, structured-output parsing, and response-schema registry, once LLM runtime exists.
|
|
|
|
|
- `internal/framework/prompt`: embedded prompt assets, prompt registry, and prompt rendering helpers, once prompt assets exist.
|
|
|
|
|
|
|
|
|
|
Domain implementations:
|
|
|
|
|
|
|
|
|
|
- `internal/modules/input/<name>`: input-stage modules that parse external input into core source documents.
|
|
|
|
|
- `internal/modules/chunk/<name>`: chunk-stage modules.
|
|
|
|
|
- `internal/modules/extract/<domain>/<name>`: extract-stage extractor modules.
|
|
|
|
|
- `internal/modules/merge/<name>` or `internal/modules/merge/<domain>/<name>`: merge-stage modules.
|
|
|
|
|
- `internal/modules/normalize/<name>` or `internal/modules/normalize/<domain>/<name>`: normalize-stage modules.
|
|
|
|
|
- `internal/modules/output/<name>`: output-stage modules.
|
|
|
|
|
- `internal/validators/<validator>`: built-in validator implementations.
|
|
|
|
|
- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
|
|
|
|
|
|
|
|
|
|
Package-private implementation constants may live near the package that owns
|
|
|
|
|
them, preferably in `constants.go` when useful.
|
|
|
|
|
|
|
|
|
|
Start with fewer, larger framework packages. Split a package only when a real
|
|
|
|
|
boundary proves itself through import direction, ownership, test seams, or
|
|
|
|
|
substantial file size. Do not create catalog, diagnostics, reporting,
|
|
|
|
|
structured-output, response-schema, output, merge, normalize, extraction, or
|
|
|
|
|
warnings packages merely because the concepts exist in the architecture.
|
|
|
|
|
|
|
|
|
|
## Stage Modules
|
|
|
|
|
|
|
|
|
|
Concrete business logic should live under `internal/modules/<stage>/...`.
|
|
|
|
|
|
|
|
|
|
Stage-oriented module layout is preferred because it makes the application
|
|
|
|
|
workflow visible in the filesystem:
|
|
|
|
|
Core packages should contain deterministic models and policy. Framework
|
|
|
|
|
packages should contain reusable orchestration and provider plumbing. Concrete
|
|
|
|
|
business logic should live under stage-oriented module packages:
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
internal/modules/input/...
|
|
|
|
|
@@ -122,120 +73,43 @@ internal/modules/normalize/...
|
|
|
|
|
internal/modules/output/...
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Use short, lowercase, idiomatic Go package names. Prefer names such as
|
|
|
|
|
`dndtranscript`, `appendorder`, and `spells` over names like `dnd_transcript`,
|
|
|
|
|
`serial_merge`, or `spell_extractor` that repeat parent-stage context.
|
|
|
|
|
|
|
|
|
|
## Input Modules
|
|
|
|
|
|
|
|
|
|
Use a hexagonal architecture style for source input.
|
|
|
|
|
Use short, lowercase, idiomatic Go package names. Avoid package names that repeat
|
|
|
|
|
parent-stage context.
|
|
|
|
|
|
|
|
|
|
Input modules translate external source formats into the core source model.
|
|
|
|
|
They may know about external schema details, source-specific metadata, and
|
|
|
|
|
format-specific validation rules. They should not own extraction-domain
|
|
|
|
|
decisions.
|
|
|
|
|
|
|
|
|
|
Other packages should interact with source input through adapter contracts and
|
|
|
|
|
core source types. Input module implementation details and external dependency
|
|
|
|
|
types must not leak into framework or extract module packages.
|
|
|
|
|
Extract modules own artifact semantics, prompt usage, structured response schema
|
|
|
|
|
selection, validator defaults, and domain-specific interpretation. They should
|
|
|
|
|
depend on framework contracts and core source/artifact types, not concrete input
|
|
|
|
|
module packages.
|
|
|
|
|
|
|
|
|
|
Input module metadata may preserve source-specific facts such as transcript
|
|
|
|
|
speaker, timestamps, Markdown heading path, page number, or block ID. Framework
|
|
|
|
|
code may carry metadata through, but should not require a specific adapter's
|
|
|
|
|
metadata shape.
|
|
|
|
|
Merge modules combine extracted candidates. Normalize modules reconcile merged
|
|
|
|
|
candidates for semantic consistency. Generic behavior may exist for simple
|
|
|
|
|
artifact types, but domain-specific behavior belongs in modules for the relevant
|
|
|
|
|
stage.
|
|
|
|
|
|
|
|
|
|
Core source metadata should remain `map[string]any`. Document well-known keys
|
|
|
|
|
as conventions, and let input modules expose typed accessor helpers for their
|
|
|
|
|
own metadata when useful.
|
|
|
|
|
Output modules serialize final artifacts and may report warnings out of band.
|
|
|
|
|
CLI, diagnostics, and reporting layers are responsible for surfacing those
|
|
|
|
|
warnings.
|
|
|
|
|
|
|
|
|
|
## Extractors
|
|
|
|
|
|
|
|
|
|
Extractors are independent modules that process source chunks or whole source
|
|
|
|
|
documents and produce one kind of structured artifact candidate.
|
|
|
|
|
Each extract module owns:
|
|
|
|
|
|
|
|
|
|
- its artifact semantics;
|
|
|
|
|
- its prompt usage;
|
|
|
|
|
- its structured response schema selection;
|
|
|
|
|
- its validator chain;
|
|
|
|
|
- any domain-specific mapping or interpretation.
|
|
|
|
|
|
|
|
|
|
Extract modules should depend on framework contracts and core source/artifact
|
|
|
|
|
types. They should not depend on concrete input module packages.
|
|
|
|
|
|
|
|
|
|
Extraction requests should carry the active source chunk plus optional ambient
|
|
|
|
|
context, such as a document synopsis, prior-chunk summaries, known entities, or
|
|
|
|
|
other module-provided state. The context may be empty for simple modules, but
|
|
|
|
|
the contract should not assume extraction is always chunk-local.
|
|
|
|
|
|
|
|
|
|
Extractors should not be the only place where chunking, merging, or
|
|
|
|
|
normalization happens. They may choose processing mode or provide domain-specific
|
|
|
|
|
merge/normalization behavior when generic behavior is insufficient, but the
|
|
|
|
|
pipeline stages themselves are framework concepts.
|
|
|
|
|
|
|
|
|
|
The runner should be able to compose, skip, resume, or run individual extractors
|
|
|
|
|
when their prerequisites are satisfied. Ordering should be explicit through
|
|
|
|
|
configuration, a default sequence, or documented orchestration rules.
|
|
|
|
|
|
|
|
|
|
Extractor selection must go through a registry or equivalent mechanism rather
|
|
|
|
|
than scattered conditionals.
|
|
|
|
|
|
|
|
|
|
## Pipeline Stages
|
|
|
|
|
|
|
|
|
|
The pipeline has six conceptual stages:
|
|
|
|
|
|
|
|
|
|
1. input: external source material becomes a `SourceDocument`;
|
|
|
|
|
2. chunk: a `SourceDocument` becomes ordered source chunks;
|
|
|
|
|
3. extract: extractors produce artifact candidates from chunks or whole documents;
|
|
|
|
|
4. merge: per-chunk candidates become a merged candidate collection;
|
|
|
|
|
5. normalize: merged candidates are reconciled for duplicates, aliases, consistency, or cross-chunk issues;
|
|
|
|
|
6. output: final artifacts are serialized.
|
|
|
|
|
|
|
|
|
|
Chunking is first-class because source documents may be too large for a single
|
|
|
|
|
LLM pass. Chunkers should preserve source-unit order and produce stable chunk
|
|
|
|
|
metadata.
|
|
|
|
|
|
|
|
|
|
Merge and normalize are separate concerns. Merge combines per-chunk results into
|
|
|
|
|
a deterministic collection. Normalize performs semantic reconciliation after
|
|
|
|
|
merge. Generic append-in-chunk-order merge and no-op normalization should be
|
|
|
|
|
available for simple artifact types, while domain-specific behavior can be
|
|
|
|
|
provided where needed.
|
|
|
|
|
|
|
|
|
|
The framework should allow serial and parallel chunk processing. The first
|
|
|
|
|
implementation may execute chunks serially for determinism, but contracts should
|
|
|
|
|
not prevent later parallel execution.
|
|
|
|
|
|
|
|
|
|
Final durable output should use one artifact file per artifact type plus a
|
|
|
|
|
run-level manifest/index file. Framework artifact flow should use a generic
|
|
|
|
|
envelope with `json.RawMessage` payloads; extract modules should own typed Go
|
|
|
|
|
structs at their boundaries and encode into that generic envelope before
|
|
|
|
|
returning to framework code.
|
|
|
|
|
|
|
|
|
|
Output-stage warnings are intentionally out-of-band. An `OutputEncoder` may
|
|
|
|
|
return warnings about serialization, formatting, truncation, or destination
|
|
|
|
|
concerns, but those warnings are appended to the runner result after encoding
|
|
|
|
|
and are not expected to appear inside the encoded artifact bytes. CLI,
|
|
|
|
|
diagnostic, or reporting layers should surface output-stage warnings from the
|
|
|
|
|
runner result.
|
|
|
|
|
|
|
|
|
|
Schemas should be versioned per extractor, with a separate envelope/manifest
|
|
|
|
|
format version.
|
|
|
|
|
|
|
|
|
|
## Validators
|
|
|
|
|
## Validation
|
|
|
|
|
|
|
|
|
|
Validators should be independently testable and composable.
|
|
|
|
|
|
|
|
|
|
Deterministic validators should run before LLM-backed validators when both are
|
|
|
|
|
present. Validator decision semantics should be explicit: each candidate
|
|
|
|
|
artifact should receive exactly one decision from each validator that evaluates
|
|
|
|
|
it.
|
|
|
|
|
artifact evaluated by a validator should receive exactly one decision from that
|
|
|
|
|
validator.
|
|
|
|
|
|
|
|
|
|
LLM-backed review belongs in module-owned validator chains, not in a separate
|
|
|
|
|
global review phase. Extract modules and normalize modules may both use
|
|
|
|
|
deterministic and LLM-backed validators.
|
|
|
|
|
LLM-backed review belongs in module-owned validator chains, not in an implicit
|
|
|
|
|
global review phase. Extract and normalize modules may both use deterministic
|
|
|
|
|
and LLM-backed validators.
|
|
|
|
|
|
|
|
|
|
Shared validator runtime mechanics belong under `internal/framework/validate`.
|
|
|
|
|
Concrete validator behavior belongs under `internal/validators/<validator>`.
|
|
|
|
|
Shared validator runtime mechanics belong in framework code. Concrete validator
|
|
|
|
|
behavior belongs in module or validator implementation packages.
|
|
|
|
|
|
|
|
|
|
## LLM Runtime
|
|
|
|
|
|
|
|
|
|
@@ -243,131 +117,64 @@ LLM provider details belong behind transport-neutral framework contracts.
|
|
|
|
|
|
|
|
|
|
Provider-specific HTTP request and response types should stay inside the LLM
|
|
|
|
|
runtime package. Prompt construction should stay in extractors, validators, or
|
|
|
|
|
shared prompt-context helpers; provider adapters should not own domain prompt
|
|
|
|
|
logic.
|
|
|
|
|
shared prompt helpers; provider adapters should not own domain prompt logic.
|
|
|
|
|
|
|
|
|
|
Errors, diagnostics, reports, and redacted config must not expose secrets.
|
|
|
|
|
Errors, diagnostics, reports, manifests, and redacted configuration must not
|
|
|
|
|
expose secrets.
|
|
|
|
|
|
|
|
|
|
## Configuration
|
|
|
|
|
|
|
|
|
|
Centralize configuration loading, processing, precedence, defaults, and
|
|
|
|
|
validation in `internal/core/config`.
|
|
|
|
|
Configuration should make pipeline composition explicit and discoverable.
|
|
|
|
|
|
|
|
|
|
The goal is to make configuration discoverable and avoid implicit or hidden
|
|
|
|
|
operational values. User-visible defaults and cross-package operational defaults
|
|
|
|
|
should be defined in config code.
|
|
|
|
|
Centralize configuration loading, precedence, defaults, and validation. Structural
|
|
|
|
|
pipeline choices should come from named pipeline definitions, not ad hoc command
|
|
|
|
|
flags. Operational overrides may be handled separately when they do not obscure
|
|
|
|
|
the configured pipeline structure.
|
|
|
|
|
|
|
|
|
|
Configuration should be organized around named pipeline profiles. A pipeline is
|
|
|
|
|
a fixed-shape template for the application workflow, not a free-form DAG or
|
|
|
|
|
general workflow program. The six-stage flow remains fixed:
|
|
|
|
|
Module registries should expose module metadata and capabilities without
|
|
|
|
|
requiring module construction. Configuration validation should fail fast when a
|
|
|
|
|
pipeline binds incompatible or unknown modules.
|
|
|
|
|
|
|
|
|
|
```text
|
|
|
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
|
|
|
```
|
|
|
|
|
Run manifests should record enough resolved pipeline provenance to make a run
|
|
|
|
|
auditable after named configuration changes over time.
|
|
|
|
|
|
|
|
|
|
A pipeline profile should bind registered modules to those stage slots:
|
|
|
|
|
## Dependencies
|
|
|
|
|
|
|
|
|
|
- one shared input module;
|
|
|
|
|
- one shared chunk module by default;
|
|
|
|
|
- one or more artifact lanes, each with extract, merge, normalize, and
|
|
|
|
|
validator behavior;
|
|
|
|
|
- one output module.
|
|
|
|
|
Prefer the Go standard library where practical.
|
|
|
|
|
|
|
|
|
|
The MVP should use one shared chunk module per pipeline. Per-lane chunk
|
|
|
|
|
overrides are a future extension and should be added only if a real artifact
|
|
|
|
|
lane needs different chunking.
|
|
|
|
|
Use external dependencies only when justified by correctness, security,
|
|
|
|
|
interoperability, or substantial complexity reduction. Good reasons include
|
|
|
|
|
widely used file formats, complex validation behavior, or secure transport
|
|
|
|
|
handling.
|
|
|
|
|
|
|
|
|
|
The CLI should select a named pipeline by ID, such as
|
|
|
|
|
`notarius run dnd-session --input session.json`. Structural module selection
|
|
|
|
|
should come from configuration, not ad hoc CLI flags. CLI flags may select a
|
|
|
|
|
subset of configured artifact lanes, such as `--only spells,npcs`, and may
|
|
|
|
|
override operational settings such as model, concurrency, output directory, or
|
|
|
|
|
diagnostics directory.
|
|
|
|
|
|
|
|
|
|
Pipeline definitions should support compact defaults:
|
|
|
|
|
|
|
|
|
|
- `chunk`: `generic`;
|
|
|
|
|
- lane `merge`: `appendorder`;
|
|
|
|
|
- lane `normalize`: `noop`;
|
|
|
|
|
- `output`: `json`;
|
|
|
|
|
- `llm_profile`: `default` where an LLM profile is needed.
|
|
|
|
|
|
|
|
|
|
Module bindings should support both string shorthand and object form. For
|
|
|
|
|
example, `extract: dnd/spells` and
|
|
|
|
|
`extract: {module: dnd/spells, llm_profile: fast}` should normalize to the same
|
|
|
|
|
internal binding type.
|
|
|
|
|
|
|
|
|
|
Module registries should expose module metadata, including flat string
|
|
|
|
|
capabilities, without requiring module construction. Config validation should
|
|
|
|
|
fail fast on unknown module keys, unknown pipeline IDs, missing required slots,
|
|
|
|
|
missing capabilities, unknown LLM profiles, empty artifact-lane sets, or
|
|
|
|
|
`--only` lane names that do not exist in the selected pipeline.
|
|
|
|
|
|
|
|
|
|
Keep capabilities as a flat string set. Do not evolve capabilities into a type
|
|
|
|
|
system unless real module interactions prove the need.
|
|
|
|
|
|
|
|
|
|
Unless documented otherwise, precedence from lowest to highest is:
|
|
|
|
|
|
|
|
|
|
1. built-in defaults
|
|
|
|
|
2. configuration file
|
|
|
|
|
3. environment variables
|
|
|
|
|
4. CLI flags
|
|
|
|
|
|
|
|
|
|
Prefer YAML configuration unless the project has a strong reason to use another
|
|
|
|
|
format. Config files should be discoverable at
|
|
|
|
|
`/usr/local/etc/notarius/config.yml`, with a CLI override via `--config`.
|
|
|
|
|
|
|
|
|
|
Configuration files should not contain raw secrets unless the application is
|
|
|
|
|
explicitly designed for that. Prefer environment variables or secret files for
|
|
|
|
|
secrets.
|
|
|
|
|
|
|
|
|
|
Stage-module-specific configuration should remain inline with the pipeline slot
|
|
|
|
|
that owns it. Do not add named module instances until repeated inline settings
|
|
|
|
|
create real drift or duplication. LLM profiles are the justified top-level
|
|
|
|
|
exception because model settings are cross-cutting.
|
|
|
|
|
|
|
|
|
|
The run manifest should record the selected `pipeline_id` and a digest of the
|
|
|
|
|
resolved pipeline definition after defaults and lane selection are applied.
|
|
|
|
|
`pipeline_id` alone is not sufficient provenance because a named pipeline can
|
|
|
|
|
change over time.
|
|
|
|
|
|
|
|
|
|
## Embedded Assets
|
|
|
|
|
|
|
|
|
|
Store embedded JSON schemas, Markdown prompts, templates, and similar assets as
|
|
|
|
|
separate files, not inline string literals, unless there is a strong reason
|
|
|
|
|
otherwise.
|
|
|
|
|
|
|
|
|
|
Embedded prompts and response schemas should have stable IDs, versions, source
|
|
|
|
|
metadata, and hashes suitable for diagnostics and run manifests.
|
|
|
|
|
|
|
|
|
|
## Errors and Logging
|
|
|
|
|
|
|
|
|
|
Errors should be actionable and preserve context. Wrap errors with operation and
|
|
|
|
|
path/resource context. CLI code should convert internal errors into concise
|
|
|
|
|
user-facing messages.
|
|
|
|
|
|
|
|
|
|
Errors and logs must not expose secrets.
|
|
|
|
|
|
|
|
|
|
Use structured logging where practical. Logs should describe operations, paths,
|
|
|
|
|
external calls, retries, and failure causes, but should not include large source
|
|
|
|
|
or artifact payloads by default.
|
|
|
|
|
|
|
|
|
|
## Context, Timeouts, and Cancellation
|
|
|
|
|
|
|
|
|
|
Long-running operations should accept `context.Context`. External calls,
|
|
|
|
|
subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
|
|
|
|
|
workflows should respect cancellation and timeouts.
|
|
|
|
|
Avoid dependencies for small conveniences. Do not let external dependency types
|
|
|
|
|
leak across internal package boundaries unless the dependency is itself the
|
|
|
|
|
explicit contract of that package.
|
|
|
|
|
|
|
|
|
|
## State, Files, and Safety
|
|
|
|
|
|
|
|
|
|
If the application writes durable state, writes should be atomic where
|
|
|
|
|
practical. Multi-step workflows should preserve enough state to support
|
|
|
|
|
inspection, retry, or resume after failure.
|
|
|
|
|
practical. Multi-step workflows should preserve enough diagnostics to support
|
|
|
|
|
inspection after failure.
|
|
|
|
|
|
|
|
|
|
Code that deletes, moves, or overwrites files must use narrow, explicit paths.
|
|
|
|
|
Avoid broad parent-directory operations. Cleanup that can cause data loss must
|
|
|
|
|
be opt-in.
|
|
|
|
|
|
|
|
|
|
## Errors and Logging
|
|
|
|
|
|
|
|
|
|
Errors should be actionable and preserve context. Wrap errors with operation and
|
|
|
|
|
path or resource context. CLI code should convert internal errors into concise
|
|
|
|
|
user-facing messages.
|
|
|
|
|
|
|
|
|
|
Errors and logs must not expose secrets. Logs should describe operations,
|
|
|
|
|
external calls, retries, and failure causes, but should not include large source
|
|
|
|
|
or artifact payloads by default.
|
|
|
|
|
|
|
|
|
|
Long-running operations should accept `context.Context`. External calls,
|
|
|
|
|
subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
|
|
|
|
|
workflows should respect cancellation and timeouts.
|
|
|
|
|
|
|
|
|
|
## Testing
|
|
|
|
|
|
|
|
|
|
Core logic should be testable without real external services. Use fakes,
|
|
|
|
|
@@ -375,18 +182,15 @@ fixtures, or local test doubles for input modules, extract modules, validators,
|
|
|
|
|
and LLM clients where practical.
|
|
|
|
|
|
|
|
|
|
Contract-first work should include fake implementations that prove interfaces
|
|
|
|
|
compose before real input modules or extract modules depend on them.
|
|
|
|
|
compose before real modules depend on them.
|
|
|
|
|
|
|
|
|
|
Once the pipeline-stage contracts exist, maintain a fixture-driven walking
|
|
|
|
|
skeleton that exercises input, chunk, extract, merge, normalize, and output
|
|
|
|
|
stages with fake modules and fake external clients. This test should protect
|
|
|
|
|
stage composition continuously while real modules are introduced over later
|
|
|
|
|
checkpoints.
|
|
|
|
|
Maintain a fixture-driven walking skeleton that exercises the full pipeline with
|
|
|
|
|
fake modules and fake external clients. This protects stage composition as real
|
|
|
|
|
modules evolve.
|
|
|
|
|
|
|
|
|
|
Config examples should be load-tested once config files exist. Important CLI
|
|
|
|
|
workflows should have parser or command tests. Adapter, extractor, and validator
|
|
|
|
|
contracts should have focused tests that do not require running the full
|
|
|
|
|
application unless end-to-end coverage is intentional.
|
|
|
|
|
Important CLI and configuration workflows should have tests. Adapter, extractor,
|
|
|
|
|
validator, and stage contracts should have focused tests that do not require
|
|
|
|
|
running the full application unless end-to-end coverage is intentional.
|
|
|
|
|
|
|
|
|
|
## Documentation
|
|
|
|
|
|
|
|
|
|
|