notarius/docs/roadmap/initial-architecture.md

# Initial Architecture Roadmap

## Status

This document captures proposed architecture and implementation sequencing for
Notarius. It describes planned work, not implemented behavior.

## Goal

Notarius should extract structured JSON artifacts from primary source inputs
using modular, LLM-backed extractors.

The first MVP should target audio transcripts generated by Seriatim. That
choice should be implemented as an input-stage module, not as a
transcript-specific assumption in the application core. Later input sources,
such as unstructured Markdown notes or Obsidian documents, should be addable
through new input and extract modules without reshaping the framework.

The first extraction domain should be D&D session analysis, starting with spell
casts. That domain should live in extract-stage modules and related schemas, not
in core framework packages.

The application should follow the same broad architecture as Audita:

- deterministic core packages for source documents, artifacts, and configuration once needed;
- input-stage modules that translate external source formats into a small internal source model;
- reusable framework packages for contracts, orchestration, LLM runtime, structured output, and validation;
- independent extract-stage modules that own domain-specific behavior;
- independent validator packages;
- embedded prompt and JSON schema assets;
- CLI orchestration that wires the pieces together without owning domain logic.

The main domain difference from Audita is that Notarius emits extracted
artifacts rather than proposing and applying transcript corrections.

## Architectural Principles

- Keep the core input model generic: ordered text units plus metadata.
- Keep source-format details in hexagonal input modules.
- Keep extraction-domain details in extract modules.
- Treat evidence as source references, not transcript references.
- Prefer narrow, useful abstractions over a universal document model.
- Preserve enough provenance for validation, replay, and downstream inspection.

## Proposed Package Shape

```text
cmd/notarius
internal/cli

internal/core/source
internal/core/artifacts

internal/framework/contracts
internal/framework/pipeline
internal/framework/validate
internal/framework/llm
internal/framework/prompt

internal/modules/input/seriatim
internal/modules/input/markdown

internal/modules/chunk/generic
internal/modules/chunk/dndtranscript

internal/modules/extract/dnd/spells
internal/modules/extract/dnd/items
internal/modules/extract/dnd/npcs
internal/modules/extract/dnd/combat

internal/modules/merge/appendorder
internal/modules/merge/dnd/spells

internal/modules/normalize/noop
internal/modules/normalize/dnd/spells

internal/modules/output/json

internal/validators/source_refs
internal/validators/schema_validity
internal/validators/domain_consistency
internal/validators/llm_review

examples
docs/internal
```

The `markdown` input module and D&D-specific chunk, merge, normalize, and
output modules are listed as likely future packages. The MVP should implement
only the stage modules needed by the checkpoint sequence.

`internal/core/config` should be added when production configuration exists.

The framework package list is intentionally consolidated. `pipeline` should own
runner orchestration, stage registries, and small merge/normalize/output helpers
until those boundaries prove they need separate packages. `llm` should own
structured output and response-schema mechanics until those concerns become too
large or import-heavy. `prompt` should own prompt assets and rendering helpers
once prompt assets exist.

## Core Concepts

### SourceDocument

Canonical internal representation of source material. This should be the object
extractors receive, regardless of whether the original input was a transcript,
Markdown file, note export, or another source type.

```go
type SourceDocument struct {
    ID       string         `json:"id"`
    Kind     string         `json:"kind"`
    Format   string         `json:"format"`
    Digest   string         `json:"digest"`
    Units    []SourceUnit   `json:"units"`
    Metadata map[string]any `json:"metadata,omitempty"`
}

type SourceUnit struct {
    ID       string         `json:"id"`
    Kind     string         `json:"kind"`
    Text     string         `json:"text"`
    Metadata map[string]any `json:"metadata,omitempty"`
}
```

Initial source-unit assumptions:

- units are ordered;
- unit IDs are stable within a source document;
- each unit has extractable text;
- adapter-specific metadata may carry speaker, timestamps, heading paths, page
  numbers, or other source details.

Core source metadata should remain `map[string]any`. Notarius should not define
a universal document model. Instead, the project should document well-known
metadata keys, such as `speaker`, `start`, `end`, and `heading_path`, as
conventions. Input modules may export typed accessor helpers for their own
metadata, such as `seriatim.SpeakerOf(unit)`, without leaking those helpers into
core framework contracts.

### Input Module / Adapter Contract

Hexagonal boundary for external source formats.

```go
type InputAdapter interface {
    Key() string
    Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
}
```

The MVP input module should target Seriatim minimal transcript JSON. Seriatim segment
fields should map as follows:

- `id` becomes `SourceUnit.ID`;
- `text` becomes `SourceUnit.Text`;
- `speaker`, `start`, and `end` become unit metadata;
- Seriatim output metadata becomes document metadata.

The core runner should not know that these units came from transcript segments.

### SourceRef

Grounding reference from an extracted fact back to source units.

```go
type SourceRef struct {
    SourceID    string `json:"source_id"`
    StartUnitID string `json:"start_unit_id"`
    EndUnitID   string `json:"end_unit_id"`
}
```

Initial source-reference validation should require:

- source ID exists for the current run;
- start and end unit IDs exist;
- start is less than or equal to end in document order;
- the referenced range is contiguous within the source document;
- every extracted fact has at least one source reference unless its schema
  explicitly allows ungrounded metadata.

Transcript-oriented output can still present these as transcript segment ranges
when the adapter metadata makes that interpretation available.

Source references should preserve the exact ranges produced by extractors and
validators. Overlapping ranges should not be merged or rewritten by generic
pipeline code. If a domain module wants a derived compact range later, that
should be additional output, not a replacement for the original evidence.

### Extractor

Reusable module contract for producing one artifact type.

```go
type Extractor interface {
    Key() string
    ArtifactType() string
    SchemaVersion() string
    Validators() []Validator
    Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
}
```

An extractor should receive either a whole source document or a source chunk,
depending on processing mode. It should return typed artifact candidates plus
warnings. It should not mutate the source document.

`ExtractionRequest` should be designed now to carry both the active chunk and
optional ambient context, even if the MVP leaves that context empty. Useful
ambient context may include a document synopsis, prior-chunk summaries, known
entities, or other module-provided state. D&D spell extraction can likely work
per chunk, but combat, NPC, and identity-oriented extraction will need broader
context. Adding the field later would force churn across every extractor.

Extract modules own domain concepts. For example, D&D spell extraction should
live under `internal/modules/extract/dnd/spells`; a future to-do extractor for
notes should live under a different extract-module path and use the same
framework contract.

### Chunker

Reusable stage contract for splitting a source document into ordered source
chunks.

Chunking is a first-class pipeline concern because source documents may exceed a
single LLM extraction pass. Chunkers should preserve source-unit order and
produce stable chunk metadata suitable for diagnostics and replay.

### Merger

Reusable stage contract for combining per-chunk artifact candidates into one
merged candidate collection.

Merge should combine outputs without doing semantic reconciliation. A generic
append-in-chunk-order merger should be sufficient for many artifact streams,
including the likely first D&D spell-cast extractor.

### Normalizer

Reusable stage contract for reconciling merged artifact candidates.

Normalize is distinct from merge. Normalizers may deduplicate repeated facts,
resolve aliases, reconcile conflicting fields, check cross-chunk consistency,
or attach normalization warnings.

### Validator

Reusable validation contract for artifact candidates.

Validators should cover:

- JSON/schema validity;
- source-reference validity;
- required-field and shape checks;
- domain consistency;
- optional LLM review for high-risk or ambiguous artifacts.

Validator output should follow Audita's decision-cardinality model: each
candidate artifact receives exactly one decision per validator.

LLM-backed review should be modeled as part of a module's validator chain, not
as a separate global review phase. Extract modules should be able to attach one
or more deterministic or LLM-backed validators. Normalize-stage modules may also
run validator chains, including LLM-backed validators, when semantic
reconciliation needs review.

### Artifact

Final approved JSON output from one or more extractors.

Artifacts should preserve enough metadata to support downstream validation,
debugging, and replay.

The pipeline should carry artifact candidates through a generic envelope with a
`json.RawMessage` payload. Extract modules should own typed Go structs at their
module boundary, then encode those typed records into the generic artifact
candidate envelope before returning to framework code. This keeps stage
contracts simple and avoids generic type plumbing across unrelated artifact
families.

Final durable output should be one file per artifact type plus a run-level
manifest/index file. This supports partial success and lets downstream consumers
read only the artifact types they need. Each artifact file should include its
artifact type, extractor key, extractor schema version, envelope format version,
records, source references, and enough provenance to connect it to the run
manifest.

Every artifact record should require source references unless that artifact
schema explicitly opts into ungrounded fields. Artifact-level metadata, counts,
run information, and other derived summary fields are exempt from the per-record
grounding rule.

Schemas should be versioned per extractor, with a separate envelope/manifest
format version. A single global schema version would couple unrelated extractor
release cadence.

### RunManifest

Per-run provenance record.

```go
type RunManifest struct {
    EnvelopeVersion  string   `json:"envelope_version"`
    PipelineID       string   `json:"pipeline_id"`
    PipelineDigest   string   `json:"pipeline_digest"`
    InputModule      string   `json:"input_module"`
    Chunker          string   `json:"chunker"`
    SourceDigests    []string `json:"source_digests"`
    Extractors       []string `json:"extractors"`
    Merger           string   `json:"merger"`
    Normalizer       string   `json:"normalizer"`
    OutputEncoder    string   `json:"output_encoder"`
    SchemaVersion    string   `json:"schema_version"`
    ValidationStatus string   `json:"validation_status"`
}
```

The manifest should eventually include model names, prompt IDs, prompt hashes,
response schema versions, config source, redacted resolved config digest,
started/completed timestamps, and diagnostics paths.

## Initial Extractor Targets

### D&D Spells

Recommended first vertical slice because it is narrow but representative.

```go
type SpellCast struct {
    Player               string      `json:"player"`
    Spell                string      `json:"spell"`
    Effect               string      `json:"effect"`
    NarrativeDescription string      `json:"narrative_description"`
    SourceRefs           []SourceRef `json:"source_refs"`
}
```

The spell extractor should be D&D-specific. The framework should not know what a
spell is.

### D&D Items

Tracks items gained, lost, transferred, consumed, or transformed.

Open questions:

- Should currency be represented as items or as its own artifact type?
- Should item ownership be a required field?
- How should ambiguous ownership changes be represented?

### D&D NPCs

Tracks NPCs interacted with, newly introduced, renamed, described, or otherwise
made relevant to campaign state.

Open questions:

- Should NPC identity resolution happen inside this extractor or in a later
  deduplication stage?
- Should location/faction/relationship facts be separate artifact types?

### D&D Combat

Likely warrants a dedicated schema rather than a generic event list.

Proposed first shape:

```go
type CombatTurn struct {
    Actor                string      `json:"actor"`
    Action               string      `json:"action"`
    Outcome              string      `json:"outcome"`
    NarrativeDescription string      `json:"narrative_description"`
    SourceRefs           []SourceRef `json:"source_refs"`
}
```

Open questions:

- Should combat be extracted as turns, rounds, encounters, or all three?
- Should mechanical fields such as damage, conditions, saves, attacks, and spell
  slots be normalized immediately or added later?
- How should uncertain initiative order be represented?

### Future Non-D&D Extractors

The architecture should support extractors outside the D&D domain. Examples:

- to-do items from Markdown or Obsidian notes;
- decisions and action items from meeting transcripts;
- named people, places, and dates from research notes.

These should be addable as extract modules without changing runner,
validator, source-reference, or LLM framework contracts.

## Configuration Model

Notarius should use named pipeline profiles selected by ID at the CLI. A
pipeline is a fixed-shape template for the known application workflow, not a
free-form list of steps:

```text
input -> chunk -> extract -> merge -> normalize -> output
```

A pipeline profile should define one shared front end and one or more artifact
lanes:

- shared input module;
- shared chunk module by default;
- artifact lanes containing extract, merge, normalize, and validator behavior;
- shared output module.

The MVP should use one shared chunk module per pipeline. Per-lane chunk
overrides can be added later if an artifact lane, such as combat, proves it
needs a different chunking strategy.

Example shape:

```yaml
llm_profiles:
  default:
    model: example-model
    max_concurrency: 4

pipelines:
  dnd-session:
    input: seriatim
    chunk: dnd/transcript
    artifacts:
      spells:
        extract: dnd/spells
        normalize: dnd/spells
      npcs:
        extract: dnd/npcs
      items:
        extract: dnd/items
```

The CLI should run named pipelines:

```sh
notarius run dnd-session --input session-014.json
notarius run dnd-session --input session-014.json --only spells,npcs
```

`--only` should select configured artifact lanes. It should not create an
ad hoc pipeline. Structural module selection should come from config, while CLI
flags may override operational knobs such as model, concurrency, output
directory, and diagnostics directory.

Initial defaults:

- `chunk`: `generic`;
- lane `merge`: `appendorder`;
- lane `normalize`: `noop`;
- `output`: `json`;
- `llm_profile`: `default` where an LLM profile is needed.

Module bindings should support both string shorthand and object form:

```yaml
extract: dnd/spells
```

```yaml
extract:
  module: dnd/spells
  llm_profile: fast
  prompt_version: v1
```

Both forms should normalize into a single internal `ModuleBinding` shape before
validation and manifest hashing.

Pipeline validation should use module metadata declared through registries.
Modules should expose flat string capability metadata, such as `speaker` or
`timestamps`, without requiring module construction. Config validation should
fail fast for:

- unknown pipeline IDs;
- unknown module keys;
- missing required slots;
- missing required capabilities;
- unknown LLM profiles;
- empty artifact-lane sets;
- `--only` lane names that do not exist in the selected pipeline.

The MVP should keep pipelines config-file-only. Built-in pipeline profiles can
be added later if the project needs embedded defaults, but that introduces
merge/override semantics that the MVP does not need.

The resolved pipeline definition should be hashed after defaults and lane
selection are applied. The run manifest should record both `pipeline_id` and
`pipeline_digest`; a pipeline ID alone is not stable provenance.

## Proposed Pipeline Flow

The application workflow should be first-class:

```text
input -> chunk -> extract -> merge -> normalize -> output
```

Proposed runner flow:

1. Load effective config.
2. Resolve the selected pipeline profile by ID.
3. Apply defaults and `--only` lane selection.
4. Validate module keys, lane definitions, LLM profiles, and capabilities.
5. Hash the resolved pipeline definition.
6. Create diagnostics run directory.
7. Resolve the configured input module through the input adapter registry.
8. Read source input.
9. Parse source input into a `SourceDocument`.
10. Validate source-document invariants.
11. Resolve the configured chunker.
12. Chunk source units into deterministic source chunks.
13. Resolve configured artifact lanes through registries.
14. Extract, merge, normalize, and validate each selected artifact lane.
15. Retain approved artifacts and rejected-artifact diagnostics.
16. Serialize output files and run-level manifest/index.
17. Write diagnostics and optional report JSON.

The runner should operate on source documents and source chunks only. Any
transcript-specific behavior should happen before the runner, inside the input
adapter, or after the runner, inside output rendering that understands source
metadata.

## Audita Patterns To Reuse

Reuse these architectural patterns:

- deterministic parsing and schema validation style;
- deterministic chunking of ordered source units;
- explicit extractor registry;
- explicit pipeline stage contracts;
- `contracts` package for transport-neutral interfaces;
- OpenAI-compatible structured LLM client;
- scheduler for bounded LLM concurrency;
- embedded prompt registry with prompt metadata and hashes;
- embedded response-schema registry with schema metadata and hashes;
- diagnostics run directory with redacted effective config;
- validator decision cardinality and deterministic validator ordering;
- CLI tests and fixture-driven integration tests.

The fixture-driven integration-test pattern should begin at checkpoint 3 with a
walking skeleton over fake modules and a fake LLM client. Later checkpoints
should replace fake pieces with real Seriatim, runtime, and D&D modules without
losing that end-to-end contract coverage.

Avoid copying these Audita concepts directly:

- transcript-specific core types;
- correction proposals;
- replacement policies;
- deterministic transcript mutation;
- correction ledger terminology.

Those concepts are specific to Audita's transcript-editing role and should be
replaced with source-document, artifact-candidate, artifact-validation, and
extraction-report concepts.

## Checkpoint Roadmap

The initial implementation should proceed through six coherent checkpoints.
Each checkpoint should leave the repository in a reviewable state, with the code
compiling and targeted tests covering the newly introduced contracts or behavior.

1. [Core Contracts And Skeleton](1-core-contracts-and-skeleton.md)
2. [Framework Composition](2-framework-composition.md)
3. [Pipeline Stages, Chunking, Merge, And Normalize](3-pipeline-stages-chunking-merge-normalize.md)
4. [Portable Audita Infrastructure](4-portable-audita-infrastructure.md)
5. [Seriatim Input Module](5-seriatim-input-module.md)
6. [D&D Spells Extractor](6-dnd-spells-extractor.md)

The first contract-level walking skeleton should arrive at checkpoint 3: fixture
input through fake input, chunk, extract, merge, normalize, and output modules
with a fake LLM client. The first useful vertical slice should arrive at
checkpoint 6: Seriatim transcript input to validated D&D spell artifact output.
Earlier checkpoints remain contract-first and may not produce useful user output
yet.

## Architecture Decisions

- Final durable output should use one artifact file per artifact type plus a
  run-level manifest/index file.
- Framework artifact flow should use a generic envelope with `json.RawMessage`
  payloads. Extract modules should use typed Go structs at their own boundaries.
- Schemas should be versioned per extractor, with a separate envelope/manifest
  format version.
- Artifact records should require source references by default. Individual
  schemas may explicitly opt into ungrounded fields. Artifact-level metadata is
  exempt.
- Source-reference ranges should be preserved exactly. Generic pipeline code
  should not merge or rewrite overlapping ranges.
- `ExtractionRequest` should carry the active chunk plus optional ambient
  context for document synopsis, prior-chunk summaries, known entities, or
  similar module-provided state.
- LLM-backed review should be part of module-owned validator chains. Extract
  modules and normalize modules may both use deterministic and LLM-backed
  validators.
- The Seriatim MVP should support only the minimal Seriatim schema. Broader
  Seriatim schema support should be added later without changing core source
  contracts.
- Core source metadata should remain `map[string]any`. Well-known metadata keys
  should be documented as conventions, and input modules may expose typed
  accessor helpers for their own metadata.
- Configuration should use named pipeline profiles selected by ID at the CLI.
- A pipeline profile should be a fixed template, not a free-form DAG: shared
  input and chunk stages, one or more artifact lanes, and shared output.
- `--only` should select configured artifact lanes without creating ad hoc
  pipelines.
- Module bindings should support string shorthand and inline object settings,
  normalized into one internal binding shape.
- Registries should expose flat capability metadata so config can fail fast on
  invalid module combinations.
- The run manifest should record both `pipeline_id` and a digest of the resolved
  pipeline definition after defaults and lane selection.

## Open Design Questions

- Which artifact types should use generic append-in-chunk-order merge, and which
  should use domain-specific merge?
- Which artifact types need domain-specific normalization for deduplication,
  identity resolution, or consistency?
- Which operational settings should be allowed as CLI/environment overrides
  without weakening pipeline provenance?

## Near-Term Documentation Tasks

Once behavior is implemented, move implemented contracts out of roadmap docs and
into canonical docs:

- `README.md` for purpose and shortest useful command;
- `docs/cli.md` for CLI behavior;
- `docs/config.md` for config fields and precedence;
- `docs/internal/` for implemented architecture and package boundaries;
- `docs/integrations/` for source input and artifact file formats;
- `examples/` for maintained source, config, and artifact examples.