Prepare roadmap for documentation policy update
This commit is contained in:
355
docs/roadmap/documentation.md
Normal file
355
docs/roadmap/documentation.md
Normal file
@@ -0,0 +1,355 @@
|
||||
# Documentation Policy And Structure Alignment
|
||||
|
||||
## Status
|
||||
|
||||
Proposed.
|
||||
|
||||
## Purpose
|
||||
|
||||
Align Narratio's documentation policy, organization, and maintenance practices
|
||||
with the current Notarius approach while preserving Narratio-specific product,
|
||||
operational, and integration needs.
|
||||
|
||||
The resulting documentation set should assign each authoritative topic to one
|
||||
canonical owner, distinguish current behavior from future work, make audience
|
||||
boundaries explicit, and minimize duplicated contracts that can drift as the
|
||||
application changes.
|
||||
|
||||
This roadmap defines the scope and intended final state. Execution guidance is
|
||||
maintained separately in the [Implementation Plan](implementation.md).
|
||||
|
||||
## Motivation
|
||||
|
||||
Narratio already has substantial user, operator, integration, and internal
|
||||
documentation, but its documentation policy predates the current Notarius
|
||||
policy. The existing policy prescribes document profiles and detailed outlines
|
||||
without defining ownership boundaries precisely enough. Several policy files
|
||||
also refer to obsolete or nonexistent paths, and some concrete inventories and
|
||||
contract details are repeated across documents.
|
||||
|
||||
The alignment should retain useful Narratio documentation while adopting the
|
||||
stronger Notarius principles:
|
||||
|
||||
- one canonical owner for each authoritative fact;
|
||||
- current behavior outside the roadmap and future behavior inside it;
|
||||
- audience-appropriate detail;
|
||||
- maintained, valid, secret-free examples;
|
||||
- explicit boundaries among user, operator, integration, architecture, and
|
||||
internal documentation;
|
||||
- a defined lifecycle for architectural decision records; and
|
||||
- verification of documentation against implemented behavior.
|
||||
|
||||
## Target Documentation Model
|
||||
|
||||
### Canonical Ownership
|
||||
|
||||
The final documentation policy should assign the following responsibilities.
|
||||
|
||||
| Topic | Canonical owner | Intended responsibility |
|
||||
| --- | --- | --- |
|
||||
| Product orientation and minimal end-to-end quickstart | `README.md` | Explain what Narratio is, why it is useful, show the shortest successful invocation, and route readers onward. |
|
||||
| Contributor entry point | `docs/development.md` | Provide a task-oriented reading guide, baseline validation commands, and links to canonical policies and contracts. |
|
||||
| Current application architecture | `docs/policy/architecture.md` | Define system shape, normative ownership, dependency direction, boundaries, invariants, safety properties, and non-goals. |
|
||||
| Documentation organization | `docs/policy/documentation.md` | Define canonical ownership, audience boundaries, maintenance rules, and the ADR and document lifecycle. |
|
||||
| Testing policy | `docs/policy/testing.md` | Define test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, and test lifecycle decisions. |
|
||||
| CLI contract | `docs/cli.md` | Define commands, arguments, flags, invocation semantics, output conventions, and exit behavior. |
|
||||
| Configuration contract | `docs/config.md` | Define discovery, precedence, schemas, fields, defaults, environment overrides, and validation rules. |
|
||||
| Operations | `docs/operations.md` | Define runtime workflows, physical state layout, resume, cleanup, permissions, recovery, and operational limits. |
|
||||
| Troubleshooting | `docs/troubleshooting.md` | Provide symptom-driven diagnosis and safe remedies, linking to the owning CLI, configuration, operations, or integration contract. |
|
||||
| External and durable integration contracts | `docs/integrations/` | Define the external formats, protocols, logical artifact shapes, and compatibility behavior Narratio relies on. |
|
||||
| Implemented component inventory | `docs/internal/overview.md` | Inventory current packages and components, summarize their implemented responsibilities, and route readers to focused internal documents. |
|
||||
| Internal component behavior | Other files under `docs/internal/` | Explain implementation flow, internal collaborators, state transitions, package-local guarantees and failures, and relevant tests. |
|
||||
| Architectural decision history | `docs/adr/` | Record significant decisions, alternatives, rationale, consequences, and supersession history when ADRs are warranted. |
|
||||
| Future work and implementation status | `docs/roadmap/` | Describe proposed, accepted, deferred, rejected, or completed work and its implementation status. |
|
||||
| Complete copyable artifacts | `examples/` | Hold maintained configuration, inputs, and other files intended to be copied or run. |
|
||||
|
||||
Documents for interfaces that Narratio does not expose should not be created as
|
||||
placeholders. In particular, public API or consumer documentation should be
|
||||
added only if a corresponding public interface exists.
|
||||
|
||||
### Structural Parity With Notarius
|
||||
|
||||
Narratio should follow the current Notarius policy layout where the same
|
||||
responsibility exists. In the aligned structure:
|
||||
|
||||
- `docs/internal/overview.md` is the internal component inventory;
|
||||
- `docs/development.md` is the first-read contributor landing page;
|
||||
- all navigation and relative links use those canonical paths;
|
||||
- `docs/adr/` and sequentially numbered ADR filenames are used when architectural
|
||||
decisions need durable records; and
|
||||
- Narratio-specific canonical documents remain, including the troubleshooting
|
||||
guide, stage documentation, and integration contracts.
|
||||
|
||||
Structural parity does not require placeholder documents, removal of useful
|
||||
Narratio-specific material, or identical prose where the applications have
|
||||
different contracts.
|
||||
|
||||
## Policy Alignment
|
||||
|
||||
### Documentation Policy
|
||||
|
||||
The documentation policy should follow the current Notarius policy, adapted to
|
||||
Narratio. It should:
|
||||
|
||||
- define one canonical owner for every contract or authoritative fact;
|
||||
- identify volatile details that must not be maintained in multiple places;
|
||||
- permit non-owning documents to provide only short, stable summaries with
|
||||
links;
|
||||
- describe implemented behavior only outside `docs/roadmap/`;
|
||||
- allow accepted ADRs to precede implementation without presenting the
|
||||
decision as implemented behavior;
|
||||
- distinguish user, operator, contributor, integration, and internal detail;
|
||||
- require complete copyable files to live under `examples/`;
|
||||
- define documentation security and privacy requirements;
|
||||
- specify ownership boundaries for orientation, CLI, configuration,
|
||||
operations, troubleshooting, integrations, architecture, and internals;
|
||||
- define the ADR format and supersession rules; and
|
||||
- require contract, example, link, and sensitive-data checks when behavior or
|
||||
documentation changes.
|
||||
|
||||
### Architecture Policy
|
||||
|
||||
The architecture policy should remain the normative owner of Narratio's system
|
||||
boundaries and invariants rather than a concrete package inventory or secondary
|
||||
testing and documentation policy. It should:
|
||||
|
||||
- preserve Narratio's explicit, stage-driven orchestration model;
|
||||
- preserve adapter, manifest, artifact, path-safety, publish-commit, security,
|
||||
privacy, and determinism invariants;
|
||||
- link to the testing and documentation policies for their general rules;
|
||||
- link to internal documentation for implemented component mechanics;
|
||||
- link to external contracts rather than redefining them; and
|
||||
- use the canonical `docs/policy/` paths consistently.
|
||||
|
||||
### Testing Policy
|
||||
|
||||
The copied Notarius testing policy is Narratio's canonical testing policy. Its
|
||||
place in the documentation set requires:
|
||||
|
||||
- linking it from architecture and contributor guidance;
|
||||
- removing or reducing duplicated general test philosophy elsewhere;
|
||||
- retaining subsystem-specific test guidance only where it helps maintain a
|
||||
concrete contract; and
|
||||
- ensuring project commands and examples remain accurate for Narratio.
|
||||
|
||||
The alignment should not require Narratio and Notarius to have identical test
|
||||
suites. The shared policy governs how Narratio evaluates test value,
|
||||
boundaries, doubles, regression protection, coverage, and sufficiency.
|
||||
|
||||
### Contributor Policy
|
||||
|
||||
The development document should be a concise contributor entry point. It should
|
||||
route maintainers and coding agents to canonical documentation based on the
|
||||
task at hand and provide the minimum repository orientation and validation
|
||||
commands needed to begin work.
|
||||
|
||||
Detailed package inventories belong in `docs/internal/overview.md`;
|
||||
architecture rules belong in the architecture policy; application contracts
|
||||
belong in their user, operator, or integration documents; and general testing
|
||||
rules belong in the testing policy. Any retained change recipes should avoid
|
||||
redefining those owners.
|
||||
|
||||
## Documentation Set Alignment
|
||||
|
||||
### README And Navigation
|
||||
|
||||
Keep the README short and outward-facing. It should own product orientation and
|
||||
one minimal successful workflow, then link to the CLI, configuration,
|
||||
operations, troubleshooting, contributor, architecture, testing, integration,
|
||||
and internal entry points as appropriate.
|
||||
|
||||
Navigation should use the final canonical paths and should not rely on obsolete
|
||||
aliases or duplicate index files.
|
||||
|
||||
### CLI, Configuration, And Operations
|
||||
|
||||
Separate these contracts consistently:
|
||||
|
||||
- CLI documentation answers how Narratio is invoked and what its command-line
|
||||
interface means;
|
||||
- configuration documentation answers how configuration is discovered,
|
||||
interpreted, defaulted, overridden, and validated; and
|
||||
- operations documentation answers what happens to runtime state and how an
|
||||
operator runs, resumes, cleans, diagnoses, or recovers the application.
|
||||
|
||||
Cross-cutting workflows should have one task-oriented owner and link to the
|
||||
other contracts rather than copying their flags, fields, defaults, or path
|
||||
definitions.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
Keep troubleshooting as a Narratio-specific canonical document. Each entry
|
||||
should begin from an observable symptom and provide a likely cause, a safe
|
||||
diagnostic step, a safe remedy, and links to the canonical contract or
|
||||
operational procedure.
|
||||
|
||||
Troubleshooting should not become a second CLI, configuration, or operations
|
||||
reference.
|
||||
|
||||
### Integrations
|
||||
|
||||
Integration documents should own the externally observable contracts Narratio
|
||||
uses: subprocess behavior, file formats, protocols, logical artifact paths and
|
||||
schemas, compatibility expectations, and upstream or downstream
|
||||
responsibilities.
|
||||
|
||||
They should describe only the portions of WhisperX, Seriatim, Audita,
|
||||
Scriptorium, object storage, or future integrations that Narratio actually
|
||||
depends on. Internal adapter mechanics belong under `docs/internal/`, while
|
||||
configuration defaults, CLI syntax, and physical runtime placement remain with
|
||||
their respective owners.
|
||||
|
||||
### Internal Documentation
|
||||
|
||||
`docs/internal/overview.md` is the canonical implemented component inventory. It
|
||||
should summarize current components and route readers to focused documents
|
||||
without restating normative architecture.
|
||||
|
||||
Focused internal documents should describe implemented behavior at useful
|
||||
component boundaries, including collaborators, data and state transitions,
|
||||
failures, and tests worth consulting. They may identify an external field,
|
||||
file, or protocol when explaining a dependency, but should link to the
|
||||
canonical contract for its definition.
|
||||
|
||||
The internal set should be reviewed for duplicate definitions of:
|
||||
|
||||
- stage order and stage contracts;
|
||||
- manifest states and transitions;
|
||||
- artifact identities, paths, and schemas;
|
||||
- workspace and object-storage layout;
|
||||
- restore, resume, cleanup, and publish behavior;
|
||||
- adapter invocation and compatibility rules; and
|
||||
- configuration fields and defaults.
|
||||
|
||||
### Examples
|
||||
|
||||
Complete copyable configuration and input files should remain under
|
||||
`examples/`. Documentation may include small illustrative fragments, but it
|
||||
should link to maintained examples instead of embedding alternate complete
|
||||
files.
|
||||
|
||||
Examples should remain secret-free, loadable, valid, and covered by automated
|
||||
tests where practical. Their commands, filenames, fields, defaults, and
|
||||
templates must agree with implemented Narratio behavior.
|
||||
|
||||
### Roadmaps And ADRs
|
||||
|
||||
Roadmaps should own future work, implementation status, and sequencing. They
|
||||
must not be treated as current behavior references. When roadmap work lands,
|
||||
the relevant current-behavior documents should be updated in the same change,
|
||||
and the roadmap status should accurately reflect completion.
|
||||
|
||||
Use ADRs for significant architectural decisions whose context, alternatives,
|
||||
and consequences should remain durable. Use the lightweight Nygard structure:
|
||||
title, status, date, context, decision, alternatives considered, and
|
||||
consequences. Accepted decision content is immutable; a later change should
|
||||
supersede it with a new ADR. Rejected architectural alternatives belong in the
|
||||
ADR, while rejected product ideas remain roadmap material.
|
||||
|
||||
No empty `docs/adr/` directory or retrospective ADR catalog is required merely
|
||||
to satisfy structural parity.
|
||||
|
||||
## Canonical-Ownership Audit
|
||||
|
||||
The existing documentation should be audited for repeated authoritative facts,
|
||||
with priority given to details most likely to drift:
|
||||
|
||||
- commands, arguments, flags, output conventions, and exit behavior;
|
||||
- configuration discovery, fields, defaults, environment variables, and
|
||||
validation rules;
|
||||
- stage order, prerequisites, invalidation, skip, force, resume, and failure
|
||||
semantics;
|
||||
- manifest states, artifact identities, and publish commit behavior;
|
||||
- local workspace, cache, spool, log, report, and remote object paths;
|
||||
- integration arguments, formats, schemas, timeouts, and compatibility rules;
|
||||
and
|
||||
- security, credential, permission, and sensitive-artifact handling.
|
||||
|
||||
For each repeated fact, choose the canonical owner defined by policy. Remove
|
||||
the duplicate definition or reduce it to the smallest stable summary needed
|
||||
for orientation, with a link to the owner.
|
||||
|
||||
This audit should preserve useful task-oriented guidance. Canonical ownership
|
||||
means eliminating parallel contract definitions, not forcing readers to
|
||||
assemble every workflow from isolated reference fragments.
|
||||
|
||||
## Intended Final State
|
||||
|
||||
When this roadmap is complete:
|
||||
|
||||
- Narratio's documentation policy closely matches the current Notarius policy
|
||||
in principles, organization, terminology, and lifecycle rules;
|
||||
- every authoritative documentation topic has one stated canonical owner;
|
||||
- `docs/internal/overview.md` is the internal component index and no references
|
||||
to `docs/internal/README.md` remain;
|
||||
- all policy and navigation links use real canonical paths;
|
||||
- architecture, documentation, testing, and contributor policies have distinct
|
||||
responsibilities and link to one another;
|
||||
- current-behavior documentation contains no unimplemented claims;
|
||||
- roadmap documents clearly own future behavior and implementation status;
|
||||
- user and operator documents avoid unnecessary implementation detail;
|
||||
- internal documents do not redefine external or user-facing contracts;
|
||||
- volatile commands, fields, defaults, schemas, paths, and guarantees are not
|
||||
maintained authoritatively in multiple places;
|
||||
- complete examples live under `examples/` and remain valid and secret-free;
|
||||
- the documentation remains concise enough to navigate but complete enough for
|
||||
users, operators, developers, integrators, and coding agents; and
|
||||
- future behavior changes can identify the documentation that must change by
|
||||
consulting the ownership table.
|
||||
|
||||
## Validation
|
||||
|
||||
Completion should include:
|
||||
|
||||
- a repository-wide review of Markdown links and canonical paths;
|
||||
- verification of documented commands and flags against the CLI
|
||||
implementation;
|
||||
- verification of configuration fields, defaults, environment overrides, and
|
||||
validation rules against the configuration implementation;
|
||||
- verification of stage, manifest, artifact, restore, resume, cleanup, and
|
||||
publish claims against implemented behavior;
|
||||
- validation of maintained examples through existing runtime config paths and
|
||||
tests;
|
||||
- `go test ./...`;
|
||||
- confirmation that non-owning documents summarize and link rather than
|
||||
redefine volatile contracts;
|
||||
- confirmation that unimplemented behavior appears only under
|
||||
`docs/roadmap/`, subject to the accepted-ADR exception; and
|
||||
- a review for credentials, private campaign content, sensitive environment
|
||||
data, and private infrastructure details.
|
||||
|
||||
If the repository has no automated Markdown link checker, validation should use
|
||||
a focused, reproducible scripted link check supplemented by manual review of
|
||||
directory links and anchors.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `docs/policy/documentation.md` expresses the Narratio-adapted canonical
|
||||
ownership model and boundary rules from the current Notarius policy.
|
||||
- `docs/policy/architecture.md`, `docs/policy/testing.md`, and
|
||||
`docs/development.md` have clear, non-overlapping responsibilities.
|
||||
- `docs/policy/development.md` has moved to `docs/development.md`, with all
|
||||
inbound links updated.
|
||||
- `docs/internal/README.md` has been renamed to
|
||||
`docs/internal/overview.md`, with all inbound links updated.
|
||||
- README, CLI, configuration, operations, troubleshooting, integration,
|
||||
internal, roadmap, example, and ADR responsibilities match the ownership
|
||||
model.
|
||||
- No known obsolete documentation paths remain.
|
||||
- High-volatility contracts have one authoritative definition.
|
||||
- Current and future behavior are clearly separated.
|
||||
- Maintained examples and documented commands agree with the implementation.
|
||||
- Documentation validation and the full Go test suite pass, or any unrelated
|
||||
pre-existing failure is recorded precisely.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing Narratio runtime behavior solely to make existing documentation
|
||||
true.
|
||||
- Implementing features described by other roadmap documents.
|
||||
- Rewriting every document for stylistic uniformity when its ownership and
|
||||
content are already correct.
|
||||
- Copying Notarius product, CLI, configuration, integration, or internal
|
||||
contracts into Narratio.
|
||||
- Creating placeholder API, consumer, ADR, or integration documents for
|
||||
interfaces that do not exist.
|
||||
544
docs/roadmap/implementation.md
Normal file
544
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,544 @@
|
||||
# Documentation Policy And Structure Implementation Plan
|
||||
|
||||
## Status
|
||||
|
||||
Proposed.
|
||||
|
||||
## Objective
|
||||
|
||||
Implement the target documentation model defined in
|
||||
[Documentation Policy And Structure Alignment](documentation.md). This is a
|
||||
documentation-only effort: do not change Go code, runtime behavior,
|
||||
configuration schemas, generated assets, or application features while
|
||||
following this plan.
|
||||
|
||||
Complete the stages below in order. Treat each stage as an independently
|
||||
reviewable change boundary and run its stated checks before proceeding. Preserve
|
||||
the technical intent of the feature roadmap and the existing current-behavior
|
||||
documentation unless repository evidence shows that it is stale.
|
||||
|
||||
## Governing Decisions
|
||||
|
||||
- Use `../notarius/docs/policy/documentation.md` as the structural and policy
|
||||
baseline for Narratio's documentation policy, adapting project names,
|
||||
canonical paths, and Narratio-specific document types.
|
||||
- Keep architecture, documentation, and testing under `docs/policy/`.
|
||||
- Move contributor orientation from `docs/policy/development.md` to
|
||||
`docs/development.md`.
|
||||
- Rename `docs/internal/README.md` to `docs/internal/overview.md`.
|
||||
- Replace the root `AGENTS.md` with the exact two-line content specified in
|
||||
Stage 4.
|
||||
- Keep `docs/troubleshooting.md` as a Narratio-specific canonical owner even
|
||||
though the current Notarius ownership table has no corresponding row.
|
||||
- Add `docs/integrations/whisperx.md`; WhisperX is an implemented external HTTP
|
||||
boundary and is the only current first-class integration missing a focused
|
||||
integration contract.
|
||||
- Keep directory index filenames not otherwise selected by this plan unchanged.
|
||||
In particular, do not rename `docs/integrations/README.md`.
|
||||
- Do not create `docs/adr/` until Narratio has an ADR to record. Do not create
|
||||
placeholder API or consumer documentation.
|
||||
- Do not copy Notarius product or runtime contracts. Use Notarius only as the
|
||||
policy and information-architecture model.
|
||||
- Outside `docs/roadmap/`, describe implemented behavior only. Roadmaps may
|
||||
describe proposed behavior but must not be cited as current contracts.
|
||||
- When documentation and implementation disagree, inspect the implementation
|
||||
and tests and make current-behavior documentation accurate. Do not alter code
|
||||
in this effort. Record a genuine runtime defect or unresolved ambiguity for
|
||||
separate work rather than silently documenting desired behavior as current.
|
||||
- Do not add a documentation tool dependency solely for this migration. Use
|
||||
repository scripts only if they already exist; otherwise use the validation
|
||||
procedure in Stage 8.
|
||||
|
||||
## Stage 1: Replace The Documentation Policy
|
||||
|
||||
### Goal
|
||||
|
||||
Make `docs/policy/documentation.md` the authoritative Narratio documentation
|
||||
ownership and lifecycle policy before changing the rest of the documentation.
|
||||
|
||||
### Work
|
||||
|
||||
1. Read the complete current files:
|
||||
- `../notarius/docs/policy/documentation.md`;
|
||||
- `docs/roadmap/documentation.md`;
|
||||
- `docs/policy/documentation.md`; and
|
||||
- the current Narratio documentation tree.
|
||||
2. Replace `docs/policy/documentation.md` with a Narratio-adapted version of the
|
||||
current Notarius policy. Preserve the Notarius section model:
|
||||
- Purpose;
|
||||
- Core Rules;
|
||||
- Canonical Ownership;
|
||||
- Boundary Rules;
|
||||
- Architecture Decision Records; and
|
||||
- Maintenance.
|
||||
3. Make these Narratio-specific adaptations:
|
||||
- use Narratio rather than Notarius throughout;
|
||||
- set the contributor owner to `docs/development.md`;
|
||||
- set architecture, documentation, and testing owners to their files under
|
||||
`docs/policy/`;
|
||||
- set the implemented component inventory to
|
||||
`docs/internal/overview.md`;
|
||||
- retain the Notarius conditional rows for public HTTP and consumer
|
||||
documentation;
|
||||
- add a troubleshooting row assigning symptom-driven diagnosis and safe
|
||||
remedies to `docs/troubleshooting.md` while keeping commands,
|
||||
configuration, and operational procedures with their existing owners;
|
||||
- retain `docs/integrations/` as the owner of external and durable contracts;
|
||||
and
|
||||
- retain the rule that nonexistent interface-specific documents are not
|
||||
required until the interface exists.
|
||||
4. In the boundary rules, explicitly separate:
|
||||
- README orientation, contributor routing, normative architecture, and the
|
||||
implemented internal inventory;
|
||||
- CLI invocation, configuration meaning, operations, and troubleshooting;
|
||||
- integration contracts and internal implementation; and
|
||||
- documentation security rules, application security invariants,
|
||||
credential-supply mechanisms, operational handling, and internal
|
||||
mechanisms.
|
||||
5. Preserve the lightweight Nygard ADR format and supersession rules from the
|
||||
Notarius policy. Do not create an ADR directory in this stage.
|
||||
6. Remove the old documentation profiles, prescribed per-document outlines,
|
||||
and obsolete paths rather than appending the new policy beneath them.
|
||||
|
||||
### Checks
|
||||
|
||||
- Search the finished policy for `Notarius`, `docs/architecture.md`,
|
||||
`docs/policy/development.md`, and `docs/internal/README.md`; none should
|
||||
remain.
|
||||
- Confirm every canonical owner in the feature roadmap appears consistently in
|
||||
the policy.
|
||||
- Run `git diff --check -- docs/policy/documentation.md`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- The policy closely follows the current Notarius policy while expressing the
|
||||
Narratio-specific ownership model.
|
||||
- It describes target canonical paths even when a later stage will perform the
|
||||
physical rename.
|
||||
- It contains no implementation sequence or future runtime claim.
|
||||
|
||||
## Stage 2: Refocus The Architecture Policy
|
||||
|
||||
### Goal
|
||||
|
||||
Make `docs/policy/architecture.md` the concise normative owner of Narratio's
|
||||
current system shape, dependency boundaries, invariants, safety properties,
|
||||
and non-goals.
|
||||
|
||||
### Work
|
||||
|
||||
1. Review the current architecture policy, the feature roadmap, the new
|
||||
documentation policy, `../notarius/docs/policy/architecture.md`, and the
|
||||
current internal overview and focused internal documents.
|
||||
2. Rewrite or reorganize `docs/policy/architecture.md` using the Notarius
|
||||
architecture policy as a style and ownership reference, not as a source of
|
||||
Narratio runtime facts.
|
||||
3. Preserve and clearly state Narratio's existing normative decisions:
|
||||
- a small, explicit, stage-driven orchestrator rather than a generic DAG or
|
||||
workflow engine;
|
||||
- Narratio ownership of orchestration, configuration resolution, run and
|
||||
session state, artifact and path modeling, manifests, resume, and publish;
|
||||
- isolation of WhisperX, Seriatim, Audita, Scriptorium, notification, and
|
||||
object-storage details behind adapters;
|
||||
- strict, centralized configuration and narrow templating;
|
||||
- centralized local and remote path construction and confined writes;
|
||||
- manifest-driven stage progress and transactional stage completion;
|
||||
- the remote publish commit boundary in which `current/run_id.txt` is
|
||||
written last;
|
||||
- private campaign-data and secret-handling invariants;
|
||||
- deterministic ordering and repeatable orchestration where practical; and
|
||||
- the existing architectural non-goals.
|
||||
4. Keep package names and concrete component inventories out of architecture
|
||||
except where a name is essential to express a boundary. Route concrete
|
||||
ownership to `../internal/overview.md` and focused internal docs.
|
||||
5. Replace the general testing inventory with a short architectural testing
|
||||
expectation and a link to `testing.md`. Package- or subsystem-specific test
|
||||
details belong in internal docs when useful.
|
||||
6. Replace documentation-rule duplication with a short requirement and a link
|
||||
to `documentation.md`.
|
||||
7. Link operational layout and lifecycle statements to `../operations.md` and
|
||||
external contract statements to `../integrations/` where useful. Do not
|
||||
duplicate their complete contracts.
|
||||
8. Remove all obsolete references, especially
|
||||
`docs/documentation/policy.md` and `docs/architecture.md`.
|
||||
9. Do not invent ADR links. Mention `docs/adr/` only as the future home of
|
||||
significant decision history.
|
||||
|
||||
### Checks
|
||||
|
||||
- Search the file for `docs/documentation/policy.md`, `docs/architecture.md`,
|
||||
and `docs/internal/README.md`; none should remain.
|
||||
- Confirm links resolve relative to `docs/policy/`.
|
||||
- Compare every retained invariant with the existing architecture policy and
|
||||
relevant current internal documentation; no invariant may be weakened by
|
||||
omission merely to shorten the document.
|
||||
- Run `git diff --check -- docs/policy/architecture.md`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Architecture owns normative system rules but not the package inventory,
|
||||
general testing policy, documentation policy, CLI reference, configuration
|
||||
reference, or operational procedures.
|
||||
- Existing Narratio safety and publish invariants remain explicit.
|
||||
|
||||
## Stage 3: Move And Rewrite The Contributor Guide
|
||||
|
||||
### Goal
|
||||
|
||||
Create the canonical first-read contributor landing page at
|
||||
`docs/development.md` and remove its obsolete policy location.
|
||||
|
||||
### Work
|
||||
|
||||
1. Move `docs/policy/development.md` to `docs/development.md` using a normal
|
||||
repository rename so history remains recognizable.
|
||||
2. Rewrite it as a concise, table-of-contents-adjacent landing page modeled on
|
||||
`../notarius/docs/development.md`, adapted entirely to Narratio.
|
||||
3. Begin with a short orientation stating that Narratio is a stage-driven Go
|
||||
orchestrator and route readers to:
|
||||
- `README.md` for product context;
|
||||
- `policy/architecture.md` for normative system boundaries; and
|
||||
- `internal/overview.md` for implemented component ownership.
|
||||
4. Add a `What To Read` table with task-specific routes for at least:
|
||||
- locating current component ownership;
|
||||
- architecture, boundaries, invariants, or safety changes;
|
||||
- documentation changes;
|
||||
- adding, reviewing, rewriting, or deleting tests;
|
||||
- CLI behavior;
|
||||
- configuration loading or user-visible configuration;
|
||||
- session workflow, restore, status, cleanup, and object storage;
|
||||
- stage behavior and pipeline sequencing;
|
||||
- adapters and external contracts;
|
||||
- manifests, artifacts, workspace paths, and publish behavior;
|
||||
- examples; and
|
||||
- proposed or unimplemented behavior.
|
||||
5. For each row, link both the internal implementation guide and public or
|
||||
external contract when those have distinct ownership. Do not restate their
|
||||
rules in the table.
|
||||
6. End with concise validation guidance:
|
||||
- use focused package tests while iterating;
|
||||
- run `go test ./...`;
|
||||
- run `go vet ./...`; and
|
||||
- run `go build ./cmd/narratio` for repository-wide changes.
|
||||
7. Remove the old repository-layout inventory, detailed change playbooks,
|
||||
dependency policy, and duplicated coding conventions. Route those topics to
|
||||
architecture, internal documentation, public contracts, examples, and the
|
||||
testing or documentation policy instead.
|
||||
8. Update direct references to the contributor guide in files already changed
|
||||
in Stages 1 and 2 if any transitional link remains. Broader navigation is
|
||||
handled in Stage 5.
|
||||
|
||||
### Checks
|
||||
|
||||
- Confirm `docs/development.md` exists and
|
||||
`docs/policy/development.md` does not.
|
||||
- Confirm all links in the new file resolve relative to `docs/`.
|
||||
- Confirm the guide contains no package inventory or second definition of a
|
||||
public contract.
|
||||
- Run `git diff --check -- docs/development.md`.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- A new maintainer or coding agent can choose the correct canonical reading
|
||||
path from one short document.
|
||||
- The contributor guide is navigational rather than a fourth normative policy.
|
||||
|
||||
## Stage 4: Replace The Repository Agent Instruction
|
||||
|
||||
### Goal
|
||||
|
||||
Make `AGENTS.md` a stable pointer to the task-specific contributor guide.
|
||||
|
||||
### Work
|
||||
|
||||
Replace the entire root `AGENTS.md` contents with exactly these two lines,
|
||||
including the line break and final period:
|
||||
|
||||
```text
|
||||
Please review `docs/development.md` for initial orientation in this repository
|
||||
and follow its task-specific reading guide.
|
||||
```
|
||||
|
||||
Do not retain the previous instruction, add headings, add generated markers, or
|
||||
append repository policy details.
|
||||
|
||||
### Checks
|
||||
|
||||
- Run `git diff --check -- AGENTS.md`.
|
||||
- Verify exact content with:
|
||||
|
||||
```sh
|
||||
diff -u <(printf '%s\n' 'Please review `docs/development.md` for initial orientation in this repository' 'and follow its task-specific reading guide.') AGENTS.md
|
||||
```
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- The comparison is empty and `AGENTS.md` contains no other text.
|
||||
|
||||
## Stage 5: Align Structural Entry Points And Navigation
|
||||
|
||||
### Goal
|
||||
|
||||
Complete the selected structural parity changes and make every primary entry
|
||||
point use the new canonical paths.
|
||||
|
||||
### Work
|
||||
|
||||
1. Rename `docs/internal/README.md` to `docs/internal/overview.md`.
|
||||
2. Refocus the renamed file as the implemented component inventory:
|
||||
- state its boundary against architecture and public/operator contracts;
|
||||
- show the high-level execution path from `cmd/narratio` through
|
||||
`internal/app`, configuration and composition, stages and adapters,
|
||||
manifests and artifacts, and durable outputs;
|
||||
- provide a compact component table for the executable, app orchestration,
|
||||
configuration, stages, adapters, manifests, artifacts/path safety,
|
||||
previous-session cache, logging, and artifact policy; and
|
||||
- retain links to every focused internal document.
|
||||
3. Keep the canonical stage order in the internal overview only as an
|
||||
implemented inventory summary. Link to stage documents for mechanics and to
|
||||
CLI/operations for user-visible execution semantics.
|
||||
4. Update `README.md` while keeping it short:
|
||||
- preserve Narratio's product description and minimal successful command;
|
||||
- link to `docs/development.md` and `docs/internal/overview.md`;
|
||||
- include the documentation, architecture, and testing policies in a
|
||||
maintainer-oriented portion of the link list without turning the README
|
||||
into contributor guidance; and
|
||||
- retain user links to CLI, configuration, operations, troubleshooting,
|
||||
integrations, and examples.
|
||||
5. Update all references throughout `README.md`, `docs/`, and roadmap files from
|
||||
`docs/policy/development.md` to `docs/development.md`, and from
|
||||
`docs/internal/README.md` to `docs/internal/overview.md`. Fix relative links
|
||||
according to the referencing file's directory rather than performing a
|
||||
blind textual replacement.
|
||||
6. Correct other obsolete structural references already known in
|
||||
`docs/roadmap/notarius-extract-stage.md`, including `docs/architecture.md`,
|
||||
so it points to `docs/policy/architecture.md` and the now-existing
|
||||
`docs/development.md`. Do not change that roadmap's feature design or claim
|
||||
that extraction is implemented.
|
||||
7. Keep `docs/integrations/README.md` as the integrations index for now; update
|
||||
its links and ownership language as necessary, but do not rename it.
|
||||
|
||||
### Checks
|
||||
|
||||
- Search the repository documentation, excluding this implementation plan and
|
||||
its feature roadmap, for `docs/policy/development.md`,
|
||||
`docs/internal/README.md`, `docs/architecture.md`, and
|
||||
`docs/documentation/policy.md`; none should remain. The two planning documents
|
||||
may name old paths when specifying the required migrations.
|
||||
- Confirm `README.md`, `docs/development.md`, and
|
||||
`docs/internal/overview.md` form a coherent orientation chain.
|
||||
- Run `git diff --check` for all changed Markdown files.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- The physical file layout matches the canonical ownership table.
|
||||
- All primary entry points and known inbound links use the new locations.
|
||||
- No compatibility stub is left at either old path.
|
||||
|
||||
## Stage 6: Audit User, Operator, And Integration Contracts
|
||||
|
||||
### Goal
|
||||
|
||||
Give volatile public and external facts one canonical owner while preserving
|
||||
useful task-oriented guidance.
|
||||
|
||||
### Work
|
||||
|
||||
1. Review `docs/cli.md`, `docs/config.md`, `docs/operations.md`,
|
||||
`docs/troubleshooting.md`, every file under `docs/integrations/`, maintained
|
||||
examples, and the corresponding Go implementations and tests.
|
||||
2. For code discovery, follow repository instructions: use the codebase
|
||||
knowledge graph first, then use text search for literal commands, fields,
|
||||
defaults, paths, messages, Markdown, YAML, and other non-code content.
|
||||
3. Make ownership conform to policy:
|
||||
- CLI owns commands, arguments, flags, invocation semantics, output
|
||||
conventions, and exits;
|
||||
- configuration owns discovery, precedence, fields, defaults, environment
|
||||
overrides, and validation;
|
||||
- operations owns runtime workflows, physical local and remote layout,
|
||||
resume, cleanup, permissions, recovery, and operational limits;
|
||||
- troubleshooting owns symptom-led diagnosis and safe fixes, linking rather
|
||||
than redefining other contracts; and
|
||||
- integrations own external protocols, file formats, request/result shapes,
|
||||
logical durable outputs, and compatibility behavior.
|
||||
4. Preserve concise cross-cutting workflows in the document whose audience
|
||||
owns the task. Replace repeated volatile definitions with stable summaries
|
||||
and direct links.
|
||||
5. Create `docs/integrations/whisperx.md`. Derive its current contract from
|
||||
`internal/adapters/whisperx`, `internal/stage/transcribe.go`, configuration,
|
||||
and tests. Cover only:
|
||||
- Narratio's purpose for the integration;
|
||||
- the HTTP adapter boundary;
|
||||
- request and response expectations;
|
||||
- retry, timeout, cancellation, validation, and failure semantics;
|
||||
- deterministic or concurrency behavior that is externally relevant; and
|
||||
- links to configuration for operator-selected values and to internal docs
|
||||
for implementation mechanics.
|
||||
Do not duplicate the configuration field table or physical workspace paths.
|
||||
6. Add WhisperX to `docs/integrations/README.md` and correct the index's
|
||||
description: integration contracts are externally observable boundaries,
|
||||
not merely implementation-level references.
|
||||
7. Review the Audita, Scriptorium, and Seriatim documents for the same boundary.
|
||||
Keep their protocol and data contracts; move or replace configuration field
|
||||
definitions and internal runner wiring with links when duplicated.
|
||||
8. Verify every documented command, flag, field, default, environment
|
||||
override, exit behavior, path, and integration guarantee against code or a
|
||||
focused test. If a current fact cannot be verified, remove unsupported
|
||||
specificity or record the issue for separate follow-up.
|
||||
|
||||
### Checks
|
||||
|
||||
- Compare CLI command and flag inventories with `internal/app` command parsing
|
||||
and its tests.
|
||||
- Compare configuration tables with `internal/config` structs, defaults,
|
||||
normalization, loading, validation, and tests.
|
||||
- Compare operational path and lifecycle claims with `internal/artifacts`,
|
||||
`internal/manifest`, `internal/app`, `internal/stage`, and their tests.
|
||||
- Compare each integration contract with its adapter and stage boundary tests.
|
||||
- Confirm troubleshooting entries use symptom, likely cause, diagnostic step,
|
||||
safe fix, and relevant links.
|
||||
- Run focused tests only when needed to verify a disputed contract; no code
|
||||
changes are authorized.
|
||||
- Run `git diff --check` for changed documentation.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Each volatile user, operator, and integration fact has one authoritative
|
||||
definition.
|
||||
- The new WhisperX contract is discoverable and implementation-accurate.
|
||||
- Task-oriented workflows remain usable without maintaining parallel reference
|
||||
tables.
|
||||
|
||||
## Stage 7: Audit Internal Documentation And Examples
|
||||
|
||||
### Goal
|
||||
|
||||
Make internal documents explain implementation without redefining architecture,
|
||||
public contracts, external contracts, or future work, and confirm maintained
|
||||
examples remain canonical.
|
||||
|
||||
### Work
|
||||
|
||||
1. Review every focused file under `docs/internal/` against its current package
|
||||
implementation and focused tests.
|
||||
2. For each internal document, retain or establish:
|
||||
- purpose and implemented owner;
|
||||
- internal collaborators and execution flow;
|
||||
- inputs, outputs, and state transitions expressed at the internal boundary;
|
||||
- package-local guarantees, failures, and safety invariants;
|
||||
- links to relevant tests or test areas when they materially help a
|
||||
maintainer; and
|
||||
- links to public, operator, configuration, or integration contracts rather
|
||||
than duplicate definitions.
|
||||
3. Audit the highest-drift topics explicitly:
|
||||
- canonical stage order, prerequisites, invalidation, skip, force, resume,
|
||||
and failure behavior;
|
||||
- session and run manifest states and transitions;
|
||||
- artifact identities, resolution rules, checksums, and current-state
|
||||
pointers;
|
||||
- workspace, cache, spool, logs, reports, and object-storage keys;
|
||||
- restore discovery, planning, conflict handling, execution, and reporting;
|
||||
- publish upload ordering and commit semantics;
|
||||
- adapter composition and subprocess behavior; and
|
||||
- path confinement, deletion, and cleanup safety.
|
||||
4. Where the same internal fact appears in several focused documents, select
|
||||
the component that owns it and use links from consumers. Do not over-prune a
|
||||
short invariant when repeating it is necessary to prevent an unsafe local
|
||||
change; in that case, state the invariant briefly and link to its normative
|
||||
owner.
|
||||
5. Review all complete YAML and input artifacts under `examples/`:
|
||||
- keep complete copyable files there rather than in prose;
|
||||
- ensure documentation links to them rather than maintaining full duplicate
|
||||
examples;
|
||||
- confirm they contain no credentials or private campaign content; and
|
||||
- verify they load and validate through existing tests where such coverage
|
||||
already exists.
|
||||
6. Do not modify application code or invent new test infrastructure. A missing
|
||||
example test may be recorded for later code work; it is not in scope for
|
||||
this documentation-only implementation.
|
||||
|
||||
### Checks
|
||||
|
||||
- Search internal docs for duplicated configuration field/default tables, CLI
|
||||
flag tables, complete external schemas, and physical operations procedures;
|
||||
replace inappropriate copies with links.
|
||||
- Compare the stage inventory in `docs/internal/overview.md` with the
|
||||
implemented stage registry.
|
||||
- Run the existing example/config validation tests identified in
|
||||
`docs/development.md` or the current test suite.
|
||||
- Run `git diff --check` for changed documentation and examples.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- Internal documentation is an accurate component-level implementation guide.
|
||||
- Architecture and public/external contracts remain authoritative at their
|
||||
respective boundaries.
|
||||
- Complete examples are centralized, valid, discoverable, and secret-free.
|
||||
|
||||
## Stage 8: Perform Repository-Wide Validation And Close The Roadmap
|
||||
|
||||
### Goal
|
||||
|
||||
Verify the migrated documentation as a coherent whole and record completion
|
||||
only after every acceptance criterion is satisfied.
|
||||
|
||||
### Work
|
||||
|
||||
1. Review `README.md`, `AGENTS.md`, `docs/development.md`, all policy files,
|
||||
all user/operator references, all integration and internal documents, all
|
||||
roadmaps, and all maintained examples as one navigation system.
|
||||
2. Search for obsolete paths and project-name leakage:
|
||||
|
||||
```sh
|
||||
rg -n 'docs/policy/development\.md|docs/internal/README\.md|docs/architecture\.md|docs/documentation/policy\.md' README.md AGENTS.md docs --glob '!docs/roadmap/implementation.md' --glob '!docs/roadmap/documentation.md'
|
||||
rg -n 'Notarius' README.md AGENTS.md docs --glob '!docs/roadmap/notarius-extract-stage.md'
|
||||
```
|
||||
|
||||
The first search must return no matches. Review every match from the second;
|
||||
Narratio may legitimately mention the Notarius product or integration, but
|
||||
copied Notarius policy language must not remain.
|
||||
3. Validate local Markdown file and directory links with a reproducible script
|
||||
that scans `README.md`, `AGENTS.md`, and `docs/**/*.md`, resolves relative
|
||||
targets from each source file, strips anchors and query fragments, ignores
|
||||
external URLs and pure anchors, and fails for missing local targets. Use an
|
||||
existing repository checker if present. Otherwise run a temporary script
|
||||
without adding it or a new dependency to the repository.
|
||||
4. Manually inspect anchor links and directory links because a basic local-path
|
||||
checker may not validate them fully.
|
||||
5. Check formatting and whitespace:
|
||||
|
||||
```sh
|
||||
git diff --check
|
||||
```
|
||||
6. Run the complete existing validation suite:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/narratio
|
||||
```
|
||||
7. Review the feature roadmap acceptance criteria one by one. Confirm that:
|
||||
- canonical ownership is explicit and reflected by actual content;
|
||||
- current behavior and future roadmap work are separated;
|
||||
- examples and commands match implementation;
|
||||
- no sensitive data was introduced;
|
||||
- no obsolete path or compatibility stub remains; and
|
||||
- this effort changed documentation only.
|
||||
8. If all criteria and checks pass, change the status in
|
||||
`docs/roadmap/documentation.md` from `Proposed.` to `Completed.` and change
|
||||
this plan's status from `Proposed.` to `Completed.`. Do not mark either
|
||||
complete while required work or an unexplained validation failure remains.
|
||||
9. If a validation failure predates this work and is demonstrably unrelated,
|
||||
record the exact command, failure, and evidence in the implementation handoff
|
||||
rather than changing application code. Documentation-link or content failures
|
||||
introduced or exposed by this migration must be fixed before completion.
|
||||
|
||||
### Completion Criteria
|
||||
|
||||
- All Stage 8 searches and documentation checks pass.
|
||||
- `go test ./...`, `go vet ./...`, and `go build ./cmd/narratio` pass, or a
|
||||
precisely evidenced unrelated pre-existing failure is reported.
|
||||
- Both roadmap documents are marked `Completed.` only after the intended final
|
||||
state is present.
|
||||
- The final handoff lists changed documents, validation commands and results,
|
||||
and any explicitly out-of-scope follow-up without claiming unperformed work.
|
||||
380
docs/roadmap/notarius-extract-stage.md
Normal file
380
docs/roadmap/notarius-extract-stage.md
Normal file
@@ -0,0 +1,380 @@
|
||||
# Notarius Extraction Stage
|
||||
|
||||
## Status
|
||||
|
||||
Proposed.
|
||||
|
||||
## Purpose
|
||||
|
||||
Add a first-class Narratio `extract` stage that runs Notarius against the
|
||||
session's final trimmed transcript, validates and collects the resulting
|
||||
structured D&D artifacts, and registers those artifacts for later use by the
|
||||
`analyze` and `publish` stages.
|
||||
|
||||
This feature should integrate Notarius through Narratio's existing stage,
|
||||
adapter, manifest, workspace, and artifact-catalog boundaries. It must not turn
|
||||
Narratio into a generic workflow engine or a second configuration language for
|
||||
Notarius pipelines.
|
||||
|
||||
## User Outcome
|
||||
|
||||
An operator can enable one configured Notarius pipeline for a Narratio
|
||||
campaign. During a normal run, Narratio will:
|
||||
|
||||
1. finish producing the session transcript tiers;
|
||||
2. invoke Notarius once with the final trimmed Seriatim JSON transcript;
|
||||
3. collect and validate the configured structured artifact lanes;
|
||||
4. record their exact files and provenance in the Narratio manifest; and
|
||||
5. make those artifacts selectable as inputs to Scriptorium artifacts in the
|
||||
later `analyze` stage.
|
||||
|
||||
The maintained D&D example should demonstrate all ten lanes emitted by
|
||||
Notarius's complete `dnd-session` pipeline.
|
||||
|
||||
## Target Stage Architecture
|
||||
|
||||
### Canonical Order
|
||||
|
||||
The canonical stage order becomes:
|
||||
|
||||
```text
|
||||
prepare -> transcribe -> merge -> polish -> normalize -> trim -> render
|
||||
-> extract -> analyze -> publish -> notify
|
||||
```
|
||||
|
||||
`extract` is deliberately after all transcript-producing stages and before
|
||||
analysis. Its source document is the manifest-resolved
|
||||
`narratio.transcript.final_trimmed` artifact, normally
|
||||
`transcripts/final.trimmed.json`. It does not consume rendered Markdown.
|
||||
|
||||
Adding the stage must update full-plan construction, explicit stage selection,
|
||||
downstream invalidation, prerequisite checks, resume behavior, run manifests,
|
||||
CLI stage validation and help, and every canonical-stage inventory. Forcing an
|
||||
upstream transcript stage must stale a previously successful `extract` stage
|
||||
and its downstream stages. Forcing `extract` must stale `analyze`, `publish`,
|
||||
and `notify` according to existing rules.
|
||||
|
||||
### Stage Boundary
|
||||
|
||||
The stage owns Narratio policy and state transitions:
|
||||
|
||||
- resolve the final trimmed transcript through the runtime artifact catalog;
|
||||
- build a Narratio-level Notarius request from validated configuration and
|
||||
run-local paths;
|
||||
- call a narrow Notarius adapter;
|
||||
- apply the configured required-output policy;
|
||||
- materialize the validated bundle into its canonical session location;
|
||||
- return manifest-ready artifact references and bounded metadata; and
|
||||
- fail without marking the stage successful when any required contract or
|
||||
materialization step fails.
|
||||
|
||||
The stage must not construct subprocess arguments, infer Notarius output
|
||||
filenames, parse provider logs, or decode individual D&D payload bodies.
|
||||
|
||||
### Adapter Boundary
|
||||
|
||||
Add a dedicated Notarius adapter package with a small interface, production
|
||||
subprocess implementation, and test fake. Its request should contain only the
|
||||
resolved Notarius binary, configuration path, pipeline ID, transcript path,
|
||||
output root, working directory, timeout, and process-log destinations needed
|
||||
for one run.
|
||||
|
||||
The adapter owns:
|
||||
|
||||
- optional `notarius config validate` preflight for the configured pipeline;
|
||||
- exact `notarius run ... --json` argument construction;
|
||||
- stdout and stderr separation;
|
||||
- context cancellation and timeout propagation through Narratio's shared
|
||||
subprocess boundary;
|
||||
- exit-status handling;
|
||||
- decoding the `notarius.run-result.v1` success receipt;
|
||||
- receipt and index path-confinement checks;
|
||||
- decoding `index.json` and resolving descriptor paths safely beneath the
|
||||
reported output directory; and
|
||||
- returning a transport-neutral result containing the bundle location,
|
||||
receipt summary, lane descriptors, pipeline-wide descriptors, warnings and
|
||||
rejection locations, and diagnostic log paths.
|
||||
|
||||
Only exit status zero permits receipt decoding. Receipt, index, or descriptor
|
||||
paths that are absolute where a logical relative path is required, or that
|
||||
escape their owning root, are integration failures. Unknown fields in a
|
||||
supported receipt or index schema should be tolerated. Unsupported schema
|
||||
versions and incompatible descriptor metadata should fail clearly.
|
||||
|
||||
The adapter must not write Narratio manifests, choose required lanes, decide
|
||||
analysis inputs, or contain D&D domain logic.
|
||||
|
||||
## Configuration Contract
|
||||
|
||||
Add a strict optional `pipeline.notarius` configuration section. Omission or
|
||||
`enabled: false` keeps the current workflow usable and causes `extract` to
|
||||
self-skip without outputs.
|
||||
|
||||
The section should provide:
|
||||
|
||||
- `enabled`: explicit opt-in;
|
||||
- `binary`: Notarius executable, defaulting to `notarius`;
|
||||
- `config_path`: required when enabled;
|
||||
- `pipeline_id`: required when enabled;
|
||||
- `timeout`: a positive stage timeout with a documented default;
|
||||
- `working_directory`: optional explicit subprocess working directory,
|
||||
defaulting to the directory containing `config_path`; and
|
||||
- an `outputs` map defining the Notarius lane artifacts Narratio promises to
|
||||
collect.
|
||||
|
||||
Each output-map key is a stable Narratio extraction key. Each value must define:
|
||||
|
||||
- the exact Notarius `lane_id`;
|
||||
- the expected `media_type`;
|
||||
- the expected `schema_id`;
|
||||
- the expected `schema_version`; and
|
||||
- optionally an expected `module_key` when the operator needs to constrain the
|
||||
producing module as part of compatibility.
|
||||
|
||||
Narratio derives the downstream source ID
|
||||
`narratio.extraction.<output-key>` from the map key. Keys and lane IDs must be
|
||||
non-empty, unique after normalization, path-safe under the existing artifact
|
||||
policy, and collision-free with built-in and configured artifact identities.
|
||||
Every configured output is required: a successful Notarius process that omits
|
||||
one, rejects it, or reports incompatible descriptor metadata fails the
|
||||
`extract` stage.
|
||||
|
||||
This explicit map keeps Narratio's consumer contract stable when a Notarius
|
||||
lane ID or schema changes and avoids hard-coding the current D&D family into a
|
||||
generic adapter. It also replaces a separate `required_lanes` list, which would
|
||||
duplicate configuration.
|
||||
|
||||
Narratio should not reproduce Notarius lane selection, references, LLM
|
||||
profiles, model settings, retries, concurrency, or prompt configuration. Those
|
||||
remain in the referenced Notarius configuration. Narratio should not expose a
|
||||
runtime lane-selection flag for `extract`; one stage invocation runs the
|
||||
configured Notarius pipeline as a unit.
|
||||
|
||||
All configured paths should become absolute during Narratio configuration
|
||||
resolution. The deterministic default working directory allows a Notarius
|
||||
profile path relative to that directory, but operator documentation should
|
||||
still recommend absolute deployment paths where practical. Notarius reference
|
||||
paths continue to follow Notarius's own configuration-relative rules.
|
||||
|
||||
## Output And Artifact Model
|
||||
|
||||
### Canonical Bundle
|
||||
|
||||
Run Notarius against a run-local output root. After all configured descriptors
|
||||
are validated, materialize the contents of the exact run-specific Notarius
|
||||
bundle into a fixed canonical session directory:
|
||||
|
||||
```text
|
||||
artifacts/notarius/
|
||||
```
|
||||
|
||||
Preserve its relative layout, including `index.json`, `manifest.json`,
|
||||
`rejected.json`, `warnings.json`, `lanes/`, and any indexed `chunk-map.json` or
|
||||
`evidence-context.json`. Materialize the complete directory as one narrow,
|
||||
transactional replacement so a failed or interrupted rerun cannot mix files
|
||||
from different Notarius runs.
|
||||
|
||||
The raw subprocess receipt and stderr log belong in the run-local `extract`
|
||||
report and log directories. The raw receipt identifies the original run-local
|
||||
Notarius bundle and must not be rewritten to pretend that the canonical copy
|
||||
was its original `output_directory`. Narratio's manifest is the durable ledger
|
||||
for the canonical materialized paths.
|
||||
|
||||
### Registered Artifact Sources
|
||||
|
||||
For each configured output, locate the lane through the canonical copy of
|
||||
`index.json` and record a manifest artifact with:
|
||||
|
||||
- source ID `narratio.extraction.<output-key>`;
|
||||
- canonical lane-file path discovered from the index;
|
||||
- producer stage and Narratio run ID;
|
||||
- checksum;
|
||||
- Notarius lane ID; and
|
||||
- descriptor media type, schema identity/version, and module key when present.
|
||||
|
||||
If the current manifest model cannot carry descriptor compatibility metadata,
|
||||
extend its artifact metadata in a backward-tolerant way rather than encoding
|
||||
that information in filenames or source IDs.
|
||||
|
||||
Also record the canonical Notarius index as a stage output or stage metadata so
|
||||
operators can discover the complete bundle, including non-lane artifacts. The
|
||||
configured lane sources are the stable interface for analysis; the index and
|
||||
bundle remain the provenance and inspection interface.
|
||||
|
||||
## Analysis And Publish Integration
|
||||
|
||||
Extend the runtime artifact catalog and configured Scriptorium input validation
|
||||
so an enabled analysis artifact can declare, for example:
|
||||
|
||||
```yaml
|
||||
inputs:
|
||||
npc_registry:
|
||||
source: narratio.extraction.npc_registry
|
||||
```
|
||||
|
||||
Resolution must remain manifest-first and verify that the recorded artifact
|
||||
was produced by a successful current `extract` stage. A required extraction
|
||||
source that is unavailable must fail analysis with guidance to configure or
|
||||
rerun `extract`; an optional source may be omitted according to the existing
|
||||
Scriptorium input contract.
|
||||
|
||||
Publish source resolution should accept configured
|
||||
`narratio.extraction.<output-key>` sources through the same artifact catalog so
|
||||
operators may publish selected structured artifacts without manually copying
|
||||
paths. The existing `--artifacts` flag remains scoped to Scriptorium artifact
|
||||
selection and must not partially execute the Notarius pipeline.
|
||||
|
||||
No current-session analysis artifact should consume an incidental file from a
|
||||
failed, stale, skipped, or superseded extraction run.
|
||||
|
||||
## Failure, Skip, Resume, And Diagnostics
|
||||
|
||||
- Missing or invalid enabled Notarius configuration fails configuration
|
||||
validation before stage execution where statically discoverable.
|
||||
- A disabled or absent Notarius configuration makes `extract` skip with clear
|
||||
stage metadata and no new outputs.
|
||||
- A missing or invalid final trimmed transcript fails `extract` before starting
|
||||
Notarius.
|
||||
- Preflight failure, nonzero Notarius exit, cancellation, timeout, malformed or
|
||||
unsupported receipt/index data, unsafe paths, incompatible descriptors,
|
||||
rejected required outputs, or missing configured lanes fails the entire
|
||||
stage.
|
||||
- Process success does not override Narratio's required-output policy.
|
||||
- A failed run retains bounded run-local receipt bytes, stderr, and the
|
||||
unpublished Notarius bundle for diagnosis, subject to Narratio's existing
|
||||
sensitive-data and cleanup policies.
|
||||
- The canonical bundle and manifest artifacts are updated only after complete
|
||||
validation and materialization.
|
||||
- Resume skips a succeeded, non-stale `extract` stage only when its
|
||||
manifest-recorded canonical index and configured lane outputs still validate.
|
||||
- Force and staleness behavior follows the ordinary stage contract; it must not
|
||||
depend on merely finding `artifacts/notarius/` on disk.
|
||||
|
||||
Transcripts, Notarius outputs, evidence context, manifests, receipts, and logs
|
||||
are private campaign material. Subprocess arguments and manifest metadata must
|
||||
not contain secrets. Credentials remain in the environment or in mechanisms
|
||||
owned by Notarius and PromptKit.
|
||||
|
||||
## Maintained D&D Example
|
||||
|
||||
Add or update a Narratio example that enables Notarius's complete
|
||||
`dnd-session` pipeline and maps these ten required lanes to stable extraction
|
||||
keys:
|
||||
|
||||
| Output key | Notarius lane ID |
|
||||
| --- | --- |
|
||||
| `item_registry` | `item-registry` |
|
||||
| `npc_registry` | `npc-registry` |
|
||||
| `location_registry` | `location-registry` |
|
||||
| `scene_descriptions` | `scene-descriptions` |
|
||||
| `item_occurrences` | `item-occurrences` |
|
||||
| `spells` | `spells` |
|
||||
| `combat_turns` | `combat-turns` |
|
||||
| `npc_occurrences` | `npc-occurrences` |
|
||||
| `location_occurrences` | `location-occurrences` |
|
||||
| `enemy_events` | `enemy-events` |
|
||||
|
||||
The example must include each lane's current media type and schema identity
|
||||
from Notarius's published contracts. It should also demonstrate at least one
|
||||
Scriptorium analysis artifact consuming one or more
|
||||
`narratio.extraction.*` sources. The example must use placeholders and relative
|
||||
paths suitable for the example tree, contain no credentials, and pass the
|
||||
repository's configuration validation tests.
|
||||
|
||||
## Compatibility Policy
|
||||
|
||||
The initial integration baseline is the public subprocess contract available
|
||||
in Notarius v0.3.0:
|
||||
|
||||
- successful JSON receipt schema `notarius.run-result.v1`;
|
||||
- production JSON bundle discovery through `index.json`; and
|
||||
- the schema IDs and versions explicitly configured for required lanes.
|
||||
|
||||
Runtime compatibility should be decided from those published contracts, not
|
||||
from textual parsing of `notarius --version`. New optional receipt or index
|
||||
fields must not break Narratio. An unsupported receipt version or lane schema
|
||||
must fail before the artifact is registered for analysis.
|
||||
|
||||
## Documentation Deliverables When Implemented
|
||||
|
||||
Update current-behavior documentation in the same change that implements the
|
||||
feature:
|
||||
|
||||
- add `docs/integrations/notarius.md` for the external CLI, receipt, bundle,
|
||||
and adapter contract, linking to Notarius's canonical documentation;
|
||||
- add `docs/internal/stage-extract.md` for stage inputs, outputs, collaborators,
|
||||
state transitions, failures, and focused tests;
|
||||
- update `docs/internal/adapters.md`, `docs/internal/artifacts.md`,
|
||||
`docs/internal/manifest.md`, and the internal stage inventory;
|
||||
- update `docs/architecture.md` to list Notarius among isolated external
|
||||
systems and preserve the adapter/stage boundary;
|
||||
- update `docs/config.md`, `docs/cli.md`, `docs/operations.md`,
|
||||
`docs/troubleshooting.md`, `README.md`, and maintained examples only to the
|
||||
extent their canonical scopes require; and
|
||||
- add the missing `docs/development.md` required by the repository's
|
||||
documentation policy, or track that repository-wide documentation gap in a
|
||||
separate roadmap if it is intentionally outside the implementation scope.
|
||||
|
||||
Outside this roadmap, do not describe `extract`, Notarius configuration, or
|
||||
`narratio.extraction.*` sources as implemented until the code exists.
|
||||
|
||||
## Testing And Validation Expectations
|
||||
|
||||
Implementation should provide focused tests for:
|
||||
|
||||
- strict configuration decoding, defaults, required fields, path resolution,
|
||||
output-map validation, normalized-key collisions, and example loading;
|
||||
- exact stage order, selection, downstream staleness, resume, force, and
|
||||
prerequisite behavior;
|
||||
- adapter command construction, deterministic working directory, environment
|
||||
inheritance, stdout/stderr separation, cancellation, timeout, and nonzero
|
||||
exits;
|
||||
- supported and unsupported receipt versions, unknown optional fields,
|
||||
malformed receipts, index decoding, and path escapes at every boundary;
|
||||
- descriptor lookup by lane ID rather than filename, expected metadata checks,
|
||||
missing/rejected configured lanes, and tolerated unconfigured lanes;
|
||||
- run-local execution, transactional canonical-bundle replacement, checksums,
|
||||
failed-run preservation, and manifest recording;
|
||||
- artifact-catalog resolution from `narratio.extraction.*` into analysis and
|
||||
publish, including required, optional, missing, stale, and skipped cases; and
|
||||
- end-to-end stage execution with a fake Notarius adapter, without live LLM or
|
||||
external subprocess requirements in the ordinary test suite.
|
||||
|
||||
Run the repository-wide Go tests, vet, build, and maintained example validation
|
||||
after focused tests pass.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- `extract` is a first-class transactional stage between `render` and
|
||||
`analyze` everywhere Narratio models stage order or state.
|
||||
- Narratio invokes Notarius only through a narrow, tested adapter.
|
||||
- The stage consumes the manifest-resolved final trimmed Seriatim transcript.
|
||||
- The Notarius configuration remains owned by Notarius; Narratio configures
|
||||
only invocation and its downstream consumer contract.
|
||||
- Every configured output is discovered through the receipt and `index.json`,
|
||||
contract-checked, materialized transactionally, and recorded with a stable
|
||||
`narratio.extraction.*` source ID.
|
||||
- The complete D&D example maps all ten current lanes and passes strict config
|
||||
validation.
|
||||
- Analysis can consume extraction sources through the existing artifact input
|
||||
model, and publish can select them through the artifact catalog.
|
||||
- Failed, partial, rejected, unsafe, stale, or incompatible output never becomes
|
||||
a current analysis input.
|
||||
- Resume and force behavior remains manifest-driven.
|
||||
- Documentation accurately describes the implemented stage, adapter,
|
||||
configuration, operations, and compatibility boundary without duplicating
|
||||
Notarius's canonical schemas.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Reimplementing Notarius extraction, prompts, schemas, references, retries,
|
||||
profiles, or lane orchestration in Narratio.
|
||||
- Allowing one Narratio run to invoke arbitrary extractor programs or multiple
|
||||
Notarius pipelines.
|
||||
- Making `extract` a configurable DAG or folding it into the Scriptorium
|
||||
`analyze` stage.
|
||||
- Partially selecting Notarius lanes through Narratio's `--artifacts` flag.
|
||||
- Decoding D&D payload bodies in the generic Notarius adapter.
|
||||
- Supporting previous-session extraction artifacts in the initial feature.
|
||||
- Requiring live Notarius, PromptKit, an LLM provider, or external services in
|
||||
the ordinary unit test suite.
|
||||
Reference in New Issue
Block a user