255 lines
11 KiB
Markdown
255 lines
11 KiB
Markdown
# Internal: Artifacts
|
|
|
|
## Purpose
|
|
|
|
Explain the artifact registry, runtime catalog, resolver, previous-input
|
|
requirements, and shared remote current-state mechanics implemented by
|
|
`internal/artifacts`. Configuration fields that accept source IDs belong in
|
|
[Configuration](../config.md); physical placement belongs in
|
|
[Operations](../operations.md).
|
|
|
|
## Built-in Source IDs
|
|
|
|
The internal registry recognizes these stable built-in source IDs:
|
|
|
|
- `narratio.transcript.base`
|
|
- `narratio.transcript.polished`
|
|
- `narratio.transcript.final`
|
|
- `narratio.transcript.final_trimmed`
|
|
- `narratio.transcript.final_markdown`
|
|
- `narratio.transcript.final_trimmed_markdown`
|
|
- `narratio.bounds.session`
|
|
|
|
Registry entries bind each ID to its producer, output kind, canonical fallback,
|
|
and content validator. The focused stage documents own their input/output flow;
|
|
[Configuration](../config.md) owns where operators may select these IDs.
|
|
|
|
## Configured, Extraction, And Previous-Session Sources
|
|
|
|
- configured source ID format: `narratio.artifact.<artifact_key>`
|
|
- extraction source ID format: `narratio.extraction.<output_key>`
|
|
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
|
|
|
|
All formats are validated by strict source-policy rules. Configured artifact and
|
|
extraction keys use `^[a-z][a-z0-9_]*$`; source parsers never normalize an
|
|
unrecognized token into a valid source. Extraction sources are registered only
|
|
from `pipeline.notarius.outputs`; the Notarius index has no selectable source
|
|
ID.
|
|
|
|
Prepared stable source IDs are `narratio.input.players`,
|
|
`narratio.input.party`, `narratio.input.glossary`, and
|
|
`narratio.input.spell_catalog`. Artifact policy owns their canonical manifest
|
|
kind and prepared filename vocabulary. Canonical party mode preserves the
|
|
party source bytes in the party record and supplies the players record from the
|
|
deterministic `derived_from_party` projection; both remain ordinary prepared
|
|
source IDs for consumers.
|
|
|
|
## Runtime Catalog
|
|
|
|
`ArtifactCatalog` tracks:
|
|
|
|
- `planned`: source registered for run context;
|
|
- `executable`: included in the effective analyze artifact set;
|
|
- `available`: the source's canonical evidence owner validates its current
|
|
manifest record and durable bytes;
|
|
- `provenance`: availability source.
|
|
|
|
Configured definitions are always registered. Without an explicit selection,
|
|
the effective analyze set contains enabled definitions. With `--artifacts`, the
|
|
exact named configured definitions become the effective set for that invocation,
|
|
regardless of their `enabled` value. The effective-set resolver itself does not
|
|
expand dependencies; the analyze work planner closes those targets over their
|
|
configured prerequisite graph. Availability is separate from executability.
|
|
Configured outputs, including non-executable prerequisites, become available
|
|
only when the versioned analyze state identifies a current result whose source,
|
|
contract, canonical configured path, size, and checksum match a confined
|
|
no-follow regular file. An incidental canonical file and a legacy aggregate
|
|
analyze output are unavailable.
|
|
Extraction entries are registered from configuration and become available only
|
|
after compatible extraction evidence is hydrated.
|
|
|
|
During an analyze invocation, a newly validated and atomically materialized
|
|
configured output is marked available with its producer run ID, contract,
|
|
checksum, and size. Later scheduled dependents therefore observe the same
|
|
semantic identity whether their prerequisite was reused from current manifest
|
|
evidence or produced earlier in the invocation.
|
|
|
|
Current provenance values:
|
|
|
|
- `generated.current_analyze_run`
|
|
- `manifest.current_analyze_artifact`
|
|
- `manifest.inputs.previous_cache`
|
|
- `current_session.previous_cache`
|
|
|
|
## Resolution Rules
|
|
|
|
Built-ins:
|
|
|
|
1. manifest producer outputs (when present)
|
|
2. canonical session-path fallback
|
|
|
|
Configured sources (`narratio.artifact.*`):
|
|
|
|
- resolve only through runtime catalog availability;
|
|
- use the shared typed analyze-evidence inspection in
|
|
`analyze_evidence.go` for prior current-session results;
|
|
- require the supported analyze-state and fingerprint versions, a `current`
|
|
record for the exact configured key and source ID, a complete contract, the
|
|
configured canonical relative path, positive stored size, and stored
|
|
checksum matching bytes read from a confined no-follow regular file; and
|
|
- treat non-current statuses, legacy or malformed records, removed keys,
|
|
unsafe or missing files, and size/checksum mismatches as unavailable without
|
|
rewriting manifest state. Catalog construction iterates current
|
|
configuration, so removed or renamed records are not advertised.
|
|
|
|
Prepared stable sources (`narratio.input.*`):
|
|
|
|
- resolve only from the current manifest's exact prepared-input record;
|
|
- require the policy-owned canonical path below the session root, a confined
|
|
non-symlink regular file, a non-empty payload, and a matching SHA-256
|
|
checksum; and
|
|
- return an immutable source/path/checksum/size identity shared by extract and
|
|
analyze rather than falling back to campaign/session source paths.
|
|
|
|
Extraction sources (`narratio.extraction.*`):
|
|
|
|
- use the shared typed bundle evidence inspection in `extraction_evidence.go`;
|
|
- require a current successful extract record with the exact configured source,
|
|
compatible contract and Notarius provenance, a confined regular durable
|
|
payload, matching checksum, and the current resolved trimmed-transcript
|
|
identity;
|
|
- remain unavailable unless catalog hydration receives valid evidence. Resume
|
|
treats absent or obsolete evidence as a rerun decision and unsafe evidence as
|
|
an error; and
|
|
- are never inferred by scanning the Notarius bundle directory.
|
|
|
|
Previous-session sources (`narratio.previous_session.artifact.*`):
|
|
|
|
- resolve only from local `previous/` cache state;
|
|
- prefer manifest-backed previous-input paths;
|
|
- fallback to existing previous-cache filesystem paths.
|
|
|
|
Source absence is evaluated by the consuming artifact input. An optional input
|
|
is omitted from that invocation; a required input fails resolution. This is
|
|
separate from a stage's lifecycle outcome.
|
|
|
|
Validation by content type:
|
|
|
|
- transcript JSON built-ins: JSON with top-level `segments` array;
|
|
- transcript Markdown built-ins: non-empty text file;
|
|
- bounds built-in: valid JSON;
|
|
- configured/previous-session artifact files: non-empty text file.
|
|
|
|
## Previous Requirement Collection
|
|
|
|
`CollectPreviousArtifactRequirements`:
|
|
|
|
- scans the effective configured artifact set;
|
|
- extracts only canonical previous-session sources;
|
|
- deduplicates by artifact key;
|
|
- merges required and optional references (required wins);
|
|
- returns deterministic ordering and source locations.
|
|
|
|
## Current-State Helpers
|
|
|
|
Artifacts package owns shared remote current-state loading mechanics used by
|
|
restore, status and validation checks, and previous-cache planning.
|
|
|
|
For a new-protocol current state, the pointer-selected immutable commit is the
|
|
complete restore authority. Callers receive its declared object identities and
|
|
must not supplement them by listing mutable session prefixes. The legacy reader
|
|
is intentionally separate and remains migration-only support.
|
|
|
|
The reader opens each small control object directly and enforces owner-specific
|
|
limits before decoding: 64 KiB for the mutable commit pointer, 4 MiB for the
|
|
immutable commit manifest, and 8 MiB for the selected session manifest. Legacy
|
|
compatibility applies a 4 KiB limit to `current/run_id.txt` and the same 8 MiB
|
|
manifest limit to `current/manifest.json`. These are exposed as
|
|
`MaxCurrentCommitPointerBytes`, `MaxRemoteCommitManifestBytes`,
|
|
`MaxRemoteSessionManifestBytes`, `MaxLegacyCurrentRunPointerBytes`, and
|
|
`MaxLegacyCurrentManifestBytes`.
|
|
|
|
Each read uses the generation and size metadata returned with its opened body.
|
|
Actual bytes remain subject to a limit-plus-one read even if size metadata is
|
|
absent or inaccurate. Immutable selections then retain their declared-size,
|
|
checksum, generation, and identity checks. No current-state control object is
|
|
downloaded through a temporary file.
|
|
|
|
Core helpers:
|
|
|
|
- `LoadCurrentState`
|
|
- `ValidateCurrentStateIdentity`
|
|
- `RemoteCommitManifest` and `CurrentCommitPointer`
|
|
|
|
Typed missing-state errors:
|
|
|
|
- `CurrentRunPointerMissingError` (`ErrCurrentRunPointerMissing`)
|
|
- `CurrentManifestMissingError` (`ErrCurrentManifestMissing`)
|
|
|
|
Identity validation supports caller-provided expectations:
|
|
|
|
- expected campaign;
|
|
- expected session ID;
|
|
- expected run ID, or pointer/manifest run-ID consistency check.
|
|
|
|
Caller policy is intentionally outside artifacts helpers:
|
|
|
|
- some callers fail on missing current state;
|
|
- some callers downgrade missing state to status/findings;
|
|
- some callers skip optional behavior when state is missing.
|
|
|
|
## Key Path Helpers
|
|
|
|
`internal/artifacts/paths.go` and S3-key helpers define canonical helpers for:
|
|
|
|
- session/work/run paths;
|
|
- previous-cache paths;
|
|
- spool/cache paths;
|
|
- S3 session/run/current-state key layout.
|
|
|
|
New publication creates run-scoped immutable objects, including
|
|
`runs/{run_id}/commit.json` and `runs/{run_id}/session-manifest.json`. The sole
|
|
mutable selector is `current/commit-pointer.json`; readers verify its selected
|
|
commit and declared object generations/checksums. Legacy current-pair loading
|
|
is confined to `current_state_legacy.go` for migration only.
|
|
|
|
Campaign, session, and Narratio run IDs are validated as portable opaque
|
|
segments at configuration and artifact boundaries before they can be used in a
|
|
workspace or S3 namespace. Previous-artifact destinations remain typed,
|
|
multi-segment relative paths and are confined beneath `previous/artifacts`; they
|
|
are not treated as opaque identifiers.
|
|
|
|
See [Workspace Internals](workspace.md) for how callers consume local helpers
|
|
and [Operations](../operations.md#local-state-layout) for the authoritative
|
|
physical layout.
|
|
|
|
## Invariants
|
|
|
|
- source ID formats are stable contracts;
|
|
- artifact resolution is deterministic and manifest-aware;
|
|
- extraction sources are available only from a compatible successful manifest
|
|
record;
|
|
- previous-session source resolution in `analyze` is local-only;
|
|
- remote current-state key construction remains centralized in artifacts helpers.
|
|
|
|
## Implementation And Tests
|
|
|
|
- Registry and resolution: `internal/artifacts/artifact_resolver.go`,
|
|
`internal/artifacts/catalog.go`, `internal/artifacts/transcripts.go`,
|
|
`internal/artifacts/extraction_catalog.go`,
|
|
`internal/artifacts/extraction_evidence.go`,
|
|
`internal/artifacts/extraction_input.go`,
|
|
`internal/artifacts/prepared_input.go`
|
|
- Current state: `internal/artifacts/current_state.go`,
|
|
`internal/artifacts/current_state_commit.go`,
|
|
`internal/artifacts/current_state_legacy.go`
|
|
- Paths and keys: `internal/artifacts/paths.go`,
|
|
`internal/artifacts/s3_keys.go`
|
|
- Previous requirements: `internal/artifacts/previous_requirements.go`
|
|
- Tests: `internal/artifacts/artifact_resolver_test.go`,
|
|
`internal/artifacts/catalog_test.go`,
|
|
`internal/artifacts/extraction_catalog_test.go`,
|
|
`internal/artifacts/current_state_test.go`,
|
|
`internal/artifacts/paths_model_test.go`,
|
|
`internal/artifacts/previous_requirements_test.go`
|