215 lines
8.6 KiB
Markdown
215 lines
8.6 KiB
Markdown
# Internal: Artifacts
|
|
|
|
## Purpose
|
|
|
|
Explain the artifact registry, runtime catalog, resolver, previous-input
|
|
requirements, and shared remote current-state mechanics implemented by
|
|
`internal/artifacts`. Configuration fields that accept source IDs belong in
|
|
[Configuration](../config.md); physical placement belongs in
|
|
[Operations](../operations.md).
|
|
|
|
## Built-in Source IDs
|
|
|
|
The internal registry recognizes these stable built-in source IDs:
|
|
|
|
- `narratio.transcript.base`
|
|
- `narratio.transcript.polished`
|
|
- `narratio.transcript.final`
|
|
- `narratio.transcript.final_trimmed`
|
|
- `narratio.transcript.final_markdown`
|
|
- `narratio.transcript.final_trimmed_markdown`
|
|
- `narratio.bounds.session`
|
|
|
|
Registry entries bind each ID to its producer, output kind, canonical fallback,
|
|
and content validator. The focused stage documents own their input/output flow;
|
|
[Configuration](../config.md) owns where operators may select these IDs.
|
|
|
|
## Configured, Extraction, And Previous-Session Sources
|
|
|
|
- configured source ID format: `narratio.artifact.<artifact_key>`
|
|
- extraction source ID format: `narratio.extraction.<output_key>`
|
|
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
|
|
|
|
All formats are validated by strict source-policy rules. Configured artifact and
|
|
extraction keys use `^[a-z][a-z0-9_]*$`; source parsers never normalize an
|
|
unrecognized token into a valid source. Extraction sources are registered only
|
|
from `pipeline.notarius.outputs`; the Notarius index has no selectable source
|
|
ID.
|
|
|
|
## Runtime Catalog
|
|
|
|
`ArtifactCatalog` tracks:
|
|
|
|
- `planned`: source registered for run context;
|
|
- `executable`: included in the effective analyze artifact set;
|
|
- `available`: local file exists and validates;
|
|
- `provenance`: availability source.
|
|
|
|
Configured definitions are always registered. Without an explicit selection,
|
|
the effective analyze set contains enabled definitions. With `--artifacts`, the
|
|
exact named configured definitions become the effective set for that invocation,
|
|
regardless of their `enabled` value; dependencies are not added implicitly.
|
|
Availability is separate from executability: a non-executable configured output
|
|
may be reused from a canonical non-empty file, while an executable definition
|
|
is generated by analyze. Extraction entries are registered from configuration
|
|
and become available only after compatible extraction evidence is hydrated.
|
|
|
|
Current provenance values:
|
|
|
|
- `generated.current_analyze_run`
|
|
- `filesystem.disabled_artifact_output`
|
|
- `manifest.inputs.previous_cache`
|
|
- `current_session.previous_cache`
|
|
|
|
## Resolution Rules
|
|
|
|
Built-ins:
|
|
|
|
1. manifest producer outputs (when present)
|
|
2. canonical session-path fallback
|
|
|
|
Configured sources (`narratio.artifact.*`):
|
|
|
|
- resolve only through runtime catalog availability.
|
|
|
|
Extraction sources (`narratio.extraction.*`):
|
|
|
|
- use the shared typed bundle evidence inspection in `extraction_evidence.go`;
|
|
- require a current successful extract record with the exact configured source,
|
|
compatible contract and Notarius provenance, a confined regular durable
|
|
payload, matching checksum, and the current resolved trimmed-transcript
|
|
identity;
|
|
- remain unavailable unless catalog hydration receives valid evidence. Resume
|
|
treats absent or obsolete evidence as a rerun decision and unsafe evidence as
|
|
an error; and
|
|
- are never inferred by scanning the Notarius bundle directory.
|
|
|
|
Previous-session sources (`narratio.previous_session.artifact.*`):
|
|
|
|
- resolve only from local `previous/` cache state;
|
|
- prefer manifest-backed previous-input paths;
|
|
- fallback to existing previous-cache filesystem paths.
|
|
|
|
Source absence is evaluated by the consuming artifact input. An optional input
|
|
is omitted from that invocation; a required input fails resolution. This is
|
|
separate from a stage's lifecycle outcome.
|
|
|
|
Validation by content type:
|
|
|
|
- transcript JSON built-ins: JSON with top-level `segments` array;
|
|
- transcript Markdown built-ins: non-empty text file;
|
|
- bounds built-in: valid JSON;
|
|
- configured/previous-session artifact files: non-empty text file.
|
|
|
|
## Previous Requirement Collection
|
|
|
|
`CollectPreviousArtifactRequirements`:
|
|
|
|
- scans the effective configured artifact set;
|
|
- extracts only canonical previous-session sources;
|
|
- deduplicates by artifact key;
|
|
- merges required and optional references (required wins);
|
|
- returns deterministic ordering and source locations.
|
|
|
|
## Current-State Helpers
|
|
|
|
Artifacts package owns shared remote current-state loading mechanics used by
|
|
restore, status and validation checks, and previous-cache planning.
|
|
|
|
For a new-protocol current state, the pointer-selected immutable commit is the
|
|
complete restore authority. Callers receive its declared object identities and
|
|
must not supplement them by listing mutable session prefixes. The legacy reader
|
|
is intentionally separate and remains migration-only support.
|
|
|
|
The reader opens each small control object directly and enforces owner-specific
|
|
limits before decoding: 64 KiB for the mutable commit pointer, 4 MiB for the
|
|
immutable commit manifest, and 8 MiB for the selected session manifest. Legacy
|
|
compatibility applies a 4 KiB limit to `current/run_id.txt` and the same 8 MiB
|
|
manifest limit to `current/manifest.json`. These are exposed as
|
|
`MaxCurrentCommitPointerBytes`, `MaxRemoteCommitManifestBytes`,
|
|
`MaxRemoteSessionManifestBytes`, `MaxLegacyCurrentRunPointerBytes`, and
|
|
`MaxLegacyCurrentManifestBytes`.
|
|
|
|
Each read uses the generation and size metadata returned with its opened body.
|
|
Actual bytes remain subject to a limit-plus-one read even if size metadata is
|
|
absent or inaccurate. Immutable selections then retain their declared-size,
|
|
checksum, generation, and identity checks. No current-state control object is
|
|
downloaded through a temporary file.
|
|
|
|
Core helpers:
|
|
|
|
- `LoadCurrentState`
|
|
- `ValidateCurrentStateIdentity`
|
|
- `RemoteCommitManifest` and `CurrentCommitPointer`
|
|
|
|
Typed missing-state errors:
|
|
|
|
- `CurrentRunPointerMissingError` (`ErrCurrentRunPointerMissing`)
|
|
- `CurrentManifestMissingError` (`ErrCurrentManifestMissing`)
|
|
|
|
Identity validation supports caller-provided expectations:
|
|
|
|
- expected campaign;
|
|
- expected session ID;
|
|
- expected run ID, or pointer/manifest run-ID consistency check.
|
|
|
|
Caller policy is intentionally outside artifacts helpers:
|
|
|
|
- some callers fail on missing current state;
|
|
- some callers downgrade missing state to status/findings;
|
|
- some callers skip optional behavior when state is missing.
|
|
|
|
## Key Path Helpers
|
|
|
|
`internal/artifacts/paths.go` and S3-key helpers define canonical helpers for:
|
|
|
|
- session/work/run paths;
|
|
- previous-cache paths;
|
|
- spool/cache paths;
|
|
- S3 session/run/current-state key layout.
|
|
|
|
New publication creates run-scoped immutable objects, including
|
|
`runs/{run_id}/commit.json` and `runs/{run_id}/session-manifest.json`. The sole
|
|
mutable selector is `current/commit-pointer.json`; readers verify its selected
|
|
commit and declared object generations/checksums. Legacy current-pair loading
|
|
is confined to `current_state_legacy.go` for migration only.
|
|
|
|
Campaign, session, and Narratio run IDs are validated as portable opaque
|
|
segments at configuration and artifact boundaries before they can be used in a
|
|
workspace or S3 namespace. Previous-artifact destinations remain typed,
|
|
multi-segment relative paths and are confined beneath `previous/artifacts`; they
|
|
are not treated as opaque identifiers.
|
|
|
|
See [Workspace Internals](workspace.md) for how callers consume local helpers
|
|
and [Operations](../operations.md#local-state-layout) for the authoritative
|
|
physical layout.
|
|
|
|
## Invariants
|
|
|
|
- source ID formats are stable contracts;
|
|
- artifact resolution is deterministic and manifest-aware;
|
|
- extraction sources are available only from a compatible successful manifest
|
|
record;
|
|
- previous-session source resolution in `analyze` is local-only;
|
|
- remote current-state key construction remains centralized in artifacts helpers.
|
|
|
|
## Implementation And Tests
|
|
|
|
- Registry and resolution: `internal/artifacts/artifact_resolver.go`,
|
|
`internal/artifacts/catalog.go`, `internal/artifacts/transcripts.go`,
|
|
`internal/artifacts/extraction_catalog.go`,
|
|
`internal/artifacts/extraction_evidence.go`,
|
|
`internal/artifacts/extraction_input.go`
|
|
- Current state: `internal/artifacts/current_state.go`,
|
|
`internal/artifacts/current_state_commit.go`,
|
|
`internal/artifacts/current_state_legacy.go`
|
|
- Paths and keys: `internal/artifacts/paths.go`,
|
|
`internal/artifacts/s3_keys.go`
|
|
- Previous requirements: `internal/artifacts/previous_requirements.go`
|
|
- Tests: `internal/artifacts/artifact_resolver_test.go`,
|
|
`internal/artifacts/catalog_test.go`,
|
|
`internal/artifacts/extraction_catalog_test.go`,
|
|
`internal/artifacts/current_state_test.go`,
|
|
`internal/artifacts/paths_model_test.go`,
|
|
`internal/artifacts/previous_requirements_test.go`
|