Files
commandkit/docs/roadmap/invalid-configuration-outcomes.md

13 KiB

Invalid configuration outcomes

Status — proposed implementation roadmap. The user has accepted the targeted integration behavior. The decisions at the end identify remaining policy and API choices for the implementation plan.

Purpose

Make standard ckconfig and ckprompt commands return usage status 2 for identifiable invalid configuration input, while returning operational status 1 for cancellation and other operational failures. Preserve actionable upstream diagnostics and supported typed error identity through the integration boundary.

CommandKit owns this CLI classification. ConfigKit and PromptKit own loading, validation, diagnostic meaning, and their public error contracts. Application commands own their domain outcomes. These responsibilities follow the architecture policy.

Current baseline

The reviewed baseline is CommandKit commit 1e92a87706e44684ee34395245e5b633474677e0, following the publication of ckconfig v0.7.0 and ckprompt v0.7.0. Both integration modules select core v0.6.0 and ConfigKit v1.2.0. ckprompt selects ckconfig v0.7.0 and PromptKit v0.21.0. Their manifests own the dependency selections.

The consumer report reproduced malformed operator YAML in Audita, including its automatic prompts.yml companion and an explicitly selected application configuration. Delegated inspection returned yaml.invalid with status 1, while Audita's processing command returned the same diagnostic with status 2. The report's reproduction is retained in Audita's upstream handoff.

The review confirmed the behavior through Audita's existing regression tests and a temporary public-API reproduction against the aligned CommandKit workspace. The latter covered catalog listing and all five configuration operations in both the application and PromptKit scopes, including wrapped ConfigKit loading errors and already-canceled contexts.

  • FromConfigKit and the PromptKit adapters preserve the ConfigKit loading error and adapt its diagnostic evidence.
  • configFailure and definitionFailure construct operational outcomes for the affected failures. Configuration comparison and validation also reach the former helper.
  • ConfigKit's public LoadError represents several kinds of loading failure, including malformed YAML, acquisition errors, and schema or callback problems. Its public diagnostic codes provide the classification evidence. Its error contract intentionally exposes safe diagnostics and preserves its own typed identity across CommandKit adaptation.
  • config sources can report discovered source evidence after failed loading, and config validate can report failed validation. Cancellation observed before acquisition leaves stdout empty in the reviewed command paths.

The current CLI contract assigns loading failures status 1, with more specific statements in the configuration comparison, requirements, validation, and smoke sections. This roadmap proposes a deliberate change to that published status behavior.

Targeted end state

Evidence available to the integration Selected outcome
ConfigKit error diagnostics that identify invalid configuration input under the selected mapping Usage, status 2
Caller cancellation or deadline observed at an existing operation checkpoint, or present in the returned error chain Operational, status 1
I/O, schema or wiring defects, callback execution failures, and other recognized operational failures Operational, status 1
An unrecognized failure, including a loading error with insufficient classification evidence Operational, status 1
Successful acquisition with warnings Follow the command's successful-result behavior

Classification uses upstream structured evidence. Diagnostic text, filenames, and source labels serve presentation and provenance. The exact input-error code mapping is proposed under Open Decisions.

The selected outcome retains the original returned cause, including ordinary wrapping and joined errors, so supported errors.Is and errors.As queries continue to work. Existing explanations, source context, diagnostic visibility, and operation-specific reporting lifecycles apply to the newly classified failures. Shared rendering and delivery failures follow the core output-failure rules, including their effect on usage outcomes.

Apply the classification to configuration acquisition failures throughout the standard suites, including failures returned by application callbacks when they carry the supported ConfigKit evidence. CLI syntax, resource lookup, request execution, and application-specific error semantics retain their owning contracts. The work concerns interpreting returned failure evidence at the CLI boundary.

Work needed

1. Define the configuration-failure classification

  • Inventory ConfigKit v1.2.0's public diagnostic codes and their documented meaning. Record the selected input-error cases explicitly using upstream constants. Review all code families, including composition, imports, inheritance, versions, overrides, secrets, and configured limits.
  • Use error-severity diagnostics as failure evidence. Warning records can accompany a failure or a successful result.
  • Handle wrapped LoadError values and standard joined error branches. A failure selects usage only when its failing evidence is classified as invalid input. An operational or unclassified failing branch makes the combined result operational. Preserve the original cause tree.
  • Account for cancellation and deadlines before choosing usage. When an existing checkpoint observes cancellation alongside an acquisition failure, preserve both causes using the established error-joining approach.
  • Keep classification within integration ownership, using the sharing approach selected below. Existing core outcome constructors and reporting provide the required mechanics.

2. Apply it throughout ckconfig

Wire the classification into acquisition failure handling for config show, config sources, config validate, config explain, and config diff.

Follow each operation's current preparation, aggregation, and reporting flow. In particular, source and validation reports can accompany a failed load. Detailed validation filters control displayed findings, while the complete underlying failure supplies classification evidence. Comparison must classify a failure on either loading side and retain the side-specific diagnostic context.

Include native ConfigKit scopes, PromptKit configuration scopes, and custom scope callbacks returning the supported evidence. Scope selection and input preflight continue through their existing contracts.

3. Apply it throughout ckprompt

Cover engine-acquisition failures in catalogs, named prompt/profile/backend inspection, requirements discovery, concrete request validation, prepared-request preview, individual smoke operations, and aggregate smoke testing.

Acquisition evidence may arrive through loaders or application request builders. Use the same classification policy wherever it reaches the standard command's failure boundary. For joined aggregate smoke errors, an operational or unclassified failure takes precedence over input failures. Follow the existing aggregate execution and report lifecycle when selecting the final outcome.

4. Verify observable integration behavior

Extend focused public-command tests where they provide regression protection. Use upstream errors as fixtures to establish CommandKit's classification and preservation behavior, following the integration testing policy.

Cover these distinct risks with a proportionate set of cases.

  • Malformed main and companion configuration produce status 2 with yaml.invalid and the typed ConfigKit cause available through wrapping.
  • Representative decoding and selected semantic-validation errors receive the intended classification. Operational and unrecognized loading diagnostics receive status 1, including mixed failing evidence and warnings alongside errors.
  • The five configuration operations and affected PromptKit command families use the policy, including comparison-side failures and aggregate smoke outcomes.
  • Cancellation and deadlines receive status 1 and preserve their supported identity. Already-canceled invocations retain the reviewed empty-output behavior. A failed acquisition combined with cancellation preserves both causes.
  • Source and validation reports retain their operation-specific failure evidence. Detailed display filters do not determine failure classification.
  • Output failures continue to follow the shared reporting contract when the command has selected usage status 2.

Run the complete deterministic suites independently for each changed integration and with the aligned local workspace as described in the developer guide. Exercise the maintained examples where their existing assertions cover affected outcomes. The Audita reproduction supplies supporting consumer evidence. CommandKit-owned tests provide the maintained regression coverage.

5. Document and prepare the integration releases

Update the CLI status table and affected operation sections with the implemented classification. Update any affected exported GoDoc and consumer guidance under their canonical owners. Explain the classification boundary and fallback through the chosen API's documentation.

Prepare ckconfig v0.8.0 followed by ckprompt v0.8.0, with ckprompt selecting the published ckconfig version. The status-contract change warrants minor releases under the compatibility policy. The release notes should explain that identifiable invalid configuration now selects status 2, alongside the operational classification and preserved error evidence. Follow the release procedure when publication is authorized. Implementation completion and publication are tracked separately.

Completion evidence

Implementation is complete when the accepted classification is exercised through the public integration commands, the focused regression cases and selected integration suites pass, and the canonical documentation describes the implemented behavior. Review the final diff for the agreed scope and retire this roadmap under the normal documentation lifecycle after recording any unfinished release work.

Open Decisions

Exact diagnostic-code mapping

Recommendation. Use an explicit mapping based on ConfigKit v1.2.0's public code semantics. Include YAML-content and decoding errors, explicit version.error, resolve.error, and validate.error findings, and other codes whose documented meaning clearly identifies invalid configuration content. Classify schema/options defects, acquisition failures, hook execution failures, invalid callback paths or records, and validation mutation as operational. Keep ambiguous limit, secret, and source-selection cases operational until their individual meaning supports input classification. Review and record those cases when preparing the implementation plan.

This uses the evidence upstream already supplies and distinguishes reported configuration rejection from callback machinery failing. Source evidence alone may be insufficient to establish whether application-supplied data came from an operator, so the rule follows the meaning recorded in upstream diagnostics.

Alternative. Add an application-supplied classification hook for cases that depend on source ownership or application policy, alongside the built-in mapping. Applications can then identify input failures using knowledge their loaders possess. This supports genuinely contextual cases, but expands the public integration contracts and creates additional configuration and precedence rules. The current reproduction is addressable through existing typed diagnostics, so the additional hook has limited demonstrated value in this scope.

Shared classification interface

Recommendation. Place one narrowly documented classification function in ckconfig and have ckprompt call it through its existing dependency. Keep command explanation construction and reporting in their owning handlers. Select its exact signature during implementation planning based on the needed cause and cancellation inputs. A shared implementation reduces drift across the two standard families and fits ckconfig's ownership of ConfigKit adaptation. The public function adds an API maintenance obligation.

Alternative. Maintain small private classifiers in both integration modules, with matching behavioral cases. This keeps the helper private and supports package-local evolution, but duplicates the diagnostic mapping and joined-error rules. Future upstream code changes would require coordinated edits in both modules. Prefer this approach if the shared signature would expose more policy or machinery than the demonstrated integration need warrants.