Conformance report¶
This file is generated by scripts/conformance_report.py from docs/requirements.yaml,
docs/reviews.yaml and a pytest-json-report run. Do not edit it by hand. How the report works
is described in docs/conformance.md.
It describes the commit that contains it. It holds nothing that varies between runs of the same commit, and CI regenerates it at every commit and fails if a single byte differs.
How to read it¶
Each row is a requirement in the register. The register says how each requirement is assessed: by automated test, by human review, or by both (docs/specs/evaluation.md). Each assessment gives its own verdict, and the two are never combined into one. A requirement not assessed one way reads not assessed in that column.
The test verdict applies the traceability rule:
- substantiated — at least one test marked
@pytest.mark.requirement("<ID>")passed in this run and none failed; - contradicted — a test marked with the identifier failed;
- unsubstantiated — no test claims the identifier, or every claiming test was skipped.
The review verdict reads the latest review recorded for the requirement in docs/reviews.yaml:
- substantiated — the reviewer found the requirement met at the commit named;
- contradicted — the reviewer found it not met;
- unsubstantiated — no review has been recorded.
Substantiated by a test means an example-based test exercised one path through the requirement. Substantiated by a review means one named person examined the scope described in the review record at one commit. Either way it is not a conformance claim, an evaluation result, or evidence that the requirement is met in general. Rows marked unsubstantiated are the honest default and the point of publishing this file: the Blueprint's Appendix D records implementers' stated intentions; this table records what a test run and the recorded reviews actually showed.
source: blueprint rows use the Blueprint's own identifiers and titles (§7). source: project
rows are identifiers this project introduced (the R3.n decomposition from agents/advisor-standards, sub-IDs
such as C13.1 that split a Blueprint requirement so a row
claims only what its tests show, and DD-* for the contract itself); the Blueprint does not use
them. A Blueprint row whose sub-IDs are substantiated is not thereby substantiated itself.
Summary¶
| Verdict | By test | By review |
|---|---|---|
| substantiated | 29 | 0 |
| contradicted | 0 | 0 |
| unsubstantiated | 25 | 8 |
| ID | Source | Title | Test verdict | Tests | Review verdict | Review |
|---|---|---|---|---|---|---|
| R1 | blueprint | Recommend and select repositories meeting funder, journal, institutional and community requirements | ✅ substantiated | test_repository_agent::test_a_generalist_is_offered_after_a_repository_of_the_discipline, test_repository_agent::test_a_kind_of_data_alone_does_not_recommend_a_repository, test_repository_agent::test_a_mandated_repository_is_found_by_identifier_and_never_shortlisted_around, test_repository_agent::test_a_members_only_repository_is_left_out_unless_the_researcher_is_a_member, test_repository_agent::test_a_repository_that_takes_no_deposits_is_never_recommended, test_repository_agent::test_certification_carries_no_weight, test_repository_agent::test_commercial_repositories_are_flagged_and_listed_after_equals_not_dropped, test_repository_agent::test_mandated_repositories_are_described_with_the_needs_they_miss, test_repository_agent::test_shortlist_is_grounded_and_every_fact_is_evidence, test_repository_agent::test_the_researchers_institutional_repository_is_offered_beyond_the_cap, test_repository_shortlists::test_a_workshop_shortlist_keeps_to_the_datasets_discipline[cylinder-flow.json-include2-exclude2], test_repository_shortlists::test_a_workshop_shortlist_keeps_to_the_datasets_discipline[moorland-sensors.json-include0-exclude0], test_repository_shortlists::test_a_workshop_shortlist_keeps_to_the_datasets_discipline[perovskite-xrd-sem.json-include1-exclude1] |
not assessed | — |
| R2 | blueprint | Generate FAIR-aligned and discipline-appropriate metadata using controlled vocabularies, ontologies and community standards | ⬜ unsubstantiated | — | not assessed | — |
| R3 | blueprint | Suggest appropriate vocabularies, ontologies and open data formats | ✅ substantiated | test_fairsharing::test_standards_over_the_real_snapshot_recommends_for_the_soil_sample, test_standards_harness::test_standards_recommends_across_kinds_with_evidence |
⬜ unsubstantiated | — |
| R3.1 | project (under R3) | Controlled vocabulary recommendations for concept linkage | ✅ substantiated | test_fairsharing::test_standards_over_the_real_snapshot_recommends_for_the_soil_sample, test_standards_harness::test_standards_recommends_across_kinds_with_evidence |
not assessed | — |
| R3.2 | project (under R3) | Ontology recommendations for schema or model alignment | ✅ substantiated | test_fairsharing::test_standards_over_the_real_snapshot_recommends_for_the_soil_sample, test_standards_harness::test_standards_recommends_across_kinds_with_evidence |
not assessed | — |
| R3.3 | project (under R3) | Open data format recommendations | ✅ substantiated | test_fairsharing::test_standards_over_the_real_snapshot_recommends_for_the_soil_sample, test_standards_harness::test_standards_recommends_across_kinds_with_evidence |
not assessed | — |
| R3.4 | project (under R3) | Field-level format standards (date/time, units, codes) | ✅ substantiated | test_standards_harness::test_standards_recommends_across_kinds_with_evidence |
not assessed | — |
| R3.5 | project (under R3) | Support for emerging standards; no hard-coded standards list; that no list is hard-coded is not tested (#5) | ✅ substantiated | test_standards_harness::test_deprecated_records_are_dropped_and_emerging_ones_kept |
not assessed | — |
| R3.6 | project (under R3) | Explicit abstention where no suitable resource exists | ✅ substantiated | test_standards_harness::test_standards_abstention_reasons_are_distinct |
not assessed | — |
| R4 | blueprint | Validate metadata quality and produce standardised records assessed against community best practices | ⬜ unsubstantiated | — | not assessed | — |
| R4.1 | project (under R4) | Completeness of a metadata record scored against declared criteria, each finding explained | ✅ substantiated | test_quality::test_complete_record_scores_one_and_every_finding_is_informational, test_quality::test_review_is_grounded_on_the_input_and_passes_the_linter |
not assessed | — |
| R5 | blueprint | Draft data documentation including READMEs and data dictionaries at publication | ⬜ unsubstantiated | — | not assessed | — |
| R6 | blueprint | Enable interoperable metadata across institutions and Data Director instances | ⬜ unsubstantiated | — | not assessed | — |
| R7 | blueprint | Support assignment of persistent identifiers (PIDs) at point of publication | ⬜ unsubstantiated | — | not assessed | — |
| R8 | blueprint | Verify data sharing at publication against Data Management Plan (DMP) commitments | ⬜ unsubstantiated | — | not assessed | — |
| R9 | blueprint | Retrofit FAIR-aligned metadata for already-published datasets and supplementary materials | ⬜ unsubstantiated | — | not assessed | — |
| R10 | blueprint | Track provenance of AI agent actions on data and metadata | ⬜ unsubstantiated | — | not assessed | — |
| R10.1 | project (under R10) | Every invocation is stored, traced and packaged as a Process Run Crate; PROV-O alignment and audit-trail completeness are not exercised (#26) | ✅ substantiated | test_standards_harness::test_model_call_after_retrieval_passes_linter_and_records_tokens, test_harness::test_a_reused_invocation_id_is_refused_and_the_stored_run_kept, test_harness::test_every_invocation_is_stored_traced_and_crated, test_harness::test_linter_runs_offline_over_written_spans, test_harness::test_retrieval_agent_passes_when_it_rests_only_on_what_it_retrieved, test_harness::test_the_crate_names_the_human_beside_the_agent, test_harness::test_the_crate_names_the_issuer_that_vouched_for_the_human |
not assessed | — |
| R11 | blueprint | Link datasets to related research outputs including publications, people, grants and software | ⬜ unsubstantiated | — | not assessed | — |
| R12 | blueprint | Discover related datasets relevant to a research project | ⬜ unsubstantiated | — | not assessed | — |
| C1 | blueprint | Security | not assessed | — | ⬜ unsubstantiated | — |
| C2 | blueprint | Privacy | ⬜ unsubstantiated | — | ⬜ unsubstantiated | — |
| C3 | blueprint | Compliance | not assessed | — | ⬜ unsubstantiated | — |
| C4 | blueprint | Sovereignty | not assessed | — | ⬜ unsubstantiated | — |
| C5 | blueprint | Interoperability — open APIs and established metadata standards | ⬜ unsubstantiated | — | not assessed | — |
| C5.1 | project (under C5) | The workbench exposes an open API (A2A agent card and message send, AG-UI run and replay); established metadata standards, exchange with repositories and a harmonised API between instances are not exercised (#24) | ✅ substantiated | test_transport::test_a2a_message_send_returns_envelope_artifact, test_transport::test_agent_card_lists_one_skill_per_agent_from_the_manifest, test_transport::test_agui_run_and_replay_emit_started_and_finished |
not assessed | — |
| C6 | blueprint | Accessibility | ⬜ unsubstantiated | — | ⬜ unsubstantiated | — |
| C7 | blueprint | Reliability | ⬜ unsubstantiated | — | not assessed | — |
| C8 | blueprint | Resiliency | ⬜ unsubstantiated | — | not assessed | — |
| C9 | blueprint | Availability | ⬜ unsubstantiated | — | not assessed | — |
| C10 | blueprint | Performance | ⬜ unsubstantiated | — | not assessed | — |
| C11 | blueprint | Scalability | ⬜ unsubstantiated | — | not assessed | — |
| C12 | blueprint | Data Governance | ⬜ unsubstantiated | — | ⬜ unsubstantiated | — |
| C13 | blueprint | AI Governance | ⬜ unsubstantiated | — | not assessed | — |
| C13.1 | project (under C13) | Every agent action is stored, traced and attributed to the agent identity that took it | ✅ substantiated | test_standards_harness::test_model_call_after_retrieval_passes_linter_and_records_tokens, test_delegation::test_a_delegated_child_is_stored_with_its_lineage_and_recorded_by_the_parent, test_harness::test_conductor_records_the_declared_mode_on_envelope_and_root_span, test_harness::test_every_invocation_is_stored_traced_and_crated, test_harness::test_retrieval_agent_passes_when_it_rests_only_on_what_it_retrieved |
not assessed | — |
| C13.2 | project (under C13) | The policy gate refuses an agent the profile does not permit and refers an action requiring approval to a named human role; what happens after referral is not exercised (#16) | ✅ substantiated | test_harness::test_action_requiring_approval_is_referred, test_harness::test_disabled_agent_fails_with_problem_details |
not assessed | — |
| C13.3 | project (under C13) | A person approves, edits then approves, or rejects each document version an agent drafts, with a reason for any departure from the draft; the harness records who decided, when and why, and refuses a decision on text other than the undecided latest version; that review is mandatory before deposit is not exercised (#197) | ✅ substantiated | test_document_decisions::test_a_decision_acts_for_the_person_signed_in_and_is_in_the_log, test_project_documents::test_a_decision_on_a_stale_version_is_refused_and_recorded, test_project_documents::test_a_malformed_decision_is_refused_and_records_nothing[given0-rejecting needs a reason], test_project_documents::test_a_malformed_decision_is_refused_and_records_nothing[given1-rejecting needs a reason], test_project_documents::test_a_malformed_decision_is_refused_and_records_nothing[given2-approving an edit needs a reason], test_project_documents::test_a_malformed_decision_is_refused_and_records_nothing[given3-can be approved, not rejected], test_project_documents::test_a_malformed_decision_is_refused_and_records_nothing[given4-at most], test_project_documents::test_a_malformed_decision_is_refused_and_records_nothing[given5-not 'maybe'], test_project_documents::test_a_malformed_decision_is_refused_and_records_nothing[given6-UUIDv7], test_project_documents::test_a_malformed_decision_is_refused_and_records_nothing[given7-SHA-256], test_project_documents::test_a_plain_approval_needs_no_reason_and_is_final, test_project_documents::test_a_rejection_keeps_its_reason, test_project_documents::test_editing_then_approving_writes_the_edit_and_its_approval_together, test_project_runs::test_a_caller_may_not_name_a_project, test_project_runs::test_a_delegated_childs_draft_is_kept_in_its_parents_project, test_project_runs::test_a_draft_in_a_project_is_given_its_facts_and_kept_as_a_proposed_version, test_project_runs::test_a_draft_that_revises_no_version_of_the_project_fails_the_run, test_project_runs::test_a_fact_the_project_does_not_have_is_removed_and_the_drafter_is_told_none, test_project_runs::test_a_failed_draft_is_not_kept, test_project_runs::test_outside_a_project_nothing_is_filled_or_recorded, test_project_runs::test_the_agent_is_never_told_its_project |
not assessed | — |
| C14 | blueprint | Explainability — human-readable explanations for recommendations and decisions | ⬜ unsubstantiated | — | not assessed | — |
| C14.1 | project (under C14) | Every recommendation and decision carries a human-readable rationale, and a model rationale naming an identity that was not retrieved is replaced; challenge and override by human roles and a reasoning log are not exercised (#25) | ✅ substantiated | test_quality::test_review_is_grounded_on_the_input_and_passes_the_linter, test_explain::test_model_output_with_ungrounded_identifier_falls_back_to_template, test_explain::test_template_rationale_names_resource_kind_and_target, test_standards_harness::test_standards_recommends_across_kinds_with_evidence |
not assessed | — |
| C15 | blueprint | Data and Metadata Quality — quality checks on outputs, human review mandatory | ⬜ unsubstantiated | — | ⬜ unsubstantiated | — |
| C16 | blueprint | Measurability | ⬜ unsubstantiated | — | not assessed | — |
| C17 | blueprint | Affordability | ⬜ unsubstantiated | — | not assessed | — |
| P14 | blueprint | Low Impact — energy footprint captured in provenance records | ⬜ unsubstantiated | — | not assessed | — |
| DD-OUTCOME | project | Outcome vocabulary (succeeded, abstained, referred, failed, suspended) with reason codes; suspended is not yet produced (#16) |
✅ substantiated | test_dmp_agent::test_no_commitments_abstains, test_documentation_agent::test_abstains_without_an_inventory_or_for_another_kind[asked0-insufficient_input], test_documentation_agent::test_abstains_without_an_inventory_or_for_another_kind[asked1-outside_agent_scope], test_ethics_agent::test_nothing_to_go_on_abstains, test_fairness_agent::test_neither_identifier_nor_title_abstains, test_files_agent::test_an_empty_inventory_abstains, test_linking_agent::test_no_identifiers_abstains, test_metadata_agent::test_abstains_without_a_repository_or_for_another_kind_or_standard, test_policy_agent::test_a_context_with_no_actor_abstains, test_policy_agent::test_actors_with_no_record_are_listed_or_abstain_when_none_has_one, test_policy_agent::test_an_unreachable_registry_is_not_an_empty_result, test_quality::test_empty_record_abstains, test_repository_agent::test_an_unreachable_registry_is_not_an_empty_result, test_repository_agent::test_needs_with_nothing_to_look_up_abstain, test_repository_agent::test_no_mandated_repository_found_abstains, test_repository_agent::test_nothing_fitting_abstains_rather_than_returning_an_empty_list, test_rights_agent::test_nothing_to_go_on_abstains, test_standards_harness::test_standards_abstention_reasons_are_distinct, test_hello::test_unknown_language_abstains_rather_than_raising, test_orchestrator_stub::test_an_unroutable_message_abstains_naming_what_it_can_route, test_harness::test_action_requiring_approval_is_referred, test_harness::test_stub_abstains_on_a_message, test_remote::test_an_agent_that_exceeds_its_timeout_is_a_failed_envelope, test_remote::test_an_agent_that_goes_away_after_registration_is_a_failed_envelope |
not assessed | — |
| DD-GROUNDING | project | The agent's account is consistent — its declared retrieval precedes any model call and every identity a payload asserts appears in its declared retrieval; that the agent retrieved anything is not shown by the linter (docs/specs/grounding.md) | ✅ substantiated | test_explain::test_model_output_with_ungrounded_identifier_falls_back_to_template, test_standards_harness::test_a_relabelled_record_is_withheld, test_standards_harness::test_chat_before_retrieval_fails_g1, test_standards_harness::test_model_call_after_retrieval_passes_linter_and_records_tokens, test_standards_harness::test_ungrounded_identifier_in_output_is_withheld, test_grounding::test_e1_evidence_content_must_hash_to_its_content_hash[delegation], test_grounding::test_e1_evidence_content_must_hash_to_its_content_hash[input_only], test_grounding::test_e1_evidence_content_must_hash_to_its_content_hash[none], test_grounding::test_e1_evidence_content_must_hash_to_its_content_hash[retrieval], test_grounding::test_g1_chat_before_or_without_retrieval, test_grounding::test_g2_asserted_identity_must_match_a_retrieval_on_id_and_hash, test_grounding::test_g2_succeeded_payload_resting_on_nothing, test_grounding::test_g3_and_g4_evidence_agrees_with_trace_and_payload, test_grounding::test_retrieval_mode_passes_when_everything_was_retrieved_and_cited, test_harness::test_chat_before_retrieval_fails_g1, test_harness::test_identity_not_retrieved_is_withheld, test_harness::test_none_agent_that_calls_a_model_is_withheld, test_harness::test_retrieval_agent_passes_when_it_rests_only_on_what_it_retrieved, test_remote::test_agent_spans_join_the_conductors_trace_under_invoke_agent |
not assessed | — |
| DD-GROUNDING-MODE | project | An agent declares its grounding mode; the conductor records it and the linter applies that mode's rules | ✅ substantiated | test_dmp_agent::test_flags_missing_unplanned_and_changed_sharing, test_documentation_agent::test_drafts_a_readme_from_the_inventory_and_marks_the_rest, test_ethics_agent::test_flags_field_names_and_asks_about_consent, test_fairness_agent::test_checks_the_record_and_suggests_retrofits, test_files_agent::test_judges_each_file_of_the_inventory_by_its_media_type, test_linking_agent::test_recognises_checks_and_normalises_identifiers, test_metadata_agent::test_drafts_a_datacite_record_with_every_property_the_repository_asks_for, test_quality::test_review_is_grounded_on_the_input_and_passes_the_linter, test_rights_agent::test_lists_fitting_licences_and_puts_restricted_access_first, test_standards_factory::test_the_spec_round_trips_through_the_manifest, test_hello::test_greeting_is_grounded_on_the_input_and_passes_the_linter, test_grounding::test_envelope_mode_must_match_the_root_span, test_grounding::test_input_only_forbids_retrieval_and_foreign_citations, test_grounding::test_input_only_permits_chat_without_retrieval, test_grounding::test_missing_or_unknown_mode_fails, test_grounding::test_none_mode_forbids_chat, test_grounding::test_retrieval_mode_passes_when_everything_was_retrieved_and_cited, test_harness::test_conductor_records_the_declared_mode_on_envelope_and_root_span, test_harness::test_input_only_agent_may_call_a_model_without_retrieving, test_harness::test_input_only_agent_that_retrieves_is_withheld, test_harness::test_none_agent_that_calls_a_model_is_withheld, test_remote::test_a_span_carrying_a_conductor_attribute_is_withheld[dd.grounding_mode-none], test_remote::test_a_span_carrying_a_conductor_attribute_is_withheld[dd.input_hash-0000000000000000000000000000000000000000000000000000000000000000], test_remote::test_a_span_outside_the_invocations_trace_is_withheld, test_contract::test_envelope_requires_a_grounding_mode, test_contract::test_input_grounded_success_must_cite_only_the_input, test_contract::test_retrieval_success_requires_non_empty_grounded_on |
not assessed | — |
| DD-GROUNDED-PAYLOAD | project | Every payload declares what it rests on (grounded_on); a payload the linter cannot ground is a violation, never a pass | ✅ substantiated | test_dmp_agent::test_flags_missing_unplanned_and_changed_sharing, test_documentation_agent::test_drafts_a_readme_from_the_inventory_and_marks_the_rest, test_ethics_agent::test_flags_field_names_and_asks_about_consent, test_fairness_agent::test_checks_the_record_and_suggests_retrofits, test_files_agent::test_judges_each_file_of_the_inventory_by_its_media_type, test_linking_agent::test_recognises_checks_and_normalises_identifiers, test_metadata_agent::test_drafts_a_datacite_record_with_every_property_the_repository_asks_for, test_policy_agent::test_each_actor_is_matched_and_every_record_is_evidence, test_policy_agent::test_policies_that_differ_are_flagged_and_grounded_on_each, test_repository_agent::test_shortlist_is_grounded_and_every_fact_is_evidence, test_rights_agent::test_lists_fitting_licences_and_puts_restricted_access_first, test_hello::test_greeting_is_grounded_on_the_input_and_passes_the_linter, test_grounding::test_payload_without_grounded_on_is_a_violation_not_a_pass, test_grounding::test_payload_without_schema_class_is_a_violation, test_harness::test_a_run_names_the_contract_and_the_class_schemas_it_was_checked_against, test_harness::test_identity_not_retrieved_is_withheld, test_contract::test_a_payload_of_any_class_validates_with_grounded_on, test_contract::test_payload_without_grounded_on_or_schema_class_is_rejected |
not assessed | — |
| DD-INPUT-ACCEPTS | project | An agent declares the input classes it accepts; a request carrying another, or an input its class schema does not admit, is a failed outcome with Problem Details | ✅ substantiated | test_standards_factory::test_an_input_class_it_did_not_declare_is_refused, test_hello::test_an_input_class_it_did_not_declare_is_refused, test_harness::test_an_agent_may_accept_several_input_classes, test_harness::test_input_of_an_unaccepted_class_is_a_failed_outcome_and_the_agent_is_not_run, test_harness::test_input_that_breaks_its_class_schema_is_not_accepted_and_the_agent_is_not_run, test_contract::test_an_input_of_any_class_validates_and_names_its_class, test_contract::test_input_without_schema_class_is_rejected, test_transport::test_a2a_input_mismatch_is_an_envelope_not_a_transport_failure |
not assessed | — |
| DD-EVIDENCE | project | Evidence names a registered canonicalisation and its hash is reproducible from the source | ✅ substantiated | test_policy_agent::test_each_actor_is_matched_and_every_record_is_evidence, test_repository_agent::test_shortlist_is_grounded_and_every_fact_is_evidence, test_standards_harness::test_a_relabelled_record_is_withheld, test_grounding::test_e1_evidence_content_must_hash_to_its_content_hash[delegation], test_grounding::test_e1_evidence_content_must_hash_to_its_content_hash[input_only], test_grounding::test_e1_evidence_content_must_hash_to_its_content_hash[none], test_grounding::test_e1_evidence_content_must_hash_to_its_content_hash[retrieval], test_harness::test_fairsharing_canonicalisation_is_order_independent_and_projected, test_harness::test_input_hash_is_stable_across_key_order_and_sensitive_to_list_order, test_sources::test_a_file_that_does_not_match_its_pin_is_a_configuration_error, test_sources::test_conductor_records_a_verified_result, test_sources::test_conductor_withholds_a_result_citing_a_record_the_source_does_not_hold, test_sources::test_config_resolves_paths_beside_the_file, test_sources::test_evidence_from_a_source_with_no_copy_is_unresolved_not_verified, test_sources::test_evidence_matching_the_source_is_verified, test_sources::test_input_and_delegation_evidence_is_left_to_the_linter_outside_retrieval, test_sources::test_s1_a_cited_record_the_source_does_not_hold, test_sources::test_s2_content_consistent_with_its_hash_but_not_with_the_source, test_contract::test_envelope_hash_is_reproducible_from_the_stored_document, test_contract::test_evidence_content_reproduces_its_hash[dd-envelope-json-v1], test_contract::test_evidence_content_reproduces_its_hash[dd-input-json-v1], test_contract::test_evidence_content_reproduces_its_hash[dd-json-document-v1], test_contract::test_evidence_content_reproduces_its_hash[json-sorted-utf8-v1], test_contract::test_unknown_canonicalisation_is_rejected, test_cli::test_verify_exits_non_zero_on_a_violation, test_config::test_real_agents_evidence_verifies_against_the_committed_copies, test_config::test_the_committed_source_copies_match_their_pins |
not assessed | — |
| DD-REGISTRY | project | Agents are discovered from a registry and one manifest describes each to every consumer | ✅ substantiated | test_policy_agent::test_it_serves_the_policy_check_and_the_mandate, test_repository_agent::test_it_serves_four_phase_3_steps_and_not_the_mandate, test_standards_factory::test_the_spec_round_trips_through_the_manifest, test_hello::test_the_template_ships_everything_the_manifest_needs, test_registry::test_a_card_built_against_another_contract_is_incompatible_not_unavailable, test_registry::test_a_card_naming_a_class_it_carries_no_schema_for_is_rejected, test_registry::test_a_card_without_the_spec_extension_is_rejected, test_registry::test_an_unreachable_agent_is_unavailable_and_the_rest_still_load, test_registry::test_config_lists_agents_by_url_and_the_registry_reads_each_card, test_registry::test_from_url_keeps_the_raw_card_it_fetched, test_registry::test_manifest_rebuilt_from_cards_equals_each_agents_own_description, test_registry::test_refresh_finds_an_agent_that_started_after_the_registry_was_built, test_registry::test_refresh_keeps_an_agent_still_down_unavailable, test_registry::test_refreshes_from_several_threads_register_an_agent_once, test_cli::test_agents_command_lists_the_registry, test_transport::test_agent_card_lists_one_skill_per_agent_from_the_manifest, test_transport::test_agent_detail_serves_the_resolved_spec_and_404s_an_unknown_id, test_transport::test_manifest_samples_and_schema_are_served |
not assessed | — |
| DD-REMOTE-AGENT | project | Agents run as separate services reached over A2A; their spans join the conductor's trace and are held to the same rules | ✅ substantiated | test_remote::test_a_span_carrying_a_conductor_attribute_is_withheld[dd.grounding_mode-none], test_remote::test_a_span_carrying_a_conductor_attribute_is_withheld[dd.input_hash-0000000000000000000000000000000000000000000000000000000000000000], test_remote::test_a_span_outside_the_invocations_trace_is_withheld, test_remote::test_agent_spans_join_the_conductors_trace_under_invoke_agent, test_remote::test_an_agent_that_exceeds_its_timeout_is_a_failed_envelope, test_remote::test_an_agent_that_goes_away_after_registration_is_a_failed_envelope, test_remote::test_the_reply_carries_the_spans_beside_the_result_and_refuses_malformed_ones |
not assessed | — |
| DD-CONVERSATION | project | A conversation is a sequence of invocations sharing a conversation_id; every turn names the agent and version that produced it | ✅ substantiated | test_orchestrator::test_a_reply_without_delegation_cites_only_the_input, test_orchestrator::test_the_model_delegates_and_its_reply_is_grounded_on_the_child, test_orchestrator::test_the_model_is_sent_the_last_turns_of_the_history, test_orchestrator_stub::test_a_greeting_is_routed_to_example_hello_and_relayed, test_orchestrator_stub::test_history_is_part_of_the_input_hash, test_conversations::test_a_version_change_between_turns_is_reported, test_conversations::test_turns_are_in_order_with_children_nested_and_history_built, test_contract::test_bad_conversation_id_is_rejected, test_contract::test_message_and_reply_validate_and_are_discriminated, test_transport::test_agui_thread_is_the_conversation_and_conversations_are_served, test_viewer::test_chat_mode_is_driven_by_the_conversation_api_and_names_every_agent_version |
not assessed | — |
| DD-DELEGATION | project | An orchestrator delegates only through the workbench; each child is a governed, stored invocation and the parent records it | ✅ substantiated | test_orchestrator::test_a_failed_or_refused_child_is_reported_and_the_parent_succeeds, test_orchestrator::test_the_model_delegates_and_its_reply_is_grounded_on_the_child, test_orchestrator::test_the_tools_are_the_agents_the_harness_lists, test_orchestrator_key_not_stored::test_the_orchestrator_key_reaches_no_stored_record, test_orchestrator_only_delegates::test_the_model_is_offered_only_delegations, test_orchestrator_only_delegates::test_the_package_depends_only_on_the_sdk_and_the_model_client, test_orchestrator_only_delegates::test_the_source_imports_only_the_standard_library_the_sdk_and_the_model_client[__init__.py], test_orchestrator_only_delegates::test_the_source_imports_only_the_standard_library_the_sdk_and_the_model_client[agent.py], test_orchestrator_only_delegates::test_the_source_imports_only_the_standard_library_the_sdk_and_the_model_client[factory.py], test_orchestrator_only_delegates::test_the_source_imports_only_the_standard_library_the_sdk_and_the_model_client[model.py], test_orchestrator_only_delegates::test_the_source_imports_only_the_standard_library_the_sdk_and_the_model_client[prompt.py], test_orchestrator_stub::test_a_greeting_is_routed_to_example_hello_and_relayed, test_orchestrator_stub::test_a_request_for_a_metadata_review_is_routed_to_the_advisor_quality, test_orchestrator_stub::test_a_request_for_standards_is_routed_to_the_advisor_standards, test_bearer_tokens::test_a_child_acts_for_its_parent_whoever_sends_the_callback, test_delegation::test_a_caller_may_not_set_lineage_or_use_an_unknown_or_expired_token, test_delegation::test_a_child_acts_for_its_parents_principal_not_its_callers, test_delegation::test_a_child_may_not_reuse_its_parents_invocation_id, test_delegation::test_a_child_refused_by_policy_is_recorded_and_relayed, test_delegation::test_a_delegated_child_is_stored_with_its_lineage_and_recorded_by_the_parent, test_delegation::test_a_delegation_agent_reads_what_it_may_invoke_from_the_harness_card, test_delegation::test_async_callers_run_parents_side_by_side_and_their_children_never_wait_on_them, test_delegation::test_delegations_are_kept_when_the_parent_fails_after_delegating, test_delegation::test_only_a_delegation_agent_is_given_a_grant, test_delegation::test_routes_without_an_authenticator_accept_only_a_live_grant, test_delegation::test_self_delegation_and_a_second_level_are_refused, test_grounding::test_delegation_mode_forbids_retrieval_and_requires_the_input, test_grounding::test_delegation_mode_passes_citing_input_and_a_recorded_child, test_grounding::test_delegation_mode_rejects_an_unrecorded_or_altered_child, test_contract::test_delegation_evidence_must_be_the_input_or_a_recorded_delegation |
not assessed | — |
| DD-ENTRY-POINTS | project | The orchestrator finds which of the Blueprint's three entry points (§8.1) applies, and for each phase step from there names the agent that serves it or the requirement no agent serves yet; the steps are taken from the figure captions, not checked against the figures | ✅ substantiated | test_orchestrator_stub::test_a_first_turn_that_names_nothing_asks_which_entry_point_applies, test_orchestrator_stub::test_a_well_formed_request_on_a_first_turn_is_routed_not_questioned, test_orchestrator_stub::test_entry_one_starts_before_collection_and_names_the_documentation_gap, test_orchestrator_stub::test_entry_two_lists_phases_two_to_six_with_the_agents_and_the_gaps, test_process_flow::test_a_plan_runs_from_the_entry_to_the_last_phase_and_names_who_serves_each_step |
not assessed | — |
| DD-POLICY | project | Institutional profile gates which agents run, assigns each its action class, and says which classes need approval | ✅ substantiated | test_delegation::test_a_child_refused_by_policy_is_recorded_and_relayed, test_harness::test_a_profile_key_nothing_enforces_is_refused, test_harness::test_action_requiring_approval_is_referred, test_harness::test_an_agent_declaring_another_class_than_assigned_fails_and_is_not_run, test_harness::test_disabled_agent_fails_with_problem_details, test_harness::test_gate_matches_the_class_the_profile_assigns_not_the_one_the_agent_declares, test_harness::test_the_orchestrator_a_profile_names_must_be_an_agent_it_enables |
not assessed | — |
| DD-POLICY-OWNER | project | The deployment chooses the institutional profile; a request cannot name one, and every envelope records the profile and digest the conductor applied | ✅ substantiated | test_delegation::test_a_child_refused_by_policy_is_recorded_and_relayed, test_harness::test_the_envelope_records_the_profile_the_conductor_applied, test_transport::test_a2a_fails_a_request_naming_a_profile_and_runs_nothing, test_transport::test_agui_reports_an_unknown_agent_or_a_request_naming_a_profile_as_run_error |
not assessed | — |
| DD-ACTS-FOR | project | Every envelope and crate names the human the invocation acted for, taken from the deployment's authentication boundary and never from the request, and a delegated invocation acts for its parent's; authentication itself is not exercised (the identity is asserted by the operator) | ✅ substantiated | test_bearer_tokens::test_a_caller_without_a_bearer_token_is_refused_not_defaulted[None], test_bearer_tokens::test_a_caller_without_a_bearer_token_is_refused_not_defaulted[headers1], test_bearer_tokens::test_a_caller_without_a_bearer_token_is_refused_not_defaulted[headers2], test_bearer_tokens::test_a_caller_without_a_bearer_token_is_refused_not_defaulted[headers3], test_bearer_tokens::test_a_caller_without_a_good_token_is_refused_with_the_problem, test_bearer_tokens::test_a_child_acts_for_its_parent_whoever_sends_the_callback, test_bearer_tokens::test_a_provider_beside_an_operator_principal_stops_start_up, test_bearer_tokens::test_a_spoilt_token_is_refused[another-audience], test_bearer_tokens::test_a_spoilt_token_is_refused[another-issuer], test_bearer_tokens::test_a_spoilt_token_is_refused[azp-missing], test_bearer_tokens::test_a_spoilt_token_is_refused[expired], test_bearer_tokens::test_a_spoilt_token_is_refused[no-subject], test_bearer_tokens::test_a_spoilt_token_is_refused[wrong-nonce], test_bearer_tokens::test_a_verified_token_names_an_authenticated_person, test_bearer_tokens::test_each_caller_is_recorded_as_the_person_their_token_names, test_delegation::test_a_child_acts_for_its_parents_principal_not_its_callers, test_harness::test_the_crate_names_the_human_beside_the_agent, test_harness::test_the_crate_names_the_issuer_that_vouched_for_the_human, test_harness::test_the_envelope_records_whom_the_invocation_acted_for, test_identity::test_an_unset_principal_stops_start_up, test_identity::test_the_stub_asserts_the_configured_principal_whoever_calls, test_contract::test_a_principal_needs_an_absolute_iri_and_a_name[0000-0002-1825-0097-Josiah Carberry], test_contract::test_a_principal_needs_an_absolute_iri_and_a_name[https://orcid.org/x y-J], test_contract::test_a_principal_needs_an_absolute_iri_and_a_name[urn:x-], test_contract::test_a_request_cannot_say_whom_it_acts_for, test_contract::test_an_authenticated_principal_names_the_issuer_that_vouched, test_contract::test_an_envelope_names_the_human_it_acted_for, test_contract::test_vouched_by_is_present_exactly_when_authenticated[asserted-https://orcid.org], test_contract::test_vouched_by_is_present_exactly_when_authenticated[authenticated-None], test_cli::test_the_cli_acts_for_the_example_person_when_no_one_is_named, test_cli::test_the_cli_does_not_complete_half_a_principal[DD_PRINCIPAL_ID-DD_PRINCIPAL_NAME], test_cli::test_the_cli_does_not_complete_half_a_principal[DD_PRINCIPAL_NAME-DD_PRINCIPAL_ID], test_transport::test_a_request_saying_whom_it_acts_for_is_refused_by_both_transports, test_transport::test_each_transport_records_the_principal_its_authenticator_names |
not assessed | — |
Attestation¶
This report is not signed. What ties the table to the code is the commit that contains this file:
CI regenerates the report at that commit and fails if it differs. To check a version, find its
commit with git log --follow -- CONFORMANCE.md and look at that commit's CI run, whose job
summary names the commit tested, the Python version and the suite's counts. The intent is to
attest the report with in-toto or Sigstore once the project has a release process; an attestation
is made after the commit exists, so it can name that commit (see
docs/specs/dependencies.md, "Rejected alternatives"). A review is attested only by its entry in
docs/reviews.yaml and the commit that added it.
Regenerating¶
uv run pytest --json-report --json-report-file=.report.json
uv run python scripts/conformance_report.py
CI runs the same two commands with --check, which fails if the output differs from the
committed file.