10. Evaluation Criteria¶

This section sets out how the Data Director Blueprint and any implementations of it will be evaluated. At v1.0, no reference implementation exists; this is a community specification at its first release. The evaluation framework is therefore defined here as a forward-looking commitment rather than an active assessment. It gives implementing organisations a clear target to design towards and gives the community a basis for assessing the Blueprint's impact as adoption grows.

10.1 Blueprint Quality Criteria¶

Quality criteria describe what a well-functioning implementation of the Data Director should demonstrate. The nine thematic areas below group the Blueprint's functional requirements (R), non-functional requirements (C), and architecture principles (P), reflecting the main dimensions across which an implementation can be assessed. Each area identifies what can be measured now and what cannot yet be measured.

Within each thematic area, criteria are distinguished by what the agent should do: its specific behaviours and outputs, and what the agent should be: its qualities and characteristics such as explainability, security, and accessibility.

Within each area, items listed as Cannot yet be measured are tagged to indicate the nature of the gap. [Agent/system development needed] indicates that the gap requires further development of the agent, tooling, methodology, or community-agreed benchmarks. [Human/user input needed] indicates that the gap can only be closed through human judgement, structured user research, longitudinal adoption evidence, or direct involvement of community stakeholders

1. Metadata and documentation quality | R2 · R3 · R4 · R5 · R7 · C15

A core output of the Data Director is high-quality metadata and supporting documentation. Two important caveats frame evaluation in this area. First, schema compliance alone is insufficient: values may be syntactically correct against a schema while being semantically wrong. Second, in domains where no controlled vocabulary, ontology, or prior FAIR practice exists, the tool's outputs will be generic, and the tool should explicitly flag this rather than presenting generic outputs as domain appropriate.

Can be measured now:

R2: Ontology term coverage is measurable: the proportion of terms present in a draft metadata record against a specific vocabulary can be counted. Coverage should be interpreted alongside term appropriateness, as precise metadata may legitimately use only a small number of terms from a large vocabulary.
R3: Vocabulary and ontology use verifiable against the FAIRsharing registry. For example, the MOMSI (Multi-Omics Metadata Standards Integration) FAIRsharing collection is confirmed useful as a cross-domain benchmarking reference.
R4: If a given schema is adopted to describe data or metadata, the description must be valid against the chosen schema. Validation against common schemas including DataCite, Dublin Core, and discipline-specific standards should be testable through automated checking.
R7: PID assignment workflow testable end-to-end against repository capabilities.

Cannot yet be measured:

[Agent/system development needed] C15: No community-agreed benchmark exists for evaluating whether AI-generated metadata meets an acceptable curation standard. However, cross-source and internal consistency checking could offer a partial way forward: when a generated field contradicts its source record, that disagreement is measurable today without a gold standard, and may serve as an automatable step ahead of fuller benchmarks.
[Human/user input needed] R5: Whether data documentation is genuinely sufficient for data reuse by a third party requires human judgement and cannot be assessed automatically.
[Agent/system development needed] Semantic correctness of metadata values: schema compliance does not confirm that values are meaningful in context, as demonstrated by the RDA Sample Type Working Group and RDA PIDINST Working Group, both of which identified difficulties aligning domain-specific concepts with DataCite.
[Agent/system development needed] Standardised benchmarking datasets for cross-implementation comparison do not yet exist. Specifying such datasets, potentially covering Dublin Core plus a core set of key disciplinary metadata standards, would allow different Data Director implementations to be evaluated against each other over time.

2. FAIR and CARE principle alignment | P1 · R9 · R11 · C15

This area assesses whether the Data Director actively advances FAIR principles across all capabilities, appropriately applies CARE principles where relevant, retrofits metadata for already-published datasets, and links data to the broader research record. FAIR maturity can be assessed using existing community tools. No equivalent methodology currently exists for CARE. Where multiple FAIR evaluation tools produce differing scores, the tool should support users in selecting the most appropriate evaluation approach for their context rather than presenting all available scores simultaneously.

Can be measured now:

P1: FAIR maturity scores assessable using community-recognised tools and comparable before and after implementation.
R11: Dataset linking to publications, grants, software, and people via ORCID and ROR verifiable through automated testing.

Cannot yet be measured:

[Human/user input needed] P1 (CARE Principles): No agreed evaluation methodology exists. Assessment requires direct involvement of Indigenous and community stakeholders and cannot be conducted through a checklist. CARE implementations are context-dependent and will require more than one profile.
[Agent/system development needed] C15: No community-agreed benchmark exists for comparing AI-generated records against manually curated equivalents.

3. Interoperability and open standards | R6 · C5 · P2 · P6 · P7

This area assesses whether the Data Director operates using open, community-maintained standards, exposes interoperable APIs across instances, and defaults to open sharing. All criteria in this area are technically testable at deployment.

Can be measured now:

R6 and C5: Metadata exchange across two or more Data Director instances without bespoke integration is testable; API conformance against a published specification is verifiable. The tool's ability to map metadata across multiple schemas, enabling deposit of the same dataset to different repositories without manual remapping by the researcher, is also testable.
P6: All standards and dependencies in use are auditable against the open standards requirement.
P7: A licence audit confirms the agent source code is publicly available under MIT or Apache 2.0.
P2: A configuration audit confirms default sharing is open, and any restrictions are accompanied by a documented legal basis.

4. Governance, provenance, and human oversight | R8 · R10 · C3 · C12 · C13 · P4 · P5

This area covers the traceability of agent actions, the enforceability of human oversight, and compliance with data governance and AI governance frameworks. Technical verification is possible for most criteria. Whether human oversight is genuinely meaningful in practice cannot be assessed technically and requires structured user research with researchers and data support professionals.

Can be measured now:

R10 and P5: Provenance logging, audit trail completeness, and use of PROV-O verifiable through automated testing and independent audit.
R8: DMP alignment verification at the point of data publication testable against machine-readable DMP outputs from compatible tools (DSW, Argos, DAMAP).
P4: Technical presence of human approval controls and blocking of irreversible actions testable.
C12: Enforceability of machine-readable governance policies verifiable through independent audit.

Cannot yet be measured:

[Human/user input needed] P4: Whether human oversight is genuinely meaningful in practice requires structured user research.
[Agent/system development needed] C13: Compliance coverage depends on which AI governance framework is adopted and varies by jurisdiction; full compliance cannot be assessed against a single universal standard.

5. Security, privacy, and sovereignty | C1 · C2 · C4 · P3 · P10

This area assesses whether security is built into the tool from the outset, whether privacy risks are detected and flagged rather than resolved autonomously, and whether data sovereignty requirements are respected across jurisdictions. All criteria are technically testable at deployment. Ongoing monitoring is additionally required beyond initial testing.

Can be measured now:

C1 and P3: Penetration testing against a recognised framework such as NIST; encryption and role-based access control audit; audit log completeness verifiable.
C2: Whether the tool correctly detects potential personal identifiers in data and metadata is testable. Whether the sharing workflow pauses and alerts the human reviewer upon detection is testable.
C4 and P10: Data residency enforcement across configured jurisdictions is testable; local and edge deployment options are verifiable.

Cannot yet be measured:

[Agent/system development needed] Ongoing monitoring is required to assess robustness against the tool being manipulated or redirected into taking actions outside its unintended scope, including indirect prompt injection embedded in ingested content such as repository records, DMPs, or README files. The OWASP Top 10 for Agentic Applications¹ and NIST's agent-hijacking evaluation work² offer starting points for testing whether existing mitigations (P4, R10) hold under adversarial input. No single methodology currently covers this scenario comprehensively at the point of deployment.

6. Explainability | C14 · P8

This area assesses whether all AI-generated outputs are accompanied by human-readable explanations, whether explanations are genuinely understandable to non-technical users, and whether a log of agent thought processes is maintained for transparency and audit.

Can be measured now:

C14 and P8: Presence of explanations alongside every AI output verifiable through automated testing.
C14: Role governance of override rights testable against defined user roles.
P8: Presence and completeness of thought process logs verifiable at deployment.

Cannot yet be measured:

C14: Whether explanations are genuinely understandable to researchers, data support professionals, and non-technical users across different disciplines requires structured user research.

7. Accessibility and inclusive access | C6 · P12

This area assesses whether the Data Director meets accessibility standards across all interfaces and is genuinely designed to include rather than exclude researchers across communities, institution sizes, and levels of data expertise.

Can be measured now:

C6: WCAG 2.1 compliance is independently auditable at deployment using recognised testing tools and assistive technology testing.

Cannot yet be measured:

[Human/user input needed] P12: Whether the tool is genuinely inclusive across communities, institution sizes, and disciplines requires longitudinal evidence of adoption patterns and impact that does not exist at v1.0.

8. Operational performance | C7 · C8 · C9 · C10 · C11 · P11

This area assesses whether the Data Director meets reliability, availability, performance, and scalability requirements across deployment contexts. Most criteria are testable now. The 99.5% uptime target (C7) applies to the user-facing front end; internal components may have differential SLAs.

Can be measured now:

Uptime against the 99.5% target (C7), recovery from component failures without data loss (C8), 24/7 availability across time zones (C9), response time benchmarking and asynchronous processing (C10), load testing with growing volumes and concurrent users (C11), and deployment testing across edge, local, and cloud environments (P11) are all testable through standard infrastructure and performance testing at deployment.

These metrics apply to service-based deployments. Where the Data Director is deployed as a local or on-demand tool on personal or institutional hardware, infrastructure metrics such as uptime targets and concurrent user load testing may not be applicable. Implementers should assess which metrics are relevant to their deployment model and document any that are out of scope with justification.

Cannot yet be measured:

[Agent/system development needed] C7: Implementing organisations must calibrate reliability targets to their deployment context. The 99.5% uptime target may not be realistic for locally hosted or resource-constrained deployments, such as those with limited computing power, IT support, or funding.

9. Impact, sustainability, and evaluation infrastructure | C16 · C17 · P9 · P13 · P14

This area covers two distinct dimensions that must not be conflated: the impact of the tool's outputs on researcher productivity and data quality, and the impact of the tool's own operation on the environment. Both must be monitored; neither yet has an agreed measurement methodology at v1.0. Affordability (C17) is identified as a future evaluation consideration in Section 10.3.

This area also covers the monitoring, logging, and feedback infrastructure needed to support ongoing evaluation of the tool's own performance and impact. Most mechanisms are technically verifiable at deployment.

Can be measured now:

C16: Usage data collection and reporting verifiable at deployment. Monitoring dashboards, alert mechanisms, and feedback collection mechanisms all verifiable at deployment.
C17: Configuration options for resource-constrained environments testable; AI processing cost architecture reviewable.
P9: Local configuration points for schemas, repository preferences, and access controls testable.
P13: Dependency viability assessable through review of licences, maintenance status, and community support. Testable by verifying that when dependencies are unavailable, unaffected workflow steps continue to function, the user is alerted, and work can be saved and resumed without data loss.

Cannot yet be measured:

[Human/user input needed] C16: Whether usage data demonstrates genuine researcher impact requires longitudinal evidence. Automated quality scoring requires community-agreed thresholds that do not yet exist. User signal mechanisms require active researcher adoption before generating useful data.
[Agent/system development needed] C17: No agreed methodology exists for assessing affordability across different institutional contexts and deployment models.
[Agent/system development needed] P14: No agreed methodology exists for assessing environmental footprint across deployment models. Edge deployment is harder to monitor for environmental impact than centralised infrastructure, where active tracking tools can be deployed and usage and inference costs can be more reliably measured.

10.2 Evaluation Process¶

At v1.0, evaluation of any implementation is self-assessed by the implementing organisation against the quality criteria set out in section 10.1. Implementing organisations are encouraged to document which criteria they have addressed, record any limitations or gaps against the measurability ratings above, and share that evidence openly with the community in line with the Blueprint's Open First and Open-Source principles.

A formal evaluation process, including community review, independent verification, and a structured conformance framework, could be developed as a priority output of the v2.0 process, informed by feedback from early adopters during and after the 30-day community review period. Until that framework exists, the MUST designations in the functional specification (Section 7.1) define the minimum that any implementation claiming alignment with this Blueprint must satisfy.

10.3 Future Evaluation Considerations¶

Five areas of evaluation methodology remain underdeveloped in this version of the Blueprint. Each is identified for further development; several are carried forward explicitly in Section 13.2 (Roadmap for Future Iterations).

Community-agreed metadata quality benchmarks

The Blueprint requires AI-generated metadata to meet community curation standards (C15, R4), but no agreed threshold currently exists against which outputs can be assessed. Evaluation must extend beyond schema compliance to encompass semantic correctness: whether metadata values are accurate and meaningful in context. This is domain-specific and cannot be validated by machines alone. The Working Group has proposed a tiered quality framework (gold, silver, bronze) as a direction for development in v2.0.

Cross-source and internal consistency checking, flagging where a generated value contradicts its source record, is measurable without a pre-agreed benchmark and may serve as a tractable first tier within this framework, consistent with established metadata quality dimensions in which logical consistency sits alongside completeness and accuracy (Bruce & Hillmann, 2004).³

Developing shared benchmarks will require structured collaboration across disciplinary communities, including through emerging frameworks such as Assess-IF and existing mechanisms such as FAIRsharing community collections and FAIR Implementation Profiles (FIPs).⁴

CARE evaluation methodology

Existing community-recognised tools support FAIR maturity assessment. No equivalent approach for evaluating CARE principle application in practice currently exists, and assessment cannot be conducted through a checklist. It requires direct involvement of Indigenous and community stakeholders and must account for significant variation in CARE interpretation across communities and domains. As a minimum, the tool should identify when CARE principles are likely to apply and direct users to appropriate guidance; full assessment methodology remains a future development.

Longitudinal impact measurement framework

Whether the Data Director achieves equitable, sustained improvement in data publication practice cannot be determined at deployment (C16). A framework for measuring researcher uptake, equity of access, and the quality of data publication over time needs to be defined and agreed upon with the community. Standardised output formats, such as those produced by Assess-IF, can support longitudinal analyses once such a framework is in place.

Affordability and environmental impact assessment

The Blueprint requires the tool to be affordable across institutional contexts (C17) and to minimise its environmental footprint (P14) but specifies no measurement approach for either. No agreed methodology exists for assessing affordability across deployment contexts, and environmental monitoring is harder for edge deployments than for centralised infrastructure.

Formal conformance framework

Conformance at v1.0 is defined by satisfying the MUST requirements in the functional specification and is self-declared by implementing organisations. A community-governed conformance framework, including defined tiers, a peer review process, and an independent verification mechanism, could be developed as a priority output of the v2.0 process and piloted with early adopter implementations, replacing the current arrangement in which implementing organisations self-assess against the criteria without independent verification

https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ ↩
https://www.nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations ↩
Bruce, T. R., & Hillmann, D. I. (2004). The Continuum of Metadata Quality: Defining, Expressing, Exploiting. In D. I. Hillmann & E. L. Westbrooks (Eds.), Metadata in Practice (pp. 238–256). Chicago: American Library Association. https://hdl.handle.net/1813/7895 ↩
https://www.go-fair.org/how-to-go-fair/fair-implementation-profile/ ↩