Data Director Workbench¶

A test bed for building and testing Data Director sub-agents. It calls each agent, checks that the agent kept to a shared contract, and records every run so that a person can inspect what the agent did and why.

Background¶

The Data Director Agentic AI Blueprint, adopted by the Research Data Alliance (RDA), describes an assistant made of sub-agents that help researchers manage their data. This repository is a proposed reference implementation of it; the Blueprint is reproduced in ../docs/blueprint/ and its requirements are cited by ID (R3, P14, …).

The workbench is not the reference implementation itself, and it contains no agent code. Each agent states what input it reads, what output it returns, and what that output is based on. Agents are separate services in ../agents/. The workbench calls each one over A2A, the Agent2Agent protocol, then checks that it kept to those statements (Agents). The calls and checks are made by the harness, a package of its own in ../harness/ that the workbench is one way in to (System and boundaries). docs/architecture.md explains how the pieces fit together.

Who it is for¶

Software engineers and testers who write Data Director sub-agents, and anyone reviewing whether those agents meet the Blueprint. Using it requires familiarity with the command line and Python tooling; no knowledge of language models is needed, since the demonstration agents use none.

Installation¶

Requirements:

Python 3.14.
uv, which installs the dependencies listed in pyproject.toml.
A Bash shell to run ../scripts/run-agents.sh. On Windows, use WSL.

No account or API key is needed. Clone the repository, then from the repository root, which is a uv workspace:

uv sync --all-packages --all-extras
uv run pytest                                          # network blocked; everything offline
uv run pytest --cov                                    # with line and branch coverage (term report; see [tool.coverage])

The tests serve every agent in memory, so they need no agent running.

Usage¶

Quick start¶

To call agents from the command line, start them first and leave them running. Then, in a second terminal in workbench/, name the person the invocations act for (see Who an invocation acts for) and run the commands:

../scripts/run-agents.sh                               # in the first terminal

# optional; without these runs act for the example person, Josiah Carberry
export DD_PRINCIPAL_ID=https://orcid.org/0000-0002-1825-0097   # your ORCID iD, or another IRI
export DD_PRINCIPAL_NAME="Josiah Carberry"                     # your name
uv run workbench agents                                # what is registered, what each accepts
uv run workbench invoke --agent advisor.quality --input samples/orda-record.metadata.json
uv run workbench invoke --agent advisor.standards --input samples/soil-chemistry.profile.json
uv run workbench invoke --agent example.hello      --input samples/hello.salutation.json
uv run workbench invoke --agent advisor.quality --input samples/hello.salutation.json   # failed: input-not-accepted
uv run workbench invoke --agent advisor.standards --input samples/empty.profile.json   # abstained
uv run workbench serve                                 # then open http://127.0.0.1:8000/viewer/

The viewer¶

The viewer has three pages, reached by the tabs under its header. It opens on Chat.

Chat (http://127.0.0.1:8000/viewer/) holds a conversation with the orchestrator, orchestrator.stub, which hands each message to other agents and shows their replies inside its own (Orchestration).
Inspect (/viewer/inspect.html) runs one agent over a sample and shows the run.
Agents (/viewer/agents.html) lists the registered agents and their cards.

Only serve can run the orchestrator: invoke does not grant the permission it needs to call other agents, so samples/director.message.json has to go through the viewer or the conversation API.

Inputs and outputs¶

An input is a JSON document whose schema_class names its input class; the samples and what each exercises are listed in samples/README.md. The request and response formats are described in docs/contract.md.

invoke prints the agent's response and the result of the grounding check, which confirms that everything the response relies on can be traced to something the agent actually read during the run (docs/grounding.md). It also writes a folder, runs/<invocation_id>/, holding the request, the response, the trace of the run and a provenance record (a Process Run Crate). To re-run the grounding check on a finished run, use workbench lint <spans.jsonl> <envelope.json>.

Every response has one of five outcomes: succeeded, abstained, referred, failed or suspended; see docs/contract.md.

Configuration¶

The defaults run entirely offline, and nothing needs configuring to run the tests. To invoke an agent, from the command line or through serve, the workbench must be told who the invocation acts for. Unless told otherwise it acts for an example person, Josiah Carberry. The workbench is configured in three places.

Who an invocation acts for¶

The Blueprint (§5.4) requires that no agent operates anonymously: every invocation is made on behalf of a named human, and the response and the provenance record say who. A request cannot name that person. Until the workbench has authentication, the operator of the deployment names one person in its configuration, and every invocation is recorded as acting for them, marked assurance: asserted to show that nobody's identity was checked. On a shared workbench serve, every caller is therefore recorded as the operator.

To act for yourself, set both of these before running invoke or serve. With neither set, every invocation is recorded as acting for Josiah Carberry (https://orcid.org/0000-0002-1825-0097), the fictitious researcher ORCID publishes for examples. Setting only one is an error: the workbench does not complete half a principal.

Variable	Meaning
`DD_PRINCIPAL_ID`	An absolute IRI for the person, such as `https://orcid.org/0000-0002-1825-0097`. Use an ORCID iD where the person has one.
`DD_PRINCIPAL_NAME`	The person's name, as it should appear in the record.
`DD_PRINCIPAL_KIND`	Optional. `person` (the default), or `accountable_role` when the principal is a role rather than an individual.

--acting-for-id and --acting-for-name on invoke or serve take the place of the first two for one command. If only one of them is set, the command stops with DD_PRINCIPAL_ID and DD_PRINCIPAL_NAME unset; if the identifier is not an IRI, it stops with the configured principal is malformed.

Environment variables¶

The workbench reads these from the environment when it starts. For local development, set DD_CONFIG_DIR=config/local (the tree scripts/dev.py setup creates, which the workbench does not own): the workbench command loads app.env and agents.env from it at start-up. A variable already set in the environment takes precedence over the files.

Variable	Default	Meaning
`DD_PRINCIPAL_ID`, `DD_PRINCIPAL_NAME`	Josiah Carberry, `https://orcid.org/0000-0002-1825-0097`	Who every invocation acts for; see above.
`DD_PRINCIPAL_KIND`	`person`	`person` or `accountable_role`.
`DD_RUNS_DIR`	`runs`	Where each run's folder is written.
`DD_WRITE_CRATE`	`1`	`0` stops the provenance record (Process Run Crate) being written.
`DD_PROFILE`	`profiles/default.yaml`	The institutional profile applied to every invocation. A request cannot name one (Policy and identity).
`DD_AGENTS_CONFIG`	`agents.yaml`	The agent registry file.
`DD_SOURCES_CONFIG`	`sources.yaml`	The pinned source copies the source check resolves evidence against (Evidence and grounding).
`DD_HARNESS_URL`	the host and port of `serve`	The address the orchestrator uses to call the harness back. Set it when agents reach the harness by another name, such as a container network or a proxy.

These defaults are the workbench's development set (src/workbench/config.py), resolved against workbench/. The harness has no defaults of its own, so another way in names these files itself (System and boundaries). Each agent is a separate process and reads its own variables, such as the standards advisor retrieval backend or a model API key, where it runs; config/examples/ lists them.

Agent registry¶

agents.yaml lists the base URL of each agent, with an optional timeout_s. The ports match those started by ../scripts/run-agents.sh. An agent that cannot be reached is listed as unavailable and the rest still run; see docs/registry.md.

Institutional profiles¶

A profile says which agents may run, which action class each is assigned, and which classes need a person's approval. The deployment chooses one profile for every invocation; a request cannot name one. The default is profiles/default.yaml. Set DD_PROFILE, or pass --profile <path> to invoke or serve, to use another. The format is described in profiles/README.md.

Command-line options¶

uv run workbench <command> --help lists every option. The ones most often changed are:

serve --host 127.0.0.1 --port 8000: the address the viewer and APIs are served on.
invoke|serve --acting-for-id <IRI> --acting-for-name <name>: who the invocations act for, in place of DD_PRINCIPAL_ID and DD_PRINCIPAL_NAME.
invoke --input-type <class>: the input class, when the document has no schema_class and the agent accepts several.
invoke --requirement <ID>: the Blueprint requirement being exercised; repeatable.

Troubleshooting¶

invoke or serve will not start: DD_PRINCIPAL_NAME unset (or DD_PRINCIPAL_ID unset). Only one of the pair is set. Set both, or neither to act for the example person; see Who an invocation acts for.
the configured principal is malformed. DD_PRINCIPAL_ID must be an absolute IRI, such as https://orcid.org/…, and DD_PRINCIPAL_KIND must be person or accountable_role.
An agent is listed as unavailable. Its service is not running or is on another port. Start it with ../scripts/run-agents.sh, and check its URL in agents.yaml.
An agent is listed as incompatible. It was built against a contract version this workbench cannot govern; the message names both. Rebuild the agent, or the workbench, against the same SDK.
failed: input-not-accepted. The agent does not read that input class. workbench agents shows what each accepts.
failed: agent-not-permitted. The profile in use does not list the agent under agents.
failed: action-class-mismatch. The agent declares a different action class from the one the profile assigns it, usually because the agent changed. The steward updates the profile.
The workbench will not start: has keys nothing enforces. The profile uses a key outside its vocabulary; see profiles/README.md.
orchestrator.stub cannot reach other agents from invoke. invoke issues no delegation grant, so the orchestrator has no way to call them. Use workbench serve; see The viewer.

Agents¶

workbench agents lists what is registered. The agents themselves, and how to add one, are described in ../agents/README.md.

Project layout¶

Path	Contents
`src/workbench/`	The `workbench` CLI, the AG-UI transport, evaluation, and the development configuration it supplies to the harness.
`../harness/`	The harness itself: the conductor, policy gate, input check, grounding linter, source check, run store, provenance and the A2A endpoint.
`sources.yaml`	The pinned source copies for the source check.
`viewer/`	The browser UI served by `workbench serve`.
`agents.yaml`	The agent registry.
`profiles/`	Institutional profiles.
`samples/`	Example inputs.
`tests/`	The test suite.
`scripts/`	Maintenance scripts.
`docs/`	The architecture overview and the walkthrough. The decisions, the register and the conformance report are at the repository level (`../docs/`, `../CONFORMANCE.md`).

Contributing, contact and licence¶

See the contribution guide and the links in the repository README. The maintainer is Joe Heffer, University of Sheffield (j.heffer@sheffield.ac.uk). The workbench is released under the MIT Licence; to cite it, use CITATION.cff.