Architecture¶

This page explains how the Data Director is put together. It starts with the whole system in one picture and adds detail one level at a time, so a reader can stop at the level they need:

The idea in one picture: what the system is for.
The parts: the packages in this repository and how they depend on each other.
One run, step by step: what the harness does every time an agent runs.
Conversations and delegation: how the orchestrator uses other agents.
Configuration and deployment: where it runs and who decides what it may do.

The reasons behind each choice are in the specifications. Terms are defined in the glossary. For one run followed with real output, see the workbench's walkthrough.

1. The idea in one picture¶

The Blueprint describes a Data Director as one assistant a researcher talks to, backed by specialist sub-agents: one recommends metadata standards, another reviews a metadata record, and so on. In this implementation every agent runs under a harness that checks what the agent did and records it.

flowchart LR
    person(["Researcher or data steward"])
    app["Data Director app<br/>(browser)"]
    harness["Harness<br/>checks and records every run"]
    agents["Agents<br/>(separate services)"]
    sources[("Sources<br/>e.g. FAIRsharing")]
    runs[("Run store<br/>envelopes, traces,<br/>provenance")]
    projects[("Project store<br/>file inventories,<br/>documents")]

    person --> app
    app --> harness
    harness --> agents
    agents --> sources
    harness --> runs
    harness --> projects

Three ideas carry through the rest of this page.

Nothing reaches an agent except through the harness. The app, the developer workbench and other systems all hand requests to the same function, the conductor. An agent is held to the same rules whichever way it is called.
An agent is a separate service with a declared contract. Each agent says what input it reads, what result it returns and what that result may rest on (its grounding mode). The harness checks the agent against that declaration; it needs no code specific to any agent.
Every run ends in one record, the envelope. It says what happened (succeeded, abstained, referred, failed or suspended), carries the result and the evidence it rests on, and names the person the run acted for. A refusal is an envelope too, not an exception.

2. The parts¶

The repository is a uv workspace of separate Python packages, plus the app's browser shell in TypeScript.

flowchart TB
    subgraph ways_in["Ways in"]
        web["app/web<br/>React shell"]
        app["app/ — dd_app<br/>the Data Director app"]
        workbench["workbench/<br/>CLI, viewer, evaluation<br/>(development only)"]
        a2a_callers["Other systems<br/>over A2A"]
    end

    harness["harness/ — dd_harness<br/>conductor, policy gate, checks,<br/>run store, provenance"]

    subgraph agent_services["agents/ — one service each"]
        director["orchestrator.stub<br/>orchestrator"]
        standards["advisor.standards"]
        quality["advisor.quality"]
        hello["example.hello<br/>template"]
    end

    sdk["sdk/ — dd_sdk<br/>contract, evidence, tracing,<br/>agent server"]

    web -- "HTTP /api" --> app
    app --> harness
    workbench --> harness
    a2a_callers -- "A2A" --> harness
    harness -- "A2A" --> agent_services
    harness -. "imports" .-> sdk
    agent_services -. "imports" .-> sdk

Solid arrows are calls at run time; dotted arrows are imports.

Part	What it is	Read more
`sdk/`	What agents and the harness share: the contract (in LinkML, with generated JSON Schema), evidence hashing, tracing helpers, and the server that puts an agent behind A2A.	`sdk/README.md`, contract
`harness/`	The governance every run goes through. Its centre is the conductor. It also serves its own A2A endpoint (`dd-harness serve`).	`harness/README.md`, conductor
`agents/`	One package and one A2A service per agent. `example-hello/` is the template for a new one.	`agents/README.md`
`app/`	The Data Director that people use (alpha). A Starlette server (`dd_app`) over the conductor, and a React shell (`app/web`) drawn from each agent's manifest.	`app/README.md`, The app
`workbench/`	The test bed for people who write agents: a CLI, the Inspect and Chat pages, and evaluation. Not part of a deployment.	`workbench/README.md`
`inventory/`	`dd-inventory`, the command a researcher runs where their data live. It writes the inventory of a dataset's files (paths, sizes, digests, PRONOM formats, table fields) and needs no other part of Data Director.	`inventory/README.md`, Projects

The dependency rules¶

The boundaries between packages are deliberate, and tests enforce the most important one.

sdk depends on nothing in the repository.
harness depends only on sdk. It never imports the app, the workbench or an agent (harness/tests/test_harness_boundary.py).
An agent depends only on sdk. The harness reaches it over A2A, not by import (Agents).
app and workbench each depend on harness, and never on each other. Neither imports an agent; each reaches every agent over A2A (The app).

This is why adding an agent needs no change to the harness, the app or the workbench: each of them learns about the agent from its agent card, the document every A2A service publishes, which carries the agent's AgentSpec (registry).

The shared formats¶

Some things are fixed so that everything else can change. The contract's request and envelope, the dd.* trace attributes, the evidence canonicalisations, the identifiers and the outcome vocabulary are settled formats; changing any of them changes a spec. The components that produce and read them (agents, retrieval back ends, model explainers, front ends) sit behind interfaces and can be replaced.

Each agent owns its own input and payload classes, declared in a LinkML file that imports the shared core (Contract). Its card carries each class's JSON Schema, pinned by a digest, so the harness can check inputs and results without a copy of the agent's code.

3. One run, step by step¶

A run is one agent invoked on one input. Every run, from any way in, goes through Conductor.invoke in harness/src/dd_harness/conductor.py.

sequenceDiagram
    autonumber
    actor P as Person
    participant W as Way in<br/>(app, workbench, A2A)
    participant C as Conductor
    participant A as Agent service
    participant S as Run store

    P->>W: input for an agent
    W->>C: request + who it acts for
    Note over C: fills in identifiers,<br/>timestamps, input hash
    C->>C: policy gate: may this agent run here?
    C->>C: input check: does it read this class?
    C->>A: A2A message (request, trace parent)
    A-->>C: outcome, payload, evidence, spans
    C->>C: grounding linter: does the result rest<br/>only on what its mode allows?
    C->>C: source check: do cited records match<br/>the pinned copies?
    C->>C: validate the envelope (JSON Schema)
    C->>S: envelope, trace, Process Run Crate
    C-->>W: envelope
    W-->>P: outcome, result and evidence

What each step is for:

Who the run acts for comes from the way in, never from the request. Until there is sign-in, it is the one person the operator configured (Policy and identity).
Policy gate. The deployment's institutional profile says which agents are enabled, which action class each is assigned and which classes need a person's approval. A request cannot choose the profile (Policy and identity, Policy and identity).
The agent runs in its own process. It returns only an outcome, an optional payload and the evidence it used. The conductor, not the agent, sets identifiers, timestamps, telemetry and the grounding mode on the envelope.
Grounding linter. Compares what the result cites with what the agent's trace shows it did, under the rules for its grounding mode: retrieval, input_only, none or delegation (grounding, Evidence and grounding). It is a consistency check, not proof.
Source check. Re-hashes each cited record against a pinned copy of its source, which does verify the citation (Evidence and grounding).
Run store. Appends the envelope to invocations.jsonl and writes a folder per run with the request, envelope, trace, both checks' verdicts and a Process Run Crate.

A run that belongs to a project is also given the project facts its input class asks for. The conductor puts them into the input, so the agent never reads the project store itself. The project store holds, for each project, an append-only log of what changed (an inventory added, a document uploaded, a repository chosen) and any uploaded documentation, kept by digest. It holds an inventory of the research data files, made where the data live, never the files themselves. Which project content may go to an agent that calls a hosted model is the profile's decision (Projects). The inventory is made by dd-inventory and kept in the project store (projects); giving agents the project's facts is not built yet (#394).

A project document (a metadata record, a README, a data dictionary) is a Document payload. When a run in a project returns one, the conductor adds it to the log as a new version. A person's edit is a version too. Each version is proposed, approved or rejected on its own, and none is removed (Projects; the conductor recording a run's draft is not built yet, #394). A person approves or rejects the latest version, with a reason to reject or edit, and the harness records the decision in the log under the principal who made it. The app only shows it, at /projects/{project_id}, and nothing resumes a run (Projects).

A project is shared by its members, recorded in the same log. An owner adds and removes members by their ORCID iD, and decides on document versions. Any member may read the whole project, run agents in it and add a turn to any of its conversations, each turn acting for whoever sent it. The harness checks membership before every run and read, and refuses a turn or an edit made from a state of the project that is no longer the latest. Agents see each person only as a pseudonymous label such as "Member 1" (Projects; not built yet, #388).

The loop ends with a deposit package for the repository the researcher chose, which may be any repository in re3data. Its requirements come from YAML files of the deployment's: the set for the metadata standard the repository declares (DataCite for most), and, for the few repositories the deployment knows well, an override with the fields its form requires and the formats, sizes and licences it accepts. The harness, not an agent, checks the project against them and writes a SHACL Validation Report. Only from a report with no violation does it build the package: an RO-Crate of the dataset that holds the approved documents, the inventory, the report and the audit trail, and no data files. The researcher adds the files when depositing (Deposit). Recording the choice is built; validation and the package are not yet, #199, #202.

If a step refuses, the agent's result is withheld but the run is still recorded, and the person gets an envelope that says why. Only a caller's own mistake (a malformed request, a reused identifier, an unknown agent) raises an error instead. The step-by-step table, with each problem type, is in conductor.md.

4. Conversations and delegation¶

The app opens on a conversation with an orchestrator: the agent whose grounding mode is delegation. Today that is orchestrator.stub, a rule-based stand-in that routes each message by the patterns in its routing.yaml. A first message that matches no pattern is asked which of the Blueprint's three entry points applies (before data collection, data being prepared for sharing, ready to deposit), and the reply lists the steps from there, each with the agents that serve it or "no agent yet" and its requirement (Orchestration). An orchestrator answers by asking other agents, but it never calls them directly. It asks the harness, so that every child run passes through every step above. A model-backed orchestrator has no other tools, no MCP server, web search or Skills, so whatever it says about a case comes from a child's result (Orchestration).

sequenceDiagram
    autonumber
    participant App as App server
    participant C as Conductor
    participant D as Orchestrator<br/>(orchestrator.stub)
    participant X as Specialist agent

    App->>C: Message for the orchestrator
    C->>C: policy gate, input check
    C->>D: request + grant token + harness A2A address
    D->>C: ctx.delegate(agent, input) with grant (A2A callback)
    C->>C: check grant, then a full run of its own
    C->>X: A2A message
    X-->>C: result
    C-->>D: child envelope reference and evidence
    D-->>C: Reply citing the child run
    C->>C: grounding linter, source check
    Note over C: revoke the grant,<br/>record children as delegations
    C-->>App: envelope with delegations

The grant is a random token, valid only while the orchestrator's run is in progress. The callback arrives on the harness's own A2A endpoint (/a2a), which in the app accepts nothing else.
Each child run is stored as a run of its own, in a trace linked to its parent, acting for the same person and gated by the same profile.
A child is never given a grant, so delegation is one level deep.
The app builds each message's history from the stored turns; a caller cannot supply one (The app). Conversations are read back from the run store.
Each conversation belongs to one project, and a project can have many conversations, so what is known about a project survives a new conversation (Projects).

The rules are in Orchestration and conductor.md.

5. Configuration and deployment¶

The deployment decides¶

The harness ships no profile, agent list or source list. Each comes from the deployment, through DD_* environment variables or defaults the way in passes to Settings.from_env:

Setting	What it names
`DD_PROFILE`	The institutional profile: which agents may run, and with what approval.
`DD_AGENTS_CONFIG`	The list of agent URLs to read cards from.
`DD_SOURCES_CONFIG`	The pinned source copies the source check uses.
`DD_RUNS_DIR`	Where runs are written.
`DD_PROJECTS_DIR`	Where project records and uploaded documentation are kept (Projects, not built yet).
`DD_REPOSITORIES_DIR`	The requirement sets for metadata standards and repository overrides (Deposit). Only `dd-harness project choose-repository` reads them so far.
`DD_PRINCIPAL_ID`, `DD_PRINCIPAL_NAME`	Whom every run acts for where nobody signs in (the CLI, a development server).
`DD_OIDC_ISSUER`, `DD_OIDC_CLIENT_ID`	The OpenID Connect provider people sign in with (Policy and identity). The app also takes the client's secret.

The app's reference set is in app/config/; the workbench's development set is in workbench/. Listing an agent makes it known; only the profile makes it permitted.

Running it¶

On a developer's machine, scripts/run-agents.sh starts each agent on its own port and uv run data-director starts the app (see the README). The shared deployment is one Ubuntu VM that pulls main and serves it once CI has passed (deploy/README.md):

flowchart LR
    browser(["Browser"])
    subgraph vm["Ubuntu VM"]
        nginx["NGINX<br/>HTTPS, password,<br/>/a2a closed"]
        app["data-director-app<br/>127.0.0.1:7860"]
        agents["data-director-agent@…<br/>hello :8101, quality :8102,<br/>standards :8105, director :8106"]
        runs[("/var/lib/data-director/runs")]
        projects[("/var/lib/data-director/projects")]
        timer["deploy timer<br/>every 5 min"]
    end
    github[("GitHub main<br/>+ CI passed")]

    browser -- HTTPS --> nginx --> app
    app -- "A2A (loopback)" --> agents
    agents -. "delegation callback" .-> app
    app --> runs
    app --> projects
    timer -- "fetch, build, switch" --> github

The app and agents listen on loopback only, and NGINX asks for a password, because callers are not yet authenticated: every run acts for the operator configured on the VM (#17). Do not expose the app or the agents beyond a trusted network until that is done. The VM holds no research data: a project there holds only an inventory of its files and any uploaded documentation (Projects).

Where to go next¶

To understand	Read
What a request and an envelope contain	`contract.md`
Each conductor step and its problem types	`harness/docs/conductor.md`
Grounding modes and the linter's rules	`harness/docs/grounding.md`
How agents are found and described	`harness/docs/registry.md`
How to write an agent	`agents/README.md`
The workbench's components, module by module	`workbench/docs/architecture.md`
How requirements are tested and reviewed	`conformance.md`
How each part is meant to work, and why	`specs/`