Skip to content

Data Director inventory

CI Coverage

dd-inventory describes a dataset's files where they live. It reads every file under a folder and writes a FileInventory, a JSON document of the core contract, that gives for each file:

  • its path relative to the folder;
  • its size and its SHA-256 digest;
  • its media type and every PRONOM identifier it matched, found with fido, and whether the match was by byte signature, container signature or extension only;
  • for a CSV or TSV table, its fields as a Frictionless Table Schema inferred from a sample of rows.

The inventory holds facts about the files and never their content. Research data are not uploaded to Data Director: the researcher adds the inventory to their project instead (Projects). The digests let the files deposited later be shown to be the files described.

$ uv run dd-inventory path/to/dataset -o inventory.json

Hidden files and folders (names beginning with .) are skipped, symbolic links are not followed, and both are counted in inventory_limitations. If fido cannot be loaded, formats are guessed from file extensions and the inventory says so.

The package does not depend on the contract or any other part of Data Director (System and boundaries). Its tests check what it writes against the contract's FileInventory schema.

Module What it does
scan.py Walks the folder, digests each file and assembles the inventory.
identify.py The one module that calls fido, with the extension fallback.
tables.py Infers a table's fields with Frictionless.
cli.py The dd-inventory command.