Provenance¶

provenance ¶

Data provenance for NUANCE datasets, modelled on W3C PROV.

Every dataset carries a :class:Provenance record:

entities: the things that exist, such as an instrument file (with SHA-256, size and modification time, or the Globus collection and path it came from) or an in-memory array (with a content hash);
activities: what was done, such as reading a file or running an analysis, with its parameters, timing and the software versions used;
agents: who and what was responsible, such as the person, the facility (NUANCE Center) and the instrument.

The record travels with the data through every analysis, is written into every file NUScope saves, and exports to PROV-JSON or RO-Crate.

Activities form a hash chain: each one stores the SHA-256 digest of its own canonical JSON, which includes the digests of the activities it follows (prev). Entity ids are content addresses (file:sha256:..., array:blake2b:...), so the chain also pins every input and output. Editing any step afterwards breaks the chain; :mod:nuscope.integrity checks it and seals and signs saved files.

Provenance ¶

Provenance()

A small PROV graph that is copied forward with every derived dataset.

amend_last ¶

amend_last(*, used: Optional[list] = None, params: Optional[dict] = None)

Add inputs or parameters to the newest step (before anything follows it) and re-hash it.

heads ¶

heads() -> list[str]

Digests of the newest steps (those no other step follows). One, unless histories were merged.

check_chain ¶

check_chain() -> tuple[bool, list[str]]

Recompute every digest and link. Returns (intact, problems).

merge ¶

merge(other: 'Provenance')

Bring in another dataset's history (for operations with several inputs).

lineage ¶

lineage() -> list[str]

Human-readable history, oldest first.

to_prov_json ¶

to_prov_json() -> dict

W3C PROV-JSON (https://www.w3.org/submissions/prov-json/).

hash_array ¶

hash_array(data) -> str

Content hash of an array (BLAKE2b over dtype, shape and bytes).

canonical ¶

canonical(obj) -> bytes

Deterministic JSON bytes for hashing and signing (sorted keys, no spaces, UTF-8).