Skip to content

Results Instance Manifest transfers field misses definition.json descriptor-based transfers #224

Description

@dk107dk

Summary

The Results Instance Manifest field transfers only ever records transfers
declared through the old, per-instance transfer-mode metadata directive.
Transfers declared the newer way, through the named-paths group
definition.json descriptor, do actually run, but are never recorded in
transfers, or anywhere else in the manifest. A csvpath whose transfers are
all defined via the group descriptor will show an empty transfers field
even though real transfers happened during the run.

Where the field is set

csvpath/managers/results/result_registrar.py, register_complete:

if self.result.csvpath.transfers:
    tpaths = self.result.csvpath.csvpaths.results_manager.transfers_manager.transfer_paths(
        self.result
    )
    mdata.transfers = tpaths

self.result.csvpath.transfers (csvpath/csvpath.py:861-863) reads only
the transfer-mode key from that one csvpath own leading-comment metadata
(csvpath/modes/transfer_mode.py). And
TransfersManager.transfer_paths() (csvpath/managers/results/transfers_manager.py:167-175)
is built on exactly that same source:

def transfer_paths(self, result) -> list[tuple[str, str, str, str, str]]:
    transfers = result.csvpath.transfers
    return self._transfer_paths(result, transfers)

What is missing

TransfersManager.do_transfers_if(result)
(csvpath/managers/results/transfers_manager.py:28-48) actually calls two
independent mechanisms for every result:

def do_transfers_if(self, result) -> None:
    try:
        self.do_transfer_mode_if(result)
    except Exception:
        ...
    try:
        self.do_description_transfers_if(result)
    except Exception:
        ...

do_transfer_mode_if is the old, per-instance transfer-mode path, the
same one that feeds mdata.transfers.

do_description_transfers_if (lines 50-74) is the newer mechanism, sourced
from the named-paths group descriptor:

def do_description_transfers_if(self, result) -> None:
    csvpaths = self.results_manager.csvpaths
    pathsmgr = csvpaths.paths_manager
    name = result.paths_name
    describer = pathsmgr.describer
    transfers = describer.get_transfers(name)
    if transfers and transfers.path_transfers is not None:
        n = result.identity_or_index
        if n is not None and n in transfers.path_transfers:
            self.do_description_transfers(result, name, transfers.path_transfers.get(n))

This walks into do_description_transfers and do_description_transfer
(lines 76-98), which build their own tpaths and call _do_transfers
directly. Those transfers really execute, files really get copied, but
nothing on this path ever writes to mdata.transfers, or to any other
manifest field. The group descriptor mechanism also supports several
distinct trigger states (on_complete_all, on_complete_valid, and an
error-triggered branch), none of which are distinguished, or even present,
in the manifest today.

Impact

Anyone reading a Results Instance Manifest to find out what transfers ran
for that csvpath only sees the old-style, per-instance ones. Any group,
descriptor-based transfers are invisible in the manifest even though they
happened. This will also matter for the in-progress references v3 work,
where a :transfers() style accessor would currently only be able to expose
half the picture.

Suggested fix direction

Make mdata.transfers collect the union of both mechanisms results:
transfer_paths(result) from do_transfer_mode_if, plus whatever
do_description_transfers/do_description_transfer actually copied from
the descriptor path. Likely means having do_description_transfer return
or accumulate its tpaths the same way do_transfer_mode_if already does,
so do_transfers_if can merge both lists before handing them to whichever
registrar sets mdata.transfers. Worth deciding at the same time whether
the trigger state, on_complete_all vs on_complete_valid vs error, should
be recorded per transfer in the manifest, since that information is
currently dropped entirely.

Found while reviewing manifest fields for the references v3 work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions