Skip to content

Inconsistent/missing remote-prefix checks in Config._assure_*_path methods #209

Description

@dk107dk

csvpath/util/config.py has several assure*_path() methods that create a local directory unless the configured path is a remote backend URI. The remote-prefix checks are inconsistent across them:

_assure_archive_path():        s3://, azure://, sftp://              (missing gs://)
_assure_transfer_root():       (none at all)
_assure_inputs_files_path():   s3://, azure://, sftp://, gs://       (complete)
_assure_inputs_csvpaths_path():s3://, azure://, sftp://, gs://       (complete)

Two concrete gaps:

  1. A gs:// archive_path falls through _assure_archive_path()'s checks and hits os.path.exists()/os.makedirs() on the raw gs://... string.
  2. transfer_root has no remote checks at all, so any of s3://, azure://, sftp://, or gs:// as a transfer_root falls through the same way.

David also raised a broader question while reviewing: sftp:// is currently grouped with s3:///azure:///gs:// in every one of these checks, but sftp is a real filesystem behind a network protocol, while the blob stores (s3, azure, gcs) only have loose, filesystem-like semantics layered on top of object storage. It is worth evaluating whether sftp:// actually belongs being treated identically to the blob-store prefixes here, or whether it should be handled on its own (e.g. because a real directory concept may be creatable/expected for sftp in a way that does not apply to the others).

Recommend, when this is picked up:

  • add the missing gs:// check to _assure_archive_path()
  • add all four remote-prefix checks to _assure_transfer_root()
  • evaluate whether sftp:// should be split out from the blob-store group across these methods (and check for the same pattern elsewhere in the codebase, e.g. file_readers.py/file_writers.py/nos.py, which use similar per-backend prefix dispatch)

Left TODO comments in csvpath/util/config.py pointing at this issue rather than fixing, per tests-only scope.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions