csvpath/util/config.py has several assure*_path() methods that create a local directory unless the configured path is a remote backend URI. The remote-prefix checks are inconsistent across them:
_assure_archive_path(): s3://, azure://, sftp:// (missing gs://)
_assure_transfer_root(): (none at all)
_assure_inputs_files_path(): s3://, azure://, sftp://, gs:// (complete)
_assure_inputs_csvpaths_path():s3://, azure://, sftp://, gs:// (complete)
Two concrete gaps:
- A gs:// archive_path falls through _assure_archive_path()'s checks and hits os.path.exists()/os.makedirs() on the raw gs://... string.
- transfer_root has no remote checks at all, so any of s3://, azure://, sftp://, or gs:// as a transfer_root falls through the same way.
David also raised a broader question while reviewing: sftp:// is currently grouped with s3:///azure:///gs:// in every one of these checks, but sftp is a real filesystem behind a network protocol, while the blob stores (s3, azure, gcs) only have loose, filesystem-like semantics layered on top of object storage. It is worth evaluating whether sftp:// actually belongs being treated identically to the blob-store prefixes here, or whether it should be handled on its own (e.g. because a real directory concept may be creatable/expected for sftp in a way that does not apply to the others).
Recommend, when this is picked up:
- add the missing gs:// check to _assure_archive_path()
- add all four remote-prefix checks to _assure_transfer_root()
- evaluate whether sftp:// should be split out from the blob-store group across these methods (and check for the same pattern elsewhere in the codebase, e.g. file_readers.py/file_writers.py/nos.py, which use similar per-backend prefix dispatch)
Left TODO comments in csvpath/util/config.py pointing at this issue rather than fixing, per tests-only scope.
csvpath/util/config.py has several assure*_path() methods that create a local directory unless the configured path is a remote backend URI. The remote-prefix checks are inconsistent across them:
Two concrete gaps:
David also raised a broader question while reviewing: sftp:// is currently grouped with s3:///azure:///gs:// in every one of these checks, but sftp is a real filesystem behind a network protocol, while the blob stores (s3, azure, gcs) only have loose, filesystem-like semantics layered on top of object storage. It is worth evaluating whether sftp:// actually belongs being treated identically to the blob-store prefixes here, or whether it should be handled on its own (e.g. because a real directory concept may be creatable/expected for sftp in a way that does not apply to the others).
Recommend, when this is picked up:
Left TODO comments in csvpath/util/config.py pointing at this issue rather than fixing, per tests-only scope.