Issue #1009: Retrieve single dataset - #1047
Conversation
| @@ -246,9 +246,9 @@ def mass_list_files_recursively(mass_path, simulation): | |||
| 'files': [] | |||
| } | |||
| datasets[dataset_id]['files'].append({ | |||
| 'filesize': elems[4], | |||
| 'filesize': elems[2], | |||
| 'filename': filename, | |||
| 'mass_path': elems[8] | |||
| 'mass_path': elems[6] | |||
There was a problem hiding this comment.
The data retrieval tool appeared to be broken when i tested it at the start of this issue. It was throwing this error:
File "CDDS/cdds/cdds/misc/retrieve_archived_data.py", line 333, in main_cdds_retrieve_archived_data
mass_file_list = mass_list_files_recursively(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "CDDS/cdds/cdds/common/mass.py", line 238, in mass_list_files_recursively
timestamp, filename) = elems[8].split('/')[-11:]
~~~~~^^^
IndexError: list index out of range
I can only assume that the 'moo', 'ls', '-Rl' command used here now has slightly different stdout formatting after the move to new MASS.
This change fixes the error so the tool works again.
|
Undertook a rename of the cdds specific tool in this commit from |
|
In this commit i created a replacement (called I considered modifying I've dropped this new function as a replacement for |
ae7a0a3 to
5c77cde
Compare
| "dataset_id", | ||
| help="Full CMIP6 dataset_id, e.g. CMIP6.CMIP.MOHC.UKESM1-0-LL.piControl.r1i1p1f2.Amon.tas.gn", | ||
| ) | ||
| if len(sys.argv) > 1 and sys.argv[1] == "get": |
There was a problem hiding this comment.
This guards against IndexError when no command is supplied - provides helpful output instead:
$ cdds/bin/crepp_retrieve_archived_dataset get
usage: crepp_retrieve_archived_dataset [-h] [--create-directories-false] [--mass-root MASS_ROOT]
[--dry-run] [--chunk-size CHUNK_SIZE]
{get,ls} dataset_id destination
crepp_retrieve_archived_dataset: error: the following arguments are required: dataset_id, destination
Arguably a bit brittle but the alternative was to use subparsers, which don't appear to be used elswhere in cdds - and they look a bit odd to me at first glance. So leaving this for ease of review unless that change is requested.
| """The :mod:`mass` module interact with the MASS archiving system.""" | ||
| import logging | ||
| import subprocess | ||
| import re |
There was a problem hiding this comment.
Didn't appear to be used
Pylance: "re" is not accessed
1 participant
Fulfils #1009