Skip to content

Error downloading distance-from-coast data in GitHub CI #1299

Description

@peanutfun

Describe the problem
Distance from coast data cannot be downloaded from NASA via the GitHub CI. The error code is either 444 or 403 (Forbidden). Everything works fine on my local machine.

Backtrace:

url = 'https://oceancolor.gsfc.nasa.gov/docs/distfromcoast/GMT_intermediate_coast_distance_01d.zip'
download_dir = PosixPath('/home/runner/climada/data'), overwrite = True

    def download_file(url, download_dir=None, overwrite=True):
        """Download file from url to given target folder and provide full path of the downloaded file.
    
        Parameters
        ----------
        url : str
            url containing data to download
        download_dir : Path or str, optional
            the parent directory of the eventually downloaded file
            default: local_data.save_dir as defined in climada.conf
        overwrite : bool, optional
            whether or not an already existing file at the target location should be overwritten,
            by default True
    
        Returns
        -------
        str
            the full path to the eventually downloaded file
        """
        file_name = url.split("/")[-1]
        if file_name.strip() == "":
            raise ValueError(f"cannot download {url} as a file")
        download_path = (
            CONFIG.local_data.save_dir.dir() if download_dir is None else Path(download_dir)
        )
        file_path = download_path.absolute().joinpath(file_name)
        if file_path.exists():
            if not file_path.is_file() or not overwrite:
                raise FileExistsError(f"cannot download to {file_path}")
    
        try:
            req_file = requests.get(url, stream=True)
        except IOError as ioe:
            raise type(ioe)("Check URL and internet connection: " + str(ioe)) from ioe
        if req_file.status_code < 200 or req_file.status_code > 299:
>           raise ValueError(
                f"Error loading page {url}\n"
                f" Status: {req_file.status_code}\n"
                f" Content: {req_file.content}"
            )
E           ValueError: Error loading page https://oceancolor.gsfc.nasa.gov/docs/distfromcoast/GMT_intermediate_coast_distance_01d.zip
E            Status: 444
E            Content: b'403 Forbidden\n'

climada/util/files_handler.py:97: ValueError

To Reproduce

  1. Run the test pipeline on GitHub.

Expected behavior
Tests pass.

Climada Version: develop

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions