Create meaningful filestructure for data, labels and model results

[alterapars]

• create coherent filestructure of each region of interest in one central bucket
• create regional disks that contain all data/labels for one region of interest (e.g. Mozambique)
• store all relevant models in bucket

[alkalait]

Given a checkpoints folder in some GC bucket:

• subfolder per user
  • the checkpoint itself
  • config used for training
  • post-mortem results (if any)

[user]__[arch]_[hyperparam1=val].ckp

[alterapars]

Central storage bucket for all relevant data now under:
gs://wfe-main
https://console.cloud.google.com/storage/browser/wfe-main

Folder Structure:
/ checkpoints
-- / model data (as described above)
/ data
-- / region
---- / labels
---- / s2_data (sentinel 2 data as .zarr)

Note to self:
gsutil -m cp -R is much faster then gsutil cp -r

[alterapars]

Setting up disks with data as a basis for training VMs on gcp:

1. Create VM in specified region/zone
2. Create disks in the same (!!) specified region/zone (extra or during the VM creation) as in docs
3. attach disk to VM (read-only):
   gcloud compute instances attach-disk <vm-name>--disk <disk-name> --mode=ro --device-name <disk-name>
4. if ill-formatted, re-format empty disk:
   sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/your_device_name
5. check for the mapping of disk <> specified disk name
   sudo lsblk -o NAME,MOUNTPOINT,SERIAL
6. for each disk, create mounting point (mkdir) and mount:
   sudo mkdir -p /mnt/disks/<mounting-point-name>
sudo mount -o discard,defaults /dev/your_device_name /mnt/disks/<mounting-point-name>

File transfer from bucket via:
gsutil -m cp -R gs://<bucket-path> <path-on-disk>

if rights are not sufficient, upgrade with:
chown -R <username>:<username><path-on-disk>

[alkalait]

Hi Julia, this optional step might also be useful:

1. Make a backup of the system config file (fstab) for mounting:
   sudo cp /etc/fstab /etc/fstab.backup
2. Edit fstab to auto-mount the new device at startup:

echo UUID=`sudo blkid -s UUID -o value /dev/your_device_name` /mnt/disks/<mounting-point-name> ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab

[alterapars]

Copying with gsutil -m cp -R gs://<bucket-path> <path-on-disk> was super slow.

@Lkruitwagen came up with the idea to use s5cmd for copying:

Usage:
With a little digging I found this post and then hacked around with s5cmd to get it working with gcp. Here are the steps:
Download a s5cmd release and untar to your favourite spot.

In the GCP Console, Cloud Storage -> Settings -> Interoperability -> Service Account HMAC , create an access key for your service account. Keep the key_id and secret handy.

In your $HOME directory, create a directory for your .aws credentials:

cd $HOME
mkdir .aws
cd .aws

4. In your new .aws folder, make a file called credentials . Store your key_id and secret in this file like so:

[default]
aws_access_key_id = <your-key-id>
aws_secret_access_key = <your-key-secret>

5. Now you can use s5cmd with the google cloud storage endpoint and move data blazingfast:
   ./s5cmd --endpoint-url https://storage.googleapis.com cp -c=1 -p=1000000 s3://your-google-bucket/subdirectory/* /path/to/my/disk/
   Don't worry, it won't be fooled by the s3://. The -c and -p parameters are explained in the blog post and github issue.

[alterapars]

Mounting the disk in ro mode:
sudo mount -o ro,noload /dev/<disk name> /mnt/disks/<mounting-point-folder>

ext4 filesystem on disk might cause trouble when mounting, therefore specify noload