Use explicit volume mount definition for kubeflow workdir

[nathan-az]

The motivation behind this PR was to better support subPath and other volume mount options for the workdir. Due to a cloud limitation, our users share a PVC, and subPaths provide level of virtual isolation. So the main change is – users specify the dict for the mount itself rather than name and path. The rest of the logic is the same (including appending to executor.volume_mounts if the user didn't explicitly include it there).

The pattern here changes to users explicitly specifying the VolumeMount they want for the workdir, expecting that volumes contains the relevant volume. e.g.

volumes = [{"name": "work-vol", "persistentVolumeClaim": {"claimName": "my-pvc"}}]
workdir_volume_mount = {"name": "work-vol", "mountPath": "/nemo_run", "subPath": team_name}

I haven't tested this yet on a real cluster, but will do so next week. PR is to get thoughts on the change in pattern. I haven't yet used the kuberay executor but will do the same soon, so if there is an analogous change will be able to do so there.

@ko3n1g let me know if you have any thoughts on the pattern, or issues with the PR.

[nathan-az]

@ko3n1g it looks like the kuberay executor just uses the first entry in volume_mounts as the workspace dir. This is even simpler than what I've got here (albeit a bit less transparent). That said, since it's less verbose, I'm happy to swap to that pattern for kubeflow too, if you prefer it to the one in this PR. Or if you prefer the explicit choice/flexibility, can change kuberay to take an arg like in this PR (but in a follow-up PR).