Deploy Spark application artifacts from Git repositories

[razvan]

In addition to custom images and app volumes, users should be able to easily deploy applications from remote Git repositories similarly to the Airflow operator. For Spark, no Git syncing is necessary.

Things to consider:

• Authentication
• Dependency handling
• Integration with Spark Connect

[alexortner]

What worked for me very stable is to pull the code upfront via an init container

podOverrides:
      spec:
        initContainers:
          - name: git
            image: alpine/git:latest
            securityContext:
              runAsUser: 0  # Run as root to write to /root/.ssh and set permissions
            resources:
              limits:
                memory: 500Mi
              requests:
                cpu: "1"
                memory: 500Mi
            env:
            # Configure Git Repository Details
            ################################################
            - name: GIT_URL
              value: "repo url"
            - name: GIT_PROJECT
              value: "project"
            - name: GIT_REPOSITORY
              value: "reponame"
            - name: GIT_PORT
              value: "xxxx"
            - name: GIT_BRANCH
              value: "branch"
            ################################################
            command: ["sh", "-c"]
            args:
              - |
                echo "👉 Cloning from ssh://git@$GIT_URL:$GIT_PORT/$GIT_PROJECT/$GIT_REPOSITORY.git  branch=$GIT_BRANCH";
                mkdir /root/.ssh;
                cp -r /ssh-keys/. /root/.ssh/
                chmod 600 /root/.ssh/ssh-privatekey;
                chmod 644 /root/.ssh/known_hosts;
                GIT_SSH_COMMAND="ssh -i /root/.ssh/ssh-privatekey -o UserKnownHostsFile=/root/.ssh/known_hosts -o StrictHostKeyChecking=yes" \
                git clone --branch "$GIT_BRANCH" --depth 1 "ssh://git@$GIT_URL:$GIT_PORT/$GIT_PROJECT/$GIT_REPOSITORY.git" /git
            volumeMounts:
              - mountPath: /ssh-keys
                name: ssh_secret_name
                readOnly: true
              - mountPath: /git
                name: git

this can event be enhanced with sparce checkout or only copy one file only into git pvc