Support syncing PySpark applications from Git

[sbernauer]

In case you have multiple Python files that form an application it is inconvenient to put everything in a ConfigMap.
It would be nice if you could use the same mechanism as implemented in Airflow using gitsync.

Related to stackabletech/opa-operator#504, which implements the same thing for OPA regorules.

Tasks

Beta Give feedback

1. Try to move commons code into operator-rs

[iscsrwm]

As a workaround you can use an initContainer in your podOverrides.spec to sync your code. Here is an example:

driver:
podOverrides:
spec:
initContainers:
- image: k8s.gcr.io/git-sync:v3.0.1
name: git-data
volumeMounts:
- name: git-source
mountPath: /tmp/git
env:
- name: GIT_SYNC_ONE_TIME
value: "true"
- name: GIT_SYNC_REPO
value: https://gitlab.com/mycompany/myapp.git
- name: GIT_SYNC_BRANCH
value: test
- name: GIT_SYNC_DEPTH
value: "1"
- name: GIT_SYNC_DEST
value: my_app
- name: GIT_SYNC_USERNAME
valueFrom:
secretKeyRef:
name: gitpuller-creds
key: user
- name: GIT_SYNC_PASSWORD
valueFrom:
secretKeyRef:
name: gitpuller-creds
key: password
volumes:
- name: git-source
emptyDir: {}
replicas: 1
config:
resources:
cpu:
min: "1"
max: "4"
memory:
limit: "2Gi"
volumeMounts:
- name: git-source
mountPath: /common

In this example your code would be available in your driver container at /common/my_app