[TRACKING] Kubeflow 1.9.0 (Notebooks WG Components)

[thesuperzapper]

Originally, this issue was tracking a 1.8.1 release, but now that 1.9.0 is so close, we will just include them in that.

We are planning to do an overall Kubeflow 1.8.1 patch release which will include a number of fixes to issues uncovered/missed in the 1.8.0 release.

Missing Stuff

There may have been other critical PRs / Issues raised since we created this issue.

We should review the recent activity kubeflow/kubeflow repo before finalizing this list:

• https://github.com/kubeflow/kubeflow/issues
• https://github.com/kubeflow/kubeflow/pulls

Other Tracking Issues

• Manifests (All Components): KF release 1.9 tracking manifests#2600
• Distributions (All Components): [Kubeflow 1.9] Distributions and Kubeflow 1.9 manifests#2611
• Roadmap (Notebooks WG): Notebooks WG roadmap for KF 1.9 manifests#2608

Deferred Changes (after 1.9.0-rc.0)

Notebook Images

• It looks like the example-notebook-servers for TensorFlow are failing to build due to an upstream pip change:
  • https://github.com/kubeflow/kubeflow/actions/runs/9216437170/job/25357796150

Notebooks/TensorBoards Controller

• tensorboard fails to connect if there is a notebook with the same name #7455
  • Fixed By: fix: resolve service name conflict for notebooks and tensorboards #7468
  • WARNING: this needs a lot of testing and work to ensure it is safe to do in-place upgrades

Central Dashboard

• Email validation regex is too restrictive #5004
  • Fixed By: (NEED A PR)
    • NOTE: we should use the same email validation as the HTML <input type="email">, which is described here: https://html.spec.whatwg.org/multipage/input.html#valid-e-mail-address

Volumes WebApp

• volume app list pvcs facing 404 error after upgrade kubeflow from v1.7 to v1.8 #7414
  • Caused by: Add PVCViewer Frontend Integration #7179 (see this part)
  • Fixed By: (NEED A PR)

---

Changes for 1.9.0

We need to finish/fix these PRs and then cherry-pick them into the v1.9-branch before cutting the 1.9.0 tag.

Notebook Images

• jupyter notebook not connect only #7370
  • Fixed By: fix: jupyterlab runtime volume permissions #7407
• Notebook containers can time out starting with very slow storage:
  • Fixed By: Prevent S6 timeouts on container start by setting S6_CMD_WAIT_FOR_SER… #7421
  • Follow Up PR: fix: increase S6_CMD_WAIT_FOR_SERVICES_MAXTI… #7496

Notebooks/TensorBoards Controller

• Hard-coded VirtualService hosts:
  • Fixed By: feat: allow setting ISTIO_HOST in notebook/tensorboard controller for Istio VirtualServices #6902
• Cannot see the list of notebooks [500] An error occurred in the backend.  #6886
  • Fixed by: fix(notebooks): add CRD validation for empty image and container #7495

PVC Viewer Controller

• chore: update pvcviewer dependencies #7561

Notebook Spawner UI (Jupyter Web App)

• Trim whitespace on custom images:
  • Fixed By: Trim the custom Jupyter image string #6529
• Notebook doesn't created after upgrade to Kubeflow 1.8 #7456
  • Fixed By: fix: 🏷️ Ensure username is returned by get_username as a string #7457
• Create notebooks page and get [500] error #7441
  • Caused by: Handle edge case when node capacity is undefined gracefully. Closes #7280. #7281
  • Fixed by: fix(crud-web-apps/jupyter): dict variable reference for node.status.capacity #7485
• Creating or starting notebook throws 500 error - KeyError: 'message' #7454
  • Caused by: jwa: Improve how JWA exposes errors #6952 (see get_status_from_container_state())
  • Fixed by: fix(notebooks): correctly handle status.containerState messages #7585

PVC Viewer UI

• Show "waiting" status on PVC Viewer Pods:
  • Fixed By: fix: return waiting on no pvcviewer status #7350

Tensorboards Web App

• fix: remove incorrect space in TWA_APP_SECURE_COOKIES config #7274

Central Dashboard

• fix: update central dashboard to NodeJS 16 #7578
• feat: big update for manage contributors page #7582
• feat: update dashboard styles and links #7583
• feat(dashboard): allow using {ns} template in externalLinks #7138
• centraldashboard: don't display All namespaces in Manage Contributors screen (#7188) #7242
• fix: incorrect references DASHBOARD_LINKS_CONFIGMAP env-var #7264

---

Future Changes for 1.9.1

Repo Cleanup and CI

• ci: read-only token-permissions for all workflows #7278

Notebooks Controller

• notebook-controller: Update dependency versions #7152
  • WARNING: this can be deferred to a future update as it will need to be rebased and retested.
• feat: always set a Pod status in Notebook CRD #7015
  • WARNING: needs to be reviewed
• fix: add lease RBAC to notebook controller role (for leader election) #6942
  • WARNING: needs to be reviewed
• feat: expose runningCount and totalCount in notebook controller metrics #6838
  • WARNING: needs to be reviewed
• feat: expose NotebookCullingCount and NotebookCullingTimestamp in prometheus metrics #6488
  • WARNING: needs to be reviewed
• fix: unnecessary notebook updates caused by compare issues #6474
  • WARNING: needs to be reviewed
• fix: add lease RBAC to notebook controller role (for leader election) #6942
  • WARNING: needs to be reviewed

Profile Controller

• profile controller fails to update profile owner #6576
  • Fixed by: Fix (profile controller): Profile controller fails to update profile owner, fixes #6576 and #5449 #7276
  • Fixed By: feat: allow updating profile owners #7547
    • WARNING: this needs a bit of work, it is NOT ready to merge
• feat(profile-controller): propagate labels from CR to namespace #7481
  • THIS SHOULD NOT BE INCLUDED
  • WARNING: including this introduces issues for ArgoCD and possible security issues by allowing users to set arbitrary namespace labels by updating profile labels.
• fix: extend profile AuthorizationPolicy for KServe models #6627
  • WARNING: needs to be updated
• fix: stop unnecessary updates of profiles #6435
  • WARNING: needs to be reviewed
• feat: make profile controller error logs more descriptive #6231
  • WARNING: needs to be reviewed

Tensorboard Controller

• fix: add error handling to tensorboard controller #7067
  • WARNING: needs to be reviewed
• fix: use "required" in tensorboard node affinity, rather than "preferred" #7024
  • WARNING: needs to be reviewed
• feat: make TensorBoard image configurable in CRD #6783
  • WARNING: needs to be reviewed

Notebook Spawner UI

• feat: add gpusCount to spawner_ui_config.yaml #7295
  • WARNING: in its current state, this is a breaking change, we need to find a way to make it backwards compatible.
• feat: add memoryVendors and arbitrary counts for notebook GPU resource limits #6657
  • WARNING: there are a bunch of PRs which do similar things, we should only merge one of them
• feat(notebooks): only display allocatable GPU configuration when creating notebook #7124
  • WARNING: needs to be reviewed
• feat(notebook): dynamically display the allocatable GPU type/num in notebook server #7134
  • WARNING: needs to be reviewed
• web-apps: Implement watch to avoid constant polling from backend to K8s #7080
  • WARNING: needs to be reviewed
• fix: notebook ui should not break for invalid PodDefaults #6983
  • WARNING: needs to be reviewed
• fix: typo when setting server_type for jupyter notebook (no in app affect) #6666
  • this change does nothing, but is fine to fix a typo
• fix: correctly display notebook images from registries with port #6351
  • WARNING: needs to be reviewed

Central Dashboard

• fix: show correct BUILD_VERSION in centraldashboard #6978
  • THIS SHOULD NOT BE INCLUDED
  • WARNING: we can probably just remove the build version section of the UI instead
• fix: double URL encoding in central-dashboard links #6564
  • WARNING: needs to be reviewed
• feat(dashboard): include fonts as assets #6009
  • WARNING: we need to confirm we are allowed to include these fonts in Apache 2.0 licenced code
• fix: show error logs for YAML parsing in notebooks ui #7497
  • WARNING: needs to be reviewed

Admission Webhook (PodDefaults)

• jupyter-web-app's PodDefault configurations are keyed by their label selector's key, not their name #7552
  • (Need PR)
• feat: allow same volume to be mounted multiple times in single PodDefault #6847
  • WARNING: needs to be reviewed

KFAM

• feat: use native RBAC subject to find RoleBindings in KFAM, rather than annotations #6839
  • WARNING: needs to be reviewed

[thesuperzapper]

/cc @kubeflow/wg-notebooks-leads @DnPlas

[DnPlas]

hey @thesuperzapper, just to check for updates on this, when can we cut the new patch release? We'd like to work on the patch release next week.

[thesuperzapper]

@kimwnasptd it looks like a number of regressions were also introduced in the Kubeflow Notebooks 1.8.0 which MUST be resolved before 1.8.1.

All are caused by trying to access parts of a YAML/object that are not defined (causing "key errors" in Python).

• Parsing container state fails:
  • Issues:
    • Creating or starting notebook throws 500 error - KeyError: 'message' #7454
  • Caused by PR:
    • jwa: Improve how JWA exposes errors #6952
      • (see get_status_from_container_state())
• Parsing GPU vendors fails:
  • Issue:
    • Create notebooks page and get [500] error #7441
  • Caused by PR:
    • Handle edge case when node capacity is undefined gracefully. Closes #7280. #7281
• Parsing Volumes List fails:
  • Issue:
    • volume app list pvcs facing 404 error after upgrade kubeflow from v1.7 to v1.8 #7414
  • Caused by PRs:
    • Add PVCViewer Frontend Integration #7179
      • (see this part)

[juliusvonkohout]

#7421 might be relevant as well.

[thesuperzapper]

It looks like there is another critical issue with 1.8.0 (and all previous versions):

• Cannot see the list of notebooks [500] An error occurred in the backend.  #6886

Which causes the Notebook UI to crash loop when a Notebook with a null/empty image is created.

[juliusvonkohout]

Regarding "WARNING: we might want to consider setting a non-infinite timeout, so that notebooks pods will eventually crash-loop, rather than hanging forever." What about just using one minute as timeout?

[juliusvonkohout]

#7325 is what would be useful for 1.9. Even if we just merge a reduced version and improve later on.

[thesuperzapper]

Let's try and put all the open issues (that we plan to fix in 1.9.0) in the milestone I just created, it will make it easier to track:

• https://github.com/kubeflow/kubeflow/milestone/4

[juliusvonkohout]

Can you add #7325 to https://github.com/kubeflow/kubeflow/milestone/4

[thesuperzapper]

Can you add #7325 to https://github.com/kubeflow/kubeflow/milestone/4

@juliusvonkohout not for the 1.9.0 release, we really need to get it moving.

We can discuss it for 1.9.1, but even then, I am not sold that we actually need such a complex solution to a problem which can be solved by just including two copies of the ConfigMap in the manifests (see #7325 (comment)).

[thesuperzapper]

I have tried to go through every open (human-authored) PR, and group them by component in this issue under the "Future 1.9.1" section.

I have created a separate milestone for 1.9.1: https://github.com/kubeflow/kubeflow/milestone/5

WARNING: none are ready to merge, and not all will make it into 1.9.1, but having them grouped should help us review them.

[StefanoFioravanzo]

Thank you @thesuperzapper , these Milestones are really helpful not only for you but for others to keep track of the pending work. Other WGs should take inspiration!

[juliusvonkohout]

Can you add #7325 to https://github.com/kubeflow/kubeflow/milestone/4

@juliusvonkohout not for the 1.9.0 release, we really need to get it moving.

We can discuss it for 1.9.1, but even then, I am not sold that we actually need such a complex solution to a problem which can be solved by just including two copies of the ConfigMap in the manifests (see #7325 (comment)).

Even merging only the first part for code maintenance without the second set of labels is useful.