Kale donation to Kubeflow

[StefanoFioravanzo]

Hello Kubeflow community! Do you remember Kale? The low-code/no-code Jupyter extension that simplifies the Data Science experience of orchestrating ML experiments from notebooks. This includes defining and deploying Kubeflow pipelines and triggering Katib hyper parameter tuning experiments with just a few clicks and code cell annotations.

If you are not familiar with it, you can read the introductory blog post or watch this video.

The project has been stale for some time and some of you may remember that Arrikto was supposed to upstream a more advanced version. Unfortunately those plans didn't pan out, Arrikto was later acquired, and the project is now locked down.

I do have administrative access to the original repository github.com/kubeflow-kale/kale and I would love to donate it to the Kubeflow community to bring it back to life.

A high-level low-code interface that quickly gets you from notebook experimentation to pipelines and other distributed training facilities is a true upselling for Kubeflow. I personally saw how appealing this approach was to end users, and several other enterprise distributions beyond Arrikto capitalized on this capability.

FUTURE WORK

I believe this approach is still relevant today and it can give Kubeflow a great value added. Jupyter notebooks are still the centerpiece for countless ML practitioners, and Kale can be Kubeflow's playground to provide high-level coding user experience.

I can definitely envision how we could evolve the current annotation-based progrmming model to better integrate with other Kubeflow components and explore ways to expose state-of-the-art capabilities such as GPT-based workflows, in-app tutorials, and integrations with upcoming Kubeflow components (e.g. model registry, spark, etc.)

WHAT'S NEEDED

The codebase is a little bit old, so I would need help with the following:

• Migrating the code under gtihub.com/kubeflow/kale
• Upgrade dependencies to resolve CVEs and update deprecated modules
• Update docker images
• Bootstrap a new documentation site
• Align the internal API with KFP v2 and the newer SDK Katib and KServe SDK versions

CALL TO ACTION

USERS: Were you (or still are!) a Kale user and want this project to be maintained by the Kubeflow community? Are you an interested user who would love a low-code solution for Kubeflow? Then please react with 👍

MAINTAINERS: Are you a code maintainer, documentation specialist, architect, or literally anything that can be helpful to the development of the project? Then please react with 🚀 and comment how you can help!

Your feedback and reactions to this issue are instrumental to validate that indeed there is still interest and to set the right direction for this project. Please comment with your opinions and ideas.

With ❤️ and 🥬 Thank you all!

[elikatsis]

@StefanoFioravanzo this is amazing news! I recalled a ton of great memories around Kale and our collaboration while reading this.

I'll definitely be watching Kale closely and will try to contribute on the development front.

Good luck!

[milosjava]

@StefanoFioravanzo I believe this project can be very useful for data scientists that are familiar with jupyter and want to use KF. I am very interested in participating.

[rochaporto]

Kale was one of the most popular features for new users in our internal kubeflow platform. We had issues with the maintenance of the component in particular during kubeflow upgrades, but would love to see this component properly integrated in the kubeflow ecosystem! We would definitely consider adding it back to our system.

[thesuperzapper]

@StefanoFioravanzo This is certainly an interesting idea.

In my mind there are a few requirements that we need to meet before accepting such a donation:

1. Ensure that Kale is still relevant to users, and that it will provide value.
2. Find at least two maintainers (preferably from different organizations), who will be responsible for maintaining it.
3. Decide which Kubeflow Working Group (WG) will own Kale, I can see arguments for any of the following:
   • Kubeflow Notebooks WG (@kubeflow/wg-notebooks-leads) - as they control the Jupyter deployments of Kubeflow, and Kale is a Jupyter plugin
   • Kubeflow Pipelines WG (@kubeflow/pipelines) - as Kale is ostensibly an extension to create Kubeflow Pipelines with a UI
   • Create a new WG - this is always a challenge, but if there are lots of willing maintainers for Kale, it might make sense.

---

I also want to note that there is a similar project to Kale called Elyra (elyra-ai/elyra) which provides a UI-like experience for Kubeflow Pipelines and Airflow, and is also owned by the Linux Foundation, under LF AI & Data. However, Elyra is also largely unmaintained (last release in March 2023).

Elyra was spearheaded by IBM, so I wonder if RedHat would be interested in picking up the torch on it (given they are now owned by IBM and have taken an interest in Kubeflow).

Perhaps we could merge Elyra and Kale into a single project under Kubeflow?

[ederign]

@thesuperzapper, we still use Elyra at ODH, but as you mentioned, the project is mostly unmaintained, and we are discussing our options forward. Certantly, Kale will be one of the options that we are going to consider. I'm checking internally, and I'll send an update here as soon as possible.