Help needed with deploying flyte in local kubernetes (rancher desktop)

[Yc-Chen]

Hi all,

I'm trying to deploy flyte locally as I do not want to use a remote S3 or SQL server. So I'm interested in using minio and locally deployed postgres. But I cannot get it running. This is what I see by accessing 'flyteconsole' pod:

I can see from the logs that requests reached the flyteconsole pod, but I only see blank error page.

flyteagent is having lots of errors:

I deployed it using command helm install -n flyte -f values-flyte.yaml --create-namespace flyte flyte/flyte.

This part is an outdated question; I managed to get envoy working.

I noticed flyte-contour-envoy is not running properly. Could it be the reason? [image](https://github.com/flyteorg/flyte/assets/3079912/252d9b6a-2972-4911-b65d-62ffeff96444)

This is the tutorial that I followed: https://docs.flyte.org/en/v1.0.0/deployment/sandbox.html
I'm aware that this tutorial is outdated but I up-to-date one assumes a kubernetes on AWS and I don't have that.

[davidmirror-ops]

Hi @Yc-Chen, thanks for reporting and sorry for the struggles. We definitely need to improve the UX on non cloud K8s environments

What happens if you try to reach http://localhost:30081/console?

[davidmirror-ops]

Unfortunately, I don't have a lot of experience (yet) running Flyte locally except for the all-in-one sandbox (flytectl demo start). It ships with local Postgres and minio instances. It will require a local Docker daemon running with at least 4 CPU cores assigned.

Once up, if you want to edit some of the parameters, you could try

kubectl edit cm flyte-sandbox-config -n flyte

Please let me know if this is of any help to you

[Yc-Chen]

Yes, using command flytectl demo start I can get a locally working flyte. But as next step I'm exploring having first a kubernetes running and then deploying flyte. Because I would like to deploy it on-prem with minio and postgres, whereas other examples are using s3 with eks. Is the flytectl open sourced? I can also look into what it does...

[Yc-Chen]

After trying many times with different Kubernetes version, I see this error always showing up with 'flyteagent' replica set, in the 'flyte' namespace about flyteagent service account not found.

[davidmirror-ops]

I see that the flyte-core chart includes the information to either disable the agent service or to automatically create the service account

https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/templates/agent/rbac.yaml

[Yc-Chen]

Now that I did many times comparison of the 2 ways to start flyte, I came to notice that the pod does not terminate:

I'm not sure if this is a relevant issue and what could be a workaround: #1564

Also is there a way that I could turn on more logging? I see the pod tried to run command pyflyte-fast-execute. I wonder if the command is stuck somewhere.

[Yc-Chen]

So I spent some more time and I eventually got the flow completely with a local k3s. I must admit I had to look into the repository to understand how to use helm charts. In the end this is my steps:

1. start the kubernetes cluster using rancher desktop. Uncheck the box 'Enable Traefik'. Otherwise there seems to be port conflicts.
2. Get the file charts/flyte-core/values-sandbox.yaml
3. Run:

helm upgrade -n flyte --create-namespace flyte-deps flyteorg/flyte-deps --install --set webhook.enabled=false,contour.enabled=true
helm upgrade -n flyte --create-namespace flyte-core flyteorg/flyte-core --install -f values-sandbox.yaml

It runs on http://localhost:30081. The kubernetes(k3s) version I have is v1.27.3.

Now I compare it to the failed deployment using helm install -n flyte -f values-flyte-sandbox.yaml --create-namespace flyte flyte/flyte, I can see webhook is missing from the flyte/flyte chart and maybe there are more differences. But at least I have a working local deployment using helm charts.

[davidmirror-ops]

@Yc-Chen Thanks for persevering and sharing! So, any other change you had to do to the base values-sandbox?
Also to confirm, you had to install both flyte-deps and flyte-core charts?

@wild-endeavor / @jeevb for visibility

[Yc-Chen]

Indeed, I have to install both flyte-deps and flyte-core; installing flyte chart only would end up with a hanging task. No other changes are made to values-sandbox

[davidmirror-ops]

@Yc-Chen on a related note: I've been going through the process of deploying Flyte to a local K8s cluster. My question to you is: how are you handling the connectivity to minio? I mean, for things like pyflyte run to work, flytekit needs access to -in this case- minio. Exposing it to the internet is not ideal so, how are you doing it?

Thanks

[Yc-Chen]

So I ended up using helm charts flyte-deps and flyte-core. Maybe it's a bit cheating because it's essentially what flytectl does...

From https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values-sandbox.yaml, you can see FLYTE_AWS_ENDPOINT: "http://minio.flyte.svc.cluster.local:9000". That's how flyte accesses minio.
And you can confirm minio is indeed exposing port 9000 based on this file: port: 9000

For more about DNS within kubernetes: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/