API server can't reach XRD validation webhook

[shuknk8s]

Still encountering same/similar issue when submitting a compositeresourcedefinition, I am planning to disable webhook but there is no service called validatingwebhook/crossplane in crossplane-system namespace , the only webhook related services I see in the crossplane-system is the following and I am not sure of ramification of deleting the webhook svc as this is a shared environment and other teams are working on on the same crossplane instance.

namespace is NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/crossplane-webhooks ClusterIP

Crossplane helm package version :
crossplane crossplane-system 2 2024-01-15 18:32:57.39296 -0600 -0600 deployed crossplane-1.14.5 1.14.5

Here is the error encountered while submitting simple composite resource definition
kubectl apply -f definition.yaml ─╯
Error from server (InternalError): error when creating "definition.yaml": Internal error occurred: failed calling webhook "compositeresourcedefinitions.apiextensions.crossplane.io": failed to call webhook: Post "https://crossplane-webhooks.crossplane-system.svc:9443/validate-apiextensions-crossplane-io-v1-compositeresourcedefinition?timeout=10s": context deadline exceeded

Originally posted by @shuknk8s in #4649 (comment)