failed to relabel pod

[RubickN]

Fission/Kubernetes version

$ fission --version
client:
  fission/core:
    BuildDate: "2022-05-24T07:28:44Z"
    GitCommit: ed1e4a5c
    Version: v1.16.0
server:
  fission/core:
    BuildDate: "2022-05-24T07:28:44Z"
    GitCommit: ed1e4a5c
    Version: v1.16.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.6-tke.20", GitCommit:"f7727df6bce8b451f33c236e8e18f2e10d9dc316", GitTreeState:"clean", BuildDate:"2022-06-13T06:53:54Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes platform (e.g. Google Kubernetes Engine)

Describe the bug
Execute function seems some errors: failed to relabel pod,
To Reproduce

Expected result

Actual result

Screenshots/Dump file
{"level":"error","ts":"2022-08-06T08:45:42.659Z","logger":"generic_pool_manager.generic_pool","caller":"poolmgr/gp.go:495","msg":"failed to relabel pod","trace_id":"94eae4669972b5524ecf5d45c5d063d0","error":"context canceled","pod":"poolmgr-python-tcompose-default-2595215137-bf678765f-vmtqm","delay":409.6,"stacktrace":"github.com/fission/fission/pkg/executor/executortype/poolmgr.(*GenericPool).getFuncSvc\n\tpkg/executor/executortype/poolmgr/gp.go:495\ngithub.com/fission/fission/pkg/executor/executortype/poolmgr.(*GenericPoolManager).GetFuncSvc\n\tpkg/executor/executortype/poolmgr/gpm.go:198\ngithub.com/fission/fission/pkg/executor.(*Executor).createServiceForFunction\n\tpkg/executor/executor.go:230\ngithub.com/fission/fission/pkg/executor.(*Executor).serveCreateFuncServices.func1\n\tpkg/executor/executor.go:134"}
{"level":"info","ts":"2022-08-06T08:52:32.260Z","logger":"generic_pool_manager.generic_pool","caller":"poolmgr/gp.go:495","msg":"relabel pod","trace_id":"94eae4669972b5524ecf5d45c5d063d0","pod":"{"metadata":{"annotations":{"executorInstanceId":"fge8ubvr","fission-name":"faas","fission-uid":"452fbc04-aced-4cf9-8191-0310bf5009a2"}, "labels":{"environmentName":"python-tcompose","environmentNamespace":"default","environmentUid":"d83e67fc-3caa-4f39-abc7-324a35894b03","executorType":"poolmgr","functionName":"fn-hostinfo","functionNamespace":"default","functionUid":"ec4b6c49-042d-4202-a265-cbe53f287652","managed":"false"}}}"}
{"level":"error","ts":"2022-08-06T08:52:32.261Z","logger":"generic_pool_manager.generic_pool","caller":"poolmgr/gp.go:495","msg":"failed to relabel pod","trace_id":"94eae4669972b5524ecf5d45c5d063d0","error":"context canceled","pod":"poolmgr-python-tcompose-default-2595215137-bf678765f-vmtqm","delay":819.2,"stacktrace":"github.com/fission/fission/pkg/executor/executortype/poolmgr.(*GenericPool).getFuncSvc\n\tpkg/executor/executortype/poolmgr/gp.go:495\ngithub.com/fission/fission/pkg/executor/executortype/poolmgr.(*GenericPoolManager).GetFuncSvc\n\tpkg/executor/executortype/poolmgr/gpm.go:198\ngithub.com/fission/fission/pkg/executor.(*Executor).createServiceForFunction\n\tpkg/executor/executor.go:230\ngithub.com/fission/fission/pkg/executor.(*Executor).serveCreateFuncServices.func1\n\tpkg/executor/executor.go:134"}

$ fission support dump

Additional context
fission-dump_1659775834.zip

[github-actions[bot]]

Thank you for your first issue! ✨😊

[RubickN]

这是来自QQ邮箱的假期自动回复邮件。   您好，您发的邮件已收到，我将尽快给您回复。

[sanketsudake]

This can happen generally if request is about to timeout.

Typically when we send a request for function there is timeout for request to complete from end to end. So we all fission internal requests need to happen during timeout lets (120 sec)
Now context is way to set deadline of all internal operations within fission.
You might have seen error something like this, when we are calling relabel pod operation during specialization of function, its exceeded timeout alllowed for overall operation, context gets cancelled, because of which you might see issue

,"logger":"generic_pool_manager.generic_pool","caller":"poolmgr/gp.go:487","msg":"failed to relabel pod","trace_id":"03de95a9751eb92d9e091a0c0f5ef19d","error":"context canceled","pod":"poolmgr-go-17277-default-7023-fd6c4547f-ctn9s"

You can try to increasing timeout for request see if that helps.
Also though pool size 100, you will also need to consider different factors like rpp, concurrency based on request per second. Also check how many cold starts/specialization are happening per minute need to see pool size accordingly

[RubickN]

这是来自QQ邮箱的假期自动回复邮件。   您好，您发的邮件已收到，我将尽快给您回复。

[sanketsudake]

Since this is not a bug specifically coverting this to a discussion.

[gw123]

I find this error is not timeout， when it happend the funciton will special error at many times。

[RubickN]

这是来自QQ邮箱的假期自动回复邮件。   您好，您发的邮件已收到，我将尽快给您回复。