Job Repeated Execution

[xlw712]

I recently set up a new k8s cluster and after configuring it to gocd, I found that while the job was executing, the job was re-executing after a period of execution. The following is a version of the current environment
gocd_version: 23.1.0
k8s_version: 1.27.0
gocd_agent: go-agent-23.1.0-16079.zip (I rebuilt it from another kernel)
The status quo is this

Is there any way we can check? I suspect that there is a communication problem between the agent and gocd

[xlw712]

@chadlwilson Please take a look.

[chadlwilson]

Look at the container logs and kubernetes events for the agent containers. Logs like the above are likely to happen if the container is dying or getting killed, after which GoCD will reschedule the job on a new agent. It could be that the agent is going unhealthy or being killed by your cluster, e.g if it is breaching its memory limit.

[xlw712]

But strangely enough, the same script, he's supposed to fail every time? Sometimes he can pull it off.
Has anyone else reported this problem?

[chadlwilson]

Sorry, I don't understand what you mean.

If you're saying that sometim3s the job passes and sometimes it fails then that could just be the way memory usage works within containers, and the way kubernetes decides to schedule your pods.

To know what is happening you need to find out why the agent pods/containers are dying. The kubernetes API and container logs have that information. If kubernetes killed a pod it will say so, and why. The logs will tell you if it died due to an external action (kubernetes) or an internal one (e.g the agent crashing).