scheduler_perf: Allow each test case to specify its timeout

[utam0k]

/kind feature
/sig scheduling

what

Allow each test case to specify its timeout. Now, it is specified 30 minutes as hard coding.

kubernetes/test/integration/scheduler_perf/scheduler_perf.go

Lines 759 to 760 in 99a8a6f

	// 30 minutes should be plenty enough even for the 5000-node tests.
	timeout := 30 * time.Minute

why

I admit that 30 minutes is enough for the default scheduler with the in-tree scheduler plugins, but scheduler_perf is also helpful for other schedulers with outside scheduler plugins. In that case, more than 30 minutes might be required to complete a test case.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[utam0k]

@kubernetes/sig-scheduling-approvers I'd be happy to contribute to this issue if it is acceptable.

[kerthcet]

User case is first, do you have any real case here.

[sanposhiho]

Agree with @kerthcet. If you want to define a test case which takes more than 30 mins to finish, it feels like that's something undesired.

Additionally to say, even if there's a usecase (let's say using ~1 hour) actually, could we just extend the global timeout to 1 hour? I don't see any necessity of per-test timeout.

[utam0k]

User case is first, do you have any real case here.

It's very important. Let me share my experience. Unfortunately, I know of a scheduler with fairly poor throughput. That is like an order of magnitude slower throughput. For example, a co-scheduling plugin would be something that could significantly reduce throughput. However, we would like to verify throughput and deadlock on a large number of nodes (possibly an endurance test). In this case, I think scheduler-perf is currently appropriate. However, it will take more than 30 minutes. If preemption is also included, there is an extra possibility of this.

Additionally to say, even if there's a usecase (let's say using ~1 hour) actually, could we just extend the global timeout to 1 hour? I don't see any necessity of per-test timeout.

Certainly, this could be the case. However, in the case of fast label, for example, it is expected to fail quickly. In this case, it seems a bit wasteful to wait an hour for a deadlock or something

@kerthcet @sanposhiho Given your extensive experience, I would appreciate hearing your perspective. Could you please share your thoughts? 🙏

[sanposhiho]

Even if the scheduler is 10 times slower (= 30 pods/s throughput),
the scheduler can handle 54000 scheduling cycles within 30 min theoretically.
Did your test actually get failed by this 30 min timeout? I mean, for example, didn't your test case include something mistaken and some Pods left unschedulable forever?
It sounds like too slow (as long as you used an appropriate sized test case), even if your test involves preemption.

[utam0k]

Did your test actually get failed by this 30 min timeout?

No, I didn't. It's fair to reopen this when I get actually this situation. Thanks for your input.