New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Actions with no listeners that are old should be removed from queue #338

Open

allada opened this issue Oct 25, 2023 · 3 comments · May be fixed by #778

Assignees

Collaborator

allada commented Oct 25, 2023 •

edited by adam-singer

If an action is old (set in config) and has no more listeners it should just be evicted from the queue, since no one is waiting for the action.

see: https://github.com/TraceMachina/turbo-cache/blob/f15146d0cf94e1a9159f14db775efe4a3e27355d/cas/scheduler/simple_scheduler.rs#L432

We should implement this using timeouts and upon timing out signal to the worker to kill the action.

adam-singer self-assigned this

Contributor

zbirenbaum commented Mar 2, 2024

What exactly qualifies an action as old?

It doesn't make much sense to me to compare the timeout for an action with time elapsed since creation, so I think it has to be one of the following:

time the worker has spent trying to complete the task (would have to be a new field I think)
time since the action was loaded from CAS

Collaborator Author

allada commented Mar 2, 2024

In this case, I think it should track how long the item has been in the queue with no clients waiting on it. Then based on the config we drop or terminate the job after X number of seconds.

Contributor

zbirenbaum commented Mar 2, 2024

In this case, I think it should track how long the item has been in the queue with no clients waiting on it. Then based on the config we drop or terminate the job after X number of seconds.

Got it, thanks for the clarification!

zbirenbaum mentioned this issue

Remove no listeners #717

Closed

2 tasks

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

e294a87

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

4b1c4de

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum linked a pull request

that will close this issue

Remove old actions with no listeners #778

Open

5 tasks

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

3ddef23

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

ecadfb0

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

d5c86a4

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

4200e73

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

28aca33

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

307cbb4

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

2e287e5

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

b12d3f5

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement kill request api on scheduler

10ee56f

Implements scheduler api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

ead0531

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be received by the
worker.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement kill request api on scheduler

846cdd6

Implements scheduler api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

042a2fb

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

8959bed

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

4ce703d

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

55b1a7d

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

18f2ca9

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum mentioned this issue

Implement worker api for killing running actions #840

Merged

5 tasks

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

b91f05d

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

95edf4b

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Add scheduler function to kill action on worker

Adds methods to send a request from the scheduler to a worker for
a running action to be killed. Allows for action cancellation
requests to be sent from the scheduler during scenarios such
as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

b9caf1e

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Add scheduler function to kill action on worker

e1e7031

Adds methods to send a request from the scheduler to a worker for
a running action to be killed. Allows for action cancellation
requests to be sent from the scheduler during scenarios such
as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Add scheduler function to kill action on worker

354e5f6

Adds methods to send a request from the scheduler to a worker for
a running action to be killed. Allows for action cancellation
requests to be sent from the scheduler during scenarios such
as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Add scheduler function to kill action on worker

c90e09f

Adds methods to send a request from the scheduler to a worker for
a running action to be killed. Allows for action cancellation
requests to be sent from the scheduler during scenarios such
as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Add scheduler function to kill action on worker

e13020b

Adds methods to send a request from the scheduler to a worker for
a running action to be killed. Allows for action cancellation
requests to be sent from the scheduler during scenarios such
as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Add scheduler function to kill action on worker

d135a7c

Adds methods to send a request from the scheduler to a worker for
a running action to be killed. Allows for action cancellation
requests to be sent from the scheduler during scenarios such
as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

c40ffe6

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

9b72cad

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

faca603

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement worker api for killing running actions

c55983d

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards TraceMachina#338

zbirenbaum added a commit that referenced this issue


Implement worker api for killing running actions (#840)

abf12e8

Implements worker api for requesting a currently running action to be
killed. Allows for action cancellation requests to be sent from the
scheduler during scenarios such as client disconnection.

towards #338

Co-Authored-By: Zach Birenbaum <[email protected]>

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement kill action requests on scheduler

966e5de

Implements scheduler api for sending a kill action request to a worker.
Allows for action cancellation during scenarios such as client
disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement kill action requests on scheduler

62c2ba7

Implements scheduler api for sending a kill action request to a worker.
Allows for action cancellation during scenarios such as client
disconnection.

towards TraceMachina#338

zbirenbaum mentioned this issue

Implement kill action requests on scheduler #842

Open

5 tasks

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Implement kill action requests on scheduler

24b7bd2

Implements scheduler api for sending a kill action request to a worker.
Allows for action cancellation during scenarios such as client
disconnection.

towards TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

57bd35a

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

624eda2

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

1bbc496

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

825aa79

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

zbirenbaum pushed a commit to zbirenbaum/nativelink that referenced this issue


Remove old actions with no listeners

d971379

Implement scheduler side removal of actions with no listeners. Adds
disconnect_timeout_s configuration field with default of 60s. If the
client waiting on a given action is disconnected for longer than this
duration without reconnecting the scheduler will stop tracking it. This
does not remove it from the worker if the job has already been
dispatched.

fixes TraceMachina#338

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment