Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rf batched command #7252

Draft
wants to merge 10 commits into
base: maint
Choose a base branch
from

Conversation

christian-monch
Copy link
Contributor

@christian-monch christian-monch commented Jan 12, 2023

Fixes #7251

This PR uses thread-locks to ensure thread-safe determination of whether a BatchedCommand instance is active or not.

TODO:

  • add regression-tests

The definition of the class BatchedCommandError
was moved to keep class definitions together in
the source.

The definition of the function _now was moved to
keep functions together in the source.
This commit defines a context manager that
handles increasing and decreasing the
_active-attribute in BatchedCommand
This commit adds a lock to synchronise the
execution of BatchedCommand.cleanup() with
the executions of __call__ on BatchedCommand
instances. That means that no instance is
cleaned while executing a command, i.e.
sending a request and waiting for a
resppnse. The cost for that is that
cleanup cannot run during execution of any
BatchedCommand.__call__ and therefore
no BatchedCommand.__init__ will
terminate during this time, so creation
of new instances can only happen if
no BatchedCommand is currently executing.
This commit moves the update of
BatchedCommand._active_last to the
increased_active-context handler
This commit removes the global BatchedCommand
cleanup-lock in favor of inidividual locks in
the BatchedCommand instances. This prevents the
instances from serializing external calls, but
requires a synchronisation point that includes
all existing instances, when a new instance
is created.

In addition this commit renames the global
instance dictionary in order to reflect that it
contains all instances that have a runner
associated.

Work in the next commits should contain:
add a limit to the cleanup duration, use a
WeakSet instead of a WeakDictionary for the
global instance collection.
This commit changes the input to the
locked_instances-context manager to a list.
This ensures that the elements don't change
during the execution of the context manager
and that all locked elements can be unlocked.
@christian-monch christian-monch added semver-internal Changes only affect the internal API CHANGELOG-missing When a PR's description does not contain a changelog item, yet. labels Jan 12, 2023
@github-actions github-actions bot removed the CHANGELOG-missing When a PR's description does not contain a changelog item, yet. label Jan 12, 2023
@codecov
Copy link

codecov bot commented Jan 12, 2023

Codecov Report

Base: 88.71% // Head: 88.53% // Decreases project coverage by -0.18% ⚠️

Coverage data is based on head (bc4fc54) compared to base (9d3818f).
Patch coverage: 95.49% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##            maint    #7252      +/-   ##
==========================================
- Coverage   88.71%   88.53%   -0.19%     
==========================================
  Files         326      326              
  Lines       44420    44426       +6     
  Branches     5919        0    -5919     
==========================================
- Hits        39407    39332      -75     
- Misses       4998     5094      +96     
+ Partials       15        0      -15     
Impacted Files Coverage Δ
datalad/cmd.py 93.54% <95.49%> (-0.23%) ⬇️
datalad/interface/run_procedure.py 0.00% <0.00%> (-100.00%) ⬇️
datalad/support/nda_.py 0.00% <0.00%> (-54.55%) ⬇️
datalad/support/collections.py 84.37% <0.00%> (-9.38%) ⬇️
datalad/cli/interface.py 97.22% <0.00%> (-2.78%) ⬇️
datalad/support/network.py 88.22% <0.00%> (-2.60%) ⬇️
datalad/support/s3.py 40.00% <0.00%> (-2.27%) ⬇️
datalad/local/run_procedure.py 94.30% <0.00%> (-1.90%) ⬇️
datalad/utils.py 86.15% <0.00%> (-1.80%) ⬇️
datalad/local/remove.py 95.31% <0.00%> (-1.57%) ⬇️
... and 14 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

This commit applies total_seconds() to
datetime.timedelta instead of erroneously
to datetime.datetime
@christian-monch
Copy link
Contributor Author

Hi @yarikoptic, I would appreciate your thoughts on this PR and the related issue #7251

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver-internal Changes only affect the internal API
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ensure thread-safety in cleaning up inactive BatchedCommand instances
1 participant