Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include UUID in ActionState #927

Merged
merged 1 commit into from
May 29, 2024
Merged

Conversation

zbirenbaum
Copy link
Contributor

@zbirenbaum zbirenbaum commented May 22, 2024

Description

Changes the field used for identifying which action an ActionState corresponds to include a uuid as well as the ActionInfoHashKey. All existing functionality is kept the same by making use of the nested ActionInfoHashKey contained within the Id. These changes will provide the basis for all usage of Id in followup changes to the scheduler. This breaks compatibility for forwarding an operation from one remote execution system to another that does not use our operation name format (ie: very unlikely, but possible).

Please delete options that aren't relevant.

  • Refactor
  • Breaking Change

How Has This Been Tested?

All existing tests have been modified to remain functionally equivalent.

Checklist

  • Updated documentation if needed
  • Tests added/amended
  • bazel test //... passes locally
  • PR is contained in a single commit, using git amend see some docs

This change is Reviewable

@zbirenbaum zbirenbaum requested a review from allada May 22, 2024 23:31
@CLAassistant
Copy link

CLAassistant commented May 22, 2024

CLA assistant check
All committers have signed the CLA.

@zbirenbaum zbirenbaum force-pushed the action-id branch 2 times, most recently from 0b0fac6 to bf424be Compare May 23, 2024 01:16
Copy link
Contributor

@adam-singer adam-singer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 9 of 9 files at r1, all commit messages.
Reviewable status: 0 of 1 LGTMs obtained, and pending CI: license/cla, pre-commit-checks, and 1 discussions need to be resolved


nativelink-util/src/action_messages.rs line 60 at r1 (raw file):

}

impl Id {

Can we make units for Id that test creation, comparison, parsing? Would also be nice to get a sense of what this field looks like as a string to be parsed.

@zbirenbaum zbirenbaum force-pushed the action-id branch 3 times, most recently from d7c88bc to 37d2ba3 Compare May 29, 2024 01:47
Copy link
Contributor Author

@zbirenbaum zbirenbaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 1 LGTMs obtained, and pending CI: Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), license/cla, macos-13, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable, and 1 discussions need to be resolved


nativelink-util/src/action_messages.rs line 60 at r1 (raw file):

Previously, adam-singer (Adam Singer) wrote…

Can we make units for Id that test creation, comparison, parsing? Would also be nice to get a sense of what this field looks like as a string to be parsed.

The format for the string is just {unique_qualifier}:{uuid}

@allada I believe when we just spoke you felt tests weren't required for this refactor as long as all existing tests pass correct?

Copy link
Collaborator

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 8 of 8 files at r2, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), license/cla, macos-13, ubuntu-22.04, and 4 discussions need to be resolved


nativelink-scheduler/src/simple_scheduler.rs line 203 at r2 (raw file):

    }
}
impl Borrow<ActionInfoHashKey> for CompletedAction {

I think you can remove this now?


nativelink-util/src/action_messages.rs line 60 at r1 (raw file):

Previously, zbirenbaum (Zach Birenbaum) wrote…

The format for the string is just {unique_qualifier}:{uuid}

@allada I believe when we just spoke you felt tests weren't required for this refactor as long as all existing tests pass correct?

I'd say yes, we probably don't need a test for this unless we want the external-visible format of OperationId to be stable across versions, but I don't think we should strive for that just yet.


nativelink-util/src/action_messages.rs line 103 at r2 (raw file):

impl std::fmt::Debug for OperationId {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        let id_str = format!("{}:{}", self.unique_qualifier.action_name(), self.id);

nit: use f.write_fmt(). It will remove a needless string allocation.


nativelink-util/src/action_messages.rs line 112 at r2 (raw file):

pub struct WorkerId(pub Uuid);

impl std::fmt::Display for WorkerId {

ditto.

Copy link
Contributor Author

@zbirenbaum zbirenbaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Vercel, pre-commit-checks, and 3 discussions need to be resolved


nativelink-scheduler/src/simple_scheduler.rs line 203 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

I think you can remove this now?

Unfortunately not. Until everything is migrated to use an OperationId map, find_recently_completed_action and probably other places will error out if you remove this code.


nativelink-util/src/action_messages.rs line 103 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

nit: use f.write_fmt(). It will remove a needless string allocation.

Done.


nativelink-util/src/action_messages.rs line 112 at r2 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

ditto.

Done.

This was referenced May 29, 2024
@zbirenbaum zbirenbaum force-pushed the action-id branch 3 times, most recently from 25fea5a to 9d3849c Compare May 29, 2024 18:17
Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).
Copy link
Contributor Author

@zbirenbaum zbirenbaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dismissed @adam-singer from a discussion.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), macos-13, ubuntu-20.04 / stable, ubuntu-22.04 / stable, windows-2022 / stable, and 2 discussions need to be resolved


nativelink-util/src/action_messages.rs line 60 at r1 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

I'd say yes, we probably don't need a test for this unless we want the external-visible format of OperationId to be stable across versions, but I don't think we should strive for that just yet.

Done.

Copy link
Contributor

@adam-singer adam-singer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 6 of 8 files at r2, 4 of 4 files at r4, all commit messages.
Reviewable status: 2 of 1 LGTMs obtained, and 2 discussions need to be resolved

Copy link
Collaborator

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 3 of 4 files at r4, all commit messages.
Reviewable status: :shipit: complete! 2 of 1 LGTMs obtained

Copy link
Collaborator

@allada allada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 9 files at r1, 1 of 4 files at r4.
Reviewable status: :shipit: complete! 2 of 1 LGTMs obtained

@zbirenbaum zbirenbaum merged commit b07ca1d into TraceMachina:main May 29, 2024
28 checks passed
nfarah86 pushed a commit to nfarah86/nativelink that referenced this pull request Jun 1, 2024
Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <[email protected]>
MarcusSorealheis pushed a commit to nfarah86/nativelink that referenced this pull request Jun 1, 2024
author Nathan (Blaise) Bruer <[email protected]> 1716060089 -0700
committer Marcus <[email protected]> 1717216579 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapUMACgkQx/6q6dPS
 ZF2kPQ//ZYXhy26xNq5g2P6U7kf+dpcXok2rzntw9620kdNCLodCm7vceDvwa1mn
 JwVnXdGO7QcrLeSagHk6LewptncYq+hOc3ZA7wUtM3bSUTwPlwJue7zNVwpMSZHD
 9pdu9CDgVWh6h7oR+JoRcyHjJ7NTkSU41aLEtwaHbDWatmtX2Cy4iX0BNk55nWaK
 BtzzuLLgiWrpOXNLL4MWlRdxLbN2YjbxW+2S3Q5eg7SnlERKU+COLoaHYUrk8v9J
 qzK35H05wDiiXHQpnWky895G3DDo/NB1znVpyOf/3sPw+OwUvACoZjORIbeh7+0W
 J5NqfX1ngWTtu0dVT1tehguAplVrBCWGJgJcVs03egVlOVSL/WthdkjW/ZkMfEk5
 pUL9Gje8SyuyEMw3tUc8EV6owwLqqyvVsaO+baVZR5hAbxp/yDaT/fewic6OXa4X
 I58NE5a7w9b5Zsem5kW+yFD5GSa11sybrqffd+ofzPmmDvlVYmJ/CaQvTZ/lQpaA
 IIZLyPYcicC3v2I1+GA//QFPJsUlK0McW+MR7L6zlY395yvS6TTA3EoCBY5KrsYD
 inSfeY2IuMKjV81ihQWd56XiK289GKkvspUmjy84ytke0luqM7Xfpu3cii+gCwoq
 8B0dUCJOqEKg1AJUlM4VOS7vAwc1tJHHrEm05/opIycQxkwNnhs=
 =6j1C
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <[email protected]> 1716060089 -0700
committer Marcus <[email protected]> 1717216536 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapRgACgkQx/6q6dPS
 ZF1U1Q/9G9ZN2PkCUxBc+oTN1BjBbNB/52+xmi+Uao39vrGh2vPIOGYbjXPP36Vv
 NUQ5cckmlsoqC1VWWgJMts27Y6oSlP1JydRlgrXmhnNV/SjcAz6r2woP+3At58hk
 iZN0lqft7lZ1Viycq8bDDEDrT56BiGGbDqO0E3UMZncwp/4s5Cw8M+0DTapIz2PU
 UcxJk2goTlKwPIItlFBUF2dTPL+nxtI95rpIdaGRrP/pn7XaRuY72k5pGMVZHsS+
 OVQBmXqYrSh3nUim+YWagvbDrYJU6lksI4ZDN6cnVgdEsF0xEWguCy3g6tR0WVUQ
 Z5Y9BkedyWt7v1tK+B5cBQgoP2m34pY8TzLFbQTLkv7mrxLIwyNqkDLVJOCPcamk
 dJ+yf5lp54vUkeRnVaTcp9IMEjF0Vjteg3WHTKNK8ZsBs9DnNzb+7HyjDa6bYcau
 e0Y5HVPuC5kcTMK95yTkAabdj1qtBr8QpJ7CErqTRBgbxpcgldd3DTNndHv5/Yjz
 QH/zctjfJcISk/Y+KDTBhZRsNF1VGAzpwsT3KnFUdcTg48xMFC6IBI/YkYywg4Ra
 ZIv6O9h0V7FivbXF5J0KqawpW4lQa6mLDzNwm7ZNODVNb5TEGYu8jsYMzkxaKVPR
 ZGEtFuBOPGgLVYcIbuAFeANBERzhw5KUMe7MBSP2DG7r+50pqQI=
 =zuq5
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <[email protected]> 1716060089 -0700
committer Marcus <[email protected]> 1717216503 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapPcACgkQx/6q6dPS
 ZF0xpQ/+ILuc9NSpJpS6BNw5IUyE4cQsquFZhKBTsX1l8/ZMkDZR6pKdO+gsUdKB
 DyA8OX8VspRAWNoxyiEfLRknwNtclr6XG1znX+WcyVjnaujcezau54dhoEAsImdn
 pgwfN/RpLh3ii1Z/MbpPlDzdDcYkOtDiqF2l4Cr2wQSs72o6qONUmCrycNdipyHa
 lED+VesZOZiZTEBOX56hMGQ/NVSPQ9hisIVjFq1J472v+opU5Ht0cu8KSCRA2lud
 2DCVwpnyjIcG9Bf/XlCMzw9lODbOTqehaGsmES6CSLngjzeojhiHqABc5qYQT6Ip
 umWgSV2/AQsNcMG69WB+uPPT1aBwT//XnBfUoSazhGvFQLn9A7oLPLarhZGUP8M1
 IND49L3OBpzbbdY3ipN9e22f4NcYg7irhy/dzLIHjtBd9CPlUTQTybmRz6KebJom
 6Ha/9CXKPcaW23NqsMNlCoWqSL+cIAEvYgEWcZrceOvkeEHppI6pSLMLyWHw9epF
 Odd3AX8EP18axCNe+GJlCOZlRFq9cNXAYuboa0HpHhBbMP7FiVuQhapR/iWTA1Pv
 2X+hjkWURzpwFz+R/ObTBMiI1YqrTHkD7fukb1ibELenCs8UeIznZ7ScV6YX72Ef
 rOoxoHMrDRbXsWpc3JG3TlEbYpdpiWN09sQQpktCwZtU/uh5JsQ=
 =DmRi
 -----END PGP SIGNATURE-----

[Breaking] Digest function now auto-detected from request (TraceMachina#899)

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.

Update images for docs (TraceMachina#930)

Update Rust crate proc-macro2 to v1.0.84 (TraceMachina#916)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <[email protected]>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <[email protected]>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update dependency protobuf to v26.0.bcr.1 (TraceMachina#887)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate quote to v1.0.36 (TraceMachina#938)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump flake and Bazel modules (TraceMachina#947)

Fixes TraceMachina#943

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update images for docs (TraceMachina#930)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <[email protected]>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <[email protected]>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.
MarcusSorealheis pushed a commit to nfarah86/nativelink that referenced this pull request Jun 1, 2024
author Nathan (Blaise) Bruer <[email protected]> 1716060089 -0700
committer Marcus <[email protected]> 1717216579 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapUMACgkQx/6q6dPS
 ZF2kPQ//ZYXhy26xNq5g2P6U7kf+dpcXok2rzntw9620kdNCLodCm7vceDvwa1mn
 JwVnXdGO7QcrLeSagHk6LewptncYq+hOc3ZA7wUtM3bSUTwPlwJue7zNVwpMSZHD
 9pdu9CDgVWh6h7oR+JoRcyHjJ7NTkSU41aLEtwaHbDWatmtX2Cy4iX0BNk55nWaK
 BtzzuLLgiWrpOXNLL4MWlRdxLbN2YjbxW+2S3Q5eg7SnlERKU+COLoaHYUrk8v9J
 qzK35H05wDiiXHQpnWky895G3DDo/NB1znVpyOf/3sPw+OwUvACoZjORIbeh7+0W
 J5NqfX1ngWTtu0dVT1tehguAplVrBCWGJgJcVs03egVlOVSL/WthdkjW/ZkMfEk5
 pUL9Gje8SyuyEMw3tUc8EV6owwLqqyvVsaO+baVZR5hAbxp/yDaT/fewic6OXa4X
 I58NE5a7w9b5Zsem5kW+yFD5GSa11sybrqffd+ofzPmmDvlVYmJ/CaQvTZ/lQpaA
 IIZLyPYcicC3v2I1+GA//QFPJsUlK0McW+MR7L6zlY395yvS6TTA3EoCBY5KrsYD
 inSfeY2IuMKjV81ihQWd56XiK289GKkvspUmjy84ytke0luqM7Xfpu3cii+gCwoq
 8B0dUCJOqEKg1AJUlM4VOS7vAwc1tJHHrEm05/opIycQxkwNnhs=
 =6j1C
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <[email protected]> 1716060089 -0700
committer Marcus <[email protected]> 1717216536 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapRgACgkQx/6q6dPS
 ZF1U1Q/9G9ZN2PkCUxBc+oTN1BjBbNB/52+xmi+Uao39vrGh2vPIOGYbjXPP36Vv
 NUQ5cckmlsoqC1VWWgJMts27Y6oSlP1JydRlgrXmhnNV/SjcAz6r2woP+3At58hk
 iZN0lqft7lZ1Viycq8bDDEDrT56BiGGbDqO0E3UMZncwp/4s5Cw8M+0DTapIz2PU
 UcxJk2goTlKwPIItlFBUF2dTPL+nxtI95rpIdaGRrP/pn7XaRuY72k5pGMVZHsS+
 OVQBmXqYrSh3nUim+YWagvbDrYJU6lksI4ZDN6cnVgdEsF0xEWguCy3g6tR0WVUQ
 Z5Y9BkedyWt7v1tK+B5cBQgoP2m34pY8TzLFbQTLkv7mrxLIwyNqkDLVJOCPcamk
 dJ+yf5lp54vUkeRnVaTcp9IMEjF0Vjteg3WHTKNK8ZsBs9DnNzb+7HyjDa6bYcau
 e0Y5HVPuC5kcTMK95yTkAabdj1qtBr8QpJ7CErqTRBgbxpcgldd3DTNndHv5/Yjz
 QH/zctjfJcISk/Y+KDTBhZRsNF1VGAzpwsT3KnFUdcTg48xMFC6IBI/YkYywg4Ra
 ZIv6O9h0V7FivbXF5J0KqawpW4lQa6mLDzNwm7ZNODVNb5TEGYu8jsYMzkxaKVPR
 ZGEtFuBOPGgLVYcIbuAFeANBERzhw5KUMe7MBSP2DG7r+50pqQI=
 =zuq5
 -----END PGP SIGNATURE-----

parent 0a33c83
author Nathan (Blaise) Bruer <[email protected]> 1716060089 -0700
committer Marcus <[email protected]> 1717216503 -0700
gpgsig -----BEGIN PGP SIGNATURE-----

 iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapPcACgkQx/6q6dPS
 ZF0xpQ/+ILuc9NSpJpS6BNw5IUyE4cQsquFZhKBTsX1l8/ZMkDZR6pKdO+gsUdKB
 DyA8OX8VspRAWNoxyiEfLRknwNtclr6XG1znX+WcyVjnaujcezau54dhoEAsImdn
 pgwfN/RpLh3ii1Z/MbpPlDzdDcYkOtDiqF2l4Cr2wQSs72o6qONUmCrycNdipyHa
 lED+VesZOZiZTEBOX56hMGQ/NVSPQ9hisIVjFq1J472v+opU5Ht0cu8KSCRA2lud
 2DCVwpnyjIcG9Bf/XlCMzw9lODbOTqehaGsmES6CSLngjzeojhiHqABc5qYQT6Ip
 umWgSV2/AQsNcMG69WB+uPPT1aBwT//XnBfUoSazhGvFQLn9A7oLPLarhZGUP8M1
 IND49L3OBpzbbdY3ipN9e22f4NcYg7irhy/dzLIHjtBd9CPlUTQTybmRz6KebJom
 6Ha/9CXKPcaW23NqsMNlCoWqSL+cIAEvYgEWcZrceOvkeEHppI6pSLMLyWHw9epF
 Odd3AX8EP18axCNe+GJlCOZlRFq9cNXAYuboa0HpHhBbMP7FiVuQhapR/iWTA1Pv
 2X+hjkWURzpwFz+R/ObTBMiI1YqrTHkD7fukb1ibELenCs8UeIznZ7ScV6YX72Ef
 rOoxoHMrDRbXsWpc3JG3TlEbYpdpiWN09sQQpktCwZtU/uh5JsQ=
 =DmRi
 -----END PGP SIGNATURE-----

[Breaking] Digest function now auto-detected from request (TraceMachina#899)

The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.

If you use `verify` store and use `hash_verification_function`,
simply change it to `"verify_hash": true`. We will now auto-detect
the hash function based on the request and if it is not set, we
use the `default_digest_hash_function` in the `global` config.

Update images for docs (TraceMachina#930)

Update Rust crate proc-macro2 to v1.0.84 (TraceMachina#916)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <[email protected]>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <[email protected]>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update dependency protobuf to v26.0.bcr.1 (TraceMachina#887)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate quote to v1.0.36 (TraceMachina#938)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump flake and Bazel modules (TraceMachina#947)

Fixes TraceMachina#943

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update images for docs (TraceMachina#930)

Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932)

In prep to move to an abstract key instead of DigestInfo, this
change cleans up EvictingMap and removes dependencies on DigestInfo.

towards: TraceMachina#931

Update Rust crate mimalloc to v0.1.42 (TraceMachina#933)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Update Rust crate parking_lot to v0.12.3 (TraceMachina#936)

Include UUID in ActionState (TraceMachina#927)

Changes the field used for identifying which action an ActionState
corresponds to include a uuid as well as the ActionInfoHashKey. All
existing functionality is kept the same by making use of the nested
ActionInfoHashKey contained within the Id. These changes will provide
the basis for all usage of Id in followup changes to the scheduler.
This breaks compatibility for forwarding an operation from one remote
execution system to another that does not use our operation name
format (ie: very unlikely, but possible).

Co-authored-by: Zach Birenbaum <[email protected]>

Add Operation State Manager API (TraceMachina#937)

Introduces API to be implemented by new scheduler state backends.
WorkerStateManager is responsible for managing action updates sent to
and from workers. ClientStateManager is responsible any state related
to the clients client(s) requesting the action execution and can add
or filter operations based on a given criteria.
MatchingEnginerStateManager can filter, update, or remove actions.

Co-authored-by: Zach Birenbaum <[email protected]>

Unbreak docker-compose workflow (TraceMachina#940)

Fixes TraceMachina#939

Update Rust crate redis to v0.25.4 (TraceMachina#944)

Update Rust crate syn to v2.0.66 (TraceMachina#946)

Bump trivially bumpable deps (TraceMachina#950)

Notably, this changes the S3 schema for multipart upload starts/stops.

Add kind-loadbalancer (TraceMachina#929)

Provide an envoy loadbalancer container to access the local kind cluster.
The loadbalancer adds the ability to reach the local kind cluster trough localhost,
so that it is possible to work with a mac on the nativelink cluster.

Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster
can now be tested on that architecture.

Users can now query the loadbalancer instead of the gateway.

The loadbalancer has the following endpoints:

localhost:8080                  -> hubble-ui
localhost:8081                  -> tekton-dashboard
localhost:8082/eventlistener    -> nativelink-eventlistener

Fix pulumi ratelimiting build error (TraceMachina#953)

On MacOS the pulumi testsuite hit the ratelimiting on github.
Disable the pulumi tests with a patch on nixpkgs.

Commit 0eed759 didn't fix this issue.

Add drake toolchain configs (TraceMachina#942)

Register metrics on PropertyModifierScheduler (TraceMachina#954)

Fixes metrics being lost due to the underlying ActionScheduler not
having it's metrics passed through the PropertyModifierScheduler.
This should fix any scaling issues related to queued actions metrics
not being available.

Use single quotes for char (TraceMachina#955)

Nightly clippy reports a style issue when using double quotes around a
single char.

Increase pre-commit timeout in CI (TraceMachina#956)

The original setting of 5 minutes was a bit too flaky. Increase it to 10
minutes so that cache misses don't break the job.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants