Scheduler: Building Block & Control Plane Service #7716
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yaron2
added
the
autoupdate
* Chore: update artifact actions The `master` tag was pointing to a very old build that sometimes failed (the "master" branch is now "main"). More appropriately pointing to a version now. Also, changed to avoid compressing artifacts that are already compressed. In the process, also updated other actions including setup-go, which comes with its own caching for Go deps so we don't need the custom cache tasks Signed-off-by: ItalyPaleAle <[email protected]> * Fix Windows artifact publishing Signed-off-by: ItalyPaleAle <[email protected]> * More small tweaks Signed-off-by: ItalyPaleAle <[email protected]> --------- Signed-off-by: ItalyPaleAle <[email protected]> Co-authored-by: Artur Souza <[email protected]> Co-authored-by: Loong Dai <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
…apr#7318) * Cherry-pick dapr#7029 Signed-off-by: ItalyPaleAle <[email protected]> * WIP: Add config options to Helm chart and configure injector Co-authored-by: ItalyPaleAle <[email protected]> Co-authored-by: Deepanshu Agarwal <[email protected]> Signed-off-by: ItalyPaleAle <[email protected]> Signed-off-by: Deepanshu Agarwal <[email protected]> * Pass Actors/Reminders options to daprd Signed-off-by: Deepanshu Agarwal <[email protected]> * Fixed passing actor service Signed-off-by: ItalyPaleAle <[email protected]> * Also pass remindersService configuration Signed-off-by: ItalyPaleAle <[email protected]> * go.mod change Signed-off-by: Deepanshu Agarwal <[email protected]> * Revert Signed-off-by: Deepanshu Agarwal <[email protected]> * Revert Signed-off-by: Deepanshu Agarwal <[email protected]> * placement address corection in tpl Signed-off-by: Deepanshu Agarwal <[email protected]> * Auto-loading of placement and reminders providers Signed-off-by: ItalyPaleAle <[email protected]> * Small fixes Signed-off-by: ItalyPaleAle <[email protected]> * Fixed bug in injector Signed-off-by: ItalyPaleAle <[email protected]> * Added log Signed-off-by: ItalyPaleAle <[email protected]> * Add headers Signed-off-by: Deepanshu Agarwal <[email protected]> --------- Signed-off-by: ItalyPaleAle <[email protected]> Signed-off-by: Deepanshu Agarwal <[email protected]> Co-authored-by: ItalyPaleAle <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Co-authored-by: Mukundan Sundararajan <[email protected]>
Signed-off-by: Cassandra Coyle <[email protected]>
…uilding block (dapr#7390) * E2E tests for the crypto building block Signed-off-by: ItalyPaleAle <[email protected]> * Fixed build with subtlecrypto Signed-off-by: ItalyPaleAle <[email protected]> --------- Signed-off-by: ItalyPaleAle <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Artur Souza <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Artur Souza <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Tune actor app health check to become healthy sooner Signed-off-by: joshvanl <[email protected]> * Reset failure count to 0 on a healthy actor app health check Signed-off-by: joshvanl <[email protected]> * Move wait group to just before go routine Signed-off-by: joshvanl <[email protected]> * Change failure threshold `int32` -> `int` Signed-off-by: joshvanl <[email protected]> * Fix actors health checker Signed-off-by: joshvanl <[email protected]> * Returns `errors.Join` for actors `Close` procedure Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Updating protos Signed-off-by: Shivam Kumar <[email protected]> * enable recursive terminate/purge Signed-off-by: Shivam Kumar <[email protected]> * Updating dtf-go Signed-off-by: Shivam Kumar <[email protected]> * updating contrib Signed-off-by: Shivam Kumar <[email protected]> * Correct proto generated files Signed-off-by: Shivam Kumar <[email protected]> * update comment Signed-off-by: Shivam Kumar <[email protected]> * implementing getOrchestrationRuntimeState Signed-off-by: Shivam Kumar <[email protected]> * Adding unit tests Signed-off-by: Shivam Kumar <[email protected]> * make modtidy-all Signed-off-by: Shivam Kumar <[email protected]> * Adding recursive option in query parameter Signed-off-by: Shivam Kumar <[email protected]> * Adding integration test for workflow Signed-off-by: Shivam Kumar <[email protected]> * fix bug in Creating sub-orchestrations Signed-off-by: Shivam Kumar <[email protected]> * Setting recursive/terminate purge to default and adding tests for same Signed-off-by: Shivam Kumar <[email protected]> * linter fixes Signed-off-by: Shivam Kumar <[email protected]> * Removing fix from this PR Signed-off-by: Shivam Kumar <[email protected]> * renaming to Signed-off-by: Shivam Kumar <[email protected]> * linter fixes Signed-off-by: Shivam Kumar <[email protected]> --------- Signed-off-by: Shivam Kumar <[email protected]> Co-authored-by: Mukundan Sundararajan <[email protected]> Co-authored-by: Loong Dai <[email protected]> Co-authored-by: Dapr Bot <[email protected]>
* Updated pinned components-contrib Signed-off-by: ItalyPaleAle <[email protected]> * Fixed integration tests Signed-off-by: ItalyPaleAle <[email protected]> * downgrade json-patch due to breaking change Signed-off-by: ItalyPaleAle <[email protected]> * Updated json-patch to 5.8.1 Signed-off-by: ItalyPaleAle <[email protected]> --------- Signed-off-by: ItalyPaleAle <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Removes vnodes from placement table message Signed-off-by: Elena Kolevska <[email protected]> * Adds vnodes calculation on the sidecar Signed-off-by: Elena Kolevska <[email protected]> * Removes log line Signed-off-by: Elena Kolevska <[email protected]> * Backwards compatibility for placement tables without vnodes Signed-off-by: Elena Kolevska <[email protected]> * Update Signed-off-by: Elena Kolevska <[email protected]> * Fixes Signed-off-by: Elena Kolevska <[email protected]> * Account for daprd at 1.13 and placement at < 1.13 Signed-off-by: Elena Kolevska <[email protected]> * Missed var Signed-off-by: Elena Kolevska <[email protected]> * Adds unit tests Signed-off-by: Elena Kolevska <[email protected]> * Cleaning up Signed-off-by: Elena Kolevska <[email protected]> * Fixes Signed-off-by: Elena Kolevska <[email protected]> * Small optimisation Signed-off-by: Elena Kolevska <[email protected]> * Adds caching for vnodes Signed-off-by: Elena Kolevska <[email protected]> * Update pkg/placement/hashing/consistent_hash.go Co-authored-by: Alessandro (Ale) Segala <[email protected]> Signed-off-by: Elena Kolevska <[email protected]> * Fixes after review Signed-off-by: Elena Kolevska <[email protected]> * Adds unit test for the Virtual nodes cache Signed-off-by: Elena Kolevska <[email protected]> * lint fix Signed-off-by: Elena Kolevska <[email protected]> * Update tests Signed-off-by: Elena Kolevska <[email protected]> * Fixes linter errors Signed-off-by: Elena Kolevska <[email protected]> * Adds tests for actor service properly handling the old and new version of placement tables Signed-off-by: Elena Kolevska <[email protected]> * small improvement Signed-off-by: Elena Kolevska <[email protected]> --------- Signed-off-by: Elena Kolevska <[email protected]> Signed-off-by: Elena Kolevska <[email protected]> Co-authored-by: Alessandro (Ale) Segala <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: yaron2 <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Mark actors healthy on no app channel or actor entities Mark actors as always healthy when there is no app channel or their have been no actor entities (actor types) provided by the application config endpoint. This is important for enabling actor related APIs such as workflows, which don't rely on the application implementing actors itself. Signed-off-by: joshvanl <[email protected]> * Adds integration tests for actor healthz endpoints Most importantly, ensuring that actors are marked as always healthy when the both the application healthz endpoint is disabled, and the application reports no actor entities. Signed-off-by: joshvanl <[email protected]> * Remove uneeded app health check store Signed-off-by: joshvanl <[email protected]> * Update tests/integration/suite/actors/healthz/endpoint/noapp.go Co-authored-by: Alessandro (Ale) Segala <[email protected]> Signed-off-by: Artur Souza <[email protected]> * Update tests/integration/suite/actors/healthz/endpoint/noappentities.go Co-authored-by: Alessandro (Ale) Segala <[email protected]> Signed-off-by: Artur Souza <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Signed-off-by: Artur Souza <[email protected]> Co-authored-by: Mukundan Sundararajan <[email protected]> Co-authored-by: Artur Souza <[email protected]> Co-authored-by: Alessandro (Ale) Segala <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* append app token to outgoing calls, if it is provided Signed-off-by: Ryan Kelly <[email protected]> * use declare and assign syntax Signed-off-by: Ryan Kelly <[email protected]> --------- Signed-off-by: Ryan Kelly <[email protected]> Co-authored-by: Ryan Kelly <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* WIP Signed-off-by: ItalyPaleAle <[email protected]> * Bunch of fixes in tests Signed-off-by: ItalyPaleAle <[email protected]> * More test fixes Signed-off-by: ItalyPaleAle <[email protected]> * More test fixes Signed-off-by: ItalyPaleAle <[email protected]> * More tweaks Signed-off-by: ItalyPaleAle <[email protected]> * More test fixes Signed-off-by: ItalyPaleAle <[email protected]> * Fixes and improvements to code legibility Signed-off-by: ItalyPaleAle <[email protected]> * Use a lock around internalActors property Signed-off-by: ItalyPaleAle <[email protected]> * 💄 Signed-off-by: ItalyPaleAle <[email protected]> * Fixed last test Signed-off-by: ItalyPaleAle <[email protected]> * Changed per review feedback: use a single struct for reminders and timers Signed-off-by: ItalyPaleAle <[email protected]> * WIP: use factory for internal actors Signed-off-by: ItalyPaleAle <[email protected]> * Improve APIs for internal actors Signed-off-by: ItalyPaleAle <[email protected]> * Updated comment Signed-off-by: ItalyPaleAle <[email protected]> --------- Signed-off-by: ItalyPaleAle <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Co-authored-by: Mukundan Sundararajan <[email protected]> Co-authored-by: Loong Dai <[email protected]>
Account for Windows specific host connection errors in the mtls standalone disable test. This makes parity with the kubernetes test. Signed-off-by: joshvanl <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
…vel` (dapr#7400) Updates the placement API Level gRPC context key from `ApiLevel` to `dapr-placement-api-level`. Although collisions are highly unlikely, it could be considered more "correct" and best practice to namespace our application specific metadata keys. This isn't a breaking change as this key hasn't been part of a release yet. Signed-off-by: joshvanl <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
…apr#7041) * Injector: Change daprd projected token audience to sentry SPIFFE ID Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> * Change SidecarConfig to use string type for SentrySPIFFEID Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Cascade terminate/Purge Workflow Support (dapr#7340) * Updating protos Signed-off-by: Shivam Kumar <[email protected]> * enable recursive terminate/purge Signed-off-by: Shivam Kumar <[email protected]> * Updating dtf-go Signed-off-by: Shivam Kumar <[email protected]> * updating contrib Signed-off-by: Shivam Kumar <[email protected]> * Correct proto generated files Signed-off-by: Shivam Kumar <[email protected]> * update comment Signed-off-by: Shivam Kumar <[email protected]> * implementing getOrchestrationRuntimeState Signed-off-by: Shivam Kumar <[email protected]> * Adding unit tests Signed-off-by: Shivam Kumar <[email protected]> * make modtidy-all Signed-off-by: Shivam Kumar <[email protected]> * Adding recursive option in query parameter Signed-off-by: Shivam Kumar <[email protected]> * Adding integration test for workflow Signed-off-by: Shivam Kumar <[email protected]> * fix bug in Creating sub-orchestrations Signed-off-by: Shivam Kumar <[email protected]> * Setting recursive/terminate purge to default and adding tests for same Signed-off-by: Shivam Kumar <[email protected]> * linter fixes Signed-off-by: Shivam Kumar <[email protected]> * Removing fix from this PR Signed-off-by: Shivam Kumar <[email protected]> * renaming to Signed-off-by: Shivam Kumar <[email protected]> * linter fixes Signed-off-by: Shivam Kumar <[email protected]> --------- Signed-off-by: Shivam Kumar <[email protected]> Co-authored-by: Mukundan Sundararajan <[email protected]> Co-authored-by: Loong Dai <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Viktor Subota <[email protected]> * Add priorityClassName parameter to Helm chart Signed-off-by: Viktor Subota <[email protected]> * Add priorityClassName to README doc Signed-off-by: Viktor Subota <[email protected]> --------- Signed-off-by: Shivam Kumar <[email protected]> Signed-off-by: Viktor Subota <[email protected]> Co-authored-by: Shivam Kumar <[email protected]> Co-authored-by: Mukundan Sundararajan <[email protected]> Co-authored-by: Loong Dai <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Co-authored-by: Viktor Subota <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Fix vnode response to placement clients. Fixes vnode response to placement clients to be backwards compatible. Makes the API level check based on cluster level, rather than on a per request basis. Remove placement API level gRPC context metadata. Signed-off-by: joshvanl <[email protected]> * Fix method used in int sa grpc test Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
…e caller (dapr#7406) * Prevent invalid traceparent ("00-00000...") from being returned to the caller Signed-off-by: Karsten Wintermann <[email protected]> * OpenTelemetry trace context not transmitted when using rawPayload with Kafka Fixes dapr#7372 Signed-off-by: Karsten Wintermann <[email protected]> --------- Signed-off-by: Karsten Wintermann <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Co-authored-by: Artur Souza <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* fix skew tests Signed-off-by: yaron2 <[email protected]> * Add patch file for e2e error condition. Signed-off-by: Artur Souza <[email protected]> * Movie skew E2E patch to the right folder. Signed-off-by: Artur Souza <[email protected]> --------- Signed-off-by: yaron2 <[email protected]> Signed-off-by: Artur Souza <[email protected]> Co-authored-by: Artur Souza <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Shivam Kumar <[email protected]> Co-authored-by: Artur Souza <[email protected]> Co-authored-by: Mukundan Sundararajan <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Remove the unnecessary `time.Sleep`s slowing the workflow int test by 20s. Resolves the data race by using an `atomic.Bool` for shared state. Signed-off-by: joshvanl <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Updates controller-runtime, remove WriteSVIDToDir PR updates go modules, specifically controller-runtime to v0.16.3 so that the webhook server can be configured to use TLS options. This removes the need for the operator to write the SVID to disk and maintain it in memory. PR removes the WriteSVIDToDir option from security. Operator cache has been updated to use new controller-runtime types. Signed-off-by: joshvanl <[email protected]> * Update deprecated funcs Signed-off-by: joshvanl <[email protected]> * Replace deprecated function in test/runner Signed-off-by: joshvanl <[email protected]> * Fix Kubernetes integration process Signed-off-by: joshvanl <[email protected]> * Include int framework operator v2alpha1 API endpoint with subscription v2 Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Co-authored-by: Dapr Bot <[email protected]>
* Temporary override to have Redis to handle event metadata. Signed-off-by: Artur Souza <[email protected]> * E2E test change to reproduce HTTP header conflict in pubsub. Signed-off-by: Artur Souza <[email protected]> * E2E test for content-length conflict with gRPC subscriber. Signed-off-by: Artur Souza <[email protected]> * Hotfix to avoid content-length header conflict in HTTP and gRPC. Signed-off-by: Artur Souza <[email protected]> * Revert "Temporary override to have Redis to handle event metadata." This reverts commit 20b60a0. Signed-off-by: Artur Souza <[email protected]> --------- Signed-off-by: Artur Souza <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: joshvanl <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Shivam Kumar <[email protected]> Co-authored-by: Mukundan Sundararajan <[email protected]> Co-authored-by: Artur Souza <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Switch to OTel ParentSampler for correct sampling behavior Signed-off-by: Andrej Kyselica <[email protected]> * fix formatting Signed-off-by: Andrej Kyselica <[email protected]> * fix lint error Signed-off-by: Andrej Kyselica <[email protected]> --------- Signed-off-by: Andrej Kyselica <[email protected]> Signed-off-by: Andrej Kyselica <[email protected]> Co-authored-by: Yaron Schneider <[email protected]> Co-authored-by: Loong Dai <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Samantha Coyle <[email protected]> Co-authored-by: Loong Dai <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: kindknow <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Subscriptions: Declarative Go Generic Loader Following Components and HTTPEndpoints, creates a generic disk & kubernetes loader for Subscriptions. Removes the adhok loaders from existing runtime pubsub packages. This is required for adding Subscriptions to the hot reloader reconciler and making a more consistent & testable manifest loader package. The disk loader loads both `v1alpha1` and `v2alpha1` Subscriptions, converting `v1alpha1` to `v2alpha1` on successful load. Since the operator returns only `v2alpha1` Subscriptions (Kubernetes API converts `v1alpha1` to `v2alpha1`), the Kubernetes loader only loads `v2alpha1`. `APIVersion() string` func has been added to the generic `meta.Resource` interface type to allow for the generic loader to determine between resource versions, supporting differentiating `v1alpha1` and `v2alpha1` Subscription resource versions. To ensure backwards compatibility of the previous disk Subscription loader, the generic disk loader now tracks the order in which manifests are loaded. This ensures that, even though `v1alpha1` and `v2alpha1` Subscriptions use separate loaders, their file position order is preserved once `v1alpha1` Subscriptions are converted to `v2alpha1`. Subscription backwards compatibility of parsing & ingestion priority is covered by existing extensive integration tests. Notice that _ZERO_ of the comprehensive subscription integration tests have been modified, proving no behaviour change has occurred for loading and actuating Subscriptions. Explanation of integration test changes: - daprd/hotreload/selfhosted/crypto: loader now mandates `apiVersion` on disk manifests which is more correct. Previous behaviour ignoring this field should be considered a bug. Since we use Kubernetes resource API schema, `apiVersion` is (and has been) _always_ required. - framework/process/grpc/operator/server.go: returning non-nil gRPC object for mocked operator Subscription RPC call prevents underlying gRPC library unmarshal nil errors- more correct than previous implementation. Part of [Subscription hot-reloading](dapr#7139). Signed-off-by: joshvanl <[email protected]> * Update errS to patherrs to avoid confusion Signed-off-by: joshvanl <[email protected]> * Updates sub import to use bare version string Signed-off-by: joshvanl <[email protected]> * Remove unneeded code comment Signed-off-by: joshvanl <[email protected]> * Change test manifests to have unique names Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Fixes warning “Running http and grpc server on single port. This is not recommended for production.” Signed-off-by: Elena Kolevska <[email protected]> * Suffixes data dirs with instance id Signed-off-by: Elena Kolevska <[email protected]> * Adds space quota parameter Signed-off-by: Elena Kolevska <[email protected]> * Sets default quota to 2GB Signed-off-by: Elena Kolevska <[email protected]> * Adds compaction parameters Signed-off-by: Elena Kolevska <[email protected]> * Updates helm charts Signed-off-by: Elena Kolevska <[email protected]> * Adds namespace to data dir name. Renames etcdID to just ID. Signed-off-by: Elena Kolevska <[email protected]> --------- Signed-off-by: Elena Kolevska <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Cassandra Coyle <[email protected]>
* Hot Reload: Subscription Adds support for Subscription Hot Reloading. If the Hot Reloading preview feature is enabled, create, update and delete events to declarative Subscriptions will be reflected in the Subscriptions runtime without needing to restart the Dapr runtime. Works in both self-hosted and Kubernetes modes. Note that, because topic routes and not indexed by Subscription resources, all Subscriptions will be cancelled and topic routes re-built and re-subscribed when a Subscription event occurs. A possible improvement would to only cancel all topic route subscriptions which are related to the Subscription pubsub or include an index of the Subscription which a topic is routed for, however this is out of scope of this change. Support for per topic route cancellation will be needed for on-demand bi-directional subscriptions. In-flight messages are not effected by Subscription hot reloading. Branched from dapr#7582 Closes dapr#7139 Signed-off-by: joshvanl <[email protected]> * Load all Subscriptions before reloading on boot Signed-off-by: joshvanl <[email protected]> * Adds unit tests for subscription client side scope filtering Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Co-authored-by: Dapr Bot <[email protected]>
* Operator: Component server side scope filtering Today, daprd's will receive Component from the operator, regardless of whether they are scoped for that Component or not. This means that clients do receive components (including its associated secrets) that they are not scoped for. Updates Operator API ComponentUpdate to perform service side Component Scope filtering based on the authenticated client App ID. When a Component is de-scoped, daprd will receive a DELETE for the previous Component manifest. When a Component is scoped-in, daprd will receive an CREATE for the new Component manifest. Updates Operator API ListComponents to perform service side Component Scope filtering based on the authenticated client App ID. Uses updated events batcher which ensures queue items are sent in order. Signed-off-by: joshvanl <[email protected]> * Add 2 daprds for operator informer tests Signed-off-by: joshvanl <[email protected]> * Fix control plane trust domain Signed-off-by: joshvanl <[email protected]> * Update github.com/dapr/kit to master Signed-off-by: joshvanl <[email protected]> * go mod tidy Signed-off-by: joshvanl <[email protected]> * Fix incorrect pointer manifest compare Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Co-authored-by: Yaron Schneider <[email protected]>
* able to send triggered job back to app via the app channel from daprd sidecar using both grpc and http protocols Signed-off-by: Cassandra Coyle <[email protected]> * change sidecar receiving job to debug level to still validate the scheduler stream Signed-off-by: Cassandra Coyle <[email protected]> * grpc test Signed-off-by: Cassandra Coyle <[email protected]> * wip Signed-off-by: Cassandra Coyle <[email protected]> * some cleanup Signed-off-by: Cassandra Coyle <[email protected]> * update test framework grpc app to add the OnJobEventFn and update test to use it. grpc appcallback test passes Signed-off-by: Cassandra Coyle <[email protected]> * wip http test Signed-off-by: Cassandra Coyle <[email protected]> * added http working test. need to make lint Signed-off-by: Cassandra Coyle <[email protected]> * update tests with stub for interface func for triggerJob to app now since its in the app channel interface Signed-off-by: Cassandra Coyle <[email protected]> * defer release of ch Signed-off-by: Cassandra Coyle <[email protected]> --------- Signed-off-by: Cassandra Coyle <[email protected]>
* go-etcd-cron Signed-off-by: joshvanl <[email protected]> * Fix multi-scheduler int test Signed-off-by: joshvanl <[email protected]> * Review comments Signed-off-by: joshvanl <[email protected]> * Rename schedule app job type to job Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]>
* restore test file diff, keep chart chagnes Signed-off-by: Cassandra Coyle <[email protected]> * fix read-only err Signed-off-by: Cassandra Coyle <[email protected]> --------- Signed-off-by: Cassandra Coyle <[email protected]>
* Disk Loader: Scope filtering Adds scope filtering to the disk manifest loader. Though this doesn't have any security benefits, it does mean manifests are filtered much sooner in the runtime when running in self-hosted mode and prevents logging/errors downstream in modules like the hot reloading reconciler. Signed-off-by: joshvanl <[email protected]> * Update internal disk loader unit test with Options param Signed-off-by: joshvanl <[email protected]> * Pass appID to hotreload disk loader for scope filtering Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* first commit Signed-off-by: yaron2 <[email protected]> * add tests Signed-off-by: yaron2 <[email protected]> * linter Signed-off-by: yaron2 <[email protected]> --------- Signed-off-by: yaron2 <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Pin latest contrib with Go 1.22 Signed-off-by: Bernd Verst <[email protected]> * Fix mysterious linter issue Signed-off-by: Bernd Verst <[email protected]> * Downgrade cosmosdb Signed-off-by: Bernd Verst <[email protected]> * Update go.mod Signed-off-by: Bernd Verst <[email protected]> * Fix CosmosDB for latest API version Signed-off-by: Bernd Verst <[email protected]> * Install gotestsum in Makefile Signed-off-by: Bernd Verst <[email protected]> * Pin Go 1.22.2 Signed-off-by: Bernd Verst <[email protected]> * Fix makefile Signed-off-by: Bernd Verst <[email protected]> * revert makefile changes Signed-off-by: Bernd Verst <[email protected]> * Update all go.mod/Makefile/Dockerfile for `1.22`. Remove `{}` chars from HTTP route fuzz generations. Signed-off-by: joshvanl <[email protected]> * Makefile: export `go/bin` to PATH for darwin Exports the `go/bin/darwin_$(arch)` directory to the PATH environment variable when running on Darwin. This ensures that binaries installed by `go install` are available in the Makefile exec path. Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: Bernd Verst <[email protected]> Signed-off-by: joshvanl <[email protected]> Co-authored-by: joshvanl <[email protected]>
* Refactor SPIFFE from `pkg/security` to `kit` Updates the `pkg/security` package to move the SPIFFE implementation to a new kit package. This new kit package is more modulated and fuller test coverage. This package has been moved so that it can be both imported by dapr & components-contrib, as well as making the package more suitable for further development to support X.509 Component auth. dapr/proposals#51 Also moves in test/utils from dapr to crypto/test for shared usage. Part of dapr/proposals#51 Uses go mod fork of dapr/kit#92 Signed-off-by: joshvanl <[email protected]> * Include SVID context with `Init`ing Component Signed-off-by: joshvanl <[email protected]> * Adds security to processor options Signed-off-by: joshvanl <[email protected]> * Update github.com/dapr/dapr to master Signed-off-by: joshvanl <[email protected]> * Update `util` to new `test` package import Signed-off-by: joshvanl <[email protected]> * Update go.sum Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]>
* Bidirectional job trigger & ack. Adds job ack from scheduler client for when job is finished processing and can be ticked. Adds mTLS authorization to scheduler API server. Adds integration tests for scheduler Jobs and Actor Reminders. Signed-off-by: joshvanl <[email protected]> * Review comments & reconnect scheduler int test Signed-off-by: joshvanl <[email protected]> * Update go-etcd-cron Signed-off-by: joshvanl <[email protected]> * Linting Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]>
…ler statefulset (#27) * Charts: Adds option to use PVC for Scheduler statefulset Adds optional `dapr_scheduler.cluster.persistentVolumeClaimName` helm chart values option to change the scheduler data dir volume to use the references PersistentVolumeClaim, rather than an empty dir, making ETCD data persistent across pod restarts. Also changes the volume and mount paths so that all schedulers share the same root mount path, but write to a sub directory of the form "/<namespace>/<scheduler-id>". Signed-off-by: joshvanl <[email protected]> * Update scheduler volume to use volumeClaimTemplate Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Adds `listen-address` CLI flags for all services servers. Address corresponding `--listen-address` CLI flags for all services servers. This sets the listening address for these TCP servers. This is added to allow users to both bind the address to localhost to prevent access to a particular server on the network, as well as for integration testing as it is more correct to bind to localhost and prevents triggering MacOS firewall popups. Signed-off-by: joshvanl <[email protected]> * Report localhost to nameresolver when internal grpc listen address is localhost Signed-off-by: joshvanl <[email protected]> * Use localhost host for placement report when internal gRPC address is localhost Signed-off-by: joshvanl <[email protected]> * Change default listen-address from `0.0.0.0` to empty string to allow for default ipv6 support and keep backwards compat. Signed-off-by: joshvanl <[email protected]> --------- Signed-off-by: joshvanl <[email protected]> Co-authored-by: Dapr Bot <[email protected]>
Signed-off-by: Cassandra Coyle <[email protected]>
* merge & fix http status code check Signed-off-by: Cassandra Coyle <[email protected]> * triggered job e2e test for http app Signed-off-by: Cassandra Coyle <[email protected]> * update test iteration nums Signed-off-by: Cassandra Coyle <[email protected]> * rm time.sleep -> assert.eventually Signed-off-by: Cassandra Coyle <[email protected]> * rm local test changes Signed-off-by: Cassandra Coyle <[email protected]> * update test name Signed-off-by: Cassandra Coyle <[email protected]> * tweaks Signed-off-by: Cassandra Coyle <[email protected]> * grpc e2d works, need to cleanup grpc test Signed-off-by: Cassandra Coyle <[email protected]> * rm grpc test and combine into http test. keep both apps tho. need to cleanup local test changes in scheduler_test Signed-off-by: Cassandra Coyle <[email protected]> * cleanup local test changes Signed-off-by: Cassandra Coyle <[email protected]> * make lint Signed-off-by: Cassandra Coyle <[email protected]> * mv things around Signed-off-by: Cassandra Coyle <[email protected]> * cleanup Signed-off-by: Cassandra Coyle <[email protected]> * thread -> goroutine Signed-off-by: Cassandra Coyle <[email protected]> * Update clients.go Signed-off-by: Cassie Coyle <[email protected]> * rm commented line Signed-off-by: Cassandra Coyle <[email protected]> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <[email protected]> Signed-off-by: Cassie Coyle <[email protected]> * PR review updates Signed-off-by: Cassandra Coyle <[email protected]> * review updates. add code todo Signed-off-by: Cassandra Coyle <[email protected]> --------- Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Cassie Coyle <[email protected]> Co-authored-by: Josh van Leeuwen <[email protected]>
* continuously retry scheduler clients if it fails upon the first try Signed-off-by: Cassandra Coyle <[email protected]> * Apply suggestions from code review Co-authored-by: Josh van Leeuwen <[email protected]> Signed-off-by: Cassie Coyle <[email protected]> * fix indentation after UI committing Signed-off-by: Cassandra Coyle <[email protected]> --------- Signed-off-by: Cassandra Coyle <[email protected]> Signed-off-by: Cassie Coyle <[email protected]> Co-authored-by: Josh van Leeuwen <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* Update README.md Signed-off-by: Martin Oehlert <[email protected]> * Update README.md Signed-off-by: Martin Oehlert <[email protected]> --------- Signed-off-by: Martin Oehlert <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
* api token fix --------- Signed-off-by: Elena Kolevska <[email protected]> Signed-off-by: Artur Souza <[email protected]> Signed-off-by: joshvanl <[email protected]> Co-authored-by: Artur Souza <[email protected]> Co-authored-by: Elena Kolevska <[email protected]> Co-authored-by: Dapr Bot <[email protected]> Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Cassandra Coyle <[email protected]>
…service list, make lint Signed-off-by: Cassandra Coyle <[email protected]>
cicoyle
force-pushed
the
feat-dist-scheduler
branch
from
2f7edec
to
8bb51b7
Compare
|
rebasing to fix the DCO in CI caused all the conflicts... working to resolve them rn |
Signed-off-by: Cassandra Coyle <[email protected]>
|
Something wonky happened with my rebasing to fix the DCO check. Going to nuke this PR and open a fresh one from my backup branch (thankfully, I made a backup of the branch before going down the worm hole trying to fix DCO) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the relevant code for:
This PR follows this official proposal, however we did implement bidirectional streaming between the Scheduler and Daprd Sidecar (the proposal will be updated to reflect this change).
This is an end to end solution where an app is able to schedule a job to the daprd sidecar and perform basic crud operations with the job. The daprd sidecar sends the job to the Scheduler control plane service, which then stores the job in its embedded etcd. At trigger time, the scheduler will send the job back to the daprd sidecar where the sidecar will send the triggered job along to the app.
This includes work to replace the actor reminder subsystem, by scheduling the actor reminders in the Scheduler control plane service embedded etcd.
Perf numbers will be provided soon.
There were several individuals involved in making this work happen - thank you to all 🚀
Issue reference
1.14 release P0
Here is the issue tracking the work. There are a few remaining items that will be completed before the release date.
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list: