Flyte v1.10.7 milestone release

[flyte-bot]

Flyte 1.10.7 Release Notes

We're excited to share the release of Flyte 1.10.7, featuring a broad spectrum of updates, improvements, and bug fixes across the Flyte ecosystem. This release marks a pivotal shift in our development approach, notably with our adoption of buf for protobuf stub generation. This move optimizes our development workflow and discontinues the automatic creation of Java and C++ stubs, making it easier to adapt the generated code for other languages as needed. Additionally, we've upgraded to gRPC-gateway v2, aligning with the latest advancements and recommendations found in the v2 migration guide.

Our sincere gratitude goes to all contributors for their invaluable efforts towards this release.

Core Improvements and Bug Fixes

• Improved error handling for transient secret sync issues, enhancing the robustness of secret management. [PR #4310]
• Introduced Sphinx build for Monodocs, improving documentation generation and integration. [PR #4347]
• Enhanced the Spark plugin by fixing the environment variable ValueFrom for pod templates, allowing for more dynamic configurations. [PR #4532]
• Optimized fastcache behavior to not cache lookups on node skip, reducing unnecessary cache hits. [PR #4524]
• Removed composition errors from branch nodes, streamlining execution paths. [PR #4528]
• Added support for ignoring warnings related to AWS SageMaker imports, improving integration compatibility. [PR #4540]
• Fixed a bug related to setting the service account from PodTemplate, ensuring correct service account usage. [PR #4536]
• Addressed flaky tests in test_monitor, enhancing test reliability. [PR #4537]
• Updated the boilerplate version and contribution guide, facilitating better community contributions. [PR #4541], [PR #4501]
• Improved documentation build processes by manually creating version files and introducing a conda-lock file for consistent environment setup. [PR #4556], [PR #4553]
• Enhanced array node evaluation frequency optimization by detecting subNode phase updates. [PR #4535]
• Introduced support for failure nodes, allowing workflows to handle failures more gracefully. [PR #4308]
• Made various updates to Go versions, plugin integrations, and GitHub workflows to enhance performance and developer experience. [PR #4534], [PR #4582], [PR #4589]
• Addressed several bugs and made improvements in caching, metadata handling, and task execution, further stabilizing the Flyte platform. [PR #4594], [PR #4590], [PR #4607]
• Streamlined development workflow with the transition to buf for generating protobuf stubs, ceasing the automatic generation of Java and C++ stubs.
• Upgraded to grpc-gateway v2, optimizing API performance and compatibility.

Plugin and Integration Enhancements

• Added new features and fixed bugs in the Spark plugin, Ray Autoscaler integration, and other areas, expanding Flyte's capabilities and integration ecosystem. [PR #4363]
• Updated various dependencies and configurations, ensuring compatibility and security. [PR #4571], [PR #4643]
• Improved the handling and documentation of plugin secrets management, making it easier for users to manage sensitive information. [PR #4732]

Documentation and Community

• Updated community meeting cadence and contribution guidelines, fostering a more engaged and welcoming community. [PR #4699]
• Enhanced documentation through various updates, including the introduction of a new architecture image for FlytePlugins and clarification of propeller scaling. [PR #4661], [PR #4741]

Full Changelog

• Fix transient secret sync error handling by @Tom-Newton in Fix transient secret sync error handling #4310
• Monodocs sphinx build by @cosmicBboy in Monodocs sphinx build #4347
• [Spark plugin] Fix environment variable ValueFrom for pod templates by @Tom-Newton in [Spark plugin] Fix environment variable ValueFrom for pod templates #4532
• fastcache should not cache lookup on node skip by @hamersaw in fastcache should not cache lookup on node skip #4524
• Removed composition error from branch node by @hamersaw in Removed composition error from branch node #4528
• ignore warnings related to awssagemaker import by @cosmicBboy in ignore warnings related to awssagemaker import #4540
• [BUG] Fix setting of service_account from PodTemplate by @pvditt in [BUG] Fix setting of service_account from PodTemplate #4536
• Fix flaky test_monitor by @pingsutw in Fix flaky test_monitor #4537
• Update boilerplate version by @flyte-bot in Update boilerplate version #4541
• remove hardcoded list of tests by @samhita-alla in remove hardcoded list of tests #4521
• manually create flytekit/_version.py file in docs build by @cosmicBboy in manually create flytekit/_version.py file in docs build #4556
• introduce conda-lock file for docs by @cosmicBboy in introduce conda-lock file for docs #4553
• Detect subNode phase updates to reduce evaluation frequency of ArrayNode by @hamersaw in Detect subNode phase updates to reduce evaluation frequency of ArrayNode #4535
• Add support failure node by @pingsutw in Add support failure node #4308
• Return InvalidArgument for workflow compilation failures in CreateWorkflow by @katrogan in Return InvalidArgument for workflow compilation failures in CreateWorkflow #4566
• Update to go 1.21 by @eapolinario in Update to go 1.21 #4534
• Update contribution guide by @pingsutw in Update contribution guide #4501
• Add flyin plugin to monodocs integrations page by @neverett in Add flyin plugin to monodocs integrations page #4582
• Use updated cronSchedule in CreateLaunchPlanModel by @pmahindrakar-oss in Use updated cronSchedule in CreateLaunchPlanModel #4564
• Writing zero length inputs by @hamersaw in Writing zero length inputs #4594
• Feature/add pod pending timeout config by @pvditt in Feature/add pod pending timeout config #4590
• Run single-binary gh workflows on all PRs by @eapolinario in Run single-binary gh workflows on all PRs #4589
• auto-generate toctree from flytesnacks index.md docs by @cosmicBboy in auto-generate toctree from flytesnacks index.md docs #4587
• add repo tag and commit associated with the build by @cosmicBboy in add repo tag and commit associated with the build #4571
• monodocs - gracefully handle case when external repo doesn't contain tags: use current commit by @cosmicBboy in monodocs - gracefully handle case when external repo doesn't contain tags: use current commit #4598
• convert commit to string by @cosmicBboy in convert commit to string #4599
• Bug/abort map task subtasks by @pvditt in Bug/abort map task subtasks #4506
• Supporting parallelized workers in ArrayNode subNodes by @hamersaw in Supporting parallized workers in ArrayNode subNodes #4567
• Don't use experimental readthedocs build.commands config by @cosmicBboy in Don't use experimental readthedocs build.commands config #4606
• Ignore cache variables by @hamersaw in Ignore cache variables #4618
• Feature/add cleanup non recoverable pod statuses by @pvditt in Feature/add cleanup non recoverable pod statuses #4607
• Agent Metadata Servicer by @Future-Outlier in Agent Metadata Servicer #4511
• Add Flyin propeller config by @eapolinario in Add Flyin propeller config #4610
• Correctly computing ArrayNode maximum attempts and system failures by @hamersaw in Correctly computing ArrayNode maximum attempts and system failures #4627
• Agent Sync Plugin by @Future-Outlier in Agent Sync Plugin #4107
• Add github token in buf gh action by @eapolinario in Add github token in buf gh action #4626
• Update flyte-binary values by @davidmirror-ops in Update flyte-binary values #4604
• Fixing cache overwrite metadata update by @hamersaw in Fixing cache overwrite metadata update #4617
• Fixing 100 kilobyte max error message size by @hamersaw in Fixing 100 kilobyte max error message size #4631
• Add Ray Autoscaler to the Flyte-Ray plugin by @Yicheng-Lu-llll in Add Ray Autoscaler to the Flyte-Ray plugin #4363
• Artifact protos and related changes by @wild-endeavor in Artifact protos and related changes #4474
• Remove protoc-gen-validate by @eapolinario in Remove protoc-gen-validate #4643
• Readme update 2023 by @davidmirror-ops in Readme update 2023 #4549
• Fixing ArrayNode integration with backoff controller by @hamersaw in Fixing ArrayNode integration with backoff controller #4640
• Avoid to use the http.DefaultClient by @andresgomezfrr in Avoid to use the http.DefaultClient #4667
• Update dns policy for sandbox buildkit instance to ClusterFirstWithHo… by @jeevb in Update dns policy for sandbox buildkit instance to ClusterFirstWithHo… #4678
• Updating ArrayNode ExternalResourceInfo ID by @hamersaw in Updating ArrayNode ExternalResourceInfo ID #4677
• Feat: Inject user identity as pod label in K8s plugin by @fg91 in Feat: Inject user identity as pod label in K8s plugin #4637
• Artifacts shell 2 by @wild-endeavor in Artifacts shell 2 #4649
• Improve Agent Metadata Service Error Message by @Future-Outlier in Improve Agent Metadata Service Error Message #4682
• move pod start/end time to a common template vars by @vraiyaninv in move pod start/end time to a common template vars #4676
• switch readthedocs config to monodocs build by @cosmicBboy in switch readthedocs config to monodocs build #4687
• Update Flyte components by @flyte-bot in Update Flyte components #4690
• Add GetTaskMetrics and GetTaskLogs to agent by @pingsutw in Add GetTaskMetrics and GetTaskLogs to agent #4662
• add algolia searchbar by @cosmicBboy in add algolia searchbar #4696
• monodocs: do not use beta releases when importing projects by @cosmicBboy in monodocs: do not use beta releases when importing projects #4712
• add cache evicted status by @pvditt in add cache evicted status #4705
• Update community meeting cadence by @davidmirror-ops in Update community meeting cadence #4699
• Remove dockerfiles from subfolder by @pingsutw in Remove dockerfiles from subfolder #4715
• Update to artifact idl - Add List Usage endpoint by @wild-endeavor in Update to artifact idl - Add List Usage endpoint #4714
• monodocs uses flytekit/flytectl index rst file by @cosmicBboy in monodocs uses flytekit/flytectl index rst file #4720
• Replace grpc gateway endpoints with post by @wild-endeavor in Replace grpc gateway endpoints with post #4717
• [BUG] Retry fetching subworkflow output data on failure by @pvditt in [BUG] Retry fetching subworkflow output data on failure #4602
• Option to clear node state on any termination by @Tom-Newton in Option to clear node state on any termination #4596
• Add org to identifier protos by @katrogan in Add org to identifier protos #4663
• Update docs for plugin secrets management by @wild-endeavor in Update docs for plugin secrets management #4732
• Reintroduce k8s client fallback to cache lookups by @hamersaw in Reintroduce k8s client fallback to cache lookups #4733
• Remove unused validate files by @eapolinario in Remove unused validate files #4644
• delete old docs by @cosmicBboy in delete old docs #4742
• Docs/Clarify propeller scaling by @wild-endeavor in Docs/Clarify propeller scaling #4741
• Update Flyte components by @flyte-bot in Update Flyte components #4744
• Add org to all flyteadmin endpoints for consistency by @katrogan in Add org to all flyteadmin endpoints for consistency #4746
• update conda lock file by @cosmicBboy in update conda lock file #4749
• Use logger with formatter by @andrewwdye in Use logger with formatter #4747
• [housekeeping] Remove pull_request_template from each subdirectory by @pingsutw in [housekeeping] Remove pull_request_template from each subdirectory #4753
• [Docs] Reapply Databricks agent docs changes from [Docs] Databricks Agent Doc #4008 by @neverett in [Docs] Reapply Databricks agent docs changes from #4008 #4751
• Fix test get logs template uri test by @eapolinario in Fix test get logs template uri test #4760
• Small formatting fixes for Databrick agents docs by @neverett in Small formatting fixes for Databrick agents docs #4758
• [housekeeping] Remove flytearchives by @pingsutw in [housekeeping] Remove flytearchives #4761
• Guard against open redirect URL parameters in login by @katrogan in Guard against open redirect URL parameters in login #4763
• Wrapping k8s client with write filter and cache reader by @hamersaw in Wrapping k8s client with write filter and cache reader #4752
• [BUG] Handle Potential Indefinite Propeller Update Loops by @pvditt in [BUG] Handle Potential Indefinite Propeller Update Loops #4755
• Update Flyte components by @flyte-bot in Update Flyte components #4768
• Deprecated Agent State to Agent Phase by @Future-Outlier in Deprecated Agent State to Agent Phase #4738
• Add docs build process readme by @ppiegaze in Add docs build process readme #4772
• GetDynamicNodeWorkflow endpoint by @iaroslav-ciupin in GetDynamicNodeWorkflow endpoint #4689
• [BUG] subworkflow timeout propagation by @pvditt in [BUG] subworkflow timeout propagation #4766
• Update artifact IDL with new time partition by @wild-endeavor in Update artifact IDL with new time partition #4737
• Proto changes by @wild-endeavor in Proto changes #4778
• Updates for onboarding docs revamp by @neverett in Updates for onboarding docs revamp #4548
• [Docs] Fix toctree links to User Guide, Environment Setup, and Contributing sections by @neverett in [Docs] Fix toctree links to User Guide, Environment Setup, and Contributing sections #4781
• docs: add FlytePlugins architecture image by @jasonlai1218 in docs: add FlytePlugins architecture image #4661
• Fix repeated items in left nav by @ppiegaze in Fix repeated items in left nav #4783
• [Docs] Remove broken link from Understand How Flyte Handles Data page (for new monodocs site) (second attempt) by @neverett in [Docs] Remove broken link from Understand How Flyte Handles Data page (for new monodocs site) (second attempt) #4757
• docs: update Flyte sandbox configuration and documentation by @jasonlai1218 in docs: update Flyte sandbox configuration and documentation #4729
• Replace Storage To Ephemeral Storage in Helm Chart by @Future-Outlier in Replace Storage To Ephemeral Storage in Helm Chart #4782
• Fix webhook typo, add podLabels, add podEnv to flyte-core Helm chart by @ddl-ebrown in Fix webhook typo, add podLabels, add podEnv to flyte-core Helm chart #4756
• Dynamic log links by @eapolinario in Dynamic log links #4774
• Remove storage as a task resource option by @Future-Outlier in Remove storage as a task resource option #4658
• feat: add apache 2.0 license to python flyteidl by @michaeltinsley in feat: add apache 2.0 license to python flyteidl #4786
• Reduce maptask transitions between WaitingForResources and CheckingSubtaskExecutions by @hamersaw in Reduce maptask transitions between WaitingForResources and CheckingSubtaskExecutions #4790
• propeller gc ttl comparison should allow 23 by @hamersaw in propeller gc ttl comparison should allow 23 #4791
• Bring Scheme back for backwards compatibility by @eapolinario in Bring Scheme back for backwards compatibility #4789
• Pass secret to invocation of go_generate gh workflow by @eapolinario in Pass secret to invocation of go_generate gh workflow #4630
• [BUG] handle potential uncaught OOMKilled terminations by @pvditt in [BUG] handle potential uncaught OOMKilled terminations #4793
• Update additional bindings for org in path to be consistent by @katrogan in Update additional bindings for org in path to be consistent #4795
• Update pyflyte serve into pyflyte serve agent by @chaohengstudent in Update pyflyte serve into pyflyte serve agent #4526
• Agent ClientSet by @Future-Outlier in Agent ClientSet #4718
• Support kuberay v1.0.0 by @Yicheng-Lu-llll in Support kuberay v1.0.0 #4656
• Update Flyte components by @flyte-bot in Update Flyte components #4803
• install latest flyteidl when building monodocs by @cosmicBboy in install latest flyteidl when building monodocs #4815
• Rewrite GetExecutionData path additional bindings for org by @katrogan in Rewrite GetExecutionData path additional bindings for org #4816
• update docs README environment setup by @cosmicBboy in update docs README environment setup #4819
• Flyte-core add missing nodeSelector values by @ddl-ebrown in Flyte-core add missing nodeSelector values #4808
• Flyte-core add missing imagePullSecrets support by @ddl-ebrown in Flyte-core add missing imagePullSecrets support #4810
• Logger disable HTML escaping by @andrewwdye in Logger disable HTML escaping #4828
• Move intro docs from flytesnacks to flyte by @ppiegaze in Move intro docs from flytesnacks to flyte #4814
• Flyte-agent configure pod securityContext by @ddl-ebrown in Flyte-agent configure pod securityContext #4785
• [Docs] add sandbox to local cluster resource path by @wild-endeavor in [Docs] add sandbox to local cluster resource path #4837
• Flyte-core add missing podEnv values by @ddl-ebrown in Flyte-core add missing podEnv values #4807
• Flyte-core Expose propeller webhook port 9443 in charts by @ddl-ebrown in Flyte-core Expose propeller webhook port 9443 in charts #4804
• Align dir structure and URL structure with left nav hierarchy by @ppiegaze in Align dir structure and URL structure with left nav hierarchy #4843
• Generate version with setuptools_scm and migrate to pyproject.toml by @pingsutw in Generate version with setuptools_scm and migrate to pyproject.toml #4799
• MNT Fixes packaging for flyteidl wheel by @thomasjpfan in MNT Fixes packaging for flyteidl wheel #4846
• Use buf to generate stubs by @eapolinario in Use buf to generate stubs #4806
• [FLYTE-486] Support selecting IDP based on the query parameter by @pmahindrakar-oss in [FLYTE-486] Support selecting IDP based on the query parameter #4838
• Update Flyte components by @flyte-bot in Update Flyte components #4847
• Add plugin_config for agent by @pingsutw in Add plugin_config for agent #4848
• Adds MANIFEST.in for flyteidl by @thomasjpfan in Adds MANIFEST.in for flyteidl #4850
• Verify unbounded inputs for all scheduled launch plan types by @katrogan in Verify unbounded inputs for all scheduled launch plan types #4867
• Fix npm publish of flyteidl package by @eapolinario in Fix npm publish of flyteidl package #4861
• Remove protoc_gen_swagger by @eapolinario in Remove protoc_gen_swagger #4860
• Create CODEOWNERS file and add docs team by @neverett in Create CODEOWNERS file and add docs team #4857
• [Docs] update outdated link to on-prem tutorial by @ALMerrill in [Docs] update outdated link to on-prem tutorial #4868
• Re-add link to hosted sandbox by @neverett in Re-add link to hosted sandbox #4856
• Fix asterisk in cron table being rendered as list item by @neverett in Fix asterisk in cron table being rendered as list item #4836
• Add notes to selfAuth with Azure docs by @davidmirror-ops in Add notes to selfAuth with Azure docs #4835
• Add protos to support cache overrides by @hamersaw in Add protos to support cache overrides #4820
• Flyte-core add support for ingressClassName in ingress by @ddl-ebrown in Flyte-core add support for ingressClassName in ingress #4805
• Update flyte docs build directions by @ppiegaze in Update flyte docs build directions #4862
• Replaced deprecated bitnami/bitnami-shell image with bitnami/os-shell by @kamaleybov in Replaced deprecated bitnami/bitnami-shell image with bitnami/os-shell #4882
• Flyte-core define pod and container securityContext by @ddl-ebrown in Flyte-core define pod and container securityContext #4809
• Leverage KubeRay v1 instead of v1alpha1 for resources by @peterghaddad in Leverage KubeRay v1 instead of v1alpha1 for resources #4818

New Contributors

• A warm welcome to our new contributors: @pvditt, @ppiegaze, @jasonlai1218, and @ddl-ebrown. Thank you for your contributions to the Flyte community!

---

This discussion was created from the release Flyte v1.10.7 milestone release.