Releases: apache/incubator-uniffle
Releases · apache/incubator-uniffle
Release v0.8.0
Apache Uniffle (Incubating) Release v0.8.0
Highlight
- Support TEZ
- Introduce Netty for shuffle data transmission
- Use off heap memory to store shuffle data.
- Introduce REST API for cluster management.
- Introduce command line for cluster management.
ChangeLog
- Change license owner to ASF by @kaijchen in #5
- Trivial code improvements by @wForget in #7
- [Minor] Store shuffleId int to be consistent with other data structure by @zuston in #10
- Introduce the asList method in ConfigOptions by @zuston in #9
- Rename package by @jerqi in #6
- Minimize apache-rat excluded files by @kaijchen in #11
- Update module names by @kaijchen in #12
- Covert PartitionAssignmentInfo to static inner class by @pan3793 in #15
- [Followup] Migrate to Junit5 by @zuston in #14
- [Bug] Fix NPE problem when process the event if application was cleared already by @colinmjj in #16
- [CI] Enable codecov report by @kaijchen in #17
- Correct the config description and fix typo by @zuston in #19
- Add CI and Codecov badges in README by @kaijchen in #20
- [Followup] Use asList method in some existing configOptions by @zuston in #18
- Move rss-integration-spark-common-test module package by @wForget in #23
- [INFRA] Improve asf.yaml to reduce the notifications by @jerryshao in #25
- [TEST] Improve code coverage in rss-common by @kaijchen in #26
- Remove redundant package by @wForget in #27
- [CI] Switch to temurin JDK by @kaijchen in #24
- [INFRA] Improve asf.yaml to reduce the notifications (another-try) by @jerryshao in #33
- Bump commons-lang3 from 3.5 to 3.10 by @wForget in #28
- Fix the log of incorrectly bound class by @wForget in #35
- [TYPO] Fix misspelled word "integration" by @kaijchen in #34
- Fix some hyperlink in README.md by @daugraph in #32
- Upgrade gRPC to support Apple Silicon by @pan3793 in #13
- Allow to specify custom tags to decide the assignment of servers by @zuston in #30
- Optimize the bash script by @zuston in #29
- [Improvement] reduce compiler warnings by @advancedxy in #46
- [Chore]: document update and build time optimize by @advancedxy in #45
- Supplement doc about assignment tags by @zuston in #47
- [Bug] Fix skip() api maybe skip unexpected bytes which makes inconsistent data by @colinmjj in #40
- [improvement] Remove experimental feature with ShuffleUploader by @colinmjj in #51
- [Improvement] Provides utility classes for creating thread factories by @smallzhongfeng in #49
- Enable spotbugs and fix high priority bugs by @kaijchen in #38
- [CI] Change default checkstyle severity to error by @kaijchen in #57
- [Style] Check indentation by @kaijchen in #56
- [Experimental Feature] MR Supports Remote Spill by @frankliee in #55
- [Improvement] Log indicate the shuffle server host:port when doing re… by @zuston in #58
- Send commit concurrently in client side by @zuston in #59
- Explicitly set the constructor with AccessManager when extending AccessChecker by @zuston in #43
- [DOC] Replace Firestorm with Uniffle by @jerqi in #60
- Introduce the extraProperties to support user-defined pluggable accessCheckers by @zuston in #42
- Log enhancement: Merge multiple logs into oneline and add more description by @zuston in #62
- [TEST] Add more unit tests in rss-common by @kaijchen in #63
- [MINOR] Comments of PartitionBalanceAssignmentStrategy miss byte units by @smallzhongfeng in #68
- [Minor] Make config keys and default values finalized by @kaijchen in #70
- [Log Improvment] Add more detailed debug info for MR client by @frankliee in #84
- [Improvement] Shutdown the grpc executors pool when closing by @zuston in #83
- Log enhancement: return error message when getting assignment servers and log exception when initializing by @zuston in #64
- [ISSUE-48] [Feature] Init Kubernetes operator directory by @jerqi in #75
- [Improvement] No need to use synchronized lock of the method scope when getting client by @zuston in #82
- [DOC] Remove Wechat group in README by @jerqi in #88
- [Performance Optimization] Improve the speed of writing index file in shuffle server by @zuston in #91
- [DOC] Update title and description in README by @kaijchen in #94
- [Improvement] ShuffleBlock should be release when finished reading by @xianjingfeng in #74
- [IMPROVEMENT][COMMON] Fix common module code style by @jerqi in #99
- [Improvement]LocalStorage init use multi thread #71 by @xianjingfeng in #72
- [Improvement] Use OR operation instead of serialization for cloning BitMaps by @kaijchen in #103
- [Improvement] Ignore partial failure on initializing local storage in shuffle server side by @zuston in #102
- [CI] Test compile in Java 11 and Java 17 by @kaijchen in #105
- Sleep less time but try more times when stopping by @xianjingfeng in #112
- [Improvement] Use ConfigBuilder to rewrite the class RssSparkConfig by @smallzhongfeng in #104
- [Improvement] Introduce config to customize assignment server numbers in client side by @zuston in #100
- Assign partition again if registerShuffleServers failed by @xianjingfeng in #115
- [ISSUE-106][IMPROVEMENT] Set rpc timeout for all rpc interface by @xianjingfeng in #113
- [MINOR][IMPROVEMENT] Avoid CoordinatorServer#initialization multiple
new Configuration()
by @zwangsheng in #118 - [Improve] Remove useless server id from
StorageManagerFactory#createStorageManager
by @zwangsheng in #119 - [MINOR][IMPROVEMENT][COORD] Fix coordinator module code style by @jerqi in #122
- [Improvement] Set heartBeatExecutorService as daemon thread by @smallzhongfeng in #121
- [JUnit] Introduce the property of trimStackTrace to show error stacktrace in mvn-test by @zuston in #126
- Make the conf of rss.storage.basePath as list by @zuston in #130
- [MINOR][IMPROVEMENT][STORAGE] Fix storage module code style by @jerqi in #131
- [Improvement] Add timeout reconnection when DelegationRssShuffleManager send the request of AccessCluster by @smallzhongfeng in #139
- [MINOR] Fix flaky test testGetHostIp by @izchen in #141
- [Improvement] Add the number of unhealthy nodes in CoordinatorMetrics by @smallzhongfeng in #147
- [ISSUE-48][FEATURE] Add Uniffle Dockerfile by @wangao1236 in #132
- [BUGFIX] Fix memory leak which cause oom by @summaryzb in #145
- [Log Improvement] Output the register...
Release v0.7.1
Apache Uniffle (Incubating) Release v0.7.1
Highlight
-
Improvements
- Refresh application when reading memory data to prevent application from being expired.
-
Major bug fixes
- Cache proxy user ugi to avoid memory leak.
- Make metric reporter usable by fixing some logic errors.
- Close ShuffleWriteClient after task was completed to exit container process.
- Make sure 'finishShuffle' invoked after all shuffle data sent.
- Update local storage metadata for all related events instead of just the first event.
- Correct a wrong metric, grpc_server_connection_number.
-
Minor bug fixes
- Avoid returning null in defaultUserApps when quote file doesn't config user.
- Fix LocalStorageManager divide by zero exception.
ChangeLog
- [ISSUE-669] improvement: refresh application when reading memory data (#741)
- Bump project version to 0.7.1-SNAPSHOT
- [ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
- Revert "[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
- [ISSUE-796][0.7] bug: Fix the issues of MetricReporter (#821)
- [MINOR][0.7] Avoid returning null in defaultUserApps when quota file doesn't config user (#822)
- [ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
- Revert "[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
- [ISSUE-772][0.7] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
- [ISSUE-715] fix(mr): The container does not exit because shuffleclient is not closed (#882)
- [ISSUE-886] fix(mr): MR Client may lose data or throw exception when rss.storage.type without MEMORY (#887)
- Revert "[ISSUE-886] fix(mr): MR Client may lose data or throw exception when rss.storage.type without MEMORY (#887)
- [MINOR] fix: Fix LocalStorageManager divide by zero exception (#900)
- [ISSUE-881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0 (#914)
- [ISSUE-933][0.7] fix: incorrect metric grpc_server_connection_number (#941)
- change version to 0.7.1 release
Release v0.7.0
Apache Uniffle (Incubating) Release v0.7.0
Highlight
- Better support for Spark AQE
- Leveraging the LOCAL_ORDER data distribution to improve performance
- Estimating assignment number of shuffle servers to improve user experience
- Better assignment policy of assigning adjacent partitions to the same shuffle server to improve performance
- Optimization of huge partition to improve stability of shuffle servers
- More bug fixes and usability improvements of K8S operator
- Add support of user quota management and more compression algorithms
- Add support of spark data eviction mechanism of stage level
- Some improvement of stability and performance
Changelog
- Add more badges in README by @kaijchen in #219
- Fix incorrect log format strings by @kaijchen in #220
- Change total lines badge url to sloc.xyz in README by @kaijchen in #222
- [MINOR] Fix warnings reported by lgtm by @kaijchen in #223
- [MINOR] Simplify creating buffer logic by @zuston in #227
- Support cancelling previous ci actions by @zuston in #225
- Use the conf of shuffleNodesNumber from jobs to be as checking factor by @zuston in #208
- Output the stderr and stdout to output file in startup script by @zuston in #226
- [ISSUE-48][FEATURE][FOLLOW UP] Add controller component by @wangao1236 in #214
- Add more metrics about requiring read memory by @zuston in #231
- Adjust the memory required times to match grpc max deadline conf by @zuston in #218
- [MINOR] Fix flaky test by @jerqi in #238
- [ISSUE-48][FEATURE][FOLLOW UP] Add yaml of components and crd exampes by @wangao1236 in #236
- Fix Flaky test GetShuffleReportForMultiPartTest by @leixm in #241
- Set the default disk capacity to the total space by @zuston in #237
- Add issue template by @jerqi in #8
- [MINOR] Fix inefficient map iteration by @kaijchen in #245
- Support deploy multiple shuffle servers in a single node by @xianjingfeng in #166
- Fast fail when reading failed in ComposedClientReadHandler by @zuston in #213
- Fix startup shell problem by @jerqi in #251
- New version 0.7.0-snapshot by @jerqi in #252
- [ISSUE-196] Fix flaky test about kerberos by @zuston in #250
- [ISSUE-48][FEATURE][FOLLOW UP] add unit test for validating rss objects by @wangao1236 in #248
- [Improvement] Add hdfs path health check to AppBalanceSelectStorageStrategy by @smallzhongfeng in #210
- [TYPO] Replace Chinese colon by ASCII colon by @kaijchen in #255
- Introduce startup-silent-period mechanism to avoid partial assignments by @zuston in #247
- Replace DISCLAIMER with DISCLAIMER-WIP by @jerqi in #258
- [ISSUE-244] Fix flaky test of CoordinatorGrpcTest.rpcMetricsTest by @zuston in #256
- Fix flaky test of ClientConfManagerTest by @smallzhongfeng in #260
- [Refactor] Optimize creating shuffle handlers by @zuston in #259
- Introduce data cleanup mechanism on stage level by @zuston in #249
- [ISSUE-48][FEATURE][FOLLOW UP] add docs for operator by @wangao1236 in #261
- Fix potenial missing reads of exclude nodes by @zuston in #269
- [ISSUE-257] RssMRUtils#getBlockId change the partitionId of int type to long by @fpkgithub in #266
- [ISSUE-273][BUG] Get shuffle result failed caused by concurrent calls to registerShuffle by @leixm in #274
- Add enum type test about case insensitive by @zuston in #280
- Support ZSTD by @zuston in #254
- [ISSUE-239][BUG] RssUtils#transIndexDataToSegments should consider the length of the data file by @leixm in #275
- Remove code quality badge and add release badge by @kaijchen in #284
- [ISSUE-163][FEATURE] Write to hdfs when local disk can't be write by @xianjingfeng in #235
- Upgrade Github actions for Node.js 16 by @kaijchen in #292
- Fix NPE in WriteBufferManager.addRecord by @wForget in #296
- Fix AbstractStorage#containsWriteHandler by @xianjingfeng in #281
- Add more test cases on LocalStorageManager.selectStorage by @zuston in #298
- [ISSUE-137][Improvement][AQE] Sort MapId before the data are flushed by @zuston in #293
- [ISSUE-283][FEATURE] Support snappy compression/decompression by @amaliujia in #304
- [ISSUE-290] Make RpcNodePort and HttpNodePort optional by @amaliujia in #305
- [ISSUE-301][Subtask][Improvement][AQE] Merge continuous ShuffleDataSegment into single one by @zuston in #303
- Cleanup RuntimeException and fetchRemoteStorage logic in ClientUtils by @kaijchen in #295
- [ISSUE-135][FOLLOWUP][Improvement][AQE] Assign adjacent partitions to the same ShuffleServer by @leixm in #307
- Correct the contributing guide link in pull-request template by @zuston in #314
- Fix bug of "Comparison method violates its general contract" by @zuston in #315
- [AQE][LocalOrder] Fix potenial bug when merging continuous segments by @zuston in #318
- [AQE][LocalOrder] Fix wrong param of expectedTaskIds in LocalOrderSegmentSplit by @zuston in #319
- [Feature] Support the estimated number of ShuffleServers required. by @leixm in #322
- [Bug] Fix potenial bug when the index reading offset is greater than data length by @zuston in #320
- [ISSUE-154][Improvement] Support Empty assignment to Shuffle Server by @rhh777 in #325
- [Bug] Fix invalid owner of host path volumes by @wangao1236 in #330
- [ISSUE-309][FEATURE] Support ShuffleServer latency metrics. by @leixm in #327
- [ISSUE-329]Catch NPE in ShuffleTaskManager#addFinishedBlockIds by @xianjingfeng in #331
- [BUG] Fix wrong method name by @leixm in #335
- [ISSUE-328] Cleanup unused shuffle servers after stage completed by @xianjingfeng in #334
- [MINOR] Migrate RankValue to the package of the common class by @smallzhongfeng in #265
- [BUG] Fix incorrect spark metrics by @zuston in #324
- [Improvement][LocalOrder] Add tests about keeping consistent with FixedSize when no skew optimization by @zuston in #336
- [INFRA] Add k8s pipeline by @jerqi in #340
- Remove unused class of RssShuffleUtils by @zuston in #345
- [ISSUE-342][Improvement] Check Spark Serializer type by @chong0929 in #344
- [Feature] Support user's app quota level limit by @smallzhongfeng in #311
- [BUG][AQE][LocalOrder] Fix the bug of missed data due to block sorting by @zuston in #347
- [ISSUE-364] Fix
indexWriter
don't close if exception thrown when close dataWriter by @xianjingfeng in #349 - [BUG] Fix flaky test of AQESkewedJoinWithLocalOrderTest by @zuston in #350
- Add collaborators by @jerqi in #351
- [BUG][FOLLOWUP] Fix flaky test of AQESkewedJoinWithLocalOrderTest by @zuston in https://github.com/...
Release v0.6.1
Apache Uniffle (Incubating) Release v0.6.1
Highlight
-
Major bug fixes
- Partition cannot be accessed in MapReduce when the reduce task number exceeds 1024.
- Get shuffle result failure caused by concurrent calls to registerShuffle.
- Inconsistent blocks caused by missing length in RssUtils#transIndexDataToSegments.
- Handle NPE in WriteBufferManager#addRecord in the same way as Spark.
- AbstractStorage#containsWriteHandler is checking the wrong Map.
indexWriter
isn't closed if exception is thrown when closing dataWriter.- Incorrect dependency of protobuf-java at compile time.
- Potential memory leak when encountering disk unhealthy.
-
Minor bug fixes
- Potenial missing reads of exclude nodes.
- Incorrect contributing link in pull-request template.
- Incorrect spark metrics.
ChangeLog
- [ISSUE-257] RssMRUtils#getBlockId change the partitionId of int type to long (#266)
- [ISSUE-273][BUG] Get shuffle result failed caused by concurrent calls to registerShuffle (#274)
- Fix potenial missing reads of exclude nodes (#269)
- [ISSUE-239][BUG] RssUtils#transIndexDataToSegments should consider the length of the data file (#275)
- Fix NPE in WriteBufferManager.addRecord (#296)
- Fix AbstractStorage#containsWriteHandler (#281)
- Correct the pull-request contributing link in template (#314)
- [BUG] Fix incorrect spark metrics (#324)
- [ISSUE-364] Fix
indexWriter
don't close if exception thrown when close dataWriter (#349) - [ISSUE-228] Fix the problem of protobuf-java incorrect dependency at compile time (#362)
- Bump project version to 0.6.1
- [BUG] Potenial memory leak when encountering disk unhealthy (#370)
Release v0.6.0
Apache Uniffle (Incubating) Release v0.6.0
Highlight
-
Optimize the assignment strategy
-
Some improvement of stability and performance
-
Add a plugin mechanism of SelectStorageStrategy
-
Add LowestIOSampleCostSelectStorageStrategy
-
Support Kerberos HDFS
ChangeLog
- Change license owner to ASF by @kaijchen in #5
- Trivial code improvements by @wForget in #7
- [Minor] Store shuffleId int to be consistent with other data structure by @zuston in #10
- Introduce the asList method in ConfigOptions by @zuston in #9
- Rename package by @jerqi in #6
- Minimize apache-rat excluded files by @kaijchen in #11
- Update module names by @kaijchen in #12
- Covert PartitionAssignmentInfo to static inner class by @pan3793 in #15
- [Followup] Migrate to Junit5 by @zuston in #14
- [Bug] Fix NPE problem when process the event if application was cleared already by @colinmjj in #16
- [CI] Enable codecov report by @kaijchen in #17
- Correct the config description and fix typo by @zuston in #19
- Add CI and Codecov badges in README by @kaijchen in #20
- [Followup] Use asList method in some existing configOptions by @zuston in #18
- Move rss-integration-spark-common-test module package by @wForget in #23
- [INFRA] Improve asf.yaml to reduce the notifications by @jerryshao in #25
- [TEST] Improve code coverage in rss-common by @kaijchen in #26
- Remove redundant package by @wForget in #27
- [CI] Switch to temurin JDK by @kaijchen in #24
- [INFRA] Improve asf.yaml to reduce the notifications (another-try) by @jerryshao in #33
- Bump commons-lang3 from 3.5 to 3.10 by @wForget in #28
- Fix the log of incorrectly bound class by @wForget in #35
- [TYPO] Fix misspelled word "integration" by @kaijchen in #34
- Fix some hyperlink in README.md by @daugraph in #32
- Upgrade gRPC to support Apple Silicon by @pan3793 in #13
- Allow to specify custom tags to decide the assignment of servers by @zuston in #30
- Optimize the bash script by @zuston in #29
- [Improvement] reduce compiler warnings by @advancedxy in #46
- [Chore]: document update and build time optimize by @advancedxy in #45
- Supplement doc about assignment tags by @zuston in #47
- [Bug] Fix skip() api maybe skip unexpected bytes which makes inconsistent data by @colinmjj in #40
- [improvement] Remove experimental feature with ShuffleUploader by @colinmjj in #51
- [Improvement] Provides utility classes for creating thread factories by @smallzhongfeng in #49
- Enable spotbugs and fix high priority bugs by @kaijchen in #38
- [CI] Change default checkstyle severity to error by @kaijchen in #57
- [Style] Check indentation by @kaijchen in #56
- [Experimental Feature] MR Supports Remote Spill by @frankliee in #55
- [Improvement] Log indicate the shuffle server host:port when doing re… by @zuston in #58
- Send commit concurrently in client side by @zuston in #59
- Explicitly set the constructor with AccessManager when extending AccessChecker by @zuston in #43
- [DOC] Replace Firestorm with Uniffle by @jerqi in #60
- Introduce the extraProperties to support user-defined pluggable accessCheckers by @zuston in #42
- Log enhancement: Merge multiple logs into oneline and add more description by @zuston in #62
- [TEST] Add more unit tests in rss-common by @kaijchen in #63
- [MINOR] Comments of PartitionBalanceAssignmentStrategy miss byte units by @smallzhongfeng in #68
- [Minor] Make config keys and default values finalized by @kaijchen in #70
- [Log Improvment] Add more detailed debug info for MR client by @frankliee in #84
- [Improvement] Shutdown the grpc executors pool when closing by @zuston in #83
- Log enhancement: return error message when getting assignment servers and log exception when initializing by @zuston in #64
- [ISSUE-48] [Feature] Init Kubernetes operator directory by @jerqi in #75
- [Improvement] No need to use synchronized lock of the method scope when getting client by @zuston in #82
- [DOC] Remove Wechat group in README by @jerqi in #88
- [Performance Optimization] Improve the speed of writing index file in shuffle server by @zuston in #91
- [DOC] Update title and description in README by @kaijchen in #94
- [Improvement] ShuffleBlock should be release when finished reading by @xianjingfeng in #74
- [IMPROVEMENT][COMMON] Fix common module code style by @jerqi in #99
- [Improvement]LocalStorage init use multi thread #71 by @xianjingfeng in #72
- [Improvement] Use OR operation instead of serialization for cloning BitMaps by @kaijchen in #103
- [Improvement] Ignore partial failure on initializing local storage in shuffle server side by @zuston in #102
- [CI] Test compile in Java 11 and Java 17 by @kaijchen in #105
- Sleep less time but try more times when stopping by @xianjingfeng in #112
- [Improvement] Use ConfigBuilder to rewrite the class RssSparkConfig by @smallzhongfeng in #104
- [Improvement] Introduce config to customize assignment server numbers in client side by @zuston in #100
- Assign partition again if registerShuffleServers failed by @xianjingfeng in #115
- [ISSUE-106][IMPROVEMENT] Set rpc timeout for all rpc interface by @xianjingfeng in #113
- [MINOR][IMPROVEMENT] Avoid CoordinatorServer#initialization multiple
new Configuration()
by @zwangsheng in #118 - [Improve] Remove useless server id from
StorageManagerFactory#createStorageManager
by @zwangsheng in #119 - [MINOR][IMPROVEMENT][COORD] Fix coordinator module code style by @jerqi in #122
- [Improvement] Set heartBeatExecutorService as daemon thread by @smallzhongfeng in #121
- [JUnit] Introduce the property of trimStackTrace to show error stacktrace in mvn-test by @zuston in #126
- Make the conf of rss.storage.basePath as list by @zuston in #130
- [MINOR][IMPROVEMENT][STORAGE] Fix storage module code style by @jerqi in #131
- [Improvement] Add timeout reconnection when DelegationRssShuffleManager send the request of AccessCluster by @smallzhongfeng in #139
- [MINOR] Fix flaky test testGetHostIp by @izchen in #141
- [Improvement] Add the number of unhealthy nodes in CoordinatorMetrics by @smallzhongfeng in #147
- [ISSUE-48][FEATURE] Add Uniffle Dockerfile by @wangao1236 in #132
- [BUGFIX] Fix memory leak which cause oom by @summaryzb in #145
- [Log Improvement] Output the re...