Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](udf) support java-udf static load #34980

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

[improve](udf) support java-udf static load #34980

wants to merge 6 commits into from
+477 −22

Conversation

zhangstar333
Copy link
Contributor

@zhangstar333 zhangstar333 commented May 16, 2024 •
edited

Proposed changes

CREATE FUNCTION print_12() RETURNS int 
PROPERTIES (
    "symbol" = "org.apache.doris.udf.AddOne", 
    "type" = "JAVA_UDF",
    "file" = "file:///mnt/ava-udf-demo-jar-with-dependencies.jar",
    "static_load" = "true", // default value is false
    "expiration_time" = "60" // default value is 360 minutes
);

if set static_load=true, the udf ClassLoader will be cache, and will be clear when expiration timeout
or use drop function stmt.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zhangstar333
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.69% (9002/25221)
Line Coverage: 27.36% (74458/272170)
Region Coverage: 26.60% (38493/144729)
Branch Coverage: 23.41% (19632/83846)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ef0a0545198cbcd25fd5581fceb9a990678cba9a_ef0a0545198cbcd25fd5581fceb9a990678cba9a/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.70% (9004/25221)
Line Coverage: 27.37% (74483/272171)
Region Coverage: 26.60% (38502/144727)
Branch Coverage: 23.43% (19643/83844)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ef0a0545198cbcd25fd5581fceb9a990678cba9a_ef0a0545198cbcd25fd5581fceb9a990678cba9a/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.69% (9005/25229)
Line Coverage: 27.37% (74521/272262)
Region Coverage: 26.61% (38526/144779)
Branch Coverage: 23.44% (19658/83870)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1919da51b9399096821b135a3e4d39ee0b4aeeab_1919da51b9399096821b135a3e4d39ee0b4aeeab/report/index.html

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.69% (9018/25265)
Line Coverage: 27.34% (74559/272689)
Region Coverage: 26.55% (38534/145134)
Branch Coverage: 23.41% (19663/84008)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3d7d7822da3bd6753347a0990787024689af7d5b_3d7d7822da3bd6753347a0990787024689af7d5b/report/index.html

@zhangstar333
Copy link
Contributor Author

run buildall

@zhangstar333
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40799 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 796f9c0287bbbad795cd4b0b6896135cd9518749, data reload: false

------ Round 1 ----------------------------------
q1	17603	4398	4337	4337
q2	2025	203	201	201
q3	10425	1263	1111	1111
q4	10190	862	889	862
q5	7481	2721	2636	2636
q6	237	136	138	136
q7	969	641	603	603
q8	9218	2141	2118	2118
q9	9981	6645	6713	6645
q10	9085	3944	3899	3899
q11	460	250	238	238
q12	435	223	232	223
q13	17879	3223	3180	3180
q14	282	220	230	220
q15	512	470	480	470
q16	504	389	392	389
q17	955	678	627	627
q18	8457	7773	7730	7730
q19	7460	1333	1264	1264
q20	649	324	319	319
q21	5015	3244	3988	3244
q22	402	347	367	347
Total cold run time: 120224 ms
Total hot run time: 40799 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4602	4425	4460	4425
q2	379	262	264	262
q3	3159	2939	2890	2890
q4	1982	1781	1622	1622
q5	5284	5447	5483	5447
q6	225	130	129	129
q7	2192	1799	1815	1799
q8	3263	3369	3342	3342
q9	8689	8596	8592	8592
q10	4073	3864	3862	3862
q11	600	518	495	495
q12	793	565	629	565
q13	15999	2992	3112	2992
q14	302	277	273	273
q15	534	483	474	474
q16	488	412	410	410
q17	1792	1486	1479	1479
q18	8032	7876	7340	7340
q19	1755	1603	1587	1587
q20	3032	1777	1783	1777
q21	4742	4711	4668	4668
q22	825	527	539	527
Total cold run time: 72742 ms
Total hot run time: 54957 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.28% (9238/25462)
Line Coverage: 27.65% (75830/274296)
Region Coverage: 26.87% (39277/146186)
Branch Coverage: 23.60% (19909/84364)
Coverage Report: http://coverage.selectdb-in.cc/coverage/796f9c0287bbbad795cd4b0b6896135cd9518749_796f9c0287bbbad795cd4b0b6896135cd9518749/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 170598 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 796f9c0287bbbad795cd4b0b6896135cd9518749, data reload: false

query1	900	377	369	369
query2	6446	2366	2517	2366
query3	6651	212	206	206
query4	19038	17496	17291	17291
query5	4130	460	448	448
query6	241	166	154	154
query7	4589	291	285	285
query8	327	278	303	278
query9	8356	2431	2412	2412
query10	438	294	305	294
query11	10624	10192	10087	10087
query12	145	88	95	88
query13	1639	369	366	366
query14	9190	7629	7497	7497
query15	241	193	185	185
query16	7813	258	256	256
query17	1641	518	515	515
query18	1951	295	264	264
query19	213	158	151	151
query20	96	87	88	87
query21	206	134	126	126
query22	4693	4283	4341	4283
query23	33690	33325	32955	32955
query24	10938	2762	2849	2762
query25	622	347	369	347
query26	1245	150	150	150
query27	2598	320	320	320
query28	7234	2094	2075	2075
query29	887	610	584	584
query30	282	152	146	146
query31	952	740	735	735
query32	89	54	55	54
query33	764	295	296	295
query34	957	459	473	459
query35	742	620	605	605
query36	1084	908	911	908
query37	153	69	66	66
query38	2829	2739	2703	2703
query39	866	794	786	786
query40	213	125	121	121
query41	54	50	52	50
query42	120	93	102	93
query43	591	557	546	546
query44	1219	733	748	733
query45	203	170	165	165
query46	1078	743	697	697
query47	1906	1771	1805	1771
query48	365	289	296	289
query49	988	412	402	402
query50	779	385	374	374
query51	6956	6785	6774	6774
query52	99	91	87	87
query53	391	292	287	287
query54	876	460	429	429
query55	72	70	72	70
query56	272	251	260	251
query57	1145	1071	1024	1024
query58	253	268	262	262
query59	3611	3400	3227	3227
query60	296	297	260	260
query61	87	85	87	85
query62	637	436	461	436
query63	324	286	293	286
query64	8884	2214	1718	1718
query65	3190	3085	3087	3085
query66	804	320	329	320
query67	15254	15011	15146	15011
query68	4544	536	516	516
query69	449	295	295	295
query70	1134	1138	1119	1119
query71	434	280	280	280
query72	7084	2715	2578	2578
query73	739	317	315	315
query74	6009	5514	5522	5514
query75	3418	2695	2689	2689
query76	2792	938	955	938
query77	480	289	290	289
query78	10219	9754	9740	9740
query79	2576	518	498	498
query80	1435	456	452	452
query81	557	217	218	217
query82	1306	104	100	100
query83	232	166	168	166
query84	246	86	87	86
query85	1463	278	261	261
query86	492	324	318	318
query87	3309	3067	3079	3067
query88	3835	2378	2355	2355
query89	484	380	387	380
query90	1831	189	193	189
query91	133	107	106	106
query92	64	49	49	49
query93	2315	505	492	492
query94	1234	190	190	190
query95	478	308	308	308
query96	584	267	266	266
query97	3189	3009	2993	2993
query98	235	224	208	208
query99	1162	832	813	813
Total cold run time: 270629 ms
Total hot run time: 170598 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 796f9c0287bbbad795cd4b0b6896135cd9518749, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.09	0.10
query5	0.51	0.49	0.50
query6	1.12	0.71	0.71
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.48	0.48
query10	0.53	0.55	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.60
query14	0.78	0.79	0.78
query15	0.82	0.81	0.81
query16	0.36	0.37	0.34
query17	1.03	1.04	0.99
query18	0.21	0.25	0.24
query19	1.84	1.78	1.67
query20	0.02	0.01	0.01
query21	15.52	0.68	0.66
query22	4.41	6.83	2.28
query23	18.32	1.34	1.20
query24	1.75	0.27	0.23
query25	0.14	0.08	0.08
query26	0.26	0.18	0.17
query27	0.08	0.07	0.07
query28	13.32	1.00	0.98
query29	13.06	3.33	3.33
query30	0.24	0.05	0.06
query31	2.88	0.38	0.38
query32	3.29	0.46	0.47
query33	2.90	2.84	2.90
query34	17.13	4.34	4.43
query35	4.49	4.51	4.52
query36	0.69	0.51	0.48
query37	0.18	0.15	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.06
query42	0.07	0.06	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.97 s
Total hot run time: 30.78 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zhangstar333
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40684 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2f9d1601837e6d60fd097d67e6aa366233fb0dc, data reload: false

------ Round 1 ----------------------------------
q1	17690	4946	4400	4400
q2	2034	204	200	200
q3	10709	1275	1169	1169
q4	10300	814	765	765
q5	7516	2784	2660	2660
q6	225	138	138	138
q7	971	644	623	623
q8	9311	2126	2132	2126
q9	9479	6730	6670	6670
q10	9219	3844	3810	3810
q11	451	254	260	254
q12	459	230	225	225
q13	18904	2977	3010	2977
q14	262	215	232	215
q15	521	483	494	483
q16	500	382	384	382
q17	995	714	711	711
q18	8362	7565	7594	7565
q19	4805	1448	1526	1448
q20	652	323	315	315
q21	5018	3220	3911	3220
q22	384	328	334	328
Total cold run time: 118767 ms
Total hot run time: 40684 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4423	4290	4252	4252
q2	377	263	270	263
q3	3018	2733	2728	2728
q4	1903	1629	1594	1594
q5	5308	5301	5309	5301
q6	220	128	130	128
q7	2179	1703	1703	1703
q8	3211	3376	3317	3317
q9	8474	8445	8398	8398
q10	3986	3771	3657	3657
q11	594	493	498	493
q12	790	611	594	594
q13	16374	2974	3010	2974
q14	303	274	257	257
q15	524	500	498	498
q16	463	414	429	414
q17	1867	1521	1494	1494
q18	7980	7650	7585	7585
q19	1716	1631	1593	1593
q20	2017	1790	1799	1790
q21	5027	4689	4796	4689
q22	623	540	562	540
Total cold run time: 71377 ms
Total hot run time: 54262 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171731 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2f9d1601837e6d60fd097d67e6aa366233fb0dc, data reload: false

query1	921	379	365	365
query2	6449	2429	2301	2301
query3	6649	209	220	209
query4	20017	17442	17267	17267
query5	4201	453	443	443
query6	250	162	162	162
query7	4600	300	293	293
query8	306	265	280	265
query9	8758	2475	2446	2446
query10	459	313	285	285
query11	10560	10045	9999	9999
query12	129	90	85	85
query13	1627	362	360	360
query14	9115	7576	6831	6831
query15	239	182	181	181
query16	7873	269	271	269
query17	1837	535	505	505
query18	1978	266	264	264
query19	204	156	151	151
query20	90	89	88	88
query21	212	140	123	123
query22	4611	4093	4009	4009
query23	33693	32829	32890	32829
query24	11074	2845	2822	2822
query25	671	347	361	347
query26	1594	153	156	153
query27	2975	320	326	320
query28	7549	2078	2109	2078
query29	986	612	597	597
query30	272	151	149	149
query31	964	720	738	720
query32	83	51	53	51
query33	769	282	299	282
query34	943	474	467	467
query35	771	612	590	590
query36	1100	922	947	922
query37	160	67	71	67
query38	2849	2768	2734	2734
query39	846	793	780	780
query40	282	126	125	125
query41	52	50	50	50
query42	118	94	97	94
query43	569	528	544	528
query44	1261	725	739	725
query45	204	164	158	158
query46	1075	765	717	717
query47	1862	1761	1778	1761
query48	361	312	303	303
query49	1151	410	404	404
query50	786	395	395	395
query51	6895	6797	6746	6746
query52	110	94	91	91
query53	357	287	296	287
query54	939	487	439	439
query55	73	74	75	74
query56	278	251	279	251
query57	1190	1028	1029	1028
query58	250	255	232	232
query59	3290	3110	3142	3110
query60	296	266	271	266
query61	89	101	87	87
query62	647	439	448	439
query63	323	285	292	285
query64	9881	2314	1712	1712
query65	3190	3077	3119	3077
query66	1372	321	340	321
query67	15621	15146	14799	14799
query68	4571	539	529	529
query69	456	309	307	307
query70	1105	1066	1066	1066
query71	391	281	279	279
query72	7204	5460	5531	5460
query73	745	331	327	327
query74	5941	5499	5463	5463
query75	3377	2649	2659	2649
query76	2573	860	924	860
query77	458	296	293	293
query78	10318	9805	9601	9601
query79	2177	508	516	508
query80	1039	459	458	458
query81	578	217	216	216
query82	748	103	99	99
query83	233	175	185	175
query84	242	86	117	86
query85	1989	267	260	260
query86	526	332	307	307
query87	3266	3113	3052	3052
query88	3862	2427	2442	2427
query89	482	391	385	385
query90	1811	189	187	187
query91	124	97	99	97
query92	66	49	53	49
query93	2377	514	510	510
query94	1291	201	190	190
query95	409	325	318	318
query96	616	271	270	270
query97	3200	2953	3039	2953
query98	227	233	223	223
query99	1064	814	826	814
Total cold run time: 274372 ms
Total hot run time: 171731 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2f9d1601837e6d60fd097d67e6aa366233fb0dc, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.09
query5	0.51	0.49	0.49
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.50
query10	0.53	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.80	0.78	0.79
query15	0.83	0.81	0.80
query16	0.37	0.35	0.36
query17	0.96	0.97	0.99
query18	0.22	0.24	0.24
query19	1.79	1.68	1.77
query20	0.02	0.01	0.00
query21	15.71	0.67	0.65
query22	3.98	8.14	1.75
query23	18.27	1.41	1.29
query24	1.65	0.28	0.24
query25	0.14	0.08	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.27	1.00	0.99
query29	13.24	3.31	3.29
query30	0.25	0.08	0.07
query31	2.81	0.39	0.38
query32	3.27	0.46	0.47
query33	2.84	2.90	2.86
query34	17.33	4.50	4.43
query35	4.55	4.53	4.60
query36	0.64	0.46	0.46
query37	0.18	0.17	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.17	0.15	0.16
query41	0.08	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.71 s
Total hot run time: 30.48 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.39% (9282/24825)
Line Coverage: 28.73% (76013/264613)
Region Coverage: 28.11% (39376/140103)
Branch Coverage: 24.52% (19922/81232)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a2f9d1601837e6d60fd097d67e6aa366233fb0dc_a2f9d1601837e6d60fd097d67e6aa366233fb0dc/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
meta-change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@zhangstar333 @doris-robot