Filter IN pushdown to different decoders #1525

YiqinXiong · 2023-08-22T12:36:13Z

Task Description

ref #1491

Solution Description

Add/Update white filter 'in' functions for RAW/DICT/RLE/CONST/INTEGER_BASE_DIFF decoders.

Basic process:

RAW:
1. batch decode
2. traverse all rows, apply filter and set result_bitmap
DICT:
1. apply filter in dict, set ref_bitset
2. traverse all refs, set result_bitmap by using ref_bitset
RLE:
1. apply filter in dict, set ref_bitset
2. traverse all refs, set result_bitmap by using ref_bitset
CONST:
1. if const value in filter params (i.e. right arguments of expr 'in'), flip over result_bitmap
2. apply filter in dict, set ref_bitset . It needs to be combined with whether the const value is in the params.
3. traverse all refs, set result_bitmap by using ref_bitset
INTEGER_BASE_DIFF:
1. batch decode
2. if max(params) < base_value , set result_bitmap all false
3. else traverse all rows, apply filter and set result_bitmap

Passed Regressions

Unittest

Passed related unittests:

unittest/storage/blocksstable/encoding/test_general_column_decoder
unittest/storage/blocksstable/encoding/test_raw_decoder
unittest/storage/blocksstable/encoding/test_const_decoder

Mysql test

Part 1. Load data

Number of data:
1,000,000 rows.

Data types:

Five tables were created for testing based on five different encoding types.

The data types included in each table are shown in the table below:

	bigint	int	char	varchar	decimal	ubigint	uint
RAW	✅	✅	✅	✅	✅	✅	✅
DICT	✅	✅	✅	✅	✅	✅	✅
RLE	✅	✅	✅	✅	✅	✅	✅
CONST	✅	✅	✅	✅	✅	✅	✅
INT_BASE_DIFF	✅	✅				✅	✅

Data organization format:

The data for each encoding is generated by a Python script through pseudorandom calculations based on different seeds. And we will do manual major freeze to make columns encoded.

In order to form the corresponding encoding, the data is arranged according to the following rules:

RAW: Completely random, values within each column are mostly different.

A	D	R	X	Y	Q	...	L	T	H

DICT: Values within the dictionary appear randomly. eg: the dictionary contains values {A, B, C}.

B	A	A	C	B	A	...	C	A	B

RLE: Values within the dictionary appear consecutively in order. eg: the dictionary contains values {A, B, C}.

A	A	A	A	B	B	...	B	C	C

CONST: 99.9% the default value C appearing, and a 0.1% probability of values {A, B} within the dictionary appearing.

C	C	C	A	C	C	...	C	B	C

INTEGER_BASE_DIFF: Random data appears within a certain range, and the minimum value within the data is taken as the base value B.

B+v1	B+v2	B	B+v3	B+v4	B+v5	...	B+vn-2	B+vn-1	B+vn

Part 2. Run

Pattern of SQL statements used for testing:

select count(col_name) from table_name where col_name in (params);

In the code above, col_name is the column name, table_name is the table name, and params is the parameter list for IN, containing params.size() parameters that need to be matched.

Variables in the test:

Scenarios:

T for values in column (RAW / INTEGER_BASE_DIFF)
F for values not in column
D for values in dict table (DICT / RLE / CONST)
C for const value (CONST)
n for number of IN params

RAW / INTEGER_BASE_DIFF:
- NO: IN (F * n)
- PART: IN (T * 0.5n, F * 0.5n)
DICT / RLE:
- NO: IN (F * n)
- PART: IN (D * k, F * (n-k)), where k=min(n/2, len(dict)/2)
CONST
- NO: IN (F * n)
- PART: IN (D * k, F * (n-k)), where k=min(n/2, len(dict)/2)
- PART_CONST_IN: IN (C, F * (n-1))

Number of IN parameters:
4 / 40 / 400

Part 3. Result

Meaning of values:
The values in the table below represent the execution time of SQL queries on the master branch divided by the execution time of the same queries on the issue branch (i.e. The speedup ratio of the issue branch compared to the master branch).

Performance test results:

Notice: The following results are obtained by fully utilizing the cache for hot queries and are averaged over 3 repetitions with discarding the first round result.

RAW:

Scenario	num of IN (*)	bigint	int	char(20)	varchar(40)	decimal(11,3)	ubigint	uint
NO	4	1.088	1.094	1.032	1.815	1.193	1.091	1.108
PART	4	1.080	1.076	1.016	1.424	1.192	1.117	1.117
NO	40	1.044	1.018	1.003	1.002	1.141	1.063	1.065
PART	40	1.032	1.009	1.004	1.002	1.132	1.085	1.085
NO	400	1.082	1.059	1.010	1.005	1.217	1.090	1.104
PART	400	1.057	1.069	1.001	1.019	1.198	1.102	1.104

DICT:

Scenario	num of IN (*)	bigint	int	char(20)	varchar(40)	decimal(11,3)	ubigint	uint
NO	4	47.600	53.400	119.545	167.273	50.545	45.100	49.200
PART	4	4.560	4.729	13.894	19.364	5.325	4.185	4.603
NO	40	51.600	57.500	129.600	165.917	56.091	49.400	54.000
PART	40	3.988	3.747	12.847	18.720	4.412	3.863	3.852
NO	400	48.100	54.500	111.909	117.313	53.091	49.700	53.100
PART	400	3.868	3.656	12.530	18.133	4.558	4.013	3.779

RLE:

Scenario	num of IN (*)	bigint	int	char(20)	varchar(40)	decimal(11,3)	ubigint	uint
NO	4	45.200	50.700	112.182	126.286	53.500	44.900	49.000
PART	4	10.894	9.814	32.909	45.240	12.000	10.646	9.839
NO	40	50.900	53.700	112.182	158.083	61.000	49.600	53.900
PART	40	7.695	6.116	24.667	32.989	8.239	7.512	6.299
NO	400	47.700	51.900	101.667	140.769	50.273	49.200	52.100
PART	400	7.226	6.035	24.053	32.523	8.261	7.506	6.490

CONST:

Scenario	num of IN (*)	bigint	int	char(20)	varchar(40)	decimal(11,3)	ubigint	uint
NO	4	44.400	49.300	149.100	245.100	51.500	44.800	48.600
PART	4	23.789	26.474	56.700	132.905	27.474	23.947	25.947
PART_CONST_IN	4	4.936	4.136	19.353	39.299	5.490	4.964	4.240
NO	40	53.900	49.000	102.273	234.833	53.800	44.500	48.600
PART	40	24.955	23.905	49.348	113.040	24.227	20.773	23.619
PART_CONST_IN	40	5.465	4.137	19.697	41.162	5.644	4.979	4.208
NO	400	44.400	49.400	139.000	203.750	52.500	52.700	57.600
PART	400	20.455	22.818	61.040	99.160	24.545	24.273	26.545
PART_CONST_IN	400	4.971	4.052	22.059	38.236	5.732	5.554	4.663

INTEGER_BASE_DIFF:

Scenario	num of IN (*)	bigint	int	ubigint	uint
NO	4	37.833	37.333	38.167	37.750
PART	4	1.160	1.195	1.139	1.177
NO	40	44.500	49.273	45.083	49.000
PART	40	1.088	1.074	1.091	1.104
NO	400	24.238	42.417	41.750	46.364
PART	400	1.130	1.112	1.130	1.138

Other Information

Result csv files:
result-issue.csv
result-master.csv
mysql_test.zip

DengzhiLiu · 2023-08-28T11:50:56Z

src/sql/engine/basic/ob_pushdown_filter.cpp

@@ -1015,6 +1077,8 @@ void ObWhiteFilterExecutor::check_null_params()
 int ObWhiteFilterExecutor::init_obj_set()
 {
 int ret = OB_SUCCESS;
+ obj_array_sorted_ = false;


这个成员不需要，保证数组有序就行

这个成员不需要，保证数组有序就行

收到, 已修改 commit 63e156e

DengzhiLiu · 2023-08-28T12:31:10Z

src/storage/blocksstable/encoding/ob_raw_decoder.cpp

@@ -843,14 +848,67 @@ int ObRawDecoder::in_operator(
 || NULL == row_index)) {
 ret = OB_INVALID_ARGUMENT;
 LOG_WARN("Pushdown in operator: Invalid arguments", K(ret), K(filter.get_objs()));
+ } else if (OB_LIKELY(can_vectorized()) && OB_LIKELY(is_inited())) {


这个判断不需要吧

这个判断不需要吧

收到, 已修改 commit a2f7326

DengzhiLiu · 2023-08-28T12:36:16Z

src/sql/engine/basic/ob_pushdown_filter.h

+ int64_t *row_ids_;
+ common::ObIAllocator *allocator_;
+ // for white filter IN batch_decode
+ common::ObDatum *white_batch_datums_;


这个可以从表达式中拿到，不需要自己alloc

收到, 已修改 commit d3522a7

DengzhiLiu · 2023-08-28T12:49:11Z

src/sql/engine/basic/ob_pushdown_filter.cpp

+int ObWhiteFilterExecutor::eval_right_val_to_objs()
+{
+ int ret = OB_SUCCESS;
+ const ObExpr &expr = *(filter_.expr_);


先判null防御

收到，已修改 commit 2284e8c

chaserzju · 2023-08-28T11:53:23Z

src/sql/engine/basic/ob_pushdown_filter.h

+
+struct ObWhiteFilterParamsCmpFunc
+{
+ OB_INLINE bool operator()(const common::ObObj &obj1, const common::ObObj &obj2) const {


obj compare函数可能返回ERROR

收到，已修改 commit ecd9691

chaserzju · 2023-08-28T11:54:15Z

src/sql/engine/basic/ob_pushdown_filter.cpp

+ // 2. make params sorted
+ if (OB_FAIL(ret)) {
+ } else {
+ std::sort(params_.begin(), params_.end(), ObWhiteFilterParamsCmpFunc());


一般cmp fun需要带入ret
sort之后如果报错需要带出错误码

收到，已修改 commit ecd9691

chaserzju · 2023-08-28T11:54:31Z

src/sql/engine/basic/ob_pushdown_filter.cpp

+ // obj > max_obj || obj < min_obj
+ is_exist = false;
+ } else {
+ is_exist = std::binary_search(params_.begin(), params_.end(), obj, ObWhiteFilterParamsCmpFunc());


同上, 尽量不用标准库

收到，未带出错误码的问题已修改 commit ecd9691

如果不使用标准库函数的话，想问一下这里OB有自己更好的实现吗，或者说自己手撸一个？用这个主要是因为看到在其他位置 std::sort 和 std::binary_search 也有被用到😂

chaserzju · 2023-08-28T13:01:25Z

src/sql/engine/basic/ob_pushdown_filter.cpp

+ } else if (OB_FAIL(cur_arg->eval(ctx, right))) {
+ LOG_WARN("failed to eval right datum", K(ret));
+ } else if (!null_param_contained_ && right->is_null()) {
+ null_param_contained_ = true;


in(xxx, null) 的时候, null是不是可以忽略掉? 这里不需要标注null param?

收到，已修改 commit 2284e8c

chaserzju · 2023-08-28T13:03:05Z

src/sql/engine/basic/ob_pushdown_filter.cpp

- check_null_params();
- if (WHITE_OP_IN == filter_.get_op_type() && OB_FAIL(init_obj_set())) {
- LOG_WARN("Failed to init Object hash set in filter node", K(ret));
+int ObWhiteFilterExecutor::eval_in_right_val_to_objs()


和 eval_right_val_to_objs代码重复比较多

收到，已修改 commit 2284e8c

…luated_datums`

DengzhiLiu · 2023-09-04T11:29:29Z

src/sql/engine/basic/ob_pushdown_filter.h

 ObPushdownWhiteFilterNode &filter_;
+ ObDatum* batch_decode_datums_;


不需要加这个成员，直接调 get_datums_from_column就行

收到，已修改 commit 94ac9b4

YiqinXiong force-pushed the issue_1491 branch 3 times, most recently from 8c26e69 to 939b84a Compare August 28, 2023 09:06

YiqinXiong marked this pull request as ready for review August 28, 2023 09:07

fixed oceanbase#1491: filter IN pushdown to different decoders

9a409c9

YiqinXiong force-pushed the issue_1491 branch from 939b84a to 9a409c9 Compare August 28, 2023 11:17

DengzhiLiu reviewed Aug 28, 2023

View reviewed changes

fixed oceanbase#1491: remove redundant obj_array_sorted_

63e156e

DengzhiLiu reviewed Aug 28, 2023

View reviewed changes

fixed oceanbase#1491: remove can_vectorized() always true

a2f7326

chaserzju reviewed Aug 28, 2023

View reviewed changes

YiqinXiong added 5 commits September 1, 2023 15:17

fixed oceanbase#1491: fix fatal bug defend filter_ null

be24058

fixed oceanbase#1491: simplify eval_right_val_to_objs and `init_eva…

2284e8c

…luated_datums`

fixed oceanbase#1491: fix typo

205f821

fixed oceanbase#1491: get datum buf from expr directly

d3522a7

fixed oceanbase#1491: return OB err code in std:: algorithm func

ecd9691

DengzhiLiu reviewed Sep 4, 2023

View reviewed changes

fixed oceanbase#1491: remove redundant member ptr, add mock for unittest

94ac9b4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filter IN pushdown to different decoders #1525

Filter IN pushdown to different decoders #1525

YiqinXiong commented Aug 22, 2023 •

edited

DengzhiLiu Aug 28, 2023

YiqinXiong Aug 28, 2023

DengzhiLiu Aug 28, 2023

YiqinXiong Aug 28, 2023

DengzhiLiu Aug 28, 2023

YiqinXiong Sep 4, 2023

DengzhiLiu Aug 28, 2023

YiqinXiong Sep 1, 2023

chaserzju Aug 28, 2023

YiqinXiong Sep 4, 2023

chaserzju Aug 28, 2023

YiqinXiong Sep 4, 2023

chaserzju Aug 28, 2023

YiqinXiong Sep 4, 2023

YiqinXiong Sep 4, 2023

chaserzju Aug 28, 2023

YiqinXiong Sep 1, 2023

chaserzju Aug 28, 2023

YiqinXiong Sep 1, 2023

DengzhiLiu Sep 4, 2023

YiqinXiong Sep 5, 2023

		ObPushdownWhiteFilterNode &filter_;
		ObDatum* batch_decode_datums_;

Filter IN pushdown to different decoders #1525

Are you sure you want to change the base?

Filter IN pushdown to different decoders #1525

Conversation

YiqinXiong commented Aug 22, 2023 • edited

Task Description

Solution Description

Passed Regressions

Unittest

Mysql test

Part 1. Load data

Part 2. Run

Part 3. Result

Other Information

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YiqinXiong commented Aug 22, 2023 •

edited