-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(core): row-first WAL segment format #4440
base: master
Are you sure you want to change the base?
Conversation
This reverts commit 3d3968e.
|
||
import static io.questdb.cairo.ColumnType.LEGACY_VAR_SIZE_AUX_SHL; | ||
|
||
public class StringTypeDriver implements ColumnTypeDriver { | ||
public static final StringTypeDriver INSTANCE = new StringTypeDriver(); | ||
|
||
public static long getPlainValueByteCount(@Nullable CharSequence value) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
long
return values is confusing and inconsistent. It should be int
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed long
to int
in VarcharTypeDriver
in 8ef6c94
For a string of Integer.MAX_VALUE
length, the bytes count will be 4294967298 which doesn't fit into an int
. So, we should probably keep long
type for the StringTypeDriver
. WDYT?
[PR Coverage check]😍 pass : 1341 / 1489 (90.06%) file detail
|
Depends on #4413 - need to introduce column conversion for row-first format once 4413 lands
Currently, we use column-oriented format for WAL segments. This means that WAL writers write into individual column files. With row-based format, they'll be writing row-wise into a single file. WAL apply job can support both formats: the difference will be in the way how it lifts WAL segments into buffers, further logic will be the same for both formats. The advantages of row-oriented format are the following:
Column-oriented WAL format is kept for efficient handling of
CREATE TABLE x AS SELECT * FROM y;
and similar column-first scenarios (not implemented yet).WAL apply job reads the transaction data from the
segment.d
and writes column values into the O3 buffers. The later data processing is the same as for the column-first format.Full design doc: https://questdb.slab.com/posts/new-row-first-wal-segment-format-4tvijkka
Implementation details
Column-oriented
WalWriter
is renamed toWalColFirstWriter
while a newWalRowFirstWriter
class is introduced. For now, this class is used by default everywhere. This is controlled with thecairo.wal.default.format
property (supported values:row
andcolumn
).Instead of the current structure, the
WalRowFirstWriter
will write the following new files as contents of thedb/{tableDirName~NN}/wal{walId}/{segmentId}/
paths.segment.d
(row-first data)ROW_FIRST_DATA
) - includes start and end offsets in the segment.d fileand no other files in the WAL segment directory.
segment.d
file format:Notes:
[descr]
are repeated sections.name: type
indicates a field.row_delimiter
field stands for the row suffix field separating rows from each other.TSBS benchmark
Dev machine
The following TSBS run covers a worst-case scenario for the new format. That's because it only uses a single table with a small number of columns (20), so the new format should have no advantage here. The goal is to make sure that there is no regression in throughput or memory usage.
Data was generated with the following command:
Ingestion was run with the following command:
Row-first format
RSS 600MB max
WAL apply job rows/s stats:
Column-first format
RSS 545MB max
WAL apply job rows/s stats:
EC2
c6a.4xlarge + 1TB gp2
Row-first format
WAL apply job rows/s stats:
Column-first format
WAL apply job rows/s stats:
1K column table benchmark
The scenario assumes a table with 1K
long
columns. The loader application may be found here.Test environment: c6a.4xlarge + 1TB gp2
Row-first format
RSS 1.2GB max
WAL apply job rows/s stats:
Ingestion: 16K rows/s on average
Column-first format
RSS 4.6GB max
WAL apply job rows/s stats:
Ingestion: 20K rows/s on average