Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REDIS BUG REPORT - ASSERTION FAILED #1081

Open
kopoev opened this issue Jan 17, 2024 · 4 comments
Open

REDIS BUG REPORT - ASSERTION FAILED #1081

kopoev opened this issue Jan 17, 2024 · 4 comments

Comments

@kopoev
Copy link

kopoev commented Jan 17, 2024

56748:M 17 Jan 2024 16:03:52.418 # <redisgears_2> v8 oom error on LocalHeap: allocation failed, is_heap_oom:true
56748:M 17 Jan 2024 16:03:52.418 # <redisgears_2> Application panicked, panicked at redisgears_v8_plugin/src/v8_backend.rs:435:17:
v8 oom error on LocalHeap: allocation failed, is_heap_oom:true

=== REDIS BUG REPORT START: Cut & paste starting from here ===
56748:M 17 Jan 2024 16:03:52.418 # === ASSERTION FAILED ===
56748:M 17 Jan 2024 16:03:52.418 # ==> redisgears_v8_plugin/src/v8_backend.rs:435 'Crashed on panic' is not true

------ STACK TRACE ------

Backtrace:
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x13f0fa)[0x7fe9e86bb0fa]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x18f888)[0x7fe9e870b888]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x18f5de)[0x7fe9e870b5de]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x18e2e6)[0x7fe9e870a2e6]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x18f342)[0x7fe9e870b342]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x1149d5)[0x7fe9e86909d5]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x10f0b8)[0x7fe9e868b0b8]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x13e199)[0x7fe9e86ba199]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x1c40b0)[0x7fe9e87400b0]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x2a3277)[0x7fe9e881f277]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x2bd612)[0x7fe9e8839612]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x2812d3)[0x7fe9e87fd2d3]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0xb747ed)[0x7fe9e90f07ed]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0xb74581)[0x7fe9e90f0581]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0xb73a88)[0x7fe9e90efa88]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0xb98619)[0x7fe9e9114619]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x60a006)[0x7fe9e8b86006]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x1ddebb)[0x7fe9e8759ebb]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x60b545)[0x7fe9e8b87545]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0xa1696f)[0x7fe9e8f9296f]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0x1c22ed)[0x7fe9e873e2ed]
/opt/redis-stack/lib/libredisgears_v8_plugin.so(+0xa11f4e)[0x7fe9e8f8df4e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7fe9ecc16ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7fe9ecca8850]

------ INFO OUTPUT ------

Server

redis_version:7.2.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:3a23a5ac67fe7008
redis_mode:standalone
os:Linux 5.15.0-91-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.4.0
process_id:56748
process_supervised:no
run_id:2664aec5a2a9753ca669fe7ba89e1770697a0d94
tcp_port:6500
server_time_usec:1705507432410754
uptime_in_seconds:1517
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:11008616
executable:/opt/redis-stack/bin/redis-server
config_file:/etc/redis/6500.conf
io_threads_active:0
listener0:name=tcp,bind=*,port=6500
listener1:name=unix,bind=/tmp/redis6500.sock

Clients

connected_clients:3
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:49200
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

Memory

used_memory:1762048536
used_memory_human:1.64G
used_memory_rss:1839947776
used_memory_rss_human:1.71G
used_memory_peak:1762644152
used_memory_peak_human:1.64G
used_memory_peak_perc:99.97%
used_memory_overhead:86644424
used_memory_startup:1192552
used_memory_dataset:1675404216
used_memory_dataset_perc:95.15%
allocator_allocated:1764576328
allocator_active:1787297792
allocator_resident:1819410432
total_system_memory:67422941184
total_system_memory_human:62.79G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:22721464
allocator_rss_ratio:1.02
allocator_rss_bytes:32112640
rss_overhead_ratio:1.01
rss_overhead_bytes:20537344
mem_fragmentation_ratio:1.04
mem_fragmentation_bytes:77919224
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:55952
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

Persistence

loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:18694815
rdb_bgsave_in_progress:0
rdb_last_save_time:1705505915
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

Stats

total_connections_received:13
total_commands_processed:7679220
instantaneous_ops_per_sec:30762
total_net_input_bytes:165084546
total_net_output_bytes:7347317
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:559.09
instantaneous_output_kbps:1.05
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:20
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:4347056
keyspace_misses:428902
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:27057
total_forks:67
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:10
dump_payload_sanitizations:0
total_reads_processed:83581
total_writes_processed:83566
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:74
reply_buffer_expands:143
eventloop_cycles:96781
eventloop_duration_sum:130566397
eventloop_duration_cmd_sum:124705478
instantaneous_eventloop_cycles_per_sec:25
instantaneous_eventloop_duration_usec:19562
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

Replication

role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:38dfbeee1a4f8ae4154e05b53a508eef5e5b8ada
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:16.139775
used_cpu_user:118.507768
used_cpu_sys_children:3.251768
used_cpu_user_children:3.083873
used_cpu_sys_main_thread:0.214147
used_cpu_user_main_thread:1.162115

Modules

module:name=search,ver=20811,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=timeseries,ver=11011,api=1,filters=0,usedby=[],using=[],options=[]
module:name=ReJSON,ver=20607,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=redisgears_2,ver=20017,api=1,filters=0,usedby=[],using=[],options=[]
module:name=RedisCompat,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=bf,ver=20610,api=1,filters=0,usedby=[],using=[],options=[]

Commandstats

cmdstat_ts.mget:calls=88,usec=12846,usec_per_call=145.98,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=1,usec=2,usec_per_call=2.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=271,usec=21712,usec_per_call=80.12,rejected_calls=0,failed_calls=0
cmdstat_ts.mrange:calls=264,usec=96423,usec_per_call=365.24,rejected_calls=0,failed_calls=0
cmdstat_FT.CREATE:calls=2,usec=509,usec_per_call=254.50,rejected_calls=0,failed_calls=0
cmdstat_exists:calls=1383957,usec=1498378,usec_per_call=1.08,rejected_calls=0,failed_calls=0
cmdstat_tfunction:calls=4,usec=1115,usec_per_call=278.75,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=9,usec=11,usec_per_call=1.22,rejected_calls=0,failed_calls=0
cmdstat_tfcallasync:calls=2,usec=420,usec_per_call=210.00,rejected_calls=0,failed_calls=0
cmdstat_ts.add:calls=3371838,usec=24215617,usec_per_call=7.18,rejected_calls=0,failed_calls=0
cmdstat_FT._LIST:calls=6,usec=59,usec_per_call=9.83,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=1011385,usec=413616,usec_per_call=0.41,rejected_calls=0,failed_calls=0
cmdstat_ts.create:calls=4708,usec=74359,usec_per_call=15.79,rejected_calls=0,failed_calls=0
cmdstat_tfcall:calls=260767,usec=124672697,usec_per_call=478.10,rejected_calls=0,failed_calls=10
cmdstat_time:calls=260417,usec=156840,usec_per_call=0.60,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=441,usec=3154,usec_per_call=7.15,rejected_calls=0,failed_calls=0
cmdstat_FT.INFO:calls=663,usec=23027,usec_per_call=34.73,rejected_calls=0,failed_calls=1
cmdstat_hset:calls=1384401,usec=40558002,usec_per_call=29.30,rejected_calls=0,failed_calls=0

Errorstats

errorstat_Unknown:count=10

Latencystats

latency_percentiles_usec_ts.mget:p50=29.055,p99=458.751,p99.9=552.959
latency_percentiles_usec_config|get:p50=2.007,p99=2.007,p99.9=2.007
latency_percentiles_usec_info:p50=78.335,p99=143.359,p99.9=179.199
latency_percentiles_usec_ts.mrange:p50=10.047,p99=1818.623,p99.9=4227.071
latency_percentiles_usec_FT.CREATE:p50=138.239,p99=372.735,p99.9=372.735
latency_percentiles_usec_exists:p50=1.003,p99=3.007,p99.9=13.055
latency_percentiles_usec_tfunction:p50=219.135,p99=362.495,p99.9=362.495
latency_percentiles_usec_ping:p50=1.003,p99=4.015,p99.9=4.015
latency_percentiles_usec_tfcallasync:p50=112.127,p99=309.247,p99.9=309.247
latency_percentiles_usec_ts.add:p50=5.023,p99=40.191,p99.9=76.287
latency_percentiles_usec_FT._LIST:p50=8.031,p99=19.071,p99.9=19.071
latency_percentiles_usec_hget:p50=0.001,p99=1.003,p99.9=4.015
latency_percentiles_usec_ts.create:p50=12.031,p99=69.119,p99.9=115.199
latency_percentiles_usec_tfcall:p50=157.695,p99=7340.031,p99.9=14942.207
latency_percentiles_usec_time:p50=0.001,p99=2.007,p99.9=7.007
latency_percentiles_usec_hgetall:p50=6.015,p99=15.039,p99.9=76.287
latency_percentiles_usec_FT.INFO:p50=30.079,p99=120.319,p99.9=258.047
latency_percentiles_usec_hset:p50=26.111,p99=92.159,p99.9=158.719

Cluster

cluster_enabled:0

Keyspace

db0:keys=1715463,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=13 addr=192.168.50.91:48020 laddr=192.168.50.91:6500 fd=16 name= age=1517 idle=3 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=2048 rbp=1024 obl=0 oll=0 omem=0 tot-mem=2952 events=r cmd=tfcall user=default redir=-1 resp=2 lib-name= lib-ver=
id=14 addr=192.168.50.91:48034 laddr=192.168.50.91:6500 fd=17 name= age=1517 idle=65 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=tfcall user=default redir=-1 resp=2 lib-name= lib-ver=
id=102 addr=/tmp/redis6500.sock:0 laddr=/tmp/redis6500.sock:0 fd=18 name= age=56 idle=0 flags=U db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=46894 qbuf-free=2252 argv-mem=8062 multi-mem=0 rbs=1024 rbp=284 obl=60 oll=0 omem=0 tot-mem=59166 events=r cmd=tfcall user=default redir=-1 resp=2 lib-name= lib-ver=

------ CURRENT CLIENT INFO ------
id=102 addr=/tmp/redis6500.sock:0 laddr=/tmp/redis6500.sock:0 fd=18 name= age=56 idle=0 flags=U db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=46894 qbuf-free=2252 argv-mem=8062 multi-mem=0 rbs=1024 rbp=284 obl=60 oll=0 omem=0 tot-mem=59166 events=r cmd=tfcall user=default redir=-1 resp=2 lib-name= lib-ver=
argc: '4'
argv[0]: '"TFCALL"'
argv[1]: '"op.subMsg"'
argv[2]: '"0"'
argv[3]: '"[{"op":"i","o":[[1796525829,54637756,34,1,9,9,0,1,1.5,0],[0,11308383,1705499366888,-1,3.1]]},{"op":"i","o":[[1810350729,54736030"'

------ EXECUTING CLIENT INFO ------
id=102 addr= laddr= fd=-1 name= age=1517 idle=1517 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=4 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=4 oll=0 omem=0 tot-mem=17560 events= cmd=hset user=default redir=-1 resp=3 lib-name= lib-ver=
argc: '0'
argv[0]: '"TS.ADD"'
argv[1]: '"op:ts:1812721689:p"'

------ MODULES INFO OUTPUT ------

search_version

search_version:2.8.11
search_redis_version:7.2.4 - oss

search_index

search_number_of_indexes:2

search_fields_statistics

search_fields_numeric:Numeric=2,Sortable=2

search_dialect_statistics

search_dialect_1:0
search_dialect_2:0
search_dialect_3:0
search_dialect_4:0

search_runtime_configurations

search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:1000000
search_max_aggregate_results:-1
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3

ReJSON_trace

ReJSON_backtrace: 0: redis_module::basic_info_command_handler
1: rejson::__info_func
2: modulesCollectInfo
at /__w/redis-stack/redis-stack/redis/src/module.c:10293:9
3: logModulesInfo
at /__w/redis-stack/redis-stack/redis/src/debug.c:1900:22
printCrashReport
at /__w/redis-stack/redis-stack/redis/src/debug.c:2183:5
4: _serverAssert
at /__w/redis-stack/redis-stack/redis/src/debug.c:1062:9
5: <redisgears_v8_plugin::v8_backend::V8Backend as redisgears_plugin_api::redisgears_plugin_api::backend_ctx::BackendCtxInterfaceUninitialised>::on_load::{{closure}}
6: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2021:9
std::panicking::rust_panic_with_hook
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:783:13
7: std::panicking::begin_panic_handler::{{closure}}
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:657:13
8: std::sys_common::backtrace::__rust_end_short_backtrace
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:170:18
9: rust_begin_unwind
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
10: core::panicking::panic_fmt
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
11: redisgears_v8_plugin::v8_backend::V8Backend::initialize_v8_engine::{{closure}}::panic_cold_display
12: redisgears_v8_plugin::v8_backend::V8Backend::initialize_v8_engine::{{closure}}
13: _ZN2v88internal2V823FatalProcessOutOfMemoryEPNS0_7IsolateEPKcRKNS_10OOMDetailsE
14: _ZN2v88internal4Heap23FatalProcessOutOfMemoryEPKc
15: _ZN2v88internal12LocalFactory11AllocateRawEiNS0_14AllocationTypeENS0_19AllocationAlignmentE
16: _ZN2v88internal11FactoryBaseINS0_12LocalFactoryEE13NewFixedArrayEiNS0_14AllocationTypeE
17: _ZN2v88internal6maglev19MaglevCodeGenerator26GenerateDeoptimizationDataEPNS0_12LocalIsolateE
18: _ZN2v88internal6maglev19MaglevCodeGenerator15BuildCodeObjectEPNS0_12LocalIsolateE
19: _ZN2v88internal6maglev19MaglevCodeGenerator8AssembleEv
20: _ZN2v88internal6maglev14MaglevCompiler7CompileEPNS0_12LocalIsolateEPNS1_21MaglevCompilationInfoE
21: _ZN2v88internal6maglev20MaglevCompilationJob14ExecuteJobImplEPNS0_16RuntimeCallStatsEPNS0_12LocalIsolateE
22: _ZN2v88internal23OptimizedCompilationJob10ExecuteJobEPNS0_16RuntimeCallStatsEPNS0_12LocalIsolateE
23: _ZN2v88internal6maglev26MaglevConcurrentDispatcher7JobTask3RunEPNS_11JobDelegateE
24: _ZN2v88platform16DefaultJobWorker3RunEv
25: _ZN2v88platform30DefaultWorkerThreadsTaskRunner12WorkerThread3RunEv
26: _ZN2v84baseL11ThreadEntryEPv
27:
28:

redisgears_2_trace

redisgears_2_backtrace: 0: redis_module::basic_info_command_handler
1: redisgears::gears_module::__info_func
2: modulesCollectInfo
at /__w/redis-stack/redis-stack/redis/src/module.c:10293:9
3: logModulesInfo
at /__w/redis-stack/redis-stack/redis/src/debug.c:1900:22
printCrashReport
at /__w/redis-stack/redis-stack/redis/src/debug.c:2183:5
4: _serverAssert
at /__w/redis-stack/redis-stack/redis/src/debug.c:1062:9
5: <redisgears_v8_plugin::v8_backend::V8Backend as redisgears_plugin_api::redisgears_plugin_api::backend_ctx::BackendCtxInterfaceUninitialised>::on_load::{{closure}}
6: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2021:9
std::panicking::rust_panic_with_hook
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:783:13
7: std::panicking::begin_panic_handler::{{closure}}
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:657:13
8: std::sys_common::backtrace::__rust_end_short_backtrace
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:170:18
9: rust_begin_unwind
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
10: core::panicking::panic_fmt
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
11: redisgears_v8_plugin::v8_backend::V8Backend::initialize_v8_engine::{{closure}}::panic_cold_display
12: redisgears_v8_plugin::v8_backend::V8Backend::initialize_v8_engine::{{closure}}
13: _ZN2v88internal2V823FatalProcessOutOfMemoryEPNS0_7IsolateEPKcRKNS_10OOMDetailsE
14: _ZN2v88internal4Heap23FatalProcessOutOfMemoryEPKc
15: _ZN2v88internal12LocalFactory11AllocateRawEiNS0_14AllocationTypeENS0_19AllocationAlignmentE
16: _ZN2v88internal11FactoryBaseINS0_12LocalFactoryEE13NewFixedArrayEiNS0_14AllocationTypeE
17: _ZN2v88internal6maglev19MaglevCodeGenerator26GenerateDeoptimizationDataEPNS0_12LocalIsolateE
18: _ZN2v88internal6maglev19MaglevCodeGenerator15BuildCodeObjectEPNS0_12LocalIsolateE
19: _ZN2v88internal6maglev19MaglevCodeGenerator8AssembleEv
20: _ZN2v88internal6maglev14MaglevCompiler7CompileEPNS0_12LocalIsolateEPNS1_21MaglevCompilationInfoE
21: _ZN2v88internal6maglev20MaglevCompilationJob14ExecuteJobImplEPNS0_16RuntimeCallStatsEPNS0_12LocalIsolateE
22: _ZN2v88internal23OptimizedCompilationJob10ExecuteJobEPNS0_16RuntimeCallStatsEPNS0_12LocalIsolateE
23: _ZN2v88internal6maglev26MaglevConcurrentDispatcher7JobTask3RunEPNS_11JobDelegateE
24: _ZN2v88platform16DefaultJobWorker3RunEv
25: _ZN2v88platform30DefaultWorkerThreadsTaskRunner12WorkerThread3RunEv
26: _ZN2v84baseL11ThreadEntryEPv
27:
28:

redisgears_2_V8AggregatedLibraryStatistics

redisgears_2_combined_memory_limit:2379776
redisgears_2_not_active:3
redisgears_2_total_heap_size:2379776
redisgears_2_active:1
redisgears_2_used_heap_size:942204

redisgears_2_PerLibraryInformation

redisgears_2_037aeaeaf4bbf26ddabe7256a8294dc52da48d575a1247b5c2598c47de7aebab:function_count(async)=1,notification_consumers_count=0,stream_consumers_count=0,api_version=1.0,pending_jobs_count=0,pending_async_calls_count=0,cluster_functions_count=0,heap_size_limit=2883584,used_heap_size=942204,function_count(sync)=11,total_heap_size=2379776

------ CONFIG DEBUG OUTPUT ------
replica-read-only yes
list-compress-depth 0
lazyfree-lazy-expire no
repl-diskless-sync yes
io-threads 1
io-threads-do-reads no
lazyfree-lazy-server-del no
repl-diskless-load disabled
activedefrag no
proto-max-bulk-len 512mb
slave-read-only yes
sanitize-dump-payload no
lazyfree-lazy-eviction no
client-query-buffer-limit 1gb
lazyfree-lazy-user-flush no
lazyfree-lazy-user-del no

------ FAST MEMORY TEST ------
56748:M 17 Jan 2024 16:03:52.456 # main thread terminated
56748:M 17 Jan 2024 16:03:52.456 # Bio worker thread #0 terminated
56748:M 17 Jan 2024 16:03:52.457 # Bio worker thread #1 terminated
56748:M 17 Jan 2024 16:03:52.457 # Bio worker thread #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       http://github.com/redis/redis/issues

If a Redis module was involved, please open in the module's repo instead.

Suspect RAM error? Use redis-server --test-memory to verify it.

Some other issues could be detected by redis-server --check-system

@kopoev
Copy link
Author

kopoev commented Jan 19, 2024

Downgrading to redis-stack 7.2.0-v3, now working without issues

@MeirShpilraien
Copy link
Collaborator

MeirShpilraien commented Jan 20, 2024

Thanks @kopoev for reporting. Any chance you can provide a reproduction steps including the js code you run? Thanks.

@kopoev
Copy link
Author

kopoev commented Feb 7, 2024

Hi @MeirShpilraien, unfortunately I am unable to isolate simple use case in order to reproduce the crash. What I observed is that most of the time, when crash happens:

  1. in log file, redis report ends at: ------ MODULES INFO OUTPUT ------. Seems like some lock happens
  2. redis process accept new TCP connections, but does not respond in any way, even on PING
  3. systemctl restart takes a few minutes in order to restart redis stack server process

Will try the new V8 build and let you know if the problem still exists.

@MeirShpilraien
Copy link
Collaborator

@kopoev its definitely an OOM issue on the V8 (though RedisGears should catch it and not crash). Did you try to configure the V8 memory parameters?
To avoid the crash I would recommended increasing the v8-library-initial-memory-limit and v8-library-memory-usage-delta to around 20M.
To catch the crash and solve it we will need a reproduction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants