Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEBUG POPULATE OOM #2914

Open
andydunstall opened this issue Apr 17, 2024 · 0 comments
Open

DEBUG POPULATE OOM #2914

andydunstall opened this issue Apr 17, 2024 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@andydunstall
Copy link
Contributor

andydunstall commented Apr 17, 2024

Describe the bug
Using DEBUG POPULATE on a master with one replica causes Dragonfly to run out of memory with high RSS

To Reproduce
Created a master and replica with 200GB each using Dragonfly 1.16.1.

Running redis-cli -p 9999 debug populate 50000000 test 1000 rand to get 50GB of data worked ok.

Then running redis-cli -p 9999 debug populate 100000000 test 1000 rand to get 100GB of data (on a 200GB machine) and Dragonfly OOM'ed

Running an INFO memory while populate was running (prior to OOM) shows high RSS and replication_streaming_buffer_bytes:

wql1kr9g7.dev.dragonflydb.cloud:6385> info memory
# Memory
used_memory:81590059392
used_memory_human:75.99GiB
used_memory_peak:72276289920
used_memory_peak_human:67.31GiB
fibers_stack_vms:4159504
fibers_count:127
used_memory_rss:134092623872
used_memory_rss_human:124.88GiB
used_memory_peak_rss:134092623872
maxmemory:199999094784
maxmemory_human:186.26GiB
object_used_memory:77293060096
type_used_memory_STRING:77293060096
table_used_memory:4233477888
num_buckets:7864320
num_entries:86264576
inline_keys:86264576
listpack_blobs:0
listpack_bytes:0
small_string_bytes:77293060096
pipeline_cache_bytes:0
dispatch_queue_bytes:2016
dispatch_queue_subscriber_bytes:0
dispatch_queue_peak_bytes:6552
client_read_buffer_peak_bytes:18688
tls_bytes:0
cache_mode:store
maxmemory_policy:noeviction
replication_streaming_buffer_bytes:71403831296
replication_full_sync_buffer_bytes:0

Repeating without replication works ok.

Environment (please complete the following information):

  • Instance: x2gd.4xlarge
  • Dragonfly Version: v1.16.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants