Memory footprint Redis vs Dragonfly

[rohanverma94]

How did you know that Redis would consume more memory ? And how io_uring in kernels 5.1+ is helping you?

[romange]

io_uring helped me to built a unified I/O framework helio, where I maintain a single polling loop per thread for both networking and disk i/o events. currently, it does not have a huge advantage over epoll because Dragonfly uses disk i/o only for snapshotting.

However, it makes a foundation for the next features that we have on our roadmap. Also with networking improvements, io_uring had this year I expect it will push Dragonfly further in terms of performance. To summarize, io_uring is an API of the future and our heads are to the future.

Re Redis memory consumption I did not understand the question. I know how Redis is built, so I know its strong and weak points. Its weakest points are the snapshotting and replication full sync procedures - read my blog for more details.

[rohanverma94]

With regards to "Redis memory consumption", I mean what made you write DF? From your blogs, I have an intuition that something is wrong with Redis memory management. Is snapshotting a real problem in Redis? Have you faced any issues in Redis which are only referenced by enterprise use cases that 'average joe' is not aware of? And also Seastar vs Helio, what made you choose a different library than what Scylla and Redpanda already use in production? I think your shared-nothing concept is inspired by Seastar or Am I wrong?

I've also written some DB engines. I usually do a fork-based save for snapshotting, the penalty is ofcourse, duplicated memory pages on write. Is this write approach or something I'll have to optimize?. Also great job on re-writing Redis Data Structures, What can you comment about memory overhead of DF implementation of Redis Data Structures?

Finally, this is the most crisp C++ code I've ever seen in a database engine. Do you have any style guides for me?

[romange]

Yes, based on my experience snapshotting in Redis is one of the serious design problems Redis has. I've expanded on that in my blog as well, btw. In short, this approach lacks back-pressure mechanisms: child process and OS allocate physical memory without parent process being in control. being in control is what differentiates database software from all other programs. And Redis is not in control, unfortunately.

Make the following test with dragonfly: run memtier_benchmark with -d 1024 --key-maximum=somebignum and try to fill up all your computer memory. At some point, DF will just refuse to write with error : "out of memory". Redis will crash. It looks like a small thing but it's not. Especially with in-memory store. With snapshotting, the problem is even more acute.

Anyway, I think using Linux CoW for snapshotting is ok for amateur software and PoCs and if you have 16GB workload it's probably also ok. But it's not ok where we are headed, i.e. TB-size workloads.

Re Seastar - I also wrote about it in my blog. I learned from Seastar a lot. In fact, I learned the principles of shared-nothing from seastar. I spent months hacking their code. But at the end, I came to the conclusion that I am not smart enough to use Seastar 😄 .
And I wrote Helio which I find simpler to use than Seastar and is much smaller. But of course, these things are subjective.

Re DF data-structures - we have a document here: https://github.com/dragonflydb/dragonfly/blob/main/docs/dashtable.md
we will continue to expand on that and write more docs with time.

Thanks for the compliment. I am ex-Googler and I really like their conservative C++ style, so I use that. https://google.github.io/styleguide/
With C++ it's important to know what not to use because the language is messy. For example, handling "out of memory" states is very hard in C++. Also in Rust. These languages are designed in such a way that OOM state is almost impossible to recognize. Btw, linux does not help in that perspective by allowing to allocate more than it actually has. Anyway, feel free to learn the code, and if you want you are welcome to contribute as well.

[rohanverma94]

One more question, Redis is based on jemalloc/pt2-malloc and DF is based on mi-malloc. As far as I know, the jemalloc is optimized in order to avoid fragmentation. So, does DF uses mi-malloc to facilitate reference counting backed structures like shared_ptr? And how do locality of reference of objects and compaction factor avoid fragmentation in DF while using mi-malloc?

Redis team reacted to DF's benchmark - https://redis.com/blog/redis-architecture-13-years-later/ , I'm eagerly waiting for your rebuttal.

[romange]

DF does not use shared_ptr. I used mi-malloc because of its simple and elegant design, friendliness with shared-nothing architecture (thread-local mi_heap_t API) and good benchmarks results. I did not validate the latter though.

There will be no rebuttal :) It's not what I do here. As an engineering community, we have rights and I would even say obligations to test various approaches. In the end, the community will decide which approach is better.

[rohanverma94]

Totally understood. It's up to the community to be a judge of any database. I too built some storage engines but, I was at a constant war with critics. So I stopped going to the forums for peace of my mind.

I would start exploring DF's code. Thanks 🙏 for answering all my queries.