-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document vm_overcommit requirement (was Seemingly High Quanity of Memory Requested) #284
Comments
Yes, it requests that memory but only uses what it needs, part of the DB pool logic. The OS properly configured should still give it. It may be that vm.overcommit needs to be turned on. I'll have to check that and document it. |
You may need to set |
For more explication. Moor builds out its own custom database and does not use an off the shelf key value store. In the future I may go back to providing the option of an alternate backend based around something like RocksDB or LLMDB, etc. But for now in order to make the transaction mgmt side to moor work performantly and be fully under my control, this is how it is. The buffer pooling for that DB uses strategies from the LeanStore and, in particular, Umbra papers: https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf In that system large quantities of virtual memory are requested, in multiple page sizes. But these are virtual memory pages, and are not actually resident process allocated. In reality the OS only actually really allocates pages from that pool as the pages are used. You can see the actual memory use under the process RSS in top etc rather than the virtual memory it has requested. This technique is used to efficiently allocate and free pages of varying size for database usage. But it does require that the overcommit_memory option in the kernel be turned on. Keep in mind this is all code under heavy development, no bug-free or production-quality guarantees are made. |
Hello, rdaum. Thank you for your work and feedback. I did a fresh install of Docker on Ubuntu 22.04 for your project.
But I also have a KVM server running, so maybe that's interfering and my situation represents a special case. |
This is the first thing I ran into when trying to play around with `moor` :) Fixes rdaum#284
This is the first thing I ran into when trying to play around with `moor` :) Fixes #284
Docker Message:
moor-daemon | 2024-02-18T09:28:15.014657Z INFO main moor_daemon: crates/daemon/src/main.rs:185: Daemon starting...
moor-daemon | The application panicked (crashed).
moor-daemon | Message: Unable to create pager: InitializationError("Mmap failed for size class block_size: 4096, virt_size 1099511627776: Cannot allocate memory (os error 12)")
crates/db/src/lib.rs:64
RelBoxWorldState::open(self.path.clone(), self.memory_size.unwrap_or(1 << 40));
I don't know Rust, but this program appears to request a terabyte of memory, which is not available on my system. I apologize if there is simply a setting I neglected.
The text was updated successfully, but these errors were encountered: