[FEATURE] mfs.cgi "RAM used" - SWAP not included

[oszafraniec]

System information

Your moosefs version and its origin (moosefs.com, packaged by distro, built from source, ...).

3.0.116 PRO

Operating system (distribution) and kernel version.

Ubuntu 18.04 LTS

Describe the problem you observed.

Master server used some SWAP memory few days ago. After starting "swapoff -a" to force it to RAM I've noticed that "RAM used" on mfs.cgi panel started to increase by amount of SWAP that was pushed to RAM.

Can you reproduce it? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Make master/follower server use SWAP, check the value of "RAM used" and then "swapoff" it back to RAM.

Include any warning/errors/backtraces from the system logs.

No warnings etc., just misinformation.

[chogata]

We show resident RAM usage in CGI INFO tab, this is quite typical. RSS + VIRT usage is shown on the charts and there it should not increase after your swapoff operation (just change proportion between light and dark blue) - did you check your charts? We could show both in INFO tab too, we just never felt the need, because typically mfsmaster should not be allowed to swap.

[oszafraniec]

Typically mfsmaster is not swapping, but we had a lot of metadata changes in cluster and metadata download operation (big file operation, file caches etc) "moved" some mfsmaster memory to swap which had impact on overall performance during metadata save (same system drive holds swap and metadata and during metadata download/save they fight for IO). I see it this way.

I think indication on mfs.cgi dashboard that mfsmaster is in SWAP would be nice to have overall as a "red light" to be checked. If I understand this can be made if this data is avaible on charts (you are correct, charts change colors but not overall sum of memory). Mayby "SWAP used" next to "RAM used" or make RAM usage in red if it includes SWAP space?

We have 196GB of RAM for masters (leader and followers), mfsmaster process takes ~105GB so there should be no swapping but if all other memory is used by caches... strange things can happen. There is an option to turn off SWAP, but it's not recommended overall.

This is what I've experienced (Leader master server). You can see that even SNMP was not responding sometimes during metadata download/save. Cluster also was slow or unresponsive. Metadata file is ~38GB.

[oszafraniec]

@chogata just found this issue: #303 - I think it's very similar situation.

[chogata]

No, issue #303 is the problem of RAM growth during metadata save, when there is another operation that changes the data drastically during saving time (like chunkserver and chunks registering). This is a specific case that we put safeguards against since then - in certain cases (registering, leadership change etc.) metadata save is just not done - it's done at the next scheduled time.
As for indicating swap usage - it's not that simple. VIRT does not equal SWAP. If the master process starts using SWAP, the proportion between RSS and VIRT will always change, but it can also change in other circumstances, not SWAP related. We had a quick look to see if there is explicit info for swap usage available, but it's not clear yet if the method we found is universal or linux-only. I'm putting this feature on the list, but it needs more research and it's a nice-to-have, rather than must-have one.

[oszafraniec]

@chogata np, make it as "nice-to-have", changed title to feature ;)