Memory leak issues #79
Comments
|
@irabinovitch Can you share what you found? |
|
I haven't found anything or done any investigation yet. |
|
Looking briefly at monitoring data when this happens CPU and memory spike. Apache2 processes look to be where thats happening from Datadog process monitoring. From a quick look at the configs it seems we're using mpm_prerfork here. One idea we could consider is to set MaxRequestsPerChild to something other than the default of 0. That would limit the number of requests before an individual Apache process is retired/restarted. I dont know that I'd call that a fix, but it would probably at least mitigate the run away memory usage. |
|
I think you mean MaxConnectionsPerChild ? Oh, I see the old name was MaxRequestsPerChild. Samesame. I can whip up a PR for that. |
|
Sure, MaxRequestsPerChild is the new name for MaxConnectionsPerChild as of 2.3.9. I think they have the exact same effect, but yes we should use the new name. Definitely shouldn't be 0, thats just the default. Not sure what a reasonable number is, and yes I'd like to find whatever we're leaking or hanging on - but this seems like a reasonable defense if someone wants to try it. |
|
This doesn't seem to have helped stability. |
|
Can you provide access to datadog or whatever logging system you have? |
|
Sent an invite. I don't think we have php tracing or profiling enabled at
the moment. Can do that if we'd like.
…On Sat, Oct 7, 2023, 12:20 AM Phillip Smith ***@***.***> wrote:
Can you provide access to datadog or whatever logging system you have?
—
Reply to this email directly, view it on GitHub
<#79 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACGW5V2UJSW7T6BECDY7CDX6BYZRAVCNFSM6AAAAAA2IBFOWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJRGQYTKOBTGY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
It's not a glaring issue but I've optimized the query, anyway. While I seek other potential culprits, let's get this merged into prod. |
* Log the state of apache processes when we restart so we can continue to understand the problem * Restart slightly less often * Update MaxConnectionsPerChild to 50, which is probably a bit more reasonable than 5 * Update server limit (see below) * Turn off KeepAlive (see below) From scale-infra, explanation on the last two above. ``` This is a confirmed bug in Apache with no configuration workaround: https://bz.apache.org/bugzilla/show_bug.cgi?id=53555 There's a pretty good description of the details, which match our behavior here: https://serverfault.com/questions/516373/what-is-the-meaning-of-ah00485-scoreboard-is-full-not-at-maxrequestworkers One thing that some people seem to have had success with is turning keepalive off. ServerLimit, for the event MPM (what we use), the recommendation is: With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default). Our MaxRequestWorkers is 50. We don't define ThreadsPerChild which defaults to 25. So if I read that correctly (and I haven't tuned Apache for a living in a looonnnggg time), we want something like 50+25=75 plus some more for shutting down processes, so like... 80? I can prep a diff for that plus keepalive and see how that goes. ``` All continuing socallinuxexpo/scale-drupal#79 Signed-off-by: Phil Dibowitz <[email protected]>
* Log the state of apache processes when we restart so we can continue to understand the problem * Restart slightly less often * Update MaxConnectionsPerChild to 50, which is probably a bit more reasonable than 5 * Update server limit (see below) * Turn off KeepAlive (see below) From scale-infra, explanation on the last two above. ``` This is a confirmed bug in Apache with no configuration workaround: https://bz.apache.org/bugzilla/show_bug.cgi?id=53555 There's a pretty good description of the details, which match our behavior here: https://serverfault.com/questions/516373/what-is-the-meaning-of-ah00485-scoreboard-is-full-not-at-maxrequestworkers One thing that some people seem to have had success with is turning keepalive off. ServerLimit, for the event MPM (what we use), the recommendation is: With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default). Our MaxRequestWorkers is 50. We don't define ThreadsPerChild which defaults to 25. So if I read that correctly (and I haven't tuned Apache for a living in a looonnnggg time), we want something like 50+25=75 plus some more for shutting down processes, so like... 80? I can prep a diff for that plus keepalive and see how that goes. ``` All continuing socallinuxexpo/scale-drupal#79 Signed-off-by: Phil Dibowitz <[email protected]>
|
OK I've done a few more things here:
|
The webserver crashes a few times a week due to running out of memory. @irabinovitch did some sleuthing and believes it's holding open old database connections and they are stacking up.
We should definitely try to sort this out as part of the 2024 launch.
The text was updated successfully, but these errors were encountered: