Skip to content

Commit

Permalink
Fixes #283 : Fixes CPU busy loop when using request_multiple.
Browse files Browse the repository at this point in the history
See #284 for background information

This will require more testing across various libcurl versions.
I doubt that the timeout is necessary for curl_multi_select
(calls select() if it can),
but leaving in one just in case of bugs, so that it will end.
- Haven't thoroughly checked for relevant libcurl bugs.

Asynchronously wait for events with a short timeout if CURLM_CALL_MULTI_PERFORM
fails. Use shorter timeouts when the total time elapsed so far is
shorter. Make the largest possible manual usleep 2ms.
  • Loading branch information
TysonAndre committed Oct 23, 2019
1 parent 58a0bbc commit 437cecf
Showing 1 changed file with 31 additions and 0 deletions.
31 changes: 31 additions & 0 deletions library/Requests/Transport/cURL.php
Original file line number Diff line number Diff line change
Expand Up @@ -214,14 +214,44 @@ public function request_multiple($requests, $options) {

$request['options']['hooks']->dispatch('curl.before_multi_exec', array(&$multihandle));

$made_progress = true;
$all_requests_start = microtime(true);
$microseconds_taken_by_last_curl_operation = 0;
do {
$active = false;

if (!$made_progress) {
// For a small fraction of slow requests, curl_multi_select will busy loop and return immediately with no indication of error (it returns 0 immediately with/without setting curl_multi_errno).
// (https://github.com/rmccue/Requests/pull/284)
// To mitigate this, add our own sleep if curl returned but no requests completed (failing or succeeding)
// - The amount of time to sleep is the smaller of 1ms or 10% of total time spent waiting for curl requests
// (10% was arbitrarily chosen to slowing down requests that would normally take less than 1 millisecond)
// - The amount of time that was already spent in curl_multi_exec+curl_multi_select in the previous request is subtracted from that.
// (E.g. if curl_multi_select already slept for 2 milliseconds, curl_multi_select might be operating normally)
$elapsed_microseconds = (microtime(true) - $all_requests_start) * 1000000;
$microseconds_to_sleep = min($elapsed_microseconds / 10, 2000) - $microseconds_taken_by_last_curl_operation;
if ($microseconds_to_sleep >= 1) {
usleep($microseconds_to_sleep);
}
}
$operation_start = microtime(true);
$made_progress = false;
$is_first_multi_exec = true;
do {
if (!$is_first_multi_exec) {
// Sleep for 1 millisecond to avoid a busy loop that would chew CPU
// See ORA2PG-498
usleep(1000);
}
$status = curl_multi_exec($multihandle, $active);
$is_first_multi_exec = false;
}
while ($status === CURLM_CALL_MULTI_PERFORM);

// curl_multi_select will sleep for at most 50 milliseconds before returning.
curl_multi_select($multihandle, 0.05);
$microseconds_taken_by_last_curl_operation = (microtime(true) - $operation_start) * 100000;

$to_process = array();

// Read the information as needed
Expand All @@ -234,6 +264,7 @@ public function request_multiple($requests, $options) {

// Parse the finished requests before we start getting the new ones
foreach ($to_process as $key => $done) {
$made_progress = true;
$options = $requests[$key]['options'];
if (CURLE_OK !== $done['result']) {
//get error string for handle.
Expand Down

0 comments on commit 437cecf

Please sign in to comment.