Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in function responseInterceptor of file handlers/response-interceptor.ts #929

Open
2 tasks done
heby281 opened this issue Aug 2, 2023 · 1 comment
Open
2 tasks done

Comments

@heby281
Copy link

heby281 commented Aug 2, 2023

Checks

Describe the bug (be clear and concise)

Memory leak is found in function responseInterceptor of file handlers/response-interceptor.ts:
// concat data stream
_proxyRes.on('data', (chunk) => (buffer = Buffer.concat([buffer, chunk])));

The buffer cannot be released especially when the response data is very large.

Step-by-step reproduction instructions

1. Prepare 2 applications: proxy server, real server.
1. Set options in proxy server
    target: {
      protocol: 'http',
      host: '127.0.0.1',
      port: 8080,
    },
    changeOrigin: true,
    router: {
      url: 'http://127.0.0.1:8080'
    },
    agent: new https.Agent({ keepAlive: true }),
    ws: false,
    selfHandleResponse: true,
    onProxyRes: responseInterceptor((responseBuffer, proxyRes, req, res) => {
      return responseBuffer;
    }
2. Send a request, and the real server respond a large response, for example, 38MB.
3. Repeat the step 2.
4. OOM will be triggered.

Expected behavior (be clear and concise)

No memory leak.
Working code:

let buffers: Buffer[] = [];
let bufferLength = 0;
_proxyRes.on('data', (chunk) => {
    bufferLength += chunk.length;
    buffers.push(chunk);
});

_proxyRes.on('end', async () => {
// ... other code
let buffer = Buffer.from('', 'utf8');
if (bufferLength > 0) {
  buffer = Buffer.concat(buffers, bufferLength);
}
const interceptedBuffer = Buffer.from(await interceptor(buffer, originalProxyRes, req, res));
// ... other code
// clear buffers in the end.
  buffers = [];
  bufferLength = 0;
});

  _proxyRes.on('error', (error) => {
  // clear buffers when error.
  buffers = [];
  bufferLength = 0;
  res.end(`Error fetching proxied request: ${error.message}`);
});

How is http-proxy-middleware used in your project?

*** /opt/***/lib/node_modules/****
`-- [email protected]

What http-proxy-middleware configuration are you using?

target: {
      protocol: 'http',
      host: '127.0.0.1',
      port: 8080,
    },
    changeOrigin: true,
    router: {
      url: 'http://127.0.0.1:8080'
    },
    agent: new https.Agent({ keepAlive: true }),
    ws: false,
    selfHandleResponse: true,
    onProxyRes: responseInterceptor((responseBuffer, proxyRes, req, res) => {
      return responseBuffer;
    }

What OS/version and node/version are you seeing the problem?

npx: installed 1 in 1.045s

  System:
    OS: Linux 5.14 SLES 15-SP4
    CPU: (16) x64 Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz
    Memory: 14.22 GB / 31.33 GB
    Container: Yes
    Shell: 4.4.23 - /bin/bash
  Binaries:
    Node: 18.12.1 - /opt/miep/bin/node
    npm: 6.14.15 - /opt/miep/bin/npm
  Managers:
    RubyGems: 2.7.6.3 - /usr/bin/gem
  Utilities:
    Curl: 8.0.1 - /usr/bin/curl
  IDEs:
    Vim: 9.0 - /bin/vim
  Languages:
    Bash: 4.4.23 - /bin/bash
    Java: 1.8.0_372 - /usr/bin/javac
    Perl: 5.26.1 - /usr/bin/perl
    Python: 3.6.15 - /opt/miep/bin/python
    Python3: 3.6.15 - /usr/bin/python3
    Ruby: 2.5.9 - /usr/bin/ruby
  Databases:
    SQLite: 3.39.3 - /usr/bin/sqlite3

Additional context (optional)

No response

@teeohhem
Copy link

Bump. We are also running into this.

@heby281 in your "working code" above, is that a patch? Do you have a solution/PR for it yet?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants