It's hard to tell what exactly is going wrong here. It looks like the http2 client should be able to handle getting the settings frame early, but clearly it isn't here for some reason. One thing I noticed is that the log you supplied includes debug logs from the C++ part of the http2 implementation, but not the JavaScript part of the implementation. I'm not sure what would cause that, but it would have some relevant information.

One thing I want to note is that this problem shouldn't happen with TLS enabled, because that changes how the beginning of a connection is handled.

One thing I noticed is that the log you supplied includes debug logs from the C++ part of the http2 implementation, but not the JavaScript part of the implementation. I'm not sure what would cause that, but it would have some relevant information.

If you mean the snippet that is visible in the ticket description, that log has been severely truncated, in the interest of making it easy to recognize other users searching for similar bugs in the future.

As for you, you will be more interested in the attached zip file, which includes the complete log 😉.

No, I'm talking about the log in the zip file. The Node http2 module has two major components: the C++ part that wraps nghttp2 and makes it interact with the Node runtime, and the JavaScript part that provides the public API. As far as I can tell, the log file you provided contains logs from the C++ part but not the JavaScript part. The clearest indication of this is that I see this line from the C++ part but not this line from the JavaScript part in the log. This is an issue because I would expect this other line from the JavaScript part to be a relevant indicator for this problem.

I haven't dug into Node's built in debug logging, so I don't know what exactly would cause this discrepancy.

One thing I want to note is that this problem shouldn't happen with TLS enabled, because that changes how the beginning of a connection is handled.

Yeah, I assumed that much.

By the way, this is not a blocker on our side. For testing purpose, patching the simple proxy server is enough, and for production use cases, I expect everyone would TLS the HTTP2/gRPC connection. I shared my findings mainly because I had already done the investigation that far, and at that point, it is easily reproducible, contrarily to most other cases of RST_STREAM/code 2 errors. But no pressure.

I was missing http2 in NODE_DEBUG.

Now, I can see HTTP2 80779: Http2Session client: created, but still no bytes already in buffer. Here's the new log file: trace.log. Search for the last ===== separator to get to the failing attempt.

Here's the command I'm using. Any flag/option I should turn on?

GRPC_VERBOSITY=DEBUG GRPC_TRACE=all NODE_DEBUG_NATIVE=HTTP2SESSION,HTTP2STREAM NODE_DEBUG=http,net,http2 grpc_proxy=http://127.0.0.1:8888 node src/greeter_client.js --target 127.0.0.1:6233 --repeat 1000 &> trace.log

I got it. The problem is that node's HTTP response parser may take multiple packets ahead while filling its parsing buffer. This is exactly what happens if the server's first HTTP2 packet comes in before node is done parsing the Http/1.1 200 Connected response. Note the comment for the head argument to the HTTP.connect event.

This works locally.