Consider automatic incoming entity collection to simplify API (discuss)

[jrudolph]

One of the oldest issues with akka-http/pekko-http is the distinction between HttpEntity.Strict and HttpEntity.Default that cannot be observed on the wire but leads to different API behavior for the user just based on the timing with which data is received.

Strict contains an in-memory copy of the full data, while Default contains a live stream that is still connected to the connection.

The rationale for this API is that we want to supply a streaming API whenever it makes sense to 1. provide data to the user in real time as it comes in and 2. to avoid having to load all data into memory but have all the nice properties of streams like end-to-end backpressure and limited memory usage.

The backside is that while the current solution gives you all the options to deal with a real time situation optimally, it also requires you to deal with all these cases (or use toStrict in more cases).

However, this often trips up users (like in #460) where depending on the environment the exact same request can lead to different runtime behavior. Often, when running tests locally, it is much more likely that requests end up as Strict while in production, Default becomes more likely. In situations like #460, the effect can be really subtle and hard to understand.

One potential solution could be to auto-strictify incoming entities for simplicity. In that case, we would always read incoming entities to their completion (up to a fixed limit) and delay dispatching the incoming HTTP message until the full message is there. The benefit would be that users of simple applications would get every message with a strict entity.

The downside is that I can see no gradual migration path once you need streaming behavior for certain entities. Then you would still have to enable streaming for the whole application which then might not be expecting streaming entities...

Alternatives:

• (even) better documentation to educate about the way things work
• try to remove Strict from the public API (very intrusive and subtle issues like Requests with Large POST Body Immediately Goes into Delay Cancellation Right After the Server Received the Request #460 will not be solved that way)
• improve library to mitigate impact / hide internals of the streaming API further

(I have not convinced myself yet that we can or should change something right now without making other cases worse but still wanted to keep a note here)