Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm starting to explore some preliminary ideas for an fs2-oriented
Queue
. FS2 provides an integration with the Cats Effect Queue, but it is neither stream- nor chunk-aware, so there are some inefficiencies and workarounds in place to get things to play well. Based on conversation in Gitter, some desirable properties of such fs2-oriented queue are: stream termination, chunk awareness, error propagation, and backpressure.Before I get any hopes up, I haven't actually implement a
Queue
in this PR. I add a new combinator,bufferThrough
, which is implemented in terms of the CE queue and I think satisfies most of the properties laid out above.bufferThrough
is pretty much theidentity
function, but elements are buffered through a queue. Actually, there is another function that already does exactly this:prefetch
. The main difference is thatbufferThrough
gives you more freedom to control backpressure semantics by supplying whatever kind of queue you want, using a trick similar to the one that @SystemFw used forAsync.cont
. Accordingly,prefetch
can be implemented in terms ofbufferThrough
.So I'm not actually introducing anything groundbreaking here! My goal is rather to draw attention to yet another dimension of streaming queues: the topology of the system.
prefetch
andbufferThrough
are both single-producer, single-consumer systems. I'm not sure yet how to generalize over the remaining three corners of the matrix, if that's even possible.An interesting question is how stream termination would work in multiple-producer systems, if it even makes sense at all. The only behavior that makes sense to me is terminating all producers and consumers whenever a single producer terminates, but that doesn't seem very useful. At that point, I think you would just use
enqueueUnterminated
anddequeueUnterminated
freely.I should've probably opened a separate issue or discussion for this, but oh well.