Is there an inverse module?

[betamos]

If pull-many takes multiple streams and combines them to one, is there something that takes one stream and splits them up in several sub-streams?

I have a case where I want to send several blobs in parallel, chunked, over a network. I have a pull-stream sink on the source machine, where I can use pull-many to combine all blob streams into one stream. On the target machine, I have a pull-stream source with incoming messages. For each message, I know which sub-stream to send it (I do my own demultiplexing). What would be the best way to achieve this? Is there a pull-stream module that could be helpful?

I'm assuming back pressure is the tricky part here, since each of the sub-streams on the target machine can only pull safely on their parent if all sub-streams are ready (otherwise the message might be for a stream which is not yet ready). I believe this would be acceptable in my case though.

[dominictarr]

that is a lot more complex than this module. so far my best suggestion is https://github.com/ssbc/muxrpc that doesn't actually do back pressure though.

Needing to solve this problem actually caused me to reconsider some design decisions in pull-stream, I have a prototype of multiplexing with back pressure, https://github.com/push-stream/push-mux but it needs an algorithm for flow control. It needs to speed up and slow down, currently it only runs at a fixed rate. TCP has a thing like this, were dropped packets cause it to go slower. Need to figure something out here though.

[betamos]

Wow, this is a pretty deep rabbit hole. I'm getting a fuller picture of the problem space now. Certainly, the general demuxing problem seems hard to solve while respecting back pressure. I understand that one-to-many becomes easier with a push-based stream, but it should also make it harder to do many-to-one (which is easy with pull streams, correct?)

The way I solved this now is that I skipped back pressure on the target entirely, and just accept everything and put it in a Pushable. I rely on that processing the incoming messages is faster than the network, otherwise, I'll run out of memory. For now, this works for me and performs well, but I might soon need back pressure.

In any case, is it easier to write a demuxer if it can make progress only if all all sinks are ready? Under some conditions, that would not perform optimally, but on the other hand it does feel like its achievable and a reasonable primitive. I think for my case that would be acceptable. If so, any pointers on how to design the API or implementation would be appreciated. I'm happy to chat or vc if you are curious about discussing the problem.

On an unrelated note, is there a buffer module for pull streams, which simply stores a certain number of values between a source and a sink?

[dominictarr]

for buffering: https://www.npmjs.com/package/pull-high-watermark and also https://www.npmjs.com/package/pull-reader if you are using binary buffers