If you feel I am "moving the goalposts" why not just prove your original case? If you are spending 5 minutes on reading the source code, why not instead spend them on proving your original assertion is correct? You can then let the readers decide if they feel I "move the goalposts".
to give you some fixed "goalposts" to aim for: Provide a solution that gives the same output byte for byte.
Also you do not seem to get the point about the amount of data. I regularly have output from a singlejob that is bigger than RAM, but rarely have output from a single job that would fill /tmp. However, the totalcombined output from all the jobs will often take up more space than /tmp.
In numbers: RAM=32 GB, /tmp=400 GB, a single job=33 GB, number of jobs=1000, jobs in parallel=8.
In other words: Running all jobs and saving the outputs into files before outputting data will not be useful for me. If you want to use FIFOs I really cannot see how you can deal with output that is bigger than RAM, unless you mix output from different jobs - which again would not be useful to me. But prove me wrong by spending 5 minutes on building the solution.
As for your example:
yes 2000000000 | parallel seq | pv > /dev/null
How would you design this, if output from different jobs are not allowed to mix?
If they are allowed to mix paralel gives you:
# bytes are allowed to mix
yes 2000000000 | parallel -u seq | pv > /dev/null
# only full lines are allowed to mix
yes 2000000000 | parallel --lb seq | pv > /dev/null
none of these use space in /tmp.
I sit back with the feeling you are willing to spend hours complaining, but not 5 minutes on proving your assertion that it can be done "easy-ish".
Prove me wrong: Spend 5 minutes on the task you believed was "easy-ish".
If it cannot be done in 5 minutes, be brave enough to admit you were wrong.
I included the example:
to give you some fixed "goalposts" to aim for: Provide a solution that gives the same output byte for byte.Also you do not seem to get the point about the amount of data. I regularly have output from a single job that is bigger than RAM, but rarely have output from a single job that would fill /tmp. However, the total combined output from all the jobs will often take up more space than /tmp.
In numbers: RAM=32 GB, /tmp=400 GB, a single job=33 GB, number of jobs=1000, jobs in parallel=8.
In other words: Running all jobs and saving the outputs into files before outputting data will not be useful for me. If you want to use FIFOs I really cannot see how you can deal with output that is bigger than RAM, unless you mix output from different jobs - which again would not be useful to me. But prove me wrong by spending 5 minutes on building the solution.
As for your example:
How would you design this, if output from different jobs are not allowed to mix?If they are allowed to mix paralel gives you:
none of these use space in /tmp.I sit back with the feeling you are willing to spend hours complaining, but not 5 minutes on proving your assertion that it can be done "easy-ish".
Prove me wrong: Spend 5 minutes on the task you believed was "easy-ish".
If it cannot be done in 5 minutes, be brave enough to admit you were wrong.