caro11ne · HackerTrans

hace 3 añosdiscuss

Do you mean like:

    parallel 'sleep {= $_=rand()*10+5; =} ; possibly_flaky {}' ::: {1..5}

The {= =} escapes to perl, so you have a full programming language available.

caro11ne

hace 3 añosdiscuss

Stop starting more jobs if load > number of cpus:

    cat list_of_file | parallel --load 100% run_intensive_process

caro11ne

hace 3 añosdiscuss

I use paralel often and find the syntax reasonable:

    cat args | parallel this-program {}
    parallel this-program {} ::: arg1 arg2 arg3 ...

The only really new thing is ::: which separates the program from the arguments if you want to give them directly.

Can you show what the syntax should be for it to be intuitive to you?

caro11ne

hace 3 añosdiscuss

Is this not covered in depth in the f.a.q? https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita...

caro11ne

hace 4 añosdiscuss

caro11ne

hace 4 añosdiscuss

> You also seem to have missed (twice!) the main thrust that [GNU Parallel is slow]

I did not miss that. I did not comment on it because I agree and so does GNU Parallel.

man parallel:

    BUGS
    [...]
       Speed
           Startup

           GNU parallel is slow at starting up - around 250 ms the
           first time and 150 ms after that.

           Job startup

           Starting a job on the local machine takes around 10 ms.
           This can be a big overhead if the job takes very few ms
           to run. Often you can group small jobs together using -X
           which will make the overhead less significant. Or you
           can run multiple GNU parallels as described in EXAMPLE:
           Speeding up fast jobs.

And man parallel_alternatives:

   DIFFERENCES BETWEEN parallel-bash AND GNU Parallel
       [...]
       parallel-bash is written in pure bash. It is really fast
       (overhead of ~0.05 ms/job compared to GNU parallel's ~3
       ms/job). So if your jobs are extremely short lived, and
       you can live with the quite limited command, this may be
       useful.

And https://www.gnu.org/software/parallel/

    Over the years GNU parallel has gotten more safety features (e.g. no silent data loss if the disk runs full in the middle of a job). These features cost performance. This graph shows the relative performance between each version.

I really do not care how fast you can produce wrong output. I care how fast you can produce correct output, and I am do not care about a specialized solution that only works for one single specialized task.

I can make a specialized solution that is faster than your specialized solution:

    $ $tm true

It gives the same output as your example, and it is way faster. But do you really feel that is a fair comparison? If you say no, then by your own arguments, I can claim you are "moving the goal posts".

> For the curious, perf ratios are actually even worse for the high volume "grep t" example made safe (21.8X slower rather than 19.4X slower - on my test machine). xargs --process-var-slot (around since 2010) is enough of a hint for anyone actually curious and there is real value to having someone solve that little puzzle themselves for their own use cases. Doing all your homework for you can take something away.

Or it might just be that your solution is not safe at all, or only works on very specialized input on your system.

I have already shown that I can do a specialized version faster than your specialized version.

As long as you do not show your work, your speed claim is just that: a claim with no evidence.

What can be asserted without evidence can also be dismissed without evidence.

> I have already highlighted 4 risks (no Perl, invisible /tmp filling default, non-drop-in xargs, slower than serial)

"No Perl": I have only once used a system without Perl: It was on an embedded system, where space was a premium. If you use a package manager to install parallel, Perl will be installed for you automatically.

"Invisible filling /tmp": I really like that behaviour, because no matter how GNU Parallel is killed, there are no files to clean up. But each to his own.

"non-drop-in xargs": Your evidence here is good and I concur, though I never hit those incompatibilites myself (apart from -n1 which is what I normally want anyway).

"slower than serial": For short-lived jobs, yes (and if your jobs are short-lived and you can live with the limitations then parallel-bash seems to be faster than xargs). In general, no. Try "seq 0.1 0.1 10 | time parallel -j 50 sleep"

I had hoped your critique would show there is a better way of running jobs in parallel. So far I can only say I am disappointed.

caro11ne

hace 4 añosdiscuss

So you knew of this limitation, but failed to mention it?!

Wow. Just wow.

I thought you were trying to show a general way to run jobs in parallel in a safe, reliable way that was faster than GNU Parallel.

You failed to do that.

Instead you showed that it is possible to run jobs faster than GNU Parallel, but in a way that is neither safe nor reliable.

(Or more correctly: If you know the exact limitations of the kernel of the OS you are currently using, it will be reliable in certain situations - but not in general).

I will pick reliable results over speed any day, thank you.

I hope the person, who mentioned safe parallel grep, will show how it is done: https://news.ycombinator.com/item?id=30891634 because I will definitely not be using your solution.

caro11ne

hace 4 añosdiscuss

I find this scary:

    $ export LC_ALL=C
    $ $tm xargs -0P1 grep $lb t < ../f      |sort |md5sum
    7.05user 9.88system 0:24.53elapsed 69%CPU (0avgtext+0avgdata 2344maxresident)k
    0inputs+0outputs (0major+4960minor)pagefaults 0swaps
    8ef2c658a70bb38438e59421231246b9  -
    $ $tm xargs -0P16 grep $lb t < ../f      |sort |md5sum
    10.16user 36.62system 0:18.30elapsed 255%CPU (0avgtext+0avgdata 2332maxresident)k
    0inputs+0outputs (0major+4980minor)pagefaults 0swaps
    c8ebf840e54ec8b5a49e159eda09e63f  -
    $ $tm parallel -X0P16 grep $lb t < ../f      |sort |md5sum
    16.97user 33.94system 0:16.36elapsed 311%CPU (0avgtext+0avgdata 51624maxresident)k
    0inputs+2069296outputs (0major+169409minor)pagefaults 0swaps
    8ef2c658a70bb38438e59421231246b9  -

It greps for lines containing t, sorts the lines and computes a hash.

Note how "xargs -P16 grep" gives the wrong answer. The output from parallel matches exactly the lines from "xargs -P1". With "-k" the lines are even in the same order (sorting removed):

    $ $tm xargs -0P1 grep $lb t < ../f      |md5sum
    7.03user 9.30system 0:16.32elapsed 100%CPU (0avgtext+0avgdata 2332maxresident)k
    0inputs+0outputs (0major+5023minor)pagefaults 0swaps
    d89b45188602c9bb08026dc2892cfa75  -
    $ $tm parallel -kX0P16 grep $lb t < ../f      |md5sum
    18.21user 36.03system 0:10.26elapsed 528%CPU (0avgtext+0avgdata 65396maxresident)k
    0inputs+2069344outputs (0major+154929minor)pagefaults 0swaps
    d89b45188602c9bb08026dc2892cfa75  -

I have not analyzed the output but I think the error is caused by the issue described here: https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w...

How anyone would ever use "xargs -P16 grep" is beyond me. I honestly do not care how fast I can get an answer, if I cannot trust the answer is correct.

I can see someone claimed they could build a safe parallel grep, but seemed not to do so: https://news.ycombinator.com/item?id=30890780#30913304 It would have been interesting to see.

caro11ne

hace 4 añosdiscuss

I use:

    iwashere() {
      sleep $(shuf -i 1-9 | head -1)
      date
      echo "===> $(hostname) <===" > ~/I_WAS_HERE
    }
    env_parallel -Sserver-0{1..5} --tag --nonall iwashere

I like that I can try out the function locally before running it remotely.

I like that I do not have to give xargs a multiline argument.

I am terrible at quoting inside quoting inside quoting, so I like that I can simply avoid the quoting. I also think my colleagues will find it easier to read (thus maintain).

comments