HackerTrans
TopNewTrendsCommentsPastAskShowJobs

Nyan

no profile record

Submissions

Comparison of Par2cmdline, Reedsolomon, and Wirehair

1f604.blogspot.com
1 points·by Nyan·3 năm trước·0 comments

[untitled]

1 points·by Nyan·3 năm trước·0 comments

[untitled]

1 points·by Nyan·3 năm trước·0 comments

comments

Nyan
·2 năm trước·discuss
Nice work - thanks for your efforts!

Yeah, the compute complexity for Reed Solomon is generally O(size_of_input * number_of_recovery_blocks)

If you don't specify the number of blocks, par2cmdline defaults to 2000, so at 25%, it's generating 500 parity blocks, which is obviously much slower than what you're generating with the other tools.

Having said that, PAR2 is generally aimed at use cases with hundreds/thousands of parity blocks, so it's going to be at a disadvantage if you're generating less than 10 - which your benchmark shows.
Nyan
·2 năm trước·discuss
Thanks for doing that.

> par2cmdline[-turbo] encode: par2 c -r25 test

That command is rather unfair to PAR2 - you should add `-b48 -n2 -u` to make the comparison fairer.

PAR2 ain't exactly fast, particularly compared to GF8 focused formats, but the numbers you initially gave seemed wildly off, so I suspected the comparison wasn't really fair.

Ideally you should also be including the version of each tool used.
Nyan
·2 năm trước·discuss
You should at least be benchmarking against par2cmdline-turbo instead (stock par2cmdline isn't exactly performance-oriented). Also, you need to list the parameters used as they significantly impact performance of PAR2.

Your benchmark also doesn't list the redundancy %, as well as how resilient it is against corruption.

One thing I note is that both ISA-L and zfec use GF8, whilst PAR2 uses GF16. The latter is around twice as slow to compute, but allows for significantly more blocks/shards.
Nyan
·3 năm trước·discuss
Someone made this: https://github.com/manticore-projects/fpng-java

Replacing zlib might give you a few percentage points' worth of difference, whilst fpnge would likely be several times faster.
Nyan
·3 năm trước·discuss
We've already got JPEG-XL as a good replacement format, but some feel that jeopardizing its adoption is a sensible cause.
Nyan
·3 năm trước·discuss
Why not use fpnge?
Nyan
·3 năm trước·discuss
If you search up guides on how to upload to Usenet, you'll often find most of the recommend creating split RARs.

Uploaders who just transfer scene releases without bothering to extract will of course stick to RAR, but there's a lot of content these days that isn't from scene groups.
Nyan
·3 năm trước·discuss
Many older formats, ZIP included, didn't standardise on an encoding, so they just use whatever the local 8-bit encoding happens to be. And if your system charset is different from the ZIP creator's, you could get messed up filenames.

This is just another cost of sticking to legacy formats.
Nyan
·3 năm trước·discuss
Hopefully that slowly stops being the standard: https://github.com/animetosho/Nyuu/wiki/Stop-RAR-Uploads
Nyan
·3 năm trước·discuss
I follow the latest stuff but haven't seen anything I'd consider to be "a very large resurgence".

Personal opinion: whilst bad official translations certainly exist, most of the time, they're fine. There's usually stuff that could be improved (for example, songs often don't get included, typesetting etc), but as far as portraying the meaning, they generally work. And this is likely a key reason why fansubs died - the official subs are often in the "good enough" category.
Nyan
·3 năm trước·discuss
Generally the translator will find a way to work around the lack of a direct equivalent, trying to convey as much of the meaning as possible.

Now you may think that these workarounds cause some meaning loss, and you'd be correct. However, the problem with this line of thinking is that this isn't unique to honorifics - there's a tonne of other stuff in the Japanese language which has no English equivalent (I don't know Italian, so can't say much on that front), and non-Japanese speaking anime fans are often unaware of these because fansubs never introduced those concepts (for example, purpose of -desu/-masu which can be used to show respect, somewhat like honorifics).

The reality is that a lot of nuance gets lost in translation, with or without honorifics; unfortunately, if nuance is important to you, there's no substitute to learning the language. As such, people who think honorifics are highly important generally give me the impression of blissful ignorance (i.e. don't know what they don't know).

Of course, you're welcome to have your own preferences regarding translations. But having some hybrid of Japanese/English subtitle to portray slightly more nuance (whilst still missing out on a lot), can seem weird (e.g. why is this particular aspect exposed, but not something else?) and isn't necessarily objectively "better" than a translation that just sticks to English.
Nyan
·3 năm trước·discuss
From memory, the original Nyaa was nyaatorrents.org -> nyaa.eu -> nyaa.se
Nyan
·3 năm trước·discuss
Your experience may be slightly out of date.

Fansubbers these days are a tiny shadow of that they used to be. The vast majority of anime piracy these days consists of ripping official streams, which include subtitles.

Of course, if you're into older shows, you'll find many to be missing on the mainstream streaming platforms, and will often need to resort to piracy and fansubs to get that. But as anime becomes more mainstream, those interested in older stuff become a smaller slice of the market.

Also, due to more mainstream adoption, those interested in "cleaner rips, better encoding, etc" are dwindling. Whilst these people are still out there, you'll find the vast majority of people are perfectly happy with low quality streaming sites (also worth pointing out that these streams still offer vastly better video quality than what you typically got in the 90s).

Enforcement wise, I doubt they're targeting the hardcore fans - they're going after the mainstream, which is a big slice of the market.
Nyan
·3 năm trước·discuss
A possibility might be that the bigger sites have paid more attention to opsec, and are harder to take down, whilst smaller players may be easier to intimidate.

Current Nyaa (nyaa.si) has been targeted [https://torrentfreak.com/mpa-lawyers-are-trying-to-shut-down...], but efforts haven't been successful so far.

> I guess because these are commercial?

Highly unlikely - it's not like rights holders get anything regardless of commercial status. Also, Nyaa runs ads on their NSFW site, so it's commercial in that respect.
Nyan
·3 năm trước·discuss
> It's jumped several domains in the past

One thing to note is that there's actually two Nyaa sites, so it's important to not confuse the two.

The original jumped several domains, and closed down in May 2017 (last domain was nyaa.se). The "current Nyaa", nyaa.si, is a clone site that launched shortly after, and hasn't changed any domains.

With the original, I recall the domains jumped were linked to various legislative changes in Europe, with the shutdown in 2017 likely due to similar reasons.
Nyan
·3 năm trước·discuss
Very useful. In fact, it speeds up a single instance (i.e. not taking advantage of SIMD) of MD5 by 20%: https://github.com/animetosho/md5-optimisation#x86-avx512-vl...
Nyan
·3 năm trước·discuss
Faster PNG en/decoders exist though, it's probably just libpng being the most commonly used, but not extensively optimised.
Nyan
·3 năm trước·discuss
I didn't infer 15% from the way it was written there. But most platforms these days have some form of CRC32 "acceleration". Adler32 is easy to compute so I'm even less concerned there.

I spent a bunch of time optimising the code in fpnge (https://github.com/veluca93/fpnge), which is often notably faster than fpng (https://github.com/nigeltao/qoir/blob/5671f584dcf84ddb71e28d...), yet checksum time is basically negligible.

Having said that, the double-checksum aspect of PNG does feel unnecessary.