This also works for comments: https://github.com/dear-github/dear-github/issues/166#issuec...
Key derivation and key stretching algorithms are
designed for secure password hashing. Naive algorithms
such as sha1(password) are not resistant against
brute-force attacks. A good password hashing function
must be tunable, slow, and include a salt.
Your approach is not too tunable, in the sense that you can't configure its memory usage, only the time it takes. ilc sort -f weechat -i <(cat a b) | ilc dedup -f weechat
I've never tested this with logs over 200MB, but sort will read the combined log into memory, which is definitely not optimal.