Intel's Storage Acceleration Library(github.com)
github.com
Intel's Storage Acceleration Library
https://github.com/01org/isa-l
6 comments
https://01.org/intel%C2%AE-storage-acceleration-library-open... has more info
I've used ISA-L as the backend implementation for erasure codes in OpenStack Swift.
There are two versions of ISA-L. The open-source one includes some optimized erasure code libraries. The other version is free (as in beer) AFAICT but it requires signing a license agreement with Intel. This version also includes some other optimized libraries for hashing, crypto, and stuff like that.
https://github.com/openstack/liberasurecode is a plugable C library that you can use for erasure coding. It supports some simple EC algorithms itself and works with jerasure and ISA-L.
https://github.com/openstack/pyeclib is a Python wrapper for liberasurecode.
There are two versions of ISA-L. The open-source one includes some optimized erasure code libraries. The other version is free (as in beer) AFAICT but it requires signing a license agreement with Intel. This version also includes some other optimized libraries for hashing, crypto, and stuff like that.
https://github.com/openstack/liberasurecode is a plugable C library that you can use for erasure coding. It supports some simple EC algorithms itself and works with jerasure and ISA-L.
https://github.com/openstack/pyeclib is a Python wrapper for liberasurecode.
This appears to be an implementation of Reed-solomon error correction code... at the assembly level. (!! handwritten assembly? I don't see that too often nowadays)
https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor...
Which is basically the mathematics behind RAID-5, RAID-6, and even hard-drive / SSD sector-level error correction. Parchive is an older open-source project that implements the same concept:
https://en.wikipedia.org/wiki/Parchive
https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor...
Which is basically the mathematics behind RAID-5, RAID-6, and even hard-drive / SSD sector-level error correction. Parchive is an older open-source project that implements the same concept:
https://en.wikipedia.org/wiki/Parchive
RAID-5 uses simply parity, RAID-6 uses something slightly more complicated, but they are not close in complexity (or capabilities) of Reed-Solomon codes -- although, if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.
You're right about Raid5.
My understanding is that Raid-6 is Reed-solomon however: https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6
> Several methods, including dual check data computations (parity and Reed-Solomon), orthogonal dual parity check data and diagonal parity, have been used to implement RAID Level 6
> if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.
Reed-Solomon codes are generalized for any number of drives with any number of failures.
There's a bitwise-efficiency thing with powers-of-two as well. I forgot the details, but I did the math back in college. I'll have to find my notes on this subject... but yeah, the matrix works for virtually everything.
I forgot whether or not Reed-Solomon becomes just a simple xor-parity check in the case of 1-failure, but it wouldn't surprise me if it were related.
My understanding is that Raid-6 is Reed-solomon however: https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6
> Several methods, including dual check data computations (parity and Reed-Solomon), orthogonal dual parity check data and diagonal parity, have been used to implement RAID Level 6
> if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.
Reed-Solomon codes are generalized for any number of drives with any number of failures.
There's a bitwise-efficiency thing with powers-of-two as well. I forgot the details, but I did the math back in college. I'll have to find my notes on this subject... but yeah, the matrix works for virtually everything.
I forgot whether or not Reed-Solomon becomes just a simple xor-parity check in the case of 1-failure, but it wouldn't surprise me if it were related.
it would be very helpful if they included a description as to what it is this library does.