I want to present a methodology I have been working on called SRAW Simple Raw. This is not a conventional compression algorithm but a preprocessing approach aimed at optimizing data before it is formatted or encoded. The key insight is that most data formats carry significant overhead in metadata headers and inefficient encoding that can be eliminated with minimal processing.
With SRAW repetitive patterns can be encoded extremely efficiently for example 1000 identical values can be represented in just a few bytes. CPU overhead is minimal making it suitable for microcontrollers and memory constrained devices. Output sizes are predictable which is critical for embedded environments. The approach can be used alongside traditional compression rather than as a replacement.
Implementation relies on smart bit packing and pattern recognition tailored to specific data types. In practice this has produced significant improvements with sensor streams telemetry and binary protocols where traditional compression either underperforms or requires excessive resources.
The underlying principle is straightforward but often overlooked optimizing data at the source can be more effective than compressing it afterwards. Removing redundancy early typically yields better efficiency than applying compression only after formatting.
I am sharing this to engage with engineers working in embedded systems IoT or other fields where data efficiency is critical. Feedback alternative approaches or similar experiences would be highly appreciated.
With SRAW repetitive patterns can be encoded extremely efficiently for example 1000 identical values can be represented in just a few bytes. CPU overhead is minimal making it suitable for microcontrollers and memory constrained devices. Output sizes are predictable which is critical for embedded environments. The approach can be used alongside traditional compression rather than as a replacement.
Implementation relies on smart bit packing and pattern recognition tailored to specific data types. In practice this has produced significant improvements with sensor streams telemetry and binary protocols where traditional compression either underperforms or requires excessive resources.
The underlying principle is straightforward but often overlooked optimizing data at the source can be more effective than compressing it afterwards. Removing redundancy early typically yields better efficiency than applying compression only after formatting.
I am sharing this to engage with engineers working in embedded systems IoT or other fields where data efficiency is critical. Feedback alternative approaches or similar experiences would be highly appreciated.
GitHub: https://github.com/DenisDolya/sraw
Documentation: https://denisdolya.github.io/sraw/