Launched this week
SmartPack
Lossless archiving. Zero 3rd-party dep. Less wasted work.
3 followers
Lossless archiving. Zero 3rd-party dep. Less wasted work.
3 followers
SmartPack is a lossless archiver built entirely with Python’s standard library. It adapts compression per block, skips incompressible data, deduplicates repeated blocks, optimizes zero-filled data, and verifies integrity with SHA-256. In my first benchmark, 64 MB of duplicate-heavy data became 8 MB vs 37 MB with xz, while mixed-data compression came within ~0.2% of xz’s size at about 3× the packing speed. No third-party dependencies.
I originally built SmartPack around a slightly stubborn question:
How far can a lossless archiver go with zero third-party dependencies?
Then I benchmarked it. 😅
A few results:
→ 14.57 MiB source/code dataset → 1.09 MiB
gzip: 1.63 MiB · xz: 1.19 MiB
→ 64 MiB duplicate-heavy dataset → 8 MiB
gzip: 64 MiB · xz: 37 MiB
→ 48 MiB mixed dataset → SmartPack came within ~0.2% of xz’s size, while packing about 3× faster.
The fun part isn’t a magical new compression algorithm.
SmartPack just tries not to do stupid work:
Compressible? Compress it.
Random? Leave it alone.
Duplicate? Store it once.
Zeros? Please don’t spend 8 MB storing nothing. 😅
The benchmark currently covers SmartPack vs gzip and xz. I’m not claiming 7-Zip or WinRAR wins until I benchmark those properly too.
Would love to see what kind of ugly dataset you can throw at it.