优化的压缩包：compress
这个包基于一个优化过的紧缩（Deflate）功能，是gzip/zip/zlib使用的包。
It offers slightly better compression at lower compression settings, and up to 3x faster encoding at highest compression level.
usage
The packages are drop-in replacements for standard libraries. Simply replace the import path to use them:
| old import | new import |
|---|---|
| compress/gzip | github.com/klauspost/compress/gzip |
| compress/zlib | github.com/klauspost/compress/zlib |
| archive/zip | github.com/klauspost/compress/zip |
| compress/deflate | github.com/klauspost/compress/deflate |
You may also be interested in pgzip, which is a drop in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.
The packages contains the same as the standard library, so you can use the godoc for that: gzip, zip, zlib, flate.
Currently there is only minor speedup on decompression (primarily CRC32 calculation).
deflate optimizations
- Minimum matches are 4 bytes, this leads to fewer searches and better compression.
- Stronger hash (iSCSI CRC32) for matches on x64 with SSE 4.2 support. This leads to fewer hash collisions.
- Literal byte matching using SSE 4.2 for faster string comparisons.
- Bulk hashing on matches.
- Much faster dictionary indexing withNewWriterDict()/Reset().
- Make Bit Coder faster by assuming we are on a 64 bit CPU.
BenchmarkEncodeDigitsSpeed1e4 571065 571799 +0.13% BenchmarkEncodeDigitsSpeed1e5 3680010 4645932 +26.25% BenchmarkEncodeDigitsSpeed1e6 34667982 45532604 +31.34% BenchmarkEncodeDigitsDefault1e4 770694 619535 -19.61% BenchmarkEncodeDigitsDefault1e5 13682782 6032845 -55.91% BenchmarkEncodeDigitsDefault1e6 152778738 61443514 -59.78% BenchmarkEncodeDigitsCompress1e4 771094 620635 -19.51% BenchmarkEncodeDigitsCompress1e5 13683782 5999343 -56.16% BenchmarkEncodeDigitsCompress1e6 152648731 61228502 -59.89% BenchmarkEncodeTwainSpeed1e4 595100 570165 -4.19% BenchmarkEncodeTwainSpeed1e5 3432796 3376593 -1.64% BenchmarkEncodeTwainSpeed1e6 31573806 30687755 -2.81% BenchmarkEncodeTwainDefault1e4 828697 674388 -18.62% BenchmarkEncodeTwainDefault1e5 11572161 6733885 -41.81% BenchmarkEncodeTwainDefault1e6 122607013 68998946 -43.72% BenchmarkEncodeTwainCompress1e4 833297 679738 -18.43% BenchmarkEncodeTwainCompress1e5 14539831 7372921 -49.29% BenchmarkEncodeTwainCompress1e6 160019152 77099410 -51.82% benchmark old MB/s new MB/s speedup BenchmarkEncodeDigitsSpeed1e4 17.51 17.49 1.00x BenchmarkEncodeDigitsSpeed1e5 27.17 21.52 0.79x BenchmarkEncodeDigitsSpeed1e6 28.85 21.96 0.76x BenchmarkEncodeDigitsDefault1e4 12.98 16.14 1.24x BenchmarkEncodeDigitsDefault1e5 7.31 16.58 2.27x BenchmarkEncodeDigitsDefault1e6 6.55 16.28 2.49x BenchmarkEncodeDigitsCompress1e4 12.97 16.11 1.24x BenchmarkEncodeDigitsCompress1e5 7.31 16.67 2.28x BenchmarkEncodeDigitsCompress1e6 6.55 16.33 2.49x BenchmarkEncodeTwainSpeed1e4 16.80 17.54 1.04x BenchmarkEncodeTwainSpeed1e5 29.13 29.62 1.02x BenchmarkEncodeTwainSpeed1e6 31.67 32.59 1.03x BenchmarkEncodeTwainDefault1e4 12.07 14.83 1.23x BenchmarkEncodeTwainDefault1e5 8.64 14.85 1.72x BenchmarkEncodeTwainDefault1e6 8.16 14.49 1.78x BenchmarkEncodeTwainCompress1e4 12.00 14.71 1.23x BenchmarkEncodeTwainCompress1e5 6.88 13.56 1.97x BenchmarkEncodeTwainCompress1e6 6.25 12.97 2.08x
- "Speed" is compression level 1
- "Default" is compression level 6
- "Compress" is compression level 9
- Test files are Digits (no matches) and Twain (plain text) .
As can be seen speed on low-matching soucesDigitsare a tiny bit slower at compression level 1, but for default compression it shows a very good speedup.