Multithreaded support in 7za
Rise to the top 3% as a developer or hire one of them at Toptal: https://topt.al/25cXVn
--------------------------------------------------
Music by Eric Matyas
https://www.soundimage.org
Track title: Over a Mysterious Island Looping
--
Chapters
00:00 Multithreaded Support In 7za
00:42 Accepted Answer Score 35
02:06 Answer 2 Score 7
02:53 Answer 3 Score 0
04:06 Answer 4 Score 0
04:18 Thank you
--
Full question
https://superuser.com/questions/433945/m...
--
Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...
--
Tags
#linux #compression #7zip
#avk47
ACCEPTED ANSWER
Score 35
According to -m (Set compression Method) switch # ZipMultiThread - 7ZIP manual & documentation, mt
defaults to on
, so there's no need to specify it at all.
However, 7zip's implementation of the DEFLATE algorithm doesn't support multi-threading!
As you have already discovered,
7za a archive.zip bigfile
only uses one core.
But .zip
files compress every file individually. When compressing several files, the multi-threading option compresses one file per core at once.
Try it and you'll see that
7za a archive.zip bigfile1 ... bigfileN
will use all available N
cores.
If you want to speed up the compression of a single file, you have two choices:
Split up
bigfile
in chunks.Use a different compression algorithm.
For example, 7zip's implementation of the BZip2 algorithm supports multi-threading.
The syntax is:
7za a -mm=BZip2 archive.zip bigfile
Also, the syntax error is caused by your attempt to use the LZM Algorithm for a .zip
container. That's not possible.
The possible algorithms for .zip
conatiners are DEFLATE(64), BZip2 and no compression.
If you want to use the LZM Algorithm, use a .7z
container. This container also handles the following algorithms: PPMd, BZip2, DEFLATE, BCJ, BCJ2 and no compression.
ANSWER 2
Score 7
This is an old question, and not the answer to the specific question, but an answer to the spirit of the question (Using all cores to compress a zip format)
pigz (parallel gzip with .zip option)
pigz -K -k archive.zip bigfile txt
This will give you a zip compatible file 7x faster for same compression level.
A quick comparisons of zip compatible and non-zip compressors using single and multiple cores.
wall times on i7-2600k to compress 1.0gb txt file on fedora 20
67s (120mb) 7za (zip,1 thread)
15s (141mb) 7za -mx=4 (zip,1 thread)
17s (132mb) zip (zip,1 thread)
5s (131mb) pigz -K -k (zip,8 threads)
9s (106mb) bsc (libbsc.com) (not zip,8 threads)
5s (130mb) zhuff -c2 (not zip,8 threads)
2s (149mb) zhuff (not zip,8 threads)
wall times to decompress
4.2s unzip -t
2.0s pigz -t
5.1s bsc d
0.5s zhuff -d
ANSWER 3
Score 0
Verified and tested: To use multithreading on 7za the parameter must be "-mmt#" not "-mmt=#", putting the equal sign makes it to ignore.
How i had discovered? After i run 7z without any parameter it shows the info about parameters, on switches it say "-mmt[N]", not "-mmt=[N]"
So if i understand well, the parameter you are typing "-mmt=2" may be misswritten and may be "-mmt2", without the equal sign.
Not sure if i understand well, my english is really poor.
By the way, why you use "7za" instead of just "7z"?
So to test the parameter i run a set of commands to do benchkmarks and that confirmed the typo error on some documentation. The correct parameter must be typed without the equal sign.
Command to do a benchmark with 7z with only one thread: 7z b -mmt1
Command to do a benchmark with 7z with only two threads: 7z b -mmt2
Command to do a benchmark with 7za with only two threads: 7za b -mmt2
Command to do a benchmark with 7za with only one thread: 7za b -mmt1
There is no equal sign on the parameter ˋ-mmt#ˋ, neither for 7z, nor 7za.
ANSWER 4
Score 0
Just use -mmt[N+1]
For example: -mmt2 is for one thread, -mmt9 is for eight threads