The Computer Oracle

Multithreaded support in 7za

--------------------------------------------------
Rise to the top 3% as a developer or hire one of them at Toptal: https://topt.al/25cXVn
--------------------------------------------------

Music by Eric Matyas
https://www.soundimage.org
Track title: Over a Mysterious Island Looping

--

Chapters
00:00 Multithreaded Support In 7za
00:42 Accepted Answer Score 35
02:06 Answer 2 Score 7
02:53 Answer 3 Score 0
04:06 Answer 4 Score 0
04:18 Thank you

--

Full question
https://superuser.com/questions/433945/m...

--

Content licensed under CC BY-SA
https://meta.stackexchange.com/help/lice...

--

Tags
#linux #compression #7zip

#avk47



ACCEPTED ANSWER

Score 35


According to -m (Set compression Method) switch # ZipMultiThread - 7ZIP manual & documentation, mt defaults to on, so there's no need to specify it at all.

However, 7zip's implementation of the DEFLATE algorithm doesn't support multi-threading!

As you have already discovered,

7za a archive.zip bigfile

only uses one core.

But .zip files compress every file individually. When compressing several files, the multi-threading option compresses one file per core at once.

Try it and you'll see that

7za a archive.zip bigfile1 ... bigfileN

will use all available N cores.

If you want to speed up the compression of a single file, you have two choices:

  1. Split up bigfile in chunks.

  2. Use a different compression algorithm.

    For example, 7zip's implementation of the BZip2 algorithm supports multi-threading.

    The syntax is:

    7za a -mm=BZip2 archive.zip bigfile
    

Also, the syntax error is caused by your attempt to use the LZM Algorithm for a .zip container. That's not possible.

The possible algorithms for .zip conatiners are DEFLATE(64), BZip2 and no compression.

If you want to use the LZM Algorithm, use a .7z container. This container also handles the following algorithms: PPMd, BZip2, DEFLATE, BCJ, BCJ2 and no compression.




ANSWER 2

Score 7


This is an old question, and not the answer to the specific question, but an answer to the spirit of the question (Using all cores to compress a zip format)

pigz (parallel gzip with .zip option)

pigz -K -k archive.zip bigfile txt

This will give you a zip compatible file 7x faster for same compression level.

A quick comparisons of zip compatible and non-zip compressors using single and multiple cores.

wall times on i7-2600k to compress 1.0gb txt file on fedora 20

67s (120mb) 7za (zip,1 thread)
15s (141mb) 7za -mx=4 (zip,1 thread)
17s (132mb) zip (zip,1 thread)
 5s (131mb) pigz -K -k (zip,8 threads)
 9s (106mb) bsc (libbsc.com) (not zip,8 threads)
 5s (130mb) zhuff -c2 (not zip,8 threads)
 2s (149mb) zhuff (not zip,8 threads)

wall times to decompress

4.2s unzip -t
2.0s pigz -t
5.1s bsc d
0.5s zhuff -d



ANSWER 3

Score 0


Verified and tested: To use multithreading on 7za the parameter must be "-mmt#" not "-mmt=#", putting the equal sign makes it to ignore.

How i had discovered? After i run 7z without any parameter it shows the info about parameters, on switches it say "-mmt[N]", not "-mmt=[N]"

So if i understand well, the parameter you are typing "-mmt=2" may be misswritten and may be "-mmt2", without the equal sign.

Not sure if i understand well, my english is really poor.

By the way, why you use "7za" instead of just "7z"?

So to test the parameter i run a set of commands to do benchkmarks and that confirmed the typo error on some documentation. The correct parameter must be typed without the equal sign.

Command to do a benchmark with 7z with only one thread: 7z b -mmt1

Command to do a benchmark with 7z with only two threads: 7z b -mmt2

Command to do a benchmark with 7za with only two threads: 7za b -mmt2

Command to do a benchmark with 7za with only one thread: 7za b -mmt1

There is no equal sign on the parameter ˋ-mmt#ˋ, neither for 7z, nor 7za.




ANSWER 4

Score 0


Just use -mmt[N+1]

For example: -mmt2 is for one thread, -mmt9 is for eight threads