User Tools

Site Tools


archive_types

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
archive_types [2024/04/27 08:45] – [How does the compression ratio of ZSTD compare to Deflate and LZMA?] johnsancarchive_types [2024/04/27 09:15] (current) – [Table] johnsanc
Line 7: Line 7:
 | {{ :set:ziptdc.png?50&nolink |tdc zip}}                   | TDC Zip               | Deflate (9)  | Specified by DAT     | A new normalized standard that preserves dates                                                          | Archives may not match existing sources                                                   | Total DOS Collection. Automatically used with DOSCenter DATs.                                                            | | {{ :set:ziptdc.png?50&nolink |tdc zip}}                   | TDC Zip               | Deflate (9)  | Specified by DAT     | A new normalized standard that preserves dates                                                          | Archives may not match existing sources                                                   | Total DOS Collection. Automatically used with DOSCenter DATs.                                                            |
 | {{ :set:zipzstd.png?50&nolink |zstd zip}}                 | ZSTD Zip              | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded                      | Not widely supported yet, but works with WinRAR and 7-zip                                 | People who want a better compression ratio and faster decompression than Deflate.                                        | | {{ :set:zipzstd.png?50&nolink |zstd zip}}                 | ZSTD Zip              | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded                      | Not widely supported yet, but works with WinRAR and 7-zip                                 | People who want a better compression ratio and faster decompression than Deflate.                                        |
-| {{ :set:sevenzipnlzma.png?50&nolink |non-solid lzma 7z}}  | Non-Solid LZMA 7z     | LZMA (9)     | Null                 | Excellent compression, raw copy supported                                                               | Extremely slow, single threaded                                                           | People who want to sacrifice some compression ratio over solid LZMA for the ability to raw copy files between archives. +| {{ :set:sevenzipnlzma.png?50&nolink |non-solid lzma 7z}}  | Non-Solid LZMA 7z     | LZMA (9)     | Null                 | Excellent compression ratio, raw copy supported                                                         | Extremely slow, single threaded                                                           | People who want to sacrifice some compression ratio over solid LZMA for the ability to raw copy files between archives. 
-| {{ :set:sevenzipslzma.png?50&nolink |solid lzma 7z}}      | Solid LZMA 7z (RV7z)  | LZMA (9)     | Null                 | Excellent compression, especially for many files packed together                                        | Extremely slow, single threaded, must unpack completely for any fix                       | People who care about compression ratio only and don't care how slow LZMA is.                                            |+| {{ :set:sevenzipslzma.png?50&nolink |solid lzma 7z}}      | Solid LZMA 7z (RV7z)  | LZMA (9)     | Null                 | Excellent compression ratio, especially for many files packed together                                  | Extremely slow, single threaded, must unpack completely for any fix                       | People who care about compression ratio only and don't care how slow LZMA is.                                            |
 | {{ :set:sevenzipnzstd.png?50&nolink |non-solid zstd 7z}}  | Non-Solid ZSTD 7z     | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded, raw copy supported  | Poorly supported, only works with zstd fork of 7-zip                                      | Default 7z structure for archives that need to be repacked in ToSort directories.                                        | | {{ :set:sevenzipnzstd.png?50&nolink |non-solid zstd 7z}}  | Non-Solid ZSTD 7z     | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded, raw copy supported  | Poorly supported, only works with zstd fork of 7-zip                                      | Default 7z structure for archives that need to be repacked in ToSort directories.                                        |
 | {{ :set:sevenzipszstd.png?50&nolink |solid zstd 7z}}      | Solid ZSTD 7z         | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded                      | Poorly supported, only works with zstd fork of 7-zip, must unpack completely for any fix  | Archives with many small files that likely won't change often, and you want fast decompression speed.                    | | {{ :set:sevenzipszstd.png?50&nolink |solid zstd 7z}}      | Solid ZSTD 7z         | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded                      | Poorly supported, only works with zstd fork of 7-zip, must unpack completely for any fix  | Archives with many small files that likely won't change often, and you want fast decompression speed.                    |
Line 26: Line 26:
   * Its designed to be highly tunable replacement for Deflate   * Its designed to be highly tunable replacement for Deflate
   * Its now well established and used for filesystem compression with ZFS and BTRFS   * Its now well established and used for filesystem compression with ZFS and BTRFS
-  * Its gaining popularity and support for ZSTD Zip archives was added to MAME v0.262 in January 2024 +  * Its gaining popularity and support for ZSTD Zip archives and CHDs was added to MAME v0.262 in January 2024 
   * It supports multi-threaded compression   * It supports multi-threaded compression
   * Its standard levels are 1-19, and ultra levels are 20-22 which require significantly more memory   * Its standard levels are 1-19, and ultra levels are 20-22 which require significantly more memory
 +  * Its updated periodically, but RomVault uses ZSTD version 1.5.5, which will not change for the foreseeable future in order to ensure deterministic behavior
  
  
 ==== How does the compression ratio of ZSTD compare to Deflate and LZMA? ==== ==== How does the compression ratio of ZSTD compare to Deflate and LZMA? ====
-ZSTD level 19 has a compression ratio right in between Deflate level 9 and LZMA level 9. However, since ZSTD is multithreaded its much faster to compress compared to LZMA and often times faster than Deflate. ZSTD also decompresses much faster than both LZMA and Deflate. Example statistics for the redump.org PlayStation collection:+ZSTD level 19 has a compression ratio right in between Deflate level 9 and LZMA level 9. However, since ZSTD is multi-threaded its much faster to compress compared to LZMA and often faster than Deflate. ZSTD also decompresses much faster than both LZMA and Deflate. Example statistics for the redump.org PlayStation collection:
  
 ^ Format          Size (GiB)  ^  Compression Ratio  ^  Space Savings  ^ ^ Format          Size (GiB)  ^  Compression Ratio  ^  Space Savings  ^
Line 40: Line 41:
 | Solid LZMA 7z  |  2,513.54    |  1.8462              45.83%         | | Solid LZMA 7z  |  2,513.54    |  1.8462              45.83%         |
  
-The diagram below shows the relationship of single-threaded compression speed and ratio. In the case of zstd compression, most use cases would involve multiple workers on modern hardware and compress much faster. The three callouts in the diagram represent the compression levels that RomVault uses for each method.  +The diagram below shows the relationship of single-threaded compression speed and ratio. Most ZSTD use cases would involve multiple workers (threads) on modern hardware and compress much faster. The three callouts in the diagram represent the compression levels that RomVault uses for each method.  
 {{:diagrams:zstd.png?nolink&1000|}} {{:diagrams:zstd.png?nolink&1000|}}
 ==== Why is ZSTD slow when compressing archives with many smaller files? ==== ==== Why is ZSTD slow when compressing archives with many smaller files? ====
archive_types.1714232716.txt.gz · Last modified: 2024/04/27 08:45 by johnsanc