archive_types []

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Archive Types ======
RomVault supports several different archive options for your ROM sets. Unlike most other ROM managers, RomVault focuses on consistency and normalizes archives. All archives that RomVault creates are deterministic. Everyone's archive for a particular set of files will be 100% hash equal if the same archive options are used.

^ Icon                                                      ^ Type                  ^ Compression  ^ Date Normalization   ^ Pros                                                                                                    ^ Cons                                                                                      ^ Recommended for...                                                                                                       ^
| {{ :set:dir.png?25&nolink |folder}}                       | File                  | None         | None, not required   | Fast, directly playable                                                                                 | Significantly more space required                                                         | People who use filesystem compression, non-compressible data                                                             |
| {{ :set:ziptrrnt.png?50&nolink |tzip}}                    | TrrntZip              | Deflate (9)  | 1996-12-24 11:32 PM  | Compatibility, historic popularity                                                                      | Weak compression ratio compared to ZSTD and LZMA                                          | Cross platform compatibility, direct playability for many emulators                                                      |
| {{ :set:ziptdc.png?50&nolink |tdc zip}}                   | TDC Zip               | Deflate (9)  | Specified by DAT     | A new normalized standard that preserves dates                                                          | Archives may not match existing sources                                                   | Total DOS Collection. Automatically used with DOSCenter DATs.                                                            |
| {{ :set:zipzstd.png?50&nolink |zstd zip}}                 | ZSTD Zip              | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded                      | Not widely supported yet, but works with WinRAR and 7-zip                                 | People who want a better compression ratio and faster decompression than Deflate.                                        |
| {{ :set:sevenzipnlzma.png?50&nolink |non-solid lzma 7z}}  | Non-Solid LZMA 7z     | LZMA (9)     | Null                 | Excellent compression ratio, raw copy supported                                                         | Extremely slow, single threaded                                                           | People who want to sacrifice some compression ratio over solid LZMA for the ability to raw copy files between archives.  |
| {{ :set:sevenzipslzma.png?50&nolink |solid lzma 7z}}      | Solid LZMA 7z (RV7z)  | LZMA (9)     | Null                 | Excellent compression ratio, especially for many files packed together                                  | Extremely slow, single threaded, must unpack completely for any fix                       | People who care about compression ratio only and don't care how slow LZMA is.                                            |
| {{ :set:sevenzipnzstd.png?50&nolink |non-solid zstd 7z}}  | Non-Solid ZSTD 7z     | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded, raw copy supported  | Poorly supported, only works with zstd fork of 7-zip                                      | Default 7z structure for archives that need to be repacked in ToSort directories.                                        |
| {{ :set:sevenzipszstd.png?50&nolink |solid zstd 7z}}      | Solid ZSTD 7z         | ZSTD (19)    | Null                 | Compression ratio between Deflate and LZMA, very fast decompression, multithreaded                      | Poorly supported, only works with zstd fork of 7-zip, must unpack completely for any fix  | Archives with many small files that likely won't change often, and you want fast decompression speed.                    |


==== Which archive type should I use? ====
It depends. Each archive type has pros and cons, but you don't need to choose just one. Choose an archive type that best fits your use case for a particular collection. For example, if you plan to extract discs from archives to load into an emulator, you may prefer ZSTD Zip due its fast decompression speed. That being said, ZSTD Zip is essentially a modernized TrrntZip and seems to be the next most popular choice since it was introduced with RomVault 3.7.0. 


==== Why is LZMA so slow? ====
Although LZMA has an excellent compression ratio, it is unfortunately single-threaded. RomVault uses maximum compression for LZMA, similar to torrent7zip. Compression and decompression are simply slow. LZMA may not be the best choice unless your goal is maximum space savings for archival purposes.


==== What is ZSTD? ====
There is a lot of existing technical documentation about ZSTD, but the most important pieces of information to know are:
  * Its a modern open source compression algorithm originally developed by Facebook
  * Its designed to be highly tunable replacement for Deflate
  * Its now well established and used for filesystem compression with ZFS and BTRFS
  * Its gaining popularity and support for ZSTD Zip archives and CHDs was added to MAME v0.262 in January 2024 
  * It supports multi-threaded compression
  * Its standard levels are 1-19, and ultra levels are 20-22 which require significantly more memory
  * Its updated periodically, but RomVault uses ZSTD version 1.5.5, which will not change for the foreseeable future in order to ensure deterministic behavior


==== How does the compression ratio of ZSTD compare to Deflate and LZMA? ====
ZSTD level 19 has a compression ratio right in between Deflate level 9 and LZMA level 9. However, since ZSTD is multi-threaded its much faster to compress compared to LZMA and often faster than Deflate. ZSTD also decompresses much faster than both LZMA and Deflate. Example statistics for the redump.org PlayStation collection:

^ Format         ^  Size (GiB)  ^  Compression Ratio  ^  Space Savings  ^
| File           |  4,640.41    |                     |                 |
| TrrntZip       |  3,025.91    |  1.5336             |  34.79%         |
| ZSTD Zip       |  2,716.79    |  1.7080             |  41.45%         |
| Solid LZMA 7z  |  2,513.54    |  1.8462             |  45.83%         |

The diagram below shows the relationship of single-threaded compression speed and ratio. Most ZSTD use cases would involve multiple workers (threads) on modern hardware and compress much faster. The three callouts in the diagram represent the compression levels that RomVault uses for each method.  
{{:diagrams:zstd.png?nolink&1000|}}
==== Why is ZSTD slow when compressing archives with many smaller files? ====
RomVault's implementation of ZSTD uses compression level 19 without any additional parameters or long distance matching. ZSTD is multi-threaded but each thread or worker is assigned a job of a specific size. ZSTD level 19 uses a job size of 32 MiB. This means any compression stream under 32 MiB will only use a single worker. Conversely a stream of 500 MiB will use up to 16 workers simultaneously.


==== Why does RomVault use ZSTD level 19 instead of 22? Why can't I choose my own? ====
There are several reasons:
  * RomVault has always focused on consistency and normalization, and purposely provides a limited set of the options that best accommodates the use cases of the community.
  * Space savings are marginal above level 19 for most use cases.
  * Levels 20-22 are considered "ultra" and have much higher memory requirements for compression which may not work for many users. (~3 GiB for a //single// worker with level 22)
  * Levels above 19 compress significantly slower due to the increased window size.
  * Levels above 19 are not as performant in most multithreaded scenarios since the job size per worker is much larger. (128 MiB to 512 MiB)


==== What's the difference between the ZSTD levels? ====
RomVault uses ZSTD level 19, which you cannot change. The details of the ZSTD levels are outlined below:

  * Window Log: The window size represented as an exponent, 2^X bytes.
  * Window Size: The size of the sliding window the algorithm uses. Larger window sizes can match patterns further away, but at the cost of higher memory utilization and speed.
  * Job Size: The amount of data allocated to a single worker (thread).
  * Strategy: The specific compression approach used.

^ Level     ^ Window Log  ^ Window Size  ^ Job Size  ^ Strategy       ^
^ Standard                                                        |||||
| 1         | 19          | 512 KiB      | 2 MiB     | ZSTD_fast      |
| 2         | 19          | 512 KiB      | 2 MiB     | ZSTD_fast      |
| 3         | 20          | 1 MiB        | 4 MiB     | ZSTD_dfast     |
| 4         | 21          | 2 MiB        | 8 MiB     | ZSTD_dfast     |
| 5         | 21          | 2 MiB        | 8 MiB     | ZSTD_greedy    |
| 6         | 21          | 2 MiB        | 8 MiB     | ZSTD_lazy      |
| 7         | 21          | 2 MiB        | 8 MiB     | ZSTD_lazy      |
| 8         | 21          | 2 MiB        | 8 MiB     | ZSTD_lazy2     |
| 9         | 22          | 4 MiB        | 16 MiB    | ZSTD_lazy2     |
| 10        | 22          | 4 MiB        | 16 MiB    | ZSTD_lazy2     |
| 11        | 22          | 4 MiB        | 16 MiB    | ZSTD_lazy2     |
| 12        | 22          | 4 MiB        | 16 MiB    | ZSTD_lazy2     |
| 13        | 22          | 4 MiB        | 16 MiB    | ZSTD_btlazy2   |
| 14        | 22          | 4 MiB        | 16 MiB    | ZSTD_btlazy2   |
| 15        | 22          | 4 MiB        | 16 MiB    | ZSTD_btlazy2   |
| 16        | 22          | 4 MiB        | 16 MiB    | ZSTD_btopt     |
| 17        | 23          | 8 MiB        | 32 MiB    | ZSTD_btopt     |
| 18        | 23          | 8 MiB        | 32 MiB    | ZSTD_btultra   |
| 19        | 23          | 8 MiB        | 32 MiB    | ZSTD_btultra2  |
^ Ultra                                                           |||||
| 20        | 25          | 32 MiB       | 128 MiB   | ZSTD_btultra2  |
| 21        | 26          | 64 MiB       | 256 MiB   | ZSTD_btultra2  |
| 22        | 27          | 128 MiB      | 512 MiB   | ZSTD_btultra2  |