benchmarking.md (4090B)
1 # Benchmarking 2 3 For speed benchmarks on single images in single or multi-threaded decoding 4 `djxl` can print decoding speed information. See `djxl --help` for details 5 on the decoding options and note that the output image is optional for 6 benchmarking purposes. 7 8 For a more comprehensive comparison of compression density between multiple 9 options, the tool `benchmark_xl` can be used (see below). 10 11 ## Benchmarking with benchmark_xl 12 13 We recommend `build/tools/benchmark_xl` as a convenient method for reading 14 images or image sequences, encoding them using various codecs (jpeg jxl png 15 webp), decoding the result, and computing objective quality metrics. An example 16 invocation is: 17 18 ```bash 19 build/tools/benchmark_xl --input "/path/*.png" --codec jxl:wombat:d1,jxl:cheetah:d2 20 ``` 21 22 Multiple comma-separated codecs are allowed. The characters after : are 23 parameters for the codec, separated by colons, in this case specifying maximum 24 target psychovisual distances of 1 and 2 (higher implies lower quality) and 25 the encoder effort (see below). Other common parameters are `r0.5` (target 26 bitrate 0.5 bits per pixel) and `q92` (quality 92, on a scale of 0-100, where 27 higher is better). The `jxl` codec supports the following additional parameters: 28 29 Speed: `lightning`, `thunder`, `falcon`, `cheetah`, `hare`, `wombat`, `squirrel`, 30 `kitten`, `tortoise` control the encoder effort in ascending order. This also 31 affects memory usage: using lower effort will typically reduce memory consumption 32 during encoding. 33 34 * `lightning` and `thunder` are fast modes useful for lossless mode (modular). 35 * `falcon` disables all of the following tools. 36 * `cheetah` enables coefficient reordering, context clustering, and heuristics 37 for selecting DCT sizes and quantization steps. 38 * `hare` enables Gaborish filtering, chroma from luma, and an initial estimate 39 of quantization steps. 40 * `wombat` enables error diffusion quantization and full DCT size selection 41 heuristics. 42 * `squirrel` (default) enables dots, patches, and spline detection, and full 43 context clustering. 44 * `kitten` optimizes the adaptive quantization for a psychovisual metric. 45 * `tortoise` enables a more thorough adaptive quantization search. 46 47 Mode: JPEG XL has two modes. The default is Var-DCT mode, which is suitable for 48 lossy compression. The other mode is Modular mode, which is suitable for lossless 49 compression. Modular mode can also do lossy compression (e.g. `jxl:m:q50`). 50 51 * `m` activates modular mode. 52 53 Other arguments to benchmark_xl include: 54 55 * `--save_compressed`: save codestreams to `output_dir`. 56 * `--save_decompressed`: save decompressed outputs to `output_dir`. 57 * `--output_extension`: selects the format used to output decoded images. 58 * `--num_threads`: number of codec instances that will independently 59 encode/decode images, or 0. 60 * `--inner_threads`: how many threads each instance should use for parallel 61 encoding/decoding, or 0. 62 * `--encode_reps`/`--decode_reps`: how many times to repeat encoding/decoding 63 each image, for more consistent measurements (we recommend 10). 64 65 The benchmark output begins with a header: 66 67 ``` 68 Compr Input Compr Compr Compr Decomp Butteraugli 69 Method Pixels Size BPP # MP/s MP/s Distance Error p norm BPP*pnorm Errors 70 ``` 71 72 `ComprMethod` lists each each comma-separated codec. `InputPixels` is the number 73 of pixels in the input image. `ComprSize` is the codestream size in bytes and 74 `ComprBPP` the bitrate. `Compr MP/s` and `Decomp MP/s` are the 75 compress/decompress throughput, in units of Megapixels/second. 76 `Butteraugli Distance` indicates the maximum psychovisual error in the decoded 77 image (larger is worse). `Error p norm` is a similar summary of the psychovisual 78 error, but closer to an average, giving less weight to small low-quality 79 regions. `BPP*pnorm` is the product of `ComprBPP` and `Error p norm`, which is a 80 figure of merit for the codec (lower is better). `Errors` is nonzero if errors 81 occurred while loading or encoding/decoding the image. 82