Benchmarks¶

Benchmarks are introduced first for developers to test certain parts of the code when modifying performance-critical code. This is done with the help of the Google Benchmark tool available at https://github.com/google/benchmark

The new benchmark has to be listed in the CMakeLists.txt file in the benchmarks folder.

Building the benchmarks¶

First, you have to add the -DBUILD_BENCHMARKS=ON to the list of CMake parameters to be able to build the benchmarks.

Running benchmarks¶

After building the bareos project, binaries for the benchmarks will be created in the build directory under build/core/src/benchmarks/.

The most obvious way to run the benchmarks would be to run the binaries directly.

Another way to run the benchmarks would be to run make benchmarks inside the build directory, which will build and run all available benchmarks, and will write benchmark results in JSON files inside the benchmarks directory (same where you would find the binaries).

As mentioned above, results can be written to a file. When running benchmarks independently, you can specify the output file path with the --benchmark_out=<filename> option, and the format with --benchmark_out_format={json|console|csv}.

Example:

Outputting the result to a json file
user@host:~$cd ~/bareosprojects/build-bareos/core/src/benchmarks/ user@host:~/bareosprojects/build-bareos/core/src/benchmarks$ ./restore_browser_stress_test --benchmark_out=restore_benchmark.json --benchmark_out_format=json
2021-06-14T13:31:03+01:00
Running ./restore_browser_stress_test
Run on (4 X 2594 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x4)
L1 Instruction 32 KiB (x4)
L2 Unified 256 KiB (x4)
L3 Unified 6144 KiB (x4)
user@host:~/bareosprojects/build-bareos/core/src/benchmarks$cat restore_benchmark.json { "context": { "date": "2021-06-14T13:31:03+01:00", "host_name": "host", "executable": "./restore_browser_stress_test", "num_cpus": 4, "mhz_per_cpu": 2594, "cpu_scaling_enabled": false, "caches": [ { "type": "Data", "level": 1, "size": 32768, "num_sharing": 1 }, { "type": "Instruction", "level": 1, "size": 32768, "num_sharing": 1 }, { "type": "Unified", "level": 2, "size": 262144, "num_sharing": 1 }, { "type": "Unified", "level": 3, "size": 6291456, "num_sharing": 1 } ], "load_avg": [0.88,0.58,0.53], "library_build_type": "release" }, "benchmarks": [ { "name": "BM_populatetree", "run_name": "BM_populatetree", "run_type": "iteration", "repetitions": 0, "repetition_index": 0, "threads": 1, "iterations": 1, "real_time": 2.6208405059000143e+01, "cpu_time": 2.6008753418000001e+01, "time_unit": "s" }, { "name": "BM_markfiles", "run_name": "BM_markfiles", "run_type": "iteration", "repetitions": 0, "repetition_index": 0, "threads": 1, "iterations": 1, "real_time": 4.0332517730003019e+00, "cpu_time": 4.0030990340000017e+00, "time_unit": "s" } ] } Comparing benchmarks¶ Google Benchmark offers a comparison script to compare different benchmarks. To use this tool, you need to clone the Google Benchmark repository, the compare.py script is located in the benchmark/tools folder. The following shell session shows basic usage of the tool (an example is for marking 10 Million files in the restore browser): Comparing the same benchmark of two different builds user@host:~$ cd ~/benchmark/tools
user@host:~/benchmark/tools$./compare.py benchmarks ~/bareosprojects/build-bareos/core/src/benchmarks/restore_browser_stress_test ~/bareosprojects/build-newcode/core/src/benchmarks/restore_browser_stress_test RUNNING: /home/user/bareosprojects/build-bareos/core/src/benchmarks/restore_browser_stress_test --benchmark_out=/tmp/tmpl3v_lavr 2021-06-14T12:24:22+01:00 Running /home/user/bareosprojects/build-bareos/core/src/benchmarks/restore_browser_stress_test Run on (4 X 2594 MHz CPU s) CPU Caches: L1 Data 32 KiB (x4) L1 Instruction 32 KiB (x4) L2 Unified 256 KiB (x4) L3 Unified 6144 KiB (x4) Load Average: 2.53, 2.74, 1.39 ---------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------- BM_populatetree 25.0 s 24.8 s 1 BM_markfiles 4.03 s 3.99 s 1 RUNNING: /home/user/bareosprojects/build-newcode/core/src/benchmarks/restore_browser_stress_test --benchmark_out=/tmp/tmpcfzev4gt 2021-06-14T12:24:54+01:00 Running /home/user/bareosprojects/build-newcode/core/src/benchmarks/restore_browser_stress_test Run on (4 X 2594 MHz CPU s) CPU Caches: L1 Data 32 KiB (x4) L1 Instruction 32 KiB (x4) L2 Unified 256 KiB (x4) L3 Unified 6144 KiB (x4) Load Average: 2.16, 2.63, 1.40 ---------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------- BM_populatetree 28.1 s 27.9 s 1 BM_markfiles 4.15 s 4.11 s 1 Comparing /home/user/bareosprojects/build-bareos/core/src/benchmarks/restore_browser_stress_test to /home/user/bareosprojects/build-newcode/core/src/benchmarks/restore_browser_stress_test Benchmark Time CPU Time Old Time New CPU Old CPU New -------------------------------------------------------------------------------------------------------------- BM_populatetree +0.1234 +0.1257 25 28 25 28 BM_markfiles +0.0297 +0.0291 4 4 4 4 What it does is for every benchmark from the first run it looks for the benchmark with the same name in the second run, and then compares the results. If the names differ, the benchmark is omitted from the diff. As you can note, the values in Time and CPU columns are calculated as (new - old) / old. The same could be done by comparing the JSON outputs of both benchmarks, or even by comparing a binary with a JSON output. Example: Comparing benchmarks with json outputs user@host:~/benchmark/tools$ ./compare.py benchmarks ~/bareosprojects/build-bareos/core/src/benchmarks/restore_benchmark.json ~/bareosprojects/build-newcode/core/src/benchmarks/restore_benchmark.json