Profile-Guided Optimization (PGO) benchmark results

[zamazan4ik]

Hi!

I evaluate Profile-Guided Optimization (PGO) performance improvements for applications (and libraries) in different software domains - all current results can be found here. According to the tests, PGO helps to improve performance in various software domains. I decided to perform PGO benchmarks on the sqlparser-rs library too since some library users can be interested in improving the library performance. Here are the results.

Test environment

• Fedora 39
• Linux kernel 6.7.4
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.76
• sqlparser-rs version: the latest for now from the main branch on commit 5da66adda99d8cf7a36c1512d41815d94ce542b5

Benchmark

For benchmark purposes, I use sqlparser-bench benchmark. Release measurements are done with cargo bench, PGO optimized with cargo pgo optimize bench, PGO training phase - with cargo pgo bench. All PGO optimizations are done with cargo-pgo.

Results

I got the following results:

• Release: https://gist.github.com/zamazan4ik/6ca7797580539b20ba40eed70ca36d90
• PGO-optimized: https://gist.github.com/zamazan4ik/2c008c45349a4ff8cf220bbeeb50bb55
• (just for reference) PGO instrumentation: https://gist.github.com/zamazan4ik/ac1f534b3db49c232b04aa222dff29db

At least in the provided by project benchmarks, there are measurable improvements.

Maybe mentioning these results somewhere in the README (or any other user-facing documentation) can be a good idea. Also, I think it will be worth performing benchmarks in more scenarios.

Please do not treat the issue as a bug or something like that - it's just a benchmark report.

[Dandandan]

That's really awesome @zamazan4ik. Thank you for sharing the results of your experiment. Looking at the results, it looks like you can expect about 15-20% improvement.

If you're looking for other projects to test PGO on, let me suggest to have a look at
https://github.com/apache/arrow-datafusion . This is a query engine (also using sqlparser-rs) where performance is critical.

[zamazan4ik]

Thank you for the suggestion! Just added arrow-datafusion to my PGO-TODO list ;)