Polars benchmarks
We don't want to improve data science privacy at the cost of your performance. This is why we have built BastionLab with speed as one of its main goals. For example, BastionLab uses Polars, a popular data science library coded in Rust, because it is far more efficient than Pandas, an even more popular one coded in Python.
Technical Specifications
To produce the following benchmarks, we compared BastionLab's performance with Polars' and Pandas' in various scenarios: BastionLab (the standard version), BastionLab run within a Trusted Execution Environement (TEE), Polars Rust (using the Lazy API which consumes less ressources), Polars Python (which is the same as Polars Rust but has python bindings; also using the Lazy API), and Pandas (the standard version).
(You can read this guide to see what TEEs are and why you could need to use BastionLab with one.)
All of the benchmarks use the same processor: AMD EPYC 7763v (with SEV-SNP disabled, except for BastionLab within a TEE which has SEV-SNP enabled).
We used a join()
operation as our baseline operation.
The memory benchmarks (memory usage) were tracked differently across Rust applications and Python applications. In Rust we used jemalloc to track memory usage and memory_profiler in Python. When comparing memory usage benchmarks, we recommend comparing Python (Polars Python against Pandas) and Rust (BastionLab against Polars Rust).
These benchmarks were performed on Azure virtual machines, the specifications of the machines (cores and memory) can be found in each benchmark table.
Benchmarking a 10Mx7 JOIN 100x5
This benchmark shows that BastionLab can perform a join operation faster than the other compared solutions - Pandas and Polars.
The mean execution times indicate that using BastionLab is comparable to using Polars. BastionLab is still a little bit more performant, being 1.14 times faster than Polars Rust and 1.19 times faster than Polars Python. BastionLab is however 7.15 times faster than Pandas!
Using BastionLab within a TEE adds a slight overhead. BastionLab (without a TEE) is 1.39 times faster than BastionLab within a TEE.
Privacy | Processor | Memory Usage (MB) | Standard deviation of Time | Mean Execution Time | Operation | Total Runs (Same Parameters) | Cores | Memory | |
---|---|---|---|---|---|---|---|---|---|
BastionLab + TEE | SEV (Encryption) + Access Control | AMD EPYC 7763v | 0.15951 | 2.29914 s | INNER JOIN | 10 | 16 | 64 GB | |
BastionLab | Access Control | AMD EPYC 7763v | 12068.93436 | 0.06813 | 1.64691 s | INNER JOIN | 10 | 16 | 64 GB |
Polars Lazy API Python | AMD EPYC 7763v | 16514.42188 | 0.05930 | 1.97007 s | INNER JOIN | 10 | 16 | 64 GB | |
Pandas | AMD EPYC 7763v | 18422.98438 | 0.54693 | 11.78010 s | INNER JOIN | 10 | 16 | 64 GB | |
Polars Lazy API Rust | AMD EPYC 7763v | 11267.53197 | 0.05090 | 1.884 s | INNER JOIN | 10 | 16 | 64 GB |
Benchmarking a 100Mx7 JOIN 100x5
When a larger join operation is performed, we see that using BastionLab remains comparable to using Polars in terms of average execution time. In this case BastionLab is 2.25 times faster than Polars Rust and 1.22 times faster than Polars Python. Here, again, BastionLab is much faster than Pandas with a 12.5 times better performance.
Running BastionLab within a TEE still presents a slight overhead, with BastionLab (without a TEE) being 1.24 times faster than BastionLab within a TEE.
Privacy | Processor | Memory Usage (MB) | Standard deviation of Time | Mean Execution Time | Operation | Total Runs (Same Parameters) | Cores | Memory | |
---|---|---|---|---|---|---|---|---|---|
BastionLab + TEE | SEV (Encryption) + Access Control | AMD EPYC 7763v | 0.07252 | 2.49139 s | INNER JOIN | 10 | 16 | 64 GB | |
BastionLab | Access Control | AMD EPYC 7763v | 22275.93819 | 0.05010 | 1.99522 s | INNER JOIN | 10 | 16 | 64 GB |
Polars Lazy API Python | AMD EPYC 7763v | 18120.21875 | 0.13787 | 2.43374 s | INNER JOIN | 10 | 16 | 64 GB | |
Pandas | AMD EPYC 7763v | 18791.93359 | 0.65590 | 24.94570 s | INNER JOIN | 10 | 16 | 64 GB | |
Polars Lazy API Rust | AMD EPYC 7763v | 14339.72344 | 0.05216 | 4.507 s | INNER JOIN | 10 | 16 | 64 GB |
To conclude, the benchmarks above show that BastionLab performs operations faster than available solutions. There is a slight overhead when using BastionLab within a TEE, but even then, it is still as fast as Polars and significantly faster than Pandas.