diff --git a/results/images/CpuCoreMedium.png b/results/images/CpuCoreMedium.png new file mode 100644 index 0000000..69c4700 Binary files /dev/null and b/results/images/CpuCoreMedium.png differ diff --git a/results/images/CpuCoreSmall.png b/results/images/CpuCoreSmall.png new file mode 100644 index 0000000..25fe8e4 Binary files /dev/null and b/results/images/CpuCoreSmall.png differ diff --git a/results/images/CpuTypeMedium.png b/results/images/CpuTypeMedium.png new file mode 100644 index 0000000..e6867f4 Binary files /dev/null and b/results/images/CpuTypeMedium.png differ diff --git a/results/images/CpuTypeSmall.png b/results/images/CpuTypeSmall.png new file mode 100644 index 0000000..a9e079f Binary files /dev/null and b/results/images/CpuTypeSmall.png differ diff --git a/results/images/OsTypeMedium.png b/results/images/OsTypeMedium.png new file mode 100644 index 0000000..cdc5611 Binary files /dev/null and b/results/images/OsTypeMedium.png differ diff --git a/results/images/OsTypeSmall.png b/results/images/OsTypeSmall.png new file mode 100644 index 0000000..dbf59fd Binary files /dev/null and b/results/images/OsTypeSmall.png differ diff --git a/results/images/TotalRamMedium.png b/results/images/TotalRamMedium.png new file mode 100644 index 0000000..f6a1e54 Binary files /dev/null and b/results/images/TotalRamMedium.png differ diff --git a/results/images/TotalRamSmall.png b/results/images/TotalRamSmall.png new file mode 100644 index 0000000..2f478f6 Binary files /dev/null and b/results/images/TotalRamSmall.png differ diff --git a/results/post-quantum-testing-results.md b/results/post-quantum-testing-results.md new file mode 100644 index 0000000..08c11d1 --- /dev/null +++ b/results/post-quantum-testing-results.md @@ -0,0 +1,269 @@ +# Post Quantum Testing Results + +# Problem Threshold + +To determine if a post-quantum signature algorithm is a viable replacement for our current algorithms, we will need to evaluate them across our criteria: + +* Client-side signature verification speed +* Server-side signature generation speed +* Increased cost + +## Performance Criteria + +* **Signature Verification (Client-side):** Our main concern is users that are on older hardware, that is \>10 years old, which may have performance issues with quantum-safe verification. We have to be certain that the verification performance across various hardware doesn’t have a steep performance drop compared to our currently used algorithms. + +* **Signature Generation (Server-side):** For signature generation we want to ensure that our signing operations don’t take significantly longer compared to current algorithms. We will test signing both sequentially and in parallel to mimic a real server side scenario. + +## Candidate Algorithms + +We want to evaluate the performance of our post-quantum algorithm choices and compare them against our current algorithms for both code and content signing. The algorithms chosen were the ones with the most efficient combination of bandwidth usage and verification speed. See our comparison table here for details: [Autograph PQ Algorithm Specs](./pq-algorithm-specs.csv) + +| Use Case | Current Algorithm | Post-quantum Algorithm | +| :---- | :---- | :---- | +| **Content Signing** | ECDSA-384 | FN-DSA-512 (Falcon) | +| **Code signing** | RSA-4096 | ML-DSA-65 (Dilithium) | + +# Conclusions + +## ML-DSA-65 + +**Recommendation:** We should switch to ML-DSA for code signing because it is already well supported and optimized for most existing hardware. Its performance and cost are also well within the acceptable range. + +* **Client-side:** ML-DSA and RSA have equivalent verification speed for small payloads (4B), but ML-DSA is slower by 45% on medium payloads (2MB). ML-DSA is only slower than RSA on older hardware that is over 10 years old. **This difference is negligible**, an extra 2ms wouldn’t affect our code signing verification time since the operation occurs infrequently. +* **Server-side:** For signature generation, small payload ML-DSA is faster than RSA by 40%, while medium payload ML-DSA is faster by 10%. +* **Cost:** Due to larger certificate and signature sizes, this could increase the bandwidth needed for signatures by 6-7x. This would be a significant amount for content signing, where data can change daily. But for code signing this is less of an impact. +* **Conclusion:** Switching to ML-DSA for code signing will add post-quantum security to autograph, in exchange for negligible user impact and slightly increased costs. + +## FN-DSA-512 + +**Recommendation:** We should switch to FN-DSA for content signing and root/intermediate certificates because it is expected to receive [NIST approval soon]((https://csrc.nist.gov/projects/post-quantum-cryptography/post-quantum-cryptography-standardization/selected-algorithms)), and has good performance. + +* **Client-side:** FN-DSA has around 20x faster signature verification speed than ECDSA on small payloads. For medium payloads, FN-DSA is 44% slower than ECDSA, mainly on hardware that is over 10 years old. This difference is very small (around 2ms), so it will not affect users. +* **Server-side:** For signature generation, small payload FN-DSA has equivalent speeds to ECDSA, but for medium payload FN-DSA is 2x faster than ECDSA. Thus, our server side signing will have a positive impact on its speed. +* **Cost:** Our weekly bandwidth for content signing goes from 3TB to 17TB, which will not increase costs significantly. +* **Conclusion:** Switching to FN-DSA for content signing will add post-quantum security for Autograph, but will increase costs and add a bit of verification latency to older devices. + +# General Performance Analysis + +## Signature Verification Average Time Per Operation + +| Algorithm | Small Payload (4 bytes) | Medium Payload (2MB) | +| :---- | :---- | :---- | +| RSA-4096 | 0.1628 ms | 4.9700 ms | +| ML-DSA-65 | 0.1643 ms | 7.1814 ms | +| ECDSA-384 | 0.9736 ms | 4.9682 ms | +| FN-DSA-512 | 0.0545 ms | 7.1921 ms | + +From this table, we can tell that FN-DSA is around 20x faster than ECDSA for small payloads. On the other hand, FN-DSA is slower by 45% for medium payload, which will slow down the verification process by at most few milliseconds. + +For the code signing we compare the average signing time for RSA and ML-DSA. In this case, ML-DSA is 20x faster than RSA for small payloads and 44% slower than RSA for medium payloads. The difference between medium payloads is 2ms which shouldn’t negatively affect our users for code signing less than 2MB. Follow-up work will be conducted to test larger code signing requests. + +## Signature Generation Average Time Per Operation + +These are the results after running our server-side signature generation testing program in parallel on GCP servers. This gets rid of any authorization overhead and gives us the closest comparison to production level signing. + +```shell + +Small Payload FN-DSA-512: 0.053 ms +Small Payload ECDSA-384: 0.061 ms +Small Payload RSA-4096: 1.751 ms +Small Payload ML-DSA-65: 1.272 ms + +Medium Payload FN-DSA-512: 0.783 ms +Medium Payload ECDSA-384: 1.254 ms +Medium Payload RSA-4096: 1.483 ms +Medium Payload ML-DSA-65: 1.386 ms +``` + +For content signing, we can see that FN-DSA has similar signature generation time to ECDSA for small payloads. FN-DSA is around 40% faster than ECDSA for medium payloads as well. Switching to FN-DSA will not negatively affect signature generation latency for Autograph. + +For code signing, ML-DSA is 50% faster for smaller payloads compared to RSA, and ML-DSA is slightly faster for medium payloads as well. Thus, ML-DSA is an improvement over RSA for code signing and will increase signing speeds. + +In conclusion, we should switch to our post-quantum candidates because they have better signature generation speeds than our current algorithms. + +# Detailed Performance Analysis + +## Signature Verification Testing Graphs + +Note: FN-DSA-512 is the NIST name of FALCON-512 + +### Algorithm Performance \- CPU Cores + +![][CpuCoreSmall] +![][CpuCoreMedium] +The data in the graph shows the trend that a greater CPU core count leads to improved verification performance for both payload types. However, we have an outlier where the 8 core is slower than 6 core CPU. This devation occures because the 8 core CPU data is solely based on the AMD FX-8300 which is based on dated architecture and is over a decade old. This processor is unable to use modern instruction sets (AVX2) which bottlenecks the performance. From this we can conclude that signature verification with our library ([OQS](https://github.com/open-quantum-safe/liboqs)) has improved performance with a higher core CPU. + +31% of our users are on 4 core CPUs and 28% are on 2 cores ([link](https://data.firefox.com/dashboard/hardware)). Our graphs show slow verification performance for 2 core CPUs because they are generally old and have outdated architecture, leading to slow single core performance. The actual verification performance drop is around 6ms which is not concerning for us. + +### Algorithm Performance \- Total RAM + +![][TotalRamSmall] +![][TotalRamMedium] +For both payload types, having more RAM doesn’t guarantee better signature verification speeds. For all 4 algorithms, machines with 16GB of ram outperformed those with 24GB. This shows that the algorithm performance isn’t affected by memory capacity and instead depends on if the CPU can pull data out of the RAM efficiently and keep its cache fed. There are a few hardware issues that can bottleneck the performance. +**Hardware Limitations:** + +* Running memory in single-channel mode due to non-matching ram sticks +* BIOS configuration errors +* Undetected faulty memory stick + +When we look at user data, 11% of users have 4GB RAM and 30% have 8GB ([link](https://data.firefox.com/dashboard/hardware)), which means they will be most impacted when we switch to post-quantum algorithms. But, the actual performance difference is within 2-4ms which will be negligible for real world signature verification. Therefore, RAM is not a hardware aspect that we should be concerned about. + +### Algorithm Performance \- Operating System + +![][OsTypeSmall] +![][OsTypeMedium] +From this graph we can see that there is no noticeable performance degradation in a specific operating system that we tested: MacOS, Windows, Linux. The variation in performance between them is mainly due to the hardware constraints of the machine. The 3 operating systems we tested make up 92% of [users](https://data.firefox.com/dashboard/hardware). + +### Algorithm Performance \- CPU Model + +| CPU Model (Older) | Release Year | +| :---- | :---- | +| Intel Core i5-2520M | 2011 | +| Intel Core i7-2670QM | 2011 | +| Intel Core i7-3667U | 2012 | +| AMD FX-8300 | 2012 | +| Intel Core i5-4200M | 2013 | +| Intel Core i7-4790K | 2014 | +| Intel Core i7-7700HQ | 2017 | +| AMD Ryzen 7 1700 | 2017 | +| Intel Core i5-8250U | 2017 | +| Apple M1 Max | 2021 | +| AMD Ryzen 5 5600 | 2022 | +| AMD Ryzen 7 7800X3D | 2023 | +| AMD Ryzen 5 7640U | 2023 | +| Apple M3 Pro | 2023 | + +![][CpuTypeSmall] +![][CpuTypeMedium] +In the CPU Model graphs, we see that modern CPUs consistently perform better than older generation CPUs. In our tests, CPUs that were older than 10 years have performance that is at least 2x slower. This is due to architectural constraints, with some processors having shared FPUs (like the AMD fx-8300) which can slow down signature verification. This combined with low core counts and poor single core performance leads to older generation CPUs performing significantly worse. + +Although older CPUs perform slower, there is at most a 7ms difference in verification speed which will be negligible for our users. Based on this data, we can conclude that switching to the post-quantum algorithms ML-DSA and FN-DSA will not significantly degrade the performance for our end users. + +## Small (4B) vs Medium (2MB) Payload + +In the small payload graphs, we can observe that ECDSA is extremely slow (\>3x) compared to any of the other algorithms. If we want to switch to a better algorithm for small payloads then FN-DSA is the clear choice, as it is 20x faster than ECDSA which is used for small-payload content-signing. For code signing, ML-DSA has similar verification performance to RSA so the performance will stay the same. In the context of small payloads we should switch because the post-quantum algorithms provide greater security and performance than our current algorithms, so there is no downside to switching. + +When we look at medium payloads we see a slight drop in performance as FN-DSA and ML-DSA are 45% slower than ECDSA and RSA respectively. Even though 45% is a significant amount, the real world time difference is 2ms which is negligible. Thus, even for medium payloads it is best to switch to post-quantum signature algorithms for increased security even at the small cost of performance. + +## Signature Generation Testing Results + +We tested our server side signature generation program to see how fast we can generate signatures and compare the post quantum algorithm signing times to current algorithms. + +Note that the ECDSA-384 and FN-DSA-512 algorithms are used for content signing, and due to a lack of a KMS implementation for FN-DSA we decided to implement both of them locally to get a better comparison. RSA-4096 and ML-DSA-65 are used for code signing and both were implemented with Google KMS. + +## Signing Operations on Local Build + +The local build signing operations were tested on a MacOS M3 Pro device with 36GB RAM. + +### Sequential Tests + +The program runs 100 signing operations sequentially, and returns the average time per operation. + +```shell +Small Payload Falcon-512: 0.125 ms +Small Payload ECDSA-384: 0.135 ms +Small Payload RSA-4096: 406.587 ms +Small Payload ML-DSA-65: 391.105 ms + +Medium Payload Falcon-512: 1.565 ms +Medium Payload ECDSA-384: 1.570 ms +Medium Payload RSA-4096: 400.628 ms +Medium Payload ML-DSA-65: 369.019 ms +``` + +### Parallel Tests + +The program ran the signing operations for each algorithm in parallel using Goroutines and waitgroups. This test was to find out how performant the signing is when the hardware is under load. We took the average time per operation of 100 tests running in parallel. + +```shell +Small Payload FN-DSA-512: 0.021 ms +Small Payload ECDSA-384: 0.019 ms +Small Payload RSA-4096: 5.601 ms +Small Payload ML-DSA-65: 6.443 ms + +Medium Payload FN-DSA-512: 0.452 ms +Medium Payload ECDSA-384: 0.311 ms +Medium Payload RSA-4096: 5.722 ms +Medium Payload ML-DSA-65: 6.530 ms +``` + +## Signing Operations on GCP + +This was tested on GCP and was done to verify if there is network latency or authorization overhead that alters the test results. + +### Sequential Tests + +The program runs 100 signing operations sequentially, and returns the average time per operation. + +```shell +Small Payload Falcon-512: 0.247 ms +Small Payload ECDSA-384: 0.245 ms +Small Payload RSA-4096: 40.807 ms +Small Payload ML-DSA-65: 26.594 ms + +Medium Payload Falcon-512: 3.240 ms +Medium Payload ECDSA-384: 4.653 ms +Medium Payload RSA-4096: 30.518 ms +Medium Payload ML-DSA-65: 24.872 ms +``` + +### Parallel Tests + +The program ran all the signing operations for each algorithm in parallel using Goroutines and waitgroups. This test was to find out how performant the signing is when the hardware is under load. We took the average time per operation of 100 tests running in parallel. + +```shell +Small Payload FN-DSA-512: 0.053 ms +Small Payload ECDSA-384: 0.061 ms +Small Payload RSA-4096: 1.751 ms +Small Payload ML-DSA-65: 1.272 ms + +Medium Payload FN-DSA-512: 0.783 ms +Medium Payload ECDSA-384: 1.254 ms +Medium Payload RSA-4096: 1.483 ms +Medium Payload ML-DSA-65: 1.386 ms +``` + +From the signature generation testing results, we can conclude that the post quantum algorithms outperform our current algorithms in most cases. In the other case, the performance difference is \~1 ms which is negligible for our use case. Thus, our chosen algorithms will not be problematic for signature generation. + +## Conclusion + +We should switch to our candidate post-quantum signature algorithms ML-DSA and FN-DSA because they provide enhanced security against quantum computers. Our verification speed will go up around 5ms for older hardware but this difference is negligible and will not affect the performance of our end users. We also saw improvements running ML-DSA and RSA in GCP compared to a local build, which shows that there is some authorization overhead while running signing operations locally, whereas our local implementation of FN-DSA and ECDSA showed no difference. + +# Limitations + +## Hardware Age and Architecture + +We have noticed that hardware with older architecture (around 10+ years) performs significantly worse for signature verification than modern hardware. Referring to the CPU age graph, there is a drop in CPU performance for the processors that are older than 2017. This drop in performance is due to architectural constraints such as shared FPUs like the AMD fx-8300 and lack of modern instruction sets like AVX2, which our post quantum algorithms rely on to speed up the verification process. + +## Mobile Testing + +We have decided to disregard mobile testing because phones have a shorter life span than computers and people will tend to change them more often. Very few users will be running a phone older than currently sold models in 2029 when we are expecting to make this switch. + +## Hardware Comparisons + +Our testing data for different hardware is biased because it is tested with different measurements. For example, when we compare 16GB vs 32GB RAM performance, the 32 GB machine might have a less powerful CPU which will cause the comparison to favour the 16GB machine. We will follow-up on this by testing algorithms with emulated hardware to isolate variables. + +## Testing Very Large Payloads + +Large payload signing happens every few weeks, so a little variation in time will not affect our users. However, the max request size can be as high as [1.5GB](https://github.com/mozilla-services/autograph/blob/43e669d2fa1b734f0de88019ca4db657595e4b7e/handlers.go#L130). We will test the larger payload sizes for code-signing. + +# Future Considerations + +## Further testing + +We would want to further test the hardware more uniformly to reduce bias in our testing results. For example, we would want to test different RAM amounts on a machine with the same specifications (CPU, OS, Cores) to reduce inaccuracies. We could use virtual machines to isolate variables, so the virtual hardware would change but the underlying silicon would stay consistent between runs. This would eliminate architecture oddities from test variables while keeping instruction set limitations and CPU frequency the same. + +# References + +* Post Quantum Research Document: [Autograph Post Quantum Research ](./pq-algorithm-research.md) +* Post Quantum Algorithm Spreadsheet: [Autograph PQ Algorithm Specs](./pq-algorithm-specs.csv) +* Firefox Desktop Hardware Survey: [https://data.firefox.com/dashboard/hardware](https://data.firefox.com/dashboard/hardware) + +[CpuCoreSmall]: ./images/CpuCoreSmall.png +[CpuCoreMedium]: ./images/CpuCoreMedium.png +[TotalRamSmall]: ./images/TotalRamSmall.png +[TotalRamMedium]: ./images/TotalRamMedium.png +[OsTypeSmall]: ./images/OsTypeSmall.png +[OsTypeMedium]: ./images/OsTypeMedium.png +[CpuTypeSmall]: ./images/CpuTypeSmall.png +[CpuTypeMedium]: ./images/CpuTypeMedium.png diff --git a/results/pq-algorithm-research.md b/results/pq-algorithm-research.md new file mode 100644 index 0000000..78c9abc --- /dev/null +++ b/results/pq-algorithm-research.md @@ -0,0 +1,246 @@ +# Autograph \- Post Quantum Research + +# Research + +Below is the research process to help us determine which post quantum signature algorithms provide the best specifications for our use case. + +## Metrics to Measure + +Some of the key metrics that we will use are public keys and signature sizes along with the NIST security category to find comparable algorithms that support our code and content signing needs. + +Private Key size is not an important deciding factor since it will be stored by Autograph and will not affect implementation cost or complexity. + +### Weekly Estimated Bandwidth + +1. Assume 500M active users each week + 1. This is more than our actual weekly active users number because of: + 1. Automation accounts that will hit remote-settings prod + 2. Cloned profiles + 3. Users that have telemetry disabled but still query remote settings + 4. Firefox forks that have telemetry disabled but query remote-settings +2. For each user, the public key will need to be downloaded once +3. For each user, a signature will need to be downloaded for each content update + 1. Let’s estimate this at 50 times per week per user. Some weeks will be higher, some weeks lower. Some collections get updated multiple times per week + +The weekly estimated bandwidth (TB) will help us understand the cost of using the algorithms, and will be a significant deciding factor. The lower bandwidth will indicate the corresponding algorithm is more **cost efficient**. + +The Weekly Verification Performance (Cycles) metric will help us understand the relative speed of the algorithms and will assist us in finding the most **time efficient** algorithm. The cycles (Cycles) are clock cycles on x86 architecture with AVX2. + +## Limitations to Consider + +We need to consider the support provided to Post Quantum algorithms by our offline HSM Enstrust nShield F3 USB. As of Sep 22, 2025, the offline HSM in Toronto has been updated to Security World 13.9.0 and supports FIPS-204 ML-DSA \[[link](https://nshielddocs.entrust.com/security-world-docs/release-notes/release-notes.html#mldsa-pkcs11)\], FIPS-203 ML-KEM \[[link](https://nshielddocs.entrust.com/security-world-docs/release-notes/release-notes.html#_ml_kem_post_quantum_algorithm_firmware_support_nse_48335)\], and FIPS-205 SLH-DSA \[[link](https://nshielddocs.entrust.com/security-world-docs/release-notes/release-notes.html#_slh_dsa_post_quantum_algorithm_firmware_support_nse_48338)\]. + +The support provided by Google KMS needs to be considered as well \[[link](https://cloud.google.com/kms/docs/algorithms)\]. Some ML-DSA and SLH-DSA algorithms are in pre-GA right now, and may be subject to change. + +## Algorithms Types + +We start off by going through and researching the major post quantum algorithm types and ruling out the types that are not favourable for Autograph. Note that we will only be focusing on the differences that will affect Mozilla’s Autograph. For additional details, this [article](https://en.wikipedia.org/wiki/Post-quantum_cryptography#Algorithms) contains information on the various types of algorithms. + +1. Lattice-based + 1. Algorithms + 1. FALCON + 1. Falcon has a better combination of lower bandwidth, faster performance, and scales better than most other algorithms at security category 1 and 5 + 2. Has a small signature and key size + 3. Relies on floating point arithmetic, which is not favourable since floating point arithmetic is handled [differently](https://falcon-sign.info/falcon.pdf#page=20) across various types of hardware and can cause inconsistencies/security [issues](https://eprint.iacr.org/2024/1709.pdf#page=2) + 2. DILITHIUM (ML-DSA) + 1. First standardized lattice based post quantum (PQ) signature with slightly larger key sizes than standard RSA algorithms, but secure even against large quantum computers + 3. Hawk + 1. Similar to Falcon in terms of key \+ signature sizes + 2. Does not rely on floating point arithmetic, meaning that we will have consistent results even across various hardware \[[link](https://hawk-sign.info/)\] + 2. Most of the lattice-based algorithms are efficient and are standardized +2. Multivariate + 1. Algorithms + 1. Rainbow ([Unbalanced Oil and Vinegar](https://www.uovsig.org/)) + 1. Rainbow algorithm has been ruled out since some security categories have been vulnerable to [attacks](https://csrc.nist.gov/csrc/media/Projects/post-quantum-cryptography/documents/round-3/official-comments/RAINBOW-round3-official-comment.pdf) + 2. MAYO + 1. Offers a middle security category choice, good [specs](https://openquantumsafe.org/liboqs/algorithms/sig/mayo.html) in terms of signature size (0.4kb-1kb) and public key size (1kb-5kb) +3. Hash-based + 1. These algorithms use hash functions to ensure quantum safe signing + 1. Hash functions transform data to code that cannot be deciphered, it is a one-way process + 2. Algorithms + 1. SPHINCS+ (SLH-DSA) standardized algorithm + 1. Has larger signature and keys sizes, making them less efficient and more computationally expensive than lattice-based \[[link](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.205.pdf#page=53)\] + 2. The bandwidth requirements are very large and range from 200TB-700TB which will be costly + 2. XMSS and LMS \- hash based stateful algorithms + 1. Complex implementation for autograph service, stateful algorithms rely on state management so we would need to track the number of keys + 2. If state is not properly tracked, it could lead to security breaches \[[link](https://blog.cloudflare.com/another-look-at-pq-signatures/)\] +4. Code-based + 1. Algorithms + 1. CROSS + 1. Has very large key sizes (100kB to 1mB for secure keys) + 2. Will not be viable for autograph post-quantum due to extremely [large](https://openquantumsafe.org/liboqs/algorithms/sig/cross.html) signature sizes for all security categories + 2. It will be better to rule this type out due to bandwidth limitations +5. Isogeny-based + 1. Algorithms + 1. SQIsign + 1. Small key and signature sizes, outperforming other PQ algorithms \[[link](https://sqisign.org/)\] + 2. Has higher signing and verification [times](https://blog.cloudflare.com/another-look-at-pq-signatures/) compared to ML-DSA, it is up to 143 times slower + 3. Computationally expensive to verify the signatures + 2. It will be better to rule this type out due to client performance +6. Symmetric key quantum resistance + 1. As long as the key size is sufficiently large, it will be quantum safe + 1. Moving from 128 to 256 bytes reduces security risks + 2. Not sufficient for our use case since public keys are required for signature algorithms + + +We will first rule out Code-based types (including CROSS) due to the high weekly bandwidth (300TB-1.3TB) and slow verification times (600K-2M cycles) which will increase costs \[[link](https://openquantumsafe.org/liboqs/algorithms/sig/cross.html)\]. + +The Isogeny-based type can also be ruled out due to the extensive signing and verification times of up to 143 times [slower](https://blog.cloudflare.com/another-look-at-pq-signatures/) than ML-DSA which can slow down autograph performance. + +The SPHINCS+ algorithm will also be ruled out for high weekly bandwidth and slow verification times compared to other algorithms, thereby increasing the cost. + +We will rule out XMSS and LMS because they are stateful algorithms, requiring us to manage state by making sure our one time keys are tracked and never reused which can lead to [security vulnerabilities](https://blog.cloudflare.com/another-look-at-pq-signatures/). + +We proceed with ML-DSA, Falcon, Hawk, and MAYO. These algorithms are well documented with relatively fast signing and verification time, along with a good balance between key sizes and security categories that we can look into. + +## Algorithm Specifications + +Post-Quantum specification sheet: [Autograph PQ Algorithm Specs](./pq-algorithm-specs.csv) + +After looking at the specifications, we can break down the advantages and disadvantages of the shortlisted algorithms ML-DSA, Falcon, Hawk, MAYO. + +1. ML-DSA + 1. Pros + 1. Has a middle security category (level 3\) which has experimental support in [Google KMS](https://cloud.google.com/kms/docs/algorithms) + 2. Has experimental offline HSM support + 3. Has the fastest verification performance compared to the other algorithms for each security category + 2. Cons + 1. Has the largest weekly bandwidth out of all the shortlisted algorithms (60TB-116TB) \[[link](./pq-algorithm-specs.csv)\] +2. Falcon + 1. Pros + 1. Second lowest weekly bandwidth usage + 2. Standardized by NIST, possible Google KMS support in future + 3. Fast verification performance + 2. Cons + 1. Not supported by [Google KMS](https://cloud.google.com/kms/docs/algorithms) at this time + 2. Relies on floating point arithmetic +3. Hawk + 1. Pros + 1. Weekly bandwidth very low, similar to Falcon + 2. Does not rely on floating point arithmetic, performance will be similar across hardware + 2. Cons + 1. Not standardized + 2. Verification performance around 45% slower than Falcon +4. MAYO + 1. Pros + 1. Offers the lowest weekly bandwidth + 2. Has a middle security category (MAYO-3) + 2. Cons + 1. Verification performance slowest out of all shortlisted algorithms (3 times slower than ML-DSA) + 2. Not supported by Google KMS at this time + 3. Similar multivariate types like Rainbow (UOV scheme) have known security [vulnerabilities](https://csrc.nist.gov/csrc/media/Projects/post-quantum-cryptography/documents/round-3/official-comments/RAINBOW-round3-official-comment.pdf) + +## Signature Algorithm Security + +### Security Category + +For our selected algorithms we need to decide on a good balance between the security category and performance/bandwidth. [Security Categories](https://csrc.nist.gov/glossary/term/security_category) are a number associated with the security strength of a post-quantum cryptographic algorithm, as specified by NIST. Categories range from 1-5, and a higher security Category indicates a more secure algorithm. + +We need to decide on a security category that is at least equivalent to the ones that we are currently using in our Signature Algorithms. We currently use ECDSA-384 for Content signing and RSA-4096 for code signing. These provide security categories [equivalent](https://postquantum.com/post-quantum/nist-pqc-security-categories/) to NIST category 3, thus we should at least aim for category 3 security to retain a comparable amount of security. + +Security category 4 and 5 are better suited as [future proofing](https://postquantum.com/post-quantum/nist-pqc-security-categories/) options because they have around 2 times slower verification performance compared to security category 3 implementations. + +Autograph should focus on NIST security category 3 implementations when available such as with MAYO and ML-DSA. If not available, like in HAWK and Falcon, we would need to consider the use case and how often we rotate the keys. If we need long term key storage then we should consider category 5, if it is short term we can consider category 1\. + +### Post Quantum vs Current Algorithms + +Current algorithms that are used by Autograph, like RSA and ECDSA are recommended to be deprecated by 2030 and disallowed by 2035 according to [NIST](https://nvlpubs.nist.gov/nistpubs/ir/2024/NIST.IR.8547.ipd.pdf#page=20). + +These algorithms are threatened by the advancement in quantum computing. By 2030, algorithms like RSA-2048 might be able to be [broken](https://nvlpubs.nist.gov/nistpubs/ir/2016/NIST.IR.8105.pdf#page=11) in a matter of hours by a quantum computer. In comparison, a classical computer would take up to a billion years to [break](https://www.btq.com/blog/how-far-away-is-the-quantum-threat) the same algorithm. Quantum safe algorithms are made to be resistant to quantum computer attacks and will take a long time to break unless groundbreaking advancements are made in quantum computing. + +## Code vs Content Signing + +Currently Autograph uses separate signature algorithms for code signing and content signing. + +With content signing we can rotate the content keys quickly and have short lived certificates, whereas with code signing we would like to store the keys for a longer period of time. We can apply something similar and use separate signature algorithms during our Post Quantum migration. + +Falcon and HAWK have lower weekly bandwidth \[[link](./pq-algorithm-specs.csv)\] compared to the other shortlisted algorithms, which makes them promising candidates for content signing. Since [Google KMS](https://cloud.google.com/kms/docs/algorithms) does not support these algorithms, we will have to store the keys outside of KMS. This is alright for content signing since we can rotate the keys frequently and have short lived certificates. + +For code signing, we want to store the keys for an extended period of time. For this, Google KMS and offline HSM support would be best to have so that keys can be stored securely long term. Thus, ML-DSA is best suited, as it offers experimental support for Google KMS for the NIST security category 3\. + +## Conclusion + +We can rule out HAWK because of the performance limitations compared to Falcon. HAWK has approximately equal weekly bandwidth to Falcon, but is around 45% slower in terms of verification performance. HAWK is also not standardized. Thus, it would make more sense to proceed with Falcon. + +MAYO was ruled out for code signing because it doesn't have Google KMS support, which is preferred for long term key storage. Moreover, similar [UOV](https://www.uovsig.org/) based multivariate schemes like Rainbow have known security [vulnerabilities](https://csrc.nist.gov/csrc/media/Projects/post-quantum-cryptography/documents/round-3/official-comments/RAINBOW-round3-official-comment.pdf), making MAYO risky to implement long term. The verification performance of MAYO-3 (category 3 security) is 340% slower than ML-DSA-65, thus we can rule out MAYO. + +The most promising algorithm for the Autograph Post Quantum code signing is the [ML-DSA-65](https://openquantumsafe.org/liboqs/algorithms/sig/ml-dsa), which is the NIST security category 3 implementation. This implementation is standardized by FIPS and receives support from Google KMS. It provides the fastest verification performance of its security category relative to other algorithms. The downside is that it requires a large weekly bandwidth (80TB). + +The best algorithm for content signing is Falcon-512, which is the NIST security category 1 implementation. The downgrade in security from a category 3 equivalent to category 1 is worth it for the performance increase, since we rotate keys frequently. It provides us with a low bandwidth (17TB) which minimizes cost, and is standardized by NIST. The verification performance and small signature size makes Falcon well suited for frequent key rotations. The downside is the lack of support from Google KMS, but we hope that it will get supported before this reaches production. Since our end-entities are short lived this would still be a non-issue if the worst happens, as we can keep our root and intermediate secured. When implementing Falcon in Autograph we need to ensure we deal with floating point inconsistencies. + +## Definitions + +### Research Stage + +No implementation found in GitHub + +### Standardized + +More documentations and implementations, submitted to NIST + +### Estimated Weekly Bandwidth + +1. Assume 500M active users each week + 1. This is more than our actual weekly active users number because of: + 1. Automation accounts that will hit remote-settings prod + 2. Cloned profiles + 3. Users that have telemetry disabled but still query remote settings + 4. Firefox forks that have telemetry disabled but query remote-settings + +2\. For each user, the public key will need to be downloaded once +3\. For each user, a signature will need to be downloaded for each collection update + +1. Let’s estimate this at 50 times per week per user. Some weeks will be higher, some weeks lower. Some collections get updated multiple times per week + +### Estimated Verification Performance + +Stated in Clock Cycles on x86 architecture with AVX2 + +### Security Category + +| Security Category | Attack Type | Example | +| :---- | ----- | ----- | +| 1 | Key search on block cipher with 128 bit key | AES-128 | +| 2 | Collision search on 256 bit hash function | SHA-256 | +| 3 | Key search on block cipher with 192 bit key | AES-192 | +| 4 | Collision search on 384 bit hash function | SHA3-384 | +| 5 | Key search on block cipher with 256 bit key | AES-256 | + +## References + +* NIST Selected Algorithms + * [https://csrc.nist.gov/Projects/post-quantum-cryptography/selected-algorithms](https://csrc.nist.gov/Projects/post-quantum-cryptography/selected-algorithms) +* FIPS 204 (ML-DSA) + * [https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.204.pdf](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.204.pdf) + * [https://csrc.nist.gov/csrc/media/Presentations/2022/high-performance-hardware-implementation-of-lattic/images-media/session-4-beckwith-high-performance-hardware-pqc2022.pdf](https://csrc.nist.gov/csrc/media/Presentations/2022/high-performance-hardware-implementation-of-lattic/images-media/session-4-beckwith-high-performance-hardware-pqc2022.pdf) +* FIPS 205 (SLH-DSA) + * [https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.205.pdf](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.205.pdf) +* FALCON + * [https://falcon-sign.info/](https://falcon-sign.info/) +* HAWK + * [https://hawk-sign.info/](https://hawk-sign.info/) +* RSA and ECDSA Specification + * [https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-78-5.pdf](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-78-5.pdf) + * [https://lf-hyperledger.atlassian.net/wiki/spaces/BESU/pages/22154986/SECP256R1+Support](https://lf-hyperledger.atlassian.net/wiki/spaces/BESU/pages/22154986/SECP256R1+Support) + * [https://nvlpubs.nist.gov/nistpubs/ir/2016/NIST.IR.8105.pdf\#page=11](https://nvlpubs.nist.gov/nistpubs/ir/2016/NIST.IR.8105.pdf#page=11) +* Comparing Hash based algorithms + * [https://eprint.iacr.org/2017/349.pdf](https://eprint.iacr.org/2017/349.pdf) +* Standardization candidates + * [https://blog.cloudflare.com/another-look-at-pq-signatures/](https://blog.cloudflare.com/another-look-at-pq-signatures/) +* Algorithm types + * [https://en.wikipedia.org/wiki/Post-quantum\_cryptography\#Algorithms](https://en.wikipedia.org/wiki/Post-quantum_cryptography#Algorithms) + * [https://www.paloaltonetworks.ca/cyberpedia/what-is-post-quantum-cryptography-pqc](https://www.paloaltonetworks.ca/cyberpedia/what-is-post-quantum-cryptography-pqc) +* Rainbow Algorithm Security Issue + * [https://csrc.nist.gov/csrc/media/Projects/post-quantum-cryptography/documents/round-3/official-comments/RAINBOW-round3-official-comment.pdf](https://csrc.nist.gov/csrc/media/Projects/post-quantum-cryptography/documents/round-3/official-comments/RAINBOW-round3-official-comment.pdf) +* Code-based algorithms + * [https://www.redhat.com/en/blog/post-quantum-cryptography-code-based-cryptography](https://www.redhat.com/en/blog/post-quantum-cryptography-code-based-cryptography) +* SQIsign + * [https://en.wikipedia.org/wiki/SQIsign](https://en.wikipedia.org/wiki/SQIsign) + * [https://sqisign.org/](https://sqisign.org/) +* Algorithm Specs + * [https://openquantumsafe.org/liboqs/algorithms/](https://openquantumsafe.org/liboqs/algorithms/) +* Estimated Performance + * [https://openquantumsafe.org/benchmarking/](https://openquantumsafe.org/benchmarking/) +* NIST Security Category + * [https://postquantum.com/post-quantum/nist-pqc-security-categories/](https://postquantum.com/post-quantum/nist-pqc-security-categories/) + * [https://nvlpubs.nist.gov/nistpubs/ir/2024/NIST.IR.8547.ipd.pdf\#page=19](https://nvlpubs.nist.gov/nistpubs/ir/2024/NIST.IR.8547.ipd.pdf#page=19) diff --git a/results/pq-algorithm-specs.csv b/results/pq-algorithm-specs.csv new file mode 100644 index 0000000..6010a2e --- /dev/null +++ b/results/pq-algorithm-specs.csv @@ -0,0 +1,42 @@ +,,,Key Sizes (in bytes),,,Estimated,,,,,,,, +Algorithm Type,Name,Parameter,Private Key,Public Key,Signature Size,Verification Performance (Cycles),Perf vs ML-DSA,Weekly Bandwidth (TB),Google KMS Support,HSM Support,Standardized,Implementation,Security Category (NIST),Notes +Lattice-Based,ML-DSA (Dilithium),ML-DSA-44,2560,1312,2420,76110,1,61.16,No,Yes,Yes,https://github.com/pq-crystals/dilithium,2, +,,ML-DSA-65,4032,1952,3309,127965,1,83.70,Yes,Yes,Yes,,3,This one seems promising as it is standardized and supported by Google KMS +,,ML-DSA-87,4896,2592,4627,206349,1,116.97,No,Yes,Yes,,5, +,,,,,,,,,,,,,, +,FALCON,FALCON-512,1281,897,666,110572,1.452792012,17.10,No,No,Yes,https://github.com/algorand/falcon,1, +,,FALCON-1024,2305,1793,1280,210549,1.020353867,32.90,No,No,Yes,,5, +,,,,,,,,,,,,,, +,HAWK,HAWK-512,184,1024,555,148224,1.947497044,14.39,No,No,No,https://github.com/hawk-sign/dev/,1,"Similar to Falcon, but doesn't require floating point arithmetic, signing time is a lot shorter" +,,HAWK-1024,360,2440,1221,302861,1.467712468,31.75,No,No,No,,5, +,,,,,,,,,,,,,, +Hash-Based,SPHINCS+,SLH-DSA-SHA2-128s,64,32,7856,645204,8.477256602,196.42,Yes,Yes,Yes,https://github.com/sphincs/sphincsplus,1, +,,SLH-DSA-SHA2-192s,96,48,16224,1739357,13.59244325,405.62,No,Yes,Yes,,3, +,,SLH-DSA-SHA2-256s,128,64,29792,2448864,11.86758356,744.83,No,Yes,Yes,,5, +,,,,,,,,,,,,,, +,XMSS,XMSS-SHA2_10_256,1373,64,2500,,,62.53,No,No,Yes,,5,It's better to not use since it is a stateful hashing-based algorithm (meaning we need to track number of signatures which increases complexity) +,,,,,,,,,,,,,, +,LMS,LMS_SHA256_H5_W1,64,60,8688,,,217.23,No,No,Yes,,5,It's better to not use since it is a stateful hashing-based algorithm (meaning we need to track number of signatures which increases complexity) +,,,,,,,,,,,,,, +Code-Based,CROSS,cross-rsdp-128-balanced,32,77,13152,1371456,18.01939298,328.84,No,No,No,https://github.com/CROSS-signature/CROSS-implementation,1, +,,cross-rsdp-192-balanced,48,115,29853,2944048,23.00666589,746.38,No,No,No,,3, +,,cross-rsdp-256-balanced,64,153,53527,5072854,24.58385551,1338.25,No,No,No,,5, +,,,,,,,,,,,,,, +Isogeny-based,SQISign,SQISign-1,353,65,148,5100000,67.00827749,3.73,No,No,No,,1, +,,SQISign-3,529,97,224,18600000,145.3522448,5.65,No,No,No,,3, +,,SQISign-5,701,129,292,35700000,173.0078653,7.36,No,No,No,,5, +,,,,,,,,,,,,,, +Multivariate,MAYO,MAYO-1,24,1420,454,188689,2.47916174,12.06,No,No,No,https://github.com/PQCMayo/MAYO-C,1,"MAYO seems interesting with the small signature size, but Public key size is very large" +,,MAYO-3,32,2986,681,434379,3.394514125,18.52,No,No,No,,3, +,,MAYO-5,40,5554,964,935674,4.534424688,26.88,No,No,No,,5, +,,,,,,,,,,,,,, +Non PQ DSA,RSA,RSA-2048,256,256,256,,0.8,6.53,,,,,, +,,RSA-3072,384,384,384,,0.8,9.79,,,,,, +,,RSA-4096,512,512,512,,0.8,13.06,,,,,, +,,,,,,,,,,,,,, +,ECDSA,ECDA P-256,32,64,64,,0.5,1.63,,,,,, +,,ECDA P-384,48,96,96,,0.5,2.45,,,,,, +,,,,,,,,,,,,,, +,,,,,,,,,,,,,, +,,,,,,,,,,,,,, +,,,,,,,,,,,,,,