Network Security Services (NSS) is Mozilla’s widely used, cross-platform cryptography library. When you verify an ASN.1 encoded digital signature, NSS will create a VFYContext structure to store the necessary data. This includes things like the public key, the hash algorithm, and the signature itself.
The maximum size signature that this structure can handle is whatever the largest union member is, in this case that’s RSA at 2048 bytes. That’s 16384 bits, large enough to accommodate signatures from even the most ridiculously oversized keys.
Okay, but what happens if you just….make a signature that’s bigger than that?
Well, it turns out the answer is memory corruption. Yes, really.
The untrusted signature is simply copied into this fixed-sized buffer, overwriting adjacent members with arbitrary attacker-controlled data.Source: Project Zero