X509Cert.set_serial breaks for Certificate SerialNumbers with MSB=1 #7
Description
in X509Cert.java line 303, set_serial makes a call to org.bouncycastle.x509.X509V3CertificateGenerator.setSerialNumber(BigInteger), which throws an exception if given a negative number.
Sample stacktrace:
Java::JavaLang::IllegalArgumentException (serial number must be a positive integer):
org.bouncycastle.x509.X509V3CertificateGenerator.setSerialNumber(Unknown Source)
org.jruby.ext.openssl.X509Cert.set_serial(X509Cert.java:303)
org.jruby.ext.openssl.X509Cert.initialize(X509Cert.java:166)
org.jruby.ext.openssl.X509Cert$i$0$1$initialize.call(X509Cert$i$0$1$initialize.gen:65535)
...
During initialization, such a certificate with a serial number having MSB=1 can be loaded via java java.security.cert.X509Certificate (line 158), this results in cert.getSerialNumber() (line 166) to yield a negative BigInteger due to it's internal 2's complement representation. Thus, a valid certificate can result in an error.
Typically, such a situation might arise from a certificate authority which generates serial numbers via UID (random bit generation) instead of an incremental approach. This results in a ~50% chance that the MSB=1, and causing this failure.
Eg, Serial number "d4 06 2d 6b c9 84 48 86 3d ea c2 cd c1 37 86 8a" (any serial where the first hex digit is 8-f)
The fix is simple, just make sure that the bouncycastle code does not receive a negative SerialNumber. This can be done by forcing BigInteger to interpret the serialNumber bits as positive number by setting the signum in the constructor, as follows:
X509Cert.java:303
-generator.setSerialNumber(bi);
+generator.setSerialNumber(new BigInteger(1, bi.toByteArray()));
note: This is difficult to reproduce using certificates generated via openssl CA, since it seems to pad the serial with extra 0s for MSB, neatly avoiding the problem.
Thank you