Domino server and Java client, SSL with self-signed certificate

Here’s the problem scenario:

  • You have a Java client program, and you want it to connect to a Domino server using HTTPS.
  • Java fails at runtime with javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  • This is happening because you have a self-signed SSL certificate on the server, which Java refuses to accept.
  • You’ve found out that you can import a self-signed SSL cert into Java’s runtime environment using keytool, but Domino uses an obscure proprietary keyring format .kyr which is only readable by IBM’s IKeyMan application, which doesn’t ship with Domino. (Or possibly you have IKeyMan, but the version you have doesn’t support .kyr conversion to other formats, or you don’t have access to the server to yank out its SSL certificate.)

The solution to all of the above:

  • Use Firefox 3 to navigate to the Domino server via HTTPS. Set up a certificate override so Firefox 3 will accept your self-signed cert.
  • In Firefox, follow through this sequence of menus, dialogs and buttons: Edit > Preferences > Advanced > Encryption > View Certificates > Authorities.
  • Find your certificate in the list, click it, click the Export… button.
  • Export into regular PEM format (the default), call it servername.pem.
  • Import it into a Java keystore using keytool. The command and output will look like this:
    $ keytool -importcert -file servername.pem
    Enter keystore password:
    Re-enter new password:
    Owner: CN=servername.example.com, O=YourOrgName, L=City, ST=State, C=US
    Issuer: CN=servername.example.com, O=YourOrgName, L=City, ST=State, C=US
    Serial number: 01decaf3
    Valid from: Thu Feb 07 00:00:00 GMT 2008 until: Fri Feb 06 23:59:00 GMT 2009
    Certificate fingerprints:
    MD5:  7A:E2:F4:BE:D3:6C:7C:79:5C:51:73:15:DD:24:92:C4
    SHA1: 95:21:31:47:8E:49:A2:18:7E:B4:B8:17:DA:50:7E:F7:D3:13:E4:57
    Signature algorithm name: MD5withRSA
    Version: 3
    Trust this certificate? [no]:  yes
    Certificate was added to keystore
  • Finally, copy and rename the newly created ~/.keystore to be $JAVA_HOME/lib/security/jssecerts on the client machine.

You should now find that Java SSL clients will connect to your server. If you want some simple test code, try listings 7 and 7 from the article about JSSE at O’Reilly ONJava.

There’s a caveat to be aware of, though. If the JVM finds a jssecerts file, it doesn’t bother to check the regular cacerts. What this means is that normal SSL certificate resolution (for non-self-signed certs) will be broken for any Java client that uses JSSE. So if you run other software that uses JSSE, it might suddenly refuse to connect to other SSL sites.

So, a better approach is to locate your Java cacerts file, and add the .pem file to that. Your server then becomes another trusted source of certificate signatures, alongside Verisign and the other root certificate providers.

The cacerts file is in $JAVA_HOME/lib/security/cacerts, and the command to add your server to it is:

keytool -importcert -keystore cacerts -storetype jks -file server.pem

There are supposed to be deployment properties which let you override the jssecerts file for a specific Java runtime. That would be the best approach, but it doesn’t seem to work with Java 6. If anyone can get it to work, I’d be delighted to know what the secret is.

Update: Andreas Sterbenz wrote a handy utility while he was working for Sun. Grab the code before Oracle take it down. He’s now working for Google, like many ex-Sun Java guys, so obviously you’ll never hear from him again.