Looking for debug information in all the wrong places

Today I took some Java code which I had been running on my laptop, and tried to deploy it to the server where it belongs.

The code in question is a command-line utility designed to be run from cron. It connects to a Domino server via Web Services, connects to a DB2 server via JDBC, and then pumps data between the two via a protocol designed to try to minimize the number of updates that need to be transferred.

I deployed using Fat Jar, which is one of the most awesome Eclipse plugins around. It makes it trivially easy to take an entire working project, including all the non-standard frameworks and libraries it depends on, and bundle it into one big .jar file. This allows me to deploy things as a single jar which can be double-clicked or run using java -jar whatever.jar. Since I use the JDBC type IV driver, I don’t even need to have DB2 installed on the deployment host; just a JRE.

To make life easier, I obtain all the runtime configuration using the Java Preferences API. I store all the invocation-to-invocation state information that way too, using a Plain Old Java Object (POJO) to wrap the Preferences API and supply all the application’s preferences and persistence needs.

In the event that the preferences aren’t there, the object creates a suitable set of preferences with default values, and tells the user where to find the XML file Java stores them in before shutting down cleanly.

So, I wasn’t expecting any big problems. Copy over the .jar and .xml files, put them in the appropriate places, and run the .jar.

Unfortunately, something unexpected happened: an SSL handshake exception. Since I’ve had SSL certificate problems with Java’s JSSE before, I immediately assumed certificates were once more to blame. I copied over a trivial Java HTTPS client, and tried connecting to the top level https URL of the server — same problem.

I then spent quite a lot of time fiddling around with certificates — connecting to the web site with Firefox, exporting the chain of certs in PEM format, converting them to DER format with OpenSSL, locating the Java CA certs on the server, feeding the DER format certs to Java’s keytool to add to the default CA certs… Still no luck. I even tried dumping the list of CAs on the server and comparing it with the list on my laptop.

Next, I belatedly wondered if there was something odd about the server’s HTTPS handshaking. It was working fine with Firefox, but what about other programs? I tried connecting with curl.

By sheer luck, I tried connecting with curl from inside an SSH session to the server. It failed, with curl: (35) Unknown SSL protocol error. Trying curl -v for more debugging info didn’t help much:

* successfully set certificate verify locations:
*   CAfile: /usr/share/curl/curl-ca-bundle.crt
  CApath: none
* SSLv2, Client hello (1):
* Unknown SSL protocol error in connection to myserver.example.com

(Hostname changed for obvious reasons.)

It was because of this lack of useful trace output that I finally, much too late, resorted to checking the server log:

SSL handshake failure, no website found for IP address [127.0.0.2] 

This format of error typically occurs when you’ve got an IP address set up for your server, say www.store.example.com, but the HTTP server only knows about www.example.com. So when you request www.store.example.com in the HTTP protocol, the server complains that there’s no website found for that address.

I stared at the error log for a while. Was it really possible that curl was issuing a Host: 127.0.0.2 header as part of the HTTP request? No, that couldn’t be it because it wasn’t getting as far as the HTTP request.

Finally, I decided to look at /etc/hosts on the server, and found:

127.0.0.1 localhost
127.0.0.2 myserver.example.com myserver

Well, that certainly explained why the HTTP server was confused. I’m still not sure I understand why the server only gets confused on loopback connections, but since fixing /etc/hosts made the problem go away all the time, I’m happy. Perhaps I’ll wake up tomorrow and it’ll all be obvious.

Right now, my hypothesis is that the entry for 127.0.0.2 is something YaST thought was a good idea, as I can’t imagine any human setting up the configuration that way.