Category Archives: System administration

Two OpenLDAP TLS gotchas

The scenario:

  • You’re using CentOS 7 or RHEL 7.
  • You’re using OpenLDAP.
  • You have TLS set up on OpenLDAP.
  • You are trying to perform a query against the server using ldapsearch.

Problem #1:

You get:

ldap_start_tls: Can't contact LDAP server (-1)
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Possible solution:

You’re using the -Z option (along with -h and -p) to specify the host and port and request TLS. That option doesn’t work.

Instead, just use the -H option and specify the parameters as an LDAPS URL.

For example:

ldapsearch -x -Z -h localhost -p 636 -b 'dc=example' '(objectClass=*)'

becomes:

ldapsearch -x -H ldaps://localhost:636/ -b 'dc=example' '(objectClass=*)'

This is actually mentioned on the OpenLDAP FAQ if you look hard enough:

Most clients now have a -Z flag which enables sending the StartTLS extended operation to the server. This extended operation initiates TLS negotiation. To use ldaps://, one must use -H ldaps://.

(Emphasis mine.) I’ve no idea what -Z is for, given that it isn’t needed if you use -H with an LDAPS URL, and doesn’t work if you don’t.


Problem 2:

Having worked out how to request a TLS connection, it still doesn’t work, so you try adding the -d 1 argument to turn on some debugging. In the client’s debug logs you see either:

TLS: error: tlsm_PR_Recv returned 0 - error 21:Is a directory

or

TLS: error: tlsm_PR_Recv returned 0 - error 13:Permission denied

Solution:

You probably have SELinux enabled. OpenLDAP doesn’t work with SELinux. Disable it and reboot the server, or work out how to persuade SELinux to let openldap open its own config files.

Playing Domino without a POODLE

If you run any kind of Internet server, you’ve hopefully heard about the POODLE vulnerability in SSL 3.

If you run a Domino server, you need to worry about this, because Firefox plan to turn off SSLv3 support in their next release in a couple of weeks and remove the code in the release after that — and Chrome will follow soon after. SSLv3 is the only secure connection supported in Domino out of the box, so that could leave you with no HTTPS support.

Don’t panic, though. IBM’s software developers have beavered away and produced interim fixes for Domino TLS 1.0 support. Another option is to run Domino behind the IBM HTTP server.

I decided against those approaches, for the following reasons:

Regarding IBM HTTP server, it’s only officially supported on Windows, and my servers are Linux. You can obtain a version of IBM HTTP server as part of WebSphere, but that’s a lot of software to download and install just for a web server, particularly when it might not work and isn’t officially supported.

As far as the interim patches go, while TLS 1.0 support solves the immediate problem, it’s far from ideal, as the current version of TLS is TLS 1.2, and TLS 1.0 is known to be not much more secure than SSLv3 when considering another attack known as BEAST.

So instead, I decided to put Apache in front of Domino, acting as a reverse proxy — much as you’d usually put a web server in front of WebSphere or any other J2EE server, or a Ruby on Rails server. Apache handles all the secure connection details, via TLS 1.2, and then forwards the request via plain HTTP to localhost, to the Domino server which is running on a different port. Some “magic” HTTP headers are used to tell Domino where the original HTTPS request came from, so from the point of view of a Domino application, everything looks exactly as if the request had gone straight to Domino.

The process of setting all this up wasn’t too hard, but it required assembling information from a variety of sources, and experimenting inside a VM until I was sure I could do it without significant downtime. So, I thought I’d write up my final process.

If you prefer, Jesse Gallagher has had success using nginx as reverse proxy for Domino.

You might also want to look at Darren Duke’s prebuild Ubuntu VM set up to proxy Domino.

Step 1 is to install Apache and OpenSSL. That’ll depend on your OS. For RHEL or CentOS, it’s a matter of yum install httpd mod_ssl openssl.

Theoretically you can use the new release of kyrtool to pull out your existing SSL key from Domino’s keyring and import it into Apache. I took the easy way out and generated a whole new key.

So, my step 2 is to generate yourself an RSA key pair for Apache:

openssl genrsa -aes128 4096 > myhostname.key

Next, you want to generate a Certificate Signing Request (CSR) for the public key portion of your new key:

openssl req -utf8 -new -key myhostname.key -out myhostname.csr 

Now you send the CSR file to your favorite SSL certificate vendor. If they give you a choice, request SHA2-256 for the hash algorithm; SHA-1 is insecire and will be removed from browsers some time next year.

In the mean time, you can generate a self-signed certificate for testing:

openssl x509 -req -days 365 -in myhostname.csr -signkey myhostname.key -out myhostname.crt

That’s one long command.

Generally for RHEL or CentOS, Apache expects to find its SSL key files in the following locations:

/etc/pki/tls/private/localhost.key # Private key
/etc/pki/tls/certs/localhost.crt   # Signed public key
/etc/pki/tls/certs/ca-bundle.crt   # Certificate chain

If you want to start Apache without needing the password you set when you created your new private key, you can create a copy of the key file with the encryption and password removed:

openssl rsa -in myhostname.key -out myhostname-decrypted.key

Make sure the decrypted file is in /etc/pki/tls/private and not readable by anyone other than root.

Step 3 is to make some Domino changes.

In the Server Document, Ports… tab, Internet Ports… sub-tab, set the HTTP port to be something other than 80. I picked 1080, but you could use pretty much any port, as it’ll be invisible to end users. While you’re there, turn off SSL as well, as we won’t be using Domino’s SSL any more.

A brief word about firewalls at this point: I’m assuming that all unknown ports, including whatever you pick, are firewalled off by default. In my case, port 1080 is blocked completely from end users. I’m also assuming that your server is allowed to talk to itself as localhost, on any port it likes.

Next, you’ll want to issue the commands

set config HTTPAllowDecodedUrlPercent=1
set config HTTPEnableConnectorHeaders=1

to the Domino console.

Step 4 is to configure Apache to act as reverse proxy for whatever port you just picked. There are a lot of instructions out there for how to do it, but I found that most of them were overly complex. All it really needs is:

ProxyRequests Off
ProxyPreserveHost On
AllowEncodedSlashes On

# Bounce port 80 to 443 (HTTPS all the things!)
RewriteEngine on
RewriteCond %{SERVER_PORT} !^443$
RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [NE,NC,R=301,L]

# Preserve remote IP info via special WebSphere variables
SetEnvIf REMOTE_ADDR (.*) temp_remote_addr=$1
RequestHeader set $WSRA %{temp_remote_addr}e
RequestHeader set $WSRH %{temp_remote_addr}e
# Tell Domino we're always on SSL
RequestHeader set $WSIS true

# Proxy to Domino
ProxyPass / http://127.0.0.1:1080/
ProxyPassReverse / http://127.0.0.1:1080/

For RHEL/CentOS, stick that info in /etc/httpd/conf.d/domino.conf. If you’re interested in the mysterious $WS headers, Vincent Kong has more info.

Step 5 is very important. If your server is running with SELinux enabled in enforcing mode — which mine all are — you need to allow Apache to initiate TCP/IP connections.

/usr/sbin/setsebool -P httpd_can_network_connect 1

This command can take a few seconds to complete. The -P means it should be persistent across reboots.

Step 6 is to update the default Apache SSL config to harden it and prevent use of old, broken encryption algorithms. Mozilla has a useful guide for this; for RHEL/CentOS, the config is in /etc/httpd/conf.d/ssl.conf.

Now for the moment of excitement. Go to the Domino console and tell http restart. When it has done so, you should find that it’s now listening on port 1080 (or whatever you chose) instead of port 80. You can verify this with

lsof -i :1080

Assuming that worked, /etc/init.d/httpd start. If you did everything right, you’ll have just switched over your Domino server to be fronted by Apache, with only a few seconds of downtime.

You should now be able to go to your server’s web URL. If you go via HTTP, you should be immediately bounced to HTTPS. You should see your Domino server’s web site output exactly as if Apache wasn’t there. However, if you check the security settings in your browser, you should find that you’re connecting via TLS 1.2!

The final step 7 is to install the certificate you eventually get from your SSL cert provider. That’s simplicity itself — just replace the .crt file you generated in /etc/pki/tls/certs with the signed one, and apachectl restart.

If you use iNotes, there are some extra pieces of Apache config needed; there’s a developerWorks article about that.