Category Archives: System administration

In the wake of shellshock

So, shellshock. It’s big. I think it’s bigger than heartbleed, because the bug has been in the code for 22 years, so there are an awful lot of systems out there with a vulnerable shell installed and nobody maintaining them properly.

One misconception I’ve seen posted across the web is that you’re not in trouble if you don’t use bash as your shell, or that you’re safe if you have dash as /bin/sh. Sadly, that’s not true. GNU tools like env will use bash to execute other programs no matter what your personal shell is. Other programs you use may invoke bash too — for example, on Ubuntu the zcat utility is a bash script. So are Firefox and Chrome.

As for exploits, basically any program which sets environment variables based on user input and uses the shell to execute something can become a vector. There’s a working exploit for Linux DHCP, so you can get pwned simply by connecting to a WiFi network. There are at least two worms spreading via the web, where you can pwn servers just by setting your web browser’s user agent to something cunning. There’s an IRC worm active, and an exploit via email has been demonstrated for some server-side software. SSH restricted mode is affected, becoming SSH totally unrestricted mode. And that’s just the stuff I know about.

At work, a colleague just asked what needs to be upgraded. My answer: Every Linux system on the planet. Prioritize ones which accept data from outside your organization (no matter how they do it). You’ll eventually want to patch everything running a Linux distibution, because the bug is also perfect for use crafting local root exploits.

Yes, it really is that bad. Even Windows systems with Cygwin are affected. Many embedded systems use busybox rather than a full shell, so (for example) your router might be safe, but I advise that you check: See if /bin/bash exists and whether /bin/bash --version reports that it really is bash. One small silver lining is that Android phones aren’t affected unless the user has chosen to install a bash shell, so we aren’t looking at a mobile phone worm apocalypse yet.

Oh, and if you patched your Linux system yesterday, patch again, because the first patch didn’t completely solve the problem, and there was a second patch released this morning.

Apple, meanwhile, still doesn’t have a patch for their old forked version of bash. Instead, they have a statement that the issue is not a problem for Mac users, so I’m sure hackers are developing OS X exploits right now.

Some people have been warning about the dangers of bash scripting for years. I’ve been bash-averse for decades; in fact, I never write shell scripts unless I absolutely have to because shell in general has so many pitfalls compared to (say) Ruby.

Of course, I’ve still had to deal with the fallout from shellshock, but I must admit I’m chuckling about the fact that purely by chance, I decided last week was the week I would finally get to grips with using Ansible to push software updates to all the servers I look after.

Another bash basher is David Jones, who has now written about why bash is bad, and provides some tips on turning your bash scripts into standard shell scripts.

If you really must continue to use bash, use unofficial bash strict mode.

Sometimes old technology is the best option

A while back, I had an interesting problem to solve. We have a set of servers on their own TCP/IP network, with addresses in the 10.0.0.0/8 private IPv4 space; it’s colloquially referred to as “the 10 network”. It’s cut off from the main corporate network; you generally need to use a VPN client to access it. It’s possible to get SSH connections enabled by special request, but only from the corporate network to the private network, only to unprivileged IDs, only between static IP addresses, and definitely not for any non-SSH protocol.

We needed to be able to send e-mail from the private network, via the corporate e-mail servers. I could set up an SMTP server on the 10 network easily enough, but there was no way it could initiate a connection to any other SMTP server. We couldn’t even initiate an SMTP connection inbound to it.

Colleagues suggested schemes involving POP3 or IMAP to pull down mail and re-send it. There was discussion whether we would be allowed to use a scripted SSH connection from a system running the VPN software, to trigger a temporary port tunnel and then prod the MTA to attempt delivery. However, I had a better idea, because I remembered how e-mail had been delivered in the 1970s through to the early 1990s, when all we had was dial-up connections.

The software in question is called UUCP, the Unix to Unix Copy Protocol. The great thing about UUCP is that all it needs is a way to send and receive bytes via a terminal-type connection. In fact, it doesn’t even need that; back when UUCP was in common use, there were quite a few networks or modems that only supported 7-bit data, so UUCP can escape data to make it work over 7-bit links too. UUCP also doesn’t need the connection to be reliable, or even there all the time—it can be triggered to perform data transfers in batch mode, then shut down.

I vaguely remembered that the free software Taylor UUCP had been given some sort of SSH support. Investigation revealed that protocol t was aimed at TCP/IP connections, where you can assume that the bytes you send will be received accurately at the other end. (Unless, of course, the entire connection fails for some reason, in which case UUCP will detect the problem and retry later.)

Taylor UUCP was still available prepackaged for RHEL, and Postfix still had all the hooks for UUCP support available. I set up the two servers, one on the corporate network and one on the 10 network. The corporate machine had been set up to allow SSH (and only SSH) connections to the 10 network. I set things up so that UUCP ran chrooted under a restricted ID, and I was also able to set the 10 network server’s UUCP user to have uucico as login shell—so even if the ID is compromised, it can’t be used to attack either server.

So now Postfix accepts mail from machines on the isolated network, and passes it to UUCP. UUCP turns the mail into UUCP jobs, and queues them. A few times an hour, a cron job on the the internal network machine starts up uucico, which calls SSH to open a connection to the uucico on the isolated server. They then exchange all the pending UUCP jobs—bidirectionally—and disconnect. Finally, UUCP on both servers processes the batches it just received, and passes the e-mail on to Postfix. And all without even any port forwarding.

It’s worth mentioning that you could go a step further, and not even allow the SSH connection, if you ran a serial cable between the two servers. IBM Notes still supports serial connections too, allowing secure e-mail between machines that can’t route TCP/IP packets.

So if you ever find yourself in a similar bind because of firewall issues or corporate security policies, remember that we used to transfer files and send e-mail before there were even TCP/IP networks to firewall. There are tutorials for how to set up UUCP over SSH, and unlike any port forwarding kludge you’re likely to come up with, UUCP is time-tested software. It may be a little arcane, but it works.