« Back to home

Wolfram Alpha

When Wolfram Alpha was made public, like everyone else I went to the site to see what it could do. The demonstration queries produced pages which looked really useful. I could instantly imagine how I might find the site invaluable.

Then I hit a wall. I thought of things I would like to see displayed by Wolfram Alpha, and for each, I wondered: how would I persuade it to show such a thing?

Let’s go through a real example.

I’m interested in the state of the economy, and how bad things have actually gotten. A fairly basic task is to find a graph of the US unemployment rate, from some recent year to the present. So I go to Wolfram Alpha and type us unemployment rate 1990-2009. The result: (insufficient data available).

The naïve user would probably take the error message at face value, and think “Wolfram Alpha doesn’t have enough unemployment data to show me the last 20 years?” and conclude that the tool is useless. Since I’m a computer scientist, however, I know that it’s just an extremely badly worded error message, and I press on.

The page shows my query with boxes around all the words except 2009, so I try again with a single year: us unemployment rate 2009. This time, Wolfram Alpha says Assuming “us unemployment rate 2009” is international data | Use as economic data instead.

Given that it says it understood what I meant by “US” and “unemployment rate”, it’s pretty dumb that it decided that was not economic data, but that link looks helpful. I click it, and get the graph I’m looking for — but only for 2009, of course. (A quick second query proves that yes, it does have data going back to 1990, and was lying when it said “insufficient data available”.)

So clearly Wolfram Alpha can do what I want, but it needs to be in some kind of special “economic data” mode. Entering my original query in the box on the economic data page doesn’t interpret my query in “economic data” mode, so how do I select that?

I try adding the words to my query: us unemployment rate 1990-2009 economic data. No deal, Wolfram Alpha now says it “isn’t sure what to do with” my input. (Another bad error message: it knows exactly what to do with my input, i.e. parse it, the real problem is that it doesn’t know how.)

Moving the “economic data” to the front doesn’t help either. Turns out, the clue is in the results page I got for a single year. There’s a box containing the words “civilian unemployment rate”. That’s the magic phrase to use instead of “unemployment” in order to put Wolfram Alpha into economic statistics mode, at least for this one query. So finally, I type us civilian unemployment rate 1990-2009 and get what I was looking for. Or rather, something close to what I was looking for: It looks like a graph of US unemployment rate. However, it starts at 1950 rather than the year I requested.

This is how it goes every time I think of a problem for which Wolfram Alpha might be a solution: the “intelligent” search interface on the front acts as a frustrating wall between me and the actual tools. Except, of course, for the trivial problems like “£36 in $”, which Google already handles quite adequately.

I decided to compare my Wolfram Alpha experience with the Google experience. I typed graph us unemployment rate and hit Enter, and the first link in the Google results was a small graph next to a link to Google’s public data site. I follow the link, and to my astonishment, it’s exactly the graph I want. It has a meaningful vertical axis. It cites the data source. It defines exactly what it means by “unemployment rate”. It even randomly chose the same range of years I picked for my example. What’s more, it has useful tools for adjusting the graph to show specific states, and hovering the mouse over the line reads off the value at that point.

I swear, I didn’t even look at Google before picking my example. I didn’t even know Google had a public data graphing site. And yes, granted, Google offers no way to adjust the date range, but still: 0 out of 10 to Wolfram, 10 out of 10 to Google.

So my initial superficial impression of Wolfram Alpha was “Hmm, nice idea, pain to try and use, doesn’t really work, maybe it’ll be worth using some day.” I filed the site away in my bookmarks in case I started hearing rumors it was useful, and went on with my work.

Sadly, I now realize that I was being overly generous with my assessment. Not only does Wolfram Alpha not work, the current design is a pretty stupid idea to start with. As others have now pointed out:

The task of “guess the application I want to use” is actually not even in the domain of artificial intelligence. AI is normally defined by the human standard. To work properly as a control interface, Wolfram’s guessing algorithm actually requires divine intelligence. It is not sufficient for it to just think. It must actually read the user’s mind. God can do this, but software can’t.

Google succeeds because for complex information searches, it directs you to a sub-site with a special-purpose interface. Wolfram Alpha tries to make the text entry box be the only interface; even when it understood my query, it didn’t take me to a graphing area or an employment statistics area, so chances are for any related query I’d have had to go through the whole frustrating query experimentation process again.

English language input barely works for Interactive Fiction, where there are at most a few hundred objects you might be referring to and maybe a hundred things you might want to do with them. Even assuming state-of-the-art AI, it’s madness to think that an English language interface to something whose problem domain is the whole of mathematics could ever be usable.

The alternative? Another quote:

… the human skull contains an organ called a “brain,” which has spent several million years learning to use tools. Therefore, if you are building a control interface, ie a tool, the prudent way to proceed is to (a) assume your users will need to learn to use your tool, (b) make it as easy as possible to learn the tool, and © make the tool as effective as possible once it is learned.

There is a caveat to this: people may be good at learning to use tools, but there’s a sizeable population who do not want to learn anything, particularly not how to use a tool, and certainly not when it might be the right tool for the job. You’ll see these people everywhere in the business world. They’re the ones using Microsoft Word, but not using styles to format their text. They send you file attachments which turn out to be an Excel spreadsheet consisting of a list of names of people arranged in one column. I’ve not yet seen anyone come up with a viable UI approach for people like that, unless you count the cluebat as a user interface.

So my thought for the day is: when you have a complex system, make your user interface be as dumb as possible.