Better string lists in XPages

Like Notes, XPages supports multi-valued fields. You can set a separator for multiple values, and tell XPages whether to trim whitespace from around each value.

Unfortunately, if you have (say) a list of string values, the separator ends up with no whitespace after it. This can look ugly:

Mon,Tue,Wed

Ideally we’d like to display the multiple values with a space after each comma, and also accept separators like newline or semicolon when entering data. Easy to do in the Notes client, not so easy in XPages.

The solution is to write a custom JSF converter. The interface is defined by the class javax.faces.convert.Converter. There are basically two methods you need to write: getAsObject takes a string and returns an object suitable for storage in the back end, and getAsString takes a back-end storage object and returns a string representation.

In the case of XPages, the back-end object for a list field is a Java Vector, and we can split and trim values in one step using a compiled regular expression:

private final static Pattern listPattern = 
  Pattern.compile("\\s*[,;\\n\\r]+\\s*", Pattern.MULTILINE);

public Object getAsObject (final FacesContext ctx, final UIComponent uicomp,
  final String val) {
  String[] vals = listPattern.split(val);
  return new Vector(Arrays.asList(vals));
}

The reverse process is slightly more complex. Until later this year XPages doesn’t have Java 8, so we can’t use StringJoiner. Instead, we’ll build the list ourselves, after checking that we’re getting the kind of value we expect:

public String getAsString (final FacesContext ctx, 
  final UIComponent uicomp, final Object obj) {
  if (!(obj instanceof Vector)) {
    throw new ConverterException(new 
      FacesMessage("An exception occurred converting data," +
        " expected a Vector<String>, got a " +
        obj.getClass().getCanonicalName()));
  }
  Vector<String> vals = (Vector<String>) obj;
  StringBuilder sb = new StringBuilder();
  for(String v : vals) {
    String ts = v.trim();
    if (!ts.isEmpty()) {
      if (sb.length() > 0) {
        sb.append(", ");
      }
      sb.append(ts);
    }
  }
  return sb.toString();
}

Next, unfortunately, you need to register the converter in faces-config.xml, by adding a <converter> element as a child of the top-level <faces-config> element, something like this:

<converter>
  <description>String list converter for multi-value fields</description>
  <converter-id>stringList</converter-id>
  <converter-class>com.ibm.us.meta.convert.StringList</converter-class>
</converter>

With that done, actually using the converter is pretty simple — just add it to the converter property of your inputText field.

<xp:inputText id="nameField1" value="#{document1.Users}">
  <xp:this.converter>
    <xp:converter converterId="stringList"/>
  </xp:this.converter>
</xp:inputText>

Since the converter handles everything, you don’t have to set the multipleSeparator or multipleTrim properties on the field any more.

Obviously the same technique can be used to implement number lists or date lists.

Search is hard

Getting search right in applications has many interesting subtleties.

One problem is that most people don’t understand boolean algebra, to the point that they don’t understand the difference between “and” and “or” in a set of search clauses. Unless your audience is highly technical, giving them a choice between “and” and “or” will only confuse them.

If you look at sites which successfully allow complex queries, they do it by using the type of UI component to express whether the choice is “and” or “or”. For example, NewEgg:

newegg

Clauses are implicitly “and”, except for sets of checkboxes. So the final search here is “Newegg Premier” AND (price $200-300 OR price $300-400).

This “implicit boolean operators” search pattern has become almost ubiquitous, though you do find the occasional site running older search software that still has drop-downs to choose “and” or “or” for each clause.

Another subtlety of search is the underlying conceptual model of the operation. To a programmer, search is straightforward: you start with an empty result set, you assemble a list of criteria, you go to the database and pick out items that match the criteria, and you add them to the result set.

That’s not the user’s conceptual model, though. To the user, you start with the set of every possible item, and you then eliminate items which don’t match the criteria. The conceptual model mirrors how you’d perform a search manually.

This becomes relevant when the user is given an advanced search interface, selects no criteria at all, and asks the system to search. Programmatically, the set of items which don’t match any criteria is the empty set; but what the user actually expects based on their mental model is all items.

Google catch this scenario by disabling the search buttons on the front page until the user types some text. They also ignore empty search string submissions using the Enter key. Interestingly, they leave the Search button visibly enabled, even though it doesn’t work. It would certainly be possible to have the button visibly disabled until the search text field contained a non-empty string; presumably they’ve done A/B testing that demonstrated that buttons initially being disabled led to user confusion.

I initially considered catching the “no criteria” case and redirecting the user to the browse UI instead. On reflection, though, I realized that selecting no criteria and hitting “Search” was a legitimate thing to do. The use case is “find out what the search results page will look like, then decide how many criteria I need to specify”.

Not a hard problem from a technical point of view, but I thought it showed an interesting subtlety involved in getting search right.

Implementing live search on a web form using XPages

Given that people find it hard to select from more than around 8 options, the “drop-down selector with live search filter” design pattern is useful for all sorts of situations. If you have a reasonably small number of options, you can do it all client-side. A couple of thousand select items can be read as JSON and filtered client-side without too much of a performance hit — at least, on the desktop, with a reasonably fast Internet connection. But what if you have thousands? Or tens of thousands? Or if you want your application to perform well on mobile devices?

I needed to implement some sort of “search, then select from the matches” selector box for web forms that would use get the server to do the work. Here’s how I did it using Domino and XPages.

The first step is making sure you have a database view which lists your data set, with suitable columns — in my example, a product ID and a product name. (IBM has a lot of products.)

There are three main components on the web form. A text field for the user to type search text into, a button to click to perform the search, and a select box for the results.

We want the search filtering to happen server side, so some sort of partial refresh will be needed. We’d like to trigger the search both when the Go button is clicked, and when the user types Enter in the text field.

So to keep things simple and avoid repeated code, let’s make the select box fetch the search text and perform the search when it’s refreshed.

One thing I’ve learned in developing XPages applications is that for robustness and performance reasons you want to write as little JavaScript as possible; so let’s assume we have a Java method which will do the actual search work and return a list of SelectItem objects. Let’s also assume that the Java code returns a safe blank list if we pass it nil or an empty string.

Since we know we’re going to be using partial refresh, we start by creating a panel to enclose the objects we’re going to be refreshing.

<xp:panel id="productSearchSelect">
  <!-- Everything else will go here -->
</xp:panel>

Here’s how we might code the select box:

<xp:listBox id="productList" value="#{document1.Product}" size="8"
  style="width:300px">
  <xp:selectItems>
    <xp:this.value><![CDATA[#{javascript:
      importPackage(com.example);
      var st = getComponent('searchText').getValue();
      return ProductSearch.search(st);
    }]]></xp:this.value>
  </xp:selectItems>
</xp:listBox>

This assumes the text field has the ID searchText. All standard stuff. However, there’s a catch. When I made the Java code log the search text and the number of results it got for debugging purposes, I discovered that it was called twice for every partial refresh.

OK, I thought: JSF lifecycle. The usual approach is to use $ instead of # for the value, so it’s only computed when the page is first rendered. However, that won’t work here, because we specifically want the value to change dynamically. (Also, the search text component doesn’t exist at that point in the JSF lifecycle.) So, how to avoid repeated evaluation during a partial refresh?

If you’re smart, you might know that it’s possible to find out if you’re in the rendering phase of the JSF lifecycle. So you wrap your ProductSearch.search call in an if statement which checks view.isRenderingPhase(), try again… and discover that it still gets called twice. Apparently XPages evaluates the selectItems property of a listBox twice during the rendering phase. I honestly have no idea if it’s supposed to, but it does.

All is not lost, however. The same JavaScript context is used for both of the evaluations, so we can use a simple JS variable to cache the result:

<xp:listBox id="productList" value="#{document1.Product}" size="8"
  style="width:300px">
  <xp:selectItems>
    <xp:this.value><![CDATA[#{javascript:importPackage(com.ibm.us.meta);
      var st = getComponent('searchText').getValue(), productList;
      if (!productList) {
        productList = ProductSearch.search(st);
      } 
      return productList;
    }]]></xp:this.value>
  </xp:selectItems>
</xp:listBox>

Now, at this point you might be thinking “Eww, a global variable!” It’s not as bad as all that, though, because we’re going to be using partial execution. That means it only needs to be uniquely named within the refresh panel’s SSJS code.

So let’s talk about the refresh. Here’s the code for the Go button:

<xp:button value="Go" id="button1">
  <xp:eventHandler event="onclick" submit="true"
    refreshMode="partial" refreshId="productSearchSelect"
    execMode="partial" execId="productSearchSelect" />
</xp:button>

Notice that I’m specifying both partial refresh and partial execution.

Partial refresh means that only the objects within the specified component (our outer panel) will be updated on the page. However, by default all the objects on the page get sent back to the server so the refresh can occur, even if you’re only refreshing part of the page. To avoid that, we additionally request partial execution, so only the objects inside the panel will be sent back to the server and their code executed.

Note that since the list box fetches the search text from the searchText text input field, that means the input text field also needs to be inside the panel. Here’s the code for the text field:

<xp:inputText id="searchText">
  <xp:eventHandler event="onkeypress" submit="true"
    refreshMode="partial" refreshId="productSearchSelect" 
    execMode="partial" execId="productSearchSelect">
    <xp:this.script><![CDATA[
      if (thisEvent.keyCode !== 13) {
        return false;
      }
      return true;
    ]]></xp:this.script>
  </xp:eventHandler>
</xp:inputText>

In spite of its position inside the eventHandler that triggers the partial refresh and runs server-side code, that piece of JavaScript is client side JavaScript. It sits on the onkeypress event for the field, and returns false (ignore the event) for any keystroke which isn’t Enter. When Enter is typed, the event causes a partial refresh with the exact same parameters as the Go button.

The final piece of the puzzle is the Java code which will actually do the search. In this example, I’m going to use a simple full text search, but obviously you could do any kind of search you like — you might ask the user to pick a brand, a type of product, or something else, and filter on that as well as product name.

Here’s the Java code for a simple full text search:

package com.example;

public class ProductSearch {

  private static final int MAX_MATCHES = 50;

  public static List<SelectItem> search(final String sstr) 
    throws NotesException {
    List<SelectItem> result = new ArrayList<SelectItem>();
    if (sstr == null) {
      return result;
    }
    String tsstr = sstr.trim();
    if (tsstr.isEmpty()) {
      return result;
    }
    Database db = DominoUtils.getCurrentDatabase();
    View view = db.getView("LookupProducts");
    view.setAutoUpdate(false);
    view.FTSearch(tsstr, MAX_MATCHES);
    ViewNavigator vnav = view.createViewNav();
    vnav.setBufferMaxEntries(MAX_MATCHES);
    ViewEntry ve = vnav.getFirst();
    while (ve != null) {
      Vector cols = ve.getColumnValues();
      String code = (String) cols.get(0);
      String name = (String) cols.get(1);
      result.add(new SelectItem(code + " " + name, name));
      ViewEntry trash = ve;
      ve = vnav.getNext(ve);
      trash.recycle();
    }
    vnav.recycle();
    view.recycle();
    db.recycle();
    return result;
  }
}

A few things to note here. Firstly, we make sure that our function always returns a non-null value. If there’s no search text, we just return an empty list of select items.

Next, notice that the first thing I do after opening a view is setAutoUpdate(false). Domino tends to try and be helpful to the na├»ve programmer, and one of the helpful things it does it try to update views on the fly if any values change. This can be terrible for performance, so it’s good to get into the habit of turning off autoupdate unless you really need it. Even IBM doesn’t change product names that quickly, so…

With the view fetched, we perform the full text search. By using the two-argument variant, we get to specify a maximum number of values to return. This is partly to ensure reasonable performance, and partly because nobody can deal with a huge number of search results in a selector.

Once the view has been filtered to the search results, we create a view navigator object. This makes enumerating data from a view much faster, as long as setAutoUpdate is false. Since we know the maximum number of view entries we’re going to have to navigate through, we can set the navigator cache to be that size.

Finally, we loop through the view entries. Notice that every single Domino object must be explicitly recycled as soon as we are done with it and its child objects. This is so that underlying C/C++ data structures get deallocated, and is just one of those things you have to get used to with Domino.

Finally, we return our list of SelectItem objects. They get passed back to the XPages control, which displays them as selectable rows.

This is a simplified example, and it can be improved in many ways. Here are a couple of suggested improvements:

  • Make the form indicate how many search results were returned, and warn if the number of results was likely truncated.
  • Make the search text field trigger a search if the user pauses for more than (say) half a second after typing a keystroke.