A common technique for getting XML data out of IBM Lotus Domino is to build an agent which outputs the DXL encoding of a document and call it via HTTP. The code typically looks like this:

Print "Content-type: text/xml" Dim session As New NotesSession Dim doc As NotesDocument [...obtain your data somehow in the variable doc...] Dim exporter As NotesDXLExporter Set exporter = session.CreateDXLExporter exporter.OutputDOCTYPE = False Dim stream As NotesStream Set stream = session.CreateStream Call exporter.SetInput(doc) Call exporter.SetOutput(stream) Call exporter.Process Print stream.ReadText()

However, there’s a subtle error in the above code. The kind of error that can make everything look fine in testing, then cause your integration work to fall over in production.
(more…)

Filed under: Domino | Comments (0)

Providing RFC3339 dates from Notes/Domino

April 6, 2009

I do a lot of integration work, and often it involves REST Web Services and other forms of XML data transfer. Generally I want to provide dates and times in RFC3339 format, or some variant of ISO8601; for example, 1996-12-19 16:39:57 -0800.

When writing a Web Service in LotusScript, it’s not too hard; I built myself a function which converts a NotesDateTime or Variant of type 7 into UTC and then formats it appropriately, and I use that all over the place. Similarly, in Java it’s easy enough to convert everything to UTC and avoid needing to work out the correct numeric time zone offset.

Then one day I had to put the value of @Modified into a view column, for export. Since view columns can only use formula language, I was faced with trying to produce the appropriate format using only @Formula and column properties. To make things worse, DST absolutely had to be handled correctly. So did time zones that aren’t on an hour boundary, such as -0330. (We don’t currently have any servers in Newfoundland, but I don’t want to get phone calls if we ever do set up a site there.)

For a while, I struggled with @Zone, @Text, and a mess of string manipulation–but it didn’t seem to behave properly. There’s something odd about the output of @Zone that makes it behave differently from an identical pure string when converted to a number.

Date/time columns have various ways to customize their output. However, there’s no way to say “always show dates and times in UTC”; that would be far too useful. There’s an “always show dates and times in local time zone”, but that changes twice a year because of the DST madness, and frankly I don’t trust it to change on the right dates. I could set the server to be in UTC–and in fact I often do–but I don’t want my application to break if deployed to a server with different locale.

Then I suddenly remembered that Notes 6 gained an @GetCurrentTimeZone formula which returns the current time zone where the code is running–either the client PC’s time zone, or the server’s time zone, whichever context it is in. The formula language also has @TimeZoneToText, which will convert a time zone value to a string like (GMT-06:00). Suddenly the answer was obvious: it’s trivial, so long as you use two columns (or two fields, in the case of a form).

The first column/field just displays the date/time value, formatted with custom preferences to look like RFC3339 minus the time zone part. The second field is then @Right(@TimeZoneToText(@GetCurrentTimeZone; "S"); "GMT");

The result is a clean ISO-formatted date/time value. Most importantly, because both fields/columns use the local time zone but display that zone numerically, the output indicates the correct instant in time even if the system changes to/from DST on the wrong day.

True, this isn’t rocket science. However, it’s one of those interesting situations where I was unable to see the obvious solution for a long time, because I was in a mindset: I was looking for a single formula to produce all the output at once. If such a thing exists, I bet it’s very large.

Filed under: Domino | Comments (0)

Is OnLive the future of the video game industry?

March 25, 2009

A company called OnLive have announced a new product. The idea is simple to understand: instead of buying a gaming PC or expensive console, they’ll render all the graphics for you on a server farm, send the video to a lightweight decoder box or software client via the Internet, and have it send back your controller input.

Basically, you’ll be playing games on a remote system via the net. Think of it as video games over VNC. The benefit is that you can play leading edge games like Crysis on dirt cheap hardware–supposedly.

A lot of people are skeptical about this supposed product, pointing out that US Internet infrastructure is pretty low bandwidth compared to other countries. Yet bandwidth isn’t the big problem; latency is the problem. (As Stuart Cheshire put it, “It’s the latency, stupid“.)

My typical ping times to a major Internet site are under 100ms–e.g. 25ms to hit Yahoo’s edge–but I’m one of the lucky ones with a good Internet connection. It’s not uncommon to have 250ms ping times. But let’s assume up to 100ms one way, and think about how this will work.

The OnLive hardware cloud will render the video. State of the art for low latency video encoding and decoding is 70ms or so for an encode/decode cycle. So the image gets to you after 100 + 70 = 170ms. You respond by moving the controller. Assuming no delay at all in reading the controller, your response gets back to the server after 270ms.

So basically, using today’s technology you’re talking over a quarter second of lag between something happening in the game and your being able to respond to it. And that’s a pretty optimistic case; note that the state-of-the-art HD video system I linked to had 500ms total latency in their demo across the public Internet.

Online video games use all kinds of tricks in order to stay responsive in spite of network lag. Even so, ping times can vary from minute to minute; the Internet offers no guarantees. When the net slows down, you tend to see bursty updates to other players’ positions, and fast gameplay gets frustrating. Some people have even taken advantage of this behavior, building “lag switches” which fool their copy of the game into thinking the network is lagging, allowing them to shoot you while you’re motionless (to them). They then turn off the lag, at which stage the game reconciles the conflicting opinions of the different players’ systems, often by killing you unfairly.

I note from the announcement that OnLive’s demos so far have shown a game being played across 5 miles of network. Their actual data centers will be up to 1,000 miles away from you, they say. I think I’ll wait until they try the thousand mile round trip before I believe it’s a real product.

OnLive say that their hardware encodes the video with only 1ms of lag. I guess that’s possible, if you throw enough expensive hardware at the problem–which brings me to the second problem: economics.

OnLive are floating the idea that it’ll be $50 a year for the service.

Now of course, they can share hardware and network costs between multiple users. However, their ability to do that is going to be severely limited by the requirement for low latency. They’ll only be able to time share my hardware node with other people in the same geographical area as me, and chances are most of us are going to be wanting to play video games at about the same times of day. I doubt they’ll be able to get better than about 4 or 5 customers per node on average.

They’ll also need much more expensive hardware than a typical web host. Gaming PC hardware is pretty high end, and low latency real time HD video encoders don’t come cheap. They’ll need a much more expensive network connection too–low latency will require the kind of multiple peering backbone connections that Yahoo and Google use, not the piece of copper your average web host uses.

Support costs will be higher too. They’ll need to hand-hold people through fixing latency problems, possibly intervening with their ISPs.

So they need much more expensive hardware, much more expensive network connections, will have higher support costs, and they’ll be able to support fewer users per hardware node. Yet they think they’ll be able to cover all that for $50 a year? The price of cheap, shoddy web hosting from GoDaddy? I don’t see any way that price is workable.

In summary: I could be wrong, but I think this OnLive thing is going to be vaporware.

Of course, you could do something like OnLive if you restricted yourself to games where latency isn’t an issue and the graphics aren’t tough to compress quickly. But then, you can play games like that on a $250 Wii console, or even a $99 PlayStation 2. So where’s the market?

Final thought: Even multiplayer online games like World of Warcraft ship with DVDs full of content to cache and render on the client. If this “video games via streaming video” idea was really a good one, wouldn’t people be doing it already?

Update 2009-03-30:
EuroGamer seems to agree

Filed under: Business | Comments (0)

Chronological pitfalls

March 16, 2009

Since the US just changed time zone, and the EU will be changing time zone soon, it seems like a good time to refresh an old posting of mine about dates and times, and how hard it can be to do the right things with them in software.

I’ve spent a lot of my career in the computer industry working on computer-based calendar systems and organizers of various kinds. Here are a few pitfalls to be aware of.

Time zones expressed as letters (e.g. CST) are ambiguous, and hence useless.
Time zone offsets are not always an exact number of hours.
Both 00:00 and 24:00 are valid times (as per ISO) and refer to different moments of a given date. 00:00 is the midnight which begins the day, 24:00 is the midnight which ends it, and is also 00:00 of the next day.
Rounding of durations isn’t obvious. People expect a duration in hours or minutes to be rounded to the nearest unit, up or down. However, they expect a duration in days to always be rounded up. Even if the event finishes after one hour of the final day, that day still counts towards the event duration.
Minutes do not always consist of 60 seconds. Sometimes there are leap seconds. For example, 2005 had a leap second on December 31st, and so did 2008. Or to look at it another way…
Never store time as a number of seconds from an ‘epoch’ date.
All of our local time zones are defined as offsets from UTC. For precision, UTC is defined by atomic clocks, whereas older time standards such as GMT were based on astronomical observations. However, UTC is kept in sync with older time standards by having leap seconds added or subtracted as necessary. Whenever UTC drifts far enough from UT1 or astronomical time, a leap second is scheduled.

Lots of systems store the date and time as the number of seconds since (say) January 1st 1970. The problem is, the exact dates on which the leap seconds in our calendar will occur cannot be determined in advance, because they depend upon the subtly varying rotational speed of the earth. The earth’s rotation changed measurably after the 2004 tsunami in Asia, for example. Hence given a date and time in the future, there is no way to calculate how to represent it exactly as a number of seconds from an epoch moment.

Another way of looking at it, is that if you have any epoch dates in 2006 stored in your computer, they’re now going to be off by a second because of the leap second announced for 2005, until your date and time library is updated.

Of course, you can ignore leap seconds. But if you do, any calculation that asks for the number of seconds between two moments in time may give the wrong answer. It won’t be very wrong, just a few seconds off, but that might be wrong enough to cause a very expensive error. (See The Fractious Leap Second Debate for more on this subject.)
Leap years. February. Get it right, OK?
Time stamps should never be given in local time unless you also specify the time zone for every single time stamp. Otherwise, your timestamps are ambiguous for the hour which recurs when people switch back from daylight savings time to standard time. Consider the following sequence:

Oct 31 01:55 Log in from 192.168.0.1
Oct 31 02:05 Log in from 192.168.0.2

Are these two events 10 minutes apart, or 70 minutes apart? Without the time zone information, you have no way to tell.

Of course, the right thing to do is timestamp in UTC.
You must always store a time zone along with a date, even if the date has no associated time.

The reason for this is simple: June 5th in Australia is not the same day as June 5th in San Jose. Specifically, June 5th in Australia is ending and everyone’s going to bed before June 5th in San Jose gets started. An event that happens on June 5th in Australia may happen on June 4th in San Jose, so you need to know the time zone of the event in order to display the date correctly to the user.

If you don’t store a time zone, then basically there’s an implicit time zone, most likely the time zone that will be in effect on the date in question, at the location where the date is being measured. For an event, that will be the location where the event is happening. Obviously, implicit time zones aren’t very helpful for end users distant from the event in question, because they have to (a) realize without prompting that the correction has to be made and the date has to be adjusted, and (b) guess what the time zone is and how it will affect the date as seen from their time zone.

In Lotus Notes, you’ll need to use a formula like this:

ost := @If(StartTime = ""; @Time(9; 0; 0); @Time(StartTime)); tm := @TimeMerge(StartDate; ost; TimeZone); @If(StartTime = ""; @Date(tm); tm);
This treats a date with no time as being basically the same as a date starting at 9am in the appropriate time zone. The choice of 9am is arbitrary and based on conventional business hours; there’s no answer which is always precisely correct. In other words…
Automatically adjusting dates and times in a calendar is hard. People often complain because their electronic calendar gives them the wrong answer when they fly half way across the world and change time zone. They think it’s a fault in the software, and that it must be badly designed. The truth is that there is no right answer.

Consider the following schedule:

2004-10-10 10:00 Team conference call.
2004-10-10 12:00 Lunch with Mike.
2004-10-10 13:00 Conference session 2: Designing Groupware

Suppose Mike and I fly 8 hours west. What happens? The first item on my schedule is a phone call. It happens at the same instant in universal time, no matter where I happen to be. Therefore to show me the right time, the software has to shift the displayed time by 8 hours. It might have to shift the date too, as a result.

The second item on my calendar is going to happen at noon in whatever time zone Mike and I happen to be in as we travel. We may not even know in advance what that time zone is going to be. So the right thing for the software to do is probably not to shift the time at all.

The third item is a physical event, but in this case the time and date are always going to be relative to the time zone in effect at the location where the event is taking place. So, the correct thing to do is almost certainly to leave the date and time alone.

So should the software adjust the display of the dates and times in this example? The “correct” answer is “definitely yes, maybe, and definitely not”.

One approach might be to have a flag in each event specifying whether the time and date should be corrected when the user changes time zone, or whether they should be allowed to float and effectively change. The flag could be part of the time zone field–if you specify a time zone field the date and time are adjusted, otherwise they float. I’ve never seen an electronic organizer that actually worked that way, though.
Lotus Notes does not change the time and date of an event when you change time zone. This is a common misconception. Notes actually leaves the date and time exactly the same; it is not modified in any way. An event stored and displayed as starting at instant T, is still stored and displayed as starting at instant T. However, the way instant T is indicated depends on what time zone you are in, so it looks like a different time or date, but it actually isn’t.

In contrast, the Palm OS date book does alter all the times and dates when you change time zones. If you change the time zone from San Jose to Australia, the system date and time will change, but all your events will be moved. If you set an alarm for a net radio show or a conference call, the alarm will probably happen at the wrong time. See immediately above for more explanation.

Filed under: Programming | Comments (2)

Web Services, SOAP, Domino, and Java 6

March 9, 2009

A while back I made a brief attempt to embark on a voyage of Web Services discovery via JAX-WS. The problem was simple enough: I had a Java program running on server A which had an object. I wanted to send that object to a Domino server running on machine B.

Domino has a nifty Web Services feature these days, so it took about 2 minutes to create the SOAP-based Web Service. I’m not exaggerating, it’s literally a two minute task. You just create a class with one or more methods that take arguments of whatever types are appropriate. Domino builds the WSDL and makes the whole thing accessible at a couple of standard URLs, handling authentication, session management, parsing, and everything else you don’t want to have to care about.

So, I had a Web Service. I thought that should have been the tough part; after all, Java is great for web programming, right? So I tried finding out how to access my Web Service from Java.

The standard API for Web Services clients is JAX-WS, which is included in Java 6. As usual, the JavaDoc is utterly useless unless you already know how to use the API, and the most useful thing on the project page is pointers to random web articles and blog postings written by other poor suckers who have worked out what the heck it does.

I eventually found out how you start developing a Java client program given some WSDL, and fed Domino’s WSDL to the wsimport program in the hope that it would generate a stub I could pull into Eclipse and annotate with a suitable @WebServiceRef.

Fat chance. The wsimport program churned away for a couple of seconds, and spat out an error: “error: rpc/encoded wsdls are not supported in JAXWS 2.0.”

It turns out that although Sun says JAX-WS is supposed to replace JAX-RPC as the way to do Web Services, it doesn’t actually support all the stuff the older API supports. In particular, it doesn’t support SOAP encoding or RPC encoding, even though Java still has full support for RPC, and even though the encodings still seem to be part of the SOAP standards.

So, you can get stuck with SOAP 1.1 and a legacy API; or you can retool all your Web Services to get rid of RPC and SOAP encoding; or you can do some horrible hackery which ruins the simplicity all this SOAP and WSDL stuff was supposed to give us in the first place.

There’s a workaround. Domino lets you change the encoding used for the SOAP messages. It doesn’t seem to be mentioned in any of the tutorials I can find, but in the penultimate tab of the Web Service property inspector there’s a drop-down list. Instead of RPC/encoded, you can use RPC/literal. Sun’s wsimport will then accept the Domino-generated WSDL and create a stub, and you can connect to the service via JAX-WS 2.x. The tricky bit is authentication, but even that is pretty simple. The whole thing ends up being about ten lines of code; the trick is knowing what those ten lines have to look like, as there seems to be a dearth of up-to-date sample code out there.

Imagine your remote Domino Web Service is called Foo, and you’re encapsulating everything in an object. It’ll look something like this:

public class WSInterface {

  private FooService service;
  private Foo foo;
  private BindingProvider binding;

  void setupConnection() {
    try {
      this.service = new FooService(new URL(SERVICE_URL), new QName("urn:DefaultNamespace", "FooService"));
    } catch (MalformedURLException e) {
      logger.severe("Malformed WSDL URL " + SERVICE_URL + " - cannot continue");
      System.exit(1);
    }
    this.foo = this.service.getDomino();
    this.binding = (BindingProvider) this.foo;
    this.context = this.binding.getRequestContext();
    this.context.put(BindingProvider.USERNAME_PROPERTY, USER_NAME);
    this.context.put(BindingProvider.PASSWORD_PROPERTY, PASSWORD);
  }
}

Once that’s done, you can access one of the functions of the Domino Web Service–call it bar(x,y)–by just calling it as a method on your Foo object:

this.foo.myfunction("Hello","world");

So my code was all working—until I discovered a big catch with SOAP.

If you change something in your Web Service API–even a change which should be invisible to clients, like removing an API call nobody is using–you need to regenerate the client code from the changed WSDL. Even if the actual logic is identical before and after, you still need to download the WSDL, regenerate Java stubs with wsimport, and recompile.

So that’s probably the last time I use SOAP, simple or not. From now on it’s REST all the way.

Filed under: Domino, Java | Comments (0)

Git

February 20, 2009

Linus Torvalds’ version control hack, git, has become flavor of the month in the open source world. Not everyone is happy.

Take the common case of “I have a bunch of stuff I’ve been working on, and want to publish it now”. Trying to achieve this with git led one developer to post a series of rants: git sucks part 1, part 2, part 3.

If your VCS needs a shell script to publish some code as a new repository, you probably need to work on your usability.

So far I’ve managed to avoid git. I think I’ll try and keep it that way.

Filed under: Linux | Comments (1)

Bitten by the specialness of Java arrays

February 18, 2009

Suppose you have some text in Java that you want to manipulate. There are classes like StringBuffer for doing so, but let’s imagine that you choose to use a char array for some special reason.

Now suppose that several pages of code later, perhaps in a different piece of code entirely, you decide to use the value of the variable. At that point, you’ve forgetten that you didn’t use a StringBuffer, so you do the usual thing and call the toString() method.

Here’s some code to simulate what happens:

char []s = "Results of some computation".toCharArray(); // Lots of code later System.out.println(s.toString());

When I compile and run the above, it outputs [C@1bc4459, which isn’t at all what you might hope.

There are two obvious questions which occur to me. The first is to wonder if this behavior of Java is deliberate. Is it considered useful in some way, or just an accident of implementation?

The second and more important question is: How can I detect such errors before code goes into production? Eclipse doesn’t catch it, and nor does PMD.

Filed under: Java | Comments (4)

VMware and Ctrl-Z

February 7, 2009

I use VMware for running legacy Windows-only software. It’s generally a fine product, but there’s one piece of UI design that just drives me insane.

What are the most frequently used keyboard shortcuts on a Windows system? Probably cut, copy, paste, and undo. Ctrl-X, Ctrl-C, Ctrl-V, Ctrl-Z.

In the VMware client, however, Ctrl-Z is “suspend the virtual machine immediately, without even a warning dialog”.

So if you realize you made a mistake in your Windows app, click on the VMware window to bring it to the front, and then type Ctrl-Z, the entire VM is immediately suspended to disk. You should have clicked twice on the Window before typing Ctrl-Z.

I say “immediately”, but in fact suspend and then unsuspend can take around a minute, which is a long time to spend swearing at a piece of software.

Please, someone from VMware, fix this.

Filed under: Linux | Comments (0)

Home Page

meta/LPAR

Sidebar

Recent Posts

Categories

Main Contents

OpenSSH flaw: workaround

Ubuntu Jaunty (9.04): Getting Java and Eclipse working

Getting UTF-8 out of Domino web agents