Of Domino and data pumps

March 29, 2011

When you have IBM Lotus Domino in your organization, sooner or later you come up with a requirement to move data between Domino and some other system–often a relational database. There are many ways to do this, and not much guidance is offered as to which to pursue, so here’s a summary of my own experience.

LEI

LEI is Lotus Enterprise Integrator. It’s basically a general purpose solution for pumping data between Domino and other systems–typically relational databases, but you can also use it to pump data between non-replicating Domino servers.

With LEI, you specify data pumping tasks in a database, schedule how often they should run, and the LEI server add-in moves the data for you. If it’ll do what you need to do, it’s very convenient. However, it does require a special LEI task to be installed on the server and run at server startup.

My experience is that LEI works well until you need to change data structures during the pump operation–say, to summarize information from response documents into the main document each time it’s copied, or to turn multi-valued fields in Notes into multiple records in a destination relational table. At that point, the limitations of its scripted activities become apparent, and you tend to need a more specific tailored solution. That said, I haven’t tried LEI 8 yet, and it may be more capable now.

ODBC

The first option anyone with a Microsoft background will think of is ODBC. It requires the NotesSQL driver, which is only supported on Windows–no unixODBC option. That’s a non-starter for me, as I don’t want to have to keep a Windows system up and running just to provide back-end database connectivity.

If that’s not enough to dissuade you, consider that you’ll also need to write some Windows software to perform the ODBC operations, so you’ll need some Microsoft dev tools, and you’ll be writing platform-specific code for ODBC access from C, or possibly C# if you’re lucky.

Oh, and reportedly the NotesSQL driver only supports strings up to 256 characters in length, which would also have made it completely useless for all my projects so far.

CORBA

The next option, working roughly in order of descending age, is CORBA. Once upon a time, CORBA was going to be the universal glue to take any client program, written in any language on any platform, and connect it to any server on any platform.

The nice thing about CORBA is that the same code works whether you’re running client and server on the same system, or running them connected across the Internet.

One of the reasons CORBA didn’t catch on is that most documentation makes it sound incredibly complicated. It really isn’t. You download the jar files, place them in your Java run-time’s path, then write Java code more or less as if you were writing a Domino agent in Java and accessing the database directly. CORBA abstracts away all the details of database drivers and network protocols.

The downside of CORBA is that the server needs to be reconfigured to support it, and the appropriate ports need to be opened for the DIIOP server task.

Although CORBA doesn’t require design changes to the database, you need to bear in mind the general performance issues for agent-based solutions–see below.

Java (notes.jar)

This is basically the same as using CORBA, but without the remote network access. You write a Java agent, and run it on the server with notes.jar in the JRE runtime path. You no longer need to configure the server for DIIOP access, but that’s because you now have to run your client code on the server. Same agent performance issues apply.

SOAP

Originally, SOAP stood for “Simple Object Access Protocol”. The acronym was quietly dropped when version 1.2 of the spec was released. I hear that SOAP is truly horrible if you have to worry about the details, but fortunately Domino abstracts most of that away.

I’ve used SOAP, but I don’t want to use it again. To find out why, you can head over to my posting from 2009 about SOAP and Domino. Basically, every time you make any change to the Web Service API, even one that should have no impact on existing remote method calls, you have to regenerate and recompile the code for every single client that uses the Web Service, and redeploy the changed client code.

If that doesn’t put you off, SOAP isn’t that bad of a solution. However, another minor limitation is that it doesn’t support the full range of Domino data types, so you end up having to pack data (including multi-valued fields) into Strings and then unpack them again. Not ideal.

JDBC

The standard Java way to access databases is JDBC. Theoretically you can access Domino data this way. However, it’s done using a JDBC wrapper over ODBC. This means you get all the disadvantages of ODBC, with the added fragility of a wrapper layer.

HTTP and XML

If you only need to get data out, and the database is web-enabled, there are Domino URLs to extract view contents in XML format. Basically, take the URL of any view, and replace the ?OpenView with ?ReadViewEntries. You should get an XML version of the contents of the view. Check the Domino help for more information.

The main problem with this approach is that you can’t get at rich text information. Also, the normal restrictions on number of view entries returned still apply, so you can end up having to page through the data via multiple HTTP requests.

While this technique doesn’t strictly need design changes to the database, if you happen to be lucky enough to find a web-accessible view that suits your needs, typically that won’t be the case.

HTTP and JSON

You can add Outputformat=JSON to ReadViewEntries URLs and get view data in JSON. Very handy for AJAX. Same issues as for HTTP and XML above, though.

Files in a directory

Domino agents can dump data into files in a directory. You can then have some other task on the server that does something with those files. You basically have an unconstrained choice of file format, so long as you’re willing to write LotusScript or Java code to generate the format you want.

The main downside is that you need access to write random files into a directory on the server. This means the agent has to be given unrestricted access. Also, the usual agent performance caveats apply (see below)

DB2

Domino 8 and up can use DB2 as a data store, instead of Domino’s own NSF storage format.

This is pretty much the holy grail as far as integration goes–you can use a Type IV JDBC connector to connect straight to DB2, do SQL queries at high speed, and so on. However, it requires that DB2 be installed on the server, that you be given access to DB2 JDBC services, and that Domino be configured to use DB2 as data store. I’ve never been in this lucky position yet, so I can’t say too much about how well things work.

C/C++

Lotus offers a C API. You can do pretty much anything with it that can be done in a Domino agent. The big downside, of course, is that you have to do it in C.

There’s also a C++ API, but it doesn’t seem to be supported for Domino 8.5 and up.

OLE

It’s possible to use OLE on Windows to access Notes/Domino data. This allows you to use pretty much any scripting language that runs on Windows, including Ruby, Python, Perl, and so on. The downside, as with ODBC, is that you are irrevocably wedding yourself to Windows.

Mathew’s hybrid HTTP and XML solution

So what do I do? I use a modification of the “HTTP and XML” approach.

Step 1 is to obtain a list of all the UNIDs of documents modified since the data pump last ran. To do this, the data pump calls a web agent, supplying a date and time in ISO 8601 format. The agent opens a view which lists all relevant documents by last modification, formatted as an ISO8601 date/time string. The agent uses NotesView.GetViewEntry(datetime, False) to find the first value which comes after the supplied date/time value in alphabetic sort order. Since ISO 8601 format sorts alphabetically into date/time order, this is also the first entry with a modification date/time larger than the supplied date/time value. The agent then iterates to the end of the view, printing the UNIDs of every document, which I put in the second column of the view so that the NotesDocument object does not have to be accessed.

I could do much the same using the ReadViewEntries URL, but if I did that I would need to make the client page through view entries N at a time, backwards, until hitting a cutoff date. That’s harder work.

Step 2 is that the client calls a second web agent, passing it the UNIDs one at a time in the URL. The agent uses NotesDatabase.GetDocumentByUNID to fetch each document and dump it out in XML format via NotesDXLExporter. Since Domino transparently handles persistent HTTP connections, this is very easy to code.

My approach needs two agents and a view. However, it doesn’t require any special access or server configuration changes. You can write your client in any language you like, and run it on any platform you like. You can also change your database design without breaking it. Obviously if you change field names or contents in a way the destination system doesn’t understand then you’ll break things, but nothing else should cause a problem.

Agent performance

There are caveats about Domino agent performance that apply to all of the data I/O solutions involving LotusScript or Java code.

Any access to a NotesDocument object causes the entire document to be loaded from the database. This is slow, so you want to avoid doing it unless you know you need to read that document–for example, to copy the data to some other system.

So for performance, you should try to make sure that you can work out which documents you need to load, by accessing only NotesViewEntry and NotesViewEntryCollection objects. Typically you’ll want to build a view which contains all the documents you want to access, in some suitable ordering.

If you’re planning to mirror all data into another system, for example, you’ll want a view ordered by last modification date. You’ll load a NotesViewEntryCollection from it, and iterate through the NotesViewEntry objects, looking at the ColumnValue array to read the modification date/time. If it’s more recent than the copy of the document in the other system, then you’ll access NotesViewEntry.Document to actually load the document from the database.

So, what if you can’t build a view, perhaps because you don’t own the database or don’t have design access?

One option is to use NotesDocumentCollection to iterate. However, this will basically mean pulling a copy of every document every time, so you’ll find that as the database gets bigger, you rapidly run into problems.

Another option is to use a search to try and narrow down the NotesDocumentCollection to only include documents you probably need to access. This involves using either NotesDatabase.Search or NotesDatabase.FTSearch to build your document collection. Of the two, Search is probably the best bet, especially if you don’t know whether the database will be full text indexed. It’ll be slow, but it probably won’t be as slow as reading every document.

So in summary:

Special case easy options

There are a few special cases where getting data out of Domino is incredibly easy.

• E-mail access: Use IMAP.
• One-time export of data from a view: File menu in the Notes client, Export. Select tabular text or CSV.
• One-time export of everything: File / Export again, and choose Structured Text. You’ll get something a bit like an RFC822 mbox.

Any options I’ve forgotten? Let me know.

Filed under: Domino, Java, Linux, LotusScript, Programming | Comments (0)

Engines of TeXnology

March 23, 2011

Back in 1978, when the TeX project began, there were no scalable fonts. If your printer supported 10 point and 12 point text, those were the two sizes you could use in your documents. Even when the Macintosh came along, you still had a fixed set of text sizes, unless you were rich enough to have a PostScript laser printer.

TeX, on the other hand, had METAFONT–a system for defining vector fonts, which could then be rasterized at any size, for any printer that could handle a bitmap image. So for years, TeX was the only way a lot of people could create documents using a wide range of typeface sizes and styles.

Then around 1990, Adobe Type Manager (ATM) went on the market. If you installed it on your Macintosh or Microsoft Windows system, you could use PostScript fonts on your desktop at any size, and ATM would handle the rasterization for your screen–and for your printer, if it wasn’t a Postscript printer. Suddenly you could have any font size you wanted, anywhere.

Fast forward 20 years, and vector fonts are ubiquitous; even my phone has vector fonts. Type 1, TrueType, OpenType… but not METAFONT. Like Plain TeX, METAFONT was way too complicated for most graphic designers. So nowadays, like everyone else, I want to be able to use the dozens of attractive vector typefaces installed on my computer, rather than the rather anemic selection of METAFONT fonts.

Then there was TeX’s way of getting output to the printer. First you ran your source TeX file through TeX itself; that spat out a DVI file with just the typeset text. Then you took the DVI file, and ran it through a converter which would insert the diagrams from their files, embed a set of fonts generated by METAFONT, and output a PostScript file. Then you either sent your PostScript file to your PostScript printer, if you were lucky enough to have one; or else you ran the PostScript through another converter (such as GhostScript) to turn it into your printer’s proprietary printer dump format (such as ESC/P or PCL, and copied that to your printer. It was all tedious but necessary in 1990. Today, it looks insane.

Then there’s the whole character encoding issue. Back in 1990, the only thing that could reliably be translated between different computer systems was plain ASCII text, so TeX had macros for curly quotes, accented characters, typographic dashes, dingbats, non-Roman alphabets, and so on. Nowadays, every major OS supports Unicode, so you can put Cyrillic or Greek text in your document on the Mac and be reasonably confident it’ll still be there if you open the file in Linux.

So TeX has been adapted to the modern world. pdfTeX got rid of the DVI stage, and went straight from TeX source files to PDF files; it later added support for microtypography. XeTeX got rid of METAFONT fonts, and allowed direct access to OS-installed vector fonts; it also added Unicode support. LuaTeX extended pdfTeX by adding Lua as an extra scripting language, allowing more complicated functionality to be supported than was feasible with TeX macros, and also added OS font support. LuaTeX has now been adopted as the successor to pdfTeX.

Since I wanted the most up-to-date Unicode and font support, I set about trying ConTeXt on Linux and Mac, using both LuaTeX and XeTeX. I created a document using a custom set of fonts–one OpenType, one Type 1, and one TrueType.

XeTeX was far easier to set up. Whereas LuaTeX requires that you set up environment variables and run a script to scan your fonts, XeTeX just calls the OS font routines. However, LuaTeX gets points because it’s the only supported engine for the latest ConTeXt release, Mark IV.

On the minus side, XeTeX on Linux is a bit buggier than on the Mac, as it’s a recent port; I had trouble with some Type 1 fonts. LuaTeX is not without bugs either: the ConTeXt mailing list recently discussed a bug triggered by a 5,000 page document, which caused LuaTeX to crash on page 3,987. That bug is fixed in a recent beta. Meanwhile, if anyone wants to try assembling a 4,000 page document in Microsoft Office, I’d be interested to know if it’s possible.

Since my documents are well under 3,987 pages, I’ve been happy with LuaTeX so far. So I had picked my TeX platform: ConTeXt running on LuaTeX. Now I had to sort out my macros and other necessary tools.

Filed under: TeX | Comments (0)

Which TeX?

March 15, 2011

When I posted that I was going back to using TeX, I mentioned that TeX had changed a lot in 20 years, but didn’t really go into too many details. Time to remedy that.

TeX is two layers of software. Underneath is the core of TeX, written in a variant of Pascal. These days it gets translated to C before being compiled to a binary. On top of the TeX core, you have a set of macros which provide all the handy \commands you use to typeset documents.

There are a number of different macro packages. Knuth’s own is known as “Plain TeX”; it’s what I used back when I last wrote TeX documents. It’s extremely flexible, and I managed to make it format my dissertation in a way that was so un-TeX-like that the examiners asked what I used. Unfortunately, Plain TeX is rather ugly to use. For example, here’s the code of a Plain TeX macro for placing two pieces of text side by side:

\def\xsplit#1#2#3#4#5{{
\setbox1=\vbox{\hsize= #1 #4}
\setbox2=\vbox{\hsize= #3 #5}
\ifdim\ht2>\ht1
\setbox1=\vbox to \ht2{\hsize= #1 #4 \vfill}
\else
\ifdim\ht1>\ht2
\setbox2=\vbox to \ht1{\hsize= #3 #5 \vfill}
\fi\fi
\hbox{\box1\hskip#2\box2}}}
\def\split#1#2#3#4{
\dimen1=\hsize
\advance\dimen1 by -#1
\advance\dimen1 by -#2
\xsplit{#1}{\dimen1}{#2}{#3}{#4}}

I remember spending most of a summer vacation poring over a copy of Knuth’s The TeXbook, slowly assembling my own set of macros. I based my page layout on that of Apple’s Macintosh Human Interface Guidelines. I had found the book particularly readable, and reasoned that Apple probably knew what it was doing as far as designing page layouts for technical documentation. Bending TeX to my will wasn’t much fun.

It’s a problem lots of people had. So to get away from all that, Leslie Lamport wrote an alternate set of macros called LaTeX. He provided standard templates for letters, technical articles, reports, books, and overhead projector slides. (Kids: Ask an old person what an overhead projector was.) LaTeX also had macros for bibliographies, tabular data, simple diagrams, indexes, and pretty much everything else needed for academic documents. Its book–LaTeX: A Document Preparation System–was about half the thickness of Knuth’s book, more focused on end users, and came with a handy quick reference card.

LaTeX spread through academia faster than Far Side cartoons. It was particularly popular with mathematicians, physicists, chemists, computer scientists, and anyone else who needed to be able to typeset mathematical equations. (It was also about the only way to typeset Tibetan in 1990, which led to my helping out some humanities students.) LaTeX is probably the most popular TeX macro package. There’s even a full graphical editor for LaTeX, so you can avoid the markup language entirely. If you have a mental image of what a TeX document looks like, chances are it’s the look of a standard LaTeX template.

I had toyed with returning to TeX a year or so ago, and picked up a copy of Lamport’s book in order to try out LaTeX. There’s a problem with LaTeX, though. If you don’t care about page design, it makes it really easy to put together a document that looks exactly as specified by its templates; but if you want to design your own page layout from scratch, you quickly enter a world of pain. That’s probably why almost all LaTeX documents look alike.

So I did some more research, and decided to try a newcomer to the TeX package market: ConTeXt. Its development started in 1990; it attempts to make TeX behave in ways familiar to people used to modern DTP packages. It’s more flexible than LaTeX, yet easier to use than Plain TeX. Here’s how you put two paragraphs side by side in ConTeXt:

\defineparagraphs[sidebyside][n=2]
\setupparagraphs[sidebyside][1][width=.45\textwidth]
\setupparagraphs[sidebyside][2][width=.45\textwidth]

\startsidebyside
First paragraph goes here.
\sidebyside
Second paragraph goes here.
\stopsidebyside

It’s still a bit long-winded, but that’s because it’s completely general. The first chunk is the setup; you can adjust the number of columns, define each column’s width differently, give different columns different text styles, and so on, and give each setup its own name and pair of \macros to apply to your paragraphs, as in the second chunk of text.

So, I had chosen a macro package; but there were more decisions to make…

Filed under: TeX | Comments (0)

Going back to something that works

March 10, 2011

Last week I had a bad experience with several pieces of office software.

It started with a simple enough task: I had some existing documentation, and I needed to extend the “How to perform common tasks” section. There were two sub-headings to add, each of which needed a few bulleted paragraphs of instructions.

I fired up LibreOffice, opened the document, and started typing–but something was wrong. When I clicked to turn my instructions into a bulleted list, the indentation was wrong. It didn’t match the similar bulleted lists above or below in the document.

I assumed I had mixed up the styles somehow, perhaps applying bullets to the wrong paragraph styles, so I checked the style of the correct paragraphs, then applied that style to the new ones. Still wrong.

Perhaps my correct paragraphs had somehow been modified? I selected both the correct and incorrect ones, and applied the style they were supposed to be in. They stayed determinedly different.

I tried fiddling with the rules manually, then re-applying the styles. The rulers reset to the tabbing defined in the styles, but the text continued to be indented incorrectly. I even tried creating a whole new style, in case the style definition had somehow become corrupt. Still the newly-type text would not match indents with the existing paragraphs.

Finally, I pondered the possibility that LibreOffice had a pretty severe formatting bug. I opened the document in IBM Lotus Symphony, which was forked from OpenOffice 1.x and (of course) reads and writes standard ODF documents. Hurrah! My paragraphs were all indented correctly!

I thought I was done. I made a few last edits, went to the top of the document, and refreshed the table of contents.

Oh dear.

Now my table of contents had garbage in, entries with a page number and no text, pointing at pages that had no headings on (including the front cover).

I cussed, saved the document, and quit Symphony. I opened the document in LibreOffice again, and refreshed the table of contents–now it looked right once more. I nervously scrolled down, and was somewhat surprised to discover that my paragraphs were all indented correctly. I hurriedly saved out a PDF before anything else went wrong.

So it was that I was forcibly reminded of how much I hate office suites, and WYSIWYG word processing in general. That well-known commercial office suite from Redmond is no better than the ODF gang; a few quick searches will unearth countless tales of horror involving corrupt files, misnumbered pages, font problems, and so on.

Meanwhile, I had another piece of documentation to start writing, and the thought of doing so in LibreOffice now filled me with a mixture of rage and dread.

I’m not a luddite, I quite like GUI software when it works. On the Mac, Apple’s iWork suite does a good job, at least with the kind of short document I find myself writing. However, I don’t have a Mac at work, so that wasn’t an option. So I returned to a piece of software I hadn’t used in 20 years: TeX.

Madness? Perhaps. I appreciate that TeX is perhaps not a tool for everyone; though it’s really no worse than word processors of the 80s, which ordinary people nevertheless learned to use. I suspect that most people just don’t have the patience any more. They don’t want to sit down and learn something, which is why we have Microsoft Word, and why hardly any office documents even use the style system properly. Which, I suppose, is how a major formatting bug gets into released code without being noticed.

It turns out that quite a lot has happened in the TeX world in the last couple of decades. Obviously computers have gotten a lot faster; I remember watching my 8MHz Atari ST grinding away at my dissertation. As I recall, it took about a second per page to typeset my text to DVI; I would then wait about 2 seconds per page to flip through and inspect the pages in the DVI viewer.

These days the console output scrolls past too quickly to read. TeX dumps its output directly to PDF, which loads instantly into an integrated PDF viewer, which of course supports antialiased text. Plus, my screen has four times the resolution. So even before considering software features, it’s a very different experience these days.

So what of the software? Modern TeX supports OpenType fonts, and accepts diagrams in PDF or bitmap format. It handles Unicode, so you don’t need to use escapes for accented letters and other special characters. It can place hypertext links in your PDF documents, and use color and other effects. And yet, TeX documents written in 1980 are not merely readable today; they can still be typeset into beautiful documents.

So from my point of view TeX has a lot of advantages:

• The source files are plain text, so I can stick them in bzr for version control.
• I can edit my documents with vim, meaning I can easily do things like searching and replacing control sequences and styles.
• The output is of higher quality than any office suite. CTAN has a sample of text typeset by Microsoft Office, next to the same text typeset by TeX. While the differences are subtle, the overall effect is that the well-typeset text is simply less fatiguing to read.
• Web page captures look a lot better in TeX than in a word processor. I can render a web page to PDF using wkhtmltopdf, crop out the bit I want, and pull it into my document–and because all the text stays as vectors, it’s searchable and looks good at any resolution. Much better than using a bitmap screen capture, and good luck pulling SVG or PDF into Office.
• Similarly, I can draw diagrams in Inkscape, save as PDF, and pull straight into my document. Much better than trying to draw anything with the tools provided in an office suite.
• I can be confident that what I write today will be readable, editable and printable for years to come, by anyone willing to put in the time to learn how to install and run TeX and edit a text file in a markup language that’s really no worse than HTML.
• TeX stays out of the way. It’s like writing HTML or wiki pages; you don’t need to mess with menus and mice to start a new section, you just type \section{My new section} and carry on.
• I can write my own macros. For example, I quickly built a macro for web site references. I provide the URL and title; the macro handles formatting the title to show that it’s a link, making it clickable, pointing it at the URL, adding a footnote, and placing the URL in the footnote in monospaced font for people who have printed the document.
• It scales. While office suites can start to crap out at a hundred pages, TeX can easily deal with thousand page reference manuals.
• It’s bug-free. Core TeX hasn’t had a significant bug found in it in years, and the length of the complete known bug list for the TeX Live DVD speaks for itself.
• It’s free software, it runs anywhere, and on today’s computers it’s fast.

If necessary, I can always turn my TeX into HTML, RTF, ODF, or whatever. But for now, I’m going to try writing new documents in TeX.

Want to see what TeX can do these days? Try some pages from a German chess book typeset with it, or take a look at sample spreads from humanities books typeset with TeX.

Filed under: TeX | Comments (0)

Java: Call by reference, or call by value?

January 18, 2011

My attention was drawn to a trainwreck of a discussion on Reddit, which was triggered by another trainwreck of a discussion on an enterprise Java community web site.

The vexed question: Is Java pass by value, or pass by reference?

My short answer, carefully phrased: Java passes both object references and primitive types by value.

I’ll now unpack that answer and explain it, and talk about why there’s so much confusion. To do so, I need to go back to the very early days of programming…

Once general purpose computers were built, it was quickly discovered that it would be really useful if you could call a chunk of code multiple times, from multiple places.

For instance, most early CPUs (including popular 8 bit CPUs like the Z80 and 6502) didn’t have a general purpose multiply instructions. It was very useful to be able to write a piece of code that would multiply two numbers together and return the result. Because you’d want to multiply numbers a lot from different parts of your program, there needed to be a standard way for the code to know where to jump to when it was done computing the answer for you.

In the early days, this was done in one of two ways. One was to have the caller put a return address into a register. When the multiply routine was done, it would jump to whatever address was in that register, resuming the main program, which could then use the result of the multiplication.

The PDP-8 used a slightly different approach: the instruction to call a subroutine would store the return address in a reserved piece of memory immediately before the subroutine. The subroutine code would be called, would find the return address immediately before it in memory, and could therefore jump back there.

The problem, of course, is that soon programmers wanted their subroutines to be able to call other subroutines, or even call themselves. Both of these methods of passing return addresses failed to allow that. The PDP-8 approach was also ugly because it meant you couldn’t store subroutines in ROM.

So by the time of the PDP-11 and 6800, the call stack had been invented. With a call stack, when you called your multiply subroutine the address of the next instruction after the call was placed on the call stack automatically by the CALL instruction. The processor then jumped to the subroutine code. The code did its multiplication and put the result in a register as before, then pulled the address off the top of the stack and called it–hence jumping the CPU back to the point immediately after the CALL instruction. Popular 8-bit home computer CPUs including the 6502 and Z80 had hardware call stack support.

This was a big improvement, but still not ideal. The problem was, the subroutines and the code that called them had to agree on which registers to use for which values before the call, and which registers to find the results in after the call.

So later CPUs, like the 68000, added instructions to make it easy to push additional values onto the stack–not just return addresses. Now to call your multiply subroutine you’d push the two values onto the stack and CALL the subroutine. The subroutine would pull two values from the stack, multiply them, and push the result onto the stack before returning. Your code would then pull the result from the stack and use it. The combination of return address and subroutine parameters was called a stack frame. This call model was the one used on the PDP-11, and hence became the call model for the C programming language.

Now, stack-based parameters work well, but there’s a potential problem once you start dealing with values that can occupy a large amount of memory–such as strings. How do you pass a string to a subroutine?

This problem had been tackled before the invention of call stacks, of course. The general answer was that rather than copying the data around everywhere, you would just pass around a pointer to the start of the string. Not only is it a lot more efficient, you also don’t use up as much stack space–and in early computers, stack space was a significant concern.

So in most languages, when you passed an integer and a string to a function, what actually happened was that the value of the integer was placed on the stack; but a pointer to the string was placed on the stack, rather than the value of all the characters in the string itself. These two approaches were named ‘call by value’ and ‘call by reference’, respectively. (For example, see “Semantic Models of Parameter Passing“, Richard E. Fairley, 1973.)

When object-oriented programming was invented, the exact same issue applied to objects. The value of an entire object, with all its methods and fields, could be very large. Rather than placing the actual value of the object on the stack, instead a reference to the object was placed on the stack.

For now, though, let’s stick with strings. When you pass a string to a C function, the function may be able to change the text of the string, but it can’t change the fact that your string variable will still be looking for the string at the same memory address once the subroutine returns.

So at the language level, function parameters are conceptually passed by value, even though those values may actually be the addresses of an object or string. Hence K&R states:

One aspect of C functions may be unfamiliar to programmers who are used to some other languages, particularly Fortran. In C, all function arguments are passed “by value”. This means that the called function is given the values of its arguments in temporary variables rather than the originals.

This description is clear for C, because C always makes pointers visible. Your string is manipulated using a char * pointer variable. It’s handed to the subroutine as a char *. It’s clear that the subroutine gets a copy of that pointer, not access to your pointer. Whatever the subroutine does, the caller’s pointer variable will still point to the same address when the subroutine is done.

Java handles parameter passing the same way in the JVM. Primitive values such as ints have their values placed on the stack; objects have the a copy of the address of the object placed on the stack. The called code can mess with the contents of the object, but when it returns, your variable will still point at the same object. That is, strictly speaking in Java the value of a variable is which object it refers to, not the value of the object.

This is, of course, a highly unusual definition of “value”.

James Gosling goes to the bank and finds all the money has been emptied from his account. When he complains, the teller says “But your account has the same value it had yesterday–your account number still references the exact same account!”

Another reason why people get confused about Java’s call semantics is that you never actually see the addresses of objects. Unlike in C, where you can mess with pointers (and indeed have to), Java hides that detail of the machine-level implementation. You’re simply not allowed to access the JVM’s pointers in the high level Java language. Variable names which are bound to the address of an object are automatically and invisibly dereferenced to the referenced object when necessary.

As I put it in my earlier article about final parameters, “The Java Thing a is really like C++ Thing *a, and Java’s a.method() is really C++’s a->method(), not a.method().”

So in Java, you don’t need to explicitly dereference your object variable in order to call one of its methods; the compiler does it for you.

So when you pass a bunch of variables to a method, Java passes both the variables which were assigned object values and the variables which were assigned primitive values by value; it makes copies of all of them to give to the method code. It’s just that the variables which were assigned object values actually containing references to the object data, which will be automatically dereferenced when necessary. Most of the time this is very convenient, and it makes the language a lot safer by prohibiting various kinds of misbehavior with pointers.

So in Java,

int x = 3;
String y = "three";

are two totally different operations. They only look like they’re the same concept because of some syntactic sleight-of-hand performed by the compiler.

If someone asks you what the value of y is after the above code runs, you might say that it’s the string “three”. In fact, though, the true value of y is a reference to the string “three”. You just never see that true value.

In C, on the other hand, the difference is explicit:

int x = 3;
char *y = "three";

The asterisk is the clue that two different sorts of things are being done here.

This distinction also comes up when considering the == operator. Java’s == compares the true underlying value of variables; x == y only if x and y are both references to the exact same object. To see if x and y have the same object value when dereferenced, you need to use a method such as equals() or compareTo(). Worse still, the default implementation of equals() is ==, unless the class overrides it.

So what would call-by-reference look like? Well, in C we can do this:

int x = 3;
f(&x);

This calls the function f, passing it a reference (the address of) the value in the variable x. The function is thus able to change the value in x, as it is then seen in the calling code.

This brings up a second reason why the discussion on Reddit is a trainwreck. People say that a language is call-by-value when a subroutine cannot change the value of a variable passed as a parameter. Unfortunately, because Java invisibly dereferences your object pointer variables, that’s not really true:

static void f(StringBuffer o) {
  o.delete(0, o.length());
  o.append("This is impossible.");
}

public static void main(String[] args) {
  StringBuffer x = new StringBuffer("Java is call by value.");
  f(x);
  System.out.println(x);
}

In the above code, f() clearly changes what we’d colloquially call the “value of x”, as we see it at Java language level. However, it does so without changing which object x points to. To rephase in Java language specification terms, before the method call x is a reference to a string object which has the value “Java is call by value.” After the method call, x is still the same reference to the same string object, which now has the value “This is impossible.” But again, you never see the references which are being compared, so the situation is confusing.

Or as I wrote in Java annoyance: final parameters:

It seems that the designers of Java wanted to make sure nobody confused their object pointers with the evil manipulable pointers of C and C++, so they decided to call them references instead. So Java books usually say that objects are stored as references, leading people to think that objects are passed by reference. They’re not, they’re passed by value, it’s just that the value [on the stack at JVM level] is a pointer.

If you’re still confused at this point… well, I can’t say I blame you. We should probably both blame James Gosling.

Filed under: Java, Programming, Uncategorized | Comments (0)

Business Conduct Guidelines

January 18, 2011

Every year, IBM has all its employees read through its Business Conduct Guidelines, certify that they have read them and will live by them, and take a quick quiz. It sometimes feels like an unnecessary chore, but there are usually a couple of sections that I find cheer me up when I read them.

The first is this one, which seems to get longer every year:

IBM strives to maintain a healthy, safe and productive work environment which is free from discrimination and harassment, whether based on race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or any other factors that are unrelated to IBM’s legitimate business interests.

The second is this one:

IBM will not make contributions, payments or otherwise give
any endorsement or support which would be considered a contribution, directly or indirectly, to political parties or candidates, including through intermediary organizations, such as political action committees, campaign funds, or trade or industry associations. [...] In many countries, political contributions by corporations are illegal, though IBM will not make such contributions even where they are legal.

You must not make any political contribution as a representative of IBM. You may not request reimbursement from IBM, nor will IBM reimburse you, for any personal contributions you make.

Even before the Supreme Court blocked limits on corporate political donations, it seemed to be taken as a law of the universe that corporations would naturally want to funnel cash towards politicians in order to get favors. In a world where some employers even go as far as telling employees who to vote for, it’s refreshing to work for a company that stays out of the political murk.

Filed under: Business, Career | Comments (0)

Multi-line comments considered harmful

December 31, 2010

Recently on the ruby-core mailing list, someone asked why Ruby doesn’t have multi-line comments. I did a few searches, and didn’t find much discussion of the evils of multi-line comments. I thought it was common knowledge that they were a bad thing and a historical mistake made by Kernighan and Ritchie, but apparently not. So, here’s my opinion on the matter.

Superficially, multi-line comments seem like a handy thing to have, especially if you often comment out blocks of code temporarily during development. However, as soon as you allow multi-line comments, you need to decide what to do about nested multi-line comments–that is, what should happen when the ‘start comment’ sequence occurs inside a comment.

Option 1 is to allow it and make it not special. This is bad because it makes it very easy to end up with a run-away comment:

/* set variables
param = 0
/* set maximum size */
maxs = 100

Here ‘param’ ends up unset. This kind of bug can be really hard to locate.

Option 2 is to allow the start-comment sequence within a comment, and make comment pairs stack, so you need as many end-comment sequences as start-comment sequences. This at least makes errors like the above cause a compile-time failure, but it still makes them a pain to locate; you end up having to use something like vim’s ‘%’ command to play match-the-comment-delimiters until you spot which pair is mismatched. That’s no fun, especially once your comments contain pages of documentation. Ask a Java programmer…

Option 3 is to disallow nested comments, in which case multi-line comments can’t be used to comment out blocks of code which contain multi-line comments. That makes them pretty useless for temporary disabling of code; it would also cripple their use in rdoc.

Single-line comments have none of the above problems. You always know the comment will end when the next line starts. You can always comment out a block of code, no matter how many comments or comments-within-comments it contains. You never get runaway comments, and you never have to match up comment delimiters. The only downside to single-line comments is that it’s harder to comment out and un-comment-out multiple lines. That’s easily solved by using any good text editor.

All of which is why C++ introduced single-line comments, and C99 added them to regular C.

So in short, multi-line comments have no significant advantages, make parsing harder, and make errors more likely. I think Matz was entirely right not to put them in Ruby. As a rule, the only place I use multi-line comments is for JavaDoc, and that’s only because I have to.

Filed under: Annoyances, Java, Programming, Ruby | Comments (2)

RFC 5424 syslogging on RHEL 5.x

November 30, 2010

Here’s how to get non-broken syslog output on RHEL:

1. yum install rsyslog
2. Modify the appropriate line of /etc/sysconfig/rsyslog to say SYSLOGD_OPTIONS="-c3"
3. Insert the following two lines at the top of /etc/rsyslog.conf:

   $template RFC5424fmt,"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%\n"
   $ActionFileDefaultTemplate RFC5424fmt

It’s probably then a good idea to disable the old syslog. You can also use rsyslog to replace klogd. Here are some other suggested additions to /etc/rsyslog.conf:

$ModLoad immark # provides --MARK-- message capability
$ModLoad imudp # provides UDP syslog reception
$ModLoad imtcp # provides TCP syslog reception
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)

Filed under: Linux, System administration | Comments (0)

Useful RPM one-liner

October 22, 2010

To get a list of all the software installed on your system, in descending order of bloatedness, with descriptions:

rpm -qa --queryformat="%10{size} %{NAME} - %{SUMMARY}\n" | sort -k1,1nr

Filed under: Linux, System administration | Comments (0)

Mandelbrot remembered

October 17, 2010

Like many other people, I first saw the Mandelbrot Set in an issue of Scientific American in 1985. Before long I had written a program in BBC BASIC to perform the computation, and set my BBC Micro’s 2MHz 6502 churning away. Because it took so long to render an image–even at the Beeb’s meager 256×160 resolution in 8 non-blinking colors–I made the code preview by rendering every 16th pixel, then every 8th, and then finally fill in all the points. As well as Mandelbrot and Julia sets, I experimented a lot with Lindenmayer systems and generation of realistic-looking plants.

About a year later I upgraded to an Atari ST, with an 8MHz 68000 and the ability to handle 320×200 resolution in 16 colors. I had Computer Concepts’ FaST BASIC, a version of BBC BASIC on a cartridge for the ST, and soon had fractals rendering on the Atari. I built a fairly elaborate menu and dialog system for adjusting parameters, and added mouse-based selection and zooming. Rendering speed was improved, but it still took a few minutes to put together a complete image.

The trend continued over the next decade: every time I got access to a newer and faster computer, one of the first things I would do would be to see how quickly it could draw fractals. This continued until the mid 90s, at which point I could move my mouse around the Mandelbrot set and see the corresponding Julia sets drawn in real time.

By then, fractal imagery had become commonplace on album covers and posters. My fascination waned. Yet Benoît Mandelbrot’s influence stayed with me; his fractal geometry had changed my view of the world. One single simple equation, simple enough for a teenager to understand–and yet, hidden inside it was infinite complexity. Simple iteration could generate images that looked like objects from the natural world. Entire universes could be encoded in a tiny algorithm, if you were clever enough.

Filed under: Programming | Comments (1)