XML History

In the late 1960s, largely as a result of the Multics time-sharing research program, IBM had a research lab in Cambridge, Massachusetts, near MIT. That lab was engaged in lots of what became important work: some of the origins of modern operating systems, the database, and many languages have a historical thread that goes through MIT and this lab. In 1968, a problem assigned to the lab was how to mix “programming instructions” with document text to allow legal documents to automatically be indexed, and composed (typeset). IBM had lots of experience in this field; indeed, their “business machines” included compositors before they developed computers.

I visited this lab as a computer science student in 1967 and 1968, where they were developing something that the next year would be tagged “Generalized Markup Language.” The original developers claim that the name was derived from the initials of their last names (which are G, M, and L, at least during a key period), but I seem to recall the name predating the formation of that group. The idea was to intersperse notes among the text as elements of computer instructions that would be “compiled” for layout and publishing.

You have to recall that this was in the center of the Lisp universe in its heady days. Lisp was arguably the most used for research and certainly the most advanced language of the time. (Many still claim so.) Lisp is built on the notion that data and programming instruction are comingled; no, that’s not quite true: in Lisp they are actually the same.

GML appeared in the midst of similar ideas that were popping all over the place, most notably in the Graphic Communications Association (GCA) Composition Committee. But instead of using the strong notion from Lisp where the “tags” were similar in form to the text, they used a weaker notion where the text was one kind of thing and the “tags” were more computer-like,

IBM in those days was both savvy and agile, and in a very short time IBM expanded GML to a product line (“Document Composition Facility” also sometimes “Framework”) for many publishing purposes. At this point, another family tree thread diverged that led to GML-derived codes that formed the basis of Wang Laboratories and the development of so-called “word processors.” But that’s another story.

Independently, but with the backing of IBM, news syndicators (such as Reuters, Associated Press, and the like) invested heavily in a markup language using GML to transmit structured stories. They drove an international standard that resulted in the Standard Generalized Markup Language (SGML), ISO 8879 in 1979. SGML by that time (and since) was seen as a robust means for structuring documents in such a way that content and presentation could be managed separately. It was pretty much the only game in town for the heavy-duty document crowd.

Meanwhile, over the next decades, various much lighter weight proprietary tagging schemes popped up all over the place as the basis for numerous word processing and desktop publishing products. Over time, the complex and costly SGML became more and more marginalized for high end users.

Then two things happened that saved SGML.

The first was a project in the US Department of Defense (DoD) that was dealing with a financial disaster. They bought complex systems in huge amounts, more than any other enterprise in the world. All of these needed manufacturing, training, and repair documents that were typically included in the system procurement. However, as the systems evolved, the paperwork lagged, often by years. And we’re talking a lot of paper here: the typical submarine for example had associated paper that weighed more and cost more (overall) than the sub itself.

Many, many billions of dollars were being wasted by out of date paper, perhaps tens of billions a year, so the idea was to digitize the documents. The very ambitious project to do this was called Computer Aided Logistics Support (CALS). I was at the Advanced Research Projects Agency (ARPA) at the time advising CALS. I lobbied against adopting SGML as a CALS standard (holding out instead for something simpler and more consistent as I’ll explain later, but with process state), but in 1987 SGML became a defense requirement under CALS. Billions of dollars poured into SGML compliance (which continues today). Naturally, this put new life into the tool base. The defense and intelligence world also sponsored lots of research—real research—to extend SGML to support complex structured documents, particularly hypertext.

I wish they had gone with my recommendation to create a new markup language that used the same syntax to specify the markup elements that it used in applying those elements. (This goes back to the strong Lisp-like idea.) As you’ll see in a moment, SGML lacks this elegance and the specification of just what the markup is has turned into a nightmare of conflicting and increasingly complex notions that makes life hard for us ATPOers. If CALS had not adopted SGML, then it would have died a natural death to be replaced by something better.

Our whole Web and document world is a result of this bad decision.

CALS spawned research into SGML-related hypertext schemes. Much of that hypertext research was conducted by the US intelligence community in projects that may never be detailed, I’m afraid. (I was involved in many of these.) Several hypertext conferences were held, starting in 1987 where some parts of some of these projects were reported. In the next couple years, several proposals were made for a hypertext language either extending or subsetting SGML. One of these took off, the simplest—HTML, Hypertext Markup Language—imagined by Tim Berners-Lee in 1989 and implemented in 1991.

That version took off because it was incredibly simple, an implementation existed, there was a preformed (scientific document) user base, there was the Internet transport vehicle in place (for many years), and the language wasn’t proprietary. Also, so the story goes, Berners-Lee credits an amazingly capable development system for the time in NeXT, the precursor of OS X. HTML looks and acts just like SGML but with a miniscule number of operators and no need for a separate “language” to specify it (because of its simplicity).

Needless to say, as with all cheap and easy solutions, the compromises catch up with you. So in more recent time, the now huge Web community forced a revisiting of SGML to invent another variant in complexity between SGML and HTML. The result was XML, the eXtensible Markup Language, what HTML (and in some respects SGML) should have been from the beginning.

XML has been since 1998 a standard for documents (vastly eclipsing SGML in the user base), database exchange and the reinvented XHMTL (HTML as an XML application). Along with XML are a bewildering array of associated languages, formats, transforms, and such. Adding to the confusion are the hundreds of “standards” (like OPML, RSS, SOAP, and the several application-specific XML formats—Tinderbox, NoteTaker, NoteBook, OmniOutliner, OpenOffice, and now iWork—that employ XML) that we in the ATPO world encounter.

Just as a historical note, in the area of information science, the entire defense and intelligence research community collapsed about a decade ago, and basic research of this kind is now in the hands of others.

About XML

If you are like me and the median ATPO reader, you need a gentle introduction to just what this beast has turned out to be, probably with some hand-holding.

XML “stuff” is found in as many as three places. One place is interspersed within your document as tags, saying for instance that “this bunch of text here is what we call a ‘title’.” If you look in an OPML file from one of our outliners, these are easy to see. It looks just like HMTL tags, which many of us have encountered.

HTML Source

The screenshot shows a rudimentary Web page and the simple HTML that generates it. The tags are those things in angle brackets.

There might be XML-related code in a second area describing in a structured way just what you meant when you said “title” in the document. Often there are complicated rules that designers specify about the relationships among all the elements that are defined: for instance you may specify that a “subsection” must be part of a “section” and there must be at least two subsections in that section for it to exist.

This part is sometimes prepended to the beginning of a document, or more often in a separate file that is used over and over. This can be pretty hairy stuff, and can drive even experts crazy. The official name is DTD, Document Type Definition, by which they mean the definitions of the “types” (the entities in the tags) in the document.

Whether formally specified or not, Tinderbox will have its own version of these definitions, as will NoteTaker, and all the others on the list. The problem of course is translating from one to the other, and what to do when one has some elements the other does not. And that’s the case between any two interesting outliners on the ATPO list.

DTDs are so hairy in fact that many XML users don’t want to even tinker with them. So they’ll use a standard one depending on the domain of application. In the document world, DocBook is a common one that many tools support. Many disciplines have DTDs which you can explore on XML.org.

Most applications that use XML (including OPML originally) don’t bother with a DTD in fact, assuming that the structure is obvious, or explaining it in plain English.

DTDs might be created (if they are done right) using another XML-related standard, the XML Schema language which provides a means for specifying the thing. There are lots of other competing standards in the DTD space which we won’t mention. Suffice it to say that it is complex, contentious, and confusing.

This is a legacy of basing XML on SGML, and using two completely different languages in the specification: one for the tagging and another to define the structure of the tags. You can blame your ATPO exchange frustrations on the Department of Defense if you want. I do.

The third area you might find XML-related stuff is in the actual presentation of the file. This won’t matter so much to ATPO readers. Applications that natively use XML (iWork, OmniOutliner, OpenOffice, and Tinderbox) have their own proprietary means for producing what we see. The other outliners translate into XML for export. But they could have used XSL-FO (Formatting Objects) as a standard way of specifying appearance.

Our problem—the problem of getting your power outliner XML from a mindmapping tool, to an outliner, and on to a page layout program and back, or getting your outliner to dynamically import from your database—is in the DTD side. Fortunately, there is another part of the XML standard that can help us. XSL, the Extensible Stylesheet Language is yet another language (!) whose purpose is to translate XML from one format to another. XSL is a family of specifications using the same language. We already mentioned XSL-FO, but there is a sibling specification called XSLT, XSL Transformations.

Whew, that’s a lot of acronyms. There are lots more where those came from, and all of it is unfriendly.

An ATPO user who wants to tie applications together has a few choices:

1. Just use OPML as is. It is a simple specification in XML that just about everybody supports. You don’t have to know what’s in it to use it as an intermediate format among applications of interest.
2. Use XML, taking the XML format of one application and translating it to another in an ad hoc manner. Maybe this would be useful if you plan, for example, on spending a year writing a book in OmniOutliner and you want it published in Publicon, so you know the originating and receiving applications.
3. Do number (2) but do it in a reusable, wholesale but more expensive way using the XSL Transformations language and perhaps some associated tools.
4. Do number (3) but have the work already done for you for each application in the ATPO Transformation specification, but you still might want to tinker with how features from one application translate to another.

Today, we’ll just introduce OPML files.

OPML Files

The good news is that OPML files are easy to look at and understand. The bad news is that they are useless for real outliner integration.

Here is a simple outline in OmniOutliner Pro:

OmniOutliner Example

and here’s what it exports as in OPML format:

OmniOutliner OPML

It is a bit more complicated than HTML, but most of it is pretty obvious, right? That bit at the top about “expansion state” records which headers were collapsed. In our case, none of the subheaders were collapsed and three were expanded. An OPML file from Frontier would have lots of other stuff in the header section, like owner and date.

But look at this OmniOutliner document, with a column:

Enhanced OmniOutliner Example

and its associated OPML file:

OPML file with Columns

Before, you may have noticed that the outline’s notes weren’t an element (a separate tag). Instead each note was an annotation within an outline element. Here, OmniOutliner has done the same thing with the column. OPML, as it turns out has no definition for a note because Frontier doesn’t do notes. Developers just know what a “note” is when annotated to an OPML outline element (and because OmniOutliner is a leader of sorts).

But “column 1” doesn’t mean anything to a importing application. It wouldn’t know what it is; in fact I could have named that column anything. OmniOutliner only puts it there so that if anyone saves an outline in OPML from OmniOutliner and opens it again in the same application, they can reconstruct the document. That’s because OmniOutliner knows what a “note” is and assumes anything else is a column.

Bad news, right? When this second example is opened in NovaMind for example, it has to import the column values as if they were notes, together with the actual notes.

OPML is best for “flat” outlines where all the information is in headers and there is no separate “notes,” “comments,” or “paragraph” type. This is the case with NoteTaker and Circus Ponies’ NoteBook.

NoteTaker exports “category” and “priority” in the place where OmniOutliner puts column information. OmniOutliner sees those as columns, which of course they are, and displays them correctly when imported. I haven’t taken the time to explore all combinations of OPML-capable outliners, but my impression is that NoteTaker and OmniOutliner are the only ones that handle “column” data.

One approach to a new standard would be for us to specify a standard collection of attributes for the OPML element, essentially expanding the standard. We’d have to have internal links, specific column types with attributes (like priority, start date, owner, cost, and others with data types: numbers, string, graphic, etc.), outlines within notes (Hog Bay does something like this in its OPML), folded state, styles, and clones.

We could fix OPML. Or we could start fresh and do it right, like the CALS folks should have done. In our case, that would mean a new XML specification with accompanying DTD.

If there’s interest in this, we could continue in future columns to explore how to understand and tweak outlining-related XML.

ATPO Tracker