Programmers and word processing

The other day, I was thinking about the mechanics of writing. Not grammar, spelling, and punctuation, but the actual work flow involved. Specifically, I was trying to gather my thoughts on why writing a software book using Microsoft Word is so incredibly painful. (Granted, I have used Word or similar tools to write fiction and shorter pieces.)

It wouldn't seem so at first, would it? Isn't word processing software the savior of the writer? Wouldn't we be using typewriters without it?

Well, yes and no. Several facts have to be considered here: 1. I'm a programmer (and a somewhat old-fashioned one, who almost always goes straight to the command line). 2. When you write a programming book, there is often a need to manipulate the text in various ways you otherwise wouldn't. This is even more true for a "larger" document like a book of a few hundred pages. 3. Programmers by nature want the computer to do things for them. It's natural to want to write code to manipulate your text.

the typewriter and the modern word processing suites. It's a forgotten generation of software -- forgotten partly because computer guys were mostly the only ones who ever used it, and today even the younger software professionals don't remember it.

I'm talking, of course, about what we commonly call a text formatter. Many of you who read this will have much more experience here than I do, but I'll share from my own limited knowledge and experience.

I'll assume some of my readers have no idea what I'm talking about. A text formatter is just a piece of software that takes plain text (mixed with some metadata relating to formatting) and produces a formatted document. Think of it as a little like HTML, but usually without nesting. Typically if a line of text started with a dot, it would be interpreted as a formatting instruction rather than as raw text.

This was before there were laser printers (or at least I had never heard of them). Although I don't go back to the early days of Unix utilities like troff and nroff, I did know about them and have touched them (in the early to mid-1980s). I've even used a similar tool on an IBM mainframe (called Scripsit, as I recall). I'm certain there were similar things on the DEC-10 and many other forgotten platforms. Even in the late 80s, when word processing was coming into its own, many of the tools still were not fully "WYSIWYG."

Remember that term? "What You See Is What You Get"? It was coined because this was rather a new concept. With text formatters, what you saw was not what you got.

Even back then, there were more sophisticated tools. TeX was around, though I didn't know it and never really saw it. LaTeX I suppose came later. At that time, dot matrix printers were not uncommon -- and on the mainframes, even line printers were still common (typically without even lowercase letters).

As for "really fancy" output such as TeX could produce, I don't know that our comp sci department or our computer center could really print such material. If I used a printer that could print italics and boldface text, I was ecstatic.

This is starting to sound like the stories about walking to school in the snow, right? And it was uphill both ways? That's not my point. The hardware was primitive, yes, and the software was primitive, too.

When WYSIWYG came along, suddenly armies of "ordinary people" escaped their typewriters and learned the joys of backspace and delete. Programmers benefited, too. And yet...

I won't even go into the disadvantages of the GUI in general. That could be a controversy in itself. Suffice to say that I earned two degrees in computer science and hardly even touched a mouse. In fact, I don't think I actually owned a mouse until four or five years later. I remember with fondness the online signature of one Arno Schaefer: (How do I type) ("for i in *.dvi do xdvi i done") (in a GUI?) But that is a little outside our discussion here.

From a programmer's perspective, a primary failing of the word processor in general is that it stores documents in a binary format rather than text. It is difficult or impossible to peer into those files from the outside.

That means that all of the traditional Unix utilities, so useful for text files, are useless for Word documents and similar file types. Which utilities? That depends on how well you know Unix and how much time you have spent at the command line.

The first utility, of course, is the text editor itself. If you want fancy formatting and WYSIWYG, the traditional editors like vim and emacs won't seem that great to you. But if you want something "lean, mean, and easy" -- maybe these are pretty great after all. (Granted, nothing ever seems "cool" until you learn to use it effectively.)

Personally I use vim (although I freely admit that emacs is more powerful, just as I admit LISP is more powerful than any of my favorite languages). It loads into memory very quickly, it loads files quickly, it saves them quickly, and it never crashes. Navigation is fast and simple (again, if you know what you're doing). I've never done real scripting in vim, but I have sometimes used macros and such.

But the editor is only the beginning. Some of the utilities I have used are: grep, wc, fold, cut, colrm, diff, sdiff, csplit, expand, head, tail, less, more, cut, sort, split, strings, tee, tr, and uniq. Sometimes I have even used awk, sed, and m4. I can combine files, split them, search them, and manipulate their contents at will.

I can redirect standard input, standard output, and standard error for these utilities. I can use pipes with them. Many of these tools understand regular expressions. I can build scripts that perform multiple commands, even with some simple branching and looping logic involved.

None of these things are possible in general with Word documents. None of these tools or techniques will work well with such a file format.

But let's go a step farther... I'm a programmer, after all. I principally work in Ruby, which is good for text processing (far more powerful and friendly than bash or awk, for instance).

I have written programs that could search and/or replace only in the body of the text, only in the source code, or in both. I have written programs to extract code fragments from the text, to insert them, to syntax-check them, and to format them. I've written programs to create test cases, to syntax-check code, and to reorder (and renumber) sections and other items. I've written code that will honor the file format (and code fragments) and spell-check the document, naturally honoring all the technical terms and abbreviations unique to my topic. I've written code to extract a table of contents, lists of figures, and lists of code listings. I've written code to compare the sizes of different sections, code to check cross-references, and code to suggest keywords for the index.

Is all this overkill? You could argue that. Some of these things are doable in MS Word, after all. But you have to do everything their_ way; you can't tweak or customize the functions; and you can't turn off all the features you don't need or want. Furthermore, you can't automate it, you have to use the mouse, and sometimes it just crashes or malfunctions for a reason you may or may not figure out.

There are many solutions, of course. If you know TeX, that is probably the ultimate tool; and there are add-ons such as LaTeX and Lyx to make it easier. I've never learned any of this; for most of my purposes, it would be a case of killing a mosquito with a hand grenade. On the other hand... if I were writing a calculus textbook, as I never will, I would want something like that to handle all the mathematical notation.

As a side note, I've never used Lyx. But I've heard it is a nice compromise between the two paradigms -- a WYSIWYG editor that also offers low-level access to the raw markup.

What's interesting is that people often "roll their own" solutions. I have done it myself (although I principally worked with plain HTML), and I have seen it done by others more than once. It's natural, as programmers are always looking for an easier or better way to do things.

Once in a while, a programmer will build a solution that is also intended to benefit others who think the same way. An excellent example is the relatively new Softcover_, created by Michael Hartl (https://www.softcover.io/). I've only begun to learn it myself, but my initial impression is that what troff was in the 70s, softcover will be in only a year or two. It looks to me like a powerful text formatter for the 21st century -- a tool that will let the author/programmer do what he wants without getting too much in the way.

Softcover was designed (in my opinion) the "right way" -- that is, it was first built for a specific purpose, and during that process, ideas were generated, and lessons were learned. This enables the developer to avoid false moves, design flaws, and "dead ends." Remember that, in the same way, Ruby on Rails was first an application (Basecamp, as I recall) before it was abstracted and generalized into a framework.

Softcover does other things right as well. It isn't just a tool or framework, but also a publishing platform for ebooks. There is at least one print book published this way as well, and I expect more.

When programmers write, the readers are often programmers as well. And we are certainly (on the whole) heavy consumers of electronic reading material. Only one American in four uses any kind of ebook reader; but do you know of even a single software person who doesn't use one?

So it's a tool, and it's a publishing platform. The business model permits the author to keep a high percentage of the sale price of the book. All this assumes, of course, that the author is actually capable of producing quality material without the help of the host of individuals at a traditional publishing house. But I've noticed that this often can be done, especially if the author seeks a little outside help now and then.

So I see Softcover as a tool, a platform, and community that connects writers to readers (and to each other). You might consider checking it out.