What is a Zipfian ordering?

I probably shouldn’t toss Zipf’s name around so casually, as the concept here isn’t 100% related to his studies. But as Zipf was (to a certain extent) a crackpot anyhow, perhaps it won’t matter. (Refer to: George K. Zipf, Human Behaviour and the Principle of Least Effort, Addison-Wesley, Cambridge MA, 1949.)

In linguistics, if we do a frequency spectrum of word usage (a ranking of words by how commonly they are used), we find that this is a nice, monotonically decreasing function. Roughly speaking, the frequency of a word is inversely related to its rank in the list.

A related principle is Benford’s Law, in its own way even more fascinating. Simon Newcomb first discovered this principle in 1881 when he noticed that the earlier pages of logarithm books were much more heavily used than the later ones.

We use this "inverse" relation in daily life and in computer science all the time. In Huffman encoding, we assign fewer bits to the more frequently occurring symbols, and more bits to the less frequent symbols. (If it were not for this convenient "trailing off" effect — if all symbols were equally likely — then the text would not compress at all.)

In daily life, whether you are a mechanic or a chef, you use tools; but you don’t ‘t use them all with equal frequency. The efficient worker in any field keeps the most commonly used tools within easy reach, at a distance inversely proportional to frequency of use.

You see where I’m going with this. I want to apply this to user interface design. Of course, it has been already; in GUIs, we routinely measure the difficulty of acquiring a target in terms of its distance and its size.

But where lists of items are concerned, user interface designers haven’t done their jobs. Consider the typical "Windows explorer" type of interface where a tree view is shown in the left pane. A list of 1000 items (or even 100) is too large; it has to be managed in some way. The usual solution (as in an image browsing program I have) is to alphabetize the items.

But what are the chances that my usage patterns will follow the alphabet? Nil, I assure you. As I now open my image software and count the top-level directories, I find there are 46 of them. As I look for the two I use most commonly, I find that they are the 32nd and 21st in the list, respectively. But it doesn’t end there. As I open directory 32, it has 66 subdirectories; the ones I use most commonly happen to be the 22nd and the 15th of these. (I especially hate it when the item I select 95% of the time happens to be near the bottom, so that I have to scroll even to see it.)

You see what I’m getting at. Specifically I want this kind of capability:

• I want to be able to order lists manually
• I want to be able to sort lists based on metadata (let the most frequently used ones "float to the top" on their own)
• I want to be able to sort chronologically by date of creation, modification, and access
• I want to be able to sort by "most frequently used" and "most recently used" and the like

That’s why I want to keep all this metadata. There are issues to be ironed out, but they are fairly minor. For example: What if we switch from manual ordering to alphabetic; create a new item; and then switch back to manual ordering? Do we put the new item at the top of the list or the bottom? Do we flag it in some way? I’m not picky about those things.

There is another benefit here. We will often find that in a Zipfian list, the top three or four items will account for over 90% of all use. In that case, why are we taking up the screen space by displaying all the others? There should be a way to "collapse" or "fold" the rest, perhaps into a special entry that said "More…" or the equivalent. Traditionally, this sort of folding or collapsing has only been done for entire levels of a tree or outline. This, however, is only tradition, not logic. Let’s add the ability to collapse infrequently used items. Again, this might be done in a fully manual way, a fully automatic way, or even a bit of both.

Now that I have said all this, I am obligated to issue a disclaimer. No doubt some reader will call to my attention a paper written thirty years ago or more, an operating system from the seventies, or a old obscure piece of application software that illustrates a deep understanding of these principles.

That’s fine. I don’t claim to have originated every aspect of these ideas. In fact, I claim that they are largely common sense.

I also claim, however, that these principles are virtually ignored in most or all software, especially where lists are concerned. The only application of this principle I can think of offhand is actually (in my opinion) a mis-application. Certain pieces of Microsoft software have menus that are truncated with supposedly the most common commands at the top, and the rest collapsed or folded. The problem I have with this is that it is neither (so far as I know) dynamic or configurable, and that whoever guessed at the usage patterns did not have me in mind. Far from helping me, this feature has hindered me more often.

So we must implement such features with caution. But let us not use too much caution; for list-oriented interfaces can hardly get much worse than they are now.