Commodification and Operating Systems

I'll warn you in advance: this is going to start off topic and go way off topic. "Catching up" with the changes to Mac OS, Windows and Linux has me thinking about the nature of technology. I feel a little bit guilty about this post because it's going to turn into a rant about Vista, and ranting about Vista is like shooting fish in a barrel. On the other hand, having used Vista, well, I have a lot of rant to give.

One of the most important things to understand about technology (and computers are no exception) is that changes in the scale of the technology change the very nature of the technology. That is, as you make computers faster and cheaper, at some point the sum of all of those small improvements changes the fundamental nature of the beast. We've seen this as the computer transformed from main frame to desktop (which is really just a change in cost and size), finding an entirely new audience, and now again as the computer changes from what we know of now as a computer to cell phones, MP3 players, and other small, mobile devices.

"Commodification" is what happens when, as things get better, cheaper, faster, etc., consumers stop caring about the marginal improvement. Back in the days of Windows 95 and 386's, there were ways you could improve the operating system and hardware in substantial ways; a doubling in processor speed and a rewrite of the operating system got you protected memory, which meant less data loss.

A few years ago we reached the point where desktop hardware became commodified. For the average user, 1.8 vs 2.2 ghz makes no difference at all. It's a question of how quickly your computer can wait for keystrokes and data from the internet. (Answer: even a lowly Celleron is light-years faster than the I/O devices it typically has to talk to. Even if you're the last kid in your class at Harvard, you're going to be bored discussing politics with a bunch of four-year-olds.) At that point things became very difficult for major vendors like IBM (sold out), HP and Compaq (merged), Gateway (bought out of it's misery), etc. The price of a desktop plummeted from over $1000 to less than $400.

I believe we've reached the point where operating systems have become a commodity as well;

• Every major operating system has all of the features of a "real" operating system - that is, protected memory, virtual memory, plug & play driver support, etc.
• The performance for normal applications is just about the same; there are some specific variations that matter in the server market, but for all practical purposes the operating system is not in the way, and the machine is much faster than users need anyway.
• Every major operating system has a similarly designed GUI experience that, once you get used to the quirks of where the close box is, is just about the same, more or less. (Mac users - keep your pants on. :-)

And this is why life is not so good for Microsoft. In a non-commodified market, you can charge a premium for incremental improvements over the competition. That's a game Microsoft can play - they have a lot of capital to invest in changing their operating system, as long as they are rewarded with a lot of cash for doing it. (And normally they are - about six billion dollars for a major OS revision, I'm told.)

The problem is that operating systems are now a commodity. Simply put, users don't need a new operating system. There are no big ticket features missing from OS X 10.4, Windows XP, or Linux 2.6. This makes Microsoft's business model fundamentally vulnerable to Linux for the first time. If the name of the game is:

• Keep costs down, as low as possible.
• Incrementally improve quality very slowly without ever causing the pain of a major OS upgrade.

That's a game Linux, with their army of distributed bug fixers and free source code, is going to win.

When I looked at Windows XP and Ubuntu 6.06 I was afraid that Linux wouldn't make traction into the desktop market. I blamed the adoption of X11, the KDE/GNOME schism, and the Linux communities' being made up of Shell nerds for the tolerable desktop experience.

But look where we are now: Vista is a vehicle for bloat. Combine "we make money by shipping major features" and "there are no more major features to ship" and you get Vista...an attempt to change a lot of things when you should have left things alone.*

By comparison, Ubuntu pretty much just works - you put the live CD on your machine, it asks you some questions and installs...it knows about more hardware, has less bugs, more drivers, and a better user experience. In a commodified operating-system space, the only thing to do is try to avoid a bad user experience - if you can't offer a really juicy carrot to users, try to avoid hitting them with a stick.

And it is in this environment that the Mac is actually gaining market share. Apple's business model has always been at odds with the industry. Complete vertical integration meant higher costs, lack of market share, and out-of-date technology - back when having more for less meant something, that was a real weakness, and explains why the Mac never dominated in market share.

But what a difference a decade makes! Hardware is now commodified (and Apple is integrated at the system-building level, leveraging cheap third party parts like they always should have). Operating systems are commodified. But on the one frontier left, quality of user experience, Apple's vertical integration gives it an immense advantage.

The question is: why does an operating system "just work"?

• Vista: it doesn't. There are too many systems and not enough testers and engineers trying to solve the problem.
• Linux: massive distributed engineering. For any given hardware system, eventually a Linux nerd will integrate it. Anyone can solve the problem of poor user experience.
• Apple: they have it easy. With only half a dozen machines in production (and maybe another two dozen legacy configurations to support) they have a much smaller configuration space to worry about than anyone else.

I don't kow what Microsoft's future is, but it can't be very good. At some point they are going to have to transition from a "major revision for cash" to an "incremental tuning" approach to operating systems. As long as they have market share, they still get the "Windows tax" - that is, their OEM pricing from major vendors on every new computer that is built. It's going to be harder and harder to convince the entire world to make a major jump (see how well XP to Vista went). In this situation, they'd be better off with a more solid operating system. It's unfortunate that they're going to have to try to sustain market share with Vista.

Their best-case scenario is that they eventually get Vista back to an XP-quality experience, in which case all they've done is spend a huge amount of R&D money and pissed off a lot of customers to maintain the status quo.

* I have mixed opinions on Vista's video-driver-model change. But that's a different post.

My Opinion: Let's Not Use RAR

I had to do some research into compression algorithms recently, because we had to squeeze the global scenery onto fewer DVDs for retail distribution.  We did this mostly by completely filling the DVDs, but we also had to use 7-zip compression to get about a 10% improvement in compression ratios.

DSFs are not the best test of compression efficiency because the format has been organized to help algorithms like zip compress them - the improvement with 7-zip and RAR was a lot less than you'd get with, say, a text file.

Anyway, my point here is: let's not use RAR - it's the new GIF.  Every now and then a file format comes along with some kind of restriction that keeps everyone from doing everything with it.  In GIF's case, you had to buy the right to create GIFs, and in the case of RAR you have to buy the right to compress RARs.

I think that having these kinds of entanglements in fundamental low level file formats (like how do we compress our data or save our images) is really bad for the software community as a whole; it balkanizes raw materials.  And file formats stick around for a long time - even though GIF is made obsolete by PNG you'll still find them all over the web.

The lure of RAR is of course higher compression ratios than zip.  But 7-zip can do the same thing, and unlike RAR, has the potential to be completely free, which means it can be completely ubiquitous.

Macintosh users understand the problem here: for the longest time "StuffIt" archives were the standard way to compress data on the Macintosh.  The file format was proprietary, so you couldn't even make your own program work directly with StuffIt archives.  Now that zip has taken over on the Mac, getting data between Mac and Win is easy - you can just zip something using the operating system and send it to all your friends.

Let's not go back into the "bad old days" of proprietary utilities and a lack of integration with regular apps.  I say: if you can stand to use zip or bzip instead of RAR, vote for what's open and has a future, not what is slightly better now but will just be a pain in the ass in three years.

Spam My Wiki, Please

I'll post more about scenery soon; this will be short and not terribly topical.

The X-Plane Plugin Wiki used to have no login requirements - anyone could just click and edit. All was good for a while, and then I logged in one day to find some of the most highly used pages stuffed to the gills with the phrase "Nigritude Ultramarine" over, and over, with links to other sites.

You can read about this phrase, why it was invented and what was going on here.

Our response was to put a user login requirement on the site, and we haven't had a spam problem since (knock on wood) although we do seem to get what appear to be bogus signup requests. (They don't really hurt anything, they just clog the user database. I'm not sure why anyone would sign up if they don't intend to actually do anything.) But a few thoughts on Nigritude Ultramarine and people's attempts to get junk spam sites into the top of Google's search listing:

• I was pleased to see a real site (this FAQ) as the number two search hit for the term...this real link from a real blog to them can be sort of a contribution to their page rank.
• I have faith that Google will continue to fight the technology arms race against seach engine optimizers...Google has gobs of money and an immense motivation to do so.
• Apparently link farmers, in an attempt to raise their page rank, have been using bots to automatically steal blog content. I haven't seen this myself yet and I've never had an X-Plane query go to a junk site. But some blogs I read have complained of this happening.

Only mildly related, there is an X-Plane Wiki, and I'll try to point people toward it next week; I've been posting things there rather randomly in an attempt to get underdocumented stuff written down somewhere.

Happy Birthday, World

Today is Rosh Hashanah, the Jewish New Year. In Jewish tradition it is also thought of as the birthday of the world. If you're reading this blog, you probably have some interest in how X-Plane attempts to simulate the world on your computer. The attempt over the years to create a higher fidelity simulation has led me to a deeper appreciation of the subtlety, complexity, and beauty of the real thing.

(Nothing makes you realize how rich and intricate the world is than trying to model it with a few million triangles and ending up with something that looks completely crude.)

X-Plane's digital world isn't the only way that we interact with a proxy instead of the real thing. When we drive instead of walk, eat packaged food from a supermarket, talk on the telephone instead of talk in person, our technology becomes a proxy for our relationship with our direct natural environment - the planet, plants, animals, and other human beings.

Now I'm not saying that any of these things is bad. I'm not about to become a dairy farmer, and without the internet we couldn't create X-Plane at all. But I think it's important on this day, and hopefully every day to take time for activities that put us in direct contact with the world. Consider a few questions:

• How does what I eat affect the world?
• How does my travel affect the world?
• What impact does my home have on the world?
• Am I leaving the world in better condition for the next generation or worse?

Please ... take a few moments to consider the world, the only home we've ever had.

When I was in the Dolomites a few years ago with Sergio we were looking at the dolomites from his friend's balcony - mile after mile of beautiful mountains and rolling hills. I looked at him and said "God has more polygons than we do." It was a joke at the time, but I think that the act of really observing the real world and realizing that the digital reality and technology we create can be a proxy and an addition but never a replacement is critical to understanding the responsibility for stewardship we have over the planet. Are we taking good care of our most precious gift?

GIF is Dead!

(This one's gonna be a little bit political I fear. More scenery-related stuff in a few days!)

The last software patent on GIF expired October 1st 2006. It's very rare that I agree with Richard Stallman on much, but I think he presents some strong arguments against software patents in this speech.

X-Plane never used GIF; we've used PNG as our main graphics format for a while now...here's a history of PNG's development, relating to the problem with GIF being patented.

A few comments of my own:

• One problem with software patents is the "landmine" effect. There's no way to know as a programmer when we've violated a software patent. (Try searching the patent database for an hour or so and I think you will agree with me on this.)
• The bar for non-obvious patents is completely miscalibrated. Software engineering is all about building black boxes out of past work and rearranging them to do new things. Allowing the arrangement of past technologies to be patentable is like allowing a mason to patent a particular stacking of bricks; to most masons it should be obvious how to stack bricks for a certain job, and any good programmer should know how to generalize past ideas to solve new problems. This isn't invention, it's implementation!
• Patents aren't based on natural law and human rights, they're based on fostering creativity. To this end patents are the exact wrong solution for software, where having basic interoperable technologies be free and cheap is good for common infrastructure. Software patents are like someone being able to charge a toll for driving at exactly 55 miles an hour. We want everyone driving the same speed because it's good for the traffic system; similarly we want people to all use the same file formats because it makes all programs more useful.

There are two cases of software patents I can imagine: short-lived ideas that will be obsolete by the time the patent is ever defended; these patents at best serve to help one company beat another over the head with lawyers; I think this is comparable to a SLAPP lawsuit.

The other case is when the idea is truly useful for its entire lifetime (see GIF, which became the only way to put transparent graphics on the web for a while). In this case the result is a stifling of interoperability and innovation and the patent is to the detriment of the industry and all of its users.

To this end I see libpng as a perfect success: a well-designed file format that's free of patents implemented in an easy-to-use high performance library that's free (X11/MIT license). This encourages and makes it easy for everyone to use PNGs which makes all programs work better. Would we be better off if Austin and I had to invent a proprietary image format (with our own inferior compression) because the major formats were patented?

(Would it be feasable to license such a format? I don't think so - see Stallman's comments on the number of components in a software program vs the cost of licensing.)

Mmmm....Marketing

I've always thought we should come up with cool marketing names for new code in the sim. Look at ATI and nVidia. When they reduce the amount of VRAM on their cards and steal your system memory, they don't call it "hey, we're cheap bastards so we left off some chips and hijacked yours instead". Instead they call it "TurboCache™" (nVidia) or "HyperMemory™" (ATI). Don't those sound much better?

Perhaps we can call the new rain effect (due out in the next RC) "MegaRain XT" or, um, "HyperSpash GT" or "Virtual Droplet Technology (VDT)"? I don't think I'm cut out for a career in product marketing. Anyway, here are a few screenshots:



There will probably still be some camera angles where the rain looks weird; it's a compromise between image fidelity and framerate.

Apropos of nothing: Ami sent me this flash video. I must warn you...Chris described it as "the corniest flash I've ever seen"....and in the world of flash that truly says a lot. Think of it as the blue pill for your, um...bits. Anyone wanna make a flash video showing the virtues of MegaMoister 2.0? ;-)

(I promise the next blog entry will contain more information and less attitude!)