Never Send a Chair To Do a Bed's Work

I commissioned PropsMan to make me a test bed for some of the new 9.20 airplane features. Clearly he has never worked in a furniture store -- this is what he sent me!

In his defense, it's very comfy, flies surprisingly well, and is a great comprehensive test of a huge pile of advanced aircraft features.  Yes, you can close and open the laptop with the mouse in the 3-d cockpit.  

(One of the new features in 920 is "no-op manipulators" - that is, the ability to make 3-d geometry "eat" clicks before they get to the panel, without having to use panel texture as the consuming surface.  So when the laptop is closed, you cannot click on the buttons that are exposed when it is open.)

My real question is: how did he know about Laminar's new secret test vehicle?

Probability and Certainty

I've been reading Fooled by Randomness (highly recommended - it will either change your life or you won't finish it because Taleb's writing style annoys you) - and it has me thinking about the nature of certainty in software development.  Consider two approaches to uncertainty and how they are completely at odds with each other.

No Weird Behavior

The "No Weird Behavior" approach goes like this: you never want a harmless behavior inside your code that you don't understand.  The reason is that if you don't understand the behavior, you don't really know that it's harmless!

In fact this isn't just a theoretical problem - I only truly started to believe in "No Weird Behavior" after fixing several very hard to find, hard to reproduce crash bugs, only to discover (once the analysis was done) that the broken code also produced a very frequent, less harmful behavior.  Had I taken the time to investigate the "weird behavior" first (despite it not being of high importance) it would have led me to a ticking time bomb.

No Weird Behavior implies that a bug isn't fixed until you know why it's fixed, and that randomly changing code until the behavior goes away is absolutely unacceptable.  This shouldn't be surprising; if a bridge was swaying would you accept an engineer who fixed it by randomly tightening and loosening screws until it stopped?

Wait And See

I get a lot of email with bug reports, questions, and reports of strange sim behavior.  These emails have some unfortunate statistical properties:

• A good number of them are resolved by the user who emailed within a day or two.
• A certain percentage are simply never resolved.  (Often I email a question that would point to a clear diagnosis and the user never writes back.  I can only speculate that the user answered the question, found the problem, fixed it, and promptly forgot about our emails.)
• A certain percentage are solved by the user, who tells me what the problem was, and it was something I would never, ever be able to help them with.  (Things like "I have this program called WickedMP3Player - it turns out if I set its visualizer setting to 'Rainbow' then X-Plane stops crashing" when I've never even heard of the WickedMP3Player program to begin with.)
• There is a high correlation between bug reports reported by a very small number of users and a resolution from the above categories, or a resolution by the user changing his or her local configuration.

So playing the odds, the right thing to do when presented by a third party with weird behavior is to wait and see who else reports it; the frequency of reports gives us information about the likely resolution.

(And for those of you who think our tech support are being lame when they ask if you've updated your drivers, they are playing the odds to the hilt - they ask you about drivers first because updating drivers fixes an absurdly huge number of the tech support calls we get.)

Getting It Wrong

So we have motivation to investigate everything (no weird behavior), motivation to ignore everything (wait and see) and a rule of thumb that the frequency of reports can help us pick which strategy is best.  Of course, sometimes the rule of thumb is completely wrong.

One user reported a crash using the current web updater (version 2.04) - I had not seen this crash anywhere and it was inside the operating system UI code, so I assumed it was a local problem, e.g. some kind of extension or add-on that caused the OS problems.

The problem, it turns out, is simply low frequency - as the incorrect code made it into 902b1, I got a few reports from more users and realized that this wasn't a local problem, it was weird behavior!  (The bug will be fixed in the 205 installer and 902b2, both of which will be out in June.)

Consider this: if you make a measurement of a known quantity with a dubious measuring device, you learn more about the measuring device than the object you are measuring.  (If you have a ruler and you don't know the units, you could determine them by measuring yourself.)

In a number of cases, we have seen serious "should-happen-all-the-time" crash bugs that get reported by very few users.  (Once we know the actual root cause of the bug, we can deduce logically whether the bug should happen to all users with the configuration or be intermittent.) We can then look back at the actual number of reports to make a judgement call on how much testing is really happening.

For example, we can make some guesses about how quickly a new Macintosh have saturated the X-Plane user base when there is a hardware-specific serious bug in just that machine.

We had this with the iMacs (where the runway lights were broken) and we could watch the machines sell - slowly at first, but then quite quickly.  (BTW, I think that 10.5.3 fixes this and anti-aliasing problems.)  We can even see who runs with anti-aliasing when there is a setting-specific bug (a lot of you do)!

In the end, I think the right approach to balancing "no weird behavior" and "wait and see" is to remember a truth about uncertainty that is very hard for humans to grasp:

The most likely outcome of an uncertain situation is not guaranteed to happen - in fact a lot of the time it won't.*

So we can play the rule of thumb and wait and see, but we always have to keep one eye toward the improbable, which is still possible!

* Blatant blog cross-promotion...the point of my big rant here was essentially the same...it's easy to expect the most likely outcome, but the unlikely outcome will happen some of the time. 

Threads and Cores

Now that multi-core machines are mainstream, you'll hear a lot of talk about "threads". What is a thread, and how does it relate to using more cores?

Definitions

A "core" is an execution unit in a CPU capable of doing one thing. So an 8-core machine might have two CPUs, each with four cores, and it can do eight tasks at once.

A "thread" is a single stream of work inside an application - every application has at least one thread. Basically a two-threaded application can do two things at once (think of driving and talking on your cellular phone at the same time).

Now here's the key: only one core can run a thread at one time. In other words, if you have an eight core machine and a one-thread application, only one core can run that application, and the other seven cores have to find something else to do (like run other applications, or do nothing).

Two more notes: a thread can be "blocked" - this means it's waiting for something to happen. Blocked threads don't use a core and don't do anything. For example, if a thread asks for a file from disk, it will "block" until the disk drive comes up with the data. (By CPU standards, disk drives are slower than snails, so the CPU takes a nap while it waits.)

So if you want to use eight cores, it's not enough to have eight threads - you have to have eight unblocked threads!

If there are more unblocked threads than cores, the operating system makes them take turns, and the effect is for each of them to run slower. So if we have an application with eight unblocked threads and one core, it will still run, but at one eighth the speed of an eight core machine.

It's not quite that simple, there are overheads that come into play. But for practical purposes we can say:

• If you have more unblocked threads than cores, the execution speed of those threads slows down.
• If you have more cores than unblocked threads, some of those cores are doing nothing.
  

Trivial Threads

When a thread is blocked, it does not use any cores. So while X-Plane has a lot of threads, most of them are blocked either most or all of the time. For all practical purposes we don't need to count them when asking "how many cores do we use". For example, G1000 support is done on a thread so that we keep talking to the G1000 even if the sim is loading scenery. But the G1000 thread spends about 99.9% of its time blocked (waiting for the next time it needs to talk) and only 0.1% actually communicating.

What Threads Are Floating Around

So with those definitions, what threads are floating around X-Plane? Here's a short list from throwing the debugger on X-Plane 9.0. (Some threads may be missing because they are created as needed.

• X-Plane's "main" thread which does the flight model, drawing, and user interface processing.
• A thread that can be used to debug OpenGL (made by the video driver, it blocks all the time).
• Two "worker" threads that can do any task that X-Plane wants to "farm out" to other cores. (Remember, if we want to use more cores, we need to use more threads.)
• The DSF tile loader (blocks most of the time, loads DSF tiles while you fly).
• At least 3 threads made by the audio driver (they all block most of the time).
• At least four threads made by the user operating system's user interface dode (they block most of the time).
• The G1000 worker thread (blocks most of the time, or all the time if you don't have the G1000 support option).
• The QuickTime thread (only exists when QuickTime recording is going on).
  

So if there's anything to take away from this it is: X-Plane has a lot of threads, but most of them block most of the time.

Core Use Now


So how many cores can we use at once? We only need to look at threads that aren't blocked to add it up. In the worst flying case I can think of:

1. The main thread is rendering while
2. The DSF tile loader is loading a just-loaded tile while
3. One of the pool threads is building forests while
4. You are recording a QuickTime movie (so the QT thread is compressing data).

Yep. If you really, really put your mind to it, you can use four cores at once. :-) Of course, two cores is a lot more common (DSF tile loading or forests, but not both at once, and no QuickTime.

Core Use In the Future

Right now some of X-Plane's threads are "task" oriented (e.g. this thread only loads DSF tiles), while others can do any work that comes up (the "pool threads", it's like having a pool car at the company, anyone can take one as needed). The problem with this scheme is that sometimes there will be too many threads and sometimes too few.

• If you have a dual-core machine, having forests building and DSF loading at the same time is bad - with the main thread that's three threads, two cores; each one runs at two-thirds speed. But you don't want the main thread to slow down by 66%, that's a fps hit.
• If you have a four-core machine, then when the DSF tile is not loading, you have cores being wasted.

Our future design will allow any task to be performed on a "pool thread". The advantage of this is that we'll execute as many tasks as we have cores. So if you have a dual-core machine, when a DSF tile load task comes along while there is forests being done, the one pool thread will alternate tasks, leaving one core to do nothing but render (at max fps). If you have a four-core machine, the DSF load and forests can run at the same time (on two pool threads) and you'll have faster load times.*

* Who cares about load time if you're flying? Well, if you crank up the settings a lot and fly really fast, the loader can get behind, and you'll see trees missing. X-Plane is always building trees in front of you as you fly and deleting the ones behind you. So using more cores to build the forests faster means you're less likely to fly right out of the forest zone at high settings.

The Relationship Problem

I just finished about 15 pages of emails, mostly to Andrew McGregor (who is the very first MeshTool user) and also Benedikt Stratmann (whose x737 is on the bleeding edge of plugin-based aircraft) and AlpilotX (we all know about his forests). Probably all three are wondering how the hell I have time to write so much on weekends. (The answer is of course that my frisbee game got rained out. Foo!)

In the meantime, probably about 300 other people who have emailed me in the last few
months are wondering why the hell they have heard nothing from me. My in-box looks like a mail server exploded. It's not pretty.

So let me blog for a moment about the "relationship problem". Simply put, there are two of us (Austin and myself) and about a thousand of you (third party developers doing cool and interesting things with X-Plane) plus significantly more users, some of whom have some very weird tech support problems.

In this environment, our algorithm for who gets "developer attention" is pretty broken and subject to total thrash...there is a huge element of random luck (who emails me when I am recompiling the sim vs. debugging a nasty bug).

I'm aware of both how hard the task Austin and I face and how frustrating it is for a third party developer because I've been on both sides. Before I worked for LR, I was a third party and I was always astounded that Austin couldn't remember what we talked about last week.

Then I started working for the company and saw what it's like. Imagine sitting at a train station watching the trains go by* (at full speed, not stopping) and someone says "last week I waved to you out the window and you waved back, remember me?"

So I would advise three things to the neglected third party:

1. Be firm - you may need to ping us again because at busy times we can't always keep track of who has asked for what.
2. Be patient - if you need something the week we're burning DVD masters for a second time (because the first set failed at the factory) then you're going to have to wait.
3. Don't take it personally...a lack of a response usually indicates overload inside the company, not a poor opinion of your work!

This blog post has rambled enough, but it may feed well into the next one.

* This analogy is totally stolen from "How Doctors Think" by Jerome Groopman - he uses it to describe the task of primary care physicians trying to spot the early signs of a very rare illness among a fast-moving train of patients who are almost entirely healthy. I strongly recommend this book particularly for Americans - we need to understand the forces at work in shaping the quality of our medical care!

Instability in Version 9

One of the reasons why the X-Plane 9 betas have had so many more crash bugs than version 8 is that we introduced loading DSFs on a second core. This feature makes scenery loads much slower and (by using the second core) impacts fps less while they happen.

The problem is that I'm still fumbling with code that will allow this in all cases. (That would be three operating systems, two hardware vendors, and a myriad of drivers, some new, some quite prehistoric.)

Beta 22 will be out soon, and will contain the fourth major rewrite of the OpenGL setup code for X-Plane 9. So far the initial tests look good, but we never know until we let a lot of users try the code and find the new edge cases.

It's relatively easy to tell if your instability is related to the use of OpenGL with threads: simply run the sim with the --no_threaded_ogl option. If things become a lot more stable, it's a threaded GL problem. Mind you --no_threaded_ogl is more of a diagnostic than a workaround; without threaded OpenGL, the sim will pause when loading scenery.

(Also, to clarify, you'll find talk on discussion groups and game forums about "threaded drivers". Threads are a programming abstraction that can utilize multiple cores. What I am talking about is X-Plane using multiple threads to load scenery - in our case this requires interfacing with OpenGL. But a threaded driver is different - it's just a graphics driver that's been optimized for multcore machines. These two concepts are totally different; you don't need a threaded driver to use X-Plane 9, and a threaded driver won't make X-Plane 8 load without pauses.)

GeForce 7 and Water Performance

A number of Windows and Linux GeForce 7 users have discovered that the command-line option --no_fbos improves their pixel-shader framerate a lot. Windows and Linux Radeon HD users have also discovred that --no_fbos cleans up artifacts in the water. Here's what's going on, at least as far as I can tell. (Drivers are black boxes to us app developers, so all we can do is theorize based on available data and often be proved wrong.)

Warning: this is going to get a bit technical.

FBO stands for framebuffer object, and simply put, it's an OpenGL extension that lets X-Plane build dynamic textures (textures that change per frame) by drawing directly into the texture using the GPU. Without FBOs we have to draw to the main screen and copy the results into the dynamic texture. (You don't see the drawing because we never tell the card "show the user".)

We like FBOs for a few reasons:

• Most importantly, FBOs allow us to draw big images in one pass even if the screen is small. For example, if we have a 1024x1024 dynamic texture but the screen is 1024x768, then withou FBOs we have to draw the image in two parts and stitch it together. That sucks. With FBOs we can just draw straight to the texture and not worry about our "workspace" being smaller than our texture. This is going to become a lot more important for future rendering features where we need really-frickin' big textures.
  
• It's faster to draw to the texture than to copy to it.
• If you're running the sim with FSAA, then we end up using FSAA to prepare all of those dynamic textures. In virtually all cases, we don't need the quality improvements of FSAA, so there's no point in taking the performance penalty. When we render right into the texture, FSAA is bypassed and we prep our dynamic textures a lot faster.

Since copying to a texture from the screen predates these new-fangled FBOs by several years, most drivers can copy from the screen to the texture very quickly; however we have hit at least one case where FBOs are much faster than copy-from-screen. That's really a rare bug, and as you'll see below, we see more weird behavior with FBOs.

When do we use FBOs vs. copying? Well, it's pretty random:

• Pixel shader reflective water and fog use FBOs.
• Cloud shadows and the sun reflection when pixel shaders are off do not use FBOs.
• The airplane panel uses FBOs if the panel is 1024x1024 or smaller; if the panel is larger than 1024x1024 we draw from the screen and patch things together. So the P180 and the C172 are using different driver techniques!!

When you run X-Plane with --no_fbos, you instruct X-Plane to ignore the FBO capability of the driver, and we use copy-from-screen everywhere.

Mipmapping

There is one more element: mipmapping. A mip map is a series of smaller versions of a texture. Mipmapping allows the video card to rapidly find a texture that is about the size it needs. Here's an example: imagine you have a building with a 128x128 texture. If you park your plane by the building, the building might take up about 100x100 pixels on the screen; your 128x128 texture is a good fit.

Now taxi away from the building and watch it get smaller out your rear window. After a while the building is only taking up 8x8 pixels. What good is that 128x128 texture? Its' much too big for the job. With mipmapping, the card has a bunch of reduced-size versions of your texture laying around...64x64, 32x32,16x16, 8x8, 4x4, 2x2, 1x1. The video card realizes the building is tiny and grabs the 8x8 version.

Why not just use the 128x128 texture? Well, we'd only have two options with this texture:

1. Examine all 16384 pixels of the texture to compute the 64 pixels on screen. That sucks...we're accessing VRAM sixty four times for each pixel. Accessing VRAM is slow, so this would kill performance.
2. Simply pick 64 pixels out of the 16384 based on whatever is nearby. This is what the card will do if mipmapping is not used (because option 1 is too slow) and it looks bad. Thsoe 64 pixels may not be a good representation of the 16384 that make up your building side.

So mipmapping lets the video card grab a small number of pixels that still capture everything we need to know about the building at low res.

We don't mipmap our dynamic textures very often; the only ones that we do mipmap are the non-pixel-shader sun reflections and the pixel-shader sun reflections.

ATI

As far as we can tell, the current ATI Catalyst 8.1 drivers do not generate mipmaps correctly for an FBO-rendered texture. This is why without --no_fbos ATI users on Windows or Linux see very strange water artifacts. --no_fbos switches to the copy path, which works correctly.

At risk of further killing my track record of driver bugs in v9, we do think this is a bug. We have good contact with the ATI Linux driver guys so I have hopes of getting this fixed.

nVidia

It appears that the process of creating mipmaps for FBO textures is not accelerated by the hardware on the GeForce 7 GPU series. This is why GeForce 7 users are seeing such poor pixel shader performance, while GeForce 8 users aren't having problems.

Now poor performance is not a bug; there's nothing in the OpenGL spec that says "your graphics card has to do this function wickedly fast". Nonetheless, what we're seeing now is unusably slow. So more investigation is needed -- given that the no-FBO case runs so quickly, I suspect the hardware itself can do what we want and it's just a question of the driver enabling the functionality. But I don't know for sure.

Algo-Gen

I may be fighting a pointless and unwinnable linguistic battle, but I have to try. People very often refer to the default city buildings in X-Plane as "auto-gen" but by any reasonable definition of "auto-gen" they are not really auto-gen at all.

Now these are all made up computer terms, so we can't really check the dictionary. But "autogen" scenery (short for automatically generated) usually refers to scenery that is created by a flight simulator itself, usually while you fly, and usually by placing 3-d detail in places that match the base terrain. This exists in FSX, and existed in X-Plane up to version 7.63.

X-Plane 8 doesn't have autogen!!!!!!! X-Plane 8 has scenery that is generated by computer programs, but X-Plane is not the computer program that is doing it. When you see a ton of buildings piled up in New York City, that is not becaues X-Plane looked at the New York city base terrain and said "hrm - some buildings would be nice."

What actually happens is we analyze New York City when we create the global scenery (before we ever burn the DVD masters) and the DSF generator places all of those buildings in New York City. X-Plane simply gets a huge list of buildings from the DSF and draws them.

I am going to try to coin the term "algogen" (algorithmically generated) to describe these buildings that (like autogen) come from a computer generating semi-random buildings from input data, but unlike autogen, algogen is a process that runs once before the scenery is made.

So how is algogen and autogen different?

• You can't change the pattern of algogen building placement by editing files in the sim. The algorithm has already been run! You can replace the buildings using an overlay (that excludes the base) or by using a library of models to substitute models.
• We are trading data size for computation. The DSF is bigger because it lists the location of every building in New York, even if they were just algogen buildings, but the job of placing those buildings is less difficult because X-Plane does not have to check each building against each road. That has been done in advance.
• Changing the scenery via an overlay doesn't change the algogen! Add an airport via an add-on and you have to exclude the buildings. (But if you send that airport to Robin, the next global render will include it and the algogen will skip the airport automatically.)

Note one of the interesting results of algo-gen: X-Plane can't tell the difference between an alg-gen building and a hand-placed one! They're all just objects in a DSF. The fact that algo-gen buildings disappear with lower settings is because the sim/require_object property in the DSF header tells the sim which objects are important, and our generator always signals the buildings based on obstacle data as important. But algogen as a process is not visible to X-Plane!

And that's why I'm spending so many words on trying to distinguish between "algogen" and "autogen" - because the processes are fundamentally different, they're very different for scenery authors to work with. As a result, authors coming from X-Plane 7 or FSX will be very surprised if they try to understand X-Plane in terms of autogen....they won't be able to find the autogen config files, and the autogen buildings won't react to other scenery changes, because they're not actually autogen at all!

Algogen is a classic pattern of "precompute" vs. "compute-while-fly". Generally precomputing gives authors more flexibility (in our case, we have an obj engine that can handle a lot of objects, so authors can make their own objects of the same density as algo-gen with the objects placed anywhere) at the expense of making it more complex to edit the existing scenery (edit the mesh and the algogen doesn't change).

When we started the v8 scenery, two things pushed me toward precomputation:

• In the past, changes in X-Plane's rendering engine had broken third party add-ons. So a precomputation strategy (by getting the scenery code out of the sim) means that the sim is doing less "interpretation" and thus the interpretation of scenery is less likely to change.
• We wanted to focus on performance, which means getting computation out of the sim whenever we could.

Now that last point isn't quite as important as it used to be...when we were doing the design (during mid X-Plane 7), dual core for everyone wasn't on the radar, so the penalty for complex computation while flying is lower (and thus we have more expensive in-flight computation, like forests and completely draped bezier curve-based polygonal pavement).

But I think precomputation is still useful. Even with dual core, the algorithm that places X-Plane's algo-gen bulidings can take one to two minutes for a 1x1 DSF tile on a very fast computer. That's still a load time that's out of the question for us; even on the second core, the DSF wouldn't be "ready" in time for you to fly it. So one use of precomputation is to run algorithms that are more expensive than you can have in real-time. (That algorithm to pack objects inside an irregularly shaped polygon made by roads and land features is not fast.)

More importantly, precomputing does give us a nice advantage in the use of storage data. We ship about 50-60 GB of final scenery, but the source data is well over 100 GB. When we run the algogen algorithm, we have access to the full set of source data: coastlines, elevation, and land use before any simplification is done and any data is thrown out. So we have the potential not only to do a more complex analysis, but to do the analysis on a larger data set.

The down-side of precomputing is that if integration of all data is saved until sim time, there is the potential for third parties to contribute separate data to the sim via add-ons and still have the integration of those data sets work well. This doesn't always work out - see complaints in online magazine reviews about combining orthophotos and new road grids in FS2K4...they don't integrate because neither of those types of resources can be integrated to match the other in real time. But autogen still does a much better job than algogen at this; algogen basically has to be recut when other data changes. (And that is our intention - if you change the road grid, exclude and replace the objects too!)

Why You Can't Have a Setting

I get a lot of requests for settings...the email is typically something like:

• Some feature in X-Plane is defaulted to X.
  
• I like it better when it's like Y.
• Can I have a setting to change the feature between X and Y.

Raymond Chen has a great posting that I think is very topical: "In order to demonstrate our superior intellect, we will now ask you a question you cannot answer."

This brings up one of the main reasons why we shy away from more settings: the more complex we make X-Plane's configuration, the less likely it is that the average user will be able to set the sim up correctly. Settings requests usually come from our most advanced users, but we also have users who have never used a computer before. Really! I've been on the tech support calls - they are very nice, but way overextended on the computer side of things. Should we allow them to pick whether scenery geometry is store in AGP memory vs. VRAM?

From our perspective, having a setting that a user doesn't understand is worse than neutral, it's actually harmful. Every one of those settings is something that can go wrong with the sim. I removed the ability to set the level of detail bias to positive (in other words, extend the visibility distance of scenery beyond its original design) after about 500 complaints of "low framerate" from users who had maxed this setting out (causing a 4x increase in 3-d processing load) without knowing (1) what the setting was, (2) what it was good for or (3) what the down-side was.

Could we present all the info to make intelligent decisions on the rendering pages? Honestly, probably not beyond a certain point...we would devolve our sim into a lecture on working sets, bottlenecks, and the OpenGL pipeline long before the user got flying. (Wait, that's my blog! Doh!!!) At some point the sim just has to do its best to do the right thing, or something similar to it. Just as Raymond points out that the default answer to any dialog box is "cancel", the default answer to any rendering setting is "all the way up."

When I tell a user who wants a setting that he or she can't have a setting because some other user will abuse it, the answer is almost always: well why don't you have two settings screens, a simple and advanced mode?

Besides the irony (of trying to solve the problem of too many settings with another settings), Raymond also points out that no location to hide an advanced setting is ever quite good enough. This is something we have struggled with, choosing command-line options more for to pragmatic reasons than because it's a great solution.

This doesn't mean you can't ask for command-line options...I am just trying to point out some of the thinking on the other side of the coin.

Is Bigger Always Better?

We've been preaching "one big texture, not lots of little textures" for a while now, and generally speaking, packing a lot of art into one big texture makes life eaiser for X-Plane, because it can draw more triangles at once before it has to tell the card to change what it's doing. Inside the company we call this the "crayon rule".

Now the total set of geometry and textures that X-Plane needs to use for one frame is the "working set" - you can think of it as the crayons that you keep out of the box because you need them all the time. And as I said before, if the working set becomes too big, your framerate dies.

Now with large panels we're seeing a new phenomenon, one of the first cases where the crayon rule might not be true. The reason is due to working set.

When you make an airplane with a large panel in version 9, you can either use ATTR_cockpit, which lets you use the entire panel as a texture, or you can use ATTR_cockpit_region, which will let you use several parts of the panel. Each ATTR_cockpit_region is a texture change, so that's more crayons. And yet ATTR_cockpit_region is usually faster.

The reason is two-fold:

1. You can often use cockpit regions that don't cover the entire cockpit texture. Large panels are rounded up to 2048 if the are larger than 1024 in any dimension, so the "wasted space" in a 1600x1600 panel is actually quite huge. If you can get away with some smaller regions, your total panel texture area is smaller because there isn't wasted space due to this rounding, and you can also skip things like Windows. Prepping the panel texure takes time, and it's done once for lit and once for non-it elements, so it adds up!
2. It turns out there are two categories of textures that contribute to the working set: static texures and dynamic ones, and their impact on VRAM is very different. Dynamic textures are much more expensive. The panel texture is dynamic and it's uncompressed, so it really costs a fortune. (32 MB of VRAM for 1600x1600. That's not a lot for a static texture but for a dynamic one that'll kill you.)

Here's the details on dynamic vs static textures: the OpenGL driver keeps a backup copy of a texture in main memory, so that if it has to purge VRAM (to make room for more stuff) it still has the texture. As it "swaps" textures, the process is to simply send textures as needed from main memory to VRAM. No big deal.

But with a dynamic texture, the texture has been modified in VRAM! So the copy in system memory is old and stale. The graphics card thus must send the texure back to main memory, consuming twice as much bus bandwidth as normal. (To free 16 MB of VRAM and refill it takes 32 MB of transfer, 16 MB to copy the old texture back to system RAM and another 16 to send the new textures to VRAM.) On non-PCIe cards, this back-transfer might be at 1/8th the speed of the transfer to the card, so this is even worse on AGP cards.

Thus the driver does its best to not throw out dynamic textures. And this is why the panel texture is so expensive. That P180 will cause X-Plane to make two 16-MB dynamic texures, and those textures will cause 32 MB of VRAM to basically be off the table. That's less space for the other textures to swap in and out of. This kind of "permanent allocation" makes the VRAM budget tighter for all other drawing operations.

Given the right combination of large panels, large res, pixel shader effects (which make more dynamic textures), clouds, and FSAA, you can easily get even a 256 MB card to a state where the free space into which static textures are shuffled becomes horribly small, and the framerate just dies.

So the moral of the story is: yes, it can be worth 4 crayons (using panel regions) to avoid the huge cost of dynamic textures from large panels.

As to static textures (regular DDS files) that are 2048x2048 - the jury is still out but my guess is they don't represent a huge performance problem. As one user pointed out to me, they're only 2 MB when compressed (maybe more with alpha) so they're not insanely huge, and they can be swapped out.

NVidia: 2 Ben: 0

I found the root cause of another NVidia specific bug, and once again it's my own stupid code. If you Google for driver bugs, you'll find plenty of grumpy developers ranting about how card X does this wrong thing and card Y does that wrong thing...I figure it's only fair to follow up and say "yep, that one was mine."

Like the previous nVidia-only crash, this was a case where X-Plane was always doing something wrong, but only some drivers had problems with the behavior. So the crash was NVidia-specific, but X-Plane caused.

I believe that this bug was manfiesting itself either as a message that "scenery shift took more than 30 seconds" or some kind of crash. One of the problems was that the diagnostics for this particular bit of code were really bad. So we've improved things a bunch...

• There is more careful error checking during scenery shift, and those error messages are reported.
• If the sim does crash, some new code will output a crash log on Windows that helps us isolate what actually happened.

Beta 12 will be out soon with the fix that caused problems on NV hardware as well as the improved diagnostics. So you may find that the sim just works better, but if it does still crash or report errors, please tell us - now we'll have log files that will let us diagnose the problem a lot faster!

"Driver Bugs"

I have spent perhaps the last two weeks tracking down driver-related problems. But the term "driver bug" is heavily overused (blog around and you'll see that many OpenGL developers get frustrated...) A few examples of the gray areas:

• Sometimes there will be a bug in application code that shows up only on certain hardware. Drivers are concerned with making video cards go fast, not spanking programmers who don't know what they're doing. This is exactly what happened to me in beta 2 with the crash-on-nVidia-with-C172 bug. This was just plain broken code in X-Plane, but for some reason the ATI drivers didn't have a problem with it (probably because they were performing an optimization which let them ignore the bogus call I made). NVidia specific, but not NVidia's fault!
• On OS X with the Radeon 9600XT, runway lights don't show up. Adding an extra line to the pixel shaders "fixes" the problem. I believe the problem is in the driver (or the shader compiler more specifically) but by changing the code to the shader we work around the problem. A change in X-Plane addresses the problem, but not X-Plane's fault!
• The "36061" errors that some users have been seeing turn out to be because (through a very convoluted chain of events) X-Plane was asking the video card to operate in a mode that it could not operate in. Turns out this can be fixed by changing X-Plane's code (fix will be in beta 12) or by getting new drivers. This one wasn't really a driver bug - more that the drivers were limited in a way X-Plane did not expect. (Our fault for being picky!)

The situation is fundamentally tricky - games are integrators of other people's technology - as such, we get blamed by the end user for a fault anywhere in the system. At the same time, it's way too easy to turn around and blame the part supplier, and unfair when the source of the bug hasn't been identified.

I am looking now at problems on Windows with dual core machines and nVidia cards. The problem goes away both by changing a registry setting that affects the driver and by changing X-Plane's code. So I think it's too soon to tell on this one.

I will reply (soon)

At this point my in-box has approximately 180 emails from the month of December regarding X-Plane 9. So while I appreciate all of the feedback we've gotten (bug reports, performance, etc.) it's going to take a while to reply. If you haven't heard from me, don't panic! I hope to answer a whole pile of emails next week.

In the meantime, I've been working on improved crash logging on Windows. Right now when we have a crash on Windows, all we know is that (1) X-Plane crashed and (2) what DLL we crashed in (which is always us or the video driver - not useful).

Coming soon, X-Plane will catch the fatal crash, examine memory to see what was going on, examine its own EXE to figure out the names of the functions going on, and output it all to a crash log that users can send us to get a much clearer picture of what's going on. This information is called a "backtrace" - we've had it for the Mac or a while (OS X provides back-traces automatically with a crash) and it's really useful.*

So my top priority is all of the users who are seeing problems during scenery load, and a new build with a back-trace should help to reveal what's really going on.

I'm also working on putting additional timing and performance information into the sim so we can learn more about why some users have poor performance. So far here's what I'm seeing:

• 8800 users seem to have great performance. If you have this card and don't have good fps, adjust your x-plane settings and NV control panel settings - this card can bring it.
• 8600 users sometimes have performance problems - not sure why.
• Older nvidia GPUs (7600, 6800) sometimes have performance problems with the new eye-candy features - I am investigating.
• The pixel shaders seem to slow down the new HD2x00 Radeons a lot more than expected...I still need to investigate this. This is the most surprising datapoint - the X1600 does very well, so I would expect newer GPUs to at least have that level of performance. I think this is something we might be able to address.
  

However not all of the reports are consistent, so I think it's too soon to make some calls on recommended hardware. The only thing that's clear is that most 8800 useres who we do careful perf experiments with end up with huge framerates.

* Microsoft provides some back-trace facilities, but since we don't use their compiler tools, we had to roll our own.

Cores and Drivers and Vsync

Random thoughts:

Update your drivers! Version 9 uses driver features that version 8 does not. Just because version 864 works doens't mean that your drivers are up to date and bug free! First thing to try when weird things happen in version 9 is a driver update.

If you have 60 fps and a rendering setting cuts it to 30, you probably have vsync on - that is, your graphics card can only run at an even divisor of your refresh rate. The next hit will be 20, then fog. Change your monitor refresh rate to 75 or 80 hz. If the framerates all change (to 80, 40, 20, etc.) it must be v-sync. Turn v-sync off for better framerates under heavy load. Nvidia users, you need to turn v-sync to off, not application controlled.

X-Plane 900b7should be able to put sustained load on three cores - if you're recording a QuickTime movie, one core draws the world, one compresses QuickTime frames, and one rebuilds 3-d as you fly. So...I guess we've already hit a point where a quad-core machine has some benefit over a dual-core machine. (I think we'll start to see more central features use more cores during the v9 run.)

The new forests rebuild 3-d very frequently - dual core users who run on "tree hugger" should see utilization of 75% or higher on both cores, depending on video card power. (If your CPU usage isn't 100% then probably your video card is holding you back - turn down FSAA.)

Reflective Airplanes

I woke up this morning to one of those funny coincidences that defines the experience of working on X-Plane: two users had emailed me. One was asking whether we'd be extending water reflection technology to airplane fuselages (like some other programs have) and the other made the case that such an extension was not necessary. The two emails arrived in sequence! (Perhaps there was a forum debate on the subject somewhere.)

First, I can tell you that if we ever have reflective airplanes, it won't be that soon. I have a number of features for version 9 that are in progress and need to be finished off before I can start anything new.

Reflective airplanes are on my "investigation list"...that is, a feature where we want to do the initial research to see if it could be implemented in a way that makes sense (for X-Plane this means: would it look good and not kill fps too badly for at least some segment of our users).

I believe the X-Plane 9 version run will start to contain more features for serious 3-d modeling of airplanes. Version 9 already features 3-d lighting in the 3-d cockpit, key frames for animation, and a ton of new datarefs to drive that animation. We're going in the direction of being able to model the plane in absurd detail.

We're also looking at the lighting model in X-Plane. We've only started this work for version 9, but consider pixel-shader-based water. Even in the "no reflections" case, the pixel-shader based water is a reflection of the real sky (as rendered) with a procedural texture to create waves. When you compare this to the version 8 water, you can see how having really close alignment of the coloring scheme for all parts of the sim creates more of a sense of realism.

So reflective airplanes are at least on my list of things to try. I have seen users do wonderful amazing "metal" textures on airplanes, but the one thing that I think holds them back is that all metal airplanes have some kind of tinting assumption to them based on the reflection used...typically these are "blue-based" (meaning they look right on a sunny day) or "gray based" (meaning they look right on a cloudy day). But if you put the plane in the other environment the texture looks a lot less convincing. Reflective textures would let authors really use the real sky color on the plane, for consistent lighting (especially when the plane's orientation changes and the blue side isn't up anymore).

On the other hand, reflections are expensive. Planes reflect light from all sides, so we would need to take reflections from all angles (the water always reflects up, which is a huge savings). For low-quality settings for water, we drop the terrain, and since the terrain only reflects at the water's edge, this is a pretty tolerable omission. An airplane reflection with "sky on the bottom" would look absurd. (Similarly, the water tends to only reflect things that aren't on camera, so the total rendering load of water + the world tends to be static. The plane would pick up a lot of 3-d objects even in orientations where they don't do much good, so plane reflections would become expensive.) And the plane reflection isn't usable for any other plane...do we build them for all planes or just the user's plane?

Certainly right now it's still too soon to tell. Not only have I not done the research into this feature, but we still don't have comprehensive performance data on the water across lots of hardware. A number of users are reporting huge framerate loss on the lowest water settings. This implies that our "render-to-texture" code is slow on some hardware but not others. (The fps loss on my laptop with the lowest water setting is less than 4%.) Render-to-texture is new to v9 and used heavily, so we need to understand how it scales for all users before we go further.

Finally, there is a whole area of 3-d techniques that X-Plane does not yet use that could make sense for airplane modeling: artist controlled fake lighting.

For example, imagine if the airplane contained a single "reflection" texture - this texture would contain a fake ground texture and alpha transparency where the sky color goes. X-Plane could then fill in the sky color (where there is transparency) only when the weather conditions change, and then apply the texture keeping the plane's orientation in mind. Such a proposal would give the plausibility of reflections (correct coloring on all parts of the plane across lighting, orientation and weather conditions) for a fraction of the cost of "real" reflections. I'm not saying this is the best idea, just that there's a lot of intermediate ground between "full reflections" and "make a static texture".

Dr. House

Lori and I are hooked on the TV show "House", where Hugh Laurie plays a really grumpy doctor who solves bizarre medical cases more or less by ESP. The characters are well written, but the medical plot line is somewhat predictable: there are four quarters to the show - in each one House except the last, House will make the wrong diagnosis and the patient will get worse right before the commercial break. (Usually this involves massive bleeding or cardiac arrest going into the long commecial block at 0:30.) None of the symptoms fit until the very end when House finds the simple right explanation they just couldn't see.

This set of nvidia crash bugs felt a lot like that - we had multiple attempted fixes, some of which didn't help at all, until finally after multiple tries I found a bug that explained all of the otherwise completely weird behavior we were seeing.

But I must admit - I have brought shame on the house of X-Plane...the buggy code was mine and the mistake was really stupid. Why nVidia on Windows? As far as I can tell the optimizers present in most OpenGL engines can change whether (and how) the bug manifests itself - different OS/driver pair: different engine with different optimizers.

Now that (at risk of massively jinxing ourselves) we have the crash bug fixed, I will resume performance work. Once we get a build done with all of the immediate performance items I want to cover, we'll start collecting user reports on in-field performance. So I should have more specific instructions on what you can do to help us isolate performance problems in the next few days.

Better Bug Reports

(Probably I'll be blogging a lot today...the load/change-planes/crash/recompile cycle I am going through while working on the crash bug is a slow one - my old Dell is long in the tooth...it leaves me a lot of time to post.)

Beta can be frustrating for both users (why don't they fix the bugs I reported) and programmers (I need more details in my bug report). Here are a few thoughts on what makes an initial bug report useful:

• Precision of reproduction. This is probably the most important thing - we get a lot of "open an airplane"-type instructions. Which airplane? It turns out that a lot of bugs depend on the particular content being used. So if you know how to make a bug happen, please describe it in the most painfully precise steps possible!
• Short is beautiful. We must know precisely how to reproduce a bug, but a procedure that takes two hours kills our productivity. So please try to determine how to reproduce the bug with the minimum number of precise steps.
• Clean system. Bugs that involve only the default content shipped with the sim are more useful for us because they're quicker to find and more likely to be due to a bug in the sim itself.
• Nuke the prefs. Bug reports that start with "delete your preferences" are good because it means the bug procedure starts from a known state (the sim defaults). We get bugs that we can't reproduce because something is subtly different in our system. Killing prefs is the quickest way to eliminate this case.

As an example, the cleanest, simplest version of the nvidia crash bug would be:
1. Delete prefs.
2. Start the sim.
3. Open the C172 using the "open aircraft" dialog box.
Result: unexpected program termination before the terrain is visible.

How Framerate Dies - Glitching

Back in the good old days (that would be X-Plane 6), X-Plane's framerate would suffer in two ways as you cranked up the rendering options:

1. For most features (more visibility, more autogen) as the CPU and GPU became more heavily loaded, the framerate would gradually decrease.
2. If you ran out of VRAM (that is, the working set of textures needed per frame was more than your card's VRAM) framerate would really die fast - think 2 fps.

The reason for this second behavior was that computers couldn't shuffle textures between main memory and VRAM fast enough to render a frame in a 30th of a second.

As computers have gotten faster, this second behavior has gone away - modern cards, with fast PCIe 16x busses, can transfer textures from system memory to VRAM pretty fast - fast enough to have the working set be (slightly) larger than VRAM and still fly. So as texture memory increases, framerate decreases more gradually.

However, a new behavior has emerged: "glitching". You may have noticed that when you've got your computer set to the ragged edge of the rendering settings, as you turn the camera, the framerate will stutter for a few frames, then return to a relatively high rate (40-50 fps).

What's happening is: the working set of textures and geometry needed by X-Plane just barely fit in VRAM. But when you turn your head, a different set of textures and geometry are needed. While the card sorts out what is needed and what isn't, it spends some time needlessly shuffling textures, and eventually reaches stability, with only what's needed in VRAM, and framerate stays high.

Glitching has emerged as a mode of performance degredation because over time we've cut down the amount of "stuff" (textures and geometry) x-plane needs to draw a frame to only what's really absolutely needed. This means there is less intersection between the working set in one view and another, and it also means you can get closer to the edge of your hardware.

So my view on glitching is basically "too bad". If the working set weren't as carefully trimmed, you wouldn't have glitches, but the framerate would be entirely low, not entirely high. The only solution is to turn down settings that increase the working set (object density, world LOD, tex res, forests...) until the computer can run without glitches.

An even stranger variant of this: users sometimes report framerate getting "stuck" at 19 fps and then coming back when they change apps. The problem is that the driver doesn't know exactly what the best order of textures to keep in VRAM vs. shuffle is...as the view changes, sometimes the driver ends up with a non-optimal decision about what stays in VRAM and what goes, causing framerate to drop. Changing which app is in the foreground fixes this by temporarily pushing a lot of items out of VRAM, at which point the driver makes a different decision by luck.

Again, the solution is simple: turn down rendering settings to get the working set smaller than VRAM.

Basically, if the working set is smaller than your amount of VRAM, you should have even framerates, proportional to rendering load.

If the working set is greater than VRAM, the driver may find an optimal way to shuffle things and only decrease fps a little, or it may find a non-optimal way to shuffle and you'll get terrible fps, and that shuffle can change on the fly, causing framerate to fluctuate all over the place.

Beta 3 - Don't Panic

X-Plane 9 is currently in beta 2. Long-time X-Plane users know that a lot of bugs get fixed between early and late betas, and they also know that a good number of bugs get added between the early betas.*

If all goes well, X-Plane 9 beta 3 will be out in a few days. My advice is: don't panic. X-Plane beta 2 crashes for a number of users, so our top priority is to fix the crash and get the fix out ASAP. If your bug doesn't get fixed in beta 3, it's probably because we're still working on it but didn't want to delay getting the crash fix to users.

Similarly, beta 3 will include some performance improvements, but more are coming. Beta 3 doesn't represent the end of our performance tuning, it represents the first beta that we can do serious analysis with. We only have a fraction of all of the supported video cards within the company, so if your computer is having performance problems, well, we'll figure out what's going on in beta 3 and then fix it.

* Our approach to bug fixing is: if a piece of code is buggy because it's subsystem has a design problem, we go in and fix the design problem, even if we're in beta. Other companies might say "no fixing design problems (which changes more code) during beta." But the way I look at it, badly designed code is just going to cause problems all the time until it's fixed, and it has to be fixed some day, so why wait.

(Why would there be badly designed code in X-Plane, or any computer program? Computer programs change over time, and the functionality they perform changes and grows. As this happens, the designs of the past no longer make sense for future requirements. In my experience most design problems come from code "outgrowing" its framework.

So our approach is to upgrade the framework as soon as it shows signs of growing pains, rather than jamming as many features into the existing framework as possible until it becomes so overcrowded that we can't get anything done.)

X-Plane 9: The Absurdity of Pretending

There have been plenty of rumors and semi-official posts regarding the upcoming major revision of X-Plane (X-Plane 9). I have been trying to keep my mouth shut about it...the problem with pre-announcing anything is that the upside to us is small (at best we do what we said) and the downside is large (at worst we don't do what we said and people get grumpy). No one complains if XP9 turns out to have no-pause scenery load and it's a surprise...but plenty of people complain if we say "there won't be pauses" and then they are.

But...the situation is becoming mildly absurd...plenty of info is out there, and saying "the upcoming major release", etc. just feels political and weaselly. Austin would be disgusted.

So listen: I am going to try to provide some info on X-Plane 9. This info is subject to change. This is what we think is going to happen to the best of our knowledge. The release is still a ways away and enormous changes will happen. When things change, do not bitch to me that "you promised X" would happen. I do not promise anything. This info is provided to try to help those making add-ons for X-Plane plan appropriately.

With that in mind...I will try to post some more details on the authoring environment in the next few days. For now, here's some very basic guidance on compatibility and hardware requirements:

• The hardware requirements will be at least as high as X-Plane 8. If your machine is gasping and wheezing on 8, it's not going to be any better on 9.
• X-Plane 9 will depend more heavily on pixel shaders. If your hardware doesn't have pixel shaders (GeForce 2-4, Radeon 7000-9200) or has really crappy shaders (GeForce FX series) you will miss out on a lot of the cool stuff in v9, and possibly have the scenery look worse (but faster) than v8 (as we move features from the CPU to pixel shaders).
• Scenery that opens in x-plane should open in X-Plane 9 unmodified - if the scenery works in 8 but not 9, report it as a bug!
• Plugins that work in v8 should work in v9 without modification.

Finally, we are trying to finish up X-Plane 861...this is a bug-fix patch for version 8 - it contains no new features, but it does fix a few nasty bugs, some of which cause crashes. So if there is any new feature, it's coming in 9, not 861. Version 8 has been out for a very long time, so I will accept no argument that v8 should have more features than it does now. It's been a long run!

(One of my main goals with 861 is to try to fix any weird behavior for third party scenery add-ons, so that a scenery add-on looks the same in v8 and v9. If we left the bug in 861 and fixed it in v9, authors would have to hack the scenery to make it work with v8, and then remove the hack and republish for v9. By trying to fix the authoring bugs in v8, at least when possible, it lets authors publish one package for both versions. Of course, v9 will have new features, so I expect some v9-only scenery will emerge pretty quickly.)

X-Plane Dark Arts: Airplane Object Translucency II

In my previous post I described the underlying problems that make translucency in airplane cockit objects tricky. That gives us the background to understand how X-Plane draws cockpit objects and why.

The actual draw order for airplane-related objects is this:

1. Far away weapons.
2. All of the airplanes that we are not inside (this means the user's plane in an external view, and AI planes all the time). For each of these planes, the external cockpit object is drawn first, then the parts, then the attached objects in order.
3. Clouds and puffs and other such environmental phenomena.
4. The geometry of our plane, if it is to be drawn and we are in an inside view.
5. The attached objects in order for our plane, if internal geometry is drawn and we are in an inside view.
   
6. Weapons that are very close to the plane.
7. The inside cockpit object, if we are inside the plane.

A few things to note about this draw order:

• Note that weapons appear twice in the category of "near" and "far". This is all about the clipping plane - if a weapon is close to us, it must be drawn with the cockpit object, late in the draw cycle, when we are doing "close" things. Weapons are treated dynamically - they change where they are in the draw cycle depending on their position in space. This is necessary because a weapon starts out real close (just off your wing) and then goes real far away.
• The user's airplane is special-cased when we are in an internal view ... at that point if external things are being drawn, they are done in the later "close" view with the cockpit.
• The external cockpit object is (strangely) quite early in the draw order. There's really no good reason for that. What's particularly annoying is that it's inconsistent with the internal draw order. The main point of internal/external cockpit objects is to let you simplify your cockpit object in external view for performance.

Is there a sane way to set up an X-Plane 860 aircraft with translucent geometry? I'm not 100% sure about this, but it looks to me from the code like the correct thing to do might be:

1. Put any translucent windshield/canopy/etc. in the last attached object.
2. Put only the interior panel (and not the canopy) in the cockpit object.

This works because in the external view the canopy is drawn last, and in the internal view, the panel is more or less guaranteed to be closer than the canopy, which is good enough.