To Tune Framerate, You Have To Hit Rock Bottom

I have blogged about this before, but I keep seeing this issue come up in the forums, so I want to go over it again. If you want to effectively tune X-Plane to trade off maximum visual quality with desired framerate, you must first reduce your rendering settings all the way to the bottom, then work your way up. Let me explain with an analogy.

I have fallen off my motorcycle and hit my head, skinned my knee, and broken my arm; the bone is sticking right out of my skin! Ouch! So I go to the hospital and the Doctor examines me. Here is how the conversation goes:

Dr: How does your arm feel?
Me: My arm hurts so much! OUCH!
Dr: And how does your leg feel?
Me: My arm is burning and stinging! Please make it stop!
Dr: Do you feel dizzy or light headed?
Me: Please fix my arm!!!!!!!!

Clearly with a bone sticking through my skin, there is no point in doing a physical examination. My arm hurts so much that I can't tell the Doctor whether I have any other aches or pain. I feel one thing: the arm.

What does this have to do with framerate? Well, framerate is just like pain. The low framerate you see is caused by only the one worst problem with your setup. If your graphics card is a little bit overloaded, you are a hair short on VRAM, and your CPU is absolutely being killed, then the low framerate you see is totally because of the CPU. The weakest link decides your framerate. And like the Doctor, if we go trying to deal with the VRAM problem, we will see no change because it's the CPU that hurts the most.

This is what I see over and over: a user is running X-Plane, his framerate is bad, and he has turned some but not all of the settings down. At this point the user is usually pretty grumpy - his visuals now look bad and his framerate is poor.

The problem is that the user hasn't turned down the setting that really matters. This is why the first thing you need to do in order to tune framerate is to turn everything down, so that you are running with framerate at least as high as what you want for your target value. Then you can turn settings up one at a time and watch which one lowers framerate.

(Don't worry, you're not going to stay at the lowest settings. The key is just to always be turning settings up, not down.)

Here are some of the settings I see that need to be turned down but often are not.

• Full screen anti-aliasing. (FSAA) Always turn FSAA down to none. FSAA will kill fps on any graphics card that is fill rate limited.
• Pixel shader effects - every one of these should be turned off to start. And when you do start them, try them one at a time and have the water reflections off - work your way up in small steps. The gap from shaders without volumetric fog, shadows, reflections and per pixel lighting to shaders with all of these effects is huge!
• Turn objects all the way down to default, or even off. If your CPU is the problem, objects could be a factor.
• Leave texture compression on in your "rock bottom" settings. Texture compression improves fps and lowers visual quality, so having the check box be on is the minimal setting.
• Don't run at a huge screen res or FOV. Run 1024x768 and 50 degrees FOV.

What happens if you turn everything down and you still see 19 fps? Now it's time to investigate your video drivers and control panel settings. If your setup is even remotely better than minimum hardware, it should scream when all of those settings are turned down. If you still see low fps, check drivers, check control panel settings. There are a lot of control panel settings, for example, that will completely kill framerate.

Too Much Water Can Be a Bad Thing

In MeshTool 2.0, you can specify how wet orthophotos are handled. There are three possibilities:

1. The orthophoto has no alpha-based water. The alpha channel will be ignored.
2. The orthophoto has alpha-based water. Draw water under the alpha, but for physics, make the triangles act "solid".
3. The orthophoto has alpha-based water. Draw water under teh alpha, and for physics make the triangles act "wet".

The reason for 2 and 3 is that the X-Plane physics engine doesn't look at your alpha channel - wet/dry polygons are decided on a per triangle basis. (The typical work-around is to use the "mask" feature in MeshTool to make some parts of the orthophoto be physics-wet and some physics-solid. This is described in the MeshTool README.)

Whenever possible: don't use alpha-based water at all. It is certainly easy to set all of your orthophotos to alpha-water + physics-solid, but there are three costs to this:

1. You eat a lot of fill rate. X-Plane manages alpha=water by drawing the water underneath the entire orthophoto, then painting over it with the orthophoto. This is fill-rate expensive. If you know there is no alpha, tell MeshTool, so it can avoid creating that "under-layer" of water.
2. If the terrain is very mountainous, you may get Z-buffer artifacts from the layering, particularly for thin, spikey mountains (which probably aren't wet anyway).
3. The reflection engine tries to figure out the "surface level" of the water, but it doesn't understand the alpha channel on top of the water. So all of that water "under" your mountain or hill is going to throw the reflection engine into hysterics.

Limiting the use of water under your orthophotos fixes all three problems.

1,2,3,4,5...6,7,8,9,10,11,12

Surely I'm not the only one with that song embedded in some deep part of my psyche, right? One thing we seem to have down with 940 is counting upward...945 is in testing now.

There are two things driving the perpetual 940 bug fix releases:

1. Third parties are using the sim heavily, so the bug fix releases have been vehicles for fixing a number of small bugs that are only reproducible with custom add-ons. (The fix for night lighting and orthophotos is an example of this.)
2. We've played a real game of whac-a-mole with the throttles. (The bug is that X-Plane will lock the two throttles together. I believe that the new 945 release candidate finally fixes this.) Having done a bit of a 3 stooges routine on this bug, we've tightened up our internal procedures a bit.

As always, if your third party add-on broke with 940, please file a bug report! The only case of this I know of is the throttles bug - 945 should fix the BK-117, the Mu, and any other plane suffering from throttle-glue.

Will there be a 946? There very well might be, and I sincerely hope it will not be because 945 missed its target. But if, two months from now, we've collected another half-dozen fixes for third party authoring, I don't see why we wouldn't release them.

Could The Name Beta Be Any More Confusing?

The version number gets bumped when we have a release and it still needs patching. Of course, some bugs get reported after we ship, and some address issues not even under consideration during release. Had we not had any fire drills, we would probably still be on 942 or 943 right now, because we've had third party reports with very specialized authoring cases after we finalized the build.

Here's what makes things even more confusing: that check box "Check for new betas as well as updates" in the installer. At any one time we may have two versions of X-Plane availalbe for download:

1. The official version (currently 944) for everyone.
2. Whatever the latest in-test build is (currently 945).

(You can even see which versions are current here.) What's confusing is that while this check box says "get betas", in truth it gets any build that is "in staging". At this instant that I blog, X-Plane 9.45 final is in staging. That is, we believe 9.45 fixes our two known bugs, we think it's done, it's a release candidate, we think it is the next American Idol^H^H^H^H^H^H^H^HX-Plane. But until we certify it, you still have to check "get betas" to download it.

What we probably need to do is re-label the check-box in some way; the "staged" approach where every build is available for test first is, I think, a valuable tool to let third party authors confirm that we fixed their bugs before the entire world gets the app.

Yet Another Tools Update

A few quick notes on the various betas floating around:

• X-Plane: with the 9.4x series we are trying to fix small one-off bugs and improve stability. The BK-117's throttles will be fixed in the next beta, and we fixed a crash with a third party orthophoto scenery pack.

  This goes for any beta, but: if you have an add-on that used to work on an older build (especially 940) and it does not work, please report this immediately! There are no intentional add-on compatibility breaks in this patch.

• WorldEditor: apparently developer preview 2 doesn't contain the latest texture coordinate editor (TCE) code, so I will try to cut a preview 3 soon. The texture coordinate editor is a new small editor that lets you specify the texture placement on draped polygons. This facility lets you create custom markings. (The current TCE in preview 2 apparently doesn't have snap to grid and a few other useful features.)

  What's holding us back from a real beta? The problem is that the DSF export of draped polygons cannot split bezier curved polygons (especially with texture coordinates) across the boundaries of DSF tiles. Since I don't have an algorithm implementation for this yet, I'm not sure how soon I will be able to fix it. For now, don't try to export a DSF polygon that spans DSF tiles.

• MeshTool 2: beta 4 seems to be a keeper. I hope to cut an RC some time in the next week. (You can see the results of MeshTool 2.0 in these FranceVFR preview pics.)

500th Post

This is my 500th post. I put off posting it all week because I wanted to post something lofty and clever. But in the end, the great is the enemy of the good - if I wait until I have a really good post, it could be weeks before I have time to write a 6000 word treatise on the relationship between quantum physics, shaders, and the price of crude oil.

The decision to publish less now or more later always comes up in software release planning - once the resource budget for a project is fixed* the only choice is ship sooner with less features or later with more features.

With both X-Plane and WorldEditor we often choose "ship now with less" for a simple reason: we are going to ship with more later, but if we ship now with less as well, people get some benefit in the meantime. WED is a perfect example: the first version could only edit airports, and shipped almost 18 months ago. Had we waited until we had overlay editing and airports, we would have had a more impressive release, but authors would have had no editing at all for 18 months. Why force the people who want to edit airports to wait 18 months for overlay features?

(An assumption in this is that the cost of actually doing a release is fairly low. Obviously we don't want to do a new release every single week!)

There is a contrary force that might argue against frequent releases: once we change a feature to make it better, users are surprised if we don't make it perfect. Users assume that if we fix some bugs in a feature but not all bugs, that it's because we didn't know about the bugs we didn't fix. (The truth is usually that we had limited resources.) This produces a very strange situation where users are sometimes happier with a product that is less featureful/more deficient/more buggy because a small improvement in real functionality introduces an expectation of a large improvement.

A second behavioral phenomenon amplifies this: in my experience users consider new bugs to be significantly "buggier" (for lack of a term) than bugs that have been around for a while. This is perfectly understandable: humans are very adaptable and we get used to a bug over time to the point where we may not consider it as "bad" as when we first saw it. Trade the old bug for a new one, and we have to become re-acclimated.

These two behaviors argue (particularly when bugs and limited functionality are involved) to make a small number of large changes that move an aspect of the program from one 'stable' configuration to the next.

* If you think that more resources can break this trade-off between features and a quick release date, I strongly recommend "The Mythical Man Month". The short version: 9 women can't have a baby in 1 month - if you want a quicker release you have to do less.

Please Report Spam

I've been very busy with X-Plane feature development...when the blog is quiet, usually something interesting is getting coded. (Actually this weekend, I've just been sick in bed, so...hrm...when the blog gets quiet, perhaps I am infected.)

Just a quick note: if you see any kind of spam or vandalism on the wikis, please let me know immediately. The wiki engine we use (mediaWiki) has some fairly advanced anti-attack capabilities, but they need to be brought in when a problem occurs. Over the last two weeks I've cleaned out repeated attacks on the SDK site and one attack on the main site, and it appears that the software installed now is working.

But if more attacks come in, the next step is filting, e.g. programming the wiki to automatically reject posts based on typical off-topic keywords like certain ED drugs, etc. So if you see a post like this, please let me know - I'll kill the post and set up the anti-spam filter.

Developer Hardware

So...just how awesome is my main development machine? Not that awesome.

Periodically users ask me what my setup is. Usually the user wants to set up a really nice machine to run X-Plane at its best and figures "let's find out what the guys who write X-Plane have."

But ... my main development machine is definitely not selected to be the best possible X-Plane machine. I put together a system to:

• Debug X-Plane productively.
• Test all aspects of X-Plane.
• Create the global scenery as fast as possible.

In practice this means a Mac Pro with 8 cores, 12 GB of RAM, a lot of hard drive, and not particularly fast CPUs. It's a machine that can churn out 8 DSFs at once. There's no need for me to keep upgrading the CPU type itself - it's the 8 way parallelism that makes DSF creation fast. (To Apple's credit, the IO in the machine is fast enough to keep all 8 cores busy!)

Being an Intel Mac, the machine is triple-bootable into Vista (someone in the company has to have it) and Ubuntu Linux.

Right now I have a Radeon 4870 in the machine and an 8800 on my shelf. I do recommend the 4870 to Mac users - it's a very nice card. But for my purposes it has one annoying problem: it takes up the space for the second graphics card slot and both power connectors...I may go back to a lower powered card so I can have one NV and one ATI card in the machine at the same time - a great configuration for debugging. (I do not recommend that any user ever mix graphics card brands..."don't try this at home", etc.)

Maxing out X-Plane isn't on the priority list. In particular, past these goals, the faster the machine, the less likely I am to notice a problem.

An example: during 930 development, for some period of time, we had accidentally set the code to allocate an extra 1 GB of RAM at startup. Oops! The embarrassing part: neither Austin and I noticed for weeks. Both of our machines have plenty of RAM, and OS X has a decent VM system, so we just ran, using a lot of memory.

Then one day I try to start X-Plane on my laptop and the whole machine nearly catches on fire. Sure enough...an extra 1 GB of RAM is being grabbed.

The moral of the story: I'd rather not have a machine that hides things from me, if it doesn't affect productivity.

X-Plane on the Pre

Palm announced 3-d gaming for the Palm Pre! Some big names there...Need for Speed, Monopoly, the Sims 3, X-Plane -- hey!! Laminar Research?!?! Who let those hacks in!??

I have the original Palm Pre (not the newly announced Palm Pre Plus) and flight is quite smooth! Thanks to the Palm team for letting us in on the roll-out.

My Wife's Technology Predictions

With a new year and CES upon us, it's go time for pundits to predict the future of the technology industry. Lori considered the question of how technology might affect X-Plane and pronounced:

Pixel shaders will get shadier.

I think she is right. It is only a question of how long until they are so shady that they are basically pitch black. (At that point, I expect a significant boost in framerates!)

The 3-d Panel Is Not Always Necessary

There is no need to use the 3-d panel if you only want 3-d cockpit.

That might be the most counter-intuitive statement in the entire universe. Let's break it down and see what's going on. Originally we had the 2-D panel. The 2-D panel was where you put your instruments for interior views. When 3-d cockpit support was added, the 2-D panel was used to create the panel texture - that is, the instruments texture for the 3-D cockpit.

Thiat worked for a while, but as handles became more complex, screens became larger, and as 3-D cockpits became a more complex, authors started to run into the system's limitations. If the author wanted to create a huge panel, then the panel texture for the 3-D cockpit was huge. Furthermore, texture space for the instruments in the 3-D cockpit was wasted because the 2-D panel had to have windows on it.

With X-Plane and 9 we added the 3-D panel to fix these problems. The 3-D panel is a second panel whose sole purpose is to provide instruments as a texture for the 3-D cockpit. When you have a 2-D panel and a 3-D panel, the 2-D panel is used to draw the panel in 2-D viewing modes and 3-D panel is used only to create the panel texture for the 3-D cockpit. With this setup the 2-D panel can be huge, and the 3-D panel can be small and compact with no windows.

The 3-D panel solves a second problem as well. Often the overhead panel is drawn in perspective on the 2-D panel this perspective view is useless for texturing a true 3-D cockpit. With a 2-D panel and a 3-D panel, the author can draw the overhead in perspective for the 2-D panel, and orthographically for the 3-D panel.

Now here's where things get complicated: if an airplane has no 3-D panel the 2-D panel is used for both the 2-D view and the 3-D cockpit. But some airplanes have only a 3-D cockpit. In this case, there is no reason to use the 3-D panel at all. You can simply use the 2-D panel for texturing the 3-D cockpit and not worry about the user seeing it; in a 3-D only plane the user will only see the panel as a texture for the 3-D cockpit.

This setup is confusing in name: how can you have a 2-D panel used for a 3-D cockpits? But it is an allowed configuration. In fact, it was the only configuration allowed in X-Plane 8.

In summary: a 3-D cockpit can use the 2-D panel or 3-D panel as its panel texture. If an airplane has no 2-D panel, then the 2-D panel can be used for the 3-D cockpits without problems. In this situation and there is no need for a 3-D panel.

(It might be better to think of the 2-D panel and three panels simply add his two panels so that you can have two different versions of the panel. The names to the panel and 3-D panel are really just labels.)

Virtuosity

Happy New Year! This entire post is going to be off topic, but hopefully worth it; I will resume the usual ranting about how video cards are slowly turning our brain into string cheese later.

I think my favorite part of working for Laminar Research and being part of the X-Plane community is the friends I have made in the X-Plane community - X-Plane really does bring a great group of people together. While I lived in California, I met Jay Oliver, and actually lived not too far from him while I was in ATC school.

Here's the thing about Jay: he is a fabulous musician. In an age where music is "produced" and so much of what we hear is digitally manipulated, there's a lot to be said for listening to a human being who is simply astoundingly good with his or her instrument. It's an experience that can't be synthesized. It was always enjoyable to get a few drinks into Jay and put him down in front of the Piano and then just listen.

I've been listening to Jay's new album for a few days now, but Austin beat me to the punch, putting it on the X-Plane announce list. Jay set up his new website here, and his album is available for sale in CD or DRM-free mp3 format. You can also listen online right on the site.

Austin's usual diet is highly electronic, usually runs at about 180 beats per minute, and is the musical equivalent of shooting Jolt cola directly into your brain. What Jay does is completely different, and it's really something special.