Friday, October 06, 2006

The X-Plane 850 Framerate Test

Now that I've covered how to use X-Plane with command-line options and how to write up performance bugs, we can discuss the framerate test, why it exists, who should use it, etc.

The X-Plane built-in framerate test serves three purposes:
  1. To let us compare our current performance to past performance easily (so we can detect code that hurts sim performance early in our development cycle). This also lets us rapidly determine whether bug reports of "my framerate sucks" are valid or not.
  2. To allow graphics-card driver writers to use X-Plane to benchmark and regress changes to their drivers.
  3. To give us an easy way to get "hard numbers" in-field - that is, when a user emails us with a tech support request saying "my sim is slow", the fps test allows us to determine whether the problem is based on configuration.
I've spent a lot of time in the X-Plane 850 release cycle looking at performance, both with the help of users (e.g. instructing users on how to look at their own machines) and my own. For all the complaining I've done I must say: THANK YOU to all of the users who have helped with performance analysis. These users put in a lot of hours doing mundane test after test and they did a great job of emailing me back information, fps reports, screenshots, and even a Shark profile or two! Even when the result is "hrm - everything is normal", this kind of information is valuable in helping us prove that our release is solid. (And all of the indications I've found indicate that 850 is solid.)

But the process was very time consuming and error-prone. The frame-rate test simplifies things for everyone; the test is standardized and requires no user intervention; it runs the exact same settings every time and is not subject to software configuration issues. The reports are automatically printed out and require no observation.

We are adding the test into 850 so that in the next release we can compare to 850 and be sure we're on track. I only wish we'd have the test in 840!

Should I Be Using the Framerate Test?

If you are not a developer/programmer and you are not asked to use the framerate test by a programmer (for example, in response to a bug report or after emailing tech support) there is no need for you to run the framerate test. You can run the test and email your friends (especially if you have higher framerate than they do) but if you have problems with the framerate test and you are using it "recreationally" please do not file a buf report or query tech support for help. The framerate test is an internal development tool; X-Plane is meant to be used to fly!

What the Test Does:

The framerate test runs X-Plane with preference reading/writing disabled, and it changes the 'default' configuration (e.g. X-Plane's configuration without preferences) based on command-line input. This gives us a specific setup of the sim regardless of how the sim was last used. The preferences are not saved to disk when the test is done, so there is no need for a user to delete preferences to get solid test numbers. User alerts are not shown (just logged) so that the test can be run from a shell script automatically. The sim logs info and quits when it finishes.

The performance test can run in two modes: analysis mode and pass-fail mode. In analysis mode, the sim is run in three 30-second tests: panel view, forward view, and forward-view paused. These three numbers help us tell the relative framerate loss due to the panel and the flight model. The fps-test is primarily aimed at the 3-d rendering engine, but on a CPU-limited machine the flight model can have a huge impact, and on a PCI/AGP bandwidth limited or VRAM-limited machine the panel can cause a noticable slowdown. (The panel also hurts if a machine is extremely blend/fill-rate limited, like a Rage 128, but you shouldn't be using X-Plane 8 on a Rage 128!)

The pass-fail test runs a single test, paused with the panel to gauge pure framerate, and returns 0 or 1 (as a command-line shell app) to the calling script based on whether the sim passes a minimum framerate. The pass-fail test is designed for automated testing; you can put X-plane into a series of standardized tests and automatically send an email if the sim's performance falls below a certain minimum.

How to Use the Framerate Test:

First, see my instructions on how to use X-Plane with a command-line option. In these examples I will simplify the X-Plane executable name; refer to the instructions for your specific operating system.

To run the frame-rate test, use this:
X-Plane --fps_test=2
Where the number is a fps-test "test number". (I will explain the numbers in detail below.) This will run the 3-part analysis test and quit. You can add other command-line options as well; for example, to compare performance with and without VBOs you can do this:
X-Plane --fps_test=2 --no_vbos
(Side note: on fps-test 1 pass-fail I see 71 fps on my MacBook Pro with VBOs and 59 without. Turning off VBOs definitely hurts performance, so only users with serious driver bugs should be using the --no_vbos option.)

To run the pass-fail test, you add a second command-line option:
X-Plane --fps_test=3 --require_fps=20
This will run the pass-fail test and return a positive result if the frame-rate is at or above 20 fps. For example, you could write a test like this:
X-Plane --fps_test=2 --require_fps=20 || echo We were below 20 fps!
Test Numbers:

The framerate test number is up to 3 digits controlling various aspects of the sim:
  • The ones digit can be 1-3 and selects the rendering settings; 1 = very low and 3 = very high. You can view the rendering settings while the fps test its running (it will ruin your test results of course to put up dialog boxes).
  • The tens digit can be 0-6 and programs in a variety of weather and visibility conditions.
  • The hundreds digit can access a few special features: 100-series tests use a far view angle and 200-series tests run at night.
We may invent more tests later on but this initial set lets us run the sim in a wide enough envelope to gather the data we need.

4 comments:

Mike Ash said...

Have you considered adding a basic GUI to this? All of our apps present a special debugging panel when started with the Option key held down which lets the user do things like enable logging and various other debugging features. I think that this would help make this more accessible to your users, particularly to people in the Mac world (where I work) where people aren't always so comfortable mucking about at the CLI. This may well be more complexity than you want, but I can imagine it being helpful in the customer support case, and I thought I'd suggest it.

Benjamin Supnik said...

A few thoughts on that...
- We've used command-line tests mainly for things where we need to induce changes before the sim has initialized OpenGL...so making these GUI features might make them inaccessible if we have a load-time driver bug.
- In the case of the fps test, similar principles apply - the fps test acts on the sim very early in te init time.
- The fps test is primarily geared at us and driver writers, so the command-line interface is top priority...I don't expect users to have to use it a lot in normal use.

There actually are a lot of debugging variables inside X-Plane...grab the DataRefEditor plugin and pick "view art controls" or "view stats" to see all the private junk X-Plane posts. We have other plugins that let us execute special commands and remotely trigger profiling. I chose to use a plugin for this rather than bake it into X-Plane because:
- The dataref mechanism has zero overhead when not in use and is completely hidden when the plugin is not installed, so we can be sure it's not interfering with users in any way. Thus we can leave it in the production system.
- We recycle the UI code I already wrote for dataref editing for plugin authors, which saved time.

So there's always a trade-off with these things regarding both how much time we spend on the tools and who the target audience is.

Caravelle said...

"This gives us a specific setup of the sim regardless of how the sim was last used."

Not really, at least on Windows.
This test keeps and uses the last screen resolution, the last color depht and the last antialias set by user.
And another difference could disturb if the user had removed KSBD scenery or not.

Benjamin Supnik said...

You are right that the fps test is susceptible to some factors beyond our control.

In the case of add-ons (plugins, drivers, removing the KSBD package, adding global scenery), and command line options, we can see all this from the log file, so with a log file rather than the numbers, this stuff is easily handled.

For driver settings, we can't tell from the log file that some settings might be overridden. However, we almost certainly would see a strange pattern of fps between multiple tests (e.g. if fps test 2 and 3 run at the same speed, you've probably forced FSAA in the driver).

*Cheers*
ben