The X-Plane 850 Framerate Test

Now that I've covered how to use X-Plane with command-line options and how to write up performance bugs, we can discuss the framerate test, why it exists, who should use it, etc.

The X-Plane built-in framerate test serves three purposes:

1. To let us compare our current performance to past performance easily (so we can detect code that hurts sim performance early in our development cycle). This also lets us rapidly determine whether bug reports of "my framerate sucks" are valid or not.
   
2. To allow graphics-card driver writers to use X-Plane to benchmark and regress changes to their drivers.
3. To give us an easy way to get "hard numbers" in-field - that is, when a user emails us with a tech support request saying "my sim is slow", the fps test allows us to determine whether the problem is based on configuration.

I've spent a lot of time in the X-Plane 850 release cycle looking at performance, both with the help of users (e.g. instructing users on how to look at their own machines) and my own. For all the complaining I've done I must say: THANK YOU to all of the users who have helped with performance analysis. These users put in a lot of hours doing mundane test after test and they did a great job of emailing me back information, fps reports, screenshots, and even a Shark profile or two! Even when the result is "hrm - everything is normal", this kind of information is valuable in helping us prove that our release is solid. (And all of the indications I've found indicate that 850 is solid.)

But the process was very time consuming and error-prone. The frame-rate test simplifies things for everyone; the test is standardized and requires no user intervention; it runs the exact same settings every time and is not subject to software configuration issues. The reports are automatically printed out and require no observation.

We are adding the test into 850 so that in the next release we can compare to 850 and be sure we're on track. I only wish we'd have the test in 840!

Should I Be Using the Framerate Test?

If you are not a developer/programmer and you are not asked to use the framerate test by a programmer (for example, in response to a bug report or after emailing tech support) there is no need for you to run the framerate test. You can run the test and email your friends (especially if you have higher framerate than they do) but if you have problems with the framerate test and you are using it "recreationally" please do not file a buf report or query tech support for help. The framerate test is an internal development tool; X-Plane is meant to be used to fly!

What the Test Does:

The framerate test runs X-Plane with preference reading/writing disabled, and it changes the 'default' configuration (e.g. X-Plane's configuration without preferences) based on command-line input. This gives us a specific setup of the sim regardless of how the sim was last used. The preferences are not saved to disk when the test is done, so there is no need for a user to delete preferences to get solid test numbers. User alerts are not shown (just logged) so that the test can be run from a shell script automatically. The sim logs info and quits when it finishes.

The performance test can run in two modes: analysis mode and pass-fail mode. In analysis mode, the sim is run in three 30-second tests: panel view, forward view, and forward-view paused. These three numbers help us tell the relative framerate loss due to the panel and the flight model. The fps-test is primarily aimed at the 3-d rendering engine, but on a CPU-limited machine the flight model can have a huge impact, and on a PCI/AGP bandwidth limited or VRAM-limited machine the panel can cause a noticable slowdown. (The panel also hurts if a machine is extremely blend/fill-rate limited, like a Rage 128, but you shouldn't be using X-Plane 8 on a Rage 128!)

The pass-fail test runs a single test, paused with the panel to gauge pure framerate, and returns 0 or 1 (as a command-line shell app) to the calling script based on whether the sim passes a minimum framerate. The pass-fail test is designed for automated testing; you can put X-plane into a series of standardized tests and automatically send an email if the sim's performance falls below a certain minimum.

How to Use the Framerate Test:

First, see my instructions on how to use X-Plane with a command-line option. In these examples I will simplify the X-Plane executable name; refer to the instructions for your specific operating system.

To run the frame-rate test, use this:

X-Plane --fps_test=2

Where the number is a fps-test "test number". (I will explain the numbers in detail below.) This will run the 3-part analysis test and quit. You can add other command-line options as well; for example, to compare performance with and without VBOs you can do this:

X-Plane --fps_test=2 --no_vbos

(Side note: on fps-test 1 pass-fail I see 71 fps on my MacBook Pro with VBOs and 59 without. Turning off VBOs definitely hurts performance, so only users with serious driver bugs should be using the --no_vbos option.)

To run the pass-fail test, you add a second command-line option:

X-Plane --fps_test=3 --require_fps=20

This will run the pass-fail test and return a positive result if the frame-rate is at or above 20 fps. For example, you could write a test like this:

X-Plane --fps_test=2 --require_fps=20 || echo We were below 20 fps!

Test Numbers:

The framerate test number is up to 3 digits controlling various aspects of the sim:

• The ones digit can be 1-3 and selects the rendering settings; 1 = very low and 3 = very high. You can view the rendering settings while the fps test its running (it will ruin your test results of course to put up dialog boxes).
• The tens digit can be 0-6 and programs in a variety of weather and visibility conditions.
• The hundreds digit can access a few special features: 100-series tests use a far view angle and 200-series tests run at night.

We may invent more tests later on but this initial set lets us run the sim in a wide enough envelope to gather the data we need.