Wednesday, September 27, 2006

HOWTO: File a Performance Bug

This blog entry describes how to file a performance bug for X-Plane. You don't have to participate in X-Plane betas, so if this procedure seems scary or too complex, I suggest simply waiting for the final sim release. But if you want to help, this procedure explains how to do it. It is not enough to just tell us "the framerate is bad now" - we will just ask for the information that this post explains how to provide.

First, to file a performance bug, you will need two clean copies of X-Plane: the current beta and the previous final release. Use the web-based installers to install clean copies of both of them. (Hint: you can download the current final release, then copy the folder and run the beta installer on the copy to save download time.) We always need a relative comparison of framerate between two versions to isolate how efficient the sim is from how fast your hardware is. If the sim is slow on your computer and has always been that way, that's not really a problem with the sim. But if the sim used to be blazingly fast and now it is not, we can fix that.

In order to get a clean test we need to control every aspect of the simulator. Fortunately these clean installs have default preferences. But you will need to go through and make sure you have:
  • The same weather settings!
  • The same rendering settings. Make sure new features are off, e.g. if you are comparing 840 and 850, disable birds in 850.
  • The same aircraft.
  • The same number of AI aircraft.
  • The same location.
  • The camera facing in the exact same position and direction!
  • The same view mode (e.g. forward, forward with panel, etc.).
For your framerate test you will basically run the sim in a fixed configuration and take some screenshots.

It is very important that the camera angle be fixed. Use the "takeoff" menu item to place the plane on a runway. Do not taxi the plane into position; even the slightest change in the positioning of the camera can have a large impact on framerate, so if there can be human error in moving the camera, then the comparison is not valid. By pre-placing the plane you can get the same camera angle every time.

Some important notes on screenshots:
  • Control-period will take a screenshot in any screen, so you can take a screenshot of your weather settings, not just the sim screen. Always use control-period, not the built-in OS screenshot mechanism.
  • Please send us the original PNG files in a big zip file; please do not crop, editor or compress them!
  • Please do not run the sim at larger than 1024x1024 as the screenshots will be clipped.
For both 850 and 840 we will need three screenshots (so for each performance report we will need six screenshots):
  1. A screenshot of the sim running. Use the data outputs screen to view frame rate, plane latitude, longitude and altitude, and camera location on-screen, so that these are captured in the screenshot.
  2. A screenshot of your rendering settings.
  3. A screenshot of your weather settings.
These last two allow us to exactly copy what you have configured. Please do not use "real weather" when you do these tests.

For each performance report, please send us the six screenshots, the log.txt after quitting both versions of the sim (so two log.txt files) and please tell us in the email the framerates of each sim both when paused and unpaused. (Take the screenshot when unpaused.) So there will be four fps numbers to report for every given test.

Note that these clean installs will not have any third party add-ons; if your performance problem is only visible with a third party add-on, please:
  • First do the performance test without them and file that pair of datapoints.
  • Then install the add-on on both and make as few configuration changes as possible and then file that data too.
So for a third party bug there might be eight fps to report - all combinations of: old and new, paused and unpaused, clean and with the add-on.

No comments: