Saturday, December 01, 2007

Better Bug Reports

(Probably I'll be blogging a lot today...the load/change-planes/crash/recompile cycle I am going through while working on the crash bug is a slow one - my old Dell is long in the tooth...it leaves me a lot of time to post.)

Beta can be frustrating for both users (why don't they fix the bugs I reported) and programmers (I need more details in my bug report). Here are a few thoughts on what makes an initial bug report useful:
  • Precision of reproduction. This is probably the most important thing - we get a lot of "open an airplane"-type instructions. Which airplane? It turns out that a lot of bugs depend on the particular content being used. So if you know how to make a bug happen, please describe it in the most painfully precise steps possible!
  • Short is beautiful. We must know precisely how to reproduce a bug, but a procedure that takes two hours kills our productivity. So please try to determine how to reproduce the bug with the minimum number of precise steps.
  • Clean system. Bugs that involve only the default content shipped with the sim are more useful for us because they're quicker to find and more likely to be due to a bug in the sim itself.
  • Nuke the prefs. Bug reports that start with "delete your preferences" are good because it means the bug procedure starts from a known state (the sim defaults). We get bugs that we can't reproduce because something is subtly different in our system. Killing prefs is the quickest way to eliminate this case.
As an example, the cleanest, simplest version of the nvidia crash bug would be:
1. Delete prefs.
2. Start the sim.
3. Open the C172 using the "open aircraft" dialog box.
Result: unexpected program termination before the terrain is visible.

No comments: