Showing posts with label off topic. Show all posts
Showing posts with label off topic. Show all posts

Thursday, January 13, 2011

LSD.glsl

Too trippy not to post:

Wednesday, January 05, 2011

I Recognize That Scenery

Thanks to Dominic for sending me this link...these guys are building an airship^H^H^H^H^Hhover^H^H^H^H, well, it's weird looking. Hrm...that airport they're landing at in the simulator looks strangely familiar.

Sunday, October 03, 2010

Tuning the Rasterizer

I'm not sure anyone cares about this kind of thing, but...




These are pictures of a test of the rasterizer. The rasterizer is code in both the DSF creator* and X-Plane that converts polygons into lines or boxes. What good is that? We use it to:
  • Plant trees inside a polygonal forest in a DSF.
  • Process all of the elevation points inside a polygonal water body when making a DSF.
Etc., etc. These were from a performance test rasterizing the Mississippi delta at 1201 x 1201; because the vector data is insanely detailed (something like 2 million line segments I think) it's a good test of performance. The blue lines represent a line fit, but the white lines are a "box fit" - that is, they ensure that not only are they "inside" the water, but the area above and below them are too.

* Programming geeks can use this code - it's in PolyRasterUtils.

Sunday, September 19, 2010

Feature Requests for Reality

With X-Plane 10 on the way, it seems that the various X-Plane forums are just filled with users listing off their favorite feature requests. If there is one uniting theme for these requests, it is this: they are pretty much all requests for X-Plane to more accurately model some neglected aspect of reality.

I would like this relationship to be two-way, with give-and-take. It doesn't seem fair that in every case, we should have to change X-Plane to model reality. So I humbly submit to the X-Plane user community: my requests for changes to reality to ease X-Plane development.

No Wind Near Water

My first request is a simple one: can we simply not have any more wind over the ocean? Ocean waves are hard to model. They are non-sinusoidal, they don't repeat, they're refractive, and they go for miles. If we could simply eliminate the wind, then we could model the ocean as a nice calm glassy lake, which would be quite a bit simpler to get right.

Only One Building

My second request is a modest one: from now on, I would like everyone to live in the same type of uniform house. It can be a nice house (several stories, with a pool in the house), but everyone's house should be the same. We have some new instancing technology for version 10; it reaches peak efficiency when a single building is repeated ad infinitum. So if you could all agree on a single house type, it would be good for everyone framerate wise.

(I realize that some of you in subdivisions in the US that already do this; hopefully the rest of the world can take a hint.)

No More Intersecting Roads

Alex and I have spent just a crazy amount of time trying to cope with what to do when two roads intersect. So my third request is: no more intersecting roads. I would like all roads to cross each other by some kind of overpass. This will eliminate a lot of artifacts.

In fact, for extra credit, if all roads could be aligned to be precisely north-south or east-west in a grid, that would be good for RAM use.

No Variation in Climate

This one isn't so important for me, but I think the art team might appreciate it. It turns out that different parts of the world have different temperature variations and rainfall, and that in turns makes the local color of grass, trees, etc. a bit different. This just makes work for our artists - they have to draw everything several times for the different climates. It would help us out a lot if you could all live in places with the same annual rainfall and temperature variation.

(Just in case this climate thing turns out to be true, I suggest everyone settle on the "hot dry" climate pattern - we have pretty good textures for that.)

Pave the Bay

Finally, when in doubt, use asphalt. We have some new shaders in version 10 that will make asphalt look pretty good.

Anyway, this is my list - I am sure the other X-Plane developers will have other requests. (I suspect Austin's life would be made easier if weather was always CAVOK, and Chris's life would be greatly simplified if there weren't any other airplanes in the sky.)

Do you have any hints as to when this new version of reality might be released, or what its minimum system requirements are?

Wednesday, May 12, 2010

The Wonderfully Filthy

From one of the podcasts I listen to I just discovered Edward Burtynsky. He takes these amazing photographs of industrial landscapes - really scary post-apocalyptic images of oil refineries, chopped up cargo ships, etc. Take a few minutes to look through some of the images.

One of the side effects of working on X-Plane scenery for the last few years is that it has made me look a lot more closely at the world. Once you try to recreate the world on a computer (and watch your digital creation fall way, way short) you realize how much intricacy and detail every-day phenomena have.

So when I saw Burtynsky's photos I immediately thought "he sees the complexity and beauty* and sadness of industrial landscapes the same way we do!"

* Beauty? I am not suggesting that the SOCAR oil fields are beautiful, a particularly good idea, or something I want more of. But I think that there can be a poetry in the image - perhaps a poetry of despair.

Thursday, April 08, 2010

Ray Tracing is the Technology of the Future...

...and it always will be!

Seriously, first, let's be clear: my opinions do not matter! X-Plane is a small program in a large market (game/graphics hardware) and as I've said before, flight simulators are not the early adopters of new tech. So (and this is a huge relief to me) I can do my job without correctly predicting the future of computer graphics.

Keep that in mind as I mouth off regarding ray tracing - I'm just some guy throwing tomatoes from the balcony. X-Plane doesn't have skin in the game, and if I prove to be totally wrong, we'll write a ray tracer when the tech scales to be flight simulator ready, and you can point to this post and have a good laugh.

With that in mind, I don't see ray tracing as being particularly interesting for games. I could make arguments that rasterization* is significantly more effecient, and will keep moving the bar each time ray tracing catches up. I could argue that "tricks" like environment mapping, shadow mapping, deferred rendering, and SSAO have continued to move effects into the rasterization space that we would have thought to be ray-tracing-only. (Heck, ray tracing doesn't even do ambient occlusion particularly well unless you are willing to burn truly insane amounts of computing power.) I could argue that there is a networking effect: GPU vendors make rasterization faster because games use it, and games use it because the GPU makers have made it fast. That's a hard cycle to break with a totally different technology.

I don't really have the stature in the world of computer graphics to say such things. Fortunately John Carmack does. Read what he has to say. I think he's spot on in pointing out that rasterization has fundamental efficiencies over ray tracing, and ray tracing doesn't offer enough real usefulness to overcome the efficiency gap and the established media pipe-line.

The interview is from 2008; a few months ago Intel announced that first-generation Larrabee hardware wouldn't be video cards at all. For all effective purposes from a game/flight simulator perspective, they basically never shipped. So as you read Carmack's contents re: Intel, you can have a good chuckle that Intels claims have proven hollow due to the lack of actual hardware to run on.

I will be happy to be proven wrong by ray tracing, or any other awesome new technology. But I am by disposition skeptical until I see it running "for real", e.g. in a real game that competes with modern games written via rasterization. Recoding old games or showing tech demos doesn't convince me, because you can recode an old game even if your throughput is 1/20th of rasterization, and you can hide a lot of sins in a tech demo.

Heck, while I'm putting my foot in my mouth, here's another one: unlimited detail. Any time someone announces the death of the triangle, I become skeptical. And their claim of processing "unlimited point cloud data in real time" strikes me as an over-simplification. Perhaps they can create a smooth level of detail experience with excellent paging characteristics (which is great!) but the detail isn't unlimited. The data is limited by your input data source, your production system, the limits of your hardware, etc. Those are the same limits that a mesh LOD system has now. In other words, what they are doing may be significantly more efficient, but they haven't made the impossible possible.

That is my general complaint with most of the "anti-rasterization" claims - they assume that mesh/rasterization systems are coded by stupid people - and yet most of the interesting algorithms for rasterization, like shadow mapping and SSAO, are quite clever. Consider these images: saying that rasterization doesn't produce nice images while showing Half Life 2 (2004, for the X-Box 360) is like saying that cars are not fuel efficient because a 1963 Cadillac got 8 mpg. The infinite detail sample images show a lot of repeated geometry, something that renderers today already do very well, if that's what was desirable (which it isn't).

Finally, is in favor of sparse voxel octrees (SVOs). SVOs strike me as the most probable of the various non-mesh-rasterization ideas floating around, and an idea that might be useful for flight simulators in some cases. To me what makes SVOs practical (and in defense of the unlimited detail folks, their algorithm potentially does this too) is that it can be mix-and-matched with existing rasterization technology, so that you only pay for the new tech where it does you some good.

* Rasterization is the process of drawing on the screen by filling in the pixels covered by a triangle with some shading.

Sunday, January 24, 2010

500th Post

This is my 500th post. I put off posting it all week because I wanted to post something lofty and clever. But in the end, the great is the enemy of the good - if I wait until I have a really good post, it could be weeks before I have time to write a 6000 word treatise on the relationship between quantum physics, shaders, and the price of crude oil.

The decision to publish less now or more later always comes up in software release planning - once the resource budget for a project is fixed* the only choice is ship sooner with less features or later with more features.

With both X-Plane and WorldEditor we often choose "ship now with less" for a simple reason: we are going to ship with more later, but if we ship now with less as well, people get some benefit in the meantime. WED is a perfect example: the first version could only edit airports, and shipped almost 18 months ago. Had we waited until we had overlay editing and airports, we would have had a more impressive release, but authors would have had no editing at all for 18 months. Why force the people who want to edit airports to wait 18 months for overlay features?

(An assumption in this is that the cost of actually doing a release is fairly low. Obviously we don't want to do a new release every single week!)

There is a contrary force that might argue against frequent releases: once we change a feature to make it better, users are surprised if we don't make it perfect. Users assume that if we fix some bugs in a feature but not all bugs, that it's because we didn't know about the bugs we didn't fix. (The truth is usually that we had limited resources.) This produces a very strange situation where users are sometimes happier with a product that is less featureful/more deficient/more buggy because a small improvement in real functionality introduces an expectation of a large improvement.

A second behavioral phenomenon amplifies this: in my experience users consider new bugs to be significantly "buggier" (for lack of a term) than bugs that have been around for a while. This is perfectly understandable: humans are very adaptable and we get used to a bug over time to the point where we may not consider it as "bad" as when we first saw it. Trade the old bug for a new one, and we have to become re-acclimated.

These two behaviors argue (particularly when bugs and limited functionality are involved) to make a small number of large changes that move an aspect of the program from one 'stable' configuration to the next.

* If you think that more resources can break this trade-off between features and a quick release date, I strongly recommend "The Mythical Man Month". The short version: 9 women can't have a baby in 1 month - if you want a quicker release you have to do less.

Wednesday, January 06, 2010

My Wife's Technology Predictions

With a new year and CES upon us, it's go time for pundits to predict the future of the technology industry. Lori considered the question of how technology might affect X-Plane and pronounced:
Pixel shaders will get shadier.
I think she is right. It is only a question of how long until they are so shady that they are basically pitch black. (At that point, I expect a significant boost in framerates!)

Monday, January 04, 2010

Virtuosity

Happy New Year! This entire post is going to be off topic, but hopefully worth it; I will resume the usual ranting about how video cards are slowly turning our brain into string cheese later.

I think my favorite part of working for Laminar Research and being part of the X-Plane community is the friends I have made in the X-Plane community - X-Plane really does bring a great group of people together. While I lived in California, I met Jay Oliver, and actually lived not too far from him while I was in ATC school.

Here's the thing about Jay: he is a fabulous musician. In an age where music is "produced" and so much of what we hear is digitally manipulated, there's a lot to be said for listening to a human being who is simply astoundingly good with his or her instrument. It's an experience that can't be synthesized. It was always enjoyable to get a few drinks into Jay and put him down in front of the Piano and then just listen.

I've been listening to Jay's new album for a few days now, but Austin beat me to the punch, putting it on the X-Plane announce list. Jay set up his new website here, and his album is available for sale in CD or DRM-free mp3 format. You can also listen online right on the site.

Austin's usual diet is highly electronic, usually runs at about 180 beats per minute, and is the musical equivalent of shooting Jolt cola directly into your brain. What Jay does is completely different, and it's really something special.

Thursday, December 03, 2009

Have You Hugged Your Driver Writer Lately?

In my contact with users, on X-Plane forums, in discussions of computer graphics, video drivers are an easy punching bad. When an app doesn't work, blame the video driver. The guys at NV and ATI don't have time to respond to every ridiculous allegation that is posted. Sometimes drivers are borked, but when it turns out to be X-Plane I try to set the record straight.

Driver writers have what might be the hardest combination of programming circumstances:
  1. Their code cannot crash or barf. X-Plane crashes, you send me some hate email. Your video driver crashes, you can't see to send me that email.
  2. The driver has to be fast . The whole point of buyng that new GeForce 600000 GTX TurboPower with HyperCache was faster framerates. If the driver isn't absolutely optimized for speed, that hardware can't do its thing.
  3. The driver writers don't have a lot of time to be fast and correct - the GeForce 700000 GTX TurboPower II will HyperCache Pro will be out 18 months from now, and they'll have to start all over again.
That's not an easy set of goals to meet. Today's video cards are basically computers on a PCIe board, and they do amazing things, but they do it thanks to a fairly complex piece of software.

Applications writers like myself get to outsource the lower level aspects of our rendering engine to driver writers. When a driver doesn't work right, it's frustrating, but when a driver does work right, it's doing some amazing things.

Tuesday, November 10, 2009

I Hear You 5-by-5

I don't usually link to non-X-Plane blogs, but I really liked this pair of posts:

http://distractible.org/2009/11/05/top-10-ways-to-annoy-your-doctor/
http://distractible.org/2009/11/08/top-10-ways-doctors-can-annoy-patients/

If you live in the US, you'll definitely appreciate it...the lists are funny and yet have a seed of painful truth in them.

So I decided to try to create my own lists.

I am only tangentially related in tech support - Randy takes on most of the work with some help from Jack. Sometimes very weird reports get escalated to me. (And most of the "let the report sit for a week" comes from me not having time to dig in.)

Anyway, please take these with a grain of salt - they're meant to be funny and exagerated. Most of our users are very, very helpful in tech support calls, despite the fact that, if you are talking to tech support X-Plane is already hosed. And Randy puts forth some amazing acts of patience in the face of some of the requests he gets. My hope here is only to show that there are two sides to the frustration in a tech support incident, and we'll all be happier if we can see the whole picture.

Five Things You Can Do To Annoy Tech Support

1. Be As Angry as Possible

Threaten to switch to Microsoft Flight Simulator. Drop the F word a few times. KEEP CAPS LOCK DOWN FOR THE ENTIRE EMAIL. Tech support definitely responds better to users who are angrier - you don't want to get sub-standard service because you were too nice, right?

2. Omit Information

If you have a second graphics card made in Kazakhstan, over-clocked and running hacked drivers you got off of the pirate bay, don't tell us. If your computer regularly catches on fire, be sure not to mention that. Did you recompile the Linux Kernel yourself after letting your pet monkey edit the thread scheduler? It's best we not know.

Extra credit: report a truly bizarre problem, provide no details on your customized configuration, wait a week and tell us how you fixed it by removing a third party program that "enhances" sound or graphics. Priceless!

3. Don't Include Past Emails In a Thread

Be sure to delete any past information from your email. Change the subject of the email so we can't tell what the original issue was. If you have more than one email, send replies from different addresses. A perfect reply would be "That didn't work" sent from an email address that you haven't used before, without your name included.

4. Email the Last Person You Talked To.

If you just finished up sorting out a shipping problem with the shipping guy, ask him how to create a plugin. If you just got info from the developers about UDP, ask them why your credit card was charged the amount it was charged.

5. Bring Up New Issues In the Middle of Old Ones.

To do this just right, wait until the thread between you and tech support is pretty deep into the meat of a complex issue. Then throw in another paragraph about something else that's gone wrong. To perfect this technique, try to pick a new problem that the person who you are emailing with isn't equipped to handle (see point 4) and keep the report vague (see point 2). You can repeat this technique to stretch out a tech support incident indefinitely.

Five Ways Tech Support Can Annoy You

1. Make the User Reinstall the OS

Reinstalling the operating system fixes approximately 0% of user problems, but it takes a really long time, and is almost guaranteed to screw something else up, usually something that wasn't broken and isn't related to X-Plane. If a user is a little bit annoyed, this is a great way to pour gasoline on the flames.

This is really a special case of the general strategy "ask the use to do something time consuming, annoying, and unlikely to help."

2. Forward the User a Huge FAQ, None of Which is Relevant to the Problem

Everyone likes form letters and impersonal service. The FAQ should be badly written, badly formatted, confusing to read, and preferably not accidentally contain the real solution to the problem. If the solution to the problem is in the FAQ, don't tell the user where in the FAQ to look.

3. Wait a Long Time Between Replies

Tech support incidents are like fine wines - they get better with age. To allow the user's annoyance to bloom into a finely honed rage, be sure to let each email 'sit' for a week before replying. This works especially well if your response is just to ask another question, setting the user up for another week's delay.

4. Blame Some Other Component

The modern PC is built by approximately 600 different vendors. Blame one of them. The beauty of this strategy is that it is one that can be used by every vendor who provided software or hardware for the PC. Also, because quite often the problem really is with another component, you can claim this with a straight face.

Tip: blame the graphics card maker - ATI and NVidia do not have the resources to pursue every complaint that an over-clocked graphics card running the latest patch to some simulator written by two guys in their bedrooms crashed with the drivers visible somewhere in the callstack. Put the blame on the GPU makers - they don't have the resources to refute you, no matter how bogus your claim.

5. Forward the User's Issue Around the Company Until It Gets Lost and Dropped

Everyone in the company has to be in on this strategy for it to work - if one of your idiot coworkers actually solves the user's problem, well that defeats the purpose. This strategy can be combined with (3) and is sort of a riff on (4) - once the user complains that they got dropped, blame everyone else in the company for the mis-communication.

Tuesday, September 29, 2009

Off Topic: Freecycle

This is off topic, and I try to keep the off-topic political crap on this blog to a minimum. But I do want to put a plug in for Freecycle.

Freecycle is basically a series of locally based mailing lists for the free exchange of things you would have thrown out. My wife and I use it every time we move - we get packing materials from other people who have just moved, and then give them back to our new local group when we are done. We also freecycle all of "that old junk" that we realize we shouldn't move with us to the next location.

If you are ever in a situation where you need to throw out items that are potentially useful but just not worth the cost of carrying anymore, please consider Freecycle - it strikes me as a reasonably reliable way to keep items out of landfills. (Especially hard-to-recycle mixed-material items like electronics.)

One note of caution: freecycle is just a group of people exchanging "stuff" in their spare time - typically it could be 2-3 days before someone can pick up your donated item. So...if you have a lot of stuff to give away, start early. I made this mistake in DC and wasn't able to give away all of the items because people couldn't pick them up soon enough.

Sunday, September 27, 2009

Closing the Washington Branch

Laminar Research has closed its Washington, DC satellite office and opened a new office outside Boston. Okay - that is completely cheeky - LR employees all work at home, and I have moved from the DC area to the Boston area.

This has put me a little bit behind schedule with, well, everything. I have four or so interesting example plugins almost ready to be published, WED is ready for some kind of developer preview, and I have some X-Plane 940 bug reports to go through. I hope to get the back-log cleared out this week. But first priority: unpacking the computers (and figuring out how to get an internet drop into the new home office).

Sunday, September 06, 2009

Pay No Attention To That Last Post

My last blog post may have expressed some negative views toward software upgrades:
I know I'll take some slack for this, but: why did you all go and grab an OS update so fast? What have you gained? The down-side is lost time and application compatibility problems...was it worth it?
That blog post was of course not my real opinion, but rather a recreation of the evil Ben. Always upgrade your software to the newest version whenever it comes out!

Thursday, September 03, 2009

Marketing And Fact

A while ago, Austin was on FSBreak, and I wrote this post as commentary on his interview. The main point I meant to make was this: from what I have heard from other engineers and seen in my own experience, most software companies prefer to develop new features on top of implementations that are known to have architectural problems. At LR, we fix the implementation architectural problem first, and that has been a net win for us.

Now that's basically a statement of my opinion on software engineering - in hindsight it probably belongs on my programming blog and not here. Unless you develop X-Plane plugins, you're not a programmer; I will try to constrain future scenery blog posts to things that non-programming X-Plane users will notice. If you are a plugin developer, you might want to look at the "Hacks of Life" posts tagged with OpenGL.

Anyway, back to the story...the responses to that blog post were all well thought out comments on X-Plane's quality control process. At the time my immediate reaction was: that's totally off topic - I'm commenting on architecture and they're talking about QA. I do think the authors made fair points.

But in hindsight, I think that there's a deeper issue: one of verifiability. Simply put, my statement that we (LR) rewrite stale implementations is nearly impossible to verify without source code access, something that you can't get for X-Plane. So from the perspective of anyone outside the company, my original statement is not falsifiable (it cannot be proven false) and thus rather useless as a statement of fact. Even though I claim that we make rapid progress on features by keeping implementations clean, you as a user don't care how we develop our features - clean architecture, more developers, or the use of time travel and voodoo dolls, it's a bit moot.

Thus the comments were off topic, but also they were moving away from an unverifiable topic and toward one that users can measure, namely the quality of X-Plane's betas.

There's a fair amount of marketing that gets put out in the tech and games industry. It's a slippery slope from giving a new, real technology a whiz-bang name (e.g. HyperZ is a real technology, and it is good for your frame-rates) to using techno-babble to make the bad seem good (e.g. HyperMemory just means that your video card lacks VRAM and is going to be slow). When new products come out, the feature list is parroted, but it's not always clear whether the new features turn into real benefits.

So what I'm going to try to do with the scenery blog is: try to keep the blog limited to verifiable, measurable aspects of X-Plane. If we ship X-Plane with "psychoactive rendering*", I'll try to explain what the heck that is and why you'd want it, and how you might notice that it's working.

* X-Plane does not have psycho-active rendering, except possibly when I make a mistake in the shaders and everything turns purple.

Monday, August 17, 2009

I am the Spam King!

If I have not emailed you back, it's probably because I've been very busy trying to interleave X-Plane coding and packing up the house. But it is also possible that my email response bounced because my web server has ended up on a bunch of anti-spam "black-lists".

Black-listing seems like a good idea at first: we'll gather up a list of all of the IP addresses from which spam comes from, then publish them. Then your local mail server can use that list to filter spam - you never see it!

In practice it doesn't work so well...example: http://www.mipspace.com/. The IP address for my server (XSquawkBox) is now on this list. Why?
MIPSpace is a list of IP Addresses associated with known commercial marketing companies.
Since my server is used for my own personal email and to run the SDK website, I'm not sure why I am on the list. I have sent them an email to clear things, but in general I hit an anti-spam/black-list bounce somewhat frequently now, and frankly I don't have time to separately try to clear my name from every guilty-until-proven-innocent blacklist that pops up and screws up my email.

If I seem disproportionately grumpy about this, it could be due to one of two reasons:
  1. Not replying to emails is generally bad customer service. (Okay, my in-box is backed up four months...that's bad.) I don't like the idea that a customer might perceive us (LR) as being unresponsive because some third party with no skin in the game decides to black-list us.

    The blacklist has no incentive to be accurate - it's not their lost customers if email doesn't go through.

  2. I'm not at all convinced that this is going to cut down unsolicited commercial mail and/or spam.

    In the spam case, spammers can send from botnets - they have access to a huge number of ever-changing IPs. Unless we are prepared to blacklist the entire internet, the blacklists are going to pick up more and more false positives while spammers find ways to harvest fresh, untainted IP addresses. The whole IP-reputation strategy assumes that IPs are hard to change. In practice, IPs are very, very easy to change.

    Commercial mail is a lost cause too - even if I am being solicited for commercial mail I don't want, no program or automatic process is ever going to tell the difference between the confirmation of my invoice and a list of discounts from the same company. When it comes to commercial mail, the reputation damage has to be done to the company, not the IP.

    (The company does have reputation to risk - if we are known as a company that doesn't honnor a "do not subscribe" policy, then customers can choose to not buy our products.)

It could also be because I ran out of Viagra and don't have a diploma from a prestigious non-acredited university.

Sunday, December 28, 2008

This Is Where SRTM Comes From

My family visited DC this weekend and we went out to Udvar-Hazy, the extension of the Smithsonian aerospace museum out near Dulles international airport. My dad took this picture.

That is one of the two radar antennas (and the telescoping arm) used to scan the earth as part of the Shuttle Radar Topography Mission (SRTM). The SRTM is basically the first really good quality most-of-the-earth elevation dataset, and it is the main (but not only) source of elevation data for the X-Plane global scenery.

The telescoping mast shown in the picture (horizontal) extends one of the two radar antennas away from the shuttle when in orbit; had they not been able to retract the antenna they would have had to detach it and leave it in space. Fortunately the mechanism worked properly, so they were able to bring the antenna back for posterity.

Tuesday, October 14, 2008

Another Programmer

With the iPhone and X-Plane 9, we've been very busy...with this in mind, I am pleased to announce the latest engineer to join Laminar Research.



"Nubblet" would like you to know that the MacBook Pro is, in fact, "hers". :-)

Friday, July 25, 2008

AWOL Again

It's that time of the year again...I'll be out of the office next week, and more importantly, away from all cell and email contact.

Austin and Randy will be at EAA - stop by and say hi if you're there. Once Austin gets back, the last few betas should resume - we're close to an RC on 920 but not quite there.

Saturday, May 31, 2008

Probability and Certainty

I've been reading Fooled by Randomness (highly recommended - it will either change your life or you won't finish it because Taleb's writing style annoys you) - and it has me thinking about the nature of certainty in software development.  Consider two approaches to uncertainty and how they are completely at odds with each other.

No Weird Behavior

The "No Weird Behavior" approach goes like this: you never want a harmless behavior inside your code that you don't understand.  The reason is that if you don't understand the behavior, you don't really know that it's harmless!

In fact this isn't just a theoretical problem - I only truly started to believe in "No Weird Behavior" after fixing several very hard to find, hard to reproduce crash bugs, only to discover (once the analysis was done) that the broken code also produced a very frequent, less harmful behavior.  Had I taken the time to investigate the "weird behavior" first (despite it not being of high importance) it would have led me to a ticking time bomb.

No Weird Behavior implies that a bug isn't fixed until you know why it's fixed, and that randomly changing code until the behavior goes away is absolutely unacceptable.  This shouldn't be surprising; if a bridge was swaying would you accept an engineer who fixed it by randomly tightening and loosening screws until it stopped?

Wait And See

I get a lot of email with bug reports, questions, and reports of strange sim behavior.  These emails have some unfortunate statistical properties:
  • A good number of them are resolved by the user who emailed within a day or two.
  • A certain percentage are simply never resolved.  (Often I email a question that would point to a clear diagnosis and the user never writes back.  I can only speculate that the user answered the question, found the problem, fixed it, and promptly forgot about our emails.)
  • A certain percentage are solved by the user, who tells me what the problem was, and it was something I would never, ever be able to help them with.  (Things like "I have this program called WickedMP3Player - it turns out if I set its visualizer setting to 'Rainbow' then X-Plane stops crashing" when I've never even heard of the WickedMP3Player program to begin with.)
  • There is a high correlation between bug reports reported by a very small number of users and a resolution from the above categories, or a resolution by the user changing his or her local configuration.
So playing the odds, the right thing to do when presented by a third party with weird behavior is to wait and see who else reports it; the frequency of reports gives us information about the likely resolution.

(And for those of you who think our tech support are being lame when they ask if you've updated your drivers, they are playing the odds to the hilt - they ask you about drivers first because updating drivers fixes an absurdly huge number of the tech support calls we get.)

Getting It Wrong

So we have motivation to investigate everything (no weird behavior), motivation to ignore everything (wait and see) and a rule of thumb that the frequency of reports can help us pick which strategy is best.  Of course, sometimes the rule of thumb is completely wrong.

One user reported a crash using the current web updater (version 2.04) - I had not seen this crash anywhere and it was inside the operating system UI code, so I assumed it was a local problem, e.g. some kind of extension or add-on that caused the OS problems.

The problem, it turns out, is simply low frequency - as the incorrect code made it into 902b1, I got a few reports from more users and realized that this wasn't a local problem, it was weird behavior!  (The bug will be fixed in the 205 installer and 902b2, both of which will be out in June.)

Consider this: if you make a measurement of a known quantity with a dubious measuring device, you learn more about the measuring device than the object you are measuring.  (If you have a ruler and you don't know the units, you could determine them by measuring yourself.)

In a number of cases, we have seen serious "should-happen-all-the-time" crash bugs that get reported by very few users.  (Once we know the actual root cause of the bug, we can deduce logically whether the bug should happen to all users with the configuration or be intermittent.) We can then look back at the actual number of reports to make a judgement call on how much testing is really happening.

For example, we can make some guesses about how quickly a new Macintosh have saturated the X-Plane user base when there is a hardware-specific serious bug in just that machine.

We had this with the iMacs (where the runway lights were broken) and we could watch the machines sell - slowly at first, but then quite quickly.  (BTW, I think that 10.5.3 fixes this and anti-aliasing problems.)  We can even see who runs with anti-aliasing when there is a setting-specific bug (a lot of you do)!

In the end, I think the right approach to balancing "no weird behavior" and "wait and see" is to remember a truth about uncertainty that is very hard for humans to grasp:

The most likely outcome of an uncertain situation is not guaranteed to happen - in fact a lot of the time it won't.*

So we can play the rule of thumb and wait and see, but we always have to keep one eye toward the improbable, which is still possible!

* Blatant blog cross-promotion...the point of my big rant here was essentially the same...it's easy to expect the most likely outcome, but the unlikely outcome will happen some of the time.