Thursday, November 19, 2009

Cake and Pie

On this week's FSBreak, Holger Sandmann described MSFS's scenery system as a "pie" made of layers. This made me think: well, in that case X-Plane must be a cake. It turns out X-Plane has been a cake for a while.

If this discussion seems tasty but confusing, let me clarify. The issue is: which parts of a scenery system are combined when the simulator runs, and which parts must be combined ("baked") in advance.

With MSFS, you can separately install a new mesh, new land class data, new coastlines, and new orthophotos. In X-Plane, all four of those elements must be pre-combined into a single "base mesh". In X-Plane, you have to bake those elements.

This means that you can't make "just" an add-on mesh for X-Plane. You have to create an add-on that addresses elevation and land class and orthophotos and water. Third party authors are quite often not very happy about this - I don't know how many times I've seen "can I replace just the elevation and not the water" posted in forums.

But the Orbx team brings up the exact reason why I thought (five years ago when designing DSF) that requiring all of the elements would be okay: often if you replace one part of the scenery and not another, the results are inconsistent. If you move a coastline but don't adjust the mesh, you may have water climbing up a mountain. If you move a mountain but don't adjust the landuse, you may have a farm at a 45 degree angle.

Cake and Pie

Not everything in X-Plane is pre-baked. As of X-Plane 940, you must pre-bake:
  • Land class
  • Wide-scale orthophotos
  • Terrain Mesh
  • Coastlines
You can separately replace:
  • All art assets (E.g. change or add types of buildings, replace the look of a given land class for an area).
  • All 3-d (forests, buildings, airports, roads*).
  • Small area orthophotos
Orthophotos are on the list twice. This is because X-Plane has two orthophoto technologies. Draped polygons (.pol files) can be overlaid anywhere, but they degrade performance if used over too large of an area. They are really meant for airport surface areas and other "detail" work. Orthophotos terrain (.ter files) can be used over huge areas with no performance problem, but must be baked.

* Roads are sort of a special case in X-Plane 9: you can replace them with an overlay, but the replacement roads must be "shaped" to the underlying terrain mesh, which means they won't work well if a custom mesh is installed. This limitation in the road code dates back to X-Plane 8, when most of us had only one CPU - pre-conforming the road mesh to the terrain shape saved load time.

Monday, November 16, 2009

The Legacy of the R300

You know it as the Radeon 9700 (and its later cheaper 9600XT, as well as the X300, X500 and X600). The R300 was a definite victory for ATI - when it came out it was vastly superior to the NVidia GeForce FX series, and for a video card it held up pretty well.

From my perspective as an application's developer, however, the R300 has some fine print that makes it difficult to deal with:
  • It features only 24-bit floating point precision (as opposed to 32-bit precision in all other shader-enabled hardware from ATI or NV). This is why the reflective water looks square and pixelated up close on these cards.
  • It has a 96 instruction limit per shader (as opposed to the 1024 instruction or larger limit in all other shader-enabled hardware from ATI or NV.) X-Plane 9's current water shader is right on the bleeding edge of exceeding this limit. In fact, the water pattern is simplified for this set of GPUs to stay within the 96 instruction limit.
  • Since the cards were really quite decent for their time, they are still in field and in use.
(This is a different problem from the Geforce FX/5 series...those GPUs were fully capable of running advanced shaders, they just couldn't do it quickly. In practice this is better for an application's developer, because users are more likely to replace a slow GeForce FX than a capable but internally limited 9700.)

This shows up in X-Plane as a pile of special cases...X-Plane 9 productizes 2.5 renderers:
  • A no shader renderer for old GPUs and buggy drivers.
  • A shader-based renderer for modern hardware.
  • A special-case on the shader based GPU to meet the limits of the R300.
I'm not sure what we will do about this in the future. But as I go through the shader code, this special casing sticks out. We don't usually special case X-Plane for a particular set of hardware, but in the case of the R300, it was for a while so widespread that it made sense to special-case for its limitations.

Saturday, November 14, 2009

You Do Not Need to Resave Airplanes - Really!

Let me set the record straight on this: you should not need to re-save an airplane to have it work in a newer version of X-Plane. If you have to do this, X-Plane is broken ... please report a bug!

(In the case of 940 - there is a big fat bug - see the end of the post.)

Here's a little bit more about what's going on under the hood.

When Austin creates a new revision of the acf format (which happens in virtually every major patch), he handles backward compatibility with old aircrafts in one of two ways:
  1. He sets the default value of a setting to match the "unused" value in the old ACF file and sets this default value to match the legacy behavior. This naturally initializes all newly introduced functionality to its "backward compatible default" for old airplanes.
  2. Where this is not possible, he writes some conversion code that maps old ACF values to new ACF values. This mapping tries to re-create the old systems functionality as closely as possible.
This forward conversion code runs in two cases:
  • When you open the airplane in Plane-Maker.
  • When you open the airplane in X-Plane.
Plane-Maker will resave the plane in the newest format, with the automatic system updates in place. But this should not be necessary because X-Plane applies the same automatic process on airplane load. This is why you should not need to resave - X-Plane will do the upgrade "on the fly".

Now about that bug...it turns out that 940 incorrectly updates 930 airplanes - the generator amperage is not correctly initialized. This is why 930 planes will run their batteries down in 940. (This bug was fixed in 941 beta 2, btw.)

What was strange was that, because of the way Plane-Maker's code was structured, this code was failing in X-Plane but succeeding in Plane-Maker. This doesn't happen very often (usually the code fails everywhere) but the result was authors noticing that their planes would start working if resaved in PM.

And that brings me back to the beginning of the post. If Plane-Maker can update the airplane but X-Plane cannot, that's a bug! Please report it as such.

I want to make sure people realize that auto-update should work, and that resaving in Plane-Maker should not be necessary. Otherwise authors will start silently resaving their airplane instead of reporting the bugs, and we'll never find them.

(Systems bugs sometimes only show up with a particular combination of systems settings. So while I do hope that we can catch all such bugs in beta, it is always possible that one peculiar model will induce a bug once the sim is released.)

Scenery Compatibility

I had a conversation with a third party developer the other day - he offered to sign a non-disclosure-agreement (NDA) and wanted to know: would custom scenery he made for X-Plane now keep working with the next major version?

Well, no NDA is necessary for that. The simple answer is: yes! (A more complete answer is: if you use the current file formats and not legacy formats from 9 years ago, then yes.) Here's a quick review of how long the various scenery and modeling file formats have been supported:
  • DSF: 5 years
  • OBJ8: 5 years
  • OBJ7: 7 years
  • ENV: 9 years
  • OBJ2: 9 years
With X-Plane 8 we rewrote a lot of the rendering engine - since then, rendering engine enhancements have been incremental, building on what we have. The next major version will be like that too: the new version will do more than X-Plane 9, but it's not going to drop existing capabilities.

So if you model a building with an OBJ or model a terrain area with a DSF, I expect that it will work unmodified with the next major version.

Modeling Formats in Detail

X-Plane 9 supports 3 revisions of the OBJ file format (X-Plane's modeling format):
  • Version 800, which is the current version. Introduced with version 8, the OBJ800 format has been extended heavily, but the format was never changed, so original version 800 objects are not incompatible.
  • Version 700, which was used with version 7.
  • Version 2, whic hwas used for most of the X-Plane 6 run.
The next major version will clearly support version 800, and will probably support version 700 as well.

Do not make new objects in the version 700 format! This format is obsolete, supported only for legacy purposes, and is an inferior format.

We may drop support for version 2 objects - I haven't seen user content with a version 2 object in a very long time. Version 2 objects date back to a time when every polygon was expensive, so content authored in version 2 is likely to look, well, ten years old.

If you do have version 2 content, you can use ObjConverter to convert it to version 800 format. (ObjConverter will also convert version 700 OBJs to version 800.)

Scenery Formats

DSF has been our scenery format for five years now, and will continue to be so. DSF has not had any format revisions - new features are supported by allowing DSF data to be tied to new art asset formats.

I do not know if we will support ENV in the next major version. Supporting ENV is relatively trivial in the code, but whenever there is a bug, we have to fix it in the legacy ENV code as well. ENV supports a 500m terrain mesh, which is completely obsolete by today's standard.

Do not make new content in the ENV format! Like OBJ 2 and 700, it is a legacy format for backward compatibility.

Wednesday, November 11, 2009

DirectX, OpenGL and X-Plane

I commented about this once a long time ago, but with DirectX 11 cards coming to market, I think some clarification may be in order. What exactly is the relationship between DirectX, OpenGL, your graphics card, and X-Plane?

OpenGL and DirectX

OpenGL and DirectX (well, technically the Direct3D part of DirectX) are both:
  • Specifications for how an application requests that graphics be drawn.
  • Specification for what will be drawn by a library or video card when those requests are made.
(Okay that's a gross simplification, but it will work for now.)

But DirectX is also something else: it specifies what the hardware must do. That's different from OpenGL. You can have an OpenGL compliant renderer that uses the CPU for everything difficult. It will be slow, but correct.

Extensions and Versions

Both DirectX and OpenGL have version increases, and the new version increase specifies new functionality. But OpenGL also has extensions. While normally new features come in a new spec verison, OpenGL's extensions allow OpenGL implementations to pick up new functionality "a la carte". OpenGL implementers can pick and choose what they add.

When Features Show Up

The trend in game GPUs has been for new features to show up in the DirectX spec first, then become available in OpenGL via an extension, and then make it into a future core OpenGL version. For example, most DirectX 10 features were available once DX10 hardware was released, in the form of extensions.

Sometimes the features flow in the opposite direction. For example, ATI's tessellation technology may make it into the DirectX 11 spec, but is already available as an OpenGL extension.

How This Affects X-Plane

To be blunt, X-Plane is not an early adopter of GPU tech. We are a small company so we have to prioritize our feature work carefully, and there's strong motivation to prioritize a feature that helps all users over a feature that helps only users with certain hardware.

So by the time we code a hardware dependent feature, the feature is usually available in OpenGL...I can't think of any cases where not using DirectX has held us up.

For performance, DirectX vs. OpenGL doesn't really matter - both provide access to the "fast path" of the harware - this is the code path where the GPU runs its fastest. At that point, it's a question of hardware, not OpenGL vs. DirectX.

So X-Plane uses OpenGL, but both are fine for rendering engine development (unless you need to be cross-platform), and both provide reasonable access to video card features for X-Plane.

Tuesday, November 10, 2009

I Hear You 5-by-5

I don't usually link to non-X-Plane blogs, but I really liked this pair of posts:

http://distractible.org/2009/11/05/top-10-ways-to-annoy-your-doctor/
http://distractible.org/2009/11/08/top-10-ways-doctors-can-annoy-patients/

If you live in the US, you'll definitely appreciate it...the lists are funny and yet have a seed of painful truth in them.

So I decided to try to create my own lists.

I am only tangentially related in tech support - Randy takes on most of the work with some help from Jack. Sometimes very weird reports get escalated to me. (And most of the "let the report sit for a week" comes from me not having time to dig in.)

Anyway, please take these with a grain of salt - they're meant to be funny and exagerated. Most of our users are very, very helpful in tech support calls, despite the fact that, if you are talking to tech support X-Plane is already hosed. And Randy puts forth some amazing acts of patience in the face of some of the requests he gets. My hope here is only to show that there are two sides to the frustration in a tech support incident, and we'll all be happier if we can see the whole picture.

Five Things You Can Do To Annoy Tech Support

1. Be As Angry as Possible

Threaten to switch to Microsoft Flight Simulator. Drop the F word a few times. KEEP CAPS LOCK DOWN FOR THE ENTIRE EMAIL. Tech support definitely responds better to users who are angrier - you don't want to get sub-standard service because you were too nice, right?

2. Omit Information

If you have a second graphics card made in Kazakhstan, over-clocked and running hacked drivers you got off of the pirate bay, don't tell us. If your computer regularly catches on fire, be sure not to mention that. Did you recompile the Linux Kernel yourself after letting your pet monkey edit the thread scheduler? It's best we not know.

Extra credit: report a truly bizarre problem, provide no details on your customized configuration, wait a week and tell us how you fixed it by removing a third party program that "enhances" sound or graphics. Priceless!

3. Don't Include Past Emails In a Thread

Be sure to delete any past information from your email. Change the subject of the email so we can't tell what the original issue was. If you have more than one email, send replies from different addresses. A perfect reply would be "That didn't work" sent from an email address that you haven't used before, without your name included.

4. Email the Last Person You Talked To.

If you just finished up sorting out a shipping problem with the shipping guy, ask him how to create a plugin. If you just got info from the developers about UDP, ask them why your credit card was charged the amount it was charged.

5. Bring Up New Issues In the Middle of Old Ones.

To do this just right, wait until the thread between you and tech support is pretty deep into the meat of a complex issue. Then throw in another paragraph about something else that's gone wrong. To perfect this technique, try to pick a new problem that the person who you are emailing with isn't equipped to handle (see point 4) and keep the report vague (see point 2). You can repeat this technique to stretch out a tech support incident indefinitely.

Five Ways Tech Support Can Annoy You

1. Make the User Reinstall the OS

Reinstalling the operating system fixes approximately 0% of user problems, but it takes a really long time, and is almost guaranteed to screw something else up, usually something that wasn't broken and isn't related to X-Plane. If a user is a little bit annoyed, this is a great way to pour gasoline on the flames.

This is really a special case of the general strategy "ask the use to do something time consuming, annoying, and unlikely to help."

2. Forward the User a Huge FAQ, None of Which is Relevant to the Problem

Everyone likes form letters and impersonal service. The FAQ should be badly written, badly formatted, confusing to read, and preferably not accidentally contain the real solution to the problem. If the solution to the problem is in the FAQ, don't tell the user where in the FAQ to look.

3. Wait a Long Time Between Replies

Tech support incidents are like fine wines - they get better with age. To allow the user's annoyance to bloom into a finely honed rage, be sure to let each email 'sit' for a week before replying. This works especially well if your response is just to ask another question, setting the user up for another week's delay.

4. Blame Some Other Component

The modern PC is built by approximately 600 different vendors. Blame one of them. The beauty of this strategy is that it is one that can be used by every vendor who provided software or hardware for the PC. Also, because quite often the problem really is with another component, you can claim this with a straight face.

Tip: blame the graphics card maker - ATI and NVidia do not have the resources to pursue every complaint that an over-clocked graphics card running the latest patch to some simulator written by two guys in their bedrooms crashed with the drivers visible somewhere in the callstack. Put the blame on the GPU makers - they don't have the resources to refute you, no matter how bogus your claim.

5. Forward the User's Issue Around the Company Until It Gets Lost and Dropped

Everyone in the company has to be in on this strategy for it to work - if one of your idiot coworkers actually solves the user's problem, well that defeats the purpose. This strategy can be combined with (3) and is sort of a riff on (4) - once the user complains that they got dropped, blame everyone else in the company for the mis-communication.

Monday, November 09, 2009

XSquawkBox 1.3

This will be a little bit "off topic", and I am normally pretty adamant about not blogging (or even talking) about XSquawkBox. This is because XSB is not something I do as part of my job for Laminar Research, and it is also something that I no longer have time for. Thus my normal highest priority is to make sure that no XSquawkBox users expect timely support or expect this from Laminar Research.

So as I am about to blog about all of the new and cool things coming in XSquawkBox 1.3, let me first make this painfully clear:
XSquawkBox is a piece of code that Wade and I write in our spare time. After finishing our regular job of coding, rather than sit back and read a book or take a walk in the park, we go back to our computers and write yet more code. We don't make any money off of XSB, and no one pays any money for XSB.

This type of freeware development has certain constraints: there is no time table, XSB will be on the back burner when real work comes up or when real life comes up. There is no support. There are no guarantees of features or functionality. If you don't like those terms, I will issuse you a full refund and you can use a different piece of freeware.
Okay - that was a little bit over the top. We want people to enjoy XSB - otherwise we wouldn't have published it. And 99.999% of XSB users are very understanding and appreciative of the development situation.

With that in mind, I can't resist posting some of what is coming in XSB 1.3 - I think it's going to be a really strong release. 1.3 will have the bug fixes people are waiting for, including a fix for the dreaded noise on push-to-talk on Windows. Also coming:
  • Wade programmed the server list to auto-populate off the net. How did we live without that?
  • COM2 can tune into voice channels too. COM2 has its own hardware selection for those with multiple sound cards or USB headsets.
  • You can turn on "labels" for traffic. I actually hate this feature, but I programmed it anyway - it's optional. I'll post why I did it some other time.
  • XSB can connect to itself for users with multiple X-Plane visuals. Simply run XSB on every computer, log in from one (the "master") and then connect to the master from each visual (the "slave"). You'll have traffic visible across monitors.
And of course the best feature of all (for me): Wade is doing builds. :-) People owe him a big thank-you..without him there would simply be no 1.3. I definitely don't have time for it...I maxed out my XSB time for a while contributing some of the features.

When will 1.3 go beta? I don't know...ask Wade. Wait - don't ask Wade. He'll get to it when he is able!

Friday, November 06, 2009

Why Not Ignore Problems?

Tom had a good question that others have asked me too: why not ignore missing scenery resources? Why does X-Plane act so cranky?

Ignorance Is Bliss

We have tried the other approach: ignore missing art assets. ENV based scenery in version 7 did not require custom objects to actually be available - missing objects were ignored.

When I was working on the ENV reader for version 8 (the ENV code needed to be retrofit into the new rendering engine) I found to my surprise that virtually every ENV-based custom scenery pack I looked at was missing at least a few of the OBJs that the ENV referenced! I don't know how this happened - it seems that in the process of working on scenery, authors started to "lose" objects and simply never noticed.

Quality Control

When we developed DSF we had a chance at a clean slate: there were no DSFs in existence so we could set the rules for art assets any way we wanted. So we picked the harshest rule possible: any missing art asset was illegal and would cause X-Plane to refuse to load the scenery package, with no way to ignore the error. Why be this rude?
  • Missing artwork failures are 100% reproducible - you don't have to try your package more than once to see the problem. If you are missing an art asset, you will have the failure every single time you run.
  • The error is found on load - you don't have to fly over the art asset to discover that it is missing.
  • Therefore if an author tests a scenery package even once, even in the most trivial way, he or she will discover the missing art asset.
  • Once the error is fixed, it is fixed forever, so a scenery pack that passes this quality control measure in development will be just fine "in the wild".
  • This rule has been in place since 8.0 beta 1 for DSFs, so there are no legacy DSF files that would have this problem.
Libraries

There is one special case worth mentioning: a scenery pack might reference an art asset in another scenery pack, and that other scenery pack might not be installed. This is why the library file format allows for "export_backup". (Read more here and here.) Export_backup is your scenery pack's way of sayingg "only use this art asset if you can't find it somewhere else. It lets you provide emergency art-work in the event the other library is not installed.

What should you use as an emergency backup art asset? It could be anything - a big floating question mark, an empty object, a poor approximation of the desired art asset. But my main point is that responsibility for location of art assets lies with the author of a pack - so if you make a scenery pack, be sure to provide backups for any libraries you use.

(If you use OpenSceneryX, the library comes with a "developer pack" - read more here. Basically they already built a "backup" library that you can put in your scenery pack to avoid nasty messages from X-plane when OSX isn't installed.)

Wednesday, November 04, 2009

Deep In the Goo

No posts for two weeks...sorry, I've been head's down with next-gen tech. It's a little bit too early to blog about this stuff.

I was able to fix a few MeshTool bugs, but I have more problem reports, so I might be able to do a MeshTool beta 3 in a few days if things go smoothly.

940 is final - there might be a 941 maintenance release - we'll know in a few weeks.

Friday, October 16, 2009

Order Independent Transparency - What It Fixes

Yesterday I blogged about Order Independent Transparency (OIT). That robot is very shiny, but what does this fix in X-Plane itself? Here are two pictures that show the problem in an X-Plane context.

This is the Cirrus jet in X-Plane 9.22.

Pretty, eh? But...look through the right two windows - look at the passenger door on the other side. Note that through the middle window the passenger door is visible. Through the right window, the entire passenger door is gone!

This is the same shot from X-Plane 940, where the problem has been corrected.

The bug you see here is incorrect draw order. In X-Plane 922, the right door (which contains the right-most near window) is drawn before the left door - hence its translucent part destroys the door.

Max fixed this by changing the draw order of the model - the new cirrus draws the inside view of both doors before the outside view of either doors; because the geometry is one-sided he can draw the 'inside' first and outside 'second'.

These two blog posts explain translucency in a lot more detail.

What X-Plane has now is "order dependent" transparency - if translucent surfaces are not drawn from far to near, you get artifacts like the missing door.

What OIT promises (and that robot demonstrates) is the ability to have translucent geometry and not have the objects behind the translucent parts disappear.

Thursday, October 15, 2009

I'm Looking Through You

Order independent transparency (OIT): first, the shiny robot.

So first, what's so special about this? Well, if you've ever worked with a lot of translucency in X-Plane, you know that it doesn't work very well - you invariably get some surfaces that disappear at some camera angles.

The problem is that current GL rendering requires translucent surfaces to be drawn from farthest to nearest, and who is far and who is near changes as the camera changes. There are lots of tricks for trying to get the draw order mostly right, but in the end it's somewhere between a huge pain in the ass and impossible.

What's cool about the robot data is that the graphics card is drawing the transparency even if it is not drawn from back to front, which means the app can just shovel piles of translucent triangles into the card and let the hardware sort it out (literally).

X-Plane is currently riddled with transparency-order bugs, and the only thing we can do is burn a pile of CPU and add a ton of complexity to solve some of them partly. That proposition doesn't make me happy.

So I am keeping an eye on hardware-accelerated OIT - it's a case where a GPU feature would make it easier for modelers to create great looking content.

Sunday, October 11, 2009

WED - Click to Split

This feature is not in the WED developer preview (because I just coded it) but: WED 1.1 will feature "click-to-split" for edges. With WED 1.1 you can option-click the edge of a polygon or line feature (but not a must-be-straight entity like a runway) to insert and drag a split point.

This will hopefully be a lot easier than the current, convoluted, WED 1.0 technique of selecting the two surrounding vertices and picking "split", then repositioning the vertex.

Hackers: this features is not yet checked into the tree, so ... building from source won't help you. It'll be available some time in the next week.

Saturday, October 10, 2009

A Developer Preview

I realize as I write this that I am going to get some comments mocking the fact that X-Plane 940 is on RC, um...13. I don't decide the milestones for X-Plane, nor do I decide the version numbers. If you want to discuss why X-Plane is 940 (and not 930 or 950) and why it goes beta and RC when it does, email Austin. What follows is all about the scenery tools, not X-Plane.

With that out of the way: I have released a WorldEditor 1.1 developer preview. So I wanted to explain in a little bit more detail what the difference is between a developer preview and a beta. Here is an approximation of the standard definitions of "milestones" - they are what I use for WED.
  1. Development: not all features are coded, no guarantees about bugs.
  2. Alpha: all features are coded, no bug is so severe that you can't at least try a feature. (For example, if WED crashed on startup, it would not be alpha, because you could not test saving files.)
  3. Beta: all requirements of alpha, also no bugs that cause program crash or data loss.
  4. Release: no open bugs.
This all applies to known bugs. Beta software may crash and cause data loss - it's just that we wouldn't have put it out as beta knowing that this happened.

WED 1.1 is still in phase (1) - development, and the build I posted is a developer preview - a cut of whatever code I had laying around. So: I can't promise it isn't going to trash your data or crash! Be even more cautious with the developer preview than you would with a normal beta. You don't want to run a five hour session without saving your work, and you want to be backing up your work often - the "save" command might trash your entire project.

Why do a developer preview if it's still so buggy? Some users who know how to compile WED from its source code are already using WED 1.1 and they seem to be enjoying it. So far it appears not to be lethally broken. Given that and the fact that most of the uncoded WED 1.1 features are usability and edge-cases, it seemed like the developer preview could be useful for getting earlier feedback.

One last note: the manual is not updated at all, nor is there any documentation on the new features. Let me be clear: no tech support or help is provided what-so-ever. Do not email me, or X-Plane tech support with "how do I use WED 1.1" questions. If you cannot figure out how to use WED 1.1 on your own, don't use the developer preview.

Wednesday, October 07, 2009

XSquawkBox Transition

We've made some progress in transitioning XSquawkBox to be under Wade's primary management - read more here.

Monday, October 05, 2009

Never Ignore A Bug You Don't Understand

First, I want to thank the X-Plane users who have gone through piles and piles of regression tests to isolate some of the peskier bugs. Often a bug only occurs on hardware that we don't have in-house, so this detailed reporting is a huge help in shipping a program that has to run on a huge number of configurations.

Sometimes a user will offer to work around a bug by changing an add-on, or just dropping the add-on. But...I have learned the hard way: never ignore a bug you don't understand.

First, the bug might run much deeper than the reported use. Perhaps the bug is actually affecting dozens of other add-ons.

If we don't understand the bug, how can we say "this is so unimportant that it can be ignored"?

Now some bugs, once diagnosed, may prove to be not worth fixing. But...until the bug is fully understood, we have to take the time to dig in. We can't just give up because the bug seems unimportant.

Tuesday, September 29, 2009

Off Topic: Freecycle

This is off topic, and I try to keep the off-topic political crap on this blog to a minimum. But I do want to put a plug in for Freecycle.

Freecycle is basically a series of locally based mailing lists for the free exchange of things you would have thrown out. My wife and I use it every time we move - we get packing materials from other people who have just moved, and then give them back to our new local group when we are done. We also freecycle all of "that old junk" that we realize we shouldn't move with us to the next location.

If you are ever in a situation where you need to throw out items that are potentially useful but just not worth the cost of carrying anymore, please consider Freecycle - it strikes me as a reasonably reliable way to keep items out of landfills. (Especially hard-to-recycle mixed-material items like electronics.)

One note of caution: freecycle is just a group of people exchanging "stuff" in their spare time - typically it could be 2-3 days before someone can pick up your donated item. So...if you have a lot of stuff to give away, start early. I made this mistake in DC and wasn't able to give away all of the items because people couldn't pick them up soon enough.

Sunday, September 27, 2009

Closing the Washington Branch

Laminar Research has closed its Washington, DC satellite office and opened a new office outside Boston. Okay - that is completely cheeky - LR employees all work at home, and I have moved from the DC area to the Boston area.

This has put me a little bit behind schedule with, well, everything. I have four or so interesting example plugins almost ready to be published, WED is ready for some kind of developer preview, and I have some X-Plane 940 bug reports to go through. I hope to get the back-log cleared out this week. But first priority: unpacking the computers (and figuring out how to get an internet drop into the new home office).

Tuesday, September 22, 2009

Normal Maps Can't Be Mirrored

If you have ever textured an airplane, then you know that you can't use the same texture for both sides of the plane if there is writing on the fuselage. The writing will be horizontally mirrored on one side.

The same thing goes for normal maps - you can't mirror a normal map without getting bogus results. Think of your normal map as a real 3-d (slightly extruded) piece of metal. If you are seeing the text go the wrong way, the metal must be facing with the exterior side pointing to the interior of the airplane.

Having your normal map flipped does more than just "inset" what should be "extruded" - it completely hoses the lighting calculations. In some cases this will be obvious (no shiny where there should be shiny) but in others you will see shine when the sun is in a slightly wrong position.

The moral of the story is: you can't recycle your textures by flipping if you want to use normal maps.

Wednesday, September 16, 2009

I Hate the Term Photo-Realistic

Now before you come at me with pitch-forks, I don't hate photo-realistic scenery, and X-Plane's performance with orthophoto-based scenery is very, very good. But...the term "Photo-Realistic"...it makes me crazy. Here's why:

First, photo-realistic scenery links the use of photographs in scenery to realism in its very name, and I don't buy it.

Yes, some photo-based scenery packages are realistic looking, by today's standard of flight simulation. Some are not. Just look at any old photo-realistic package to see what I mean...realistic is a relative term, defined by how much fidelity we expect, and that expectation has steadily gone up. Even with a modern package, a photo-based scenery pack might not be realistic if the photos are not used well.

(For example, is a package that uses orthophotos on the mesh but provides no 3-d in a city still considered realistic now? What kind of review would such a package get?)

Nor do photos have a monopoly on realism. They can look nice when well used, but I would put Sergio's custom panel work up against any photo-based panel. (Sergio does not manipulate photos for his panel, he constructs them from scratching. He has thousands of photos for reference, but the pixels you see are not originally from any photo.)

Second, the term photo-realistic (in the scenery world) is most commonly applied to scenery that applies orthophotos to the terrain mesh in a non-repeating way. But orthophoto base meshes don't have a monopoly on the use of photographs, which can be used to form land-class textures or to texture objects.

Okay, so "hate" is a strong word. But I feel some frustration whenever I see scenery discussed in terms of "photorealism".

Tuesday, September 15, 2009

Pie in the Sky: Future Rendering Tricks

I've had a few inquiries about environment maps, normal/specular maps in more places than just OBJs, etc. The short answer is: a lot of these future rendering engine enhancements are good ideas that we like, but we have other features that are already partially implemented that we need to productize first.

In particular, one of those half-finished features not only could use to be shipped, but also affects the way just about every other rendering effect works. So better to get these features finished first, and build new effects within the context of these "new rules".

Here are some of the ideas that I've heard kicked around:
  • Environment maps on the plane - I like it, it's not that hard to do, and the framerate hit could be ramped up and down with detail. See above about finishing other features first.
  • Next-gen texturing on runways (wet runways, environment maps, bump mapping) - I like all of it. The runways really need to be addressed comprehensively, not piece-wise, in order to find a rendering configuration that meets our scalability and efficiency needs. (In other words, we can't just burn tons of VRAM on the runways, and we need a way to render them that works on low and high end computers with one set of art assets.)
  • Normal mapping on the ground. I like it, but I wonder if this isn't part of a bigger idea: procedural texturing on the ground. E.g. if we want to add detail on the ground, can we add it with multiple layers at different resolutions with a shader adding yet more detail.
Generally speaking, it's not that hard to push a feature out from one part of the rendering engine to the other. For example, normal maps on facades (if anyone cares :-) would make sense and be a fairly trivial change. I hesitate on polygons only because the ground might require something a little bit more sophisticated.

Going Crazy With Choices

Here's a straw man of why polygons might be different: draped polygons can't have specular shininess right now, and I don't think anyone is complaining. So it's a bit of a waste to use the alpha channel of a normal map for shininess. Perhaps it could be used for something else like an environment mapping parameter.

Hrm...but we would like to have environment mapping on airplane objects too someday. Well, we could go two ways with that. We could just use the alpha channel for shininess and environment mapping...not totally unreasonable but it wouldn't let us have a glossy non-reflective material, e.g. aluminum vs. shiny white paint.

Math nerds will realize that the blue channel in normal maps can be "reconstituted" from the red and green channels by the GPU (at the cost of a tiny bit of GPU power). That would give us two channels to have fun with - blue and alpha. Perhaps one could be shininess and one could be an environment mapping material.

Well, shininess is still no good on the ground. But...perhaps that would be a good place to store dynamic snow accumulation? Hrm...

My point of this stream-of-consciousness rant is that the design of any one rendering engine feature is heavily influenced by its neighbors. We'll get all of these effects someday. If there are features that are really easy, we can get them into the sim quickly, but the only obvious one I see now is using bump maps on other OBJ-like entities (which at this point would mean facades).

Monday, September 14, 2009

Removing Third Parties is a Diagnostic Test, Not a Cure!

In response to so many bug reports I say:
Please temporarily remove your third party scenery, airplanes and plugins and retry. Do you still see this bug?
And in response I get:
No - the bug was with XXX. Thanks!
But...but...but...we're not done yet! The point of removing the third party add-on is divide-and-conquer: to see if the bug is in the core of X-Plane or is caused by an X-Plane-add-on interaction.

If the add-on used to work but no longer does, I want to know where I can get the add-on, who created it, and when it used to work, so I can investigate further! If an add-on stops working because X-Plane was updated and the add-on is not doing anything naughty, that is still an X-Plane bug.

So please: don't give up on the bug just because removing the add-on makes it go away...there is still more to debug!

Sunday, September 13, 2009

Marking the Ground? Use a .pol or .lin.

The rule is very simple: if you want to put a marking on the ground, your results will be much better if you use a .pol (draped polygon) or .lin (draped line) than if you use an OBJ. In fact, an OBJ is probably the worst way to put markings on the ground.

Requirements for Good Looking Markings

In order for markings to look good you need to have three things happen:
  1. You need to make sure your markings are at the exact same level as the ground itself.
  2. You need to use polygon offset to tell X-Plane to tell the video card that these are "coplanar" (at the same height) triangles that must be managed in a special way.
  3. You must guarantee that the draw order is: the stuff under your markings, your markings, then the 3-d buildings and other 3-d stuff.
With these requirements we can compare .pol/.lin files to OBJs:
  1. .pol and .lin files always "drape" to the ground, so they always meet rule 1 perfectly. With an OBJ you can set the height of your OBJ to 0 but on sloped terrain this won't be correct.
  2. .lin and .pol files are always "polygon offset" automatically, so they always meet rule 2 perfectly. With an OBJ you need to use ATTR_poly_os.
  3. ATTR_layer_group (or LAYER_GROUP) tell the OBJ or pol/lin in what order to draw, so you can set this correctly in all cases. The default values if you don't specify a layer group are more appropriate to the task of markings on the ground when you use a pol/lin - that is, by default they meet rule 3 perfectly. By comparison, an OBJ may not be in the right draw order.
So we can see from these 3 rules that you will always get the rules right just by using a pol or lin, but you have to be very careful and may still not get the rules right when using an OBJ.

Performance? Think .pol

When you have a large number of small markings, the performance of a .pol is going to be significantly superior to the performance of an OBJ. For example, imagine an airport where you want to draw on-pavement signs for all taxiways and runways. With an OBJ, to get the height right, you'd need to use a large number of small objects. (With one large object, it will be nearly impossible to make the OBJ markings be on the ground when far from the object center.)

When you use a large number of .pol markings that share one common texture and all have the same .pol file, X-Plane merges them behind the scene into one huge object-like pile of triangles. The cost of drawing all of those polygons will be similar to the cost of drawing just one object! That's a huge speed win.

Floating Objects Are Wrong

What I see most commonly in scenery packs that are sent to me and have thrashing problems are OBJs that have a height other than 0. This is simply the wrong way to create overlaid geometry in X-Plane, and it will produce artifacts in a wide variety of situations. At best, the "floating" objects will cause airplanes driving over the marking to look like they "sink in". At worst, the offset you pick will be too small for the video card's resolution and you'll get thrash anyway.

There is no one right vertical offset for all scenarios, and even if there was, it would still look ugly! See the above rules for what an OBJ really has to do.

How Do I Get a .pol into my scenery.

Well that is the $10,000 question. I must admit that I don't know what Overlay Editor's capabilities are in this regard. WED 1.1 will be able to add draped polygons, including texture coordinate editing of the polygons. I've been working on the texture coordinate editor in WED this weekend and am hoping to get some kind of WED 1.1 preview built this week.

With WED 1.1 the process is fairly simple:
  • Create your texture.
  • Create a single .pol text file that names the texture, so you can use it.
  • In WED 1.1 you can select the .pol from the list of resources for your scenery pack, and then use the "polygon create tool" to simply draw the draped polygons into place.
  • Once the draped polygons are created, you can select the polygon and open the "texture coordinate editor" tab to edit the way the texture is applied to the polygon.
My hope is that this process will be easier than creating markings using a 3-d editor - you can still edit the texture coordinates, but you can do so directly in WED.

Wednesday, September 09, 2009

Higher-Order Beziers? Probably Not

I was revisiting Bézier curves today. First I should say that the current scheme for handling Bézier curves in apt.dat and DSF is horribly convoluted - I am looking to replace the syntax with something less weird for the next set of features that use them. But I was also looking at the problem of generating curves from high-density line data.

Did you know that a Bézier curve can have any number of control points? This picture illustrates:

Each curve has one more control point than the last. Zero control points is of course a line segment, and two control points is a "cubic" Bézier, which is what we have now. (Note that having one control point in WED does not give you the one-point quadric Bézier you see here. WED gives you a cubic Bézier with one control point on top of an end point.)

But it also show: having tons of control points are not particularly useful for modeling a shape in detail - a string of connected cubic Bézier is probably just as good. And the high-order Bézier doesn't really trace out the control points, so it's not a good way to convert line segments into a curve.

Monday, September 07, 2009

Parameterized Lights, Generic Lights and Plugin Controlled Lights

There are a few changes to 940 regarding airplane lights...I will try to get some permanent documentation on the Wiki, but here's the basic ideas:

There is a new "type" of light in the OBJ8 format, called a parameterized light. A parameterized light is somewhere between a named light (totally as-is, can't be modified, simple to use) and a custom light (totally complex, can do anything, requires a lot of work). In a parameterized light, you control just one or two aspects of the light.

Parameterized lights are aimed at airplanes, not scenery, because typically parameterized lights are customizable and slow.* The goal is to give airplane authors some flexibility without having to invent a huge number of named lights.

Consider, for example, landing lights. A landing light could vary based on what switch controls it (we have 16 now), how big it is (many authors have pointed out that one size does not fit all) or how wide it's view angle should be. (Lights that are inset in a structure might not be easily viewable from the side.) With a parameterized light, we can provide one light definition with 3 parameters instead of a huge matrix of lights.

Generic lights are a new collection of 64 lights that can be used for any purpose, sort of like misc wings, misc bodies, and sliders. The main difference between a landing light and a generic light is that the landing light halo won't show up on the runway when a misc light is turned on. They are meant to be used for logo lights, inlet lights, etc. A series of new named lights will "listen" to the generic switches.

(Tip: combine ATTR_light_level with generic lights to have a light turn on and your lit texture appear at the same time.)

Finally, there is now a plugin override for the beacons and strobes (and in the systems model there can be up to 4 separate sets of beacons and strobes flashing at different times). With parameterized lights you can make two sets of strobes and use a plugin to control when they flash.

The combination of these three things let an author create an airliner that models all of the various lights and their behaviors.

* Slow needs some qualifications here. There are two code paths for lights, the fast and slow path. The slow path IS pretty fast, just not as fast as the fast path. The fast path is expected to be able to draw at least 10,000 lights in a single frame on low-end hardware, while the slow path is expected to be able to draw at least 500 lights per frame on low-end hardware.

500 lights is a lot for one airplane, especially if you have to place them by hand. And most modern computers will easily do thousands of slow lights.

Basically slow lights are not appropriate for scenery objects in the library that might be placed a huge number of times: OBJs attached to roads (e.g. street lights), OBJs used for buildings, taxiway lights. The are plenty fast for airplanes. In the X-Plane world, slow doesn't really means slow, just slower than something else.

Yank That Stick Back

A quote from the .org:
I actually did try the demo before I bought it, and I found it sorta arcadish as I took off with a King Air 200 and performed barrel rolls on t/o lol! That kind of tuned me out!
I don't mean to pick on that particular poster - we hear that a lot, particularly from people who use MSFS. Andy Goldstein points to Bob Hoover as a counterpoint.

More generally, the criticism is that since X-Plane will allow you to do things in commercial airliners that you've never seen in real life, X-Plane must not be realistic. And phrased like that, you can see the possible flaws in the reasoning:
  1. How do you know your control inputs match real life?
  2. How do you know the real plane is physically incapable of doing such a thing?
The first is the age-old problem of consumer joysticks - to really maneuver an airliner hard you have to put a lot of pressure on the controls - they put up a lot more resistance than a $20 SideWinder.

The second is an issue of falsifiability again - the absence of evidence is not evidence of absence.

Now in truth there are things that X-Plane models and X-Plane does not model. If you load up the airframe too much with fail-on-over-G on and the G limits are set correctly, X-Plane will start removing parts as you try to fly a 747 like an F-16. But we don't necessarily simulate some of the internal things that might go wrong. For example, accelerations on the fuel system might potentially cause flow to the engines to fail, stalling an engine. I don't know how much of this Austin simulates, but we certainly don't simulate the geometry of the fuel lines and the fuel as a fluid flow. So there may be cases where a maneuver is impossible for logistic but not aerodynamic reasons (e.g. you could do it but you'd lose your engines).

With that in mind, here are a few youtube links:
  • Bob Hoover flies a Commander like an aerobatic glider - thanks Andy!
  • Roll an airliner? It's safe, really!
  • The 757 has a lot of power - thanks Andrew.
  • I have no idea what happened here...I thought the Airbus computer prevents you from doing this kind of thing.
On that last video: the first time I flew with Austin in his Cirrus (this is before he traded it for a Columbia) he pulled the same maneuver: take-off, very slow rate of climb to pick up speed, and then: yank. Our climb rate was well over 2000 fpm in a single-engine prop for a while.

(Logistic note: please don't interpret this post as an invitation to contact me regarding any aspect of the flight model - when it comes to the physics engine I am just another user, with no special insight. Physics is Austin's domain, definitely not mine!)

Proximate And Ultimate Crashes

One last comment about driver crashes: I try to be fair about the distinction between causation and correlation when it comes to driver bugs; the piece of software that "fixes" a bug may not be the piece of software that contains a bug, and work-arounds happen in both directions between drivers and apps.

Similarly, seeing "ATI" or "NVOGL" in your crash report does not imply a driver bug. It does imply that the crash happened in the driver, but the driver crash may be a proximate cause, while the ultimate cause may be in X-Plane. (Sometimes the stack crawl is also just plain wrong, depending on the severity of the crash.)

OpenGL drivers need to be very, very fast; the whole point of buying that shiny new GPU was to get faster framerates. So driver writers have to make trade-offs about how much error checking they do when an application passes in junk data. If I were writing a driver and debating an error check that would slow performance and prevent a crash of an application that passed illegal data (but not the machine), I might very well skip the error check and say "if the application doesn't want to crash, it should pass better data."

Of course, it's not up to me, and it's not always up to the driver writers; the OpenGL specification actually dictates what outcome are acceptable in various error conditions. Quite often, "program termination" (read: the application crashes but the machine does not) is an allowed response to certain kinds of bad data from the application to the driver, particularly where checking the data would slow performance for all applications and the application should know better.

Sunday, September 06, 2009

Pay No Attention To That Last Post

My last blog post may have expressed some negative views toward software upgrades:
I know I'll take some slack for this, but: why did you all go and grab an OS update so fast? What have you gained? The down-side is lost time and application compatibility problems...was it worth it?
That blog post was of course not my real opinion, but rather a recreation of the evil Ben. Always upgrade your software to the newest version whenever it comes out!

Saturday, September 05, 2009

Caught With My Pants Down

My friends say I have become a technological curmudgeon...whenever a new gadget or device or operating system comes out, I just grumble about how that it just means more bugs and mutter about how a byte only had 7 bits when I was young.

How did this happen? Pretty simple. New driver comes out, new bugs and interactions with X-Plane, rinse, wash, repeat. This is not a criticism of the driver writers. Those guys have an impossible job: write code that is super high performance in an environment that is highly unforgiving (an X-Plane crashes X-Plane; a video driver bug crashes your entire machine), then do it again when the next hardware comes out 9 months later. As the apps guy integrating on top of this ever-running treadmill of new hardware and drivers (think I Love Lucy) I sometimes dream of a week when none of the software that X-Plane runs on top of changes.

This is not one of those weeks. Snow Leopard came out, and it looks like Apple has been quite successful in keeping the price low; I'm seeing plenty of bug reports against 10.6 and it's only been a few days. Apple ships video drivers with the OS, so Snow Leopard isn't just a new OS, it's new drivers too, as well as possible changes to the OpenGL "middle" layer that Apple provides.

Now about my pants...over the last few years I have been accumulating graphics card + operating system combinations, so I can see a higher percentage of hardware-related bugs "in the lab".* But my wife and I are in the process of relocating to Boston, and this has temporarily separated me from a significant subset of my hardware. The result: driver bugs are coming in and I can't easily reproduce them.

To paraphrase Airplane, I picked the wrong week to quit sniffing OS betas.

(I know I'll take some slack for this, but: why did you all go and grab an OS update so fast? What have you gained? The down-side is lost time and application compatibility problems...was it worth it?)

*Just because it's hardware related doesn't mean it's a driver bug - X-Plane can be the broken app too! Sometimes drivers work around X-Plane bugs and sometimes X-Plane works around driver bugs. I mention this because plenty of apps developers complain on forums about "driver bugs" - but how many of them repost when they discover a null ptr in their code? A driver and an app are two pieces of software that have to be integrated, and bugs crop up on both sides.

Friday, September 04, 2009

Scalability and apt.dat

Beta users, see the bottom of this post for how scalability turns into possible bugs.

In computer science, a program or architecture is scalable if it doesn't totally vomit up a lung as its constituent parts become bigger. For a cleaner definition, see Wikipedia, source of all internet knowledge.

An ant is not scalable - if you made an ant 100 times larger in every dimension, its tiny legs would break under its new weight. (An ant is not scalable because its weight grows faster than its structural strength. Thus elephants are not built like ants.) Geeks: scalability is to computer science as marginal cost is to economics.

Before X-Plane 940, the apt.dat file was distinctly not scalable. The entire file was loaded into memory; as users created more and more taxiway lines and signs and details, we simply used more and more memory. This approach isn't very scalable because authors have the potential to grow the apt.dat file faster than our system requirements can increase.

X-Plane 940 fixes this by not loading the entire apt.dat file into memory. Instead, only essential airport information is loaded into memory, along with a note as to where in the file the airport lives. Whenever an airport actually has to be built into a 3-d mesh while you fly, we go back to the apt. file and load the rest of the data for the one airport we are building, use it, and throw it out. Since 3-d airport meshes are built on a second core, the cost of loading one airport off disk is pretty harmless.

The problem with this fix is that it introduced a new scalability problem. Consider:
  • Meshes in 940 are built on as many CPU cores as you have - some users have 8!
  • Each CPU core could be working on a different airport, depending on how many are nearby.
  • Each airport has to load up the apt.dat file to get the extra airport data.
This means that at times on an 8 core machine we could easily have allocated 50 * 8 = 400 MB of memory just to temporarily hold 8 copies of the apt.dat file.*

This is of course completely silly - there's no reason to load the whole apt.dat file to get one airport, and the fix that is going into beta 8 lets the airport loader surgically grab just one airport. Thus we will be scalable again, because adding more cores won't cause memory usage to go up.

Beta Users: Please keep an eye out for X-Plane running out of memory - if it starts to do so in the next beta it means that some part of this code change munged memory management. We're running stress tests on the sim now, but touching the low level memory nd file handling code late in beta isn't something I like to do.

* While loading 8 copies of the apt.dat file is wasteful of memory, it is not slow; X-Plane uses memory mapped file I/O, so reading a small part of a large file is very fast - just not very virtual-memory efficient.

Thursday, September 03, 2009

Marketing And Fact

A while ago, Austin was on FSBreak, and I wrote this post as commentary on his interview. The main point I meant to make was this: from what I have heard from other engineers and seen in my own experience, most software companies prefer to develop new features on top of implementations that are known to have architectural problems. At LR, we fix the implementation architectural problem first, and that has been a net win for us.

Now that's basically a statement of my opinion on software engineering - in hindsight it probably belongs on my programming blog and not here. Unless you develop X-Plane plugins, you're not a programmer; I will try to constrain future scenery blog posts to things that non-programming X-Plane users will notice. If you are a plugin developer, you might want to look at the "Hacks of Life" posts tagged with OpenGL.

Anyway, back to the story...the responses to that blog post were all well thought out comments on X-Plane's quality control process. At the time my immediate reaction was: that's totally off topic - I'm commenting on architecture and they're talking about QA. I do think the authors made fair points.

But in hindsight, I think that there's a deeper issue: one of verifiability. Simply put, my statement that we (LR) rewrite stale implementations is nearly impossible to verify without source code access, something that you can't get for X-Plane. So from the perspective of anyone outside the company, my original statement is not falsifiable (it cannot be proven false) and thus rather useless as a statement of fact. Even though I claim that we make rapid progress on features by keeping implementations clean, you as a user don't care how we develop our features - clean architecture, more developers, or the use of time travel and voodoo dolls, it's a bit moot.

Thus the comments were off topic, but also they were moving away from an unverifiable topic and toward one that users can measure, namely the quality of X-Plane's betas.

There's a fair amount of marketing that gets put out in the tech and games industry. It's a slippery slope from giving a new, real technology a whiz-bang name (e.g. HyperZ is a real technology, and it is good for your frame-rates) to using techno-babble to make the bad seem good (e.g. HyperMemory just means that your video card lacks VRAM and is going to be slow). When new products come out, the feature list is parroted, but it's not always clear whether the new features turn into real benefits.

So what I'm going to try to do with the scenery blog is: try to keep the blog limited to verifiable, measurable aspects of X-Plane. If we ship X-Plane with "psychoactive rendering*", I'll try to explain what the heck that is and why you'd want it, and how you might notice that it's working.

* X-Plane does not have psycho-active rendering, except possibly when I make a mistake in the shaders and everything turns purple.

Test Your Add-Ons Now!

X-Plane 940 beta 7 is out. Now is the time to test your airplanes, scenery, and plugins. We've reached a point where we think we have the new systems code working. Please try the new beta, test your add-ons, and if something works in 930 but is broken in 940, file a bug immediately!

Friday, August 28, 2009

Two Warnings About Normal Maps

Two warnings about normal maps:
  1. Make sure that the RGB color underneath transparent sections does not turn black or white! Some image editing programs (in particular Photoshop) will lose the color beneath a transparent area.

    With a normal map, this is very bad - black and white are not legitimate normal map colors, and the result will be bogus normal vectors under the non-shiny part. Normal maps affect more than just specular shiny hilights - the normal map affects all lighting, so having black or white under your transparent (non-shiny) parts is bad news.

    To check whether this has happened, I recommend Graphics Converter, which will show you your alpha and RGB channels separately, exactly as they are in the file.

  2. Make sure your RGB value are normalized. The "length" of the normal (as encoded in RGB) must come out to a distance of 1. This is virtually impossible to do using PhotoShop or an image-oriented program...I suggest you use a real plugin to PhotoShop or Blender to create normal maps that are correctly "normalized".

It is also very possible that X-Plane's gamma correction is distorting normal maps, but that's one for me to fix.

Thursday, August 27, 2009

Plugin Broken? It's Temporary

I just found out from a user that Chase View Delux doesn't work with X-Plane 940 beta 4. Here's what's going on:
  • Plugins often end up running X-Plane code - when you make a call to the X-Plane SDK, the SDK sometimes passes the call right to X-Plane.
  • X-Plane 940 beta 4 has heavy error checking on all of the X-Plane code.
Put these together and...X-Plane 940 beta 4 is putting heavy checking on plugins that wasn't there before.

Typically the kinds of problems caught by the error-checking code inside a plugin are not critical, but aren't bad things for authors to fix. Since we will turn off heavy error checking as soon as we get the systems code fixed, plugins will start working again..there's no need for plugin authors to recut anything.

A Slow Motion Car Crash

I'm driving at 55 mph on the highway. I drive over a nail, lose a tire, skid off the road, crash through the guard rail, plunge off a cliff, and die. That's not much fun. But when you get to the accident site, you'll probably be able to piece together what happened. The skid marks, the nail, the hole in the guard rail, the car wreckage below - you can connect the dots.

Now...let's say I'm driving 500 mph. Same nail, same out of control crash. But this time it's going to be a lot harder to tell what happened. Lord knows where the nail ends up, the distance from the nail strike to exiting the highway is going to be a lot bigger, and the car is going to be in smaller pieces scattered over a wider distance. It's going to take a lot longer to piece together what went wrong.

That, in a nutshell, describes the motivation for an X-Plane beta with all of the debug and safety checks on. X-Plane's normal operation is like the car doing 500 mph - when it crashes and dies, there's very little left that can be used to figure out what went wrong. When there is a bug in the code that destabilizes the sim, finding it via crashes in release builds takes a lot of developer time and slows the whole beta down.

With the safety checks on, X-Plane still crashes when something goes wrong - but the bodies and wreckage are all a lot closer to the scene of the crime, and the evidence left around is in much better shape.

One of the side effects of the safety builds is that they catch "harmless" coding mistakes - (harmless in quotes - the bug might seem harmless but who knows if that will always be true). XSquawkBox now quits the sim with an ugly alert box because it reads off the end of a piece of the airplane data structure via the plugin system. This hasn't hurt things in the past, but it's not really correct. Beta 5 should fix the underlying problem, letting you run XSquawkBox again. (The fix will be in X-Plane, not XSquawkBox.)

Tuesday, August 25, 2009

A Tip For Updating Electrical Systems

X-Plane 940 has a new electrical systems model and it has a few important differences from 930:
  1. You must specify exactly how many buses your plane has. 930 provided two buses but then did a bunch of cross-tying behind the scenes in case you didn't have enough power sources.
  2. X-Plane 940 requires that each battery and generator be on exactly one bus.
  3. X-Plane 940 will not allow systems to be powered by non-existent buses.
Now this can have some strange side effects. Consider the default king-air with two generators and one battery:
  • In 930 it has two buses, battery feeds both buses, and each generator feeds one. You could have systems split by bus and they would work unless you lost one generator and the battery.
  • 940 defaults this plane to one bus, because on battery power only one bus will be fed.
  • This means that in 940 all of your systems will be reset to bus 1.
But wait...if you import into Plane-Maker, the import will trash your system bus selections before you can increase the number of buses to 2. What can you do?

Here's a work-around: before you update your plane, make sure you have two battery and two generator switches on your panel. Then open in 940. The import will set 2 buses and your systems will be preserved.

Of course, by the next few betas this may all be moot because we may get something less crazy in there.

Driver Thrash

I'm seeing a number of bug reports where weird artifacts are showing up in 940...missing pieces of runway, flickering triangles...all sorts of good stuff!

I believe that this is due to some kind of bug relating to threading, X-Plane and the video drivers. I won't say whose fault it is because I really don't know. I do know that the bug appears to not happen on OS X. (But this could simply be because the threads time out differently on OS X.)

The changes to the rendering engine for 940 from 930 are substantial and aggressive - it'll take us a little bit to fix these things.

When you wonder how come programs don't use all 8 of your cores yet, well...this is why...multi-core programming is complex, tricky, tedious to debug, and often involves substantial changes from the original code.

Monday, August 24, 2009

12 GB For the RenderFarm

The global scenery gets cut on "the renderfarm", which is our name for the cluster of computers, each loaded with all of the input data for the global scenery. These computers chug for a few days to churn out the whole planet.

With the next global render I am trying an experiment: using one 8-core Mac Pro as the RenderFarm. We've never had more than 8 processors in the farm at a time; in the past to get 5 processors might have taken 3 machines. The appeal of a single machine is ease of setup; no data to sync between machines, no sorting out which machine did which tile and merging it all back.

Today I upgraded the Mac Pro's memory (again) to 12 GB. I thought the logic of why I did this might be of interest to X-Plane users who are trying to figure out "should I have more memory"?

Basically there are three memory limits we care about:
  1. The virtual memory limit per process - generally 3 GB per process for a 32-bit application. If an application wants more memory tan this, regardless of what you have, it is dead.
  2. The virtual memory limit for the whole machine. Since the machine virtual memory limit is a function of hard disk space, normal users will never care about this - we can have a huge amount of virtual memory.
  3. The physical memory actually used by the sum of all programs actually running. Once all programs need more physical memory than you have, they start using hard drive, and they get really, really, really slow.
In the case of the render farm, our processing program runs on one DSF tile at a time. But with 18,000+ tiles we can take advantage of more cores by processing 8 tiles (using 8 copies of the program running at once) on 8 cores.

This is where memory comes in. Before the upgrade my machine had 4 GB of memory, allowing each of the 8 tiles to use a little bit less than 512 MB of RAM before we ran out of physical memory and started paging. (The OS takes a little bit for itself.)

Normally this is all good -- a typical run might only use 300 MB of RAM. But every now and then we hit New York City or Boston and the RAM use spikes out to 1500 MB or more.

This is okay if just one process hits New York, but what if a bunch of processes hit a big city at once? We blow past the physical RAM of the machine and every tile becomes slow at once. And because they are all slow at once, they take a very long time (hours) to clear this state.

(This state of thrash due to many processes is like 8 people trying to go through a doorway at once. If they would just take turns, they'd be through in a second. But by forcing 8 at once they get stuck. The operating system won't pause a few of the high-memory programs to let the others complete.)

Since the RenderFarm has to run overnight unattended, I upgraded RAM to 12 GB, for 1500 MB per rpocess before thrash. My hope is that for this investment, we'll be able to run the processing through without a human to unjam it.

What About X-Plane

So would a RAM upgrade for X-Plane help? Well we can apply what we know from above to figure it out. Generally, you need enough RAM that X-Plane + all other programs won't run out of physical memory. Since X-Plane is 32-bit (and can only use 3 GB of RAM) you are likely to be fine with 4 GB when running X-Plane + the OS. Any more and X-Plane can't take advantage.

The exception might be if you need to run X-Plane and another big program like PhotoShop at the same time. At that point you might want enough RAM for both to run at the same time.

Sunday, August 23, 2009

Lego Brick Code

When Sergio first proposed generic instruments, his model was "lego bricks". The idea was to provide a number of very basic parts for panel makers and let panel makers mix and match. The result would be huge flexibility for airplane authors without code bloat.

The problem with non-generic instruments is that there is such a huge variety of behavior among airplanes...if Plane-Maker were to have options for every possible plane, the "special equipment" screen would require 3000 tabs and be completely unusable. Hence the need for a smaller unit of modeling: the lego brick.

The prop disc is the first feature I have done that is meant to be used only by a plugin, e.g. "lego brick" code. The X-Plane systems code sometimes suffers from the same "code bloat" problem as the instruments: a ton of very specific, very tweaky behaviors that interact in strange ways and become very difficult to manage. It's not that the systems code is bad code - it's that the scope of the problem is simply too large. That is, you can't expect X-Plane to cleanly simulate the systems of every airplane ever in a ton of detail through an a la carte menu of check-boxes.

The idea of the prop disc is: someone (LR or otherwise) can write a plugin that encodes a certain style of prop disc. That plugin can then be picked up and moved around like a generic instrument between planes (perhaps with a corresponding text file to control it).

If someone else comes up with a different/better prop-disc algorithm, compatibility isn't an issue...that person writes a new prop disc plugin and the airplane author selects the one desired. Think of it as sort of a portable flight model that stays with your plane.

So we win in three ways:
  • Anyone can write the prop disc algorithm, not just LR.
  • The code lives with the plane, to avoid compatibility problems.
  • More than one plugin can exist, giving authors an a la carte menu.
That's the theory at least.

Saturday, August 22, 2009

Tweaking the Prop Disc in 940

X-Plane 940 allows plugins to customize the prop disc. Details here on the wiki.

I put these datarefs in so that modelers wouldn't have to try to model prop discs with OBJs. The problems with prop discs are many:
  • They need to be billboarded, and X-Plane does not provide datarefs for manual billboarding inside an airplane (particularly not to the engine's coordinate system, which can be transformed by all sorts of fun stuff).
  • They often need variable translucency, which OBJ does not have.
  • They cause all sorts of depth buffer errors, which OBJs cannot manage.
In short, prop discs are weirdly special-cased enough that I thought it would be better to provide a general set of parameter datarefs for prop discs and let X-Plane do the drawing.

These options are not available in Plane-Maker. Why not? That'll be my next post.

940 Beta 2 - Recovert Airplanes

I have ranted in the past about the importance of not treating beta builds of X-Plane as having finalized file formats. Generally a beta should be used to explore, experiment, and test old content, but not to create new finished work or ship product. File formats sometimes change during beta to work around bugs we find.

Another reason not to depend on the file formats of new betas is that sometimes we screw up. In the case of 940 beta 1, the code that converts 930 airplanes forward to the new 940 electrical system is pretty buggy (hence the reports of electrical systems doing wonky things, panel instruments disappearing, and general weirdness).

This one is our bug to fix and will be fixed in beta 2 (at least we think). But to "get the fix" authors will need to open their 930 saved airplane in 940. If you already re-saved the airplane in 940 then the 930->940 conversion code won't be re-run.

I'll try to post some info on the new electrical system on the Wiki, but for now: if in beta 2 you have a bug with a plane that used to work in 930, send us the 930 version of the plane so that we can convert it and watch the conversion screw up. If the plane is already in 940 format we don't know what our conversion code broke and what you edited in Plane-Maker.

Friday, August 21, 2009

Normal Maps in X-Plane 940

X-Plane 940 now supports normal maps on OBJ models (both scenery and airplane). I'll get more formal docs up once the rest of my office is moved and unpacked but here's the details for now:

The normal maps are in "blender" format see here. The alpha channel is optional; if it is present, it serves to modulate the level* of specularity. Opaque means full specularity, transparent means none. You can use this feature to make some parts of an object shiny and some dull on a per-pixel level.

Shininess level is modulated by both ATTR_shiny_rat and the alpha channel, so you need ATTR_shiny_rat 1.0 and an opaque alpha channel (or no alpha channel) to see full specularity.

Normal maps are only available for objects and only appear if pixel shaders are on and per-pixel lighting is enabled.

Normal maps should be PNG format, not DDS - they will not be texture compressed because S3TC compression tends to kill them. (There are some modern formats for normal map compression supported by the newer cards but we don't use them yet.)

* Specular level: most serious 3-d programs let you control both the specular exponent, which controls how "tight" the specular hilights are, and the specular level, which controls how bright they are. X-Plane only lets you control specular level; if specular hilights exist, they are always as the maximum exponent for the sharpest specular hilights.

Thursday, August 20, 2009

More Threads

X-Plane 940 beta 1 is out...it will take a little bit to get the release notes and docs on the website completely up-to-date. We're still dealing with loose ends from our migration to the new web site, and most of my office is packed up for a move to the Boston area. I'll try to get docs up as fast as I can given the chaos flying about.

Given the interest multi-core stirred up in previous notes, I will mention one change to 940: with this build we've added yet more multi-core to X-Plane.

In 931, X-Plane will use as many cores as you have to load textures, but only one to build "3-d scenery" (a loose category for the work we do when we make airport taxiways and lines, build forests, and extrude roads).

In 940, this "3-d scenery" is also done on as many cores as you have. This should speed up load times a bit, particularly under very heavy tree settings, and hopefully keep the forest engine running faster for users with more cores.

It also sets us up for long-term growth; X-Plane's visual quality is sometimes limited by the time to build 3-d meshes...being able to do this work on many cores means we can use higher quality algorithms.

Consider for example the roads: my original "road extruder" (the code that converts a vector road into a 3-d model, called an extruder because it builds a 3-d road from a cross section like one of those play-dough toys) made beautiful intersections with stop-lines and cross walks and lots of other great stuff.

It was also really slow. And at the time the sim wouldn't fly at all while roads were extruded, so speed was of primary concern. So I replaced it with the much dumber extruder you see today, where intersections are basically ignored.

Now that we have 3-d scenery build on multiple cores, we can begin to provide rendering options that take more CPU time but produce higher quality results. The trees and airport layouts already do this (in that they take more time and produce slower, more detailed, higher triangle count sceneery at higher rendering setting for the same input DSF ad apt.dat file). With more cores, we can continue this strategy with roads and other parts of the sim without worrying about overloading the one core that was doing this work.

Of course just because we can use 8 cores doesn't mean we do...you won't see 8 cores maxed out very often, particularly if you have simple scenery and a very fast machine.

Monday, August 17, 2009

I am the Spam King!

If I have not emailed you back, it's probably because I've been very busy trying to interleave X-Plane coding and packing up the house. But it is also possible that my email response bounced because my web server has ended up on a bunch of anti-spam "black-lists".

Black-listing seems like a good idea at first: we'll gather up a list of all of the IP addresses from which spam comes from, then publish them. Then your local mail server can use that list to filter spam - you never see it!

In practice it doesn't work so well...example: http://www.mipspace.com/. The IP address for my server (XSquawkBox) is now on this list. Why?
MIPSpace is a list of IP Addresses associated with known commercial marketing companies.
Since my server is used for my own personal email and to run the SDK website, I'm not sure why I am on the list. I have sent them an email to clear things, but in general I hit an anti-spam/black-list bounce somewhat frequently now, and frankly I don't have time to separately try to clear my name from every guilty-until-proven-innocent blacklist that pops up and screws up my email.

If I seem disproportionately grumpy about this, it could be due to one of two reasons:
  1. Not replying to emails is generally bad customer service. (Okay, my in-box is backed up four months...that's bad.) I don't like the idea that a customer might perceive us (LR) as being unresponsive because some third party with no skin in the game decides to black-list us.

    The blacklist has no incentive to be accurate - it's not their lost customers if email doesn't go through.

  2. I'm not at all convinced that this is going to cut down unsolicited commercial mail and/or spam.

    In the spam case, spammers can send from botnets - they have access to a huge number of ever-changing IPs. Unless we are prepared to blacklist the entire internet, the blacklists are going to pick up more and more false positives while spammers find ways to harvest fresh, untainted IP addresses. The whole IP-reputation strategy assumes that IPs are hard to change. In practice, IPs are very, very easy to change.

    Commercial mail is a lost cause too - even if I am being solicited for commercial mail I don't want, no program or automatic process is ever going to tell the difference between the confirmation of my invoice and a list of discounts from the same company. When it comes to commercial mail, the reputation damage has to be done to the company, not the IP.

    (The company does have reputation to risk - if we are known as a company that doesn't honnor a "do not subscribe" policy, then customers can choose to not buy our products.)

It could also be because I ran out of Viagra and don't have a diploma from a prestigious non-acredited university.

Friday, July 31, 2009

Switch and Bail

I will be out of the office next week - it's August, time to head for the mountains...no cell coverage, no internet, no computers, no electricity...no mountains actually. (This is upstate New York...mountainous to someone from Boston but not mountains compared to what Sergio normally deals with.)

As you know, LR often shows stellar judgement in managing release risk. So in true X-Plane tradition, I swapped in the new revamped website last night, just in time to head for the hills and avoid the fall-out.

Here's the short version:
  • I am wicked stupid.
  • When I did the swap last night, I screwed up the files that manage the auto-update functionality, hosing the updater.
  • I fixed these files this afternoon, once the message got back to me.
So if you saw weird stuff happen with the updater or the sim last night, please try again - it should be fixed. If it is still broken, please send an email to Austin and myself - one of us will hopefully be around.

I have also reskinned scenery.x-plane.com and wiki.x-plane.com to match the new site. If you find artifacts in those two sub-sites, please email me - I'll fix it as soon as I can. The wiki site is a MediaWiki skin - it was pretty tricky to get it working, so it may be a bit before I work all the kinks out.

Wednesday, July 29, 2009

Scenery Tools Bug Base

I have set up a public bug database for the scenery tools. The reason is that I don't have much continuous time to work on the scenery tools - it's very stop and go and will be for a while. The bug base lets you create a permanent record of a problem that I can't lose track of in the heat of whatever is on fire today.

WED is almost ready for a beta, but I am just completely swamped with work right now...for WED 1.0 I worked full time on fixing WED bugs for a significant time when WED went beta; this time around I won't have that kind of time - at least not this year.

So...we'll see how the bug base works out. My hope is that I can post WED, put it down, and pick up work on it intermittently with the bug base providing a record of what remains to be done.

Regarding other tools, there will be at least one more MeshTool beta with an improved shapefile processing algorithm that will handle broken shapefiles better. The ac3d plugin has some bugs filed against it but it's possible that they'd be deferred past the 3.2 plugin.

Accuracy, Plausibility, and OSM

I have been working on OpenStreetMap data for X-Plane this week. Use of OSM for global scenery is going to be a bit different from projects like X-VFR or other specific custom OSM-based scenery.

The issue at hand is accuracy vs. plausibility...
  • Accuracy: how much error is there between what exists in the real world and what exists in the scenery. Is that road in the right location? Is it the right type of road?
  • Plausibility: does the scenery as a whole look reasonable? Is that road on land or is it in the water? Is that river running up a mountain?
The global scenery needs to prefer plausibility over accuracy. Because we can't check and manually fix errors in the source data for the entire world, and because we don't cut the global scenery very often, it is important that the global scenery err on the side of reduced accuracy (remember, the global scenery isn't that accurate in the first place) rather than plausibility problems that will clearly be ugly and distracting.

The implication for OSM-based global scenery: not everything in OSM is going to show up in the global scenery. This would be true anyway simply due to the need to keep the global scenery compact. (I trust that OSM will grow to the point where it can source scenery larger than we can ship for the entire world.) But the global scenery generator may need to err on the side of not including data that might have plausibility problems.

Fortunately it is possible to build custom scenery from OSM as well. I don't see OSM-based global scenery as replacing efforts like X-VFR and others; rather custom scenery will always be able to use more OSM data , checking the data for accuracy, rather than reducing the data to maintain plausibility.

One technical note: I am working on an extention to the road .net file format that would allow road networks to be draped over terrain. This would allow overlay packages to add/replace road grids without having to know the shape of the base scenery mesh, and make it easier to both create custom road networks and to create the tools that manipulate them.

Tuesday, July 21, 2009

OSM: What You Can Do

I've been working with OpenStreetMap (OSM) data this week. The great thing about OSM (besides already containing a huge amount of road data) is that you can edit and correct data - that is, OSM manages the problem of "crowd sourcing" world map source data.

I get email from people all the time, saying "how can I help fix my local area of the global scenery". With OSM, you can help., by improving OSM's source data.

Here are two things that will matter in the quality of generated scenery:
  1. The oneway tag. Roads that are one-way need to have this tag, or the conversion to X-Plane might have an incorrect two-way road in place. If you don't see the one-way arrows on the OSM map rendering then this tag might be missing.
  2. The layer tag. When roads cross, "layer" tells OSM which one is on top (and that they do not intersect). Similarly, if a highway is underground, it's because it has a negative layer. If the layer tag is missing, complex intersections will probably render as junk.
In the US, a lot of OSM is built off an import of the TIGER census road map. Unfortunately TIGER in its previous released form does not contain one-way or layering information. So particularly for US cities, adding these tags will improve the road rendering a lot!

Sunday, July 19, 2009

Broken Materials in 931

There is a bug in the newly released 931. When:
  • Clouds are set to stratus and
  • Pixel shaders are on and
  • The airplane has an OBJ that uses ATTR_diffuse_rgb
the attribute is ignored, which typically makes things appear white. The primary example that has been reported to me is the throttle quadrant of the Piper Malibu.

This is simply a bug in the shaders (which is failing to apply the diffuse tinting to ambient-only lighting conditions); I have a fix, but I'm not sure how soon it will be released. We will probably do a small bug-fix release with this and one or two other things, but this is yet to be finalized, since Austin is out of the office this week.

Thursday, July 16, 2009

Bad Scenery Data

I receive bug reports periodically relating to how X-Plane and the scenery tools handle badly formed scenery files. Here's the rules:
  • It is a bug in the scenery tools if they create illegal scenery data.  They should at least flag the condition that is leading to an illegal file and refuse to proceed.
  • It is a bug in the scenery tools if they crash when trying to import illegal scenery data. Crashing can mean data loss, which is bad.
  • X-Plane's behavior with regards to illegal scenery data is undefined!  This means it is not a bug for X-Plane to show a certain behavior with bad input data.
  • There is no guarantee that X-Plane's response to illegal scenery data will remain constant over multiple patches, or even multiple executions of the sim.
This third point has some dangerous consequences:
  • X-Plane might handle illegal data in a way that an author views as useful; this "useful" side effect might go away in a future version.
  • X-Plane might crash in response to illegal data...sometimes.
  • There is no guarantee that X-Plane will provide useful diagnostics.
The reason for this scheme is that X-Plane is under pressure to get scenery loaded quickly, so at some point, there is a limit to how much validation it can do.

With that in mind, I do try to validate scenery when a problem is very common and the validation is not very expensive, or when there are serious data quality problems.  But as we move to having the tools do more validation, we can have validation where we need it: immediately, for the authors working on the scenery.

Sunday, July 12, 2009

Grand Canyon

To answer the most basic questions:
  • This is a base mesh orthophoto scenery I made with MeshTool, as a test.
  • The source DEM is 10m NED, the source imagery is 1mpp DOQQ, down-scaled to 3mpp. I gave MeshTool a point budget of 500,000 mesh points per DSF tile, and it used them all.
  • This version of MeshTool (2.0 beta 2) should be out in the next 3 days.
  • That's X-Plane 930RC4. The framerate really is around 100 fps.
  • There are no dynamic real-time shadows. Rather, the orthophotos have the shadows "baked" in because they're photos.
  • There are artifacts at the joins of the orthophotos because I spent time fixing projection errors.






Clearly we need more than 25 nm visibility in some cases!