DataRefs, the Cirrus, and Opt-In

There's a bug in X-Plane 900 RC2 that will be fixed in RC3.  Basically when you're flying in the Cirrus with a friend (also using the Cirrus via multiplayer) you'll see your own control deflections on his or her plane.

Understanding this bug gets into the way datarefs and object animation work.  In simple terms, a dataref is an access point to data provided by some other part of the sim...the flight model, or another plugin.  An object references a dataref as its source for animation values.  So if you want to animate the wings, you use sim/flightmodel/controls/lail1def or sim/flightmodel2/wing/aileron1_deg.

Now the question is: when you access this dataref in a multiplayer context, which aircraft's ailerons are you talking about?  It turns out the answer depends.

• For all datarefs except sim/flightmodel2/ and sim/cockpit2/, if the dataref needs to access a plane, the access is to the user's plane (flight 0), always.
• For sim/flightmodel2/ and sim/cockpit2/, the access is usually to flight 0.  But if we are drawing an aircraft, these datarefs refer to the aircraft being drawn!

(Note: this second feature only works correctly in RC3.)

So when you want to animate the wings of your plane in X-Plane 9, by using the sim/flightmodel2/ datarefs in your object, you'll get the behavior you want: animation with the first plane's flightmodel when your plane is loaded as plane one, animation with the second plane's flightmodel when your plane is loaded as plane two, etc.

Of course the rule of thumb is simple: if working on v9, always prefer sim/flightmodel2/ and sim/cockpit2/ when possible for animation work.

Could we have simply made the sim/flightmodel/ and sim/cockpit/ datarefs have this "pick the right airplane" behavior?  Possibly, but I took the more conservative opt-in approach.  Basically any time we want to change the behavior of the sim we can do one of three things:

1. Change it globally - all third party content is affected; don't like it, too bad.  This is how pixel shaders work; if shaders are on you can't disable the different fogging they provide for your airplane or object.
2. Opt-in - you have to change your content to get the new behavior; old content defaults to unchanged.  This is how the new datarefs work.  You have to edit your content to use the new datarefs to get the new behavior.
3. Opt-out - you have to change your content to avoid the new behavior.  For a while cockpit lighting was like this, but I received so many reports of version 8 airplanes looking totally strange that I changed the cockpit lighting choices to opt-in.

Generally opt-in is the safest thing we can do to leave previous content working; if we make a change globally then we break any content that doesn't work well with our changes.  Datarefs are used so heavily that a fundamental change in how they work would probably not be safe.

Given a choice between opt-in and opt-out I've come to prefer opt-in; both require the same amount of work for LR (we have to write code to support the old and new behavior) so why not do the thing that provides better compatibility?

Finally a general historical note: sim/cockpit2/ and sim/flightmodel2 are not just there to provide new behavior with objects; they are designed to give us a clean start in providing only authoring-related datarefs in an easier-to-understand format.  The original datarefs were meant only for programmers, so we didn't worry too much about usability.  Since then, datarefs have become the communications medium for panels and a lot of 3-d animation.

So not only do sim/cockpit2 and sim/flightmodel2/ provide correct animation behavior, but they're also easier to read.

An important distinction between sim/cockpit2 and sim/flightmodel2 is that they partition the datarefs based on perception vs. reality:

• Generally sim/cockpit2 shows datarefs as the pilot sees them in the cockpit, and actions as the pilot commands them.
• Generally sim/flightmodel2 shows the plane's behavior as it's really simulated, and actions as they actually happen.

In other words, when the pitot tube ices up, sim/flightmodel2 shows the real airspeed and sim/cockpit2 shows the incorrect airspeed.  When the hydraulic system fails, sim/cockpit2 shows that the pilot wants to bank, and sim/flightmodel2 shows that the ailerons aren't really moving much.

What this means is that you should use sim/cockpit2 for cockpit objects and generic instruments and sim/flightmodel2 for objects attached to the airplane exterior.  This will assure that failure simulation works correctly (e.g. failures are perceived correctly by the pilot) and the external control surfaces match the plane's physical behavior.

If you've studied sim/cockpit2 a lot, you know it's pretty incomplete.  We'll be adding the rest of the cockpit systems in 910.  I would have liked to get the entire set of airplane changes into 9.0 but there's a lot more to do - not just more datarefs, but also manipulators for 3-d objects and a new lighting model for aircraft interiors.  I think 910 will finish what 900 has started in terms of new airplane features.

Final random note: RC3 will allow you to pick any dataref for a generic instrument, not just sim/cockpit2 and sim/flightmodel2.  This is mainly so people writing plugins can easily hook their datarefs up to the panel by editing datarefs.txt instead of typing them by hand.

Road Overlays and Draping

X-Plane 9 allows you to put roads in an overlay DSF, but this isn't actually that useful yet.  The problem is that roads are specified using MSL elevations - that is, they are positioned in absolute space.  So you have to know where the ground is to build a road, which defeats the whole idea of an overlay.

This dates back to X-Plane 8.0 - originally all scenery load was done when a DSF was loaded, with the sim paused.  In that condition, keeping scenery load fast was a high priority.  So I set roads up to be "pre-draped" (that is, their heights are pre-determined when building the DSF). Back in the day it was a win.

Now we have multiple cores and DSF loads in the background, so we very easily could "drape" a DSF...but it gets weirder.  A road's sub-type is specified in the DSF - and sub-types are used to define bridges.

Now that's not incompatible with a "draped" scheme - imagine a road whose height is specified AGL, so a height of 0 means "run the road along the base mesh" and a height of 10 means "10 meters above the base mesh."  Now you would set the road type to overpass as the height went from 0 up to 10 meters, then back to 0, forming a bridge.

The only problem is what happens if the ground is really weird looking.  In practice what we want to do is keep the road's slope as constant as possible and build out bridges based on that. The effect can be hard to see, but if you look at bridges in the US scenery, you'll see that sometimes the banks to a river slope down but the road stays horizontal, forms a bridge, and then stays flat as the banks come up again.  This happens because our road processing code (in the DSF builder) tries to limit road slope.

That's not an easy effect to create in an overlay...you'd basically have to increase the road AGL at the rate the base mesh's AGL is decreasing (if only you could know that).

So the questions I am debating now are:

• Are AGL-draped roads useful if we don't know what the base mesh will be like?
• Does it make sense to have some other abstraction to help smooth roads when the base mesh is bumpy (and can we make something that is both simple and useful)?

Like most philosophical things I debate, this isn't scheduled to go into the sim that soon. Improved shaders, a more complete airplane SDK, and a better texture pager are all ahead of road improvements, which pushes roads out quite a bit I think.

You Can't Replace Objects in Liveries

The livery system lets you replace the image files used by an aircraft's exterior paint, and it lets you replace the image files used by the objects that are attached to the plane via the "objects" folder.

But the livery system does not let you replace the objects themselves.

(It also doesn't let you replace or modify the actual ACF file or generally change the functional behavior of the plane. The livery system is just for paint, not airplane mods.)

More Threads - the Installer

Nine women cannot have a baby in one month - that's the classic example that gets thrown around computer science for the difficulty of parallelization - that is, just because we have ten times as many resources doesn't mean we're going to go ten times as fast.

Problems of scalability via parallelization have become very important for graphics engines as everybody and their mother now has at least two cores, and users with more serious hardware are going to have four.

I get asked a lot: "will X-Plane utilize X cores..." where X is a number larger than two. My general answer is: sometimes, maybe, and probably more in the future. I can't make strong predictions for what we'll ship in the future, but the general trend for the last 18 months has been us using more cores every time we go into a piece of code to do major architectural changes.

I've been doing a lot of work on the installer this week - the first major overhaul of the installer since we originally coded it all the way back at X-Plane 8.15. And the new installers and updaters will try to take advantage of multiple CPUs where possible. A few cases:

• The X-Plane updater runs an MD5 checksum over the entire X-Plane folder to determine which version of the various file components you have and whether they need to be updated. This s not a fast process. I am working on threading this so that more CPUs can work on the problem at once. It looks like there will be only modest benefits from this because the process is also highly bottlenecked on the disk drive.
• The installation engine from the DVD will use more than one CPU to decompress files. For zip compression this wasn't very important, but the scenery will be compressed via 7-zip compression to get us down to disk DVDs. 7-Zip compresses DSFs about 10% smaller per file than zip, but it's horribly slow to decompress, so being able to throw twice the CPU at it is a big win.

Now on one hand, our top performance goals are for the sim, not the installer. On the other hand, faster installations are good. But my main point here is: when we wrote new code four years ago, we assumed one CPU and a nice graphics card. We now assume at least two cores and possibly more, and that informs the design of every new feature, not just the rendering engine. If we don't create a multi-core-friendly design, we're effectively leaving at least 50% of the CPU on the table.

When Will X-Plane Stop Saying It Is Beta?

If you look at the about box for X-Plane 9 RC1, it still lists itself as a beta.

Here's the thing: that label is dynamic. Right now when X-Plane calls up the server to see if there is a new patch, it notices that its current version (900Rc1) is newer than the latest "final version". (This is because we haven't declared any version of 9.xx final yet.)

So it thinks for a second and goes "oh - I'm newer than the latest final version, I must be a beta." And that yellow beta label appears.

When we finally declare a version final (which involves tweaking the versions listed on the servers) the existing code will then look at the server and go "oh, I'm the latest version" and that beta label will disappear.

The Global Scenery Folder

Let me answer two questions right now:

1. If you bought version 9 beta DVDs, you will not have to buy new ones when version 9 goes final.
2. If you bought version 9 beta DVDs, you do not need to manually move around any of the global scenery files that our installers put on your hard drive.

Now about this global scenery folder...

X-Plane stores its scenery in packages - a scenery package is a folder that contains all of the files needed for a given scenery element, whether custom or default. In X-Plane 7, scenery packages were only used for custom scenery, and always lived in the Custom Scenery folder directly next to the X-Plane application. This provided an easy way for users to drag third party packages into a single location to install them. (Before the custom scenery folder, installing scenery could involve a lot of "put this file here, and that file there".)

X-Plane 8 added a second location, the Default Scenery folder, which is hidden away inside the resources folder; it acts as a repository for scenery packs we ship with the sim, such as the one that contains various airport elements, and the default road packs.

In X-Plane 7 the global scenery lived directly in the resources folder in its own home; in X-Plane 8 the global scenery lives in packages inside the default scenery folder...this allows us to use one piece of code (the package loader) to load all scenery, custom and global.

The only real differences between the default scenery and custom scenery folders was priority and intended use; custom scenery is loaded with higher priority than default scenery (so you see your custom add-ons replace our default work), and the intention is that the default scenery is for Laminar to update via the web-updater and custom scenery is for you to put things into that you download off the web, etc.

With X-Plane 9 we are introducing a third location for scenery packages: the "Global Scenery" folder. It will live next to the Custom Scenery folder and X-plane application and be the location for global scenery. Its priority is intermediate - higher than default but lower than custom.

Now here's the funky thing: the X-Plane 9 beta DVDs place the global scenery into the custom scenery folder. If you've ever looked at the ranking of scenery packages, you'll see that they come out lower priority than all custom scenery, regardless of the alphabetical sorting that is usually used. The sim recognizes the peculiar placement of the global scenery and effectively labels it "global".

Future DVD pressings will simply place the global scenery in the new global scenery folder. Thus we get to the original two answers:

1. You won't have to buy new DVDs if you have the betas; the sim will continue to work with either the beta or the newer layout for global scenery for the entire v9 run.
2. Because both layouts work you don't need to move anything around. I suggest you not move any files to lower the risk of breaking things.

Scenery Tools Release

I have posted the first "combined" scenery tools release. Basically ObjView, ObjConverter, and DSF2Text (now named DSFTool) are packaged together with DDSTool (for making DDS images from PNGs) and the new MeshTool (for making base meshes); a UI application called XGrinder provides drag & drop support for most of the converters.

The tools releases will be by month (e.g. March 2008) and each release will contain the latest for all of the tools in the release. WED and the AC3D plugin will continue to be separate downloads.

I have also posted another snapshot of the source code, and generally I'll try to do a source code post whenever I do a tools post to keep the two in sync.

The AC3D plugin has been updated; I hope to have a new WED posted in the next few days.

X-Plane 9: What Comes Next

X-Plane 9.00 is going to be going final pretty soon - we're on RCs right now. But this is hardly the end of the road; rather it's the first step for a number of new development areas. What comes next?

Tools: I am working on tools releases now; a few users already have alpha copies of MeshTool. My hope is to have some new tools posted tomorrow, with more by the end of the week

More rendering engine/shader work: X-Plane 9 only begins some of the shader work we want to do; more engine enhancements will be coming in 910. Like 9.00, more advanced rendering will consume more hardware, and will be scalable via rendering settings.

Systems and Airplane SDK: X-Plane 9 introduced a lot of new features for airplane modeling, but there's still more to do. I hope to have the panel region system and advanced lighting options all fleshed out for 9.10. Austin is doing a lot of work on systems modeling. We'll be making more new airplane-related datarefs to tie it all together.

Orthophoto Scenery: I don't know if this is going to happen or not, but we get more and more requests for large-orthophoto-scenery support in X-Plane; MeshTool is only going to increase that desire by making it possible to cover large areas with orthophotos without sacrificing framerate. So I would like to do some work on the texture pager, but I do not know if I'll be able to do that in time for 9.1.

My hope is to get 9.1 into beta early so that we can start getting third party aircraft authors involved with trying new features. If you are working on an advanced airplane, keep in touch!

Version 9 - Probably Too Late

If you have a third party add-on and you haven't tested it against X-Plane 9, well...honestly it's probably a little bit too late to fix compatibility bugs now. We're in RC, and only changes to fix critical bugs are going into the sim; everything else will have to wait for 9.1.

So if you have an add-on and you haven't tested it against 9.0, for crying out loud, do it now! Submit the bug anyway - if we can't fix it for 9, we can fix it for 9.1. But...after almost 14 weeks in beta, we've got to close this build up.

BTW a minor side note on fuel pumps...

In X-Plane 864, an airplane will run even if the electric fuel pump is off. (However, if the fuel pump failure is triggered and the electric fuel pump is off, either by switch or electrical-system failure, then you're out of fuel.)

X-Plane 900RC1 restores this behavior; in beta 25, an electrical failure would turn off all sources of fuel. This is wrong for a plane like the C172 where gravity can feed the engine.

In the long term X-Plane does need a more complex abstraction (e.g. does this plane have gravity feed or engine drawn fuel, or engine driven pumps, etc. etc.) but for now the 864 model, while inaccurate and incomplete, causes less problems than beta 25, which was simply wrong for a number of planes.

Installer Theory

Let me go into the installers in a little more detail before I start a mini-riot. Basically there are three "installation" operations that we (Laminar Research) support:

1. Installing a new full sim from a DVD.
2. Installing a new demo from the internet.
3. Updating any installation from the internet.

The problem with the current installer model is that tasks two and three (get a new demo, update an existing installation) are handled by a single application. This means that the web-demo-installer/updater cannot know which choice (2) or (3) the user is trying to do (since both are legitimate) and therefore user error can result in the wrong operation.

The most common problem we have is users installing new demos when they meant to update a DVD install. The result is an old copy of X-Plane with scenery, a new copy without scenery, and a very confused user who has to contact us for help.

The solution I am working on is simple: provide separate applications for all three tasks.

1. Like before, the DVD installer comes on your DVD. It installs the DVD, nothing else.
2. A demo installer does nothing but install the demo. You get the installer from the web, probably from the "demo" page (which would have to no longer be an "update" page too). The demo installer always does a full install, and will stop you from trying to install on top of existing installations.
3. The updater is always fetched by the app and updates the app. Thus the paradigm is that the app is "self-updating" (even though most of the work is done by a helper application). That updater (fetched by the sim) can only update the copy of the sim that fetched it.

Dealing With Problems

The above work-flow is meant to address most users doing the usual thing without technical problems. We must also consider how we will deal with tech support problems, and finally what the impact is on advanced users. There are two cases where we sometimes have to help users out:

• If the DVD installer for some reason will not work for a user's computer, we usually provide a downloadable DVD installer. You would insert the DVD, run this "DVD installer" and thus install the DVD. This is not the normal case, but we will probably continue to provide DVD installers specifically for users with tech support problems (based on the tech support incident), just like we have been in the past.
• We will provide the updater directly for users who need to update the sim but cannot launch the sim. This is the exception case, so a direct link to the updater would not be globally advertised on the main page of the sim. Rather it would be a support link, again only provided to users who really need it. 98% of the user base will be able to update from the sim.

When you run the updater from the web (the support case) it will prompt you to locate your X-Plane folder if needed.

Advanced Users

Will you be able to keep multiple copies of X-Plane around? Absolutely. But you'll have to manage your operations in terms of the "three operations" (make a new install from the DVD, make a new demo install from the web, update any one version you have laying around).

If you want to get a new web demo, you'll have to use a separate application than you would for an update. I don't think this is a huge problem; generally your best bet for keeping an old copy of X-Plane (when running a new beta) is to simply copy the entire X-Plane folder, rather than downloading the contents from the net.

Fun With Installers

So first, if I promised you scenery tools about now (MeshTool, AC3D, or any of the other things I've been asked for), please wait two weeks and ask again. I thought Austin would be cutting master DVDs while I would have time for tools, but instead I will cut the DVDs, so tools will have to wait until the masters are done.

To calm any fear: if you have X-Plane 9 DVDs, you will not need to buy new ones. We will be periodically updating our master DVDs (just like we always have), but we've taken a number of steps to ensure that everything new since the original "Beta 1" DVDs will be deliverable by reasonably small incremental web updates.

If you are reading this blog, you're probably an advanced X-Plane user, possibly one who makes airplanes or scenery or even plugins, and you're probably comfortable with computers and know a lot about the sim. Please bear in mind as I describe some of the changes we're making in the installer that we have users who have never used a computer before, and purchased their first computer to run X-Plane.

Simply put, the primary goal of the installer is to allow the non-experts to use X-Plane too.

Anyway, with this post I'd like to call attention to a feature that I believe very few people know about (because it was broken for a long time and yet we didn't get many reports).

You can update X-Plane 8 or 9 from the About Box - when you pick "About X-Plane" the sim will check our update servers for new versions, and if one is found, it will give you an update button. This update button will then download the very latest installer and launch it, setting it to update the copy of X-Plane you were running.

X-Plane 9 beta 25 will be out soon; when it comes out, please try updating by using the About Box in beta 24. Let me know if the process works correctly.

(What's so great about updating from the about box? Well, one of the biggest tech support calls we get is users who do not successfully patch their existing copy of x-plane, but use the web updater to fetch a whole new X-Plane, which of course has no scenery. When you get the installer from the about box, you always get the latest correct installer, and the sim tells the installer which copy to live. In the future we'll be removing the "destination" button when you run this way, so it's as simple as "go".)

Don't use ATTR_cockpit outside the cockpit objects

I've blogged about this before, but...let me be totally clear:

Don't use ATTR_cockpit in objects that are not one of the two cockpit objects for your airplane.

Don't use ATTR_cockpit in the attached misc. objects for your plane - move the parts of the mesh that require ATTR_cockpit into the cockpit object.

Don't use ATTR_cockpit in scenery objects.

The OBJ spec basically says as much when it says "don't use cockpit features" outside of cockpits.

Now what goes wrong if you violate this varies with the betas vs. X-Plane 8, but I can tell you this: no version of X-Plane has ever shipped that will correctly handle ATTR_cockpit in attached objects for all cases.  There's always been bugs in this not-such-a-good-idea code path; it's just the severity has varied over time.

Hardware Profiles

X-Plane 9 currently recognizes (roughly) three categories of graphics hardware:

• Non-Pixel-Shader hardware (GeForce 2,3,4, Radeon 7000-9200)
• First-Generation Shader Hardware (GeForce FX 5nnnn, Radeon 9500-9800, X300-X600)
• Later-Generation Shader Hardware (GeForce 6, 7, 8, Radeon X200, Radeon X700+)

That first bucket is pretty simple: those cards don't support programmable pixel shaders (as we know them today) and can't run any shader effects. The "use pixel shaders" check box doesn't appear in the rendering settings.

The distinction between the later two is a little bit more subtle. Basically the first generation of pixel shader cards (the 9700 and friends) support only 96 instructions for each pixel shader; this puts us right on the edge of sometimes not being able to draw all of our effects; we have to simplify the water slightly to make it work. The next generation of chips (X850 and friends) doesn't have this limitation.

By comparison, while NVidia cards have been able to handle long shaders from day one, the GeForce 5's shader performance is really poor.

So we bucket all of these chips as "first-gen". When we detect this we:

• Simplify shaders slightly (gets us out of trouble with the 9700).
• Don't default to shaders being on when the sim is first booted (because the framerate will probably be unusably slow).

Even though the 9700 provided very usable shader performance in its day, by the standards of modern GPUs, this older chip isn't that fast, so it's probably for the best that we not enable reflective water by default on machines with these cards.

By comparison, X-Plane deals with almost all other capabilities on an a-la-carte basis; particualr features are enabled if the right menu of hardware features is available. We do this to try to deal more flexibly with the wide variety of cards that are out there. Some examples:

• You'll get hardware accelerated runway lights if your card supports pixel shaders and sprites (virtually all shader-enabled cards have sprites).
• You'll get sun-glare effects if your card supports pixel counting (virtually all modern cards can do this).
• The non-pixel-shader rendering code will show more detail if your card supports more texture units (this is only an issue with very old hardware).

I've been looking over hardware profiles a lot lately, and I suspect that the next big "jump" in hardware will be the DX10-compliant cards (GeForce 8, Radeon HD). There's a lot of fine print in what the various cards can do between all of the pre-DX10 cards; at some point when we decide what menu of features we'll require for rendering, we need to simplify.

My guess is that when we start to have "really advanced" pixel shaders that require hardware more sophisticated than what we need now, we'll simply require a DX10 card. Otherwise we'll have to sort through 8 different profiles of fine print, only to attempt to partially support cards that probably won't be fast enough anyway.

(That is to say, a feature is only useful for us if it can run reasonably quickly. It doesn't make sense for us to try to make a special "simplified" version of a rendering feature for, say, the X850 if every X850 is going to turn it off every time for framerate reasons.)

If any of this turns into hardware buying advice, I suppose it would be this:

• If you are deciding between a DX10 and DX9 card (e.g. between the HD2400 and X1900, or GeForce 8600 vs 7900) go for the newer generation DX10 ards (HD or GeForce 8); if the card has decent performance you'll also be setting yourself up for future features.
• As always, pay attention to the fine print of the model numbers, particularly the configuration. Lower model number cards basically have fewer parallel components than higher number ones and that leads directly to lower framerate.

I see an add online for the GeForce 8500 for $70 and 8600 for $90. But if you look at the links below, you'll see that the 8500 has only half the shaders of the 8600 - that's going to be a huge performance difference for $20.

(So the moral of this story is: try to get an HD or GeForce 8 card, but don't dip into the really low end cards because they're stripped down too far for X-Plane use.)

These pages (NV, ATI) on Wikipedia list specs for a whole pile of cards and can be useful to decode the fine print.