More Threads

X-Plane 940 beta 1 is out...it will take a little bit to get the release notes and docs on the website completely up-to-date. We're still dealing with loose ends from our migration to the new web site, and most of my office is packed up for a move to the Boston area. I'll try to get docs up as fast as I can given the chaos flying about.

Given the interest multi-core stirred up in previous notes, I will mention one change to 940: with this build we've added yet more multi-core to X-Plane.

In 931, X-Plane will use as many cores as you have to load textures, but only one to build "3-d scenery" (a loose category for the work we do when we make airport taxiways and lines, build forests, and extrude roads).

In 940, this "3-d scenery" is also done on as many cores as you have. This should speed up load times a bit, particularly under very heavy tree settings, and hopefully keep the forest engine running faster for users with more cores.

It also sets us up for long-term growth; X-Plane's visual quality is sometimes limited by the time to build 3-d meshes...being able to do this work on many cores means we can use higher quality algorithms.

Consider for example the roads: my original "road extruder" (the code that converts a vector road into a 3-d model, called an extruder because it builds a 3-d road from a cross section like one of those play-dough toys) made beautiful intersections with stop-lines and cross walks and lots of other great stuff.

It was also really slow. And at the time the sim wouldn't fly at all while roads were extruded, so speed was of primary concern. So I replaced it with the much dumber extruder you see today, where intersections are basically ignored.

Now that we have 3-d scenery build on multiple cores, we can begin to provide rendering options that take more CPU time but produce higher quality results. The trees and airport layouts already do this (in that they take more time and produce slower, more detailed, higher triangle count sceneery at higher rendering setting for the same input DSF ad apt.dat file). With more cores, we can continue this strategy with roads and other parts of the sim without worrying about overloading the one core that was doing this work.

Of course just because we can use 8 cores doesn't mean we do...you won't see 8 cores maxed out very often, particularly if you have simple scenery and a very fast machine.

I am the Spam King!

If I have not emailed you back, it's probably because I've been very busy trying to interleave X-Plane coding and packing up the house. But it is also possible that my email response bounced because my web server has ended up on a bunch of anti-spam "black-lists".

Black-listing seems like a good idea at first: we'll gather up a list of all of the IP addresses from which spam comes from, then publish them. Then your local mail server can use that list to filter spam - you never see it!

In practice it doesn't work so well...example: http://www.mipspace.com/. The IP address for my server (XSquawkBox) is now on this list. Why?

MIPSpace is a list of IP Addresses associated with known commercial marketing companies.

Since my server is used for my own personal email and to run the SDK website, I'm not sure why I am on the list. I have sent them an email to clear things, but in general I hit an anti-spam/black-list bounce somewhat frequently now, and frankly I don't have time to separately try to clear my name from every guilty-until-proven-innocent blacklist that pops up and screws up my email.

If I seem disproportionately grumpy about this, it could be due to one of two reasons:

1. Not replying to emails is generally bad customer service. (Okay, my in-box is backed up four months...that's bad.) I don't like the idea that a customer might perceive us (LR) as being unresponsive because some third party with no skin in the game decides to black-list us.

   The blacklist has no incentive to be accurate - it's not their lost customers if email doesn't go through.

2. I'm not at all convinced that this is going to cut down unsolicited commercial mail and/or spam.

   In the spam case, spammers can send from botnets - they have access to a huge number of ever-changing IPs. Unless we are prepared to blacklist the entire internet, the blacklists are going to pick up more and more false positives while spammers find ways to harvest fresh, untainted IP addresses. The whole IP-reputation strategy assumes that IPs are hard to change. In practice, IPs are very, very easy to change.

   Commercial mail is a lost cause too - even if I am being solicited for commercial mail I don't want, no program or automatic process is ever going to tell the difference between the confirmation of my invoice and a list of discounts from the same company. When it comes to commercial mail, the reputation damage has to be done to the company, not the IP.

   (The company does have reputation to risk - if we are known as a company that doesn't honnor a "do not subscribe" policy, then customers can choose to not buy our products.)

It could also be because I ran out of Viagra and don't have a diploma from a prestigious non-acredited university.

Switch and Bail

I will be out of the office next week - it's August, time to head for the mountains...no cell coverage, no internet, no computers, no electricity...no mountains actually. (This is upstate New York...mountainous to someone from Boston but not mountains compared to what Sergio normally deals with.)

As you know, LR often shows stellar judgement in managing release risk. So in true X-Plane tradition, I swapped in the new revamped website last night, just in time to head for the hills and avoid the fall-out.

Here's the short version:

• I am wicked stupid.
  
• When I did the swap last night, I screwed up the files that manage the auto-update functionality, hosing the updater.
• I fixed these files this afternoon, once the message got back to me.

So if you saw weird stuff happen with the updater or the sim last night, please try again - it should be fixed. If it is still broken, please send an email to Austin and myself - one of us will hopefully be around.

I have also reskinned scenery.x-plane.com and wiki.x-plane.com to match the new site. If you find artifacts in those two sub-sites, please email me - I'll fix it as soon as I can. The wiki site is a MediaWiki skin - it was pretty tricky to get it working, so it may be a bit before I work all the kinks out.

Scenery Tools Bug Base

I have set up a public bug database for the scenery tools. The reason is that I don't have much continuous time to work on the scenery tools - it's very stop and go and will be for a while. The bug base lets you create a permanent record of a problem that I can't lose track of in the heat of whatever is on fire today.

WED is almost ready for a beta, but I am just completely swamped with work right now...for WED 1.0 I worked full time on fixing WED bugs for a significant time when WED went beta; this time around I won't have that kind of time - at least not this year.

So...we'll see how the bug base works out. My hope is that I can post WED, put it down, and pick up work on it intermittently with the bug base providing a record of what remains to be done.

Regarding other tools, there will be at least one more MeshTool beta with an improved shapefile processing algorithm that will handle broken shapefiles better. The ac3d plugin has some bugs filed against it but it's possible that they'd be deferred past the 3.2 plugin.

Accuracy, Plausibility, and OSM

I have been working on OpenStreetMap data for X-Plane this week. Use of OSM for global scenery is going to be a bit different from projects like X-VFR or other specific custom OSM-based scenery.

The issue at hand is accuracy vs. plausibility...

• Accuracy: how much error is there between what exists in the real world and what exists in the scenery. Is that road in the right location? Is it the right type of road?
• Plausibility: does the scenery as a whole look reasonable? Is that road on land or is it in the water? Is that river running up a mountain?

The global scenery needs to prefer plausibility over accuracy. Because we can't check and manually fix errors in the source data for the entire world, and because we don't cut the global scenery very often, it is important that the global scenery err on the side of reduced accuracy (remember, the global scenery isn't that accurate in the first place) rather than plausibility problems that will clearly be ugly and distracting.

The implication for OSM-based global scenery: not everything in OSM is going to show up in the global scenery. This would be true anyway simply due to the need to keep the global scenery compact. (I trust that OSM will grow to the point where it can source scenery larger than we can ship for the entire world.) But the global scenery generator may need to err on the side of not including data that might have plausibility problems.

Fortunately it is possible to build custom scenery from OSM as well. I don't see OSM-based global scenery as replacing efforts like X-VFR and others; rather custom scenery will always be able to use more OSM data , checking the data for accuracy, rather than reducing the data to maintain plausibility.

One technical note: I am working on an extention to the road .net file format that would allow road networks to be draped over terrain. This would allow overlay packages to add/replace road grids without having to know the shape of the base scenery mesh, and make it easier to both create custom road networks and to create the tools that manipulate them.

OSM: What You Can Do

I've been working with OpenStreetMap (OSM) data this week. The great thing about OSM (besides already containing a huge amount of road data) is that you can edit and correct data - that is, OSM manages the problem of "crowd sourcing" world map source data.

I get email from people all the time, saying "how can I help fix my local area of the global scenery". With OSM, you can help., by improving OSM's source data.

Here are two things that will matter in the quality of generated scenery:

1. The oneway tag. Roads that are one-way need to have this tag, or the conversion to X-Plane might have an incorrect two-way road in place. If you don't see the one-way arrows on the OSM map rendering then this tag might be missing.
2. The layer tag. When roads cross, "layer" tells OSM which one is on top (and that they do not intersect). Similarly, if a highway is underground, it's because it has a negative layer. If the layer tag is missing, complex intersections will probably render as junk.

In the US, a lot of OSM is built off an import of the TIGER census road map. Unfortunately TIGER in its previous released form does not contain one-way or layering information. So particularly for US cities, adding these tags will improve the road rendering a lot!

Broken Materials in 931

There is a bug in the newly released 931. When:

• Clouds are set to stratus and
• Pixel shaders are on and
• The airplane has an OBJ that uses ATTR_diffuse_rgb

the attribute is ignored, which typically makes things appear white. The primary example that has been reported to me is the throttle quadrant of the Piper Malibu.

This is simply a bug in the shaders (which is failing to apply the diffuse tinting to ambient-only lighting conditions); I have a fix, but I'm not sure how soon it will be released. We will probably do a small bug-fix release with this and one or two other things, but this is yet to be finalized, since Austin is out of the office this week.

Bad Scenery Data

I receive bug reports periodically relating to how X-Plane and the scenery tools handle badly formed scenery files. Here's the rules:

• It is a bug in the scenery tools if they create illegal scenery data.  They should at least flag the condition that is leading to an illegal file and refuse to proceed.
• It is a bug in the scenery tools if they crash when trying to import illegal scenery data. Crashing can mean data loss, which is bad.
• X-Plane's behavior with regards to illegal scenery data is undefined!  This means it is not a bug for X-Plane to show a certain behavior with bad input data.
• There is no guarantee that X-Plane's response to illegal scenery data will remain constant over multiple patches, or even multiple executions of the sim.

This third point has some dangerous consequences:

• X-Plane might handle illegal data in a way that an author views as useful; this "useful" side effect might go away in a future version.
• X-Plane might crash in response to illegal data...sometimes.
• There is no guarantee that X-Plane will provide useful diagnostics.

The reason for this scheme is that X-Plane is under pressure to get scenery loaded quickly, so at some point, there is a limit to how much validation it can do.

With that in mind, I do try to validate scenery when a problem is very common and the validation is not very expensive, or when there are serious data quality problems.  But as we move to having the tools do more validation, we can have validation where we need it: immediately, for the authors working on the scenery.

Grand Canyon

To answer the most basic questions:

• This is a base mesh orthophoto scenery I made with MeshTool, as a test.
• The source DEM is 10m NED, the source imagery is 1mpp DOQQ, down-scaled to 3mpp. I gave MeshTool a point budget of 500,000 mesh points per DSF tile, and it used them all.
• This version of MeshTool (2.0 beta 2) should be out in the next 3 days.
• That's X-Plane 930RC4. The framerate really is around 100 fps.
• There are no dynamic real-time shadows. Rather, the orthophotos have the shadows "baked" in because they're photos.
• There are artifacts at the joins of the orthophotos because I spent time fixing projection errors.
  

Clearly we need more than 25 nm visibility in some cases!

Invisible Hard Surfaces

I have received several requests for a transparent runway with a physical surface type. That request is just strange enough that we need to look back and ask, "how did we get here?"

High Level and Low Level Modeling

The "new" airport system, implemented in X-Plane 850 (with a new apt.dat spec to go with it) is based on a set of lower level drawing primitives, all of which are available via DSF. In other words, if Sergio and I can create an effect to implement the apt.dat spec, you can make this effect directly with your own art assets using a DSF overlay. This relieves pressure on the apt.dat spec to become a kitchen sink of tiny details.

The goal of apt.dat is to make a visually pleasing general rendering of airport data. DSF overlays provide a modeling facility.

Little Tricks

It turns out there are two things the apt.dat file "does" with the rendering engine that you can't do in an overlay DSF:

1. The apt.dat file registers runways in the airport dialog box (for starting flights, positioning the airplane, etc.).

2. When the apt.dat reader places OBJs to form approach lights, it can offset their "timing base", which is why the rabbit flashes in sequence.

(If you were to place a sequence of approach lights with rabbits in an overlay, every single light would flash at the same time because the DSF overlay format does not have a way to adjust the object's internal timing parameter.)

The solution: the transparent runway. The idea of the transparent runway is to create with the apt.dat file the two aspects of a runway that you can't build with a DSF overlay: the approach lights and the entry in the global airport dialog box. Transparent runways leave the drawing and surface up to you.

My thinking at the time was that the actual runway visuals and physics would be implemented together via either draped polygons or a hard OBJ.

Orthophotos and Bumps

So why do authors want a transparent but hard runway? The answer is orthophotos. With paged orthophotos, it is now possible to simply put down orthophotos for the entire airport surface area (whether as overlays or a base mesh) at some high resolution (our runways are 10 cm per pixel - I'm not sure if the whole airport area can be done at that resolution) and not have any special overlays for the runways. The transparent + hard runway would change the surface type.

I'm not sure if this is a good idea, but I'm pretty sure that this feature belongs in overlay DSFs and not the apt.dat file.

• Such a technique (varying hard surfaces independent of a larger image) is useful for more than just airports (and certainly more than just runways).
• The technique is unnecessary unless a DSF overlay is in use.
• Unlike nearly all of the rest of the apt.dat file, such an abstraction (invisible but bumpy) is much more a modeling technique and less a description of a real world runway.

I'm not sure we would even want the runway outline to be the source of hard data. If there are significant paved areas outside the runway then a few larger hard surface polygons might be more useful.

Where Do I Find the 930 Datarefs

The dataref documentation on the X-Plane SDK website is updated for each release of X-Plane 9.

X-Plane 930 has not yet been released. It is a "release candidate" but since we haven't signed off on it yet, 922r1 is still the most current real release of X-Plane, and it's what users get when they update without asking for a beta. So the website has 922 datarefs.

Since version 9, every release of X-Plane (including betas and RCs) has a copy of datarefs.txt in the plugins folder that is correct for that release. In other words, the docs that ship with X-Plane and the sim itself are always in sync.

So for now use datarefs.txt in the 930 RC 2 plugins folder! When 930 goes final, the website will be updated.

ASTER For Custom Scenery

A few days ago, the ASTER GDEM was released. Basically ASTER GDEM is a new elevation set with even greater coverage than the SRTM. Basically both SRTM and ASTER (I'll drop the GDEM - in fact ASTER prodcues more than just elevation data, but the elevation data is what gets flight simmers excited right now) are space-based automated measurement of the earth's height. But since ASTER is on a satellite (as opposed to an orbiting space shuttle) it can reach latitudes closer to the poles.

So what does this mean for scenery? What does it mean for the global scenery? A few thoughts:

• ASTER data is not yet very easy to get. You can sign up with the USGS distribution website but you're limited to 100 tiles at a time, with some latency between when you ask and when you get an FTP site. Compare this to SRTM, which can be downloaded automatically in its entirety, or ordered on DVD. ASTER may reach this level of availability, but it's not there yet.

• ASTER is, well, lumpy. (Nasa says "research grade", but you and I can say "lumpy".) Jonathan de Ferranti describes ASTER and its limitations in quite some detail. Of particular note is that while the file resolution is 30m, the effective resolution of useful data will be less.

  SRTM has its defects, too, but ASTER is very new, so the GIS community hasn't had a chance to produce "cleaned up" ASTER. And clean-up matters; it only takes one really nice big spike in a flat flood plane to make a "bug" in global scenery. I grabbed the ASTER DEMs for the Grand Canyon. Coverage was quite good, despite the steep terrain angle (steep terrain is problematic by design for SRTM) but there were still drop-out areas that were filled with SRTM3 DEMs, and the filled-in area was noticeable.

• By the numbers, ASTER is not as good as NED; I imagine that other country-specific national elevation datasets are also both more accurate and more precise than ASTER.

• The licensing terms are, well, unclear. The agreements I've seen imply a limited set of research uses for the data. The copyright terms are not well specified.
  

So at this point I think ASTER is a great new resource for custom scenery, where an author can grab an ASTER DEM in a reasonable amount of time, check it carefully, and thus have access to high quality data for remote parts of the Earth, particularly areas where locally grown data is not available or not high quality.

In the long term, ASTER is a huge addition to the set of data available because of its wide-scale coverage of remote areas, and because it can fill holes in SRTM. (ASTER and SRTM suffer from different causes for drop-outs, so it is imaginable that there won't be a 1:1 correlation in drop-outs.)

But in the short term, I don't think ASTER is a SRTM replacement for global scenery; void-filled SRTM is a mature product, reasonably free of weirdness (and sometimes useful data). ASTER is very new, and exciting, but not ready for use in global scenery.

The Will to Rewrite

FSBreak interviewed Austin last week....it's an interesting listen and they cover a lot of ground. A few comments on Laminar's approach to developing software:

That Code Stinks!

Austin is absolutely correct that we (LR) write better software because neither of us are shy about telling the other when a piece of code stinks. But I think Austin deserves the credit for creating this environment. An "ego-free" zone where people can criticize each other honestly and freely is a rare and valuable thing, in many domains, not just music.

When I first came to LR, Austin created this environment by responding positively to feedback, no matter how, um, honest. When I first came to LR 100% of the code was written by him and 0% by me. Thus if I was going to say "this piece of code really needs to be different", it was going to be Austin to either run with it or try to defend his previous work.

To his credit, Austin ran with it, 100% of the time. I can't think of a single time that he didn't come down on the side of "let's make X-Plane better". That set the tone for the environment we have now: one that is data driven, regardless of who the original author is.

I would say this to any programmer who faces a harsh critique of code: good programmers write bad code! I have rewritten the culling code (the code that decides whether an OBJ really needs to be drawn*) perhaps four times now in the last five years. Each time I rewrote the code, it was a big improvement. But that doesn't make the original code a mistake - the previous iterations were still improvements in their day. Programming is an iterative process. It is possible to write code that is both good and valuable to the company and going to need to be torn out a year later.

A Rewrite Is Not A Compatibility Break

Austin also points to the constant rewriting of X-Plane as a source of performance. This is true too - Austin has a zero tolerance policy toward old crufty code. If we know the code has gotten ugly and we know what we would do to make the code clean, we do that, immediately, without delay. Why would you ever put off fixing old code?

Having worked like this for a while, I am now amazed at the extent to which other organizations (including ones I have worked in) are willing to put off cleaning up code organization problems.

Simply put, software companies make money by changing code, and the cost is how long the changes take. If code is organized to make changing it slower, this fundamentally affects the financial viability of the company! (And the longer the code is left messy, the more difficult it will be to clean up later.)

But I must also point out one critical detail: rewriting the code doesn't mean breaking compatibility! Consider the OBJ engine, which has been rewritten more times than I care to remember. It still loads OBJ2 files from X-Plane 620.

When we rework a section of the sim, we make sure that the structure of the code exactly matches what we are working on now (rather than what we were working on two years ago) so that new development fits into the existing code well. But it is not necessary to drop pieces of the code to achieve that. I would describe this "refactoring" as straightening a curvy highway so you can drive faster. If the highway went from LA to San Francisco, it still will - just in less time.

In fact, I think the issue of compatibility in X-Plane's flight model has a lot more to do with whether the goal is to emulate reality or past results. This debate is orthoganal to refactoring code on a regular basis.

* Since OBJs are expensive to draw and there are a huge number of OBJs in a scenery tile, the decision about whether to draw an OBJ is really important to performance. Make bad decisions, you hurt fps. Spend too long debating what to draw, you also hurt fps!

Do Not Work Around Our Bugs!

For most of its beta run, X-Plane 930 beta 14 didn't handle engine power limiting very well*. Here's the short version of the saga.

• Real planes sometimes have systems to limit total power output, because the power output of the engine (whether torque or internal temperature) can exceed safe operating limits at sea level).
  
• With X-Plane 9.00 you could set a critical altitude for an airplane - below this altitude, X-Plane would limit the power output of the jet. The idea is (roughly) to simulate these limiters by derating the engine's power output below this "critical altitude".
• This feature was really only meant for reciprocating engines - when Austin discovered in 9.20 that people were using this for turbines (understandable, given that there was no alternative) he simply disabled the feature for turbines. That wasn't so good.
• To resolve the situation a little bit more cleanly, X-Plane 9.30 has a setting per airplane called "FADEC - automatically keep engines from exceeding max allowable power or thrust" that, when checked, gives you version 9.00 style behavior.

Now this was mostly good except for one problem: the betas would default this setting to "off" when loading an old plane. Since version 9 always acted as if the "FADEC"** was on, this meant that old planes would need to be edited.

Finally with beta 14 we switched things: beta 14 and newer default old planes to have the FADEC checkbox on, so that planes match their old behavior - you can turn the check-box off if you don't want this behavior.

There is one final hitch: if you already went in and edited your airplanes for the earlier betas, you will find that they are now set wrong. You will have to reset the check-box. If you go back to the original, unedited, 920-compatible airplane you will find they "just work".

I mention all of this to make two points:

1. File formats for new features are subject to change during beta. In this case, the file format for the new FADEC check-box (introduced in 930) changed at beta 14. The OBJ syntax for dynamic lighting changed during beta too. Don't do "bulk" work (e.g. change a large number of planes in the same way ) on your fleet based on betas - you might have to redo that work again! Wait until the sim goes final. That's when the file formats are locked up.
2. Don't work around bugs in the sim. I have seen so many forum posts where there is a trivial bug in the sim (e.g. the sim is just screwed up in a simple way) and authors go in and update scenery to work around the bug. File a bug, then wait! If you work around the bug, we can't fix the bug, and if we don't fix the bug, then the bug just bites other users.

* Disclaimer: I don't do systems, I don't know anything about airplanes, so this whole discussion will be heavily simplified. The point of this post is not to get into a discussion of FADECS - in fact, don't even bother to post about FADECs, I'm not going to approve them. If you want to talk about FADECs and engine modeling, email Austin. The point of this post is one about file formats and compatibility.

** FADEC isn't a very good name for this feature - the feature generically limits power, without specifying a mechanism. My understanding is that some airplanes have mechanical limters, like a pressure valve on a turbo. Some planes have no limiters at all...ask a pilot "can you cook the engine by pushing the throttle too far" he will say "I'm not going to be the one to find out."

Optimization By Check-Boxes

In the next X-Plane 930 beta (it should post today I think) the rendering settings have two new check-boxes: one to enable the "dynamic" airplane shadow and one to enable per-pixel lighting.

In the last week a number of users emailed me performance numbers via the fps test, and from what I can tell, 99% of performance problems can be attributed to these two new features chewing up resources in a way that 922 did not. When the features are both disabled, from what I can tell, the sim should be as fast or faster than 922.

The new beta will also limits dynamic shadows to your aircraft - beta 13 will calculate a dynamic shadow for every aircraft, which is unacceptably slow when you have a lot of AI planes enabled.

I may still be able to improve the performance of the per-pixel-lighting shader, but fundamentally per-pixel lighting is going to be more expensive than per vertex. The average X-Plane scene might have 200,000 to 500,000 vertices. At absolute minimum resolution, no FSAA, you're going to have over 700,000 pixels even if there is no "over-draw" - you could easily have 10x that fill rate with only a modest increase in overdraw, full-screen anti-aliasing and window size. Simply put, per-pixel lighting is more work.

Please bear in mind: without per-pixel lighting X-Plane's pixel shader is extremely simple. If you have a "low-end" card this could give you the illusion of GPU power when there is really not much under the hood.

Examples of low-end cards: GeForce 7300, GeForce 8400, GeForce 9400, Radeon X300, Radeon X1300, Radeon HD2400. All of these cards are the younger brother of a fairly capable card, but with fewer pixel shader units/cores. If each unit is doing very little work, you don't need that much pixel-filling power...but when we go to a "real" shader, the difference between a GeForce 8400 and 8800 becomes very, very apparent. Simply put, even with optimization the GeForce 7300 (for example) will never run a huge monitor with per pixel lighting and high FSAA.

To Copy Or To Reference

In designing interfaces for building planes, writing plugins, etc. one of the main design questions that keeps coming up is: to copy or to reference? Should authors simply refer to an art asset, piece of data, or code in order to utilize it, or should the author copy a snapshot into the custom add-on. There isn't one right answer. Here are the main considerations.

Performance and Efficiency

One of the obvious considerations is efficiency: in some cases we might be able to provide better performance when an art asset is referred from a common source.

For example, in some cases X-Plane will consolidate VRAM use based on actual files, so a library object is loaded once no matter how many packages use it, but is loaded many times if a package copies it.

(In other cases X-Plane will actually merge multiple copies of a resource - referencing is only a win in some cases.)

An indirect consideration: if an art asset is provided by Laminar Research and is used by reference, then a new update can provide a new, better optimized art asset - see below.

Dependencies and Contracts

When someone uses an art asset, algorithm, etc. by reference, it creates an implicit contract by the provider of the asset by reference to not change the properties of the asset. By comparison, when the asset is copied, the contract is only to support the format that the asset is encoded in.

This is the main reason why I am often against providing new assets by reference, whether it is a new dataref, texture, etc. Often I will simply send a user a snippet of code, rather than making X-Plane's version available via a dataref. The idea is that copying does not create a new interface (and thus a new "contract") between X-Plane and the add-on.

Copyright and Legal Issues

For historical reasons, the US legal system describes the privileges of intellectual property owners by regulating the act of copying. (To say that this is a bit quaint in the digital age doesn't even scratch the surface, but that's a rant for another post.) The result of this particular regulation of copying (but not of referencing) is that the decision to provide an asset by copy vs. reference has legal implications. If the author does not want to go through licensing, referencing may be the only option.

The Constraints of Hardware

In a previous post (in which I tried to argue that threading is a "how" and not a "what" when it comes to feature requests) a user made this comment:

That is, that I feel you are a bit too concerned about the fact that XP has to be possible to run on a 2001 year machine. This really halts the development although you could add options to turn this and that off.

I'd like to side-step around the details of cost-benefit analysis (e.g. do the sales from low-end systems pay for the development of a renderer with lower system requirements) but take a second to focus on three general issues:

• Is there a cost to developing a scalable renderer?
• How does the trend of hardware development affect hardware?
• How do marketing forces affect both of the above?

Scalability

Is there a cost to writing a renderer that can run on a wide range of hardware? Absolutely. Obviously we have to write more code to do that.

But there is an additional cost: there are some rendering engine design decisions that have to be made system-wide. It's not practical to provide different scenery files for different hardware (since we are limited by distribution on DVD). In some cases we have to pick a non-ideal data layout (for the highest end hardware) to support everyone.

But: before you raise up arms against your fellow X-Plane user who is holding you down with his GeForce 2 MX and P-III 800 mhz machine, bear in mind that the problem of picking a data format is a bit unavoidable. Even if we targeted the highest-end machines and told everyone else to jump in a lake, those decisions would appear to target rather quaint machines only a year into the version run. At some point we have to pick a line in the sand.

There is some light at the end of the tunnel when it comes to scalability: as computers become (on average) bigger and faster, we can start to defer at least a little bit of the work of scenery generation to while the sim is running. When we first designed the new sceney system (for X-Plane 8) most users did not have dual-core machines, so the doing work on the scenery was very expensive. We preprocessed as much as possible. This isn't as necessary any more.

So are high-end users limited by having one renderer that fits all sizes? Perhaps a little bit, but any design choice is only going to fit one hardware profile perfectly, and hardware is a moving target; today's shiny new toy is tomorrow's junk.

Hardware Growth

Every two years (to be very loose about things) the number of transistors we can pack on a chip doubles. This "transistor dividend" can be turned into more cores for a CPU, or more shading units (which are now really just cores) for a GPU.

And this gets to the heart of why I don't think we can say "forge the low-end" any time soon. Imagine that we support 6 years of hardware with X-Plane, and the best hardware is 8 times as powerful as the low-end hardware. Fast-forward two years - we drop two-years of hardware and two-years of new ATI and NV graphics cards come out. What is the result?

Well, the newest hardware is still 8x as powerful as the old hardware, but the difference in the polygon budget between the two has now doubled! In other words, the gap in absolute performance is doubling every two years, driving the two ends of our hardware spectrum farther apart. (Absolute performance is what Sergio and I have to worry about when we design a new feature. How many triangles can we use is an absolute measurement.)

If we say "okay forget it, only 3 years of supported hardware" that gets us out of jail for a little while, but eventually even the difference between the newest and slightly off-the-run hardware will be very large.

A gap in hardware capability is inevitable and it will only get worse!

Market Divergence

You may have noticed that the above paragraph makes a really gross assumption: that the lowest end hardware we support is the very best card on the market from a certain number of years ago. Of course this isn't true at all. The lowest end hardware we support was probably pretty lame even when it was first made. The GeForce FX 5200 was never, even for a microsecond, a good graphics card. (It was, however, quite cheap even when first released.)

So the gap we really have is between the oldest low-end and newest high-end hardware, which is really quite wide. Consider that in May 2007 the GeForce 8800 Ultra was capable of 576 GFLOPs. Two months later (July 2007) the GeForce 8300 GS was released, packing a whopping 22 GFLOPs. In other words, in one video card generation the gap between the best and worst new card NVidia was putting out was 26x! (I realize GFLOPs isn't a great metric for graphics card performance - really no one metric is adequate, but this example is to illustrate a point.)

Let's go back in time a few years. In February 2002, NVidia released the GeForce 4 Ti (high-end) and MX (low-end. The slowest MX could fill 1000 MT/s, while the fastest Ti could fill 2400 MT/s. That's a difference in fil rate of "only" 2.4x.

What's going on here? Commodification! Simply put, graphics cards have reached the point where a lot of people just don't care. Very few users need the full power of a GeForce 8800, so a lot of lower-end machines are sold with low-end technology - more than adequate for checking email and watching web videos. This creates a market for low-end parts and creates a wider "gap" for X-Plane. Dedicated returning X-Plane users might do the research and buy the fastest video card, but plenty of new users already have the computer, and it might have something unfortunately (like a Radeon X300 or Intel GMA950) already on the motherboard.

As X-Plane's hardware needs diverge from the needs of mainstream computer users, we can expect some but not all of our users to have the latest and greatest. We can also expect plenty of new users to have underpowered machines.

Let me go out on a limb (I am not a technologist or even a hardware guy, so this opinion isn't worth the bits it is printed on) and suggest this: we're going to see a commodification fall-off in the number of cores everyone has too. Everyone is going to have two cores because it is cheap to put a second core on the main CPU if it lets you get rid of a whole array of special-purpose hardware. Give me multi-core and maybe I can get away with software-driven rendering (who needs hardware acceleration), software-driven sound (goodbye DSP chips), maybe I can even find cheaper ways to build my I/O. But 16 cores? The average user doesn't need 16 cores to check email and run Windows 7.

So as transistors continue to shrink and it becomes possible to pack 8 or 16 cores on a die, I expect some people to have this and others not to. We'll end up in the same situation as the graphics chips.

Summing It Up

To sum it up, sure there may be some drag on X-Plane in supporting a wider range of hardware. But it's an inevitable requirement, because hardware shifts in capability even during a single version run, and as hardware becomes faster, the gap between -end and cheap systems gets wider.

Glass vs. Glass (Translucent)

In Plane-Maker you will see that there are two "glass" lighting modes for instruments - one called "glass" and the other called "glass (translucent)".

What's the difference?  The difference is how they respond to their lighting rheostat being turned down.

• Glass becomes dim by mixing the RGB colors of the overlays toward black.
• Glass (tranlsucent) becomes dim by mixing the alpha channel of the overlays toward transparent.

Which one do you want?  It depends on what you are doing.  The rules I suggest are:

• Use "glass" if you are using the layer as a "mask" to block elements behind it.
• Use "glass (translucent) if you are using the layer as an "overlay" to draw on top of other elements.

A Few More Manipulators

I'm not sure if these will make it into X-Plane 9.30 (we're trying to close down features, but it doesn't do much good to hold off features that help people make airplanes) but...while I was in Italy I created a few more manipulator types.

The set of manipulators let you change the value of a dataref directly with a mouse-click - the various flavors control how the dataref is changed.

These manipulators are the natural corollary to the command manipulator, which runs a command on a click.

Why have both? Commands are good, but they don't cover 100% of sim functionality (just like datarefs don't cover 100% of sim functionality).  By having both, it will be possible to control switches and buttons that are best accessed by a dataref change.

For the basics on commands vs. datarefs, see here.

The generic instruments already write to both commands (via the trigger instrument) and datarefs (via the rotary instrument) - these new manipulators provide the same functionality for 3-d cockpits.

Multi-Threading Is a Weird Feature Request

Over and over, whether it is a feature request list for X-Plane or another simulator, I see the same thing: "multi-core support" or "multi-threading" as a feature request.

Now before I continue, I must remind everyone: X-Plane is already multi-threaded and will take advantage of multi-core hardware. How much we use those cores depends on the type of scenery loaded.

The problem is that multi-threading (as a way to use multi-core hardware) is a solution technique, not a problem statement.  What is threading going to be used for?  If I simply program the other 7 cores of your computer to calculate PI to 223,924 digits have I met the feature request?  This probably isn't what anyone wants.

Implicit in the request for multi-core is (I speculate) a request for better frame-rate.  (I did see one user who wanted multi-core to be used for a more accurate flight model.  This strikes me as a poor trade-off for hardware based on my understanding of the flightmodel - we would use a lot of hardware for only a marginal accuracy improvement - but I commend the user for stating the problem and not just a possible solution.) But is multi-threading the best way to get framerate?

If I had two patches to X-Plane, one that doubled fps by using two cores and one that doubled fps by using more efficient code, which would be better?  To me the obvious answer is: the code that is more efficient.  It will run on any hardware (not just multi-core) and if you have multi-core hardware, we still have that second core free for some other functionality.

So to me the feature request should be something like: "higher framerate - and yes I have multi-core hardware".  Or perhaps "more visual detail at the same framerate - and yes I have multi-core hardware".

All feature requests need to be in terms of problem statements, not possible solutions.  This lets us find the set of problems that can be solved together in a coherent manner, and it lets us pick a solution that meets our engineering goals.

Masks for Glass Instruments

X-Plane's panel system does not have true "masking" based on a bitmap.  You can clip an instrument to its rectangular region, but most masks are made by overlaying another layer or instrument on top of the moving parts.  Examples include the circular mask for the moving map and the outside of the artificial horizon dial.

If you are using ATTR_cockpit then what you see in the OBJ is just like what you see in the 2-d panel, and the masking problem is simple: pick a mask that matches your background.  In particular, if the back of the panel is black, your mask must be black; if the back of your panel has a gradient, the mask must contain a copy of a slice of the gradient.

But there are two cases where this rule does not work the way you might expect.

Masks for 2-d Spotlights

The 2-d spotlight textures (panel-2, panel-3, etc.) have a strange property: they light the background of the panel (the "burned in layer") per-pixel but the overlays are lit per vertex. This is a cheat to keep the frame-rate high.

Normally this is not a problem - the moving parts are small and look different enough from the background that the lighting mismatch is not visible.  But if you have a mask in the 2-d panel and spot lights, it will not match!

Unfortunately there is not much you can do about this.  The only thing you can do is to keep the spotlight color uniform over the entire mask region.

Masks for Cockpit Regions

When you use ATTR_cockpit_region, the lighting model for your 3-d cockpit changes: instead of drawing the panel (as you saw it in 2-d), X-Plane calculates the albedo (day-time) and emissive ("_LIT") components of the panel separately and then combines them with real 3-d lighting.

The good news is that the spot light problem is no longer a problem.  Since the spot lights are 3-d and are applied to these final "panel textures", a mask that matches the background will blend perfectly.

But in order to mask, you need to know which part of the panel texture you are masking (albedo or emissive!).  If you are masking the albedo texture (e.g. a mechanical artificial horizon), create a mask that looks just like the panel background.

But for a glass instrument, the moving parts go into the emissive layer.  Your mask must be pure black!  Where did I get that from?  The emissive layer adds light to the object.  Black is an absence of adding light.  So pure black 'erases' any light-only elements (which include all glass EFIS instruments, etc.).

One nice thing about this strategy: you can build a custom glass instrument (with a black mask) and put it over any background. This means you can reuse your art assets no matter how they are positioned on the panel.

I'm Back

After almost three weeks in Europe, I am now back home!

First, I would like to thank the entire team that put together the French X-Plane Congress - it was a wonderful event - you guys did an amazing job!  It was great to see so many X-Plane people in one place (including the elusive Marginal!). The French X-Plane community is a strong group and it shows.

I would also like to thank Cris for setting up an informal Italian X-Plane get-together (complete with the Zurich delegation :-) for our last night.  We didn't get much sleep, but it was great to see so many people again before we flew back home.

At this point my in-box looks like someone dropped a bomb on it...it'll take a while to dig everything out.  The first priority of business is X-Plane 930 - we'll do a series of betas over the next week or two that will hopefully put the release to bed. Once 930 is done, I can probably beta WED 1.1 and get back to work on scenery.

Per Pixel Lighting Isn't Free

I've had a little bit of time to look at X-Plane 930 performance. The data isn't 100% conclusive yet, but one performance issue sticks out like a sore thumb: per-pixel lighting hurts fps.

Now, part of this is that the per-pixel lighting shaders are not yet optimized (and perhaps are not terribly well written). I need to take some time to see if I can get some more performance out of them.

But...per-pixel lighting isn't free - when per-pixel lighting is on, the video card is simply doing a lot more work than it used to. Consider: a typical X-Plane scene might have 250,000 vertices on screen at once.  At a minimum, you have at least 750,000 pixels on screen*.  Make your window bigger and that number goes up - fast!  Turn on 16x FSAA and watch the pixel count get even larger.  So the number of lighting calculations done by your graphics card are at least 3x higher with per-pixel lighting and potentially 50x higher.  Even if your graphics card has a lot of power, that's going to cost a bit.

So one option I am considering is making per-pixel lighting a rendering option. This would allow users who want 922-level fps to simply turn it off. In my tests so far, turning off per-pixel lighting gets fps to within a few percent of 922.

(The only reason to have shaders on but per-pixel lighting off would be to have a cheap version of the reflective water. In the long term I want to limit the number of a la carte rendering settings, but for now it seems reasonable to support v9.00 base configurations through the entire version run.)

* In practice, not every pixel on screen requires full shading, e.g. the sky does not require complex shading.  But some parts of the screen may be shaded multiple times.  This is called "overdraw".  For example, with a runway we pay for our shaders twice - first with the ground underneath the runway, then with the runway itself.

Who Are You?

Today Lori and I head to France - we will do a bit of site-seeing before the conference, and then Austin and I will go to Italy to work with Sergio.

So first, my email connectivity will be dubious at best for almost an entire month - if you send me a bug report and hear nothing for several weeks, you're just going to have to wait...our internet connection in Italy doesn't really deserve the name "internet connection", and often doesn't work at all. (If you need tech support, contact info at x-plane dot com - you should not be trying to reach me for tech support in the first place!)

I fixed a number of beta bugs this week; Austin will probably cut beta 11 before the conference, and then there will be a pause while we're off the net.  Once we get back, hopefully we can kill 930 off.  (Most of the crash reports I've received turn out to be a single bug in the OBJ-handling code, fixed in beta 11, so hopefully the next beta will be stable.)

Now here is my request to everyone at the conference: please ... go easy on me if I cannot remember who you are.

The tricky thing about a conference like this is that I mostly know people in the X-Plane community by an email address; if I see a face it is only once every few years.  So if I jumble who you are, what you look like, and what you do with X-Plane, I apologize in advance.  It is not a reflection on the merits of what you do, but rather an indication of the disorganized state of my (soon-to-be-jet-lagged) brain.

X-Plane 930 Performance and Crashes

I have received a number of emails bringing up crashes and performance problems in the X-Plane 930 betas - some of the writers are concerned that 930 might be a lame patch, going final with crashes and lousy performance.

To assuage this concern, let me make a few comments on where we are in the beta process, the likely future schedule, and the problems themselves.

The Schedule

X-Plane 930 has been an absurdly long beta. Going into the beta I had the mindset that we should take the beta slowly to have time to discover driver bugs on a wide variety of hardware - why rush and miss something?

I think we took this too far. To run a "slow" beta we have run other development simultaneous to the beta, but that in turn has stretched the beta to epic lengths.

We are starting to try to clamp down and close out the beta now, but it is going to get interrupted again. Austin and I will be traveling to attend the X-Plane conference in France, and from there we will spend two weeks working with Sergio in Italy. Given how rarely we go to Europe, we cannot pass up the opportunity to work with Sergio in person - we have a few problems in the sim where getting the three of us in one room is the best course of action.

Unfortunately our internet connectivity during the trip will be limited, and we can only bring some of our equipment, so closing out the beta while on the road is really not an option. Thus there will be yet another beta delay. Hopefully when we return, we can close the beta out for good.

Performance Problems

I have seen a number of emails regarding framerate with 930. A few notes on framerate and betas:

I try to save framerate for last in a beta. Most performance problems have two possible causes.

1. We communicate with the video card driver in a way that is fast on our systems but astoundingly slow on other systems. We discover this from slow performance in a particular piece of the code on other hardware.
2. The new beta does something new that is more expensive than what the old build did, and users have not figured out how to (or do not have a way to) turn this more expensive option off.

The solution to case 1 is to use another driver call; the solution to case 2 is to make sure the rendering options provide a way to turn the feature off. (We simply cannot guarantee that a new, nicer looking feature run without a fps penalty - we can only give you a choice between better visuals and faster fps.)

Either way, framerate work tends to be the last thing on my beta list for this reason: other bug fixes may cause framerate problems, typically in category 1 - that is, a bug fixes makes use of a new driver call that we find out has hurt performance. Thus I try to do all performance fixes at the end of beta when we won't be adding new code.

This means that in practice, I have spent nearly zero time looking at performance. I am just starting that process this week, so it will be a little bit before I find problems.

Unfortunately often performance problems manifest only in the hardware I do not own - despite having a pile of computers in my office (a pile that seems to grow deeper and less manageable every year) there are just a ton of systems out there. So a lot of the performance bugs will get fixed by users trying experiments and reporting back to me - a slow process despite some of the really great efforts by our users.

Crashes

Crashes sometimes are manifestations of gross code defects, but often they fall into the category of driver problems too. I will be working to piece together the puzzle of strange behavior over the next few weeks; usually the solution is to not do some action that we thought was legal but fails in some hardware cases.

Don't Panic

As always, my final message regarding the beta is: don't panic. When it gets quiet over the next few weeks, it is because of travel, and even once Austin and I are back in the office, it will be slightly slow going to piece together problems on hardware other than our own.

Viva La France!

This year Austin, Sergio and I will be attending the X-Plane 09 Conference this year, near Paris. I am excited by the turn-out - a number of people in the X-Plane community who all do great work will be there, in some cases meeting each other in person for the first time. And there will be cheese!

The conference is officially French, and I must admit, I don't speak a word of the stuff. Fortunately, my beautiful and very patient wife will be there, and she speaks French quite well. So I have asked her to translate the rest of this blog post - a greeting to my French readers.

Bonjour. Ecoutez: Ben croit que je traduis le reste de son article sur ce blog, mais il ne parle pas du tout le français. Lorsqu'on était à Cannes, il a essayé d'apprendre à dire "Le chat est sur la chaise" - ce qui lui a fallu deux semaines! Donc il n'a aucune idée de ce que je suis en train d'écrire, et il ne saura pas non plus si traduis sa présentation correctement ou non.

Lorsqu'il commence sa présentation, je vous expliquerai toutes ses mauvaises habitudes. Est-ce que vous l'avez vu travailler? C'est tellement bizarre! D'habitude il ne porte pas de pantalon. Il s'asseoit devant l'ordinateur en buvant du café et en maudissant les "DSFs" - qu'est-ce que c'est? Je lui ai dit qu'il faut porter un pantalon à la conférence.

I am a very lucky man! I will see you all there.

I Lost My Objects

It seemed like opposite-day at Laminar Research...Austin saying that an error shouldn't quit the sim and me saying the error was never okay, ever. Well, I relented: with X-Plane 930 beta 8, if your scenery pack is missing objects, you can still fly.  Instead you get a single error message like this:

That is the "non-fatal" error dialog box - you will see it only once for each scenery pack with a problem for each time you run the sim.  It means that at least one thing is wrong with your scenery pack, but you need to look at Log.txt to see what's wrong. For example, you might see this in the Log.txt file:

Failed to find resource 'KSBD_example.obj' at 'Custom
Scenery/KSBD Demo Area/KSBD_example.obj'
Failed to find resource 'KSBD_example.obj' at 'Custom
Scenery/KSBD Demo Area/custom objects/KSBD_example.obj'
Failed to find resource 'KSBD_example.obj' at
'Resources/objects/KSBD_example.obj'
Failed to find resource 'KSBD_example.obj'
at 'Resources/KSBD_example.obj'
***Error with scenery file "Custom Scenery/KSBD
Demo Area/Earth nav data/+30-120/+34-118.dsf"
(/Volumes/RAID/code/design/HLutils/Files/io_dsf.cpp: 503.)
Unable to locate object: KSBD_example.obj

In this case, X-Plane couldn't find the object KSBD_example.obj - the sim is also listing all of the places it looked.  Note that only the first location is a good location - the other 3 are legacy search paths that date all the way back to version 6.  It is likely that in the next major version we will trim down our search paths significantly.

A few comments on this whole situation:

• Authors, do not ignore error messages like the dialog box above - every one of them indicates a condition serious enough that we think you should fix it. Non-fatal errors like these may crash future versions of the sim, or your content may simply stop working.

  If you file a bug against a future version of the sim saying your scenery pack used to work and is now broken, and we find that the old scenery pack had errors, we're not going to fix the bug - we're going to laugh maniacally and dance around you in a circle while singing "told you so".

  Okay - we're very unlikely to do that - but if you have errors in your scenery pack, you're doing something wrong and you need to fix it - treatment of illegal data is not stable between versions of the sim!!

• I was never very sympathetic to this whole bug report because X-Plane has never accepted a DSF with missing objects - this has been a fatal* error since X-Plane 8.0 when DSF was introduced. So I simply don't understand why there are any scenery packs floating around with objects missing.  Why would you place an object if you don't want to see it? The whole issue strikes me as a total failure to check quality by authors, since even running your pack once would reveal this kind of problem every time!

• The motivation to make missing objects illegal comes from version 7 and ENVs.  When looking at ENV scenery, I found that a large number of ENV scenery packs were missing at least some of their objects (an error that was silently ignored in ENV). It seemed like we were hiding an error and the result was authors not noticing simple mistakes that might "lose" an object (e.g. renaming an OBJ file).

  Hence the "harsh" policy for DSFs - it was in response to a real problem with existing scenery!

• You don't need to have missing objects just because you use library objects from another scenery pack (that might not be around).  Use the EXPORT_BACKUP command in your library and a single blank OBJ as a place-holder for the objects you want from a library that might be missing.  OpenSceneryX provides a stub library that authors can include so that their scenery will load without errors even if the OpenSceneryX library is not installed.
  

Anyway, Austin was right to make the error non-fatal.  Besides being a little bit nicer for users who don't know (or care) why their pack is gone) it lets authors get a list of all missing objects with only one run of the program.

* Fatal?  In computer terms, a fatal error is one that makes the program quit, e.g. the error is fatal because it kills the program.

A Modern Cessna

With X-Plane 930 beta 8, we finally integrated the latest version of Max's Cessna 172 SP. Grab the new beta and try it - he's added a number of new features that demonstrate some of the newer X-Plane 9 airplane features.

• Real 3-d lighting in the 3-d cockpit. Note how the map light tends to illuminate only some of the cockpit as it fades out.
• 2-d back-lighting on all of the major steam gauges.
• A bunch of parts can now be dragged in 3-d, including the door handles. This is done via manipulators.
  
• Walls! In 3-d cockpit viewer mode you won't be able to leave the airplane until you actually open the doors. The cockpit viewpoint is constrained.
• The model has the glass parts separated out for correct shadowing, and the glass works correctly from all viewpoints.
• Panel uses cockpit regions for accurate lighting.
  

The cirrus jet has been similarly updated. I plan to use the Cessna for a series of tutorials showing how to use these recent X-Plane features.

ATTR_light_level Changed!

I have said this before, but now it's finally true: new file specifications are subject to change in the middle of beta!

In particular ATTR_light_level has changed slightly from beta 7 to beta 8. If you are using this feature in your objects, you will need to update your objects.

A new ac3d beta will be posted later today that supports the updated syntax.

You can read about the syntax here.

So How Big of a Mesh Can You Build?

I get asked about the maximum mesh density in X-Plane a lot.

First, I must note that X-Plane's mesh is adaptive - the triangles are not arranged in a grid, but rather they're arranged to maximize the quality of the mesh with minimal triangle count, while preserving the outlines of water bodies and airports. These pictures show a dynamic mesh fit to a section of the grand canyon (both with and without the shaded DEM for reference):

Second, the actual maximum mesh density is almost certainly limited by system resource constraints. The maximum spatial resolution in a DSF is about 0.6 millionths of a foot. You're going to exceed the maximum number of vertices, or much more likely, the maximum amount of available memory in X-Plane long before you have a grid at that resolution.

(That doesn't meant that resolution isn't usable - remember, the idea of the adaptive mesh is to have very high densities in very improtant places, but not everywhere.)

But in practice this is all moot if you don't code your DSFs by hand - the question that really matters is: what is the highest density mesh that you can use with the current scenery tools.

Until recently, MeshTool was limited to a 1201x1201 (90 meter) input DEM. The input DEM is a simple grid, and the finished DEM is never higher resolution than the input. So until now, it would have been just about impossible to make a really high res mesh, even if X-Plane could handle it.

The most recent changes to MeshTool fix this - MeshTool can now handle a 10801 x 10801 (10 meter) input DEM, which X-Plane should be okay with as well.

This new version will hopefully go into beta relatively soon.

ATTR_light_level vs. Generic Instruments

Propsman caught something:

...is modifying the value of a batch of ATTR_light_level tris comparable [performance-wise] with toggling the state of a backlit generic instrument? Instinct tells me that you must have the latter more streamlined than the former, but maybe not?

He is right: in the current implementation, ATTR_light_level is probably a bit more expensive than using generic instruments. This may not be true in the future though.

• The generic instrument code is pretty tight.
• Right now ATTR_light_level sometimes has to adjust shaders, which can be expensive.
• In the future, ATTR_light_level has the potential to be very heavily optimized, while the generic instrument code will always be CPU based.

But to put it in perspective, all instrument drawing is slow compared to scenery drawing - in the scenery world we draw 50,000 triangles of identical OpenGL state in a row, and modern cards do that very, very well.  In the panel, we have to put in a lot of CPU time to figure out how to draw each quad or tri-strip.  Fortunately you probably don't have 50,000 individually programmed flashing lights in your panel.  Heck - there's "only" 3608 datarefs published by the sim.

Perhaps other questions are important when picking ATTR_light_level vs. panel texture:

• Which is more useful: to be able to have several variant images and variant images that are not "lights" (this is only possible by generics) or the ability to vary the light level gradually and not just have on or off (this is only possible with ATTR_light_level)?
• Which is simpler to author given the rest of the panel?

In other words, it's all pretty "slow", but fortunately "slow" isn't that slow.  If your light has to blink, you may want to pick what looks best and is straightforward to author.

Datarefs Vs. Commands IV: Duplication

In my previous posts I have tried to explain the difference between commands and datarefs, and when you might use each.  To review:

• A dataref represents information. You can always read it, and you might be able to change it.
• A command represents an action.  You can always invoke the action, but you can't tell if it worked without looking at a dataref.

So...why is there so much overlap and duplication?

Dataref Vs. Dataref

There is duplication in the datarefs because we don't delete old datarefs when we add newer, improved ones. The old datarefs stay in place to keep old plugins working. Here are a few reasons why we've added new datarefs:

• The cockpit2/ and flightmodel2/ sections were added as a new, simpler, easier to use interface for authors in version 9.  (Read more here and here and here.)
• In some cases, the old dataref was a bit-field while the new one is a simple integer. While plugins can use bitfields, modelers cannot animate using bit fields.
• In some cases, the old dataref did not represent a clean view of the data. Some old datarefs exposed X-Plane internal structures that are not appropriate for long-term use.

To see this in action, let's look at the autopilot.  How many ways are there to set the autopilot mode?

1. sim/cockpit/autopilot/heading_mode. This is the original heading mode, and it is marked deprecated, because it exposes a bunch of internal X-Plane autopilot values.
   
2. sim/cockpit/autopilot/autopilot_state. This is the ideal autopilot dataref for plugins. It provides all functions, but since it is a bit-field it is not useful for authors.
   
3. sim/cockpit2/autopilot/heading_mode. This is a clone of the original heading_mode into the cockpit2 domain. Honestly I am not sure how it got there - I know it was me who put it there, but it sure is a dumb idea; the original dataref is deprecated, so it was stupid of me to duplicate it!
   
4. sim/cockpit2/autopilot/heading_state. This is coming in 930 and provides a heading-state enum set appropriate for authors...basically an enum that matches the two heading bits of the autopilot_state dataref that programmers were using.
   

How do you sort through this? Three rules of thumb:

• Try to use sim/cockpit2 and sim/flightmodel2 when possible.
• More recent datarefs are usually better.
• Use the most useful dataref you can find.

Commands Vs. Commands

Sometimes there is some duplication of commands, e.g.

sim/engines/carb_heat_on                           Carb heat on.
sim/engines/carb_heat_off                          Carb heat off.
sim/engines/carb_heat_toggle                       Carb heat toggle.

Here it's a lot more obvious why there are multiple commands: they affect the carb heat in multiple ways. Typically this is done because commands are mapped to joysticks and other USB hardware; some hardware generates a button press when a command is toggled, but some hardware generates two commands, one for the off and one for the on position.

The rule of thumb is: use the command that gives you the action you want.

Commands Vs. Datarefs

Very often there will be a command and a writable dataref.  Typically we need them both:

• The command is needed to let users set up their joystick and keyboard.
• The dataref predates version 9 - writing it was the only way to invoke an action.

Newer datarefs are more likely to be read-only, as we put new "changing the sim" functionality into commands.  To go back to our autopilot example, we have on command: sim/autopilot/heading that lets us arm heading mode.  This command is probably preferable to any of the datarefs for changing the autopilot state.

My previous post discusses writing to a dataref. vs. actuating a command in more detail.

Datarefs Vs. Commands III: What Is My command Doing?

It turns out that understanding what a command is doing gets really complicated.

The key idea to untangling it all is this:

• Commands have duration, and that duration is the amount of time you "hold down" the button or key that actuates the command.
  
• Not all commands do things for the entire duration.

An example will clarify this:

• When you press and hold down the 'p' key to pause the sim, the sim pauses (or unpauses) instantly the moment you press the p key.  Holding the p key down for a long time does not change this.  Pause is a "momentary" command.
  
• You have to keep the starter button/key held down for a few seconds to start an engine. If you press it and release it, the starter motor will only run for a fraction of a second, which is not enough time to start an aircraft engine.  Engine start is a "duration" command.
  

Virtual Datarefs

A "virtual dataref" isn't really a dataref at all - it's a string you can enter into an OBJ or generic instrument that looks like a dataref, but actually is something else.  There is one "set" of virtual datarefs in X-Plane right now.

In X-Plane, you can enter

CMND=some/command/name

into a generic instrument or obj animation - this creates a "virtual dataref" around the command's "activation status".  The virtual dataref acts like a read-only integer-type dataref whose value is 0 if the command is not being pressed right now and 1 if it is.

For example, if you animate a push button using CMND=sim/starters/engage_starter_1 (key framing the animation to be "out" when the virtual dataref is 0 and 1 when it is in) then you will have a button that appears to be pressed whenever the starters are engaged.  This will happen no matter how the starter is engaged - your animation will happen whether the user presses a joystick button, holds down a key, clicks on a manipulator, or a plugin runs the command.

Basically, virtual datarefs that provide "activation status" of commands exist so that you can animate the buttons in your virtual cockpit to match what the user is doing with the mouse, keyboard, etc.  These datarefs are read-only; use trigger generic instruments and command manipulators to actually run the command.

For Programmers

In that last section I pointed out that a virtual dataref is not a real dataref at least 3 times. Why am I harping on this?  Well, programmers, you cannot use XPLMFindDataRef to access virtual datarefs yourself.  Sorry.

Virtual datarefs exist to give authors of OBJs and panels access to command activation status, something they would not normally have.

Plugin programmers don't need this - if you are programming a plugin you simply use XPLMRegisterCommandHandler and you will be told when the command is being actuated and released.  Using a command handler is a lot more efficient than reading a dataref because you will receive a call only when the command is pressed, instead of having to check the dataref value every frame.  If you need to monitor a lot of commands, the callbacks are a lot more efficient.

Command Activation Is Not The Same As System Activation

Let's go back to the case of the "pause" command and review.  Here's what we know about the pause command.

• The command sim/operation/pause_toggle will change the sim's pause state.  The instant this command is pressed, the sim will pause (if it is runnning) or unpause (if it is paused).
  
• Holding down the sim/operation/pause_toggle command has no effect beyond its initial press. Hold it down for an hour, it doesn't matter.
• The virtual dataref CMND=sim/operation/pause_toggle tells you if the pause command is being held down at any instant.

Here's what we know about the pause dataref.

• The dataref sim/time/paused is 1 if the sim is paused, 0 if it is not.
  
• This dataref cannot be written!
  

So here's what I mean when I say: command activation is not the same as system activation.

• CMND=sim/operation/pause_toggle tells you if the pause button is being held down.
  
• sim/time/paused tells you if the sim is actually paused or not.
  
• They are simply not the same!

This becomes important when you start to model the buttons in airplanes.  Take for example an autopilot button for altitude hold.  In many planes, the button can be pushed in, and the moment you do, the altitude hold "arm" light will turn on.  Keep the button held in and it doesn't have any more effect.  Altitude hold arm is a momentary command, but it has a system status indicator light.

I receive a number of questions about how to model this - and the answer is: take advantage of the fact that command activation (is the button pushed in) and system activation (is the light on) are not the same.

• You would use a read-only dataref for the autopilot state to turn on the indicator light.  (There are two ways to do this: use ATTR_light_level to turn a _LIT texture on and off, or use panel texture and map a generic instrument like a rotary or an annunciator.)
  
• You would use a command to make the button work.  Probably you'd use a command manipulator on the button mesh.
  
• You would key frame the button's animation based on the virtual dataref wrapped around the command you are using with the manipulator.

The result will be a button that lights up when the AP is armed but pushes in while it is being pressed (whether via the mouse in the 3-d cockpit or a joystick button press or a keyboard button press).

In my final post, I'll comment on how much overlap there is between datarefs and commands.

Datarefs Vs. Commands II: Which One Should I Use?

There is a lot of overlap between the datarefs and commands; very often there is both a dataref (telling information about some part of the sim) and a command (which takes action to change some part of the sim).  Which should you use?

Here are some general guidelines:

• If you want to set up a joystick or keyboard, you have to use a command. The joystick and keyboard configuration dialog box lets you associate actions with a keystroke or button press, not information!
  
• If you need to show the status of a system (E.g. "is the landing gear down") use a dataref. I will cover this issue in more detail in part 3, but basically only datarefs show you information.

The ambiguous case is whether to use a dataref write or a command to change a system when both exist.

• If there is a command that exactly does what you want to do, prefer the command over the dataref.  For example, it is better to arm the autopilot using the commands than the datarefs.  Changing the autopilot state often involves changing a lot of variables at once in complex ways.  When you issue the command, that work is done for you, correctly, every time.
  
• If the command is not really suitable for your purpose, use a dataref.  For example, to change the engine throttle position, do not use the command sim/engines/throttle_up to move it up "a little bit."  Use the dataref sim/cockpit2/engine/actuators/throttle_ratio to set the throttle to the precise position you want.  The throttle-up command exists so that users with no joystick or mouse wheel can fly with the keyboard by pressing the F1-F2 keys (bound to throttle-up, throttle-down).  It is not meant to precisely control the throttle position!

(I will discuss why there are so much overlap between commands and datarefs in part 4.)

DataRefs Vs. Commands I: What's The Difference

What is the difference between a dataref and a command? They serve different purposes in X-Plane, but it's easy to get them confused, especially because the names can look so similar. If you only take one thing away from this comparison, it should be:

• Datarefs are information.
  
• Commands are actions.

Datarefs

A dataref is a single bit of published information. For example, the user's indicated airspeed, as seen by the pilot, is a dataref, stored in:

sim/cockpit2/gauges/indicators/airspeed_kts_pilot

Datarefs have names that do not change. Datarefs made available by X-Plane start with sim/ while datarefs made available by plugins start with another prefix. Datarefs have been in X-Plane since the release of the plugin system in version 6.70.

You can always read a dataref, but sometimes you can change it. Trying to change a dataref usually has one of three actions:

• If the dataref is not writable at all, nothing happens.
  
• If the dataref is writable, it will change.
  
• Sometimes a dataref may be writable, but only after changing some other sim configuration. For example, you can only "write" to the control surface deflection datarefs after setting the control surface override dataref to 1. (If you don't set this override, X-Plane will constantly write its own ideas of the control surface positions to the control surface datarefs and your changes will be lost.)

You can read and write datarefs:

• With the DataRefEditor plugin (if you just need to find some information).
  
• Via generic instruments.
  
• Via OBJ8 animations and manipulators.
  
• Via a plugin via the XPLMDataAccess APIs.
  

Commands

A command is an action that the sim can take on your behalf. For example, the command

sim/autopilot/altitude_arm

arms the autopilot for altitude hold.

Like datarefs, commands have permanent names, starting with sim/ for X-Plane or other prefixes for plugins. Commands have been available in X-Plane since version 9.0.

You can always actuate a command, but there is no guarantee that it will do anything. For example, the engine starter command won't start the engine if the plane has electrical starters and the battery is dead.

You can use commands by:

• Binding them to your joystick and keyboard, in the joystick settings dialog box.
  
• Using a generic trigger instrument.
  
• Using a manipulator in an OBJ.
  
• Using the XPLMUtilties APIs - these are SDK 2.0 features.

Plugins

Plugins can add both new datarefs and new commands to the sim. Plugins can also change the behavior of all built-in sim commands, and can change the information in some datarefs.

Where Do I Find Datarefs And Commands

X-Plane's default commands and datarefs are listed in the text files Commands.txt and Datarefs.txt in the Resources/plugins folder. (Note: providing the command list is new to X-Plane 930.) The dataref list is also available on the X-Plane SDK Wiki.

Up next: when should I use a command and when should I use a dataref?

The X-Desktop and iPhone Are Not Zero-Sum Games

On the menu this morning: first a whiny rant, then a nerdy one.

Someone pointed me at this post. Before I go into my geeky diatribe about leveraging code, a quick note: it is true that at this point Austin is working heavily on the iPhone - probably more on the iPhone than the desktop. It is also true (but not mentioned) that Laminar is investing more man power into X-Plane for the desktop now than it ever has in the past. (With the iPhone we've grown our workload with a second "front" for products, but we've increased staffing a little bit too.)

It is also true that X-Plane 9 free updates have been less frequent. This doesn't mean there's less code going into them - it just means that we're doing 4-5 months of coding and 3 months of beta instead of 2 months of codindg and 1 month of beta. I'm not sure if this is better or why it is happening (each release has to be individually planned for the circumstances) but one observation:

Given how many video cards and drivers are out there and how they react differently to the X-Plane code, I wouldn't want a beta process less than 2-3 months, because I want to know that it's had time to run on a wide variety of hardware. If our beta has to be at least 2-3 months, then a 3-month release cycle would have us in beta all the time!

My whiny rant (the short version) is this: Laminar Research isn't in a position (as a company selling a product) to post all of the details of how our business operates internally. So posts that say "Laminar should do X" (where X is a business decision) drive me a bit nuts, because I can't post a reasonable reply.

Now here's my real point: development for the desktop and the iPhone are not a zero sum game. Clearly we leveraged the desktop sim to create the iPhone app. But it goes both ways; work on the iPhone also benefits the desktop.

Here is one example: when beta 8 comes out, Plane-Maker's "export to object" will a much more complete OBJ, in OBJ8 format, with animations already in place for things like wing controls surfaces!

That's a feature that people have been asking about for a while - both to make CSL objects* and as a way to save time when moving from a Plane-Maker drawn plane toa custom one. (You would first export an animated OBJ8, then start adding details in a 3-d modeling program. With beta 8, you won't have to rebuild your control surfaces from scratch.)

But...it's also a feature that is critical to the iPhone! The iPhone version of X-Plane uses animated OBJ8 files as well; this feature helped us internally to prepare planes for the iPhone version, but it's also a requested feature from our authors.

So my response to those who say that the iPhone has taken away from X-Plane development is that it is not true - not only because we are developing more heavily for the desktop, but because sometimes iPhone development is for the desktop.

* I do not remember whether the current compiled versions of libXplaneMP use OBJ7 or OBJ8 objects. Since libXplaneMP uses the OBJ code from XPTools, it should be (relatively) straight forward to update libXplaneMP and the clients that use it to use OBJ8 directly. In the meantime, you can use ObjConverter to convert an exported OBJ8 back to OBJ7 for CSL use.

Scenery Tools Progress? Yes.

I've been hacking at the scenery tools over the last few days...and I think we're making some progress.  I mention this because it can be hard to see the work getting done - there is nothing for months, then a new version of a tool.  (Well, now you can watch the code change as it happens.  If you don't think this like watching paint dry, you might be a geek!)  I also mention it because two of the biggest improvements aren't my own - they're insanely helpful user contributions.

The AC3D Plugin

Version 3.2 is now in beta.  Version 3.2 catches the plugin up with all of the latest OBJ tricks, including dataref-controlled LIT levels, manipulators, hard surfaces, etc.  This might be more useful for airplane modelers than scenery modelers, but I think the variable _LIT channels can be used for some interesting scenery effects too.

WorldEditor

WED 1.1 will go beta once X-Plane 930 is out of beta, and will feature (very primitive) overlay editing.  It's not a thing of beauty (yet) - the overlay editing features are leaner than the airport editing tools were.  But 1.1 will allow you to edit an overlay and airport in one workspace.

I can tell you now that the area where WED 1.1 is weak is preview.  But I don't want to hold off the ability to edit overlays in WED "at all" just because it's not as nice as it could be (and someday will be).

It should be possible to work on an overlay project in both WED and OverlayEditor - for example, you could build a taxiway layout in WED, using the overlay capabilities for custom pavement and lines, but use OverlayEditor to place 3-d objects (with a 3-d view).

MeshTool

To be blunt, MeshTool in its current form is really hard to use.  Part of that is expected - MeshTool is a "back-end" tool like DSF2Text, aimed to simplify the problem of creating base a base mesh for some other program.  But even as a back-end, MeshTool is difficult to use because it has bugs in its polygon processing, which often result in cryptic and hard-to-diagnose crashes when used with even relatively simple input data.

The good news is that I have my base mesh generation code running with a new polygon processing library (CGAL) - the switchover from my old buggy code to CGAL was done by Andrew and contributed back as a patch.  I've been throwing all sorts of weird data at the polygon cutting algorithms and they are rock solid.

The mesh generation code is part of MeshTool, so the next build of MeshTool will feature these processing improvements.  That's where Janos' extensive work on the build environment comes in - he's basically turned the scenery code from a big pile of my mad ramblings into a sane multi-platform project.  This will mean a much simpler, faster process for releasing tools, particularly on Windows and Linux.

(Previously once I got a tool working, I would have to waste a few days trying to get the code to work on Windows too, slowing the whole process down.)

I expect to recut the "scenery tool" pack (that is, MeshTool, DSF2Text, and all of those other smaller apps) some time during WED beta, after 930 goes final.  If all goes well, there should be native versions of the command-line tools for Linux.

MeshTool Needs a UI

Fixing MeshTool's polygon processing is necessary, but not at all sufficient to make it a useful tool; to be really useful, it needs a real user interface.  I think there are two strong possibilities for this:

1. WED could develop base mesh editing.  Under this scheme, you would be able to draw land use polygons in WED and create a "DEM" object for the elevation.
2. MeshTool could be an output option for QGIS (via some kind of plugin or script).

I don't know which it will be yet.  The advantage of WED is that an author could have all scenery elements in one place.   The advantage of QGIS is that it contains advanced GIS features that I will probably never put in WED.  It may be that QGIS's UI is simply not intuitive to less advanced users - I don't have good perspective on this.