Two Warnings About Normal Maps

Two warnings about normal maps:

1. Make sure that the RGB color underneath transparent sections does not turn black or white! Some image editing programs (in particular Photoshop) will lose the color beneath a transparent area.

   With a normal map, this is very bad - black and white are not legitimate normal map colors, and the result will be bogus normal vectors under the non-shiny part. Normal maps affect more than just specular shiny hilights - the normal map affects all lighting, so having black or white under your transparent (non-shiny) parts is bad news.

   To check whether this has happened, I recommend Graphics Converter, which will show you your alpha and RGB channels separately, exactly as they are in the file.

2. Make sure your RGB value are normalized. The "length" of the normal (as encoded in RGB) must come out to a distance of 1. This is virtually impossible to do using PhotoShop or an image-oriented program...I suggest you use a real plugin to PhotoShop or Blender to create normal maps that are correctly "normalized".

It is also very possible that X-Plane's gamma correction is distorting normal maps, but that's one for me to fix.

Plugin Broken? It's Temporary

I just found out from a user that Chase View Delux doesn't work with X-Plane 940 beta 4. Here's what's going on:

• Plugins often end up running X-Plane code - when you make a call to the X-Plane SDK, the SDK sometimes passes the call right to X-Plane.
• X-Plane 940 beta 4 has heavy error checking on all of the X-Plane code.

Put these together and...X-Plane 940 beta 4 is putting heavy checking on plugins that wasn't there before.

Typically the kinds of problems caught by the error-checking code inside a plugin are not critical, but aren't bad things for authors to fix. Since we will turn off heavy error checking as soon as we get the systems code fixed, plugins will start working again..there's no need for plugin authors to recut anything.

A Slow Motion Car Crash

I'm driving at 55 mph on the highway. I drive over a nail, lose a tire, skid off the road, crash through the guard rail, plunge off a cliff, and die. That's not much fun. But when you get to the accident site, you'll probably be able to piece together what happened. The skid marks, the nail, the hole in the guard rail, the car wreckage below - you can connect the dots.

Now...let's say I'm driving 500 mph. Same nail, same out of control crash. But this time it's going to be a lot harder to tell what happened. Lord knows where the nail ends up, the distance from the nail strike to exiting the highway is going to be a lot bigger, and the car is going to be in smaller pieces scattered over a wider distance. It's going to take a lot longer to piece together what went wrong.

That, in a nutshell, describes the motivation for an X-Plane beta with all of the debug and safety checks on. X-Plane's normal operation is like the car doing 500 mph - when it crashes and dies, there's very little left that can be used to figure out what went wrong. When there is a bug in the code that destabilizes the sim, finding it via crashes in release builds takes a lot of developer time and slows the whole beta down.

With the safety checks on, X-Plane still crashes when something goes wrong - but the bodies and wreckage are all a lot closer to the scene of the crime, and the evidence left around is in much better shape.

One of the side effects of the safety builds is that they catch "harmless" coding mistakes - (harmless in quotes - the bug might seem harmless but who knows if that will always be true). XSquawkBox now quits the sim with an ugly alert box because it reads off the end of a piece of the airplane data structure via the plugin system. This hasn't hurt things in the past, but it's not really correct. Beta 5 should fix the underlying problem, letting you run XSquawkBox again. (The fix will be in X-Plane, not XSquawkBox.)

A Tip For Updating Electrical Systems

X-Plane 940 has a new electrical systems model and it has a few important differences from 930:

1. You must specify exactly how many buses your plane has. 930 provided two buses but then did a bunch of cross-tying behind the scenes in case you didn't have enough power sources.
2. X-Plane 940 requires that each battery and generator be on exactly one bus.
3. X-Plane 940 will not allow systems to be powered by non-existent buses.

Now this can have some strange side effects. Consider the default king-air with two generators and one battery:

• In 930 it has two buses, battery feeds both buses, and each generator feeds one. You could have systems split by bus and they would work unless you lost one generator and the battery.
• 940 defaults this plane to one bus, because on battery power only one bus will be fed.
• This means that in 940 all of your systems will be reset to bus 1.

But wait...if you import into Plane-Maker, the import will trash your system bus selections before you can increase the number of buses to 2. What can you do?

Here's a work-around: before you update your plane, make sure you have two battery and two generator switches on your panel. Then open in 940. The import will set 2 buses and your systems will be preserved.

Of course, by the next few betas this may all be moot because we may get something less crazy in there.

Driver Thrash

I'm seeing a number of bug reports where weird artifacts are showing up in 940...missing pieces of runway, flickering triangles...all sorts of good stuff!

I believe that this is due to some kind of bug relating to threading, X-Plane and the video drivers. I won't say whose fault it is because I really don't know. I do know that the bug appears to not happen on OS X. (But this could simply be because the threads time out differently on OS X.)

The changes to the rendering engine for 940 from 930 are substantial and aggressive - it'll take us a little bit to fix these things.

When you wonder how come programs don't use all 8 of your cores yet, well...this is why...multi-core programming is complex, tricky, tedious to debug, and often involves substantial changes from the original code.

12 GB For the RenderFarm

The global scenery gets cut on "the renderfarm", which is our name for the cluster of computers, each loaded with all of the input data for the global scenery. These computers chug for a few days to churn out the whole planet.

With the next global render I am trying an experiment: using one 8-core Mac Pro as the RenderFarm. We've never had more than 8 processors in the farm at a time; in the past to get 5 processors might have taken 3 machines. The appeal of a single machine is ease of setup; no data to sync between machines, no sorting out which machine did which tile and merging it all back.

Today I upgraded the Mac Pro's memory (again) to 12 GB. I thought the logic of why I did this might be of interest to X-Plane users who are trying to figure out "should I have more memory"?

Basically there are three memory limits we care about:

1. The virtual memory limit per process - generally 3 GB per process for a 32-bit application. If an application wants more memory tan this, regardless of what you have, it is dead.
2. The virtual memory limit for the whole machine. Since the machine virtual memory limit is a function of hard disk space, normal users will never care about this - we can have a huge amount of virtual memory.
3. The physical memory actually used by the sum of all programs actually running. Once all programs need more physical memory than you have, they start using hard drive, and they get really, really, really slow.

In the case of the render farm, our processing program runs on one DSF tile at a time. But with 18,000+ tiles we can take advantage of more cores by processing 8 tiles (using 8 copies of the program running at once) on 8 cores.

This is where memory comes in. Before the upgrade my machine had 4 GB of memory, allowing each of the 8 tiles to use a little bit less than 512 MB of RAM before we ran out of physical memory and started paging. (The OS takes a little bit for itself.)

Normally this is all good -- a typical run might only use 300 MB of RAM. But every now and then we hit New York City or Boston and the RAM use spikes out to 1500 MB or more.

This is okay if just one process hits New York, but what if a bunch of processes hit a big city at once? We blow past the physical RAM of the machine and every tile becomes slow at once. And because they are all slow at once, they take a very long time (hours) to clear this state.

(This state of thrash due to many processes is like 8 people trying to go through a doorway at once. If they would just take turns, they'd be through in a second. But by forcing 8 at once they get stuck. The operating system won't pause a few of the high-memory programs to let the others complete.)

Since the RenderFarm has to run overnight unattended, I upgraded RAM to 12 GB, for 1500 MB per rpocess before thrash. My hope is that for this investment, we'll be able to run the processing through without a human to unjam it.

What About X-Plane

So would a RAM upgrade for X-Plane help? Well we can apply what we know from above to figure it out. Generally, you need enough RAM that X-Plane + all other programs won't run out of physical memory. Since X-Plane is 32-bit (and can only use 3 GB of RAM) you are likely to be fine with 4 GB when running X-Plane + the OS. Any more and X-Plane can't take advantage.

The exception might be if you need to run X-Plane and another big program like PhotoShop at the same time. At that point you might want enough RAM for both to run at the same time.

Lego Brick Code

When Sergio first proposed generic instruments, his model was "lego bricks". The idea was to provide a number of very basic parts for panel makers and let panel makers mix and match. The result would be huge flexibility for airplane authors without code bloat.

The problem with non-generic instruments is that there is such a huge variety of behavior among airplanes...if Plane-Maker were to have options for every possible plane, the "special equipment" screen would require 3000 tabs and be completely unusable. Hence the need for a smaller unit of modeling: the lego brick.

The prop disc is the first feature I have done that is meant to be used only by a plugin, e.g. "lego brick" code. The X-Plane systems code sometimes suffers from the same "code bloat" problem as the instruments: a ton of very specific, very tweaky behaviors that interact in strange ways and become very difficult to manage. It's not that the systems code is bad code - it's that the scope of the problem is simply too large. That is, you can't expect X-Plane to cleanly simulate the systems of every airplane ever in a ton of detail through an a la carte menu of check-boxes.

The idea of the prop disc is: someone (LR or otherwise) can write a plugin that encodes a certain style of prop disc. That plugin can then be picked up and moved around like a generic instrument between planes (perhaps with a corresponding text file to control it).

If someone else comes up with a different/better prop-disc algorithm, compatibility isn't an issue...that person writes a new prop disc plugin and the airplane author selects the one desired. Think of it as sort of a portable flight model that stays with your plane.

So we win in three ways:

• Anyone can write the prop disc algorithm, not just LR.
• The code lives with the plane, to avoid compatibility problems.
• More than one plugin can exist, giving authors an a la carte menu.

That's the theory at least.

Tweaking the Prop Disc in 940

X-Plane 940 allows plugins to customize the prop disc. Details here on the wiki.

I put these datarefs in so that modelers wouldn't have to try to model prop discs with OBJs. The problems with prop discs are many:

• They need to be billboarded, and X-Plane does not provide datarefs for manual billboarding inside an airplane (particularly not to the engine's coordinate system, which can be transformed by all sorts of fun stuff).
• They often need variable translucency, which OBJ does not have.
• They cause all sorts of depth buffer errors, which OBJs cannot manage.

In short, prop discs are weirdly special-cased enough that I thought it would be better to provide a general set of parameter datarefs for prop discs and let X-Plane do the drawing.

These options are not available in Plane-Maker. Why not? That'll be my next post.

940 Beta 2 - Recovert Airplanes

I have ranted in the past about the importance of not treating beta builds of X-Plane as having finalized file formats. Generally a beta should be used to explore, experiment, and test old content, but not to create new finished work or ship product. File formats sometimes change during beta to work around bugs we find.

Another reason not to depend on the file formats of new betas is that sometimes we screw up. In the case of 940 beta 1, the code that converts 930 airplanes forward to the new 940 electrical system is pretty buggy (hence the reports of electrical systems doing wonky things, panel instruments disappearing, and general weirdness).

This one is our bug to fix and will be fixed in beta 2 (at least we think). But to "get the fix" authors will need to open their 930 saved airplane in 940. If you already re-saved the airplane in 940 then the 930->940 conversion code won't be re-run.

I'll try to post some info on the new electrical system on the Wiki, but for now: if in beta 2 you have a bug with a plane that used to work in 930, send us the 930 version of the plane so that we can convert it and watch the conversion screw up. If the plane is already in 940 format we don't know what our conversion code broke and what you edited in Plane-Maker.

Normal Maps in X-Plane 940

X-Plane 940 now supports normal maps on OBJ models (both scenery and airplane). I'll get more formal docs up once the rest of my office is moved and unpacked but here's the details for now:

The normal maps are in "blender" format see here. The alpha channel is optional; if it is present, it serves to modulate the level* of specularity. Opaque means full specularity, transparent means none. You can use this feature to make some parts of an object shiny and some dull on a per-pixel level.

Shininess level is modulated by both ATTR_shiny_rat and the alpha channel, so you need ATTR_shiny_rat 1.0 and an opaque alpha channel (or no alpha channel) to see full specularity.

Normal maps are only available for objects and only appear if pixel shaders are on and per-pixel lighting is enabled.

Normal maps should be PNG format, not DDS - they will not be texture compressed because S3TC compression tends to kill them. (There are some modern formats for normal map compression supported by the newer cards but we don't use them yet.)

* Specular level: most serious 3-d programs let you control both the specular exponent, which controls how "tight" the specular hilights are, and the specular level, which controls how bright they are. X-Plane only lets you control specular level; if specular hilights exist, they are always as the maximum exponent for the sharpest specular hilights.

More Threads

X-Plane 940 beta 1 is out...it will take a little bit to get the release notes and docs on the website completely up-to-date. We're still dealing with loose ends from our migration to the new web site, and most of my office is packed up for a move to the Boston area. I'll try to get docs up as fast as I can given the chaos flying about.

Given the interest multi-core stirred up in previous notes, I will mention one change to 940: with this build we've added yet more multi-core to X-Plane.

In 931, X-Plane will use as many cores as you have to load textures, but only one to build "3-d scenery" (a loose category for the work we do when we make airport taxiways and lines, build forests, and extrude roads).

In 940, this "3-d scenery" is also done on as many cores as you have. This should speed up load times a bit, particularly under very heavy tree settings, and hopefully keep the forest engine running faster for users with more cores.

It also sets us up for long-term growth; X-Plane's visual quality is sometimes limited by the time to build 3-d meshes...being able to do this work on many cores means we can use higher quality algorithms.

Consider for example the roads: my original "road extruder" (the code that converts a vector road into a 3-d model, called an extruder because it builds a 3-d road from a cross section like one of those play-dough toys) made beautiful intersections with stop-lines and cross walks and lots of other great stuff.

It was also really slow. And at the time the sim wouldn't fly at all while roads were extruded, so speed was of primary concern. So I replaced it with the much dumber extruder you see today, where intersections are basically ignored.

Now that we have 3-d scenery build on multiple cores, we can begin to provide rendering options that take more CPU time but produce higher quality results. The trees and airport layouts already do this (in that they take more time and produce slower, more detailed, higher triangle count sceneery at higher rendering setting for the same input DSF ad apt.dat file). With more cores, we can continue this strategy with roads and other parts of the sim without worrying about overloading the one core that was doing this work.

Of course just because we can use 8 cores doesn't mean we do...you won't see 8 cores maxed out very often, particularly if you have simple scenery and a very fast machine.

I am the Spam King!

If I have not emailed you back, it's probably because I've been very busy trying to interleave X-Plane coding and packing up the house. But it is also possible that my email response bounced because my web server has ended up on a bunch of anti-spam "black-lists".

Black-listing seems like a good idea at first: we'll gather up a list of all of the IP addresses from which spam comes from, then publish them. Then your local mail server can use that list to filter spam - you never see it!

In practice it doesn't work so well...example: http://www.mipspace.com/. The IP address for my server (XSquawkBox) is now on this list. Why?

MIPSpace is a list of IP Addresses associated with known commercial marketing companies.

Since my server is used for my own personal email and to run the SDK website, I'm not sure why I am on the list. I have sent them an email to clear things, but in general I hit an anti-spam/black-list bounce somewhat frequently now, and frankly I don't have time to separately try to clear my name from every guilty-until-proven-innocent blacklist that pops up and screws up my email.

If I seem disproportionately grumpy about this, it could be due to one of two reasons:

1. Not replying to emails is generally bad customer service. (Okay, my in-box is backed up four months...that's bad.) I don't like the idea that a customer might perceive us (LR) as being unresponsive because some third party with no skin in the game decides to black-list us.

   The blacklist has no incentive to be accurate - it's not their lost customers if email doesn't go through.

2. I'm not at all convinced that this is going to cut down unsolicited commercial mail and/or spam.

   In the spam case, spammers can send from botnets - they have access to a huge number of ever-changing IPs. Unless we are prepared to blacklist the entire internet, the blacklists are going to pick up more and more false positives while spammers find ways to harvest fresh, untainted IP addresses. The whole IP-reputation strategy assumes that IPs are hard to change. In practice, IPs are very, very easy to change.

   Commercial mail is a lost cause too - even if I am being solicited for commercial mail I don't want, no program or automatic process is ever going to tell the difference between the confirmation of my invoice and a list of discounts from the same company. When it comes to commercial mail, the reputation damage has to be done to the company, not the IP.

   (The company does have reputation to risk - if we are known as a company that doesn't honnor a "do not subscribe" policy, then customers can choose to not buy our products.)

It could also be because I ran out of Viagra and don't have a diploma from a prestigious non-acredited university.