Cobweb Duster

Posted: 04 Oct 2022

So one of the big, big issues with modding Psychonauts right now is that if we ever get custom level tools working (the pieces are there, we just don’t have an editor yet) there’s some BIG problems with what assets you’ll need to actually ship with those custom levels in a mod.

See the game’s final build is kind of built with the idea in mind that every level will be loaded from a pack file, and that pack file will include everything that level needs. All the textures, meshes, etc.

So besides certain common assets (like Raz’s assets, for example) everything each level needs is packed into that respective level’s pack file. Results in some asset duplication but it cuts down load times a lot (which I hear Microsoft was quite serious about for certfication)

Unfortunately, for modders this means that if you’re making your own custom level and you want to use existing assets (which you will, unless you’re planning on making everything in your level 100% custom), you’d also have to ship those vanilla assets with your mod.

Lots of issues with that. Remember, this is an issue with not just models and textures, but even scripts, which define important entities. Besides that…

• First, it’s a pain to keep track of which vanilla assets you’re using, extract them, ship them with your mod.
• Second, this means that mods too will have big issues with asset duplication. A ton of mods will be shipping the exact same default textures and that’ll get in the way and clash with custom stuff and lead to load order headaches galore. We have the tech to do better.
• Third, it is of course pretty questionable to be shipping the game’s assets with mods like that. I mean, no-one probably really minds, but it’s preferable to have a better option that does not involve shipping chunks of the game like that.

How do we fix this? Well the most obvious option might be to try and add some way for the game to dig through every pack file to find any files that are missing. This is… probably doable, but it’d be slow and awful. I’m not gonna do that.

So, option B. We have a first-time setup process that will unpack all the game’s assets to a folder and then add some code hooks in Astralathe to allow the game to search from that folder no matter what level is loaded. Make every asset in the game globally accessible.

The game is perfectly capable of loading unpacked assets for debugging, it’s a functionality that I already heavily leverage and mangle to make Astralathe do its thing in the first place. It’ll just need a tiny bit of nudging to work.

So, here’s the plan. On first run, the launcher will prompt the user on if they want to run this first time setup. This is optional because while modding is Astralathe’s focus, it does include fixes to issues with the game, and some people might only want those fixes.

(You might wonder why Cobweb Duster is a seperate thing and the setup isn’t done by the launcher or Astralathe - simple, I wanted to utilise PsychoPortal to handle the assets and PsychoPortal is C# while Astralathe is written in C++)

For models and scripts, they can be unpacked as-is and the game will handle them. Textures we’ll get to later but for now let’s assume that everything is unpacked and ready.

On launch, Astralathe applies an incredibly dirty little hack. See the game does have its own mounting and load order system, it mounts “TestResource” (doesn’t exist in the final build), then the final game’s “WorkResource”.

Astralathe hooks the function responsible and injects this gross but functional code to make the game mount native_assets at the bottom of the tree. If the game is trying to find a file, then the game’s final stop will be checking if it exists in native_assets.

And now, whenever a mod needs to use a native file that’s not already loaded from a pack file, the game will be able to search native_assets for it, find it and then load it! This method is working for scripts and models, but I’ve not been able to test textures yet.

At least, that’s how it all works in theory. Adding this first time setup step is not 100% ideal for sure, but I think it’s the best option available. It’s fairly simple too, for end users this is just a matter of confirming it and then waiting for it to finish.

I’ve not yet got a fully functional implementation working. Models and scripts actually are working, but textures are the hard part.

You see, while the game stores scripts as compiled Lua which it can load loose, models in a native format (PLB) which it can also load loose and it stores textures in a native format (which might be called TX1? Or just TX? Or nothing at all?) that it cannot load loose.

Now, the format the game uses for texture isn’t too complicated. The actual pixel data is just DDS image data. But it’s wrapped in a custom format that specifies the format of the pixel data, as well as contaning animation info and frames.

Fortunately that means that converting it back to DDS is generally pretty simple, just a bit tedious. And a really good thing is that the game actually can loose load animated textures too, as long as it’s accompanied by an ATX file (which is just a text file).

So that’s the plan. First time user launches Astralathe. Astralathe prompts them if they want to run the setup. If they say yes, CobwebDuster starts up. It extracts the scripts and models, and converts the textures. Astralathe then applies a small patch to bring it all together.

Converting the texture is simple on paper but as mentioned before is a bit of a pain. While Psychonauts’ texture format just stores a number that tells the engine what format the data is in, DDS files are a bit more of a pain and that format flag needs to be translated.

Which means that this single number which tells the engine the format of an image data becomes something like this. A little bit annoying when you’ve got to write it all out but not the biggest deal.

After hitting some major confusion about why specific textures weren’t serialising properly, I finally got textures to extract back to useful formats! All that’s left is to see if the game will accept these.

If you’re curious about what the problem was, the issue was that I was setting the mipmap count of the DDS based on that of the ingame texture. This is bad because in Psychonauts, “0 mips” means “as many as possible”. In DDSes (if I recall) it means there’s only the top level.

Interestingly, it seems the only textures that are affected (i.e the only ones with mips set to 0) are the level lightmap textures and a single figment in Meat Circus. Quite peculiar.

A quick trip ingame aaaand…. it’s working! I can summon Oatmeal into the camp reception without needing any supplemental mods to ensure his assets are loaded! The game can find them fine!

I will note that my handling of animated textures is just a bit weird. See the game just stores all the frames and the relevant data, and not the original filenames of those frames, so some improvisation is needed which results in… silliness. Image

Thankfully this isn’t an issue in any way. The ATX references the texture’s frames by name so as long as the ATX references these slightly messed up names correctly the game will get along fine. I tested to make sure this works by turning Oatmeal into some kind of fire elemental.

So, that’s all great, and this is all well and good, but this does have downsides and it wouldn’t be right if I didn’t talk about them.

There is of course the obvious downside of requiring the user to go through this extra work. I have done what I can to make it as painless as possible, but it still takes a bit of work and stuff could theoretically go wrong.

Not to mention, this is a .NET Core program which means people will need the runtime for it to run. I think a lot of people should have it installed already, maybe without even knowing, but it is still a factor and I’m still wondering what the right course is to deal with that.

It takes some time to extract so the user will be left waiting for a bit. And it’ll need just over 400MB of free space which is… not really a big issue and a lot less than I thought it’d need!

Load times will be affected. There’s a reason the final game is built the way it is and we are actively tearing that down here. Modern PCs shouldn’t have much of an issue though, I just hope the game can deal with it, which brings me to…

Stability! I still don’t know how stable Astralathe is in its current state, and now I’m adding more to the pile of duct tape. Again, we are twisting and warping the game’s code to work in ways it was not meant to. Who knows how much it can withstand here?

There’s a reason that every Astralathe release past a certain one has been marked BETA. I’m still trying to evaluate how stable things are which is difficult when the userbase consists mainly of… me. And maybe one or two others experimenting with it here and there.

I’ve encountered a handful of random crashes and I’m not sure what’s causing them because they are hard to reproduce. The last one I saw was in code that shouldn’t have been touched by Astralathe, so I have no idea what’s going on.

The base game is normally super stable for me but it does still crash here and there without Astralathe! So who knows, maybe the crashes I’ve seen are regular crashes that’d happen anyway?

When Double Fine was watching a speedrunner show off stuff back in 2015, there was a part in the video where the game crashed after Raz died in the Basic Braining log tunnel. That was pretty funny. To this day I wonder what caused it. I particularly love the joking shouts of “Debug, debug!”

Getting off topic there but yeah, point is my code is awful, I am doing fairly awful things to the code of the game, and I don’t know how liable that is to all come crashing down.

Worst part is kind of that unless I am there on site to attach a debugger at the crash, I have no way of digging into it unless it’s 100% reproducible. I can’t figure out if there’s a way I can patch the game to create crash dumps when it fails. I’ve tried.

Well, regardless, after I figure out the .NET runtime issue I’ll be packing this up as a new release - Astralathe v1.5-BETA1. Uh, Astralathe v1.4 ain’t getting a non-beta release. I’d call this release v1.4-BETA4 but by now I’ve made too many changes to justify it staying v1.4.

I’m hoping that some day Astralathe V2.0 will release with the main change being a major code overhaul to clean everything up, but it’s very optimistic to assume I would clean up my code.

Unfortunately turns out that the texture I was testing animations with was already loaded in this level, they don’t actually work, which means I have more work to do to figure out where in the chain the problem is.

Is the problem with the extraction process or is it an issue with the way the game is trying to load it? That’s the big question here. If it’s an extraction issue it means that I need to fix the extractor. If it’s a game issue it means patching the game code.

A quick glance at the code and it may be an issue with where the game is looking for the file?

When the game is told to load a texture, after doing some safety checks, it will look for the requested file but change the extension to “ATX”. If the ATX exists, the game knows it is an animated texture and will use a different code path.

The thing I’m suspicious of that check in the middle - it’s checking a configuration option called “DVD Environment” and changing an argument passed to EFileManager::Exists accordingly

Said argument is the search location - the game has various different places it can look for files and this parameter lets you specify exactly where you want the game to look.

Trouble is, I’m not certain what the search locations mean. In the original code this was probably an enum so the location would have names, we don’t get that in a decompile so we have to figure it out ourselves.

In the switch statement used later down the line that changes functionality based on this parameter, 1 and 3 (and 2!) correspond to the same chunk of code, and truth be told… it’s fairly mangled in the decompile, it’s gonna take some effort to untangle.

I’m gonna take a shortcut here - Astralathe already hook EFileManager::Exists. It was used for something at some point I think, but then it was no longer needed. For a while it was used just for debugging but then that was disabled too and it just remained.

It can be useful again here though, it can help properly narrow down if this is where the issue is occurring.

Yeah, there’s a reason I completely disabled this debug output. Even locking it behind a config option to enable verbose output, it’s very spammy. It does demonstrate the game’s process well though - you see it checking for ATX files, seeing they don’t exist, etc.

Oh, and logging to the console actually slows the game down too. It’s blocking, so the game hitches with all this output.

Anyway, we spawn in our modified Oatmeal and observe what happens. I’m using the texture of Milla’s ball return launchers because… it’s the first one I found. Sure enough, the game can’t find the ATX at location 3, then searches location 1 for the DDS and can’t find that either

Not finding the DDS makes sense - it really doesn’t exist. But if we make a DDS file with that name, it does actually find it fine.

Now here’s something notable - the game will always search location 1 for DDSes, and it can find them in native_assets. But with “DVD environment” enabled, it’ll search at location 3 for ATX files and fail to find them. I wonder…

I do what I’ve been doing best thus far - apply an awful hack! And it works.

I’m sure there’s a better way to do this, but here’s what I got. I check if the game’s search for the file failed. If it did, I check if it was looking in location 3.

If it was, then I step in and make it try again in location 1 instead. If that works, then I go ahead and… nudge the game to make it start using location 1 to get that file instead. (All this logging will be removed)

It seems that searching the specific location for this file is only relevant to the game when checking if it exists. If you nudge the game to say “No, really, you’ll be able to reach this file if you try” then it all works out.

This… is probably not the best solution to this problem? I can’t really imagine any side effects that might come of it right now at least. I could force DVD environment to be disabled but I’m worried that would have side effects.

Does make me curious about what the locations actually are. I’m a bit too tried to try and untangle the big chunk of code used to handle locations 1, 2 and 3. Maybe later.

But hey this should mean that this feature is actually working properly for real this time. It’s not a pretty solution but hey it’s a miracle that we’re able to stitch this stuff onto the game to begin with.