« State of the Video Game IndustryPredictions for Skyrim and UESP »

Hacking the ESM/ESP Files

Hacking the ESM/ESP Files

  09:55:49 am, by   , 472 words  
Viewed 46803 times since 03/10/11
Categories: Programming, UESP

If you obsessively read everything on UESP you may have noticed one or two cryptic posts like this and this lately. What are these ESM and ESP files, and why are Daveh, Nephele and I trying to pull them apart?

Take a look in your Oblivion program directory and you'll find a few different files. There's Oblivion.exe itself, which is the main game engine itself. In the data directory you'll find several ".bsa" files. This seems to stand for "Bethesda Softworks Archive" and each one is a compressed file containing the graphics textures, sounds, speech and meshes (the files that define what makes an image solid). You'll also find Oblivion.esm and, depending on how many plugins you use, several ".esp" files. These are Elder Scrolls Main and Plugin files, and contain information about NPCs, weapons, armor, quests, places and so on, as well as information about the landscape. In other words, these are the files that contain all the information we need for UESP.

The file format was created for Morrowind and evolved a little for Oblivion. It's also used in Fallout 3 and Fallout: New Vegas, and has changed a little more for those games too. At the moment, the assumption is that a similar format will be used for Skyrim. This is partly down to wishful thinking, but then again as the files are purely about data storage there's no reason to change a winning format to radically.

Each file consists of records arranged in a huge list for Morrowind, and into something a little more like a tree for the other games. A record is made up of a label ("ARMO", "NPC_", "CELL", etc) and several fields that provide the data. You can get more information about the format for Oblivion here, although it's quite technical and slightly out of date.

So why do we need to hack these files?

Simply, there are too many items to create them all by hand with any degree of accuracy. That means we have to get our bots to do it, but NepheleBot and RoBoT need data to work with, and the only realistic way of doing that is by pulling it directly out of the game files. Using the Construction Set would reintroduce a manual component that would lead to mistakes, and looking at the files and getting information by eye doesn't always work because some of it's compressed and it would be a really, really awful job anyway.

To give you some idea of scale, for Oblivion and its official plugins, there are 1,277,347 records with a whopping 4,682,059 fields. Try sorting through that lot by hand.

Come 11 November, while you're enjoying playing the game for the first time, spare a thought for those of us who will be digging into its guts to bring you the best information we can!



Comment from: DKong27 [Member]

Good luck with that guys! If I end up getting Skyrim for PC, I would get the CS and everything so I could possibly help with any verifications and such. I hope it is similar to the way Oblivion is set. It was hard enough to figure that stuff out… I still can’t get the hang of worldspace editing. :’(
Oblivion was the first game I ever modded for so I can’t compare it to say, Morrowind. But again, I’m sure you guys will get it all worked out. :D

03/10/11 @ 12:10 pm
Comment from: Kalis Agea [Member]
Kalis Agea

Like DKong said, good luck! I may also help with verifications and, further on down the road, actually work with the files themselves. I became pretty experienced with the OB and MW CS after a few months or so of modding, and even though SR has a whole new engine it should be just as easy to get the hang of. But, as always, my main priority when Skyrim comes out will be Lore. :)

21/10/11 @ 10:03 am

Form is loading...