If you obsessively read everything on UESP you may have noticed one or two cryptic posts like this and this lately. What are these ESM and ESP files, and why are Daveh, Nephele and I trying to pull them apart?
Take a look in your Oblivion program directory and you’ll find a few different files. There’s Oblivion.exe itself, which is the main game engine itself. In the data directory you’ll find several “.bsa” files. This seems to stand for “Bethesda Softworks Archive” and each one is a compressed file containing the graphics textures, sounds, speech and meshes (the files that define what makes an image solid). You’ll also find Oblivion.esm and, depending on how many plugins you use, several “.esp” files. These are Elder Scrolls Main and Plugin files, and contain information about NPCs, weapons, armor, quests, places and so on, as well as information about the landscape. In other words, these are the files that contain all the information we need for UESP.
The file format was created for Morrowind and evolved a little for Oblivion. It’s also used in Fallout 3 and Fallout: New Vegas, and has changed a little more for those games too. At the moment, the assumption is that a similar format will be used for Skyrim. This is partly down to wishful thinking, but then again as the files are purely about data storage there’s no reason to change a winning format to radically.
Each file consists of records arranged in a huge list for Morrowind, and into something a little more like a tree for the other games. A record is made up of a label (“ARMO”, “NPC_”, “CELL”, etc) and several fields that provide the data. You can get more information about the format for Oblivion here, although it’s quite technical and slightly out of date.
So why do we need to hack these files?
Simply, there are too many items to create them all by hand with any degree of accuracy. That means we have to get our bots to do it, but NepheleBot and RoBoT need data to work with, and the only realistic way of doing that is by pulling it directly out of the game files. Using the Construction Set would reintroduce a manual component that would lead to mistakes, and looking at the files and getting information by eye doesn’t always work because some of it’s compressed and it would be a really, really awful job anyway.
To give you some idea of scale, for Oblivion and its official plugins, there are 1,277,347 records with a whopping 4,682,059 fields. Try sorting through that lot by hand.
Come 11 November, while you’re enjoying playing the game for the first time, spare a thought for those of us who will be digging into its guts to bring you the best information we can!
