What's the big deal with the NES format?

[FitzRoy]

I've never taken a huge interest in the NES, but I'm interested in this. I really need someone to just explain it to me in layman's terms. I know what a header is and I know what it means for the SNES. So if you can explain to me using the SNES as a comparison, I'm sure I would understand better.

Basically, why are NES memory mappers so much worse of a problem compared to SNES. What makes a generic memory map in NES emulators impossible?

Is a database feasible?

What is UNIF vs iNES? Which is the better solution?

What did old emulators do that is now being corrected?

[Deathlike2]

I've never taken a huge interest in the NES, but I'm interested in this. I really need someone to just explain it to me in layman's terms. I know what a header is and I know what it means for the SNES. So if you can explain to me using the SNES as a comparison, I'm sure I would understand better.

A header is just information. It obviously is not used for the functionality of any emu (at least for the SNES anyways).

Basically, why are NES memory mappers so much worse of a problem compared to SNES. What makes a generic memory map in NES emulators impossible?

I believe there was a std created when the SNES was out, hence you can use a generic memory map for hirom/lorom, etc. The same could not be said for the NES. I believe special memory mappers were made due to limitations.. I forget what they were, but you can google on the subject. The NES simply didn't have it easy.

http://en.wikipedia.org/wiki/Multi-Memory_Controller

Is a database feasible?

Maybe, but there's a ton of a mappers out there.

What is UNIF vs iNES? Which is the better solution?

Dunno, it seems like iNES is the way to go, not that I care too much. I'm sure some NES author could comment on it.

What did old emulators do that is now being corrected?

Hacks, and mapper specific stuff. I think it was Nesticle or some really fast NES emus that hacked certain roms simply to get it working on the emu (and clearly that was the wrong way to go). Nowdays, I don't think that happens anymore, but it's just figuring out how each damned mapper maps. There's a whole NES dev community out there still trying to figure this stuff out.

[FitzRoy]

Okay, so the memory map information for a cart is lost when you dump the cart. That is, it's not contained in the original data. When we talked about this for the SNES, Nach wanted to put this information in the header of the rom as opposed to a database bundled with an emulator. When I download some NES roms, I see separate UNF files bundled with the roms sometimes. Are these like separate headers filled with memory map information? Does the current NES rom format even have a header? I would assume that it wouldn't because there were never any copiers for the system.

[creaothceann]

http://nesdev.parodius.com/bbs/ might have some answers (see NESemdev).

[Deathlike2]

Okay, so the memory map information for a cart is lost when you dump the cart. That is, it's not contained in the original data. When we talked about this for the SNES, Nach wanted to put this information in the header of the rom as opposed to a database bundled with an emulator. When I download some NES roms, I see separate UNF files bundled with the roms sometimes. Are these like separate headers. Does the current NES rom format even have a header? I would assume that it wouldn't because there were never any copiers for the system.

For SNES, it's not needed. Every proper memory map should work for every rom correctly right out of the box. The same should go for the NES roms as well. You could put what you need to do as part of the mem map into a database, but not per game. I don't have any idea what the UNF stuff is handled or dealt with, but it seems confusing at this point. Obviously no game had headers (headers are copier specific stuff).

Something worth quoting:

Supported cartridge formats are .nes and .unif. To get beyond the limitation of the .nes format Nestopia maintains an internal database that will attempt to fix incorrect headers as well as provide necessary info for certain cartridge types. No changes are made to the physical file. This feature can be disabled in the preferences dialog.

Edit: Mednafen's docs - http://mednafen.com/documentation/

Mednafen supports the iNES, FDS(raw and with a header), UNIF, and NSF file formats. FDS ROM images in the iNES format are not supported; it would be silly to do so and storing them in that format is nonsensical.

[FitzRoy]

I think I'm beginning to get the big picture. Thank you. I've been working a lot with DAT files lately, and have been trying to decipher the big problem with the NES. The biggest limitation to DAT files is that they can't ignore added header data and simply calculate the CRC32 of the original data. So in order to verify all your roms with the DAT, they absolutely must be headerless.

Normally this isn't a problem for most people as headers don't serve a purpose other than for copiers. However, when a standard like iNES gets created, suddenly the header becomes important towards getting the game to work in emulators. So what happened was that datters were forced to index for the NES using the CRC32 of the iNES headered game file. Despite the header containing integral cart information that doesn't come with the ROM data, the manner in which the data is added to the rom is inherently subjective (and prone to revisions and rom tool dependencies). So I'm a tad disappointed at the prospect of indexing sizes and CRC32s of a rom with stuff added to it, but it's not that big of a deal.

A database doesn't seem feasible for the NES like it did for the SNES. If I understand it correctly now, bsnes only has to worry about something like a hacked "Ys III" rom that doesn't match up with the original's CRC32/MD5. But for the NES, every hack or public domain rom would need an entry in order to work properly. That's just not practical.

As you can see, I've resorted to making definite statements, so if I've got something wrong still, let me know.

[xamenus]

What is UNIF vs iNES? Which is the better solution?

UNIF was an attempt to replace the iNES format. Unfortunately, it never really picked up, as many emulators aren't compatible with it (especially the older ones, which some people still use). I think the format itself is still incomplete as well.

Recently, the people at Nesdev updated the specifications of the iNES format. This new spec has less limitations, and is compatible with the older NES emulators. So at the moment, the new iNES format is a better choice than UNIF.

The thing is, the only likely way the new iNES format will become widely standardized is if Cowering adds support for it in GoodNES. Fortunately, at least Nestopia supports it.

[byuu]

Games are all dumped the same way. The data on the ROM chips are dumped to a file. That's all the information we have on them. Copiers have their own issues (over/under dumps, their own custom header info, etc).

Now the problem is, neither the NES nor SNES has any idea about the concept of a cartridge, other than the connector port. When you read/write to a cart region of the memory map, it requests that data from the cartridge bus, and now it's up to the cartridge to return the correct data. It does that via memory mappers. In the SNES, most carts use the MAD-1, or compatible memory mappers (LSnnn series), which just say "ok address 808000 should go to ROM chip 0, offset 0", and forward the requests to those chips. However, this isn't always the case. There's also a lot more than just ROM chips out there. There's DSPs, RAM, and completely specialized chips that can also be mapped into the address space: it's totally up to the cartridge.

This process was pretty well standardized with the SNES, so HiROM and LoROM memory maps work for most games. But there are plenty of variants. Most of the time, you don't notice those variants because the games don't access those less common addresses. With the NES, not so much. There's about ~150 known memory mappers that all have different attributes. Some mappers even do weird stuff like control custom IRQ handlers. The mappers simply aren't nearly as compatible as they were with the SNES.

Now, the end problem is that we've lost all data about the actual cart PCB layouts except what is stored inside the ROM cart headers. This info, at least for the SNES, is extremely generic and limited (read: useless). The real SNES doesn't even pay attention to it.

For that, we need a header to tell us about the cart. iNES' header format was too limited and you couldn't store all the data you needed to distinguish some of the finer details of carts. UNIF's header format was too complex, and required a bunch of weird interleaving and variable-width fields that it never really caught on. Now everyone's trying to extend iNES, to support the rest of the cartridges out there. I'm rooting for that. I support the idea of small, fixed-width headers at the top of ROMs, because they're easy to ignore for external tools (eg, ROM patchers).

A database, whether for NES or SNES, has the same limitations. Just those limitations are more obvious on the NES for the above reasons. But unfortunately, here's the main issue:
- the new NES header format is still being worked on
- the new SNES header format doesn't even exist

So, in the mean time, you can either include a database and make it possible to play games such as Ys III, or you can just leave those games broken, and have inaccurate memory maps for everything else. I chose the former for now. I suppose if I had 50+ bugs in my buglist, one more wouldn't hurt and it would be a non-issue. But as it stands, if I remove bsnes' database, my completely broken game list will roughly quadruple in size.
As soon as we have a standard SNES header format that includes the PCB info I need, I'll be more than happy to remove bsnes' database. So far, we haven't even come up with a rough draft of such a header format, yet.
Even when the new format catches on, once the database is removed, all games in the old format will stop working again. Not a good end-user experience, but hopefully since all the SNES devs are in a tight-knit community, we can come up with something drastic to help get the new format out there as soon as possible (eg all emulators display a big warning when loading the old file format, or refuse to play them all together). I certainly couldn't get away with that one on my own. ZSNES and SNES9x will absolutely have to jump on board.

[PHoNyMiKe]

nes mappers are extra hardware in the cartridge. the nes can only address 32KB? of data on a cartridge. most games are bigger than this so the mapper 'bankswitches' to the other data. so when you first plug in the cart, the nes sees the first 32KB? of graphics and code. then the game will actually write to the cartridge, which tells the cart to bankswitch to another part of the rom chip. now the nes still only sees 32KB? but it's different data than before. only problem is the hardware to do this can be made in any way.

it's much like a porno website. imagine having a 15" 800x600 screen for your comp. you can only look at say 1 nice picture at a time. to see more you have to scroll down. this is in the old days. skip ahead to the snes era, it's like having a 60" 800x7,200 vertical lcd where you can look at all the pictures at once, without scrolling. mappers do this scrolling.

[byuu]

The S-DD1 performs bankswitching as well (though it's only used for Star Ocean), though you are correct that the vast, vast majority of NES games do bankswitch and virtually no SNES games bankswitch. It's not because SNES games wouldn't benefit from more ROM, it's that the costs of more ROM at the time were far too prohibitive. Lots of games could have had full vocal audio and fullscreen graphical illustrations / short movies if we weren't so limited for space.

[PHoNyMiKe]

GBA flash carts larger than 32MB use bank switching as well. the GBA can only see 32MB at one time in it's memory map. so when you have say a 128MB flash chip in the cart, you need to bankswitch to different sections of the chip. to do this in old visoly carts, you'd write a few 16bit values to a certain location on the cart (yes, write to ROM as far as the gba knows.) but the 'mapper' would detect these writes and interpret them as a bankswitch command (that's how they designed the cart.) this way you could set the starting point of the gba memory space to the location of any game on the cart.

sorta like setting a porn website as your homepage. you can set any page you want and it will come up just fine when you open your browser. bankswitching allows you to view porn on your computer!

[whicker]

phonymike is the Master Bates of techno-sexual analogies.